Best way to store DB config in Node.Js / Express app - javascript

What would be the best way to store DB config (username, password) in an open source app that runs on node.js / Express? Two specific questions:
Shall I put it into a separate config.js file in /lib folder, for example, and never include it into the master repository that is publicly available on GitHub?
To inlcude the config, is it as simple as require('./config.js') from the file that needs it or is there a better way of doing it?
PS sorry if the questions seem a bit simple or not so well formulated, but I'm just starting :)

Here's how I do it:
Create a config.js which contains objects representing your configs:
var config = {
development: {
//url to be used in link generation
url: 'http://my.site.com',
//mongodb connection settings
database: {
host: '127.0.0.1',
port: '27017',
db: 'site_dev'
},
//server details
server: {
host: '127.0.0.1',
port: '3422'
}
},
production: {
//url to be used in link generation
url: 'http://my.site.com',
//mongodb connection settings
database: {
host: '127.0.0.1',
port: '27017',
db: 'site'
},
//server details
server: {
host: '127.0.0.1',
port: '3421'
}
}
};
module.exports = config;
Then in my index.js (or wherever really),
var env = process.env.NODE_ENV || 'development';
var config = require('./config')[env];
Then process with that object, e.g.
var server = express();
server.listen(config.server.port);
...

For running toy apps where I need to hide db credentials, I use the dotenv module.
Place your sensitive info in a .env file (which is .gitignored), place require('dotenv').config(); in your app; dotenv creates entries in process.env that you can refer to.
.env file:
DATABASE_PASSWORD=mypw
DATABASE_NAME=some_db
To refer to the values:
process.env.DATABASE_PASSWORD

Not sure whether this is the best practice, but personally I have a config.json file where I store my db connection information. Then I do the following:
// options.js
var fs = require('fs'),
configPath = './config.json';
var parsed = JSON.parse(fs.readFileSync(configPath, 'UTF-8'));
exports.storageConfig= parsed;
Then from a different file I do the following:
var options = require('./options');
var loginData = {
host: options.storageConfig.HOST,
user: options.storageConfig.user,
password: options.storageConfig.password
};

I do put in args. just like the port of so many node.js example.
you most likely forever, pm2, nodemon to run your app. so this variable is not check in as part of your source code. and they are globally available too.
process.env.PORT
process.env.DATABASE_USER
process.env.DATABASE_PASSWORD
PORT=3000 DATABASE_HOST=localhost DATABASE_USER=admin DATABASE_PASSWORD=mypassword node app.js
export PORT=3000
export DATABASE_HOST=localhost
export DATABASE_PORT=27017
export DATABASE_USER=admin
export DATABASE_PASSWORD=mypassword
node app.js
var server = app.listen(process.env.PORT, function() {
});
var mongoClient = new MongoClient(new Server(process.env.DATABASE_HOST, process.env.DATABASE_PORT));

To inlcude the config, is it as simple as require('./config.js') from the file that needs it or is there a better way of doing it?
This is the right way to store config files.
The best approach would be to write your entire application like an ordinary node.js module, and write a small start-up file that calls it. This idea also allow you to use different database drivers using dependency injection.
Good, but not perfect solution is the environment. It is shared among all application, so if you have certain data you want to be available to all of them, this is the best bet. But if you have a config for one particular app, not much so.
PS: And please, don't use JSON for this. It's the worst idea possible. :)

I found this a nice way to handle my config, considering different environments:
config.coffee
exports.setEnvironment = (env) ->
switch env
when "development"
exports.DEBUG_LOG = true
exports.DB_PORT = '27017'
# ...
when "testing"
exports.DEBUG_ERROR = true
exports.DEBUG_CLIENT = true
# ...
when "production"
exports.DEBUG_LOG = false
# ...
else console.log "environment #{env} not found"
server.coffee:
config = require('./config')
config.setEnvironment env

Using environment variables
You can use export to set environment variables in OSX and Linux. The following is an example of setting a value in the SESSION_SECRET key.
export SESSION_SECRET="keyboard cat"
In Windows, you can use set.
set SESSION_SECRET="keyboard cat"
You can also set environment variables each time you run them.
SESSION_SECRET="keyboard cat" node secret-env.js
Use process.env of node.js to access environmental variables within code.
var express = require('express')
var session = require('express-session')
var app = express()
app.use(session({secret: process.env.SESSION_SECRET}))
Request a argument from the command-line
The best way to protect confidential information is not to store it in a setup file.
If the command-line requests configuration information as an argument using the noopt package, the secret information does not need to exist as a file.
The following is an example of requesting a session key as an argument using the noopt package.
var nopt = require("nopt")
var longOpts = {
"sessionSecret": String,
}
var shortOpts = {
"s": ["--sessionSecret"],
}
var parsed = nopt(longOpts, shortOpts, process.argv, 2)
console.log("session secret is:", parsed.sessionSecret)
node secret-arg.js --sessionSecret "keyboard cat"
node secret-arg.js -s "keyboard cat"
Advantages : It is safer to expose confidential information than to hardcoding or having it as a configuration file.
Disadvantages : There is a hassle of increasing the amount of information to be entered each time the app is launched.
If you try to create and solve a script, the problem that the password still exists in the script remains.

Related

NodeJS HTTPS server serving over HTTP instead of HTTPS when running inside Docker container

I'm creating a website using NextJS and Docker so that I can easily deploy it. I used npx-create-next-app to initialize it and used this Dockerfile (slightly modified) to containerize it. Since I wanted to use SSL with my server without going through the hassle of setting up a proxy, I followed this article, and setup the custom server.
This worked fine when I ran it outside of a docker container, and performed as expected, serving over HTTPS. However when I containerized it, and tried to open the webpage over HTTPS, I came up with SSL_ERROR_RX_RECORD_TOO_LONG, but I could open the page using just HTTP (which I could not do when running outside of a container). Some googling led me to this question, from which I concluded that when running outside of a docker container, the custom server runs the server over HTTPS, as expected, however when I containerize it, it starts running HTTP, even though no code has been changed.
I'd expect the behavior to be the same when running locally or containerized.
At first I assumed this was due to invalid key and cert values in httpsOptions however I wasn't able to find anything that would make them invalid, and I don't see how that would cause this strange behavior. I tried changing the Docker run environment from node:alpine-16 to just node:latest to see if it had something to do with the parent image, but that was fruitless.
One other minor issue I had is that console.log does not seem to output to the container's log for some reason, I tried googling this but didn't find much of anything pertaining to it. This has made debugging much harder as I can't really output any debug data. The only log I get when running inside of a container is Listening on port 3000 url: http://localhost:3000, which I assume is output by some library/package as it isn't anywhere in my code.
Here is my custom server code in case it would be helpful:
const https = require('https');
const fs = require('fs');
const { parse } = require('url');
const next = require('next');
const dev = process.env.NODE_ENV !== 'production';
const hostname = "127.0.0.1";
const port = process.env.PORT || 3000
const app = next({ dev, hostname, port })
const handle = app.getRequestHandler()
const httpsOptions = {
key: fs.readFileSync('./cert/privkey.pem'),
cert: fs.readFileSync('./cert/fullchain.pem')
};
app.prepare().then(() => {
https.createServer(httpsOptions, async (req, res) => { // When running on docker this creates an HTTP server instead of HTTPS
const parsedUrl = parse(req.url, true)
const { pathname, query } = parsedUrl
await handle(req, res, parsedUrl)
}).listen(port, (err) => {
if(err) throw err
console.log(`Ready on https://localhost:${port}`)
})
})
Link to a reproducible example here.
The thing is, based on your sample repo, that your server.js file that is in the root of your repo gets overwritten in the image because of this line in the Dockerfile:
COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
So the actual server.js that is running in the container is the server.js that is created by the yarn build command and it looks like this (you can exec into the container and see it for yourself):
const NextServer = require('next/dist/server/next-server').default
const http = require('http')
const path = require('path')
process.env.NODE_ENV = 'production'
process.chdir(__dirname)
// Make sure commands gracefully respect termination signals (e.g. from Docker)
// Allow the graceful termination to be manually configurable
if (!process.env.NEXT_MANUAL_SIG_HANDLE) {
process.on('SIGTERM', () => process.exit(0))
process.on('SIGINT', () => process.exit(0))
}
let handler
const server = http.createServer(async (req, res) => {
try {
await handler(req, res)
} catch (err) {
console.error(err);
res.statusCode = 500
res.end('internal server error')
}
})
const currentPort = parseInt(process.env.PORT, 10) || 3000
server.listen(currentPort, (err) => {
if (err) {
console.error("Failed to start server", err)
process.exit(1)
}
const nextServer = new NextServer({
hostname: 'localhost',
port: currentPort,
dir: path.join(__dirname),
dev: false,
customServer: false,
conf: {"env":{},"webpack":null,"webpackDevMiddleware":null,"eslint":{"ignoreDuringBuilds":false},"typescript":{"ignoreBuildErrors":false,"tsconfigPath":"tsconfig.json"},"distDir":"./.next","cleanDistDir":true,"assetPrefix":"","configOrigin":"next.config.js","useFileSystemPublicRoutes":true,"generateEtags":true,"pageExtensions":["tsx","ts","jsx","js"],"target":"server","poweredByHeader":true,"compress":true,"analyticsId":"","images":{"deviceSizes":[640,750,828,1080,1200,1920,2048,3840],"imageSizes":[16,32,48,64,96,128,256,384],"path":"/_next/image","loader":"default","loaderFile":"","domains":[],"disableStaticImages":false,"minimumCacheTTL":60,"formats":["image/webp"],"dangerouslyAllowSVG":false,"contentSecurityPolicy":"script-src 'none'; frame-src 'none'; sandbox;","remotePatterns":[],"unoptimized":false},"devIndicators":{"buildActivity":true,"buildActivityPosition":"bottom-right"},"onDemandEntries":{"maxInactiveAge":15000,"pagesBufferLength":2},"amp":{"canonicalBase":""},"basePath":"","sassOptions":{},"trailingSlash":false,"i18n":{"locales":["en"],"defaultLocale":"en"},"productionBrowserSourceMaps":false,"optimizeFonts":true,"excludeDefaultMomentLocales":true,"serverRuntimeConfig":{},"publicRuntimeConfig":{},"reactStrictMode":true,"httpAgentOptions":{"keepAlive":true},"outputFileTracing":true,"staticPageGenerationTimeout":60,"swcMinify":true,"output":"standalone","experimental":{"middlewarePrefetch":"flexible","optimisticClientCache":true,"manualClientBasePath":false,"legacyBrowsers":false,"newNextLinkBehavior":true,"cpus":7,"sharedPool":true,"profiling":false,"isrFlushToDisk":true,"workerThreads":false,"pageEnv":false,"optimizeCss":false,"nextScriptWorkers":false,"scrollRestoration":false,"externalDir":false,"disableOptimizedLoading":false,"gzipSize":true,"swcFileReading":true,"craCompat":false,"esmExternals":true,"appDir":false,"isrMemoryCacheSize":52428800,"fullySpecified":false,"outputFileTracingRoot":"","swcTraceProfiling":false,"forceSwcTransforms":false,"largePageDataBytes":128000,"enableUndici":false,"adjustFontFallbacks":false,"adjustFontFallbacksWithSizeAdjust":false,"trustHostHeader":false},"configFileName":"next.config.js"},
})
handler = nextServer.getRequestHandler()
console.log(
'Listening on port',
currentPort,
'url: http://localhost:' + currentPort
)
})
And as you see it starts a http server not a https. Also this is why the console.log("lksdfjls"); in your own server.js will not get executed.
What I would suggest is to leave node as it is, running on http://localhost:3000 and set up a reverse proxy that would forward incoming requests to this node backend that is accessible only from the reverse proxy. And of course reverse proxy would handle TLS termination. A docker compose setup would be more convenient for this so you could put the reverse proxy container (nginx for example) in the compose project too and map a directory from the docker host where your cert files are stored into the reverse proxy container at runtime - DO NOT BAKE CERTS OR ANY OTHER SECRETS INTO ANY IMAGE, not even if it is an internally used image only because it could leak out accidentally any time.
Also you could just manually run the two container with docker run but compose would make life easier it has a lot of capabilities for example you could scale compose services up and down so your backend service would run not in one but many containers. But if this would be a high load and/or business critical production stuff then you are better off with a better (real) container orchestrator like kubernetes, docker swarm, nomad etc but today as I see it the de facto container orchestrator is kubernetes.

export in js is not working

It is a very small issue but I can't find out: I have 2 file in the same folder: config.js and server.js. The last one uses the config.js with the following code:
...
var config = ('./config');
...
//Configuration
var port = process.env.PORT || 8080;
mongoose.connect(config.database);
By running the server I have problem with connection due to a wrong address, infact the following error is shown
MongoError: failed to connect to server [undefined:27017] on first connect ...
The config.js contains just:
module.exports = {
'secret' : 'test123',
'database' : 'mongodb://127.0.0.1:27017/test'
};
And If I replace the mongoose.connect with the following code
mongoose.connect('mongodb://127.0.0.1:27017/test');
everything works fine... Why it can't resolve the database name?
You must to use var config = require(./config.js) and config.js must be on the same dir if you do that, the problem is you forgot require

How to use socket.io browserify?

How do I use socket.io with browserify?
When socket.io is installed in a node app, it creates this url path for ths socket.io script:
/socket.io/socket.io.js
But what would be the real path of that file (relative to the node_modules dir) which needs to be passed to browserify?
Is it e.g.
socket.io-client/socket.io.js
or maybe
socket.io/lib/client.js
In the documentation of socket.io-client it says "Socket.IO is compatible with browserify." But they don't say how.
If you struggled to get it working with browserify as a window global use this code for the integration:
var io = require('socket.io-client');
window.io = io;
Here's a minimal client:
// client.js
var socket = require('socket.io-client')();
socket.on('message', ...);
Which you can Browserify:
browserify client.js > bundle.js
The path will be exactly the same since it's the server who serve the socket.io client library (and I guess you're not browserifying the server, are you?).
But I use a more convenient solution: check this out.
io = require 'socket.io-client'
class Network
constructor: (game, refresh_infos) ->
#sock = io()
#...
pending: (name, cb) ->
#name = name
#sock.emit 'pending', name: name
#sock.on 'new_game', (data) => cb data
Abrakadabra!
Import client and then assign it to variable with:
var client = require('socket.io/lib/client');
You can then call client.

How to share configuration variables in Nodejs application

I'm new to JavaScript/Nodejs. How can I share my configuration across the Nodejs application. For example: I have a config/config.coffee
path = require("path")
module.exports = {
development:
db: 'mongodb://localhost/hello'
root: rootPath = path.normalize(__dirname + '/..')
}
I included config.coffee in my app.coffee.
express = require("express")
# Load configurations
env = process.env.NODE_ENV || 'development'
config = require("./config/config")[env]
require('./config/boot')
app = express()
Now I want to include config variable into my config/boot.coffee. How can I do it? I don't want to re-include config/config.coffee into config/boot.coffee. Here is the my config/boot.coffee file:
env = process.env.NODE_ENV || 'development'
config = require("./config")[env]
fs = require("fs")
mongo = require("mongoose")
# Bootstrap db connections
mongo.connect config.db
# Bootstrap models
models_path = config.root+"/app/models"
fs.readdirSync(models_path).forEach( (file)->
require(models_path + '/' + file) if ~file.indexOf('.coffee')
)
# Bootstrap services
services_path = config.root+"/app/services"
fs.readdirSync(services_path).forEach( (file)->
require(models_path + '/' + file) if ~file.indexOf('_service.coffee')
)
Sorry for bad English :(
You might want to check out nconf, which helps you keep a kind of "waterfall" approach to application configuration, which allows you to mix your configuration from different sources very transparently.
You can see nconf in action in this project I wrote, unbox, which is basically boilerplate I use for applications I write on Node. You can check out how configuration is loaded here.
You could use something like grunt-pemcrypt for increased security by checking in the secure, encrypted file, and saving the encryption key somewhere safe.
12factor also has a nice approach to application configuration you might want to look into.
I believe NodeJS caches your require's, so calling require('config') again won't cause any performance degradation.
http://nodejs.org/api/globals.html#globals_require

flatiron.js / union - where to put app.use() configuration?

Taking the simple example from Union, I am wondering where I can put configuration code that usually goes in app.configure, like passport.js:
app.configure(function() {
// Initialize Passport! Also use passport.session() middleware, to support
// persistent login sessions (recommended).
app.use(passport.initialize());
app.use(passport.session());
});
Any ideas? server and router don't accept use().
Union appears to use the before collection for this:
var server = union.createServer({
before: [
connect.session({ secret: 'keyboard cat' }), // for `passport.session()`
passport.initialize(),
passport.session(),
// etc.
]
});
From the "API" documentation:
#option before {Array}
The `before` value is an array of middlewares, which are used to route and serve incoming
requests. For instance, in the example, `favicon` is a middleware which handles requests
for `/favicon.ico`.
Union supports connect middlewares via the before property, as previously mentioned by others. However, union does not handle application configuration; flatiron does. The api, however, is significantly different from express.
For example, configuring an application may look something like this:
var path = require('path'),
flatiron = require('flatiron'),
app = flatiron.app,
plugins = flatiron.plugins,
connect = require('connect'), // most connect middlewares work with flatiron ootb
passport = require('passport');
// Use flatiron's http plugin (not the same as a middleware!)
app.use(plugins.http);
// configuration consists of key/value pairs, not of function blocks associated with
// certain "environments".
// Here's *a* way you can handle environment-based configs; there are others!
app.config.file(path.resolve(
__dirname,
'config',
(process.env.NODE_ENV || 'config') + '.json'
));
// Use our config to set the secret
app.http.before.push(connect.session({
secret: app.config.get('secret') || 'keyboard cat' //default
}))
app.http.before.push(passport.initialize());
app.http.before.push(passport.session());
I haven't tried running this example (I'm sure there are more details here) but hopefully this gives you an idea.
I just built a wrapper to integrate Passport.js with Flatiron.js.
https://npmjs.org/package/flatiron-passport
https://github.com/travist/flatiron-passport
Please read the README.md on how to use it and apply it to your application.
I have tested it on LocalStrategy, but it should work for other strategies.
Please let me know otherwise.

Categories

Resources