I would like to know how to add milliseconds to every logs printed by the Nestjs Logger.
And if possible, is there a way to bind the Nestjs Logger to each request/response like Fastify do with Pino ?
Thanks in advance !
Related
I have a Express backend behind AWS Cloudfront. How properly set trust proxy for AWS Cloud Front:
app.set('trust proxy', function (ip) {
if ( ???????????? ) return true; // trusted IPs
else return false;
});
AWS Cloudfront use tons of ip address and is insecure validate all AWS ip address because anyone with an AWS EC2 instance have a valid IP.
The Problem
As you mentioned AWS CloudFront uses a long list of IP Address ranges. It's mentioned in their documenation. You can see them via this one liner (source, requires jq which you can get from brew in MacOs.):
curl 'https://ip-ranges.amazonaws.com/ip-ranges.json' | jq -r '.prefixes[] | select(.service=="CLOUDFRONT") | .ip_prefix'
(Update: or directly from http://d7uri8nf7uskq.cloudfront.net/tools/list-cloudfront-ips as mentioned in their doc.)
Right now, April 2021, it is giving me 122 ranges.
The Solution
You can make an AJAX call to this file in Node, parse the JSON file, get the list as an array of string (cloudFrontIps), and pass that to app.set('trust proxy', ['loopback', ...cloudFrontIps]).
Good news!
The good news is someone else has already done it! Check https://github.com/nhammond101/cloudfront-ip-ranges out.
Final Notes
It's obvious, but worth mentioning that getting this list in asynchronous! So, you might want to delay (e.g. await) your app start until this list is available. It's not a must though -- calling app.set after the HTTP server is up should work, thought for that short duration you will be recording CloudFront's IP.
You might want to call this file and get the new list periodically. The package is suggesting every 12 hours, using setTimeout.
My understanding is calling app.set on a running server will make the new list applicable on future calls immediately, without needing to restart. I am getting this impression by how X-Forward-For is examined on every request, and how app.set is calling compileTrust function on it's invocation. So, TL;DR: You shouldn't be needing to restart the server every 12 hours for this!
I look at express's code and it seems like app.set overrides (and not appends) the list every time you call it. So if you have some IPs of your own (e.g. your VPC's CIDR in AWS ELB), you have to manually add it to the list every time you call this app.set in your setTimeout.
Is there any way to terminate the EC-2 instance after a particular time like 2 hours from the time of creation.
I am using NodeJS to perform the operations on AWS EC-2.
Is there any parameter I have to pass when I create the instance or I need to create a separate function for this method like some built-in method for auto-terminating the EC-2 instance.
Thanks in advance!
You can achieve this using AWS Lambda anonymous functions. You can have Lambda create your EC2 instance, and then have it delete it 2 hours later through use of a 'step function'. More info here: AWS Step Functions
The easiest method would be to have the instance terminate itself.
Upon startup, it can run a simple script like:
sleep 7200
sudo shutdown -h
To have the instance terminate on the sudo shutdown command, set the Shutdown Behavior (when the instance is launched) to Terminate.
This script could be triggered from a User Data script when the instance is started. Run the script in the background, such as:
./shutdown_script &
The benefit of doing it this way is that the instance is responsible for turning itself off. Any other option would require a timer to be set somewhere (eg Amazon CloudWatch Events), which starts to get more tricky.
I am using memcached as a backing store for an ExpressJS session - chosen via an app configuration setting. I would like to fall back from memcached to memory if the memcached host cannot be contacted. (This isn't necessarily a production strategy as memcached is very reliable - it's more for when I forget to boot the Docker instance in dev, but could still be a production fail-safe.)
I first thought I could "app.use" a new session instance, and try to remove the first one, but I have read that it's difficult (if possible) to "un-use" Express middleware i.e. to swap the middleware in-place in the chain, or generally tinker with the chain once it's been setup.
The problem there is after the connection timeout period, the app middleware has been setup with many further services installed after the session middleware.
My second thought was can I re-configure the Express Session instance itself and change the store after it's been created? I could not see any way in the documentation.
My third idea was to wrap the express-session in a new "swappable store" class, but I'm wary of the scope of wrapping the entire interface.
For example in my app setup:
app.use(services.session.middleware());
// then a lot of other middleware...
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({extended: false}));
app.use(cookieParser());
app.use(express.static(path.join(rootPath, 'public')));
...etc
And in the session service, which selects and configures the session instance:
function middleware() {
// default / Memory Store options
const opts = {
resave: false,
saveUninitialized: false,
...etc
}
};
// Install any configured backing store
if(config.session.storage === "memcached"){
const MemcachedStore = require('connect-memcached')(session);
opts.proxy = 'true';
opts.store = new MemcachedStore({
hosts: ...,
secret: ...
});
const errorHandler = (type, details)=> {
/* HERE I WOULD LIKE TO RE-MAKE THE SESSION USING MEMORY
* AND DISCARD THE MEMCACHED ONE
* (THIS HAPPENS AFTER APP BOOT HAS FINISHED)
*/
console.error( String.format("Memcached {3} with host:{0} details:{1}.", details.server, details.messages.join( '' ), type));
}
opts.store.client.on('failure', details => errorHandler('failure', details));
opts.store.client.on('issue', details => errorHandler('issue', details));
}
return session(opts);
}
I think there are several approaches you can try for this. The first is to configure a storage system based on which configuration you have supplied on startup (probably via a module like config or nconf). The second is to run a quick check when booting up the app to make sure it can access the memcache service, and if it can't then fallback to memory with an error.
I would be fairly weary of doing either of these, since you're using docker and it should be easy to boot memcache. This is because you'll be introducing code which might trigger in production should there be some connection issue, and then you might find yourself accidentally serving sessions out of memory rather than something like memcache potentially without realising.
I'll expand on both strategies here and provide a third possibly better option.
1. Choose the cache system based on a config
This should be fairly straight forward, simply extract your configuration into some sort of config manager / environment variables (checkout config or nconf). When starting the application and connecting your session middleware, you can pull out all the possibly configurations, see which exist and attach one based on that. This is similar to how your if (config.session.storage === 'memcache") looks at the moment. Just use a fallback of not configuring one and the express-session middleware will fall back to memory. This way you can leave out the configuration completely and just always use memory for development.
2. Run a test before connecting to the desired service
In combination with the above, if memcache details are provided you could run a quick test by attempting to store something in memcache on startup. Perhaps new Date(); to signal when the application booted up? If this throws an error, then just don't attach the MemcachedStore to the express-session options and you can safely destroy the MemcachedStore.
3. Throw an error if you cannot connect to Memcached
This is in further combination to #2. If you identify that memcache configurations are provided, then I would personally do a check to see if you can contact the serivce and if not then throw an error and stop the application. This would mean that in development you immedietely know the issue, and in production you would as well and can trigger automatic alerts for yourself based on the fact that the application failed to start.
This is probably the best and most robust solution, generally doing a silent fallback is not a great idea when talking about connected services as things can go wrong and you have no idea. I appreciate that this is for development purposes and you need to be pragmatic, but if it saves you accidentally serving all sessions from your servers memory then this would be super beneficial.
We're using the karma-pact plugin to run our pact JS client tests, based on the example from https://github.com/pact-foundation/pact-js/blob/master/karma/mocha/client-spec.js .
In the example there's a timeout in the before(), I believe to ensure the mock service has started before running the tests (see comment "required for slower Travis CI builds").
I'm reluctant to set a fixed timeout in our tests as it'll either be too short or too long in different environments (e.g. CI vs local) and so I was looking for a way to check if the server has started.
I'd tried using the pact API https://github.com/pact-foundation/pact-node#check-if-a-mock-server-is-running , however this appears to start a new mock server which conflicts with the one started by the karma-pact plugin (an Error: kill ESRCH error is reported when trying to run pact.createServer().running from within a test).
Is there a way to determine if the mock server has started up e.g. by waiting for a URL to become available? Possibly there's a way to get a reference the mock server started by the karma-pact plugin in order to use the pact-node API?
Actually the simplest way is to wait for the port to be in use.
Karma Pact by default will start the Mock on port 1234 (and you can specify your own). Once the port is up, the service is running and you can proceed.
For example, you could use something like wait-for-host to detect the running mock service:
var waitForPort = require('wait-for-port');
waitForPort('localhost', 1234, function(err) {
if (err) throw new Error(err);
// ... Mock Service is up - now we can run the tests
});
I have a node service which uses AMQP to perform a remote procedure call, so it publishes a messages to a given amqp exchange and expects a response on a given amqp queue with the results of that call.
I am looking for a good unit test strategy which mocks the amqp connection can send a message based on a given input. I have looked into amqp-mock module but this seems to require a connection to a real amqp server and this was something I wanted to avoid.
Has anyone implemented anything similar or have a good strategy for this?
You easily mock require by using the "a" module:
// Example faking require('./foo') in unit test:
var fakeFoo = {};
var expectRequire = require('a').expectRequire;
expectRequire('./foo').return(fakeFoo);
// in sut:
var foo = require('./foo'); // returns fakeFoo
You can give node-amqp-mock a try.