The following code should write back everything the user puts in:
process.stdin.on('data', blob => {
process.stdout.write(blob)
})
Indeed it does, but when I change on to once I expect the script to only write back one line and then exit:
process.stdin.once('data', blob => {
process.stdout.write(blob)
})
However, it does not exit. Nothing ever happens again, no lines are written back, but the script keeps on running.
Looking at the documentation of once, I would expect it to exit since the documentation states:
The next time eventName is triggered, this listener is removed and then invoked.
Wouldn't that mean that the event loop queue becomes empty?
node process doesn't exit after firebase once seems relevant but the accepted answer now states:
Note that this is no longer applicable. Node.js will no longer hang when using once(), although it will be held open as long as there are active listeners subscribed to the remote server.
Which doesn't seem the case here.
Related
Background
I have a Firebase Cloud Function which sometimes can take a while to complete (20-60 seconds).
It gets triggered by a write to Firebase, which means that it starts processing at the onCreate event.
Problem
I have been doing some scripted testing by creating a new record in Firebase every N seconds, but it seems that if N is less than 20 seconds, the next onCreate trigger just doesn't fire.
In other words I end up in a situation like this:
Firebase:
record1
record2
record3
record4
Results written by the triggered function to another node in Firebase:
result-from-record1
...
record2, record3, record4 does not seem to trigger the function again.
Homework
I have re-checked Firebase documentation, but I cannot seem to find any information that explains this case.
There is some information about quotas for connected users, but it's only about connected users, not about the same triggers firing many times before the previously triggered function completes.
Questions
What is the default behavior of Firebase triggered functions in case it gets triggered while the previously triggered function is still running?
Is there any way to maybe cancel the running function if it gets triggered by a new onWrite?
Is there any queue of those triggered and running functions? (this queue doesn't seem to be the one)
What is the default behavior of Firebase triggered functions in case it gets triggered while the previously triggered function is still running?
There is no guarantee about how functions are invoked - they could happen in sequence on a single server instance, or they could run in run in parallel on multiple server instances. The order of invocation of functions is also not guaranteed.
Is there any way to maybe cancel the running function if it gets triggered by a new onWrite?
No.
Is there any queue of those triggered and running functions? (this queue doesn't seem to be the one)
There is no visible queue. Internally, Cloud Functions is using pubsub to manage the stream of events emitted by the database, but this is an implementation detail, and you have no direct control over how it works.
As for why your function doesn't seem to execute when you expect - there's not enough detail in your question to make a guess. Without seeing actual code, as well as the specific steps to take to reproduce the issue, it's not possible to say.
You might want to watch my video series on how Cloud Functions works in order to better understand its behavior.
I am new to Node JS and am trying to understand the concurrent / asynchronous execution models of Node.
So far, I do understand that whenever an asynchronous task is encountered in Node, that task runs in the background ( e.g an asynchronous setTimeout function will start timing) and the control is then sent back to other tasks that are there on the call stack. Once the timer times out, the callback that was passed to the asynchronous task is pushed onto the callback queue and once the call stack is empty, that callback gets executed. I took the help of this visualization to understand the sequence of task execution. So far so good.
Q1. Now, I am not being able to wrap my head around the paradigm of event listeners and event emitters and would appreciate if someone could explain how even emitters and listeners fall into the picture of call stack, event loops and callback queues.
Q2. I have the following code that reads data from the serial port of a raspberry pi.
const SerialPort = require('serialport');
const port = new SerialPort('/dev/ttyUSB0',{baudRate: 9600}, (err) => {
if (err) {
console.log("Port Open Error: ", err);
}
} )
port.on('data', (data) => {
console.log(data.toString());
})
As can be seen from the example, to read data from the serial port, an 'event-listener' has been employed. From what I understand, whenever data comes to the port, a 'data' event is emitted which is 'responded to' or rather listened to by the listener, which just prints the data onto the console.
When I run the above program, it runs continuously, with no break, printing the data onto the console whenever a data arrives at the serial port. There are no continuously running while loops continuously scanning the serial port as would be expected in a synchronous program. So my question is, why is this program running continuously? It is obvious that the event emitter is running continuously, generating an event whenever data comes, and the event listener is also running continuously, printing the data whenever a 'data' event is emitted. But WHERE are these things actually running, that too, continuously? How are these things fitting into the whole picture of the call/execution stack, the event loop and the callback queue?
Thanks
Q1. Now, I am not being able to wrap my head around the paradigm of event listeners and event emitters and would appreciate if someone could explain how even emitters and listeners fall into the picture of call stack, event loops and callback queues.
Event emitters on their own have nothing to do with the event loop. Event listeners are called synchronously whenever someone emits an event. When some code calls someEmitter.emit(...), all listeners are called synchronously from the time the .emit() occurred one after another. This is just plain old function calls. You can look in the eventEmitter code yourself to see a for loop that calls all the listeners one after another associated with a given event.
Q2. I have the following code that reads data from the serial port of a raspberry pi.
The data event in your code is an asynchronous event. That means that it will be triggered one or more times at an unknown time in the future. Some lower level code will be registered for some sort of I/O event. If that code is native code, then it will insert a callback into the node.js event queue. When node.js is done running other code, it will grab the next event from the event queue. When it gets to the event associated with data being available on the serial port, it will call port.emit(...) and that will synchronously trigger each of the listeners for the data event to be called.
When I run the above program, it runs continuously, with no break, printing the data onto the console whenever a data arrives at the serial port. There are no continuously running while loops continuously scanning the serial port as would be expected in a synchronous program. So my question is, why is this program running continuously?
This is the event-driven nature of node.js in a nutshell. You register an interest in certain events. Lower level code sees that incoming data has arrived and triggers those events, thus calling your listeners.
This is how the Javascript interpreter manages the event loop. Run current piece of Javascript until it's done. Check to see if any more events in the event loop. If so, grab next event and run it. If not, wait until there is an event in the event queue and then run it.
It is obvious that the event emitter is running continuously, generating an event whenever data comes, and the event listener is also running continuously, printing the data whenever a 'data' event is emitted. But WHERE are these things actually running, that too, continuously?
The event emitter itself is not running continuously. It's just a notification scheme (essentially a publish/subscribe model) where one party can register an interest in certain events with .on() and another party can trigger certain events with .emit(). It allows very loose coupling through a generic interface. Nothing is running continuously in the emitter system. It's just a notification scheme. Someone triggers an event with .emit() and it looks in its data structures to see who has registered an interest in that event and calls them. It knows nothing about the event or the data itself or how it was triggered. The emitters job is just to deliver notifications to those who expressed an interest.
We've described so far how the Javascript side of things works. It runs the event loop as described above. At a lower level, there is serial port code that interfaces directly with the serial port and this is likely some native code. If the OS supports a native asynchronous interface for the serial port, then the native code would use that and tell the OS to call it when there's data waiting on the serial port. If there is not a native asynchronous interface for the serial port data in the OS, then there's probably a native thread in the native code that interfaces with the serial port that handles getting data from the port, either polling for it or using some other mechanism built into the hardware to tell you when data is available. The exact details of how that works would be built into the serial port module you're using.
How are these things fitting into the whole picture of the call/execution stack, the event loop and the callback queue?
The call/execution stack comes into play the moment an event in the Javascript event queue is found by the interpreter and it starts to execute it. Executing that event will always start with a Javascript callback. The interpreter will call that callback (putting a return address on the call/execution stack). That callback will run until it returns. When it returns, the call/execution stack will be empty. The interpreter will then check to see if there's another event waiting in the event queue. If so, it will run that one.
FYI, if you want to examine the code for the serial port module it appears you are using, it's all there on Github. It does appear to have a number of native code files. You can see a file called poller.cpp here and it appears to do cooperative polling using the node.js add-on programming interface offered by libuv. For example, it creates a uv_poll_t which is a poll handle described here. Here's an excerpt from that doc:
Poll handles are used to watch file descriptors for readability, writability and disconnection similar to the purpose of poll(2).
The purpose of poll handles is to enable integrating external libraries that rely on the event loop to signal it about the socket status changes, like c-ares or libssh2. Using uv_poll_t for any other purpose is not recommended; uv_tcp_t, uv_udp_t, etc. provide an implementation that is faster and more scalable than what can be achieved with uv_poll_t, especially on Windows.
It is possible that poll handles occasionally signal that a file descriptor is readable or writable even when it isn’t. The user should therefore always be prepared to handle EAGAIN or equivalent when it attempts to read from or write to the fd.
In a NodeJS 6.10.2/SailsJS 0.12.13 based JavaScript application I experience since several months a strange error behavior.
In a Sails controller, I try to retrieve a property of a literal object:
console.log(someObject.someProperty);
console.log("I am still here!");
However, in my case someObject is undefined. So, I'd expect to get an error like 'Cannot read property someProperty of undefined.' - and then either Node.js to stop completely or the code to go on (with the next console.log).
Instead, the code simply stops executing at that point and I get a strange warning: "(node:4822) Warning: Possible EventEmitter memory leak detected. 11 close listeners added. Use emitter.setMaxListeners() to increase limit." It is however, unpredictable how often this error occurs. Somethings only once, somethings about 20 times right after each other.
What I found out so for is that it is somehow connected to the question whether there was already a response or not. Consider the following:
mySailsControllerFunction: function(req, res) {
console.log(someObject.someProperty);
console.log("I am still here!");
res.json({"foo":"dahoo"});
}
This will result in Sending 500 ("Server Error") response: ReferenceError: someObject is not defined - exactly what I expect.
However, now I first send some response and then trying to access my non existing property, turning the code into:
mySailsControllerFunction: function(req, res) {
res.json({"foo":"dahoo"});
setTimeout(function () {
console.log("Yeah!");
console.log(someObject.someProperty);
console.log("I am still here!");
},1000);
}
then I often get simply nothing: 'Yeah!' displayed, but nothing comes afterwards. The event listener error is sometimes there, sometimes not. Very strange.
Additionally, and strange enough, the problem seems to be somehow connected to the time passed since the start of Sails. I put the code you see above inside a Sails controller function which is called immediately after the clients re-connect. I then played around with the timeout values, restarting the Sails server several times. Outcome: If I set the timeout to 1s, in 4 of 5 tests, I will get the correct error behavior. For 10 seconds it is about 50%, for 30s the error will always be ignored without any console output.
However, if I put my test code outside of the Sails controller, I always get the correct error behavior by Node. So, I'm quite sure this is a wrong behavior of Sails, not Node.
Disclaimer: I don't know Sails. So it may or may not be related, but my answer may offer a clue.
From the Sails documentation:
http://sailsjs.com/documentation/reference/response-res/res-json
This method is terminal, meaning it is generally the last line of code
your app should run for a given request (hence the advisory usage of
return throughout these docs).
Thus, when you use res.json({"foo":"dahoo"});, Sails probably sends the response back to the client, closing the call sequence, which, if it uses Promises or some other async mechanism, may kind of "swallow" further code, as also suggested in an above comment. This is probably internal coding in Sails, so it's not immediately obvious from the outside WHY your second code block specifically doesn't work.
So you should stick to the first pattern: access your property first, and put res.json() at the end of the controller function.
For reference: I finally solved that issue.
There were, somehow hidden in the code, process exit handlers defined:
process.on('exit', myErrorFunction.bind());
process.on('SIGINT', myErrorFunction.bind());
process.on('uncaughtException', myErrorFunction.bind());
The problem was: The function in which these lines were in was bound to a cronjob. So, each time the cronjob executed, new handlers were registered. So, my assumption above (before vs. after response) was wrong: In fact everything worked till the cronjob was executed for the first time. From then on, it didn't. And eventually, the warning was fired (correctly!).
I would have never found out without this answer: Make node show stack trace after EventEmitter warning
You have to add one line of code to get the stack trace:
process.on('warning', e => console.warn(e.stack));
Additionally, speaking of stack traces: In the Sails serverError response (api/api/responses/serverError.js), it is convenient to access it like this:
module.exports = function serverError (data, options) {
console.log(data.stack);
/* ... */
};
I'm trying to listen for data changes in my firebase using firebase's package for Node. I'm using the on() method which is supposed to listen for changes non-stop (as opposed to once() method that only listens to the first occurrence of a specific event ) My listener.js file on the server is exactly like this:
var Firebase=require('firebase');
var Ref= new Firebase('https://mydatabase.firebaseio.com/users/');
Ref.on('child_changed',function(childsnapshot,prevchildname){
Ref.child(childsnapshot.key()).push("I hear you!");
} ) ;
But it only works the for the first occurrence and throws a fatal memory error after a second occurrence.
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
I'm very new to server side programming and don't know what to do. I must be missing something important. Should I set up special server settings with node first? or maybe make a daemon that runs a script with once() method every second or so ?
I'm pretty sure you're creating an endless loop here:
You push a value to https://mydatabase.firebaseio.com/users/
the on('child_changed' event fires in your script
your script pushes a new child under the value
so we go back to step 2 and repeat
It will happen quite rapidly too, since Firebase clients fire local events straight away.
It looks like you're trying to create a chat bot. Which means you more likely want to create sibling messages:
var Firebase=require('firebase');
var ref= new Firebase('https://mydatabase.firebaseio.com/users/');
ref.on('child_changed',function(childsnapshot,prevchildname){
ref.push("I hear you!");
}) ;
Note that it is pretty inefficient to use StackOverflow to debug code. Since you seem to be on Windows, I recommend installing Visual Studio and its node tools. They have a great debugger that allows you to step through the code. Setting a breakpoint in your callback (so in the line with ref.push), will quickly show you what is going wrong.
Perhaps the underlying issue is how the node-kafka module I am using has implemented things, but perhaps not, so here we go...
Using the node-kafa library, I am facing an issue with subscribing to consumer.on('message') events. The library is using the standard events module, so I think this question might be generic enough.
My actual code structure is large and complicated, so here is a pseudo-example of the basic layout to highlight my problem. (Note: This code snippet is untested so I might have errors here, but the syntax is not in question here anyway)
var messageCount = 0;
var queryCount = 0;
// Getting messages via some event Emitter
consumer.on('message', function(message) {
message++;
console.log('Message #' + message);
// Making a database call for each message
mysql.query('SELECT "test" AS testQuery', function(err, rows, fields) {
queryCount++;
console.log('Query #' + queryCount);
});
})
What I am seeing here is when I start my server, there are 100,000 or so backlogged messages that kafka will want to give me and it does so through the event emitter. So I start to get messages. To get and log all the messages takes about 15 seconds.
This is what I would expect to see for an output assuming the mysql query is reasonably fast:
Message #1
Message #2
Message #3
...
Message #500
Query #1
Message #501
Message #502
Query #2
... and so on in some intermingled fashion
I would expect this because my first mysql result should be ready very quickly and I would expect the result(s) to take their turn in the event loop to have the response processed. What I am actually getting is:
Message #1
Message #2
...
Message #100000
Query #1
Query #2
...
Query #100000
I am getting every single message before a mysql response is able to be processed. So my question is, why? Why am I not able to get a single database result until all the message events are complete?
Another note: I set a break point at .emit('message') in node-kafka and at mysql.query() in my code and I am hitting them turn-based. So it appears that all 100,000 emits are not stacking up up front before getting into my event subscriber. So there went my first hypothesis on the problem.
Ideas and knowledge would be very appreciated :)
The node-kafka driver uses quite a liberal buffer size (1M), which means that it will get as many messages from Kafka that will fit in the buffer. If the server is backlogged, and depending on the message size, this may mean (tens of) thousands of messages coming in with one request.
Because EventEmitter is synchronous (it doesn't use the Node event loop), this means that the driver will emit (tens of) thousands of events to its listeners, and since it's synchronous, it won't yield to the Node event loop until all messages have been delivered.
I don't think you can work around the flood of event deliveries, but I don't think that specifically the event delivery is problematic. The more likely problem is starting an asynchronous operation (in this case a MySQL query) for each event, which may flood the database with queries.
A possible workaround would be to use a queue instead of performing the queries directly from the event handlers. For instance, with async.queue you can limit the number of concurrent (asynchronous) tasks. The "worker" part of the queue would perform the MySQL query, and in the event handlers you'd merely push the message onto the queue.