I created a Node.js API.
When this API gets called I return to the caller fairly quickly. Which is good.
But now I also want API to call or launch an different API or function or something that will go off and run on it's own. Kind of like calling a child process with child.unref(). In fact, I would use child.spawn() but I don't see how to have spawn() call another API. Maybe that alone would be my answer?
Of this other process, I don't care if it crashes or finishes without error.
So it doesn't need to be attached to anything. But if it does remain attached to the Node.js console then icing on the cake.
I'm still thinking about how to identify & what to do if the spawn somehow gets caught up in running a really long time. But ready to cross that part of this yet.
Your thoughts on what I might be able to do?
I guess I could child.spawn('node', [somescript])
What do you think?
I would have to explore if my cloud host will permit this too.
You need to specify exactly what the other spawned thing is supposed to do. If it is calling an HTTP API, with Node.js you should not launch a new process to do that. Node is built to run HTTP requests asynchronously.
The normal pattern, if you really need some stuff to happen in a different process, is to use something like a message queue, the cluster module, or other messaging/queue between processes that the worker will monitor, and the worker is usually set up to handle some particular task or set of tasks this way. It is pretty unusual to be spawning another process after receiving an HTTP request since launching new processes is pretty heavy-weight and can use up all of your server resources if you aren't careful, and due to Node's async capabilities usually isn't necessary especially for things mainly involving IO.
This is from a test API I built some time ago. Note I'm even passing a value into the script as a parameter.
router.put('/test', function (req, res, next) {
var u = req.body.u;
var cp = require('child_process');
var c = cp.spawn('node', ['yourtest.js', '"' + u + '"'], { detach: true });
c.unref();
res.sendStatus(200);
});
The yourtest.js script can be just about anything you want it to be. But I thought I would have enjoy learning more if I thought to first treat the script as a node.js console desktop app. FIRST get your yourtest.js script to run without error by manually running/testing it from your console's command line node yourstest.js yourparamtervalue THEN integrate it in to the child.spawn()
var u = process.argv[2];
console.log('f2u', u);
function f1() {
console.log('f1-hello');
}
function f2() {
console.log('f2-hello');
}
setTimeout(f2, 3000); // wait 3 second before execution f2(). I do this just for troubleshooting. You can watch node.exe open and then close in TaskManager if node.exe is running long enough.
f1();
Related
I think we need some help here. Thanks in advance.
I have been doing programming in .Net for desktop applications and have used Timer objects to wait for a task to complete before the task result are shown in a data grid. Recently, we switched over to NodeJs and find it pretty interesting. We could design a small application that executes some tasks using PowerShell scripts and return the data to the client browser. However, I would have to execute a Timer on the client browser (when someone clicks on a button) to see if the file, that Timer receives from the server, has "ENDOFDATA" or not. Once the Timer sees ENDOFDATA it triggers another function to populate DIV with the data that was received from the server.
Is this the right way to get the data from a server? We really don't want to block EventLoop. We run PowerShell scripts on NodeJS to collect users from Active Directory and then send the data back to the client browser. The PowerShell scripts are executed as a Job so EventLoop is not blocked.
Here is an example of the code at NodeJs:
In the below code can we insert something that won't block the EventLoop but still respond to the server once the task is completed? As you can see in the code below, we would like to send the ADUsers.CSV file to the client browser once GetUsers.PS1 has finished executing. Since GetUSers.PS1 takes about five minutes to complete the Event Loop is blocked and the Server can no longer accept any other requests.
app.post("/LoadDomUsers", (request, response) => {
//we check if the request is an AJAX one and if accepts JSON
if (request.xhr || request.accepts("json, html") === "json") {
var ThisAD = request.body.ThisAD
console.log(ThisAD);
ps.addCommand("./public/ps/GetUsers.PS1", [{
name: 'AllParaNow',
value: ScriptPara
}])
ps.addCommand(`$rc = gc ` + __dirname + "/public/TestData/AD/ADUsers.CSV");
ps.addCommand(`$rc`);
ps.invoke().then((output) => {
response.send({ message: output });
console.log(output);
});
}
});
Thank you.
The way you describe your problem isn't that clear. I had to read some of the comments in your initial question just to be sure I understood the issue. Honestly, you could just utilize various CSV NPM packages to read and write from your active directory with NodeJS.
I/O is non-blocking with NodeJS, so you're not actually blocking the EventLoop. You can handle multiple I/O requests, since NodeJS will just create threads for each one,
and continue execution on the main thread until the I/O operations complete and send back the data to its function reference, adding them as functions to the callstack and resuming program execution from those function's references. After you get the I/O data, you just send it back to the client through the response object. There should be no timers needed.
So is the issue once the powershell script runs, you have to wait for that initial script to complete before being able to handle pending requests? I'm still a bit unclear...
I am working on a project where my Electron App interacts with a physical device using serial commands, via serialport. The app sends a string to the device, the device executes the command (which can take ~30s) and then sends back a string to signify completion and results from that operation.
My goal is to automate a series of actions. For that, basically the following needs to be done asynchronously, so that the render thread doesn't get blocked:
Start a loop
Send a string to the device
Wait until a specific response comes back
Tell the render thread about the response, so it can update the UI
Afterwards, repeat with the next string.
Actually, multiple different commands need to be send in each loop cycle, and between each one the app has to wait for a specific string from the device.
This is kind of related to my last question, What's the correct way to run a function asynchronously in Electron?. From that, I know I should use web workers to run something asynchronously. However, my plan turned out to involve more problems than I anticipated, and I wanted to ask what would be a good way to implement this, having the whole plan in mind and not just a certain aspect of it.
I am especially not sure how to make the worker work with serialport. The serial device it needs to interact with is a child of the render process, so sending commands will probably be done over web worker messages. But I have no idea on how to make the worker wait for a specific response from the device.
(Since this question is of a more general nature, I am unsure whether I should provide some code snippets. If this is to general, I can try to write some pseudo code to make my problem more clear.)
I would go for a promise-based approach like this:
let promiseChain = Promise.resolve();
waitForEvent = function(){
return new Promise(resolve=>{
event.on("someEvent", (eventData => {
resolve(eventData)
}))
})
}
while(someLoopCondition) {
promiseChain = promiseChain
.then(sendToSerialPort(someString))
.then(waitForEvent)
.then(result=>{
updateUI(result)
})
}
Let's say I have a parent node process that forks a child process. I want to create a request-response mechanism between them, so that the child can send a message (JSON) to the parent and wait for a response and the other way as well.
I don't want to use HTTP because I want the mechanism to be as tight and low-latency as possible, and HTTP usually has some overhead. Also, HTTP is a client-server protocol and I want something bi-directional. (I'd be happy to know if the overhead is negliglbe or as bad as using pipes, TCP sockets directly, or WebSockets in this situation; it'd make my job easier).
The end-result I want is something like this. Note that this code is in TypeScript for clarity, not because the answer has to be in TypeScript too.
class Communicator {
// Callback for when a message is received by the Communicator
// Resolve returns a response to the sender, reject returns an error
onMessage : (data : any) => Promise<any>;
// Sends an object to the other process, and returns a promise
// that resolves with the response or rejects with the error reason
send(data : any) : Promise<any>;
}
In these case, I'm going to assume we use the native process.send mechanism, but if another system like WebSockets are better, it'd be a bit different:
// PARENT
let cp = require("child_process").fork("./child.js");
let comm = new Communicator(cp);
await comm.send({property : 1});
.
// CHILD
let comm = new Communicator(process);
await comm.send({property : 1});
The request/response system should support both Success-type and Error-type responses, as well as Error non-responses that indicate the underlying protocol failed.
How should I implement this in Node? There are lots of issues I can think of if I tried to do this naively, since multiple requests and responses can be running in parallel. Is there a library that already does this? Should I just use HTTP?
One possible solution to the problem appears to be to use WAMP over WebSockets:
WebSocket request-response subprotocol
However, since in this case we're talking about IPC and processes running inside a single machine, we're not really constrained with regard to the transfer protocol (we could even use raw sockets if we really wanted). There is also a built-in messaging system between parent and child processes.
So maybe it's possible to use WAMP over something else, or come up with another solution.
With regard to WAMP, one package we can use is autobahn.
I want to know if there is any benefit in passing off db or other async calls to a worker process or processes. Specifically I'm using heroku and postgres. I've read up a good bit on node.js and how to structure your server so that the event loop isn't blocked and that smart architecture doesn't leave incoming requests hanging longer than 300ms or so.
Say I have the following:
app.get('/getsomeresults/:query', function(request, response){
var foo = request.params.query;
pg.connect(process.env.DATABASE_URL, function(err, client, done) {
client.query("SELECT * FROM users WHERE cat=$1", [foo],
function(err, result){
//do some stuff with result.rows that may take 1000ms
response.json({some:data})
});
});
});
Being that postgresql is async by nature is there any real benefit to creating a worker process to handle the processing of the results set from the initial db call?
You don't gain any benefit for running async functions in another process because the real work (running the SQL query) is already running in another process (postgres). Basically, the async/event-oriented design pattern is a lightweight process manager for things that run outside your process.
However, I noticed in your comment that the processing in the callback function does indeed take up a lot of CPU time (if that's really the case). That portion of code does benefit from being run in another process - it frees the main process to accept incoming requests.
There are two ways to structure such code. Either run the async function in a separate process (so that the callback doesn't block) or just run the relevant portion of the callback as a function in a separate process.
Calling client.query from a separate process won't give you a real benefit here, as sending queries to the server is already an asynchronous operation in node-pg. However, the real problem is the long execution time your callback function. The callback runs synchronously in the main event loop and blocks other operations, so it would be a good idea to make this non-blocking.
Option 1: Fork a child process
Creating a new process every time the callback is executed is no good idea, since each Node.js process needs its own environment, which is time consuming to set up. Instead it would be better to create multiple server processes when the server is started and let them handle requests concurrently.
Option 2: Use Node.js clusters
Luckily Node.js offers the cluster interface to achieve exactly this. Clusters give you the ability to handle multiple worker processes from one master process. It even supports connection pooling, so you can simply create a HTTP server in each child process an the incoming requests will be distributed among them automatically (node-pg supports pooling as well).
The cluster solution is also nice, because you don't have to change a lot in your code for that. Just write the master process code and start your existing code as workers.
The official documentation on Node.js clusters explains all aspects if clusters very well, so I won't go into details here. Just a short example for a possible master code:
var cluster = require("cluster");
var os = require("os");
var http = require("http");
if (cluster.isMaster)
master();
else
worker();
function master() {
console.info("MASTER "+process.pid+" starting workers");
//Create a worker for each CPU core
var numWorkers = os.cpus().length;
for (var i = 0; i < numWorkers; i++)
cluster.fork();
}
function worker() {
//Put your existing code here
console.info("WORKER "+process.pid+" starting http server");
var httpd = http.createServer();
//...
}
Option 3: Split the result processing
I assume that the reason for the long execution time of the callback function is that you have to process a lot of result rows and that there is no chance to process the results in a faster way.
In that case it might also be a good idea to split the processing into several chunks using process.nextTick(). The chunks will run synchronously in several event-loop frames, but other operations (like event-handlers) can be executed between these chunks. Here's a rough (and untested) scetch how the code could look like:
function(err, result) {
var s, i;
s = 0;
processChunk();
// process 100 rows in one frame
function processChunk() {
i = s;
s += 100;
while (i<result.rows.length && i<s) {
//do some stuff with result.rows[i]
i++;
}
if (i<result.rows.length)
process.nextTick(processChunk);
else
//go on (send the response)
}
}
I'm not 100% sure, but I think node-pg offers some way to receive a query result not as a whole, but split into several chunks. This would simplify the code a lot, so it might be an idea to search into that direction...
Final conclusion
I would use option 2 in the first place and option 3 additionally, if new requests still have to wait too long.
I have seen the potential of NodeJS recently and try this for a small task to use .
about the following:
i want to create a web service , which allows you to start and stop multiple client instances (multiple nodejs tasks who should run, until i send a stop request to a instance):
so a http server modul waits for start/stop requests (with params) (maybe with express?) like
[start]
http://taskservice/start?user=abc&pass=123
the service should create a instance of a task like:
var task = new MyTask();
task.login(user, pass);
task.on("loginsuccess", function() {
task.startwork();
};
[stop identified by user]
http://taskservice/stop?user=abc
The problem is, I do not know how this can be best implemented . It must always be able to create any number of tasks and the tasks must run until I say stop . (the tasks require very little performance)
Does anyone have an idea or a small code example ?
I am not sure if this is the proper way to do it but I think you can do it with forever. Check this link; http://www.codingdefined.com/2014/08/use-of-forever-in-nodejs.html
You can stop and start child nodejs processes through your root script.