I wrote a node.js script to fetch some prices from exchanges. It looks like this:
async function main() {
async function func() {
var start_time = performance.now();
for (let route of routes) {
var result_amount = await calc(route, amount_wei);
if (result_amount[5] > amount_start * 1) {
console.log("Good Trade");
}
while (true) {
await func();
}
}
and one route (route of routes) looks like this:
[
"quick / sushi - 1x1",
token_Address_usdc,
token_Address_dai,
token_Address_usdc,
"-",
"-",
"USDC - DAI - USDC",
]
So first I am fetching the output if I swap usdc to dai on quickswap. Then from dai to usdc on sushiswap. I save the output in an array (result_amount) and give it back to the main program (Now result is compared to the start amount).
I do have like 10 trading routes and the program needs about 20 seconds, so 2 seconds per route.
The routes are absolutely independent from each other, so it should be possible to fetch all routes at the same time right?
I have read something about multi threads with workers, but I have to say, I didn't get it. Can someone help me with this problem?
Thank you
It's important to understand that node.js is generally single threaded and not multithreaded. Even though asynchronous operations sometime give the impression of parallelism, it is not a requirement. Meaning, just because an operation is asynchronous, does not mean it has to run in its separate thread.
The routes are absolutely independent from each other, so it should be possible to fetch all routes at the same time right?
It depends, the easiest case would be complete independence, where the result of said request isn't used in conjunction with the results of other fetches.
You really didn't provide too much detail, so it's not really possible to answer your question.
What I can recommend, though, is: always try to avoid multithreading if possible.
ControlAltDel's answer is a good starting point.
Stop using await and use Promise.all
This will allow you to wait for all of your data to come in in parallel, rather than serially
function func() {
var promises = [];
for (let route of routes) {
promises.push (calc(route, amount_wei));
}
Promise.all(promises).then(function(completedItems) {
completedItems.forEach(function(val) {
var result_amount = val;
if (result_amount[5] > amount_start * 1) {
console.log("Good Trade");
}
}
});
}
Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 8 months ago.
Improve this question
So i tried to watch a tutorial and make a web scraper, i combined like 5 tutorial worth code into my code so i suppose it's messy.
This is the code:
const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
const fs = require('fs');
const { html } = require('cheerio');
const cheerio = require('cheerio');
const { setInterval } = require('timers/promises');
require('dotenv').config();
const accountSid = process.env.TWILIO_ACCOUNT_SID;
const authToken = process.env.TWILIO_AUTH_TOKEN;
const client = require('twilio')(accountSid,authToken);
puppeteer.use(StealthPlugin());
const oldData = "oldData";
async function scrape(){
const browser = await puppeteer.launch({
headless: true,
defaultViewport:{
width: 1920,
height: 1080
}
});
const page = await browser.newPage();
await page.goto('page');
const pageData = await page.evaluate(() => {
return{
width: document.documentElement.clientWidth,
height: document.documentElement.clientHeight,
html: document.documentElement.innerHTML,
};
})
const $ = cheerio.load(pageData.html);
const element = $(".accordion__header-inner:last");
console.log(element.text());
};
const handle = setInterval(scrape(), 4000);
scrape();
I hide some parts that i didn't want to be seen.
So, when i run the code function it logged the element thing twice.
When i delete the interval thing it works only one.
That is the only difference, i want to log the element thing in every 5 seconds.
Any help is appreciated and i hope this isn't a poorly written question.
In your setInterval you are executing the function and passing it's return value as argument, instead of passing function itself as argument.
It should be:
const handle = setInterval(scrape, 4000);
Note, you won't be use this inside the function, because it will be called inside setInterval scope.
If you need to use this, then use bind():
const handle = setInterval(scrape.bind(scrape), 4000);
You seem to be calling the "scrape()" method in two places. Once in the SetInterval() method and then immediately again on the line after. That looks like your problem.
I have read a few answers to similar questions, but haven't been able to understand why my application behaves this way.
I am updating a CLI application to be modular. The main script imports a function called questionOne after a console.log:
const questionOne = require('./modules/questionOne');
console.log('\n');
console.log('Which DB do you want to use?'.inverse);
questionOne();
questionOne uses readline-sync to ask a question and execute code depending on the answer. But, when I run my application... the question is asked first, and then Which DB do you want to use runs after the user has asked the question. Why?
For Reference
The code for questionOne:
const colors = require('colors');
const readlineSync = require('readline-sync');
//Data Modules
const { dbList } = require('../data/dbList.json');
const db = readlineSync.keyInSelect(dbList);
//people = 0 || offers = 1 || networks = 2
const questionOne = () => {
if (db === 0) {
console.log('\n');
console.log('Which Collection would you like to use?'.inverse.bold);
// const colRef = readlineSync.keyInSelect();
// dbPeople(colRef);
} else if (db === 1) {
console.log('You picked offers');
} else if (db === 2) {
console.log('You picked networks');
} else {
process.exit();
}
};
module.exports = questionOne;
I understand that I can put the console.log inside the module. But I am curious why javascript behaves this way?
When you first to require('./modules/questionOne');, that runs all the code at the top level of the module you are loading. And, it runs that code synchronously, meaning it doesn't return until that top level code returns.
So, in your example, that's going to run all the code at the top level of your questionOne module. That top level code includes this line:
const db = readlineSync.keyInSelect(dbList);
So, that line of code will run before the module has finished loading and before it returns from this:
const questionOne = require('./modules/questionOne');
So, that should explain why the first thing that happens is that you get the prompt from const db = readlineSync.keyInSelect(dbList);. If you want to display something before that, then put that before it in the code.
Then, after that promise is done, the only other things that happen in your questionOne module are the definition of the questionOne function and the assignment module.exports = questionOne;. At that point, the require('./modules/questionOne'); returns and the value from the module.exports is assigned to your questionOne variable in your main module.
Then, and only then, the rest of your main module runs and it executes:
console.log('\n');
console.log('Which DB do you want to use?'.inverse);
questionOne();
I understand that I can put the console.log inside the module. But I am curious why javascript behaves this way?
It's just executing code in the order encountered. Nothing here appears asynchronous so it's just simple run one line of code after the other as they are encountered.
I have this js code:
const python = spawn('python', [path.join(settings.PROJECT_DIR, '/tests_explorer/scripts/product/product.py').replace(/\\/g, "/"), 'node.js', 'python']);
python.stdout.on('data', function (data) {
console.log('Pipe data from python script ...');
dataToSend = data.toString();
console.log(dataToSend)
});
It should execute my python Script product.py:
import sys
import general.general //my own utils
print('#Hello from python#')
print('First param:'+sys.argv[1]+'#')
print('Second param:'+sys.argv[2]+'#')
name = general.id_generator(9)
print(name)
But the node js seems skipped to run my python,
if I don't import general.general which is my own utils to generate random name, the python well executed and I can see the print data
Is anyone know how to solve this?
Thankyou
Check if it's present on the path (process.env.PATH), if python even exists on your system (it may be python3 or python2) and if the path to the file is correct and perhaps not OS dependent:
const { spawn } = require('child_process');
const py = spawn('python3', ['-c', 'print("hello")']);
py.stdout.on('data', (data) => {console.log(`stdout: ${data}`);});
It works in CLI / REPL because it keeps the process open. However, in a file, it doesn't, therefore the event-based programming doesn't receive the event before the event loop manages to shutdown (kind of like while (true) {} in the background with consistent interrupts so your code can execute in-between).
For that you can utilize spawnSync() which will block the program until the process is spawned, data interchanged and process closed.
const { spawnSync } = require('child_process');
const py = spawnSync('python3', ['main.py']);
console.log(py.output.toString());
Or like in the linked question, setTimeout() or other system hacks to basically wait for the event loop to process your event (in this case on("data", ...) due to the async nature of the child_process module.
I need in node.js function
result = execSync('node -v');
that will synchronously execute the given command line and return all stdout'ed by that command text.
ps. Sync is wrong. I know. Just for personal use.
UPDATE
Now we have mgutz's solution which gives us exit code, but not stdout! Still waiting for a more precise answer.
UPDATE
mgutz updated his answer and the solution is here :)
Also, as dgo.a mentioned, there is stand-alone module exec-sync
UPDATE 2014-07-30
ShellJS lib arrived. Consider this is the best choice for now.
UPDATE 2015-02-10
AT LAST! NodeJS 0.12 supports execSync natively.
See official docs
Node.js (since version 0.12 - so for a while) supports execSync:
child_process.execSync(command[, options])
You can now directly do this:
const execSync = require('child_process').execSync;
code = execSync('node -v');
and it'll do what you expect. (Defaults to pipe the i/o results to the parent process). Note that you can also spawnSync now.
See execSync library.
It's fairly easy to do with node-ffi. I wouldn't recommend for server processes, but for general development utilities it gets things done. Install the library.
npm install node-ffi
Example script:
var FFI = require("node-ffi");
var libc = new FFI.Library(null, {
"system": ["int32", ["string"]]
});
var run = libc.system;
run("echo $USER");
[EDIT Jun 2012: How to get STDOUT]
var lib = ffi.Library(null, {
// FILE* popen(char* cmd, char* mode);
popen: ['pointer', ['string', 'string']],
// void pclose(FILE* fp);
pclose: ['void', [ 'pointer']],
// char* fgets(char* buff, int buff, in)
fgets: ['string', ['string', 'int','pointer']]
});
function execSync(cmd) {
var
buffer = new Buffer(1024),
result = "",
fp = lib.popen(cmd, 'r');
if (!fp) throw new Error('execSync error: '+cmd);
while(lib.fgets(buffer, 1024, fp)) {
result += buffer.readCString();
};
lib.pclose(fp);
return result;
}
console.log(execSync('echo $HOME'));
Use ShellJS module.
exec function without providing callback.
Example:
var version = exec('node -v').output;
There's an excellent module for flow control in node.js called asyncblock. If wrapping the code in a function is OK for your case, the following sample may be considered:
var asyncblock = require('asyncblock');
var exec = require('child_process').exec;
asyncblock(function (flow) {
exec('node -v', flow.add());
result = flow.wait();
console.log(result); // There'll be trailing \n in the output
// Some other jobs
console.log('More results like if it were sync...');
});
Native Node.js solution is:
const {execSync} = require('child_process');
const result = execSync('node -v'); // 👈 this do the trick
Just be aware that some commands returns Buffer instead of string. And if you need string just add encoding to execSync options:
const result = execSync('git rev-parse HEAD', {encoding: 'utf8'});
... and it is also good to have timeout on sync exec:
const result = execSync('git rev-parse HEAD', {encoding: 'utf8', timeout: 10000});
This is not possible in Node.js, both child_process.spawn and child_process.exec were built from the ground up to be async.
For details see: https://github.com/ry/node/blob/master/lib/child_process.js
If you really want to have this blocking, then put everything that needs to happen afterwards in a callback, or build your own queue to handle this in a blocking fashion, I suppose you could use Async.js for this task.
Or, in case you have way too much time to spend, hack around in Node.js it self.
This is the easiest way I found:
exec-Sync:
https://github.com/jeremyfa/node-exec-sync
(Not to be confused with execSync.)
Execute shell command synchronously. Use this for migration scripts, cli programs, but not for regular server code.
Example:
var execSync = require('exec-sync');
var user = execSync('echo $USER');
console.log(user);
Just to add that even though there are few usecases where you should use them, spawnSync / execFileSync / execSync were added to node.js in these commits: https://github.com/joyent/node/compare/d58c206862dc...e8df2676748e
You can achieve this using fibers. For example, using my Common Node library, the code would look like this:
result = require('subprocess').command('node -v');
my way since 5 years is to have 2 lines ;
const { execSync } = require('child_process');
const shell = (cmd) => execSync(cmd, {encoding: 'utf8'});
Then enjoy:
shell('git remote -v')
or
out = shell('ls -l')
.. so on
I get used to implement "synchronous" stuff at the end of the callback function. Not very nice, but it works. If you need to implement a sequence of command line executions you need to wrap exec into some named function and recursively call it.
This pattern seem to be usable for me:
SeqOfExec(someParam);
function SeqOfExec(somepParam) {
// some stuff
// .....
// .....
var execStr = "yourExecString";
child_proc.exec(execStr, function (error, stdout, stderr) {
if (error != null) {
if (stdout) {
throw Error("Smth goes wrong" + error);
} else {
// consider that empty stdout causes
// creation of error object
}
}
// some stuff
// .....
// .....
// you also need some flag which will signal that you
// need to end loop
if (someFlag ) {
// your synch stuff after all execs
// here
// .....
} else {
SeqOfExec(someAnotherParam);
}
});
};
I had a similar problem and I ended up writing a node extension for this. You can check out the git repository. It's open source and free and all that good stuff !
https://github.com/aponxi/npm-execxi
ExecXI is a node extension written in C++ to execute shell commands
one by one, outputting the command's output to the console in
real-time. Optional chained, and unchained ways are present; meaning
that you can choose to stop the script after a command fails
(chained), or you can continue as if nothing has happened !
Usage instructions are in the ReadMe file. Feel free to make pull requests or submit issues!
EDIT: However it doesn't return the stdout yet... Just outputs them in real-time. It does now. Well, I just released it today. Maybe we can build on it.
Anyway, I thought it was worth to mention it.
you can do synchronous shell operations in nodejs like so:
var execSync = function(cmd) {
var exec = require('child_process').exec;
var fs = require('fs');
//for linux use ; instead of &&
//execute your command followed by a simple echo
//to file to indicate process is finished
exec(cmd + " > c:\\stdout.txt && echo done > c:\\sync.txt");
while (true) {
//consider a timeout option to prevent infinite loop
//NOTE: this will max out your cpu too!
try {
var status = fs.readFileSync('c:\\sync.txt', 'utf8');
if (status.trim() == "done") {
var res = fs.readFileSync("c:\\stdout.txt", 'utf8');
fs.unlinkSync("c:\\stdout.txt"); //cleanup temp files
fs.unlinkSync("c:\\sync.txt");
return res;
}
} catch(e) { } //readFileSync will fail until file exists
}
};
//won't return anything, but will take 10 seconds to run
console.log(execSync("sleep 10"));
//assuming there are a lot of files and subdirectories,
//this too may take a while, use your own applicable file path
console.log(execSync("dir /s c:\\usr\\docs\\"));
EDIT - this example is meant for windows environments, adjust for your own linux needs if necessary
I actually had a situation where I needed to run multiple commands one after another from a package.json preinstall script in a way that would work on both Windows and Linux/OSX, so I couldn't rely on a non-core module.
So this is what I came up with:
#cmds.coffee
childproc = require 'child_process'
exports.exec = (cmds) ->
next = ->
if cmds.length > 0
cmd = cmds.shift()
console.log "Running command: #{cmd}"
childproc.exec cmd, (err, stdout, stderr) ->
if err? then console.log err
if stdout? then console.log stdout
if stderr? then console.log stderr
next()
else
console.log "Done executing commands."
console.log "Running the follows commands:"
console.log cmds
next()
You can use it like this:
require('./cmds').exec ['grunt coffee', 'nodeunit test/tls-config.js']
EDIT: as pointed out, this doesn't actually return the output or allow you to use the result of the commands in a Node program. One other idea for that is to use LiveScript backcalls. http://livescript.net/