I have a python script which returns a JSON file as output by taking a FILE as input.
I have 10 files, I am using spawnSync inside for loop and inside loop I have fs.readFile for reading JSON file which is coming from the python script.
But the problem is spawnSync is blocking fs.readFile until it executes python scripts with all 10 files. Since both spawnSync and fs.readFile are inside for loop, I want fs.readFile to read a JSON file as soon as first python script executes and outputs JSON file.
But it is not happening. spawnSync is blocking and it is continuing with next file to execute python script.fs.reafFile should prints data as soon as the file gets executes. please help, Here is my code snippet.
var spawn = require('child_process').spawnSync;
var fs = require('fs');
var filename = ['first.txt','second.txt','third.txt',....]
for(var i=0;i<10;i++)
{
var myscript = spawn('python',['/pathToPython/myPython.py',filename[i]]);
fs.readFile('/pathToPython/' + filename[i] + '.json','utf8',function(err,data){
if(err){
console.log(err);
}else{
console.log(data);
}
});
}
If you are rely to use third party module then I recommend to use async.eachSeries the method of the async module to resolve this issue
var filename = ['first.txt','second.txt','third.txt',....]
async.eachSeries(filename, function(item, next) {
var myscript = spawn('python', ['/pathToPython/myPython.py', item]);
fs.readFile('/pathToPython/' + item + '.json', 'utf8', function(err, data) {
if (err) {
console.log(err);
} else {
console.log(data);
next();
}
});
})
Related
I have one js file that sends a couple variable to ejs and that works, but when I try to run a second node js process running another js file, it seems like it won't look in the process that I run second.
What do I need to do to get the ejs file to recognize and receive variable from the second js file?
On the webpage, I get <var> is not definedand it traces back to a render from the first js file even though I'm rendering it from the second js file.
I'm sending it using the following code:
app.get('/', (req, res) => {
res.render('main', {
variable: value
}
});
edit:
Trying my best to add an example...
first.js:
var variable1 = value1;
app.get('/', (req, res) => {
res.render('main', {
variable1: value1
}
});
second.js:
var variable2 = value2;
app.get('/', (req, res) => {
res.render('main', {
variable2: value2
}
});
When I load main.ejs, it says it can't find variable2 in first.js.. why is it looking in first.js for the variable, even though I'm passing it from second.js?
I have first.js that calculates some variables and passes it to main.ejs. I also have second.js that runs separately from first.js that does the same thing.
It doesn't run separately… or at all.
HTTP is based on the simple system of:
A client (like a web browser) asks for something
The server (like your Express.js based program) responds with something
And those somethings are single somethings.
So the browser sends a request for /, and then Express looks through the registered routes until it finds a match.
app.get('/', (req, res) => {
res.render('main', {
variable1: value1
}
});
It runs that function and generates a response.
It never gets to the other function registered for the same route: It has already found a route that allowed it to provide a response. If it kept going and sent another response, then it would be a completely different execution of the EJS script, and a completely different response that the browser wouldn't be expecting.
Write one route for handling /.
Call render once and pass it an object with all the data you want
Split the code up into functions if you want to make it easier to manage. One of the functions can be the route handler itself, and it can call the others and get the data they return. If things get really complex, you can move the functions into separate modules and require them.
You should implement worker threads https://nodejs.org/api/worker_threads.html to make two node process communicate to each other
// first.js
const { Worker } = require('worker_threads')
function runProcess (workerData) {
return new Promise((resolve, reject) => {
const worker = new Worker('./second.js', { workerData })
worker.on('message', resolve)
worker.on('error', reject)
worker.on('exit', (code) => {
if (code !== 0) {
reject(new Error(`Worker stopped with exit code ${code}`))
}
})
})
}
async function run () {
const result = await runProcess('variable value to send')
console.log(result)
}
run()
// second.js
const { workerData, parentPort } = require('worker_threads')
parentPort.postMessage({ received: workerData })
I have to choose one of these languages:
Python (with Selenium or any suggestion)
Javascript (with node with any module)
Bash (with curl for example)
To do the following:
Make a request to an API (Scrapy cloud) and get some value, in my case I just need the id of the response:
{"count": 1, "id": "195457/7/19", "width": 32, "height": 27, "started_time": "2017-06-22T08:20:26", "total": 1, "status": "ok"}
And then make another request with the id to download that provides a download to a file with a CSV/JSON format.
What I tried:
Python:
With Selenium (Firefox driver) open and get the id, it works fine but when I try to download the file with the next API request it asks me for what I want to do with the file (download or open with...). So, as I have to interact with the dialog it is not viable.
Javascript:
I found a module to download files but it is just to download files as images from image web URLs and not for download a file (like the Linux wget command).
Bash:
With curl it works but I can just get the whole response and then I cannot get the id value so I cant continue with what I want. Also I tried to download de file of the second step and this works fine with a simple curl -o myfile.csv URL
Any help would be appreciated. Thanks for reading!
Here is a node version. Its quite broad but the 2 main functions are the callApi and downloadFile.
I dont know the structure of your API url so for me now I have mocked some simple ones - change to what you need.
You will need to npm install request and update the variables to match your API.
index.js
const request = require('request');
const http = require('http');
//const https = require('https'); maybe required
const fs = require('fs');
const apiEndPoint = 'http://scrapycloud?someparam=';
const fileName = 'data.csv';
const assetEndPoint = 'http://assetUrl?id=';
// This will call your api and get the asset id then calls the downloadFile function.
function callApi(assetId, callback) {
request(apiEndPoint + assetId, function (error, response, body) {
if (error) {
return callback(error);
}
const info = JSON.parse(body);
const assetId = info.id;
downloadFile(assetId, callback);
});
}
// This function creates a writeSteam to save a file to your local machine, performs a http request to the assets and pipes it back into the write stream
function downloadFile(assetId, callback) {
var file = fs.createWriteStream(fileName);
//use the following line if your requests needs to be https
//var request = https.get(assetEndPoint + assetId, function (response) {
var request = http.get(assetEndPoint + assetId, function (response) {
response.pipe(file);
file.on('finish', function () {
file.close(callback);
});
}).on('error', function (err) {
fs.unlink(dest);
if (callback) callback(err.message);
});
}
// Called when everything is finished or an error
function complete(err) {
if (err) {
return console.log(err);
}
console.log('file downloaded');
}
// Starts the process, pass it an id and a callback
callApi('123131', complete);
i'm pretty new into NodeJs. And i am trying to read a file into a variable.
Here is my code.
var fs = require("fs"),
path = require("path"),
util = require("util");
var content;
console.log(content);
fs.readFile(path.join(__dirname,"helpers","test.txt"), 'utf8',function (err,data) {
if (err) {
console.log(err);
process.exit(1);
}
content = util.format(data,"test","test","test");
});
console.log(content);
But every time i run the script i get
undefined and undefined
What am i missing? Help please!
As stated in the comments under your question, node is asynchronous - meaning that your function has not completed execution when your second console.log function is called.
If you move the log statement inside the the callback after reading the file, you should see the contents outputted:
var fs = require("fs"),
path = require("path"),
util = require("util");
var content;
console.log(content);
fs.readFile(path.join(__dirname, "helpers", "test.txt"), 'utf8', function (err, data) {
if (err) {
console.log(err);
process.exit(1);
}
content = util.format(data, "test", "test", "test");
console.log(content);
});
Even though this will solve your immediately problem, without an understanding of the async nature of node, you're going to encounter a lot of issues.
This similar stackoverflow answer goes into more details of what other alternatives are available.
The following code snippet uses ReadStream. It reads your data in separated chunks, if your data file is small it will read the data in a single chunk. However this is a asynchronous task. So if you want to perform any task with your data, you need to include them within the ReadStream portion.
var fs = require('fs');
var readStream = fs.createReadStream(__dirname + '/readMe.txt', 'utf8');
/* include the file directory and file name instead of <__dirname + '/readMe.txt'> */
var content;
readStream.on('data', function(chunk){
content = chunk;
performTask();
});
function performTask(){
console.log(content);
}
There is also another easy way by using synchronous task. As this is a synchronous task, you do not need to worry about its executions. The program will only move to the next line after execution of the current line unlike the asynchronous task.
A more clear and detailed answer is provided in the following link:
Get data from fs.readFile
var fs = require('fs');
var content = fs.readFileSync('readMe.txt','utf8');
/* include your file name instead of <'readMe.txt'> and make sure the file is in the same directory. */
or easily as follows:
const fs = require('fs');
const doAsync = require('doasync');
doAsync(fs).readFile('./file.txt')
.then((data) => console.log(data));
I easily accomplished this with the fs.readFileSync but I want to do this asynchronously. My code follows.
function send(err, str){
if(err){
console.log(err);
}
var template = ejs.render(str, 'utf8', {name: data.name});
transporter.sendMail({
from: myEmail,
to: anotherEmail,
subject: mySubject,
html: template,
attachments: images
}, function(err, response) {
if(err){
console.log(err);
}
});
}
fs.readFile('emailTemplate.ejs', send);
So I made my own callback for fs.readFile so that when the file has been read it will render the email, putting the proper name in and then send it off with nodemailer. However, it does not like this. It gets by the error if no problem but render throws the following error when it tries to render the template.
TypeError: Object (Followed by the entire HTML of the template) has no method 'indexOf'
at Object.exports.parse (/home/ubuntu/workspace/node_modules/ejs/lib/ejs.js:144:21)
at exports.compile (/home/ubuntu/workspace/node_modules/ejs/lib/ejs.js:229:15)
at Object.exports.render (/home/ubuntu/workspace/node_modules/ejs/lib/ejs.js:289:10)
at send (/home/ubuntu/workspace/routes/email.js:171:28)
at fs.readFile (fs.js:272:14)
at Object.oncomplete (fs.js:108:15)
Doing it synchronously works fine though.
var str = fs.readFileSync('emailTemplate.ejs', 'utf8');
var template = ejs.render(str, {
name: data.name
});
Can anyone give me any insight into why this is happening?
The documentation of fs.readFile and fs.readFileSync says
If no encoding is specified, then the raw buffer is returned.
Because you provide the encoding with the synchronous version, but do not with the asynchronous one they both differ in behaviour.
If you try this:
fs.readFile('emailTemplate.ejs', {encoding: "utf8"}, send);
it should work.
Try setting the encoding of the fs.readFile call, e.g.:
fs.readFile('emailTemplate.ejs', 'utf8', send);
When calling readFile asynchronously there is no default encoding, and instead returns the raw buffer. Currently, this buffer is being sent to the EJS render call and failing.
See the node documentation for readFile for more information.
I'm trying to improve my understanding of callbacks/error handling/async structure of Node.js working on challenges at nodeschool.io.
I have one local module file and one actual program file. Here they are:
my_module.js
module.exports = function(path, extension, callback) {
var fs = require('fs');
if (path) {
fs.readdir(path, function(err, list) {
if (err) return callback(err);
var filtered = list.filter(function(item) {
//console.log(item.split("."));
if (item.split(".")[1] == extension) return item
})
callback(null, filtered);
})
} else {
callback("===> Please provide a directory path.")
}
}
program.js
var my_module = require('./my_module');
var path = process.argv[2];
var extension = process.argv[3];
my_module(path, extension, function(err, data) {
if (err) return new Error(err);
console.log(data.join("\n"));
})
This program works just fine. BUT when it should give an error, it doesn't.
Inside my_module.js, if path variable is empty I want to give the error "Please provide a directory path.". The weird thing is, when I write console.log instread of callback, I can show the message. But when I call the callback as callback("===> Please provide a directory path.") nothing happens. No errors, it's silent.
Why is this? And how to fix that?
Thanks
It is silent because in your program.js, you are not consuming the error.
Think program.js is going to consume your module. module.js doesn't know what to do with error. it just passes the err and data to your program.js which consumes it. So its the responsibility of program.js to do whatever it wants to with the error. Something like log it on console, etc.
Try this:
my_module(path, extension, function(err, data) {
if (err) return console.log(err);
console.log(data.join("\n"));
})
You may want to throw that error rather than return it, depending on the error handling pattern for your application.
var my_module = require('./my_module');
var path = process.argv[2];
var extension = process.argv[3];
my_module(path, extension, function(err, data) {
if (err) throw new Error(err);
console.log(data.join("\n"));
})
As a recommendation, since process.arg[] contains user-specified input, I'd actually recommend doing the parameter checking with user-friendly (pretty) error messages (rather than an exception) BEFORE calling my_module(). my_module() itself can then be modified to check and throw an exception if any bogus parameters are passed in.