This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
Closed 2 years ago.
I'm new to javascript and especially node and would appreciate some help.
I have the following code which looks up delivery status for various packages. Everything works fine except the final step, which is writing the results to a CSV file using the csv-writer package from npm.
I process each line of the source CSV and populate an array with the results. The array is declared at the top of the file as a const (final_result) outside of any function. In my understanding, this means it can be accessed by all the functions in the code. Is this right?
The problem I'm having is that when I get to the final step (writing the results to a new CSV file), the final_result length is 0 and contains no data !
When I console-log in the first and second functions, there is data showing.I can also see the array being populated as each row is processed
What am I missing or is there a better way (best practice? ) to achieve this. Thanks.
const createCsvWriter = require('csv-writer').createObjectCsvWriter;
const yt = require('yodel-tracking');
const csv = require('csv-parser');
const fs = require('fs');
const final_result = []; // This is showing as empty at the end ?!
const csvWriter = createCsvWriter({
path: 'out.csv',
header: [
{ id: 'Location', title: 'Location' },
{ id: 'Ref', title: 'Ref' },
{ id: 'Scan', title: 'Scan' },
{ id: 'Status', title: 'Status' }
]
});
//Functions in order of how they are called
//1 of 3
function run() {
fs.createReadStream('yodeldata.csv')
.on('error', () => console.log('error'))
.pipe(csv())
.on('data', row => {
output(row); //calling function 2 of 3
});
}
//2 of 3
function output(row) {
yt(row['REF'], 'GB', function(err, result) {
let data = result.parcel_tracking.parcel_status[0];
if (!err) {
let obj = {
Ref: `${row['REF']}`,
Location: `${data.location}`,
Scan: `${data.scan_date} ${data.scan_time}`,
Status: `${data.status_description}`
};
final_result.push(obj); //Pushing each result to the array here
console.log(final_result.length); // I can see the length increasing sequentially here
}
});
}
//3 of 3
function resultToCsv() {
console.log('Number of records to be written to csv: ' + final_result.length); // Why is this Zero ?? Expected an array of objects here
csvWriter
.writeRecords(final_result)
.then(() => console.log('The CSV file was written successfully'));
}
//Function calls
run();
resultToCsv();
This issue here is that when you do this:
run();
resultToCsv();
Those two functions are being called immediately. Because run() is using asynchronous methods, it won't have finished reading the stream before resultToCsv() is called. So you want to wait for the stream to be done being read before calling that.
function run() {
fs.createReadStream('yodeldata.csv')
.on('error', () => console.log('error'))
.pipe(csv())
.on('data', row => {
output(row); //calling function 2 of 3
})
.on('close', resultToCsv);
}
Related
i want to fetch the old values from file Json if my script turns off from work and then insert the new values into the json file. i read the file and then converted the JSON file the code works safely, but i have a problem that it prints for me once
const dataFile = fs.readFileSync('data.json', 'utf8')
let decodeJson = await JSON.parse(dataFile);
decodeJson.table.push({name: 'test'})
this line is inside forEach loop so it is supposed to print 25 lines for each word test cuz after i want to write it in the same file data.json
this line for forEach
getData.data.forEach(async details => {
let currentPage = details.title;
const dataFile = fs.readFileSync('data.json', 'utf8')
if (dataFile === '') { // check if file json empty
try {
obj.table.push({
name: details.title
})
let encodeJson = await JSON.stringify(obj);
fs.writeFileSync("data.json", encodeJson)
console.log('file is empty')
} catch (err) {
console.log(err);
}
} else {
//read data with JSON from file
try {
let decodeJson = await JSON.parse(dataFile);
// i want here keep old values + push new values (details.title)
decodeJson.table.push({
name: 'test'
})
//fs.writeFileSync('data.json', JSON.stringify(decodeJson))
} catch (err) {
console.log(err);
}
}
})
i tried some attempts but i failed and it still print me only one object...why?
let data = []
data.push('test')
decodeJson.table.push(data)
this is the result i got in consolelog
table: [
{ name: 'Kimi ga Nozomu Eien' },
{ name: 'Kita e.: Diamond Dust Drops' },
{ name: 'Loveless' },
{ name: 'Blood+' },
{ name: 'Re: Cutey Honey' },
{ name: 'Solty Rei' },
{ name: 'Juuni Kokuki' },
{ name: 'Shaman King' },
{ name: 'X/1999' },
{ name: 'X' },
{ name: 'Mahou Sensei Negima!' },
{ name: 'Maria-sama ga Miteru' },
{ name: 'Boukyaku no Senritsu' },
{ name: 'Ima, Soko ni Iru Boku' },
{ name: 'Peace Maker Kurogane' },
{ name: 'Pita Ten' },
{ name: 'Power Stone' },
{ name: 'Mononoke Hime' },
{ name: 'RahXephon' },
{ name: 'Samurai 7' },
{ name: 'Scrapped Princess' },
{ name: 's.CRY.ed' },
{ name: 'Shingetsutan Tsukihime' },
{ name: 'Slam Dunk' },
{ name: 'Strange Dawn' },
[ 'test' ]
]
}
also my goal that i want to do is if script stopped and start back work. it returns back the old object that was stored in the Json file + keeps adding a new object from the API i'm still a newbie and learning json so please treat me kindly
forEach loops are synchronous
As already commented, neither JSON.parse nor JSON.stringify is asynchronous, so you don't need to use await for them.
Using await is causing the problem: array.prototype.forEach treats its argument as a synchronous function and discards any value returned from it. Effectively this means any promises returned from an async function provided are discarded without waiting for the promise to be settled.
Also async function executing an await operator synchronously return a promise the first time an await operator is executed within them.
Hence if data.json is empty, the await in
let encodeJson = await JSON.stringify(obj);
returns a promise before creating data.json content, which is sufficient for the forEach loop to continue. All remaining iterations will do the same and when resumed will overwrite the data.json file created in the previous iteration. However, since each iteration updated an outer variable obj, the last version of data.json should be correct.
Similar conditions apply to an existing data file however: each loop iteration reads the existing file and returns a promise when
await JSON.parse(dataFile);
is executed, allowing the next iteration to proceed and read the same input data and overwrite the file written by the previous iteration.
Major lessons are to not use forEach for asynchronous work that needs to complete within a single iteration, and not to use await in code that must execute synchronously.
In this particular case, leaving out the await operators, staying with synchronous writes, and removing async before the loop function declaration should be sufficient to solve the issue. A better solution may be to read data.json once before starting the loop, and write it once after finishing the loop.
What I'm trying to do
Requests come into my server to download a file containing data. The downloading part is in the front-end and works. I grab the data on my backend and then I want to write it into an existing template and return the data.
This is the handler for the request.
async handle(request: Request, response: Response) {
try {
const fileName = 'test.xlsx'
const binary = objectsToTemplateWorkBook()
response.setHeader(
'Content-Disposition',
'attachment; filename=' + fileName
)
response.setHeader('Content-Type', 'application/vnd.openxmlformats')
response.end(binary, 'binary')
} catch (error) {
console.log(error)
response.send(error)
}
}
This is the function that is supposed to write the data into the template.
export const objectsToTemplateWorkBook = ():
Promise<any> => {
var XlsxTemplate = require('xlsx-template')
var dataBlob
// Load an XLSX file into memory
const blob = fs.readFile(
path.join(__dirname, 'test_template.xlsx'),
function (err, data) {
console.log(__dirname)
// Create a template
var template = new XlsxTemplate(data)
// Replacements take place on first sheet
var sheetNumber = 1
// Set up some placeholder values matching the placeholders in the template
var values = {
people: [
{ name: 'John Smith', age: 20 },
{ name: 'Bob Johnson', age: 22 },
],
}
// Perform substitution
template.substitute(sheetNumber, values)
// Get binary data
dataBlob = template.generate()
// ...
}
)
return dataBlob
}
The function seems to write the data to the template because if I log the dataBlob inside the fs.Readfile method it shows me the file. However, the return dataBlob always returns undefined. I know this is due to the async nature, but I have no idea how to fix it quite honestly. So my question to you is: how can I get the dataBlob to my handler to send it as a response?
You can't get the return from a callback function like you're doing here, since they run asynchronously, their return will never be acessible because the external return will be executed before the inner code.
To solve this specific problem you can you the fs.readFileSync function, that executes synchronously and returns a value, that being the buffer you need to pass in your xlsxTemplate constructor. This way, the code turns into:
export const objectsToTemplateWorkBook = ():
Promise<any> => {
var XlsxTemplate = require('xlsx-template')
var dataBlob
// Load an XLSX file into memory
const data = fs.readFileSync(path.join(__dirname, 'test_template.xlsx'))
console.log(__dirname)
// Create a template
var template = new XlsxTemplate(data)
// Replacements take place on first sheet
var sheetNumber = 1
// Set up some placeholder values matching the placeholders in the template
var values = {
people: [
{ name: 'John Smith', age: 20 },
{ name: 'Bob Johnson', age: 22 },
],
}
// Perform substitution
template.substitute(sheetNumber, values)
// Get binary data
dataBlob = template.generate()
// ...
return dataBlob
}
With this you get access to the file buffer returned from the synchronous read file and is able to perform the rest of your operations. Hope it helps :D
The Node.js function below takes:
an object, shop which contains a regular expression
an array of filenames
The function will read each csv file listed in the array and test a cell in the first row with a regular expression, returning a new array of matching filenames.
function matchReport(shop, arr) {
return promise = new Promise(resolve => {
var newArray = [];
for(var j=0;j<arr.length;++j) {
let filename = arr[j];
csv()
.fromFile(filename)
.then(reportData => {
if (reportData[0]['Work'].match(shop.productRegex)) {
newArray.push(filename);
}
if (j === arr.length) {
resolve(newArray);
}
});
}
}).then(matches => {
return {
'shop' : shop.name,
'reports' : matches
}
}).catch(e => {
console.log(e);
});
}
Very rarely the function will return with the correct behavior which is this:
{ shop: 'shop1',
reports: [ '../artist-sales-report-2020-11-12(1).csv' ] }
{ shop: 'shop2',
reports:
[ '../artist-sales-report-2020-12-03.csv',
'../artist-sales-report-2020-09-01.csv' ] }
More often it returns with reports missing, like below:
{ shop: 'shop1',
reports: [ '../artist-sales-report-2020-11-12(1).csv' ] }
{ shop: 'shop2',
reports: [ '../artist-sales-report-2020-12-03.csv' ] }
I understand where the problem is taking place, within the csv reportData block. I understand that it is an asynchronous issue and I have tried to write more elaborate if..then or switch statements as a hack solution with no luck. It seems a little sloppy and cluttered to me to create more promises inside of this promise but I have been unsuccessful at that as well.
Using async/await and your disliked nested promises you could simplify your code to something like this, which should always await all results. I made the assumption that your problem is the fromFile method, which feels like it is itself asynchronous since it uses a then that you are not awaiting.
async function matchReport(shop, arr) {
const matches = await Promise.all(arr.map(async filename => {
const reportData = await csv().fromFile( filename );
if( reportData[0]['Work'].match(shop.productRegex) ){
return filename;
}
}));
return {
'shop': shop.name,
'reports': matches.filter( Boolean )
};
}
question is possibly a duplicate but I haven't found anything that provides an appropriate answer to my issue.
I have an ExpressJS server which is used to provide API requests to retrieve data from a MongoDB database. I am using mongoosejs for the MongoDB connection to query/save data.
I am building a route that will allow me to find all data that matches some user input but I am having trouble when doing the query. I have spent a long while looking online for someone with a similar issue but coming up blank.
I will leave example of the code I have at the minute below.
code for route
// -- return matched data (GET)
router.get('/match', async (req, res) => {
const style_data = req.query.style; // grab url param for style scores ** this comes in as a string **
const character_data = req.query.character; // grab url param for character scores ** this comes in as a string **
// run matcher systems
const style_matches = style_match(style_data);
res.send({
response: 200,
data: style_matches
}); // return data
});
code for the query
// ---(Build the finder)
const fetch_matches_using = async function(body, richness, smoke, sweetness) {
return await WhiskyModel.find({
'attributes.body': body,
'attributes.richness': richness,
'attributes.smoke': smoke,
'attributes.sweetness': sweetness
});
}
// ---(Start match function)---
const style_match = async function (scores_as_string) {
// ---(extract data)---
const body = scores_as_string[0];
const richness = scores_as_string[1];
const smoke = scores_as_string[2];
const sweetness = scores_as_string[3];
const matched = [];
// ---(initialise variables)---
let match_count = matched.length;
let first_run; // -> exact matches
let second_run; // -> +- 1
let third_run; // -> +- 2
let fourth_run; // -> +- 3
// ---(begin db find loop)---
first_run = fetch_matches_using(body, richness, smoke, sweetness).then((result) => {return result});
matched.push(first_run);
// ---(return final data)---
return matched
}
example of db object
{
_id: mongoid,
meta-data: {
pagemd:{some data},
name: whiskyname
age: whiskyage,
price: price
},
attributes: {
body: "3",
richness: "3",
smoke: "0",
sweetness: "3",
some other data ...
}
}
When I hit the route in postman the JSON data looks like:
{
response: 200,
data: {}
}
and when I console.log() out matched from within the style match function after I have pushed the it prints [ Promise(pending) ] which I don't understand.
if I console.log() the result from within the .then() I get an empty array.
I have tried using the populate() method after running the find which does technically work, but instead of only returning data that matches it returns every entry in the collection so I think I am doing something wrong there, but I also don't see why I would need to use the .populate() function to access the nested object.
Am I doing something totally wrong here?
I should also mention that the route and the matching functions are in different files just to try and keep things simple.
Thanks for any answers.
just posting an answer as I seem to have fixed this.
Issue was with my .find() function, needed to pass in the items to search by and then also a call back within the function to return error/data. I'll leave the changed code below.
new function
const fetch_matches_using = async function(body, richness, smoke, sweetness) {
const data = await WhiskyModel.find({
'attributes.body': body,
'attributes.richness': richness,
'attributes.smoke': smoke,
'attributes.sweetness': sweetness
}, (error, data) => { // new ¬
if (error) {
return error;
}
if (data) {
console.log(data)
return data
}
});
return data; //new
}
There is still an issue with sending the found results back to the route but this is a different issue I believe. If its connected I'll edit this answer with the fix for that.
In the project that I am working on, built using nodejs & mongo, there is a function that takes in a query and returns set of data based on limit & offset provided to it. Along with this data the function returns a total count stating all the matched objects present in the database. Below is the function:
// options carry the limit & offset values
// mongoQuery carries a mongo matching query
function findMany(query, options, collectionId) {
const cursor = getCursorForCollection(collectionId).find(query, options);
return Promise.all([findManyQuery(cursor), countMany(cursor)]);
}
Now the problem with this is sometime when I give a large limit size I get an error saying:
Uncaught exception: TypeError: Cannot read property '_killCursor' of undefined
At first I thought I might have to increase the pool size in order to fix this issue but after digging around a little bit more I was able to find out that the above code is resulting in a race condition. When I changed the code to:
function findMany(query, options, collectionId) {
const cursor = getCursorForCollection(collectionId).find(query, options);
return findManyQuery(cursor).then((dataSet) => {
return countMany(cursor).then((count)=> {
return Promise.resolve([dataSet, count]);
});
);
}
Everything started working perfectly fine. Now, from what I understand with regard to Promise.all was that it takes an array of promises and resolves them one after the other. If the promises are executed one after the other how can the Promise.all code result in race condition and the chaining of the promises don't result in that.
I am not able to wrap my head around it. Why is this happening?
Since I have very little information to work with, I made an assumption of what you want to achieve and came up with the following using Promise.all() just to demonstrate how you should use Promise.all (which will resolve the array of promises passed to it in no particular order. For this reason, there must be no dependency in any Promise on the order of execution of the Promises. Read more about it here).
// A simple function to sumulate findManyQuery for demo purposes
function findManyQuery(cursors) {
return new Promise((resolve, reject) => {
// Do your checks and run your code (for example)
if (cursors) {
resolve({ dataset: cursors });
} else {
reject({ error: 'No cursor in findManyQuery function' });
}
});
}
// A simple function to sumulate countMany for demo purposes
function countMany(cursors) {
return new Promise((resolve, reject) => {
// Do your checks and run your code (for example)
if (cursors) {
resolve({ count: cursors.length });
} else {
reject({ error: 'No cursor in countMany' });
}
});
}
// A simple function to sumulate getCursorForCollection for demo purposes
function getCursorForCollection(collectionId) {
/*
Simulating the returned cursor using an array of objects
and the Array filter function
*/
return [{
id: 1,
language: 'Javascript',
collectionId: 99
}, {
id: 2,
language: 'Dart',
collectionId: 100
},
{
id: 3,
language: 'Go',
collectionId: 100
}, {
id: 4,
language: 'Swift',
collectionId: 99
}, {
id: 5,
language: 'Kotlin',
collectionId: 101
},
{
id: 6,
language: 'Python',
collectionId: 100
}].filter((row) => row.collectionId === collectionId)
}
function findMany(query = { id: 1 }, options = [], collectionId = 0) {
/*
First I create a function to simulate the assumed use of
query and options parameters just for demo purposes
*/
const filterFunction = function (collectionDocument) {
return collectionDocument.collectionId === query.id && options.indexOf(collectionDocument.language) !== -1;
};
/*
Since I am working with arrays, I replaced find function
with filter function just for demo purposes
*/
const cursors = getCursorForCollection(collectionId).filter(filterFunction);
/*
Using Promise.all([]). NOTE: You should pass the result of the
findManyQuery() to countMany() if you want to get the total
count of the resulting dataset
*/
return Promise.all([findManyQuery(cursors), countMany(cursors)]);
}
// Consuming the findMany function with test parameters
const query = { id: 100 };
const collectionId = 100;
const options = ['Javascript', 'Python', 'Go'];
findMany(query, options, collectionId).then(result => {
console.log(result); // Result would be [ { dataset: [ [Object], [Object] ] }, { count: 2 } ]
}).catch((error) => {
console.log(error);
});
There are ways to write this function in a "pure" way for scalability and testing.
So here's your concern:
In the project that I am working on, built using nodejs & mongo, there is a function that takes in a query and returns set of data based on limit & offset provided to it. Along with this data the function returns a total count stating all the matched objects present in the database.
Note: You'll need to take care of edge case.
const Model = require('path/to/model');
function findManyUsingPromise(model, query = {}, offset = 0, limit = 10) {
return new Promise((resolve, reject) => {
model.find(query, (error, data) => {
if(error) {
reject(error);
}
resolve({
data,
total: data.length || 0
});
}).skip(offset).limit(limit);
});
}
// Call function
findManyUsingPromise(Model, {}, 0, 40).then((result) => {
// Do something with result {data: [object array], total: value }
}).catch((err) => {
// Do something with the error
});