i want to fetch the old values from file Json if my script turns off from work and then insert the new values into the json file. i read the file and then converted the JSON file the code works safely, but i have a problem that it prints for me once
const dataFile = fs.readFileSync('data.json', 'utf8')
let decodeJson = await JSON.parse(dataFile);
decodeJson.table.push({name: 'test'})
this line is inside forEach loop so it is supposed to print 25 lines for each word test cuz after i want to write it in the same file data.json
this line for forEach
getData.data.forEach(async details => {
let currentPage = details.title;
const dataFile = fs.readFileSync('data.json', 'utf8')
if (dataFile === '') { // check if file json empty
try {
obj.table.push({
name: details.title
})
let encodeJson = await JSON.stringify(obj);
fs.writeFileSync("data.json", encodeJson)
console.log('file is empty')
} catch (err) {
console.log(err);
}
} else {
//read data with JSON from file
try {
let decodeJson = await JSON.parse(dataFile);
// i want here keep old values + push new values (details.title)
decodeJson.table.push({
name: 'test'
})
//fs.writeFileSync('data.json', JSON.stringify(decodeJson))
} catch (err) {
console.log(err);
}
}
})
i tried some attempts but i failed and it still print me only one object...why?
let data = []
data.push('test')
decodeJson.table.push(data)
this is the result i got in consolelog
table: [
{ name: 'Kimi ga Nozomu Eien' },
{ name: 'Kita e.: Diamond Dust Drops' },
{ name: 'Loveless' },
{ name: 'Blood+' },
{ name: 'Re: Cutey Honey' },
{ name: 'Solty Rei' },
{ name: 'Juuni Kokuki' },
{ name: 'Shaman King' },
{ name: 'X/1999' },
{ name: 'X' },
{ name: 'Mahou Sensei Negima!' },
{ name: 'Maria-sama ga Miteru' },
{ name: 'Boukyaku no Senritsu' },
{ name: 'Ima, Soko ni Iru Boku' },
{ name: 'Peace Maker Kurogane' },
{ name: 'Pita Ten' },
{ name: 'Power Stone' },
{ name: 'Mononoke Hime' },
{ name: 'RahXephon' },
{ name: 'Samurai 7' },
{ name: 'Scrapped Princess' },
{ name: 's.CRY.ed' },
{ name: 'Shingetsutan Tsukihime' },
{ name: 'Slam Dunk' },
{ name: 'Strange Dawn' },
[ 'test' ]
]
}
also my goal that i want to do is if script stopped and start back work. it returns back the old object that was stored in the Json file + keeps adding a new object from the API i'm still a newbie and learning json so please treat me kindly
forEach loops are synchronous
As already commented, neither JSON.parse nor JSON.stringify is asynchronous, so you don't need to use await for them.
Using await is causing the problem: array.prototype.forEach treats its argument as a synchronous function and discards any value returned from it. Effectively this means any promises returned from an async function provided are discarded without waiting for the promise to be settled.
Also async function executing an await operator synchronously return a promise the first time an await operator is executed within them.
Hence if data.json is empty, the await in
let encodeJson = await JSON.stringify(obj);
returns a promise before creating data.json content, which is sufficient for the forEach loop to continue. All remaining iterations will do the same and when resumed will overwrite the data.json file created in the previous iteration. However, since each iteration updated an outer variable obj, the last version of data.json should be correct.
Similar conditions apply to an existing data file however: each loop iteration reads the existing file and returns a promise when
await JSON.parse(dataFile);
is executed, allowing the next iteration to proceed and read the same input data and overwrite the file written by the previous iteration.
Major lessons are to not use forEach for asynchronous work that needs to complete within a single iteration, and not to use await in code that must execute synchronously.
In this particular case, leaving out the await operators, staying with synchronous writes, and removing async before the loop function declaration should be sufficient to solve the issue. A better solution may be to read data.json once before starting the loop, and write it once after finishing the loop.
Related
I Have the follow object which iterating it I can add a new key and value, but when I have to make a database search and take each result and add as a new key + value does not work, the Eslint point a Assignment to property of function parameter 'object'.eslintno-param-reassign, bellow is the example without make a database search.
const obj = {
Cod: 43453,
Informations: [
{
codPay: 802112,
value: 80
},
{
codPay: 802113,
value: 80
},
{
codPay: 802114,
value: 80
},
{
codPay: 802115,
value: 80
},
{
codPay: 802116,
value: 80
},
{
codPay: 802117,
value: 80
},
{
codPay: 802118,
value: 80
},
{
codPay: 802119,
value: 80
}
],
status: 'OK'
}
obj["Informations"].forEach((f,index) => {
f.more_info = index
})
console.log(obj)
By making database search:
obj["Informations"].forEach(asyn(f) => {
const result = db.any("Select info From table where cod $1",f.codPay")
f.more_info = result[0].info
})
by openning this db.any the values is coming correct but I cannot add new key, where Am I missing?
The no-param-reassign rule is there to warn you that mutating the argument may cause undesirable effects elsewhere in the code. For example, there may be some other part of the code that relies on the assumption that each Informations sub-object contains only codPay and value properties, and nothing else (no more_info property). If such code existed, your addition of a more_info property could cause that code to stop working.
A good alternative is to use .map, to create entirely new objects instead of mutating the original ones. That will fix the linter warning.
Another issue is that it looks like db.any returns a Promise, in which case you should wait for it to resolve first. To do this in a loop, use Promise.all to wait for all Promises in an array to resolve.
const informationsWithMoreInfo = await Promise.all(
obj.Informations.map(
obj => db.any("Select info From table where cod $1", obj.codPay)
.then(result => ({ ...obj, more_info: result[0].info }))
)
);
// do stuff with informationsWithMoreInfo
For the await to work, make sure the containing function is async - either that, or call .then on the Promise.
The Node.js function below takes:
an object, shop which contains a regular expression
an array of filenames
The function will read each csv file listed in the array and test a cell in the first row with a regular expression, returning a new array of matching filenames.
function matchReport(shop, arr) {
return promise = new Promise(resolve => {
var newArray = [];
for(var j=0;j<arr.length;++j) {
let filename = arr[j];
csv()
.fromFile(filename)
.then(reportData => {
if (reportData[0]['Work'].match(shop.productRegex)) {
newArray.push(filename);
}
if (j === arr.length) {
resolve(newArray);
}
});
}
}).then(matches => {
return {
'shop' : shop.name,
'reports' : matches
}
}).catch(e => {
console.log(e);
});
}
Very rarely the function will return with the correct behavior which is this:
{ shop: 'shop1',
reports: [ '../artist-sales-report-2020-11-12(1).csv' ] }
{ shop: 'shop2',
reports:
[ '../artist-sales-report-2020-12-03.csv',
'../artist-sales-report-2020-09-01.csv' ] }
More often it returns with reports missing, like below:
{ shop: 'shop1',
reports: [ '../artist-sales-report-2020-11-12(1).csv' ] }
{ shop: 'shop2',
reports: [ '../artist-sales-report-2020-12-03.csv' ] }
I understand where the problem is taking place, within the csv reportData block. I understand that it is an asynchronous issue and I have tried to write more elaborate if..then or switch statements as a hack solution with no luck. It seems a little sloppy and cluttered to me to create more promises inside of this promise but I have been unsuccessful at that as well.
Using async/await and your disliked nested promises you could simplify your code to something like this, which should always await all results. I made the assumption that your problem is the fromFile method, which feels like it is itself asynchronous since it uses a then that you are not awaiting.
async function matchReport(shop, arr) {
const matches = await Promise.all(arr.map(async filename => {
const reportData = await csv().fromFile( filename );
if( reportData[0]['Work'].match(shop.productRegex) ){
return filename;
}
}));
return {
'shop': shop.name,
'reports': matches.filter( Boolean )
};
}
This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
Closed 2 years ago.
I'm new to javascript and especially node and would appreciate some help.
I have the following code which looks up delivery status for various packages. Everything works fine except the final step, which is writing the results to a CSV file using the csv-writer package from npm.
I process each line of the source CSV and populate an array with the results. The array is declared at the top of the file as a const (final_result) outside of any function. In my understanding, this means it can be accessed by all the functions in the code. Is this right?
The problem I'm having is that when I get to the final step (writing the results to a new CSV file), the final_result length is 0 and contains no data !
When I console-log in the first and second functions, there is data showing.I can also see the array being populated as each row is processed
What am I missing or is there a better way (best practice? ) to achieve this. Thanks.
const createCsvWriter = require('csv-writer').createObjectCsvWriter;
const yt = require('yodel-tracking');
const csv = require('csv-parser');
const fs = require('fs');
const final_result = []; // This is showing as empty at the end ?!
const csvWriter = createCsvWriter({
path: 'out.csv',
header: [
{ id: 'Location', title: 'Location' },
{ id: 'Ref', title: 'Ref' },
{ id: 'Scan', title: 'Scan' },
{ id: 'Status', title: 'Status' }
]
});
//Functions in order of how they are called
//1 of 3
function run() {
fs.createReadStream('yodeldata.csv')
.on('error', () => console.log('error'))
.pipe(csv())
.on('data', row => {
output(row); //calling function 2 of 3
});
}
//2 of 3
function output(row) {
yt(row['REF'], 'GB', function(err, result) {
let data = result.parcel_tracking.parcel_status[0];
if (!err) {
let obj = {
Ref: `${row['REF']}`,
Location: `${data.location}`,
Scan: `${data.scan_date} ${data.scan_time}`,
Status: `${data.status_description}`
};
final_result.push(obj); //Pushing each result to the array here
console.log(final_result.length); // I can see the length increasing sequentially here
}
});
}
//3 of 3
function resultToCsv() {
console.log('Number of records to be written to csv: ' + final_result.length); // Why is this Zero ?? Expected an array of objects here
csvWriter
.writeRecords(final_result)
.then(() => console.log('The CSV file was written successfully'));
}
//Function calls
run();
resultToCsv();
This issue here is that when you do this:
run();
resultToCsv();
Those two functions are being called immediately. Because run() is using asynchronous methods, it won't have finished reading the stream before resultToCsv() is called. So you want to wait for the stream to be done being read before calling that.
function run() {
fs.createReadStream('yodeldata.csv')
.on('error', () => console.log('error'))
.pipe(csv())
.on('data', row => {
output(row); //calling function 2 of 3
})
.on('close', resultToCsv);
}
I want the "topic1" to be the value of my breed name and key, but when I try to put this.topic1 to replace the manual typing, it shows nothing.
Or there are any other method to have my button name same as my retrieve API param, and sent it name when I click it?
new Vue({
el: '#app2',
components: { Async },
data() {
return {
topic1: null,
topic2: null,
currentBreed: 0,
breeds: [
{ name: this.topic1 , key: this.topic1 },
{ name: "German Shepherd", key: "germanshepherd" },
{ name: "Husky", key: "husky" },
{ name: "Pug", key: "pug" },
{ name: "(Error)", key: "error" },
]
}
},
async created() {
try {
this.promise = axios.get(
"https://k67r3w45c4.execute-api.ap-southeast-1.amazonaws.com/TwitterTrends"
);
const res = await this.promise;
this.topic1 = res.data[0].Trends;
this.topic2 = res.data[1].Trends;
} catch (e) {
console.error(e);
}
},
async mounted () {
let test = this.topic1;
},
computed: {
breedKey() {
return this.breeds[this.currentBreed].key;
}
}
})
There are several problems here.
The data function is called just once, when the corresponding Vue instance is created. Within that function you can get a reference to its Vue instance via this. At that point some properties, such as those corresponding to props, will already exist. However, others won't.
The object returned from data is used to create new properties on the instance. In this case you're creating 4 properties: topic1, topic2, currentBreed and breeds. Vue creates those properties based on that returned object, so they won't exist until after the data function is run.
So when you write { name: this.topic1 , key: this.topic1 }, within that data function you're attempting to access a property called topic1 that doesn't exist yet. As such it will have a value of undefined. So you're creating an entry equivalent to { name: undefined , key: undefined },.
Further, there is no link back to topic1. That object won't be updated when the value of topic1 changes.
It's also worth noting a few points about timing.
The data function will be called before the created hook, so the axios call isn't made until after the data properties are populated.
An axios call is asynchronous.
Using await may make the code a little easier to read but the 'waiting' is mostly just an illusion. The remaining code inside the function won't run until the awaited promise is resolved but that won't cause anything outside of the function to wait. await is equivalent to using then.
The component will render just after the created hook is called. This is synchronous, it won't wait for the axios request. The mounted hook will then be called, all before the axios call has completed.
All of this means you may need to adjust your template to handle the case where the axios call hasn't completed yet as it will initially render prior to the values of topic1 and topic2 being available.
Specifically addressing the breeds property you have a few options. One is to inject the values in once the value has loaded:
breeds: [
{ name: "" , key: "" }, // Initially empty values
{ name: "German Shepherd", key: "germanshepherd" },
// ...
const res = await this.promise;
this.topic1 = res.data[0].Trends;
this.topic2 = res.data[1].Trends;
this.breeds[0].name = this.breeds[0].key = this.topic1;
Another is to use a computed property for breeds (you'd remove it from the data for this):
computed: {
breeds () {
return [
{ name: this.topic1 , key: this.topic1 },
{ name: "German Shepherd", key: "germanshepherd" },
{ name: "Husky", key: "husky" },
{ name: "Pug", key: "pug" },
{ name: "(Error)", key: "error" },
]
}
}
As we're using a computed property it will be updated when topic1 changes as it's a reactive dependency.
Using a computed property is probably the most natural solution in this case but there are other tricks you can use to get this to work.
For example, you could use property getters for the two properties in that first breed object (that's JavaScript property getters, nothing to do with Vue):
data () {
const vm = this;
return {
topic1: null,
topic2: null,
currentBreed: 0,
breeds: [
{
get name () {
return vm.topic1;
},
get key () {
return vm.topic1;
}
},
{ name: "German Shepherd", key: "germanshepherd" },
{ name: "Husky", key: "husky" },
{ name: "Pug", key: "pug" },
{ name: "(Error)", key: "error" },
]
}
},
I'm not advocating this approach for your use case but it is an interesting way to do it that can sometimes be useful. The key thing to note is how the dependency on topic1 is evaluated only when the properties name and key are accessed, not when the data function is executed. This allows topic1 to be registered as a dependency of whatever is accessing name and key, e.g. during rendering.
In the project that I am working on, built using nodejs & mongo, there is a function that takes in a query and returns set of data based on limit & offset provided to it. Along with this data the function returns a total count stating all the matched objects present in the database. Below is the function:
// options carry the limit & offset values
// mongoQuery carries a mongo matching query
function findMany(query, options, collectionId) {
const cursor = getCursorForCollection(collectionId).find(query, options);
return Promise.all([findManyQuery(cursor), countMany(cursor)]);
}
Now the problem with this is sometime when I give a large limit size I get an error saying:
Uncaught exception: TypeError: Cannot read property '_killCursor' of undefined
At first I thought I might have to increase the pool size in order to fix this issue but after digging around a little bit more I was able to find out that the above code is resulting in a race condition. When I changed the code to:
function findMany(query, options, collectionId) {
const cursor = getCursorForCollection(collectionId).find(query, options);
return findManyQuery(cursor).then((dataSet) => {
return countMany(cursor).then((count)=> {
return Promise.resolve([dataSet, count]);
});
);
}
Everything started working perfectly fine. Now, from what I understand with regard to Promise.all was that it takes an array of promises and resolves them one after the other. If the promises are executed one after the other how can the Promise.all code result in race condition and the chaining of the promises don't result in that.
I am not able to wrap my head around it. Why is this happening?
Since I have very little information to work with, I made an assumption of what you want to achieve and came up with the following using Promise.all() just to demonstrate how you should use Promise.all (which will resolve the array of promises passed to it in no particular order. For this reason, there must be no dependency in any Promise on the order of execution of the Promises. Read more about it here).
// A simple function to sumulate findManyQuery for demo purposes
function findManyQuery(cursors) {
return new Promise((resolve, reject) => {
// Do your checks and run your code (for example)
if (cursors) {
resolve({ dataset: cursors });
} else {
reject({ error: 'No cursor in findManyQuery function' });
}
});
}
// A simple function to sumulate countMany for demo purposes
function countMany(cursors) {
return new Promise((resolve, reject) => {
// Do your checks and run your code (for example)
if (cursors) {
resolve({ count: cursors.length });
} else {
reject({ error: 'No cursor in countMany' });
}
});
}
// A simple function to sumulate getCursorForCollection for demo purposes
function getCursorForCollection(collectionId) {
/*
Simulating the returned cursor using an array of objects
and the Array filter function
*/
return [{
id: 1,
language: 'Javascript',
collectionId: 99
}, {
id: 2,
language: 'Dart',
collectionId: 100
},
{
id: 3,
language: 'Go',
collectionId: 100
}, {
id: 4,
language: 'Swift',
collectionId: 99
}, {
id: 5,
language: 'Kotlin',
collectionId: 101
},
{
id: 6,
language: 'Python',
collectionId: 100
}].filter((row) => row.collectionId === collectionId)
}
function findMany(query = { id: 1 }, options = [], collectionId = 0) {
/*
First I create a function to simulate the assumed use of
query and options parameters just for demo purposes
*/
const filterFunction = function (collectionDocument) {
return collectionDocument.collectionId === query.id && options.indexOf(collectionDocument.language) !== -1;
};
/*
Since I am working with arrays, I replaced find function
with filter function just for demo purposes
*/
const cursors = getCursorForCollection(collectionId).filter(filterFunction);
/*
Using Promise.all([]). NOTE: You should pass the result of the
findManyQuery() to countMany() if you want to get the total
count of the resulting dataset
*/
return Promise.all([findManyQuery(cursors), countMany(cursors)]);
}
// Consuming the findMany function with test parameters
const query = { id: 100 };
const collectionId = 100;
const options = ['Javascript', 'Python', 'Go'];
findMany(query, options, collectionId).then(result => {
console.log(result); // Result would be [ { dataset: [ [Object], [Object] ] }, { count: 2 } ]
}).catch((error) => {
console.log(error);
});
There are ways to write this function in a "pure" way for scalability and testing.
So here's your concern:
In the project that I am working on, built using nodejs & mongo, there is a function that takes in a query and returns set of data based on limit & offset provided to it. Along with this data the function returns a total count stating all the matched objects present in the database.
Note: You'll need to take care of edge case.
const Model = require('path/to/model');
function findManyUsingPromise(model, query = {}, offset = 0, limit = 10) {
return new Promise((resolve, reject) => {
model.find(query, (error, data) => {
if(error) {
reject(error);
}
resolve({
data,
total: data.length || 0
});
}).skip(offset).limit(limit);
});
}
// Call function
findManyUsingPromise(Model, {}, 0, 40).then((result) => {
// Do something with result {data: [object array], total: value }
}).catch((err) => {
// Do something with the error
});