I have a werid problem. What i want to do is, when calling an api, i want to make a request to the postgresql db with a bulk insert. First, i execute the loop which reads a file and extracts data from it to form the values array. Then, i want to generate a bulk insert request with pg-promise library. But when the code executes, what i get is that it tries to generate a request before the loop even starts, and it throws an error about an empty array. What the?
async import(req, res) {
var rd = readline.createInterface({
input: fs.createReadStream('.../file.csv'),
console: false
});
const createQuery = `INSERT INTO
table('columns')
VALUES ?`;
const values = [];
rd.on('line', function(line) {
//stuff
values.push({
//stuff
});
}
});
const cs = new pgp.helpers.ColumnSet(['columns'], {table: 'table'});
const query = pgp.helpers.insert(values, cs);
I've removed the details, but i hope this provides enough info. I've tried to put console logs before, in and after the loop, and first, the stuff before and after the loop gets logged, the error is thrown, and then the loop executes and logs stuff. Do i not understand or miss something?
Just making the function async doesn't accomplish anything for you here by itself. You have to find out what asynchronous operations you can await in order to serialize your asynchronous operations.
In the latest versions of node.js, you can use the for await () structure with the asynchronous iterator in the readline interface to process all the lines.
async import(req, res) {
var rd = readline.createInterface({
input: fs.createReadStream('.../file.csv'),
console: false
});
const createQuery = `INSERT INTO
table('columns')
VALUES ?`;
const values = [];
for await (const line of rd) {
values.push({...})
}
const cs = new pgp.helpers.ColumnSet(['columns'], {table: 'table'});
const query = pgp.helpers.insert(values, cs);
}
FYI, you can see an example of this in the readline doc.
You also don't need to use async at all as you can solve it by just putting your last two lines into an event listener for the close event:
import(req, res) {
var rd = readline.createInterface({
input: fs.createReadStream('.../file.csv'),
console: false
});
const createQuery = `INSERT INTO
table('columns')
VALUES ?`;
const values = [];
rd.on('line', function(line) {
//stuff
values.push({
//stuff
});
}
}).on('close', () => {
// done with all line processing here
const cs = new pgp.helpers.ColumnSet(['columns'], {table: 'table'});
const query = pgp.helpers.insert(values, cs);
});
}
When you call rd.on() you are just establishing an event with a callback to be called when the line event occurs. All your code is doing is establishling the callback but then proceeding with the rest of your code which then tries to insert the values in the database. You need to move the code that inserts into the database inside the callback of your rd.on() after you loop through all your values and push them into the array.
However, I'm not familiar of what the line even is in the case of a file. If it is truly line by line for the file then you obviously can't bulk insert there. My suggestion at that point would be to move that step into its own asyn function and await the result of that function before doing the insert.
Related
I'm trying to perform a fetch request within a transaction but when the code executes I receive the following error.
Error: Cannot modify a WriteBatch that has been committed.
The steps the function is performing are the following:
Compute document references (taken from an external source)
Query the documents available in Firestore
Verify if document exists
Fetch for further details (lazy loading mechanism)
Start populating first level collection
Start populating second level collection
Below the code I'm using.
await firestore.runTransaction(async (transaction) => {
// 1. Compute document references
const docRefs = computeDocRefs(colName, itemsDict);
// 2. Query the documents available in Firestore
const snapshots = await transaction.getAll(...docRefs);
snapshots.forEach(async (snapshot) => {
// 3. Verify if document exists
if (!snapshot.exists) {
console.log(snapshot.id + " does not exists");
const item = itemsDict[snapshot.id];
if (item) {
// 4. Fetch for further details
const response = await fetchData(item.detailUrl);
const detailItemsDict = prepareDetailPageData(response);
// 5. Start populating first level collection
transaction.set(snapshot.ref, {
index: item.index,
detailUrl: item.detailUrl,
title: item.title,
});
// 6. Start populating second level collection
const subColRef = colRef.doc(snapshot.id).collection(subColName);
detailItemsDict.detailItems.forEach((detailItem) => {
const subColDocRef = subColRef.doc();
transaction.set(subColDocRef, {
title: detailItem.title,
pdfUrl: detailItem.pdfUrl,
});
});
}
} else {
console.log(snapshot.id + " exists");
}
});
});
computeDocRefs is described below
function computeDocRefs(colName, itemsDict) {
const identifiers = Object.keys(itemsDict);
const docRefs = identifiers.map((identifier) => {
const docId = `${colName}/${identifier}`
return firestore.doc(docId);
});
return docRefs;
}
while fetchData uses axios under the hood
async function fetchData(url) {
const response = await axios(url);
if (response.status !== 200) {
throw new Error('Fetched data failed!');
}
return response;
}
prepareMainPageData and prepareDetailPageData are functions that prepare the data normalizing them.
If I comment the await fetchData(item.detailUrl), the first level collection with all the documents associated to it are stored correctly.
On the contrary with await fetchData(item.detailUrl) the errors happens below the following comment: // 5. Start populating first level collection.
The order of the operation are important since I do now want to make the second call if not necessary.
Are you able to guide me towards the correct solution?
The problem is due to the fact that forEach and async/await do not work well together. For example: Using async/await with a forEach loop.
Now I've completely changed the approach I'm following and now it works smoothly.
The code now is like the following:
// Read transaction to retrieve the items that are not yet available in Firestore
const itemsToFetch = await readItemsToFetch(itemsDict, colName);
// Merge the items previously retrieved to grab additional details through fetch network calls
const fetchedItems = await aggregateItemsToFetch(itemsToFetch);
// Write transaction (Batched Write) to save items into Firestore
const result = await writeFetchedItems(fetchedItems, colName, subColName);
A big thanks goes to Doug Stevenson and Renaud Tarnec.
Greetings of the day to everyone,
So I'm having a really hard time with Firebase cause there's just so many versions and things going on. Its extremely complicated. I wanted to achieve some functionality which is not available through the client modular web 9 version.
So I have been trying to use the Cloud functions to just get the list of all the collections in a document.
My cloud function looks like this so far -
const functions = require("firebase-functions");
const admin = require('firebase-admin');
const { object } = require("firebase-functions/v1/storage");
admin.initializeApp();
const db = admin.firestore();
exports.addMessages = functions.https.onCall(async (data, context) => {
// Grab the text parameter.
const original = data.text;
var test;
const writeResult = admin.firestore().collection('users').doc(original)
const collections = await writeResult.listCollections();
collections.forEach(collection => {
console.log('Found subcollection with id:', collection.id);
test = collection.id;
});
// Send back a message that we've successfully written the message
return { id: test }
});
I call this function on my front end simply like this --
const functions = getFunctions();
const addMessage = httpsCallable(functions, 'addMessages');
const Cloud = () => {
addMessage({ docPath: `users/${userProfile.uid}` })
.then(function (result) {
var collections = result.data.collections;
console.log(collections);
})
.catch(function (error) {
// Getting the Error details.
var code = error.code;
var message = error.message;
var details = error.details;
// ...
});
}
However, I get a 500 ERROR. I have checked the payload and everything, the data is being passed and the function is being called but there is some other kind of bug that I seem to be missing.
If anyone has any ideas please do let me know.
First off, you're calling addMessage with an Object parameter and then trying to access it's payload by the text field of data, which is undefined. You could either pass users/${userProfile.uid} as a string parameter or assign data.docPath to original.
Also, test will only contain the last collection id, since it's being overwritten forEach of the collections. So, may I suggest you make test a string array and push() each collection id to it?
Finally, on the callback of addMessage, there is no data field to access in result. In the case you decide to use an array, result will simply be the array you returned from the cloud function.
P.S. I'm not sure I see a reason why this wouldn't work with the V9 modular SDK.
This question already has answers here:
Using async/await with a forEach loop
(33 answers)
Closed 3 years ago.
await is not blocking as expected, when a block of code updates db (using postgres / node )
https://node-postgres.com
I have a list of async function calls, each call udpates a database, and each subsequent call works on data updated by the previous call.
There are about eight calls in a row, and each call must update the complete set of data it is working with, 100% to completion, before going to the next.
I tried to make everything not async, but it appears I am forced to make everything async/await because of the library I am using (postgres / node).
Each function call must complete 100% before going on to the next function call, because the next step does a select on rows where a field is not null (where the previous step fills in a value).
I have an await in front of each call, that does something (see code below):
loads the db from a csv,
next step selects all rows just inserted, calls an API and updates the database,
and so on,
but at one point, when the next function executes, NONE of the rows have been updated (as I trace through and verify, a SQL statement returns nothing back),
the code seems to pass right through going to the second function call, not blocking, honoring the await, and completing it's code block.
If I comment out some of the latter rows (dependent on the previous), and let the program run to completion, the database gets updated.
There is nothing functionally wrong with the code, everything works, just not from beginning to completion.
After running two function calls at the beginning, letting that run, I can then comment out those rows, uncomment the later rows in the flow, and run again, and everything works as expected, but I cannot run to completion with both uncommented.
What can I do to make sure each function call completes 100%, has all updates completed in the database, before going to the next step?
async/await is not working for me.
this is not pseudo-code it's the actual code, that is executing, that I am working with, the function names changed only. It is real working code, cut-n-pasted direct from my IDE.
// these are functions I call below (each in their own .js)
const insert_rows_to_db_from_csv = require('./insert_rows_to_db_from_csv')
const call_api_using_rows_from_function_above = require('./call_api_using_rows_from_function_above')
const and_so_on = require('./and_so_on')
const and_so_on_and_on = require('./and_so_on_and_on')
const and_so_on_and_on_and_on = require('./and_so_on_and_on_and_on')
// each of the above exports a main() function where I can call func.main() just // like this one defined below (this is my main() entry point)
module.exports = {
main: async function (csvFilePath) {
console.log('service: upload.main()')
try {
const csvList = []
let rstream = fs.createReadStream(csvFilePath)
.pipe(csv())
.on('data', (data) => csvList.push(data))
.on('end', async () => {
let num_rows = csvList.length
//step one (if I run these two, with step two calls below commented out, this works)
await insert_rows_to_db_from_csv.main(csvList);
await call_api_using_rows_from_function_above.main();
// step two
// blows up here, on the next function call,
// no rows selected in sql statements, must comment out, let the above run to
// completion, then comment out the rows above, and let these run separate
await work_with_rows_updated_in_previous_call_above.main(); // sets
await and_so_on.main();
await and_so_on_and_on.main();
await and_so_on_and_on_and_on.main();
})
} catch (err) {
console.log(err.stack)
} finally {
}
}
};
here is the one liner I am using to call the insert/update to the DB:
return await pool.query(sql, values);
that's it, nothing more. This is from using:
https://node-postgres.com/
npm install pg
PART 2 - continuing on,
I think the problem might be here. This is where I am doing each
API call, then insert (that the next function call is dependent upon), some code smell here that I can't sort out.
processBatch(batch) is called, that calls the API, gets a response back, and then within there it calls `handleResponseDetail(response), where the insert is happening. I think the problem is here, if there are any ideas?
this is a code block inside:
await call_api_using_rows_from_function_above.main();
It completes with no errors, inserts rows, and commits, then the next function is called, and this next function finds no rows (inserted here). But the await on the entire main() .js blocks and waits, so I don't understand.
/**
* API call, and within call handleResponse which does the DB insert.
* #param batch
* #returns {Promise<*>}
*/
async function processBatch(batch) {
console.log('Processing batch');
return await client.send(batch).then(res => {
return handleResponseDetail(res);
}).catch(err => handleError(err));
}
// should this be async?
function handleResponseDetail(response) {
response.lookups.forEach(async function (lookup) {
if (typeof lookup.result[0] == "undefined") { // result[0] is Candidate #0
++lookup_fail;
console.log('No response from API for this address.')
} else {
++lookup_success;
const id = await insert(lookup);
}
});
}
Given the code block from your Part 2 edit, the problem is now clear: all of your insert()s are being scheduled outside of the blocking context of the rest of your async/await code! This is because of that .forEach, see this question for more details.
I've annotated your existing code to show the issue:
function handleResponseDetail(response) { //synchronous function
response.lookups.forEach(async function (lookup) { //asynchronous function
//these async functions all get scheduled simultaneously
//without waiting for the previous one to complete - that's why you can't use forEach like this
if (typeof lookup.result[0] == "undefined") { // result[0] is Candidate #0
++lookup_fail;
console.log('No response from API for this address.')
} else {
++lookup_success;
const id = await insert(lookup); //this ONLY blocks the inner async function, not the outer `handleResponseDetail`
}
});
}
Here is a fixed version of that function which should work as you expect:
async function handleResponseDetail(response) {
for(const lookup of response.lookups) {
if (typeof lookup.result[0] == "undefined") { // result[0] is Candidate #0
++lookup_fail;
console.log('No response from API for this address.')
} else {
++lookup_success;
const id = await insert(lookup); //blocks handleResponseDetail until done
}
}
}
Alternatively, if the order of insertion doesn't matter, you can use Promise.all for efficiency:
async function handleResponseDetail(response) {
await Promise.all(response.lookups.map(async lookup => {
if (typeof lookup.result[0] == "undefined") { // result[0] is Candidate #0
++lookup_fail;
console.log('No response from API for this address.')
} else {
++lookup_success;
const id = await insert(lookup);
}
})); //waits until all insertions have completed before returning
}
To reiterate, you cannot easily use .forEach() with async/await because .forEach() simply calls the given function for each element of the array synchronously, with no regard for awaiting each promise before calling the next. If you need the loop to block between each element, or to wait for all elements to complete processing before returning from the function (this is your use case), you need to use a different for loop or alternatively a Promise.all() as above.
What your main function currently does is merely creating stream, assigning listeners and instantly returning. It does not await for all the listeners to resolve like you are trying to have it do
You need to extract your file reading logic to another function, which will return a Promise that will resolve only when the entire file is read, then await for that Promise inside main
function getCsvList(csvFilePath) {
return new Promise((resolve, reject) => {
const csvList = []
fs.createReadStream(csvFilePath)
.pipe(csv())
.on('data', (data) => csvList.push(data))
.on('end', () => {
resolve(csvList)
})
.on('error', (e) => reject(e))
})
}
module.exports = {
main: async function (csvFilePath) {
try {
const csvList = await getCsvList(csvFilePath)
await insert_rows_to_db_from_csv.main(csvList);
await call_api_using_rows_from_function_above.main();
await work_with_rows_updated_in_previous_call_above.main();
await and_so_on.main();
await and_so_on_and_on.main();
await and_so_on_and_on_and_on.main();
} catch (err) {
console.log(err.stack)
} finally {
}
}
};
I'm trying to retrieve data from the Yummly API through Amazon Alexa using ws-yummly in Node.js deployed on AWS Lambda. I'm fairly new to all aspects of this, but new to javascript in particular (Python is my 'native' language).
Here is what I have for my recommendation intent:
"RecommendationIntent": function () {
// delegate to Alexa to collect all the required slots
let filledSlots = delegateSlotCollection.call(this);
if (!filledSlots) {
return;
}
console.log("filled slots: " + JSON.stringify(filledSlots));
// at this point, we know that all required slots are filled.
let slotValues = getSlotValues(filledSlots);
console.log(JSON.stringify(slotValues));
const mainIngredientQuery = slotValues.mainIngredient.resolved;
async function main (queryWord) {
const resp = await Yummly.query(queryWord)
.maxTotalTimeInSeconds(1400)
.maxResults(20)
.minRating(3)
.get();
const names = resp.matches.map(recipe => recipe.recipeName);
const speechOutput = String(names[0]);
this.response.speak(speechOutput);
this.emit(":responseReady");
}
main(mainIngredientQuery).catch(error => console.error(error))
},
This is in the index.js file deployed on AWS lambda. I have isolated the problem to the async function. I have tested the function locally and it returns to console.log a list of recipe names. I want to have Alexa say these names. Or at least one name.
If I put the speechOutput assignment inside (as it is now), then I get an error that the 'Speechlet Response is set to null'.
If I tell it to 'return names' and set the external assignment to names or names[0] I get object promise or undefined (respectively).
Everything else in my program works fine and test these two bits apart they work, but putting them together doesn't work. I think that this is a syntax or placement error, but I don't understand the structure or formatting well enough yet (still learning) to understand what to try next.
How about using Promise.then like this:
async function main (queryWord) {
const resp = await Yummly.query(queryWord)
.maxTotalTimeInSeconds(1400)
.maxResults(20)
.minRating(3)
.get();
const names = resp.matches.map(recipe => recipe.recipeName);
return String(names[0]);
}
main(mainIngredientQuery)
.catch( error => console.error(error) )
.then( data => {
this.response.speak( data );
this.emit(":responseReady");
});
I'm updating this in case anyone else has the same problem.
If you notice, in my original code, I had an async function inside the intent. That didnt work because the intent itself was/is a function. By making the intent function an async function instead, I was able to solve the problem.
The following is working code for an async/await Alexa Intent.
The full index.js is available on my github if you want to take a look, but that will be a more advanced final version. The code below immediately follows up on the original question.
"RecommendationIntent": async function main () {
// delegate to Alexa to collect all the required slots
let filledSlots = delegateSlotCollection.call(this);
if (!filledSlots) {
return;
}
console.log("filled slots: " + JSON.stringify(filledSlots));
// at this point, we know that all required slots are filled.
let slotValues = getSlotValues(filledSlots);
console.log(JSON.stringify(slotValues));
const mainIngredientQuery = slotValues.mainIngredient.resolved;
const resp = await Yummly.query('chicken')
.maxTotalTimeInSeconds(1400)
.maxResults(20)
.minRating(3)
.get();
const names = resp.matches.map(recipe => recipe.recipeName);
console.log(names);
const speechOutput = names[0];
this.response.speak(speechOutput);
this.emit(":responseReady");
},
I have some code that looks like this:
async function promptHandler(source) {
source.subscribe(function(line) {
console.log(`line == ${line}`);
});
let matchingTests = await getMatchingTests('ROGL');
}
This prints out the contents of the source Observable, which is listening to a ReadStream of a txt file. When the function as it is above is called, I see the output of the file. However, if I call subscribe() after getMatchingTests() gets called, like this:
async function promptHandler(source) {
let matchingTests = await getMatchingTests('ROGL');
source.subscribe(function(line) {
console.log(`line == ${line}`);
});
}
I don't see the contents of the txt file. I know that the matchingTests variable contains the successful results of getMatchingTests, so I don't think it's preventing Node from executing that line.
I'm guessing that something about the getMatchingTests async function call is messing with the source Observable, but I'm not seeing how.
Here's my source code:
let fileStream = createReadStream(file)
.pipe(split());
let source = new Observable(o => {
fileStream.on('data', line => {console.log('data'); o.next(line);});
fileStream.on('error', err => o.error(err));
fileStream.on('end', () => {console.log('end'); o.complete();});
});
My intuition here is that the source observable is a hot source, and that by the time await has returned with the matching tests, your text file is already read. So when you subscribe at that point, there is no line to read, they were read before you subscribed to the source.
UPDATE :
Given your code, if the ordering is a problem for your use case, you can consider moving the filestream creation into the observable factory, i.e.
let source = new Observable(o => {
let fileStream = createReadStream(file)
.pipe(split());
fileStream.on('data', line => {console.log('data'); o.next(line);});
fileStream.on('error', err => o.error(err));
fileStream.on('end', () => {console.log('end'); o.complete();});
});
That way, the stream will be created and started only when you subscribe.