I have something very strange happening in my program. I have consol.logged the crap out of it with timestamps to try and figure out what is happening. Basically my program randomly stops fetching data. I have ensured a new stream of data is there but I have to refresh the entire page or resave the program to restart everything when it gets hung up. On top of that, there are no errors or flags telling me why it stops. I tried to isolate the issue but it is something to do with the async function most likely. Here is the code....
function App() {
const data = async() => {
try {
console.log('try block initiated')
const apiResponse = await fetch(ipAddress)
console.log(apiResponse);
console.log("BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB")
const responseText = await apiResponse.text();
console.log(responseText)
console.log("CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC")
if (loading === true){
setLoading(false);
}
console.log("DDDDDDDDDDDDDDDDDDDDDDDDDDDDDD")
return responseText;
} catch (error) {
console.log('catch initiated')
setError(true);
}
};
console.log("AAAAAAAAAAAAAAAAAAAAAAA")
try{
console.log("EEEEEEEEEEEEEEEEEEEEEEEEEEE")
data().then(response=>setMoisture(response));
console.log("FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF")
} catch(error){
console.log('gotcha')
setError(true);
}
let value = moisture;
console.log(value);
count += 1;
console.log(count);
return(
<div className="App">
<WeatherCard moisture={value}/>
</div>
);
}
Here is what the console looks like right before it stop fetching new data...
HERE IS A POSSIBLE SOLUTION. After many headaches and researching async functions. I think this is going to work. I am going to let this run all night and see if it ever gets caught up but so far, Here is the solution...
const data = async() => {
try {
const apiResponse = await fetch(ipAddress)
const responseText = await apiResponse.text();
console.log("CCCCCCCCCCCCCCCCCCC");
console.log(responseText);
if (loading === true){
setLoading(false);
}
data().then(response=>setMoisture(response));
return responseText;
} catch (error) {
console.log("EEEEERRRRRRRRRROOOOOOOOOOORRRRRRRRRRRR")
setError(true);
}
};
console.log("AAAAAAAAAAAAAAAAAAAA");
if (moisture === "initialState"){
data().then(response=>setMoisture(response));
}else{
console.log("QQQQQQQQQQQQQQQQQQQQQQQQ")
}
Here is what I changed... I made an initial state called "initialSate and if the variable was this value, it runs the async function. I then NEVER CALL THIS ASYNC AGAIN OUTSIDE OF THE ASYNS ITSELF. Instead I call itself right before returning a value from the function. This works because my program is running while the async function is running. However, the program is sequential to itself and the async is sequential to itself. They are NOT sequential together. So, the async function now only gets called again once the async function finishes its first run. Simultaneaously the rest of my program is running. Before I think it was getting caught because my program was trying to call the async function before it had finished the first time. This is a pure guess because I have no way to proving that but it makes sense. If this overnight run fails. I will pull this from the solution. Otherwise, I think this will work. FINGERS CROSSED!
This question already has answers here:
Using async/await with a forEach loop
(33 answers)
Closed 3 years ago.
await is not blocking as expected, when a block of code updates db (using postgres / node )
https://node-postgres.com
I have a list of async function calls, each call udpates a database, and each subsequent call works on data updated by the previous call.
There are about eight calls in a row, and each call must update the complete set of data it is working with, 100% to completion, before going to the next.
I tried to make everything not async, but it appears I am forced to make everything async/await because of the library I am using (postgres / node).
Each function call must complete 100% before going on to the next function call, because the next step does a select on rows where a field is not null (where the previous step fills in a value).
I have an await in front of each call, that does something (see code below):
loads the db from a csv,
next step selects all rows just inserted, calls an API and updates the database,
and so on,
but at one point, when the next function executes, NONE of the rows have been updated (as I trace through and verify, a SQL statement returns nothing back),
the code seems to pass right through going to the second function call, not blocking, honoring the await, and completing it's code block.
If I comment out some of the latter rows (dependent on the previous), and let the program run to completion, the database gets updated.
There is nothing functionally wrong with the code, everything works, just not from beginning to completion.
After running two function calls at the beginning, letting that run, I can then comment out those rows, uncomment the later rows in the flow, and run again, and everything works as expected, but I cannot run to completion with both uncommented.
What can I do to make sure each function call completes 100%, has all updates completed in the database, before going to the next step?
async/await is not working for me.
this is not pseudo-code it's the actual code, that is executing, that I am working with, the function names changed only. It is real working code, cut-n-pasted direct from my IDE.
// these are functions I call below (each in their own .js)
const insert_rows_to_db_from_csv = require('./insert_rows_to_db_from_csv')
const call_api_using_rows_from_function_above = require('./call_api_using_rows_from_function_above')
const and_so_on = require('./and_so_on')
const and_so_on_and_on = require('./and_so_on_and_on')
const and_so_on_and_on_and_on = require('./and_so_on_and_on_and_on')
// each of the above exports a main() function where I can call func.main() just // like this one defined below (this is my main() entry point)
module.exports = {
main: async function (csvFilePath) {
console.log('service: upload.main()')
try {
const csvList = []
let rstream = fs.createReadStream(csvFilePath)
.pipe(csv())
.on('data', (data) => csvList.push(data))
.on('end', async () => {
let num_rows = csvList.length
//step one (if I run these two, with step two calls below commented out, this works)
await insert_rows_to_db_from_csv.main(csvList);
await call_api_using_rows_from_function_above.main();
// step two
// blows up here, on the next function call,
// no rows selected in sql statements, must comment out, let the above run to
// completion, then comment out the rows above, and let these run separate
await work_with_rows_updated_in_previous_call_above.main(); // sets
await and_so_on.main();
await and_so_on_and_on.main();
await and_so_on_and_on_and_on.main();
})
} catch (err) {
console.log(err.stack)
} finally {
}
}
};
here is the one liner I am using to call the insert/update to the DB:
return await pool.query(sql, values);
that's it, nothing more. This is from using:
https://node-postgres.com/
npm install pg
PART 2 - continuing on,
I think the problem might be here. This is where I am doing each
API call, then insert (that the next function call is dependent upon), some code smell here that I can't sort out.
processBatch(batch) is called, that calls the API, gets a response back, and then within there it calls `handleResponseDetail(response), where the insert is happening. I think the problem is here, if there are any ideas?
this is a code block inside:
await call_api_using_rows_from_function_above.main();
It completes with no errors, inserts rows, and commits, then the next function is called, and this next function finds no rows (inserted here). But the await on the entire main() .js blocks and waits, so I don't understand.
/**
* API call, and within call handleResponse which does the DB insert.
* #param batch
* #returns {Promise<*>}
*/
async function processBatch(batch) {
console.log('Processing batch');
return await client.send(batch).then(res => {
return handleResponseDetail(res);
}).catch(err => handleError(err));
}
// should this be async?
function handleResponseDetail(response) {
response.lookups.forEach(async function (lookup) {
if (typeof lookup.result[0] == "undefined") { // result[0] is Candidate #0
++lookup_fail;
console.log('No response from API for this address.')
} else {
++lookup_success;
const id = await insert(lookup);
}
});
}
Given the code block from your Part 2 edit, the problem is now clear: all of your insert()s are being scheduled outside of the blocking context of the rest of your async/await code! This is because of that .forEach, see this question for more details.
I've annotated your existing code to show the issue:
function handleResponseDetail(response) { //synchronous function
response.lookups.forEach(async function (lookup) { //asynchronous function
//these async functions all get scheduled simultaneously
//without waiting for the previous one to complete - that's why you can't use forEach like this
if (typeof lookup.result[0] == "undefined") { // result[0] is Candidate #0
++lookup_fail;
console.log('No response from API for this address.')
} else {
++lookup_success;
const id = await insert(lookup); //this ONLY blocks the inner async function, not the outer `handleResponseDetail`
}
});
}
Here is a fixed version of that function which should work as you expect:
async function handleResponseDetail(response) {
for(const lookup of response.lookups) {
if (typeof lookup.result[0] == "undefined") { // result[0] is Candidate #0
++lookup_fail;
console.log('No response from API for this address.')
} else {
++lookup_success;
const id = await insert(lookup); //blocks handleResponseDetail until done
}
}
}
Alternatively, if the order of insertion doesn't matter, you can use Promise.all for efficiency:
async function handleResponseDetail(response) {
await Promise.all(response.lookups.map(async lookup => {
if (typeof lookup.result[0] == "undefined") { // result[0] is Candidate #0
++lookup_fail;
console.log('No response from API for this address.')
} else {
++lookup_success;
const id = await insert(lookup);
}
})); //waits until all insertions have completed before returning
}
To reiterate, you cannot easily use .forEach() with async/await because .forEach() simply calls the given function for each element of the array synchronously, with no regard for awaiting each promise before calling the next. If you need the loop to block between each element, or to wait for all elements to complete processing before returning from the function (this is your use case), you need to use a different for loop or alternatively a Promise.all() as above.
What your main function currently does is merely creating stream, assigning listeners and instantly returning. It does not await for all the listeners to resolve like you are trying to have it do
You need to extract your file reading logic to another function, which will return a Promise that will resolve only when the entire file is read, then await for that Promise inside main
function getCsvList(csvFilePath) {
return new Promise((resolve, reject) => {
const csvList = []
fs.createReadStream(csvFilePath)
.pipe(csv())
.on('data', (data) => csvList.push(data))
.on('end', () => {
resolve(csvList)
})
.on('error', (e) => reject(e))
})
}
module.exports = {
main: async function (csvFilePath) {
try {
const csvList = await getCsvList(csvFilePath)
await insert_rows_to_db_from_csv.main(csvList);
await call_api_using_rows_from_function_above.main();
await work_with_rows_updated_in_previous_call_above.main();
await and_so_on.main();
await and_so_on_and_on.main();
await and_so_on_and_on_and_on.main();
} catch (err) {
console.log(err.stack)
} finally {
}
}
};
I have some problem developping my own Node.JS application. I'm new to this kind of code architecture and not so good with asynchronous as well.
I currently have an event in my index.js script, calling for another script database.js. This database script will handle every db call inside my application.
index.js
const db = require('database.js');
//Called when a userjoin the application, this is working well//
db.userJoinedFunction("username");
I was planning adding other function call inside my "userjoinedFunction". For example : establishing first the DB connection --> 2nd checking if the user is already inside the DB --> 3d if not, create the user entry --> 4 if yes, do welcome back message.
Since I have to establish the DB connection multiple time, I wanted to reuse some code inside theses function like this :
database.js
module.exports.userJoinedFuntion = userJoinedFunction
function userjoinedFunction(username){
Init_database_connection();
if(UserAlreadySignedup(username)){
WelcomeMessage(username);
}
else {
AddUserToDatabase(username);
}
}
but obviously i'm doing it wrong because asynchronous Node.js application can't be that simple... How can I have that type of reusable architecture ? Do you have articles I can read about the subject ?
Thank you very much for your help !
EDIT 29/05/2019
with the response of novomanish I tried this :
async fonction run(){
await(call("number 1");
await Init_database_connection();
await(call("number 2");
}
async function call(text){
console.log(text);
}
async function Init_database_connection(){
db = new sqlite3.Database("path", sqlite.OPEN_READWRITE, (err) =>{
if(err){ console.log(err); }
else { console.log("DB Connected"); }
});
}
my output are :
- number one
- number two
- DB Connected
I checked many article and documentation, I can't find a good way of reusing function inside a "main function" (like in my example : the function run()). And i'm surprised that waiting for finished DB query isn't apparently thing...
You need async/await in your functions. So the new code would look like:
async function Init_database_connection(){...}
...
async function userjoinedFunction(username){
await Init_database_connection();
if(await UserAlreadySignedup(username)){
return WelcomeMessage(username);
}
return AddUserToDatabase(username);
}
Regarding your edit, Init_database_connection should have returned a promise for await to work, like this:
async function Init_database_connection(){
return new Promise((resolve, reject)=>{
db = new sqlite3.Database("path", sqlite.OPEN_READWRITE, (err) =>{
if(err){
console.log(err);
reject();
}else {
console.log("DB Connected");
resolve()
}
});
})
}
All of the examples for using the mssql client package/tedious driver are for async/callbacks/promises but I'm only developing a microservice that will see limited use and my understanding of asynchronous functions is still a bit fuzzy.
Here's what I have for trying to use async/await :
Report generation class:
const mssql = require('mssql');
const events = require('events');
class reporter {
constructor(searcher, logger) {
// Pass in search type and value or log the error of none defined
this.lg = logger
if (searcher.type && searcher.content) {
this.lg.lg("reporter created", 3)
this.srchType = searcher.type;
this.srchContent = searcher.content;
} else {
this.lg.lg("!MISSING SEARCH PARAMETERS", 0);
this.err = "!MISSING SEARCH PARAMETERS";
}
}
proc() {
//DB Connect async
async () => {
try {
await mssql.connect('mssql://username:password#localhost/database')
this.result = await mssql.query`select * from mytable where id = ${this.searcher}`
} catch (err) {
// ... error checks
}
}
return this.result;
}
}
Then called:
//Pass to reporter for resolution
var report1 = new reporter(searcher, logs);
report1.proc();
I'm sure this is probably a pretty bad way to accomplish this, so I'm also open to any input on good ways to accomplish the end goal, but I'd still like to know if it's possible to accomplish synchronously.
You can't do it synchronously. Figuring out this async stuff is definitely worth your time and effort.
async / await / promises let you more-or-less fake doing it synchronously
const report1 = new reporter(searcher, logs);
report1.proc()
.then ( result => {
/* in this function, "result" is what your async function returned */
/* do res.send() here if you're in express */
} )
.catch ( error => {
/* your lookup failed */
/* inform the client of your web service about the failure
* in an appropriate way. */
} )
And, unwrap the async function in your proc function, like so:
async proc() {
try {
await mssql.connect('mssql://username:password#localhost/database')
this.result = await mssql.query`select * from mytable where id = ${this.searcher}`
} catch (err) {
// ... error checks
}
return this.result;
}
await and .then are analogous.
Kind of an updated answer that continues off of O. Jones' answer.
The current version of Node.js (v15+) has support for top-level await, meaning you can run it all sequentially.
import mssql from 'mssql';
await mssql.connect('mssql://username:password#localhost/database')
const result = await mssql.query`select * from mytable where id = ${this.searcher}`
But it should still be avoided since you want to catch for errors instead of letting it crash.
In current versions of Node.js, if an await/promise rejects, and isn't caught with a .catch(), then the uncaught promise will terminate your application with the error
I am creating a Node.js module that takes a list of movie titles and fetches their respective metadata from omdbapi.com.
These lists are often very large and sometimes (with my current slow internet connection) the connection stalls due to too many concurrent connections. So I set up a timeout/abort method that restarts the process after 30 seconds.
The problem I'm having is that whenever I lose internet connection or the connection stalls, it just bails out of the process, and doesn't restart the connection.
Example:
async function getMetadata () {
const remainingMovies = await getRemainingMovies();
for (let i = 0; i < remainingMovies.length;i++) {
const { data, didTimeout } = await fetchMetadata(remainingMovies[i]);
// Update "remainingMovies" Array...
if (didTimeout) {
getMetadata();
break;
}
}
if (!didTimeout) {
return data;
}
}
This is obviously a simplified version but essentially:
The getMetadata Function gets the remainingMovies Array from the global scope.
Fetches the metadata from the server with the fetchMetadata Function.
Checks if the connection timed out or not.
If it did it should restart the Function and attempt to connect again.
If it didn't timeout then finish the for loop and continue.
I guess you want something similar to below script. Error handling using try/catch for async/await which probably is what you are looking for as a missing puzzle.
async function getMetadata() {
const remainingMovies = await getRemainingMovies();
remainingMovies.map(movie => {
try {
return await fetchMetadata(movie);
} catch (err) {
return getMetadata();
}
});
}