Develop Reference

Why isn't this function fast when we collect result from async call?

const services = ['a', 'b', 'c', 'd'];
async function foo(service) {
// function to get one jobUrl data called by map later on
const getJob = async ({ url: jobUrl }) => {
const response = await fetch(jobUrl, { headers: headers });
const { result, timestamp: timeStamp } = await response.json();
if (result === 'SUCCESS') {
return { jobUrl, timeStamp };
}
};
// Find the list of jobs for a service
let url = `http://example.com`;
const response = await fetch(url, { method: 'POST', headers: headers });
const { jobs } = await response.json();
const unfiltered = await Promise.all(jobs.map(getJob));
const successful = unfiltered.filter(v => v);
let latestJob = successful.sort((obj1, obj2) => obj2.timeStamp - obj1.timeStamp)[0].jobUrl;
let arr = latestJob.split('/');
let recent = { service: service, job: arr[arr.length - 2] };
console.log(recent);
// return recent;
}
When I run the following piece of code it takes only 4 seconds for finding the latest job.
for (const service of services) {
foo(service);
}
Whereas, when I run the following piece of code it takes 15-16 seconds for finding the latest job. For this I uncomment the return recent; line of code in the last line of function foo and comment out the console.log(recent) line.
const latest = [];
for (const service of services) {
latest.push(await foo(service));
}
My understanding is that for the second piece of code since we are waiting for all the async calls to finish it takes longer. If that is the case, is there a way to speed this up?

You may want to use Promise.all or Promise.allSettled.
That way you can pass an array of async functions (promises) and await them. That way, functions are not running sequentially but async.

stored array outside of async function in nodejs fetch

How do I get the value of the failed array outside of the function? Because I need this array as an input for other functions.
failed = [];
async function fetchFromApi(i) {
var ApiEndpoint = "https://jsonplaceholder.typicode.com/todos/" + i
var httpRequest = {
method: "GET"
}
var response = await fetch(ApiEndpoint, httpRequest);
if (!response.ok) { // Necessary for error handling, fetch does not return an error if HTTP request fails
failed.push(i);
console.log(failed)
} else {
let symbolData = await response.json();
console.log(JSON.stringify(symbolData))
return JSON.stringify(symbolData);
}
};
fetchFromApi("abc")
console.log(failed)
The console.log("failed") in the code above gives me an empty array []. I want it to be getting ["abc"] when this variable being called outside of the function
UPDATE EDIT (new follow-up question):
I tried #SlothOverlord 's solution and it seems to work below is the example.
const myFunction = async (j) => {
failed = [];
async function fetchFromApi(i) {
var ApiEndpoint = "https://jsonplaceholder.typicode.com/todos/" + i
var httpRequest = {
method: "GET"
}
var response = await fetch(ApiEndpoint, httpRequest);
if (!response.ok) { // Necessary for error handling, fetch does not return an error if HTTP request fails
failed.push(i);
} else {
let symbolData = await response.json();
console.log(JSON.stringify(symbolData))
return JSON.stringify(symbolData);
}
};
await fetchFromApi(j)
return failed
}
myFunction("abc").then(data=>console.log(failed))
However, when I add in a forEach statement within the function, it breaks and returns an empty array again. See example below. What's going on here?
const myFunction = async (j) => {
failed = [];
async function fetchFromApi(arrays) {
arrays.forEach(async function(i) {
var ApiEndpoint = "https://jsonplaceholder.typicode.com/todos/" + i
var httpRequest = {
method: "GET"
}
var response = await fetch(ApiEndpoint, httpRequest);
if (!response.ok) { // Necessary for error handling, fetch does not return an error if HTTP request fails
failed.push(i);
// console.log(failed)
} else {
let symbolData = await response.json();
console.log(JSON.stringify(symbolData))
return JSON.stringify(symbolData);
}
});
}
await fetchFromApi(j)
// console.log(failed)
return failed
}
myFunction(["aaa", "abc"]).then(data=>console.log(failed))

You need to await the fetchFromApi("abc") function.
Async await only works INSIDE the function, not outside of it.
So what happened is:
fetchFromApi("abc") is called
console.log(failed) is empty
fetchFromApi("abc") finished
The solution is to wrap the function in async and await the fetchFromApi call
const myFunction = async () => {
//...
failed = [];
async function fetchFromApi(i) {
//...
}
await fetchFromApi("abc")
console.log(failed)
}

fetchFromApi is an async method, you need to await it if you want to see it's result logged.
JavaScript has a concurrency model based on an event loop, which is responsible for executing the code, collecting and processing events. when the async call will have it's result it will be placed in the callback queue and on the next iteration (tick), it will be popped out from there and placed in the execution stack to be processed. when you use await, the exection call will be halted at that point and will progress until the async call has it's result.
failed = [];
async function fetchFromApi(i) {
var ApiEndpoint = "https://jsonplaceholder.typicode.com/todos/" + i
var httpRequest = {
method: "GET"
}
var response = await fetch(ApiEndpoint, httpRequest);
if (!response.ok) { // Necessary for error handling, fetch does not return an error if HTTP request fails
failed.push(i);
console.log(failed)
} else {
let symbolData = await response.json();
console.log(JSON.stringify(symbolData))
return JSON.stringify(symbolData);
}
};
await fetchFromApi("abc")
console.log(failed)
JS event loop - https://developer.mozilla.org/en-US/docs/Web/JavaScript/EventLoop

Async Loop Not Honoring Async

I have been a bit stuck on an Async function.
What I am trying to accomplish - I am creating a batchProcessing function (batchGetSubs) which will loop through a set of files, read a ID, then make an API request, wait for a response (THE ISSUE) and then to write to a new file with the formatted data.
The issue - I have tried both Async and Await, as well as pushing promises and trying to use a Promise.all to wait for promises to be resolved, but with no success. Current behavior is that I get all of my console.logs that are in the Promise.all section before the API calls actually return all the data. I have used these articles as reference:
Asynchronous Process inside a javascript for loop
Promise All with Axios
Javascript Promise push value into array (only from function or outside?)
The Code -
async function batchGetSubs(data, command) {
console.time('batchGetSubs')
iteration = 1;
let dataArray = []; promises = [];
// LOOP THROUGH FILES, THEN GET TENANTS BY SUB
for (i = iteration; i < totalIterations; i++) {
let msIds = await loopAndDump(iteration);
// LOOP THROUGH TENANTIDS AND GET SUBSCRIPTIONS
msIds.map(async item => {
let newRecord = await getSubsByTenantId(item);
promises.push(await newRecord);
});
}
Promise.all([promises]).then(() => { // FIXME: WHY IS THIS NOT WAITING FOR ALL RESPONSES?
console.log(p1SubscriptionArray),
console.timeEnd('batchGetSubs')
});
}
async function getSubsByTenantId(msTenantId) {
let newRecord; let subscriptionArray = [];
let bearerToken = p1BearerToken;
let totalSubs = 0;
const subIdConfig = {
method: 'get',
url: ``,
headers: { 'Authorization': bearerToken }
}
await delay();
await axios(subIdConfig)
.then(async res => {
console.log('AXIOS RESPONSE', res.data);
if (res.data.items) {
let subItems = res.data.items;
console.log('^^^^^^^^^^^^^^^^^^', res.data.items)
// LOOP THROUGH AND ADD TO SUBSCRIPTION ARRAY
subItems.map(async subscription => {
if (subscription.friendlyName) {
totalSubs++;
subscriptionArray.push(subscription.friendlyName);
}
});
newRecord = { "tenantId": msTenantId, "totalSubs": totalSubs, "subscriptionFriendlyName": subscriptionArray };
} else {
// NO SUBS
newRecord = { "tenantId": msTenantId, "totalSubs": totalSubs, "subscriptionFriendlyName": ['NONE'] };
let statusCode, errorMessage;
if (error && error.response === undefined) { // GETTING STATUS -4039 or -4077 INSTEAD OF 429 WHEN CHECKING SUBS. FORCE A RESEND.
if (error.errno && error.errno === -4039 || error.errno && error.errno === -4077) statusCode = 429; errorMessage = error.code;
} else {
statusCode = error.response.status;
errorMessage = error.response.statusText;
}
console.error('ERROR:: SUBIDCONFIG SECTION THROWING ERROR: ', statusCode, portal, errorMessage);
// SORT NON-200 CALLS BASED ON STATUS CODE
switch (statusCode) {
case 403:
status403.push('subs', newRecord);
break;
case 404:
status404.push('subs', newRecord);
erroring = true;
break;
case 429:
status429.push('subs', newRecord);
erroring = true;
break;
default:
statusOther.push('subs', newRecord)
erroring = true;
break;
}
}
})
.catch(err => {
newRecord = { "tenantId": msTenantId, "totalSubs": totalSubs, "subscriptionFriendlyName": ['NONE'] };
console.error('ERROR: REQUEST IN GETSUBSBYTENANTID(): ', err)
})
.then(res => {
console.log('DO WE HAVE ANY INFORMATION? ', newRecord);
p1SubscriptionArray.push(newRecord);
resolve();
});
}

I only checked the first function, where you had put your question:
WHY IS THIS NOT WAITING FOR ALL RESPONSES?
For several reasons:
The promise array is still empty when you call Promise.all. This is because you only do a push after an await, and so that push happens asynchronously (read: later).
Even when the promises array gets populated, it will not have promises objects, but resolved values (i.e. newRecord values)
Even if promises would have been array of promises, you don't pass that array correctly to Promise.all: you wrap that array in yet another array, which then only has one entry, and that entry is not a promise, but an array.
Not related to your problem, but:
please make it a habit to explicitly declare all your variables. You did not do this for iteration, promises, nor i.
Only use .map when you do something with the return value. For pure iteration use .forEach or for. In this case, you can use the return value to extend the promises array.
If you intend to call batchGetSubs and need to know when all is done, then make sure it returns a useful promise: return Promise.all()
Here is the suggested correction for that function:
async function batchGetSubs(data, command) {
console.time('batchGetSubs')
let iteration = 1; // Declare!
let dataArray = [];
let promises = []; // declare
// declare i
for (let i = iteration; i < totalIterations; i++) {
let msIds = await loopAndDump(iteration);
// Use the return value of the map method. No need for async callback
promises.push(...msIds.map(item => {
// Get the promise, not the resolved value, as that will come too late:
return getSubsByTenantId(item);
}));
}
// promises is already an array; don't make it an array of arrays.
// And: return the resulting promise: it may be useful to the caller.
return Promise.all(promises).then(() => {
console.log(p1SubscriptionArray),
console.timeEnd('batchGetSubs')
});
}

You do a await and then in the same time.
You do a promise/then or await/async
do
async function getSubsByTenantId(msTenantId) {
let newRecord; let subscriptionArray = [];
let bearerToken = p1BearerToken;
let totalSubs = 0;
const subIdConfig = {
method: 'get',
url: ``,
headers: { 'Authorization': bearerToken }
}
await delay();
const r = await axios(subIdConfig)
const reponse = r.data
console.log(response)
}
if you want to try catch do this :
async function getSubsByTenantId(msTenantId) {
let newRecord; let subscriptionArray = [];
let bearerToken = p1BearerToken;
let totalSubs = 0;
const subIdConfig = {
method: 'get',
url: ``,
headers: { 'Authorization': bearerToken }
}
await delay();
let r
try {
r = await axios(subIdConfig)
} catch (error) {
console.log(error)
}
const reponse = r.data
console.log(response)
}

Make axios call from another axios promise and save returned data

I am making an axios call from promise of another axios call, the code looks like below, the call is basically from the first method to the second method.
UPDATED CODE BASED ON SUGGESTION
It simply says: Can not use keyword 'await' outside an async function then I tried
var data = async () => {
await this.checkParentLoggerLevel();
};
still did not work
async updateLevel(logger, level, index) {
alert(index);
axios
.post(
'url'
)
.then(() => {
var data = await this.checkParentLoggerLevel();
alert('waiting');
alert(
'This will be executed before the second methods returns HEllo'
);
alert(data);
});
},
Second method:
async checkParentLoggerLevel() {
alert('inside checkParentLoggerLevel ');
return await axios
.get('url')
.then(() => {
alert('returning hello');
return 'hello';
});
},
My aim to to save the returned hello to data variable in first method.This is not working. The other problem is after the this.checkParentLoggerLevel() methods call code execution continues and does not wait for the retuned value.

This happens because inside checkParentLoggerLevel you are not waiting for the axios promise to complete. You can do it like:
async checkParentLoggerLevel() {
alert('inside checkParentLoggerLevel ');
return await axios
.get('url')
.then((res) => {
return 'hello';
});
}
Also, you need to await inside updateLevel like:
async updateLevel() {
axios
.post(url)
.then(async (res) => {
var data = await this.checkParentLoggerLevel();
alert("This will be executed before the second methods returns HEllo");
alert(data);
});
}

You should chain the promises, so:
updateLevel() {
axios
.post(url)
.then(res => {
return this.checkParentLoggerLevel.then(data => ([res, data]);
})
.then(([res, data]) => {
// here
});
}
Or simply with async await:
async updateLevel() {
const res = await axios.post(url);
const data = await this.checkParentLoggerLevel();
// Do whatever you want
}

properly using async and await

The function below calls several asynchronous functions in a for loop. It's parsing different CSV files to build a single JavaScript object. I'd like to return the object after the for loop is done. Its returning the empty object right away while it does the asynchronous tasks. Makes sense, however I have tried various Promise / async /await combinations hopes of running something once the for loop has completed. I am clearly not understanding what is going on. Is there a better pattern to follow for something like this or am I thinking about it incorrectly?
async function createFormConfig(files: string[]): Promise<object>
return new Promise(resolve => {
const retConfig: any = {};
for (const file of files) {
file.match(matchFilesForFormConfigMap.get('FIELD')) ?
parseCsv(file).then(parsedData => {
retConfig.fields = parsedData.data;
})
: file.match(matchFilesForFormConfigMap.get('FORM'))
? parseCsv(file).then(parsedData => retConfig.formProperties = parsedData.data[0])
: file.match(matchFilesForFormConfigMap.get('PDF'))
? parseCsv(file).then(parsedData => retConfig.jsPdfProperties = parsedData.data[0])
: file.match(matchFilesForFormConfigMap.get('META'))
? parseCsv(file).then(parsedData => {
retConfig.name = parsedData.data[0].name;
retConfig.imgType = parsedData.data[0].imgType;
// console.log(retConfig); <- THIS CONSOLE WILL OUTPUT RETCONFIG LOOKING LIKE I WANT IT
})
: file.match(matchFilesForFormConfigMap.get('PAGES'))
? parseCsv(file).then(parsedData => retConfig.pages = parsedData.data)
: console.log('there is an extra file: ' + file);
}
resolve(retConfig); // <- THIS RETURNS: {}
});
This is the code I'm using to call the function in hopes of getting my 'retConfig' filled with the CSV data.
getFilesFromDirectory(`${clOptions.directory}/**/*.csv`)
.then(async (files) => {
const config = await createFormConfig(files);
console.log(config);
})
.catch(err => console.error(err));
};

First, an async function returns a Promise, so you dont have to return one explicitely.Here is how you can simplify your code:
async function createFormConfig(files: string[]): Promise<object> {
// return new Promise(resolve => { <-- remove
const retConfig: any = {};
// ...
// The value returned by an async function is the one you get
// in the callback passed to the function `.then`
return retConfig;
// }); <-- remove
}
Then, your function createFormConfig returns the config before it has finished to compute it. Here is how you can have it computed before returning it:
async function createFormConfig(files: string[]): Promise<object> {
const retConfig: any = {};
// Return a Promise for each file that have to be parsed
const parsingCsv = files.map(async file => {
if (file.match(matchFilesForFormConfigMap.get('FIELD'))) {
const { data } = await parseCsv(file);
retConfig.fields = data;
} else if (file.match(matchFilesForFormConfigMap.get('FORM'))) {
const { data } = await parseCsv(file);
retConfig.formProperties = data[0];
} else if (file.match(matchFilesForFormConfigMap.get('PDF'))) {
const { data } = await parseCsv(file);
retConfig.jsPdfProperties = data[0];
} else if (file.match(matchFilesForFormConfigMap.get('META'))) {
const { data } = await parseCsv(file);
retConfig.name = data[0].name;
retConfig.imgType = data[0].imgType;
} else if (file.match(matchFilesForFormConfigMap.get('PAGES'))) {
const { data } = await parseCsv(file);
retConfig.pages = data;
} else {
console.log('there is an extra file: ' + file);
}
});
// Wait for the Promises to resolve
await Promise.all(parsingCsv)
return retConfig;
}

async functions already return promises, you don't need to wrap the code in a new one. Just return a value from the function and the caller will receive a promise that resolves to the returned value.
Also, you have made an async function, but you're not actually using await anywhere. So the for loop runs through the whole loop before any of your promises resolve. This is why none of the data is making it into your object.
It will really simplify your code to only use await and get rid of the then() calls. For example you can do this:
async function createFormConfig(files: string[]): Promise<object> {
const retConfig: any = {};
for (const file of files) {
if (file.match(matchFilesForFormConfigMap.get('FIELD')){
// no need for the then here
let parsedData = await parseCsv(file)
retConfig.field = parsedData.data
}
// ...etc
At the end you can just return the value:
return retConfig