I have been a bit stuck on an Async function.
What I am trying to accomplish - I am creating a batchProcessing function (batchGetSubs) which will loop through a set of files, read a ID, then make an API request, wait for a response (THE ISSUE) and then to write to a new file with the formatted data.
The issue - I have tried both Async and Await, as well as pushing promises and trying to use a Promise.all to wait for promises to be resolved, but with no success. Current behavior is that I get all of my console.logs that are in the Promise.all section before the API calls actually return all the data. I have used these articles as reference:
Asynchronous Process inside a javascript for loop
Promise All with Axios
Javascript Promise push value into array (only from function or outside?)
The Code -
async function batchGetSubs(data, command) {
console.time('batchGetSubs')
iteration = 1;
let dataArray = []; promises = [];
// LOOP THROUGH FILES, THEN GET TENANTS BY SUB
for (i = iteration; i < totalIterations; i++) {
let msIds = await loopAndDump(iteration);
// LOOP THROUGH TENANTIDS AND GET SUBSCRIPTIONS
msIds.map(async item => {
let newRecord = await getSubsByTenantId(item);
promises.push(await newRecord);
});
}
Promise.all([promises]).then(() => { // FIXME: WHY IS THIS NOT WAITING FOR ALL RESPONSES?
console.log(p1SubscriptionArray),
console.timeEnd('batchGetSubs')
});
}
async function getSubsByTenantId(msTenantId) {
let newRecord; let subscriptionArray = [];
let bearerToken = p1BearerToken;
let totalSubs = 0;
const subIdConfig = {
method: 'get',
url: ``,
headers: { 'Authorization': bearerToken }
}
await delay();
await axios(subIdConfig)
.then(async res => {
console.log('AXIOS RESPONSE', res.data);
if (res.data.items) {
let subItems = res.data.items;
console.log('^^^^^^^^^^^^^^^^^^', res.data.items)
// LOOP THROUGH AND ADD TO SUBSCRIPTION ARRAY
subItems.map(async subscription => {
if (subscription.friendlyName) {
totalSubs++;
subscriptionArray.push(subscription.friendlyName);
}
});
newRecord = { "tenantId": msTenantId, "totalSubs": totalSubs, "subscriptionFriendlyName": subscriptionArray };
} else {
// NO SUBS
newRecord = { "tenantId": msTenantId, "totalSubs": totalSubs, "subscriptionFriendlyName": ['NONE'] };
let statusCode, errorMessage;
if (error && error.response === undefined) { // GETTING STATUS -4039 or -4077 INSTEAD OF 429 WHEN CHECKING SUBS. FORCE A RESEND.
if (error.errno && error.errno === -4039 || error.errno && error.errno === -4077) statusCode = 429; errorMessage = error.code;
} else {
statusCode = error.response.status;
errorMessage = error.response.statusText;
}
console.error('ERROR:: SUBIDCONFIG SECTION THROWING ERROR: ', statusCode, portal, errorMessage);
// SORT NON-200 CALLS BASED ON STATUS CODE
switch (statusCode) {
case 403:
status403.push('subs', newRecord);
break;
case 404:
status404.push('subs', newRecord);
erroring = true;
break;
case 429:
status429.push('subs', newRecord);
erroring = true;
break;
default:
statusOther.push('subs', newRecord)
erroring = true;
break;
}
}
})
.catch(err => {
newRecord = { "tenantId": msTenantId, "totalSubs": totalSubs, "subscriptionFriendlyName": ['NONE'] };
console.error('ERROR: REQUEST IN GETSUBSBYTENANTID(): ', err)
})
.then(res => {
console.log('DO WE HAVE ANY INFORMATION? ', newRecord);
p1SubscriptionArray.push(newRecord);
resolve();
});
}
I only checked the first function, where you had put your question:
WHY IS THIS NOT WAITING FOR ALL RESPONSES?
For several reasons:
The promise array is still empty when you call Promise.all. This is because you only do a push after an await, and so that push happens asynchronously (read: later).
Even when the promises array gets populated, it will not have promises objects, but resolved values (i.e. newRecord values)
Even if promises would have been array of promises, you don't pass that array correctly to Promise.all: you wrap that array in yet another array, which then only has one entry, and that entry is not a promise, but an array.
Not related to your problem, but:
please make it a habit to explicitly declare all your variables. You did not do this for iteration, promises, nor i.
Only use .map when you do something with the return value. For pure iteration use .forEach or for. In this case, you can use the return value to extend the promises array.
If you intend to call batchGetSubs and need to know when all is done, then make sure it returns a useful promise: return Promise.all()
Here is the suggested correction for that function:
async function batchGetSubs(data, command) {
console.time('batchGetSubs')
let iteration = 1; // Declare!
let dataArray = [];
let promises = []; // declare
// declare i
for (let i = iteration; i < totalIterations; i++) {
let msIds = await loopAndDump(iteration);
// Use the return value of the map method. No need for async callback
promises.push(...msIds.map(item => {
// Get the promise, not the resolved value, as that will come too late:
return getSubsByTenantId(item);
}));
}
// promises is already an array; don't make it an array of arrays.
// And: return the resulting promise: it may be useful to the caller.
return Promise.all(promises).then(() => {
console.log(p1SubscriptionArray),
console.timeEnd('batchGetSubs')
});
}
You do a await and then in the same time.
You do a promise/then or await/async
do
async function getSubsByTenantId(msTenantId) {
let newRecord; let subscriptionArray = [];
let bearerToken = p1BearerToken;
let totalSubs = 0;
const subIdConfig = {
method: 'get',
url: ``,
headers: { 'Authorization': bearerToken }
}
await delay();
const r = await axios(subIdConfig)
const reponse = r.data
console.log(response)
}
if you want to try catch do this :
async function getSubsByTenantId(msTenantId) {
let newRecord; let subscriptionArray = [];
let bearerToken = p1BearerToken;
let totalSubs = 0;
const subIdConfig = {
method: 'get',
url: ``,
headers: { 'Authorization': bearerToken }
}
await delay();
let r
try {
r = await axios(subIdConfig)
} catch (error) {
console.log(error)
}
const reponse = r.data
console.log(response)
}
Related
const services = ['a', 'b', 'c', 'd'];
async function foo(service) {
// function to get one jobUrl data called by map later on
const getJob = async ({ url: jobUrl }) => {
const response = await fetch(jobUrl, { headers: headers });
const { result, timestamp: timeStamp } = await response.json();
if (result === 'SUCCESS') {
return { jobUrl, timeStamp };
}
};
// Find the list of jobs for a service
let url = `http://example.com`;
const response = await fetch(url, { method: 'POST', headers: headers });
const { jobs } = await response.json();
const unfiltered = await Promise.all(jobs.map(getJob));
const successful = unfiltered.filter(v => v);
let latestJob = successful.sort((obj1, obj2) => obj2.timeStamp - obj1.timeStamp)[0].jobUrl;
let arr = latestJob.split('/');
let recent = { service: service, job: arr[arr.length - 2] };
console.log(recent);
// return recent;
}
When I run the following piece of code it takes only 4 seconds for finding the latest job.
for (const service of services) {
foo(service);
}
Whereas, when I run the following piece of code it takes 15-16 seconds for finding the latest job. For this I uncomment the return recent; line of code in the last line of function foo and comment out the console.log(recent) line.
const latest = [];
for (const service of services) {
latest.push(await foo(service));
}
My understanding is that for the second piece of code since we are waiting for all the async calls to finish it takes longer. If that is the case, is there a way to speed this up?
You may want to use Promise.all or Promise.allSettled.
That way you can pass an array of async functions (promises) and await them. That way, functions are not running sequentially but async.
How do I get the value of the failed array outside of the function? Because I need this array as an input for other functions.
failed = [];
async function fetchFromApi(i) {
var ApiEndpoint = "https://jsonplaceholder.typicode.com/todos/" + i
var httpRequest = {
method: "GET"
}
var response = await fetch(ApiEndpoint, httpRequest);
if (!response.ok) { // Necessary for error handling, fetch does not return an error if HTTP request fails
failed.push(i);
console.log(failed)
} else {
let symbolData = await response.json();
console.log(JSON.stringify(symbolData))
return JSON.stringify(symbolData);
}
};
fetchFromApi("abc")
console.log(failed)
The console.log("failed") in the code above gives me an empty array []. I want it to be getting ["abc"] when this variable being called outside of the function
UPDATE EDIT (new follow-up question):
I tried #SlothOverlord 's solution and it seems to work below is the example.
const myFunction = async (j) => {
failed = [];
async function fetchFromApi(i) {
var ApiEndpoint = "https://jsonplaceholder.typicode.com/todos/" + i
var httpRequest = {
method: "GET"
}
var response = await fetch(ApiEndpoint, httpRequest);
if (!response.ok) { // Necessary for error handling, fetch does not return an error if HTTP request fails
failed.push(i);
} else {
let symbolData = await response.json();
console.log(JSON.stringify(symbolData))
return JSON.stringify(symbolData);
}
};
await fetchFromApi(j)
return failed
}
myFunction("abc").then(data=>console.log(failed))
However, when I add in a forEach statement within the function, it breaks and returns an empty array again. See example below. What's going on here?
const myFunction = async (j) => {
failed = [];
async function fetchFromApi(arrays) {
arrays.forEach(async function(i) {
var ApiEndpoint = "https://jsonplaceholder.typicode.com/todos/" + i
var httpRequest = {
method: "GET"
}
var response = await fetch(ApiEndpoint, httpRequest);
if (!response.ok) { // Necessary for error handling, fetch does not return an error if HTTP request fails
failed.push(i);
// console.log(failed)
} else {
let symbolData = await response.json();
console.log(JSON.stringify(symbolData))
return JSON.stringify(symbolData);
}
});
}
await fetchFromApi(j)
// console.log(failed)
return failed
}
myFunction(["aaa", "abc"]).then(data=>console.log(failed))
You need to await the fetchFromApi("abc") function.
Async await only works INSIDE the function, not outside of it.
So what happened is:
fetchFromApi("abc") is called
console.log(failed) is empty
fetchFromApi("abc") finished
The solution is to wrap the function in async and await the fetchFromApi call
const myFunction = async () => {
//...
failed = [];
async function fetchFromApi(i) {
//...
}
await fetchFromApi("abc")
console.log(failed)
}
fetchFromApi is an async method, you need to await it if you want to see it's result logged.
JavaScript has a concurrency model based on an event loop, which is responsible for executing the code, collecting and processing events. when the async call will have it's result it will be placed in the callback queue and on the next iteration (tick), it will be popped out from there and placed in the execution stack to be processed. when you use await, the exection call will be halted at that point and will progress until the async call has it's result.
failed = [];
async function fetchFromApi(i) {
var ApiEndpoint = "https://jsonplaceholder.typicode.com/todos/" + i
var httpRequest = {
method: "GET"
}
var response = await fetch(ApiEndpoint, httpRequest);
if (!response.ok) { // Necessary for error handling, fetch does not return an error if HTTP request fails
failed.push(i);
console.log(failed)
} else {
let symbolData = await response.json();
console.log(JSON.stringify(symbolData))
return JSON.stringify(symbolData);
}
};
await fetchFromApi("abc")
console.log(failed)
JS event loop - https://developer.mozilla.org/en-US/docs/Web/JavaScript/EventLoop
I am trying to get some data through an api call to a database
but I have problem returning the data to the main function
async function getKnackRecords(objID, filter, sortObj, recordID) {
let api_url = `https://api.knack.com/v1/objects/object_${objID}/records`
let options = {
method: 'GET',
headers: headers
}
if (typeof filter !== 'undefined') {
api_url += '?filters=' + encodeURIComponent(JSON.stringify(filter)) + '&sort_field=' + sortObj.field + '&sort_order=' + sortObj.order
} else {
api_url += '/' + recordID
}
let response = fetch(api_url, options)
console.log(response) // executed 1st
let records = response.then(await function(response) {
console.log(response) // executed 3rd
data = response.json()
return data.then(await async function(data) {
console.log(data.records) // executed 5th
return data.records
})
})
console.log(records, 'outer') // executed 2nd
return response
In the main function call the getKnackRecord function
let p = await getKnackRecords(objID, filter, sortObj)
console.log(p, 'main') // executed 4th
I can't fix the execution order. I want to return the records from the api call but had been unsuccessful. Any help is much appreciated!
You don't need, and should not use async/await inside a promise.
Async/await use promises under the hood.
I'd suggest you to get comfortable with Promises before you try to use async/await.
Here is the MDN documentation for promises
https://developer.mozilla.org/fr/docs/Web/JavaScript/Reference/Global_Objects/Promise
Your function with vanilla Promises:
function getSnackRecords(...myArgs){
const url = ...
const options = {...}
return fetch(api_url, options)
.then(response => response.json())
.then(data => data.records)
}
...
getSnackRecords(...).then(records => doSomething(records))
With async/await:
async function getSnackRecords(...myArgs){
const url = ...
const options = {...}
const response = await fetch(api_url, options)
const data = await response.json()
const records = data.records
return records
}
...
async function someFn(){
const records = await getSnackRecords(...)
//do something
}
In very short, inside an async function you can await for an asynchronous task to be performed instead of wrapping it in .then
You could return a Promise from your getKnackRecords() function.
Than, outside of it, after requesting it you're good to go with another .then() chain:
const getKnackRecords = (objID, filter, sortObj, recordID) => {
const filt = encodeURIComponent(JSON.stringify(filter));
const api_url = `https://jsonplaceholder.typicode.com/${objID}` + (
recordID ?
`/${recordID}` :
`?filters=${filt}&sort_field=${sortObj.field}&sort_order=${sortObj.order}`
);
return fetch(api_url).then(res => res.json()); // <Promise>
};
getKnackRecords("todos", null, null, "24").then(resJSON => {
console.log(resJSON);
});
Been on this for over 24 hours, everything seems to work as I want but the promise keeps returning null.
[
null,
null
]
here are my codes:
let vettedBatch = currentBatch.Items.map((current) => {
getUser(current.userId).then((res) => {
// return resolve(JSON.parse(res.body));
let body = JSON.parse(res.body);
if (body.hasOwnProperty("Item")) {
return body;
} else {
//if user does not exist on the users table based on ID, lets take his transaction out of the fail-safe table
console.log(
`user with id number ${current.userId} with transaction id ${current.txId} do not exist or must have been disabled`
);
User.deleteWithdrawalTx(current.txId).then(() => {
console.log(
`transaction id ${current.txId} delete for unknown user with userId ${current.userId}`
);
});
}
});
});
You need to use Promise.all:
const data = [];
const promises = currentBatch.Items.map(async current => {
return await getUser(current.userId)
});
Promise.all(promises)
.then(res => {
res.map(item => {
let { body } = JSON.parse(item);
if (body.hasOwnProperty("Item")) {
data.push(body);
} else {
console.log('message');
}
})
})
Instead of mixing async/await and Promise syntax, I would suggest you to stick to one.
Here would be your code written fully in async/await syntax:
const getData = async () => { // async function instead of new Promise()...
return Promise.all(currentBatch.Items.map(async (current) => { // make map async and await it with Promise.all()
const res = await getUser(current.userId); // await instead of .then()
let body = JSON.parse(res.body);
if (body.hasOwnProperty("Item")) {
return body;
} else {
console.log(`user with id number ${current.userId} with transaction id ${current.txId} do not exist or must have been disabled`);
await User.deleteWithdrawalTx(current.txId); // await instead of .then()
console.log(`transaction id ${current.txId} delete for unknown user with userId ${current.userId}`);
// You should return something here too, but I dont know what you want to return, so...
}
}));
}
let vettedBatch = await getData(); // await the async function
Your problem is actually a deviation of this question: How do I return the response from an asynchronous call?. Should be fixed in my answer, but I still suggest you to check out the linked thread.
The function below calls several asynchronous functions in a for loop. It's parsing different CSV files to build a single JavaScript object. I'd like to return the object after the for loop is done. Its returning the empty object right away while it does the asynchronous tasks. Makes sense, however I have tried various Promise / async /await combinations hopes of running something once the for loop has completed. I am clearly not understanding what is going on. Is there a better pattern to follow for something like this or am I thinking about it incorrectly?
async function createFormConfig(files: string[]): Promise<object>
return new Promise(resolve => {
const retConfig: any = {};
for (const file of files) {
file.match(matchFilesForFormConfigMap.get('FIELD')) ?
parseCsv(file).then(parsedData => {
retConfig.fields = parsedData.data;
})
: file.match(matchFilesForFormConfigMap.get('FORM'))
? parseCsv(file).then(parsedData => retConfig.formProperties = parsedData.data[0])
: file.match(matchFilesForFormConfigMap.get('PDF'))
? parseCsv(file).then(parsedData => retConfig.jsPdfProperties = parsedData.data[0])
: file.match(matchFilesForFormConfigMap.get('META'))
? parseCsv(file).then(parsedData => {
retConfig.name = parsedData.data[0].name;
retConfig.imgType = parsedData.data[0].imgType;
// console.log(retConfig); <- THIS CONSOLE WILL OUTPUT RETCONFIG LOOKING LIKE I WANT IT
})
: file.match(matchFilesForFormConfigMap.get('PAGES'))
? parseCsv(file).then(parsedData => retConfig.pages = parsedData.data)
: console.log('there is an extra file: ' + file);
}
resolve(retConfig); // <- THIS RETURNS: {}
});
This is the code I'm using to call the function in hopes of getting my 'retConfig' filled with the CSV data.
getFilesFromDirectory(`${clOptions.directory}/**/*.csv`)
.then(async (files) => {
const config = await createFormConfig(files);
console.log(config);
})
.catch(err => console.error(err));
};
First, an async function returns a Promise, so you dont have to return one explicitely.Here is how you can simplify your code:
async function createFormConfig(files: string[]): Promise<object> {
// return new Promise(resolve => { <-- remove
const retConfig: any = {};
// ...
// The value returned by an async function is the one you get
// in the callback passed to the function `.then`
return retConfig;
// }); <-- remove
}
Then, your function createFormConfig returns the config before it has finished to compute it. Here is how you can have it computed before returning it:
async function createFormConfig(files: string[]): Promise<object> {
const retConfig: any = {};
// Return a Promise for each file that have to be parsed
const parsingCsv = files.map(async file => {
if (file.match(matchFilesForFormConfigMap.get('FIELD'))) {
const { data } = await parseCsv(file);
retConfig.fields = data;
} else if (file.match(matchFilesForFormConfigMap.get('FORM'))) {
const { data } = await parseCsv(file);
retConfig.formProperties = data[0];
} else if (file.match(matchFilesForFormConfigMap.get('PDF'))) {
const { data } = await parseCsv(file);
retConfig.jsPdfProperties = data[0];
} else if (file.match(matchFilesForFormConfigMap.get('META'))) {
const { data } = await parseCsv(file);
retConfig.name = data[0].name;
retConfig.imgType = data[0].imgType;
} else if (file.match(matchFilesForFormConfigMap.get('PAGES'))) {
const { data } = await parseCsv(file);
retConfig.pages = data;
} else {
console.log('there is an extra file: ' + file);
}
});
// Wait for the Promises to resolve
await Promise.all(parsingCsv)
return retConfig;
}
async functions already return promises, you don't need to wrap the code in a new one. Just return a value from the function and the caller will receive a promise that resolves to the returned value.
Also, you have made an async function, but you're not actually using await anywhere. So the for loop runs through the whole loop before any of your promises resolve. This is why none of the data is making it into your object.
It will really simplify your code to only use await and get rid of the then() calls. For example you can do this:
async function createFormConfig(files: string[]): Promise<object> {
const retConfig: any = {};
for (const file of files) {
if (file.match(matchFilesForFormConfigMap.get('FIELD')){
// no need for the then here
let parsedData = await parseCsv(file)
retConfig.field = parsedData.data
}
// ...etc
At the end you can just return the value:
return retConfig