TLDR: How to use ES6 fetch to download synchronously?
I'm trying to write a script in Node to download data from an API till there is no more to download e.g. the endpoint has a dataset of size roughly 12000 but only provides 100 per call, and I need all the data. So I've decided to have it downloaded synchronously, and only stop when the json returned is finally empty.
// function to make one api GET call
getData = (offset) => {
return fetch('http://...'+'?start=' + offset)
.then(...)
}
// make api calls until json is finally empty,
// indicating that I've downloaded all the data
offset = 0
const results = getData(offset)
while (results.length != 0) {
// combine results...
i += 100 // move offset
results = getData(i)
}
Because I don't know precisely how large the data is and at which offset it ends, whether or not to make another call depends on the last one.
The code above fails because the promise from getData() does not resolve in time for the while loop. In another language, this would be okay as getData blocks until it is completed. I've tried to await getData but it needs to be in an async (which I don't know where to place, I already have promises). Is there any way to force getData() to block until it is resolved?
you can mark your getData() function as async
async getData() {
return fetch('http://...'+'?start=' + offset)
.then(...)
}
then await for the returned promise completion
await const results = getData(offset)
Or you can modify your code to handle the logic of whether to make another call in the promise callbacks. Something like
function fetchData(offset) {
if (offset < 1000) {
return Promise.resolve([1,2,3].map(i=>offset+i));
} else {
return Promise.resolve([]);
}
}
function needsMoreData(results) {
return results.length > 0;
}
function fetchNextDataIfNeeded(results, offset) {
if (needsMoreData(results)) {
return fetchRemainingData(offset + 100)
.then(nextResults=>[...results, ...nextResults]);
}
return Promise.resolve(results);
}
function fetchRemainingData( offset) {
return fetchData(offset)
.then(results=>fetchNextDataIfNeeded(results, offset));
}
function fetchAllData() {
return fetchRemainingData(0)
.then(results=>console.log(results));
}
fetchAllData();
See https://jsfiddle.net/rwycbn5q/1/
I recently found myself in a similar situation, but some functions cannot be made synchronously. But you could do something like this:
let results=[];
getData(idx)=>{
fetch("url").then(data=>{
if(data!=={}){ //Or some other continuation check
results.push(data); //Or whatever you want to do with the data
getData(idx+100);
}
})
}
getData(0);
Related
I'm making a request to a service that generates several strings in rapid succession. My problem arise from having to return a promise with this service, as I do not control the service process that returns the string.
I want to emphasize the fact that I necessarily need to return a promise.
Right now, what I have is the main function (handler.ts) and it doesn't mantain any business logic, including only the following:
public static callback: any;
public static async init(configuration, callback): Promise<any> {
this.callback = callback;
...
return new Promise((resolve, reject) => {
try {
const result = await Service.bootstrap(configuration)
return resolve(result)
} catch(err) {
reject(err)
}
}
}
This handler calls the service, which has the bootstrap function that performs a call to a .js file that obtains some information from my computer and then returns a string about it.
import loadComputerInformation from 'load.js'
public async bootstrap() : Promise<any> {
loadComputerInformation();
function useComputerInfoString() {
// window.getInfo is generated by loadComputerInformation(), and I can not control
// when does it return it, as it generates several strings with rapid succession
// and until the function exists, I will not get my string.
if (typeof window.getInfo !== 'function') {return;}
const data = window.getInfo();
if (data.finished) {
clearTimeout(timeoutId);
const infoString = data.computerInfo;
Service.axiosCall(infoString);
}
}
// I have to set an interval to perform the call several times, and until it resolves it
// it will not stop performing this call.
const timeoutId = setInterval(useComputerInfoString, 500);
}
return;
}
Therefore, the problem that I'm facing is that my promise gets lost in another thread, and I can not return the value from Service.axiosCall(infoString), which is just a standard axios call, but that necessarily needs the infoString.
Adding the axios call function just in case it is useful. Right now, I'm using the callback passed to the handler.js to return the axios call, but I want to be able to include it in the Promise without the necessity of a callback function
public static async axiosCall(blackbox): Promise<any> {
await axios.post('https://somecall.com/info', blackbox)
.then((response) => { this.callback(element)
return element);
}
}
Any idea of how to solve this?
Highlight
Please note that loadComputerInformation() asynchronously loads Window.getInfo(), but it does not resolve only one value, but several, therefore I can not await on Window.getInfo() because at the beggining it does not exist, and will only return undefined
Also, right now, the code is up and running, but the way it is returning the value is with the callback and not as a promise.
Try a bootstrap function whose returned promise resolves with the response returned from an axios call.
This suggestion (based on information in the post) is vanilla JavaScript to demonstrate how to the problem might be approached. Obviously modification to integrate it into the application will be needed:
const bootstrap = () => new Promise( resolve => {
loadComputerInformation();
let state = "load";
const timer = setInterval( ()=> {
if( state == "load") {
if( typeof window.getData != "function") {
return;
}
state = "get";
}
let data;
if( state == "get") {
data = window.getData();
if(!data.finished) {
return;
}
// state = "request";
}
clearInterval(timer);
resolve( axios('post', "https:example.com/info", data.computerInfo) );
}, 100); // 1/10th second?
};
Note this answer has been modified to use a state machine to wait for getData to be loaded asynchronously, then to wait for a call to getData to return a data object with a finished property, before resolving the promise returned with the promise returned by the axios call - the first answer was simpler right up until it needed to wait for getData code to be loaded asynchronously.
Beneath the question lies a problem that may have to be written off to code debt: JavaScript is event driven. loadComputerInformation appears to have been written for its results to be polled. It is hard to imagine a justifiable reason why it was left in that state.
Please bear with me I've been dropped into a new project and trying to take it all in. I've been making progress over the last couple of day but can't seem to get over this last hump. Hopefully I can explain it correctly.
I'm loading up a web form and need to make a call out to the API to get some information that may or may not be present based on the data currently loading up. I've simplified my page to basically be this.
...Get some information the user wants and start to do some work to load
up the page and set up the form.
...old stuff working fine...
//Time for my new stuff
var testValue
async function test() {
await http.post(appConfig.serviceRootUrl + '/api/XXX/YYY',
{ mProperty: myObject.collectionInObject.itemInCollection }).then(function (result) {
if (result.length < 1) {
testValue= false;
}
else if (result[0].infoIWant.trim().length > 0) {
testValue= true;
}
});
}
test();
//Originally above in the if I was just seeing if I got a result
//and setting testValue to true/false but changed it for debugging
//and I am surely getting back correct values when the data exists
//or result.length zero when no data for it
...Do a bunch of more stuff that is old and working correctly....
//Test the new stuff up above
alert(testValue);
Most of the time I get back the correct true or false in the alert but once in a while I get back undefined. I'm guessing the undefined is because it is getting to the alert before the async/await finishes. I was under the impression it won't go past the line where I call "test();". I thought it was in effect making it halt anything below test(); until the await finished. Originally it was a bit more complex but I keep stripping it down to make it (hopefully) more basic/simple.
What am I missing in my thoughts or implementation?
Any help greatly appreciated as I'm chasing my tail at this point.
This isn't how async functions work. The function only appears to wait inside the function itself. Outside the function it is called and returns a promise synchronously.
In other words if you write:
let t = test()
t will be a promise that resolves when test() returns. In your current code, if you want to respond outside the function you would need something like:
async function test() {
let result = await http.post(appConfig.serviceRootUrl + '/api/XXX/YYY',
{ mProperty: myObject.collectionInObject.itemInCollection })
if (result.length < 1) return false
else if (result[0].infoIWant.trim().length > 0) return true;
}
// test is an async function. It returns a promise.
test().then(result => alert("result: " + result ))
Edit based on comments
Here's a working version using Axios for the http.post command:
async function test() {
let result = await axios.post('https://jsonplaceholder.typicode.com/posts',
{ mProperty: "some val" })
return result.data
}
// test is an async function. It returns a promise.
test().then(result => console.log(result ))
<script src="https://cdnjs.cloudflare.com/ajax/libs/axios/0.18.0/axios.min.js"></script>
How about doing this?
...Get some information the user wants and start to do some work to load
up the page and set up the form.
...old stuff working fine...
//Time for my new stuff
var testValue
async function test() {
let promise = new Promise( (resolve, reject) => resolve( http.post(appConfig.serviceRootUrl + '/api/XXX/YYY',
{ mProperty: myObject.collectionInObject.itemInCollection }).then(function (result) {
if (result.length < 1) {
testValue= false;
}
else if (result[0].infoIWant.trim().length > 0) {
testValue= true;
}
})));
await promise;
alert(testValue);
}
test();
//Originally above in the if I was just seeing if I got a result
//and setting testValue to true/false but changed it for debugging
//and I am surely getting back correct values when the data exists
//or result.length zero when no data for it
...Do a bunch of more stuff that is old and working correctly....
//Test the new stuff up above
If you use .then() syntax, don't await, and vice versa. I was incorrect about browser compatibility with async/await, seems I haven't kept up with browser scripting, in favor of Node. But also, since you're using jQuery, $.ajax() might be a good option for you, because you don't need async/await or .then(), and you can do like so:
$.ajax(appConfig.serviceRootUrl + '/api/XXX/YYY', {
method: 'POST',
data: { mProperty: myObject.collectionInObject.itemInCollection }
}).done(function(data) {
//use result here just as you would normally within the `.then()`
})
I hope this is more helpful than my original answer.
Actually I'm not sure that Title of my question is 'correct', if you
have any idea with it, you could leave a comment and I'll rename it.
I am trying to rewrite my old function which make http-requests and insert many object at mongoDB via mongoose. I already have a working version of it, but I face a problem while using it. Basically, because when I'm trying to insertMany 20 arrays from 20+ request with ~50'000 elements from one request it cause a huge memory leak. Even with MongoDB optimization.
Logic of my code:
function main() {
server.find({locale: "en_GB"}).exec(function (err, server) {
for (let i = 0; i < server.length; i++) { //for example 20 servers
rp({url: server[i].slug}).then(response => {
auctions.count({
server: server[i].name,
lastModified: {$gte: response.data.files[0].lastModified}
}).then(function (docs) {
if (docs < 0) {
//We don't insert data if they are already up-to-date
}
else {
//I needed response.data.files[0].url and server[i].name from prev. block
//And here is my problem
requests & insertMany and then => loop main()
})
}
})
}).catch(function (error) {
console.log(error);
})
}
})
}
main()
Actually I have already trying many different things to fix it. First-of-all I was trying to add setInterval after else block like this:
setTimeout(function () {
//request every server with interval, instead of all at once
}, 1000 * (i + 1));
but I create another problem for myself because I needed to recursive my main() function right after. So I can't use: if (i === server[i].length-1) to call garbage collector or to restart main() because not all server skip count validation
Or let's see another example of mine:
I change for (let i = 0; i < server.length; i++) from 3-rd line to .map and move it from 3-rd line close to else block but setTimeout doesn't work with .map version, but as you may already understand script lose correct order and I can't make a delay with it.
Actually I already understand how to fix it at once. Just re-create array via let array_new = [], array_new.push = response.data.files[0].url with use of async/await. But I'm not a big expert in it, so I already waste a couple of hours. So the only problem for now, that I don't know how to return values from else block
As for now I'm trying to form array inside else block
function main() {
--added let array_new = [];
[v1]array_new.url += response.data.files[0].url;
[v2]array_new.push(response.data.files[0].url);
return array_new
and then call array_new array via .then , but not one of these works fine for now. So maybe someone will give me a tip or show me already answered question #Stackoverflow that could be useful in my situation.
Since you are essentially dealing with promises, you can refactor your function logic to use async await as follows:
function async main() {
try {
const servers = await server.find({locale: "en_GB"}).exec()
const data = servers.map(async ({ name, slug }) => {
const response = await rp({ url: slug })
const { lastModified, url } = response.data.files[0]
const count = await auctions.count({
server: name,
lastModified: { $gte: lastModified }
})
let result = {}
if (count > 0) result = { name, url }
return result
}).filter(d => Object.keys(d).length > 0)
Model.insertMany(data)
} catch (err) {
console.error(err)
}
}
Your problem is with logic obscured by your promises. Your main function recursively calls itself N times, where N is the number of servers. This builds up exponentially to eat memory both by the node process and MongoDB handling all the requests.
Instead of jumping into async / await, start by using the promises and waiting for the batch of N queries to complete before starting another batch. You can use [Promise.all] for this.
function main() {
server.find({locale: "en_GB"}).exec(function (err, server) {
// need to keep track of each promise for each server
let promises = []
for (let i = 0; i < server.length; i++) {
let promise = rp({
url: server[i].slug
}).then(function(response) {
// instead of nesting promises, return the promise so it is handled by
// the next then in the chain.
return auctions.count({
server: server[i].name,
lastModified: {
$gte: response.data.files[0].lastModified
}
});
}).then(function (docs) {
if (docs > 0) {
// do whatever you need to here regarding making requests and
// inserting into DB, but don't call main() here.
return requestAndInsert();
}
}).catch(function (error) {
console.log(error);
})
// add the above promise to out list.
promises.push(promise)
}
// register a new promise to run once all of the above promises generated
// by the loop have been completed
Promise.all(promises).then(function () {
// now you can call main again, optionally in a setTimeout so it waits a
// few seconds before fetchin more data.
setTimeout(main, 5000);
})
})
}
main()
Is there any other way other than setTimeout() to wait for the result come back after api call and then call that api again immediately after result is received?
In my case, I have to call n times times (depending on user input) and the api call takes 20-40 seconds to process and return result from the backend. If I send another request, while the previous one is still being processed in the backed, then it breaks the backend.
What is the right way to do call api once result is received and fully processed?
sendRequest() {
let self = this;
function f() {
if (self.state.callPerSite < self.state.selectedSites.length) {
self.processData(transid);
setTimeout(f, 20)
}
} else {
self.setState({progress: 100});
}
}
f();
}
processData(transid) {
rest.apiCall({transactionId: transid}, 'json').then(results => {
if (!firstCallIsProcessed) {
firstCallIsProcessed = true;
}
callPerSite++;
this.setState({
resultsData: results.data,
callPerSite: callPerSite
});
}).catch(err => this.onError(err));
Just a few changes:
Make your function processData return the promise, by just prefixing the return keyword:
return rest.apiCall( // ....
Chain a call to f with the then method on that return value without further need for calling setTimeout:
self.processData(transid).then(f)
That's it.
You want to look at the new await and async methods.
Check this out: https://javascript.info/async-await
I have two functions:
function manipulateData(data,callback)
function isDataValid(data)
The first one is supposed to manipulate the data and return nothing, and the second one is supposed to return true of false depending on the data's structure, without manipulating it at all.
Assuming three things:
-The manipulation doesn't require a valid data.
-The data's validation process is quite demanding IO wise.
-The data's validation process should receive the original data.
How can I use the verification function as the callback argument, so I'd be able to verify it in the background, and only throw an error from the first function, if the data is found invalid?
I want it to behave like this, where the first statement runs in the background and doesn't block the data manipulation.
function manipulateData(data, callback){
if(callback(data)==false){ return some error }
...manipulate data without waiting for the verification...
}
I hope it is indeed doable and that I'm not missing a crucial logical part in the callback mechanism.
You can do this with callbacks or Promises. With callbacks you want to use something like async, but I'd definitely go with Promises (or async/await).
You said that validation was I/O heavy so we'll assume that is an async function. I've assumed that the manipulation is not, but it doesn't really matter once you use Promise.all to wrap them up.
The following is tested in Node 6.10. It tests validation failure and success and I'm pretty sure I've got it testing with each function finishing before the other, but again Promise.all takes care of that for you anyway.
function sleep(milliseconds) {
var start = new Date().getTime();
for (var i = 0; i < 1e7; i++) {
if ((new Date().getTime() - start) > milliseconds){
break;
}
}
}
function manipulateData (data, zzz) {
if (zzz) {
sleep(zzz);
}
return 'manipulated data';
}
function isDataValid (data) {
return new Promise((resolve, reject) => {
if (data) {
resolve('ok');
} else {
reject(new Error('nok'));
}
});
}
function updateData (data, zzz) {
return Promise.all([
isDataValid(data),
manipulateData(data, zzz)
]);
}
function testIt (data, zzz) {
return updateData(data, zzz)
.then(([result, data]) => console.log(`Success! ${result} - ${data}`))
.catch(err => console.log(`Fail! ${err}`));
}
testIt(true, 0)
.then(() => testIt(false, 0))
.then(() => testIt(true, 1000))
.then(() => testIt(false, 1000));
Success! ok - manipulated data
Fail! Error: nok
Success! ok - manipulated data
Fail! Error: nok