I want to create a queue where I always execute two httpRequests in parallel. When one request completes, the next should start so that always two run in parallel.
I want to merge the results and return it as a whole, when all requests have finished.
My first attempt was to create a Subject (queue) where I pass the urls with queue.next(url). Immediately after theat I complete the subject. I use mergeMap to limit the execution to two requests at a time (queueObservable).
I Subscribe to the queueObservable and add the response to an array (list).
Then I await the promise of the queueObservable, which should resolve when the observable completes.
But what is happening is, that the promise gets resolved, even before the first next callback is executed and before the observable is completed.
In my second attempt, I created a new Promise and resolve it in the completed callback of the queueObservable. This approach is working fine, but I don't understand why the first approach is not working.
Can you explain it to me?
And is there a smarter solution to solve this problem?
Kind regards,
David
import { of, Subject } from 'rxjs';
import { delay, mergeMap } from 'rxjs/operators';
const testFunction1 = async () => {
const queue = new Subject();
const list = [];
const queueObservable = queue.pipe(mergeMap(val => of(val).pipe(delay(500)), 2));
queueObservable.subscribe((next) => {
console.log(next);
list.push(next);
}, () => { },
(complete) => {
console.log('Observable1 completed');
});
queue.next('HTTP1: 1');
queue.next('HTTP1: 2');
queue.next('HTTP1: 3');
queue.next('HTTP1: 4');
queue.complete();
await queueObservable.toPromise(); // I expect this to resolve when all values passed the mergeMap pipe.
return list;
}
const testFunction2 = async () => {
const list = [];
await new Promise(resolve => {
const queue = new Subject();
const queueObservable = queue.pipe(mergeMap(val => of(val).pipe(delay(500)), 2));
queueObservable.subscribe((next) => {
console.log(next);
list.push(next);
}, () => { },
() => {
console.log('Observable2 completed'); // This gets called when all values passed mergeMap pipe.
resolve();
});
queue.next('HTTP2: 1');
queue.next('HTTP2: 2');
queue.next('HTTP2: 3');
queue.next('HTTP2: 4');
queue.complete();
});
return list;
}
testFunction1().then(res => {
console.log('Over1: ' + JSON.stringify(res));
});
testFunction2().then(res => {
console.log('Over2: ' + JSON.stringify(res));
});
Edit:
This diagram shows basically what I want to archieve. The piped observable (queueObservable) completes, when all other observables are completed. When I call toPromise on the piped observable it should only resolve when the piped Observable is completed (At least thats my understanding of the toPromise functionality).
Could you do something like this (fiddle)
const urlsSink = new Subject();
const responses = urlsSink.pipe(bufferCount(2), mergeMap((urls) => {
return combineLatest(urls.map(url => makeHttpRequest(url)));
}));
Then you can subscribe to responses and push your URLs into urlsSink. bufferCount will group URLs into groups of two. combineLatest will combine the two HTTP observables into one combined HTTP observable.
This is because you are delaying the values, the values are not yet emitted but you completed the subject. Once the subject complete, the promise should be resolved.
Try this
setTimeout(() => queue.complete(), 2000);
This will force case 1 to work but this is not suitable for making request.
seems queueObservable's promise follow queue's completion.
you could use toArray operator to merge to result.
const queueObservable = from(listOfURL).pipe(
mergeMap(val => of(val).pipe(delay(500)), 2),
mergeMap(val => fetch(url + val)),
toArray(),
);
Related
I got this question in an interview. A weather API endpoint can return the weather data for Pincode, city, state, and country.
Given a list of APIs and their priorities, call them parallelly and return the data for the API, which has high priority.
APIs = [
{ url: "api.weather.com/pin/213131", priority: 0 },
{ url: "api.weather.com/state/california", priority: 2 },
{ url: "api.weather.com/city/sanfrancisco", priority: 1 },
{ url: "api.weather.com/country/usa", priority: 3 },
];
In the above list, the priority order is pin > city > state > country. This means calling all the APIs parallelly; if the API with Pincode returned the data first, resolve the Promise immediately; if not, it should resolve the next higher priority one.
I thought of Promise.race(), but that won't consider a priority. Then I thought of waiting till a timeout occurs, and below is my code for the same. It simply waits till timeout, and if the high priority one is resolved first, then the real Promise will be resolved. After a specific timeout, it simply resolves with the first high priority response. But the interviewer wants me to implement it without a timeout.
Below is the code with a timeout. Does anyone know how to implement it without a timeout and more generic fashion?
function resolvePromiseWithPriority(APIS, timeout) {
let PROMISES = APIS.sort((a, b) => a.priority - b.priority).map((api) => {
return () =>
new Promise((resolve, reject) => {
fetch(api.url)
.then((data) => resolve(data))
.catch((err) => reject(err));
});
});
let priorities = [...APIS.map((item) => item.priority)];
let maxPriority = Math.min(...priorities);
let minPriority = Math.max(...priorities);
let results = [];
let startTime = Date.now();
return new Promise((resolve, reject) => {
PROMISES.forEach((promise, priority) => {
promise()
.then((data) => {
results[priority] = data;
let gap = (Date.now() - startTime) / 1000;
if (gap > timeout) {
// resolve the current high priority promise, if no promises resolved before the timeout, resolve the first one resolved.
// If all promises are rejected, reject this promise.
if (!results[minPriority] instanceof Error)
resolve(resolve[minPriority]);
for (let item of results) {
if (!item instanceof Error) resolve(item);
reject("No promises resolved !!");
}
} else {
if (priority === maxPriority) {
// if the high priority promise gets it's data, resolve immediately.
resolve(results[priority]);
}
if (priority < minPriority) {
minPriority = priority;
}
}
})
.catch((err) => {
results[priority] = err;
});
});
});
}
On the understanding that ...
Given a list of APIs and their priorities, call them parallelly and return the data for the API, which has high priority.
expands to ...
Given a list of APIs and their priorities, call them in parallel and return the data from the highest priority API that successfully delivers data; if no API is successful, then provide a custon Error.
then you neither want nor need a timeout, and the requirement can be met with three Array methods .sort(), .map() and .reduce() as follows:
function resolvePromiseWithPriority(APIS) {
return APIS
.sort((a, b) => a.priority - b.priority); // sort the APIS array into priority order.
.map(api => fetch(api.url)) // initialte all fetches in parallel.
.reduce((cumulativePromise, fetchPromise) => cumulativePromise.catch(() => fetchPromise); // build a .catch().catch().catch()... chain
}, Promise.reject()) // Rejected promise as a starter for the catch chain.
.catch(() => new Error('No successful fetches')); // Arrive here if none of the fetches is successful.
}
The .sort() and .map() methods are fully described by comments in the code.
The catch chain formed by the .reduce() method works as follows:
the starter Promise is immediately caught and allows the Pincode fetch a chance of providing the required data.
if fetch failure (Pincode or otherwise) gives the next highest priority fetch a chance of providing the required data, and so on, and so on, down the chain.
as soon as a successful fetch is encountered, the chain adopts its success path; all further catches in the chain are bypassed; with no thens anywhere in the chain, the promise returned by resolvePromiseWithPriority() delivers the successful fetch's data; any lower priority successes are effectively ignored as they would only ever be considered in the (bypassed) catches;
if all of the fetches fail, a terminal catch injects your custom 'No promises resolved' error.
Here's what I came up with:
function resolvePromiseWithPriority(apis) {
const promises = apis
.sort((a, b) => a.priority - b.priority)
.map(api => fetch(api.url).then(res => res.ok ? res.json() : Promise.reject(res.status));
const contenders = [];
let prev = null;
for (const promise of promises) {
prev = Promise.allSettled([
prev ?? Promise.reject(),
promise
]).then(([{status: prevStatus}]) => {
if (prevStatus == 'rejected') return promise;
// else ignore the promise
});
contenders.push(prev);
}
return Promise.any(contenders);
}
The contenders wait for all previous promises (those with a higher priority) and resolve (to the same result as the current promise) only once all those rejected. Then they're passed into Promise.any, which will go with the result of the first contender to fulfill, or an AggregateError of all request failures in order of priority.
Notice a contender fulfills to undefined if any of the previous promises did fulfill, but this doesn't really matter - Promise.any would go with the first one. The code would work also with Promise.allSettled([prev, promise]).then(() => promise), but imo checking prevStatus makes the intention a bit clearer.
The usage of Promise.allSettled (over prev.then(() => promise, () => promise)) and Promise.any ensures that the code causes no unhandled rejections.
Assuming you already have requests promises sorted by the priority field in sortedPromises array we just need to get the result of the first successful request:
function getFirstSuccessfulResponse(sortedPromises) {
let response = null;
for (const promise of sortedPromises) {
try {
response = await promise;
return response;
} catch (err) {
console.warn('Could not get data from the request')
}
}
}
I am making multiple calls with Promise.
My API endpoints to fetch are:
https://www.api-football.com/demo/v2/statistics/357/5/2019-08-30
https://www.api-football.com/demo/v2/statistics/357/5/2019-09-30
https://www.api-football.com/demo/v2/statistics/357/5/2019-10-30
See the code
export function getTeamsStats(league, team, type) {
return function(dispatch) {
const url = "https://www.api-football.com/demo/v2/statistics";
let dates = ["2019-08-30", "2019-09-30", "2019-10-30"];
const getAllData = (dates, i) => {
return Promise.allSettled(dates.map(x => url + '/' + 357 + '/' + 5 + '/' + x).map(fetchData));
}
const fetchData = (URL) => {
return axios
.get(URL)
.then(res => {
const {
matchsPlayed: { total: teamsTotalMatchsPlayed},
} = res.data.api.statistics.matchs;
const matchsPlayed = teamsTotalMatchsPlayed;
dispatch(receivedTeamsStat(matchsPlayed, type));
})
.catch(e => {
console.log(e);
});
}
getAllData(dates).then(resp=>{console.log(resp)}).catch(e=>{console.log(e)})
}
}
Then in my component, I put in an array the matches played from this specific team ( Sao Paulo in this example ) from the initial date 30-8-2019 to 30-10-2019
const [dataHomeTeam, setDataHomeTeam] = useState([]);
useEffect(() => {
if (!team.matchsPlayed) {
return ;
}
setDataHomeTeam(prev =>
prev.concat([
{
matches: team.matchsPlayed,
}
])
);
},[team.matchsPlayed]);
console.log('Data Array', dataHomeTeam);
The problem is that normally the in the first render of the page I have the right order of the matches made from 30-8-2019 to 30-10-2019
See the console log image
But sometimes not, see here
So the question is, how can I make sure that the Promise is returning me the right order of the requests?
I am using Promise.allSettled, the multiple asynchronous tasks that are not dependent on one another to complete successfully, but the order is not always right.
This should be resolved via Promise.all because as described in documentation
Returned values will be in order of the Promises passed, regardless of
completion order.
The problem you have is that you are setting your state by calling dispatch for each promise based on it's completion and since promises can finish at any given time (e.g. 3rd request could finish first because they are sent at the same time), dispatch will be called and your state management will set that as first response (that's why you "sometimes" get such behaviour, because it's up to the network which one will finish first).
This would mean that either you should change your dispatch to receive array of already finished promises, or call dispatch one-by-one once they are finished which is shown in code below - resolve all and dispatch in order:
export function getTeamsStats(league, team, type) {
return function (dispatch) {
const url = "https://www.api-football.com/demo/v2/statistics";
let dates = ["2019-08-30", "2019-09-30", "2019-10-30"];
const getAllData = (dates, i) => {
return Promise.all(dates.map(x => url + '/' + 357 + '/' + 5 + '/' + x).map(fetchData));
}
const fetchData = (URL) => {
return axios
.get(URL)
.then(res => {
const {
matchsPlayed: { total: teamsTotalMatchsPlayed },
} = res.data.api.statistics.matchs;
return teamsTotalMatchsPlayed;
})
.catch(e => {
console.log(e);
});
}
getAllData(dates).then(resp => {
console.log(resp)
// 'resp' here are all 'teamsTotalMatchsPlayed' in correct order (here I mean order of call, not promise completion)
// so just dispatch them in order
resp.map(matchsPlayed => receivedTeamsStat(matchsPlayed, type));
}).catch(e => {
console.log(e)
})
}
}
Please note that I've maybe made some syntax error, but you get the idea.
The ideal way of achieving your requirment is by using Promise.all(). It is because of two main reasons,
To maintain the return value in the order of the Promises passed, regardless of completion order.
Returned values will be in order of the Promises passed, regardless of
completion order.
Refer to the Return value section in the link
To reject the returned promise (short circuit) if any of the promises in the iterable are rejected.
This is also important as well. We don't need to wait for all the asynchronous iterable promises to be resolved/rejected, if the first iterable promise of fetchData rejects we can short circuit and reject the returned promise.
On the other hand Promise.allSettled(),
Promise.allSettled() method returns a promise that resolves after all of the given promises have either fulfilled or rejected, with an
array of objects that each describes the outcome of each promise.
It also doesn't maintain the order in the returned promise.
Promise.allSettled() method never short-circuits. Always Promise fulfilled, never rejected.
Refer to the following comparison table between Promise.all() and Promise.allSettled(),
<script src="https://gist.github.com/Seralahthan/9934ba2bd185a8ccfbdd8e4b3523ea23.js"></script>
How, you have done,
function updateUI(value) {
console.log(value);
// Do something to update the UI.
}
// Note that order of resolution of Promises is 2, 1, 3
const promise1 = new Promise((resolve) => setTimeout(resolve, 200, 1)).then(updateUI);
const promise2 = new Promise((resolve) => setTimeout(resolve, 100, 2)).then(updateUI);
const promise3 = new Promise((resolve) => setTimeout(resolve, 300, 3)).then(updateUI);
const promises = [promise1, promise2, promise3];
Promise.allSettled(promises).
then((value) => console.log('Nothing to do here', value));
// Output: 2, 1, 3
// Here we update the UI as soon as the result is obtained. As a result, the UI is also updated in the
// order in which the promise was resolved.
In other words, we wait not only for the network call but we wait for both the network call and the UI update to complete for each id which is not you want.
How you should have done instead,
// Note that order of resolution of Promises is 2, 1, 3 (Same as previous)
const promise1 = new Promise((resolve) => setTimeout(resolve, 200, 1));
const promise2 = new Promise((resolve) => setTimeout(resolve, 100, 2));
const promise3 = new Promise((resolve) => setTimeout(resolve, 300, 3));
const promises = [promise1, promise2, promise3];
Promise.allSettled(promises).
then((results) => results.forEach((result) => updateUI(result.value)));
// Output: 1, 2, 3
// Here, we wait for all the network requests to complete and then loop through the results and update the UI.
// This ensures that the result is in order.
If you don't want to wait for all Promises to resolve and want to update the UI as soon as one resolves and still maintain order, then you need to pass the position of the element in the array and then use that position to update the element in place at the given position in the array.
Hope this helps.
export function getTeamsStats(league, team, type) {
return function (dispatch) {
const URL = "https://www.api-football.com/demo/v2/statistics";
let DATES = ["2019-08-30", "2019-09-30", "2019-10-30"];
return Promise.all(
DATES.map(date => axios.get(`${URL}/357/5/${date}`))
.then(responseList => {
responseList.map((res) => {
const {
matchsPlayed: { total: teamsTotalMatchsPlayed },
} = res.data.api.statistics.matchs;
const matchsPlayed = teamsTotalMatchsPlayed;
dispatch(receivedTeamsStat(matchsPlayed, type));
});
})
.catch((e) => console.log(e))
);
};
}
Promise.all preserves's the order. You should wait for all the api promise's to resolve first, before action dipatching.
Helpful Article: Promise.all: Order of resolved values
Tried with below code not wait for post call success jumping to next iteration before response comes.
Requirement:Need to have next iteration after the success of two api(POST/PATCH) calls
for (item of data) {
A(item)
}
A(value) {
const resp = this.post(url, {
'rationale': value['rationale']
})
.mergeMap(tempObj => {
value['detail'] = tempObj['id']
return this.patch(url, value['extid'], value)
})
.subscribe()
}
Recently I have used the toPromise function with angular http to turn an observable into a promise. If you have the outer loop inside an async function, this may work:
// This must be executed inside an async function
for (item of data) {
await A(item)
}
async A(value) {
const resp = await this.post(url, {
'rationale': value['rationale']
})
.mergeMap(tempObj => {
value['detail'] = tempObj['id']
return this.patch(url, value['extid'], value)
}).toPromise();
}
Use from to emit the items of an array. Use concatMap to map to an Observable and only map to the next one when the previous completed.
const resp$ = from(data).pipe(
concatMap(value => this.post(url, { 'rationale': value['rationale'] }).pipe(
switchMap(tempObj => {
value['detail'] = tempObj['id']
return this.patch(url, value['extid'], value)
})
))
)
I used switchMap instead of mergeMap to indicate that the feature of mergeMap to run multiple Observables simultaneously isn't used.
You could use:
<form (ngSubmit)="submit$.next(form.value)" ...
In your component:
submit$= new Subject();
ngOnInit {
submit$.pipe(
exhaustMap(value =>
this.post(url, {'rationale': value.rationale}))
.pipe(concatMap( response => {
value.detail = response.id;
return this.patch(url, value.extid, value);
}))).subscribe(); // rember to handle unsubcribe
}
The reason I use exhaustMap generally post and path are mutating calls, so that operator ensures first submit is process ignore the rest while processing AKA avoid double submit
An even better way is using ngrx effects, which if you dont know it yet I recomend to learn it
submit$ = createEffect(
() => this.actions$.pipe(
ofType(FeatureActions.submit),
exhaustMap( ({value}) => // if action has value property
this.post(url, { rationale : value.rationale}))
.pipe(concatMap( response => {
value.detail = response.id;
return this.patch(url, value.extid, value);
})),
map(result => FeatureActions.submitSuccess(result))
)
);
I know that promise.all() fails when even 1 of the promise is failed. I only want to try for failed promises and don't want to run promise.all() again.
Any recommendations on how I can achieve this in minimal way?
Promises are eager construct and model a value obtained asynchronously,
a Promise is produced using some kind of producer, like fetch for instance.
If you retain a reference to this producer then you can replay the nmechanism
that produced the Promise in the first place.
// producer function
function getData (arg) {
const result = new Promise();
return result.then(value => {
return { ok:true, value };
}, error => {
return {
ok: false,
value: error,
// retry is a function which calls the producer with the same arguments
retry: () => getData(arg)
};
})
}
Then if you have something like:
const data = [];
// Promise<{ok: boolean, value: any, retry?: function}>
// No promises will fail in this array
const asyncResults = data.map(getResults);
Promise.all(asyncResults)
.then((results) => {
const successes = results.filter(res => res.ok);
const retrys = results.filter(res => !res.ok).map(res => res.retry()); // retry all failed promises
})
Memory leaks, stack overflow: because I retain a reference to original arguments in order to retry and the algorithm is recursive there could be a memory leak. However the algorithm cannot "stack overflow":
getData calls do not get "deeper" over time (see retry definition)
the asyncrhonicity of the algorithm prevent this behaviour if a promise was never resolved
old data is properly discarded when accessing the results as const resultData = results.filter(res => res.ok).map(res => res.value);
However the algorithm could take a long time to settle if a promise keep on not getting resolved and prevent access to the rest of the values.
In an alternative I suggest you take a look at another async primitive, not yet part of the language (maybe some day) : Observables which are designed for this kind of tasks: lazy, retry-able async operations.
You may use async package and wrap all promise calls with closure with done argument.
Then simply resolve results.
const async = require('async');
const promiseAll = promises => {
return new Promise((resolve) => {
// preparing array of functions which has done method as callback
const parallelCalls = promises.map(promise => {
return done => {
promise
.then(result => done(null, result)
.catch(error => {
console.error(error.message);
done();
});
}
});
// calling array of functions in parallel
async.parallel(
parallelCalls,
(_, results) => resolve(results.filter(Boolean))
);
});
};
router.get('/something', async (req, res) => {
...
const results = await promiseAll(promises);
...
});
Or we can simply do Promise.all without using async package:
router.get('/something', async (req, res) => {
...
const results = (
await Promise.all(
promises.map(promise => {
return new Promise(resolve => {
promise.then(resolve).catch(e => resolve());
});
});
)
).filter(Boolean);
...
});
I need to make several async calls that depend on each other. I initially wrote the code and used Promise.all to make async in steps. I looped through my data and created a async method in order to put all of the needed actions into an array to pass into Promise.all(). This works fine, but how could I do the same using Observables. I've read that forkJoin is the equivalent of Promise.all, but how could i loop through the data and wrap my async function, and then execute it before moving to the next flatMap?
public getMonthly(){
return this.http.get(url)
.flatMap(response => {
// need to convert this?
let actions = response.json().map(this.asyncMonthlyRecord);
return Promise.all(actions);
})
.flatMap(()=> this.queryMonthly())
.map(this.convertFromSQl)
.catch((error:any)=> Observable.throw(error || 'Server Error'));
}
private asyncMonthlyRecord = (record):Promise<any> => {
return this.setUsage(record,'HILowMonthly');
}
private queryMonthly(){
return this.storage.query('SELECT * FROM HILowMonthly')
}
getMonthly().subscribe(x => console.info(x)); // logs data from SQLite perfectly...
I think what you want is something like this
Rx.Observable.of({ itemIds: [1, 2, 3, 4, 5 ]})
.mergeMap(response => {
const promises = response.itemIds
.map(id => {
return new Promise((resolve, reject) => {
// Wait for some random time, then resolve the promise.
const timeout = Math.floor(Math.random() * 5000);
setTimeout(() => {
console.log(`Finished promise ${id}`); // debug only
resolve({id, resolved: true})
}, timeout);
});
});
// Wait until all Promises have been resolved, then return all
// of them in a single and ordered array.
return Rx.Observable.forkJoin(promises);
})
.subscribe(x => console.log(x));
Working code on jsbin
Notice that the promises resolve in an arbitrary order but are returned in the correct order.
The commented code in the jsbin example also shows each promise could be resolved individually and merged back into the original stream in case the order of the promises is not important.