How can I make an appendable observable queue? - javascript

I have an array where I need to make single requests using very single data set of this array. The problem that I found difficult to solve was to schedule the calls. Means, only when a request finishes, the next request starts. I was looking for an RxJs queue, but I couldn't simply find a solution.
Example:
function makeRequest(body): Observable<any> {
return someAsyncRequest(body);
}
array.forEach((entry) => {
makeRequest(entry);
});
// This is just an example how the setup is. This code does not work.
// What I need is a queue like feature in RxJs to append requests and wait before the previous one is finished.

You have quite few options, but I suppose concat or forkJoin fits you best. concat calls second API only after previous completes, while forkJoin will do the same, but only if none of them errors. If any of them errors, it will not return anything.
Example with concat:
concat(...array.map(entry => makeRequest(entry)).subscribe()
p.s. import concat as static operator:
import { concat } from 'rxjs'

if you want to have things go out one by one, but still receive all the results at once, i recommend concat -> reduce. looks like this:
concat(...array.map(entry => makeRequest(entry))).pipe(
reduce((completed, curResponse) => completed.concat([curResponse]), [])
).subscribe(allResponses => console.log(allResponses))
this structure achieves the single emission that forkjoin will give you but will do the requests one by one and then gather them once all complete. if you want the results one by one as they complete though, then just concat gets the job done as shown by others

ForkJoin can meet your requirment
‘forkJoin’ waits for each HTTP request to complete and group’s all the observables returned by each HTTP call into a single observable array and finally return that observable array.
public requestDataFromMultipleSources(): Observable<any[]> {
let response1 = this.http.get(requestUrl1);
let response2 = this.http.get(requestUrl2);
let response3 = this.http.get(requestUrl3);
return forkJoin([response1, response2, response3]);
}
subscribe to single observable array and save the responses separately.
this.dataService.requestDataFromMultipleSources().subscribe(responseList => {
this.responseData1 = responseList[0];
this.responseData2 = responseList[1];
this.responseData3 = responseList[2];
});

MDN Promise.all see this you can do it even without observable
Promise.all(array.map(item => makeRequest(item))).then(values =>{
// your logic with recieved data
})

Related

.subscribe is not being executed in the correct order

Using Angular and the RxJS library.
I have a simple method:
method(){
const data = this.data;
const moreData = this.moreData;
this.anotherMethodOne(data)
this.anotherMethodTwo(moreData)
}
As you can see I am calling two other methods. anotherMethodOne() should be executed before anotherMethodTwo() - and it is.
Within anotherMethodOne() I have a .subscribe:
anotherMethodOne(data) {
<!-- at this point it jumps out of this method and continues to proceed in the parent method, but returns later -->
this.service.getService(id).subscribe(res => {
res
}
}
As soon as my debugger hits the .subscribe call it jumps back into the parent method() and proceeds to execute this.anotherMethodTwo(moreData). It then jumps back into this.anotherMethodOne(data) to finally complete the .subscribe - however, by this time its too late.
I need to make sure this.anotherMethodOne(data) is completed fully before this.anotherMethodTwo(moreData) runs or ensure that the .subscribe runs before it proceeds any further.
I started looking at switchMap and concat in the RxJS library, but unsure how that would be implemented in my case.
You cannot guarantee an observable would've emitted before proceeding with it's emitted value. So you need to return the observable from the methods and subscribe where it's value is required.
For observable depending on the emissions of other observables, you could use higher order mapping operator like switchMap. More info about them here.
method(){
const data = this.data;
const moreData = this.moreData;
this.anotherMethodOne(data).pipe(
switchMap((someValue) =>
this.anotherMethodTwo(moreData)
)
).subscribe({
// callbacks
});
}
anotherMethodOne(data): Observable<any> {
return this.http.get('url');
}
anotherMethodTwo(moreData): Observable<any> {
return this.http.get('url');
}
When it comes to asynchronous functions, you should stay with the reactive approach and not try to determine when a certain code has finished, but instead just wait for it to finish and then invoke the callback or the responsible handler subsequently. Think about this. What is the subscribe never returns anything - is that okay? What if it takes a really long time to return anything - is that okay too?
But to fix the problem at hand, simply move anotherMethodTwo into the subscription. Then when something is returned, anotherMethodTwo is run.
method(){
const data = this.data;
const moreData = this.moreData;
this.anotherMethodOne(data)
}
anotherMethodOne(data) {
this.service.getService(id).subscribe(res => {
console.log(res)
this.anotherMethodTwo(moreData)
}
}
Going down this path would be similar to async callback hell which nesting style is not recommended but it will do for now.

Is using setState in every iteration of a loop bad practice?

Here is a little code snippet:
async componentDidMount() {
...
this.state.postList.forEach(element => {
this.fetchItem(element);
});
}
async fetchItem(query) {
...
this.setState( previousState => {
const list = [...previousState.data, data];
return { data: list };
});
}
I'm curious to know if using setState in every iteration of a forEach loop is a bad idea or not. I'm suspecting that it impacts performance, but I'd like to know for sure because this seemed like the simplest solution to this problem.
Here's an alternative approach: Update your fetchItem to just return the item. In your componentDidMount use Promise.all to get all the items and then commit them to the state in a single operation.
async componentDidMount() {
const items = await Promise.all(this.state.postList.map(element => fetchItem(element)));
this.setState({data: items});
}
async fetchItem(query) {
const item = await getItem(query) // however you accomplish this
return item;
}
I'm curious to know if using setState in every iteration of a forEach loop is a bad idea or not.
If it would be directly inside of an iteration then definetly yes, as React than has to merge all the updates you make during the iteration, which will probably take much more time than if you would just set the state after the loop.
In your case however, you do start an asynchronous action in each iteration, and as all asynchronous tasks finish at different times, the updates aren't run all at once. The main benefit of your approach is that if these asynchronous tasks take some time (e.g. if you fetch a lot of data for each), then some information can already be shown, while some are still loading. If all those asynchronous calls only load a small amount of data, well then you should actually change your API to deliver all the data at once. So it really depends on your usecase wether thats good or bad.

Wait till all Observables are completed

I have few Observables like this one in my code.
this.server.doRequest().subscribe(response => console.log(response)
error => console.log(error),
() => {
console.log('completed');
});
There could be any number of these Observables,
so I need to write a function that checks if each Observable is done otherwise waits till each is finished.
I'm assuming I can create an array push every new Observable there and when it's completed remove it by index. But is it good solution?
Where I want to use it. For example I have a page where user upload photos any amount asynchronously and then he press Finish button. Once he pressed Finish button I need to wait till ALL dynamically created Observables are completed.
you should use higher order observables for this, your exact use case will dictate the exact operator, but forkJoin seems a good candidate:
forkJoin(
this.server.doRequest1(),
this.server.doRequest2(),
this.server.doRequest3(),
this.server.doRequest4()
).subscribe(vals => console.log('all values', vals));
forkJoin won't emit till all innter observables have completed. making it the operator of choice for waiting for multiple observables to complete. You can also feed it an array of observables. There are multiple other operators that may fulfill your case too, such as concat, merge, combineLatest or a few others.
edit based on more details:
in the use case described in your update, you'll still want to use a higher order observable, but forkjoin is not what you want. you'll want to use a local subject to accomplish the goal as wanting to kick off each observable as it is selected and waiting for them all to be done complicates things a little (but not too much):
suppose you had a template like:
<button (click)="addPhoto()">Add Photo</button>
<button (click)="finish()">Finish</button>
where the add photo button gets the users photo and all that, and finish is your completion, you could have a component like this:
private addPhoto$ = new Subject();
constructor() {
this.addPhoto$.pipe(
mergeMap(() => this.uploadPhoto()),
).subscribe(
(resp) => console.log('resp', resp),
(err) => console.log('err', err),
() => console.log('complete')
);
}
private uploadPhoto() {
// stub to simulate upload
return timer(3000);
}
addPhoto() {
this.addPhoto$.next();
}
finish() {
this.addPhoto$.complete();
}
if you run this code, you'll see that the photo adds will emit in the subscribe handler as they complete, but complete will only fire once all the photo uploads have completed and the user has clicked finish.
here is a stackblitz demonstrating the functionality:
https://stackblitz.com/edit/angular-bsn6pz
I'd create a dictionary (in javascript that would be a JSON with observable names as boolean properties) where you push each observable on "create" and a method which should execute on completion of each observable, which will iterate through that dictionary and if all completed do something.
That will ensure parallelism and final execution after all completed.
var requests = {
doRequest1: false,
doRequest2: false,
doRequest3: false
};
var checkIfCAllCompleted = name => {
requests[name] = true;
for (var property in requests) {
if (object.hasOwnProperty(property)) {
if (!property) {
return;
}
}
}
// all properties are true - do something here
console.log("here");
}
this.server.doRequest1().then(() => checkIfCAllCompleted("doRequest1"));
this.server.doRequest2().then(() => checkIfCAllCompleted("doRequest2"));
this.server.doRequest3().then(() => checkIfCAllCompleted("doRequest3"));

How to prevent multiple http requests from firing all at once

I have an array of objects. For each object I need to trigger an asynchronous request (http call). But I only want to have a certain maximum of requests running at the same time. Also, it would be nice (but not neccessary) if I could have one single synchronization point after all requests finished to execute some code.
I've tried suggestions from:
Limit number of requests at a time with RxJS
How to limit the concurrency of flatMap?
Fire async request in parallel but get result in order using rxjs
and many more... I even tried making my own operators.
Either the answers on those pages are too old to work with my code or I can't figure out how to put everything together so all types fit nicely.
This is what I have so far:
for (const obj of objects) {
this.myService.updateObject(obj).subscribe(value => {
this.anotherService.set(obj);
});
}
EDIT 1:
Ok, I think we're getting there! With the answers of Julius and pschild (both seem to work equally) I managed to limit the number of requests. But now it will only fire the first batch of 4 and never fire the rest. So now I have:
const concurrentRequests = 4;
from(objects)
.pipe(
mergeMap(obj => this.myService.updateObject(obj), concurrentRequests),
tap(result => this.anotherService.set(result))
).subscribe();
Am I doing something wrong with the subscribe()?
Btw: The mergeMap with resultSelector parameter is deprecated, so I used mergeMap without it.
Also, the obj of the mergeMap is not visible in the tap, so I had to use tap's parameter
EDIT 2:
Make sure your observers complete! (It cost me a whole day)
You can use the third parameter of mergeMap to limit the number of concurrent inner subscriptions. Use finalize to execute something after all requests finished:
const concurrentRequests = 5;
from(objects)
.pipe(
mergeMap(obj => this.myService.updateObject(obj), concurrentRequests),
tap(res => this.anotherService.set(res))),
finalize(() => console.log('Sequence complete'))
);
See the example on Stackblitz.
from(objects).pipe(
bufferCount(10),
concatMap(objs => forkJoin(objs.map(obj =>
this.myService.updateObject(obj).pipe(
tap(value => this.anotherService.set(obj))
)))),
finalize(() => console.log('all requests are done'))
)
Code is not tested, but you get the idea. Let me know if any error or explanation is needed
I had the same issue once. When I tried to load multiple images from server. I had to send http requests one after another. I achieved desired outcome using awaited promise. Here is the sample code:
async ngOnInit() {
for (const number of this.numbers) {
await new Promise(resolve => {
this.http.get(`https://jsonplaceholder.typicode.com/todos/${number}`).subscribe(
data => {
this.responses.push(data);
console.log(data);
resolve();
}
);
});
}
}
Main idea is here to resolve the promise once you get the response.
With this technique you can come up with custom logic to execute one method once all the requests finished.
Here is the stackblitz. Open up the console to see it in action. :)

Chaining observables with flatMap

I'm working with observables and the flatMap operator, I wrote a method which makes and API call and returns an observable with an array of objects.
Basically what I need is to get that array of objects and process each object, after all items are processed. I want to chain the result to make an extra API call with another method that I wrote.
The following code does what I need:
this.apiService.getInformation('api-query', null).first().flatMap((apiData) => {
return apiData;
}).subscribe((dataObject) => {
this.processService.processFirstCall(dataObject);
}, null, () => {
this.apiService.getInformation('another-query', null).first().subscribe((anotherQueryData) => {
this.processService.processSecondCall(anotherQueryData);
});
});
But this approach isn't optimal from my perspective, I would like to do chain those calls using flatMap but if I do the following:
this.apiService.getInformation('api-query', null).first().flatMap((apiData) => {
return apiData;
}).flatMap((dataObject) => {
this.processService.processFirstCall(dataObject);
return [dataObject];
}).flatMap((value) => {
return this.apiService.getInformation('another-api-query', null).first();
}).subscribe((value) => {
this.processService.processSecondCall(value);
});
The second API call executes once for each item on the apiData array of objects. I know I'm missing or misunderstanding something. But from the second answer of this thread Why do we need to use flatMap?, I think that the second flatMap should return the processed apiData, instead is returning each of the object items on that Array. I would appreciate the help.
Thank you.
What you want is the .do() operator, and not flatMap(). flatMap() is going to transform an event to another event, and in essence, chaining them. .do() just executes whatever you instruct it to do, to every emission in the events.
From your code, there are 2 asynchronous methods (calls to the api) , and 2 synchronous (processService). What you want to do is :
Call to the first API (async), wait for the results
Process the results (sync)
Call to the second API (async), wait for the results to come back
Process the results (sync)
Hence your code should be :
this.apiService.getInformation('api-query', null)//step1
.first()
.do((dataObject) => this.processFirstCall(dataObject))//step2
.flatMap(() => this.apiService.getInformation('another-api-query', null))//step3
.first()
.do(value => this.processService.processSecondCall(value))//step4
.subscribe((value) => {
console.log(value);
});
I wrote in comment the steps corresponding to the list above. I also cleaned up unnecessary operators (like your first flatMap is kinda redundant).
Also, in the event you want to transform your data, then you should use .map() instead of .do(). And I think that is the place where you are confused with .map() and .flatMap().
The issue I think your encountering is that flatmap should be applied to an observable or promise. in your second code example, you are returning data within the flatmap operator which is then passed to the following flatmap functions, whereas these should be returning observables.
For example:
this.apiService.getInformation('api-query', null).first()
.flatMap((dataObject) => {
return this.processService.processFirstCall(dataObject);
}).flatMap((value) => {
return this.apiService.getInformation('another-api-query', null)
}).subscribe((value) => {
this.processService.processSecondCall(value);
});
See this post for futher clarification.

Categories

Resources