I am a novice with Rxjs, I'm trying to do a request to my API with limits passed in parameters.
My problem is sometimes the returned result is empty for some reasons. The thing I need to do is retry this API call with updated parameters (skip param)
pollService.getPoll$(skip, limit).subscribe((pollList) => {
doStuff;
},
(error) => {
doStuff;
});
I read some topics about the RetryWhen RXJS function but it is about errors when the request fail and you want to retry the same one but I ve no errors and I don't want to retry the same request, I also saw topics about Replay function but it is not very clear to me.
Can someone explain me what to do here please !!
Thanks
Alex
Consider utilizing the expand operator as demonstrated below:
import { EMPTY, of } from "rxjs"
import { expand } from "rxjs/operators"
public startPolling(skip: number, limit: number): void {
of([])
.pipe(
expand(x => x.length < 2 ? pollService.getPoll$(skip--, limit) : EMPTY)
)
.subscribe(pollList => {})
}
Update:
public poll = (skip: number, limit: number): void => {
defer(() => getPoll$(1 + skip--, limit))
.pipe(
repeat(),
first(x => {
if(x.length < 2){
// update some variable in the component
return false;
}
return true;
})
)
.subscribe(pollList => { })
}
If I understand correctly, your backend is paging data using the skip and limit parameters.
So, if you have a skip value that is too high, you want to reduce it automatically.
There are many, many ways to solve this problem in RxJS:
you could insert a switchMap after getPoll$. SwitchMap would return a new observable, either wrapping the result if it's ok (with of(pollList)), or returning pollService.getPoll$(newSkipValue, newLimitValue)
you could map the result and throw an Error if the result doesn't pass validation. Then you could catchError and return the result of a new call to getPoll$
However, what I suggest is modelling the call differently. I would use a Subject as a source of requests, and switchMap to execute the requests.
// inside the component or service
interface GetPollRequest {
skip: number;
limit: number;
}
private _callSource = new Subject<GetPollRequest>();
public triggerCall(skip: number, limit: number) {
this._callSource.next({skip, limit});
}
constructor(...) {
this._callSource.pipe(
// every time _callSource emits, we call the server
switchMap(({skip, limit) => pollService.getPoll$(skip, limit).pipe(
map(pollList => ({ pollList, skip, limit }))
),
tap(({pollList, skip, limit}) => {
// update the request in any way you need. You need to make sure
// that the next automatic trigger doesn't repeat indefinitely,
// or you'll simulate a DOS attack to your backend
if (pollList.length < 2) this.triggerCall(skip - 2, limit);
})
).subscribe(pollList => // update the component status);
}
Using this pattern, you use subjects as triggers (or custom events, they are pretty much the same), and you wrap them up during constructor time.
SwitchMap is used to create an observable (in this case, performing a request) every time the source emits.
Tap is used to perform an operation (pretty much like a subscribe), embedded in the chain of transformations inside the pipe.
Related
I have an observable that fetches an array of items (32 each time) from an API and emits a new response until there are no items left to fetch.
I want to process said list of items one by one as soon as i get the first batch until im done with ALL items fetched.
When i'm done with the complete list, i want to repeat the process indefinitely.
Here's what i have so far:
_dataService
.getItemsObservable()
.pipe(
switchMap((items) => {
const itemList = items.map((i) => i.itemId);
return of(itemList);
}),
concatMap((item) =>
from(item).pipe(
concatMap((item) => {
// do something here
}
)
)
),
repeat()
).subscribe()
Any idea on what can i do? Right now what happens is it will loop over the first batch of items and ignore the rest
Replay wont call the service again, it will reuse the original values. Try switchMap from a behaviour subject and make it emit after you have processed the values. Really not sure why you would turn each item into an observable to concatMap. Just process the items after they are emitted.
const { of, BehaviorSubject, switchMap, delay } = rxjs;
const _dataService = {
getItemsObservable: () => of(Array.from({ length: 32 }, () => Math.random()))
};
const bs$ = new BehaviorSubject();
bs$.pipe(
delay(1000), // Behavior subject are synchronous will cause a stack overflow
switchMap(() => _dataService.getItemsObservable())
).subscribe(values => {
values.forEach(val => {
console.log('Doing stuff with ' + val);
});
bs$.next();
});
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/7.8.0/rxjs.umd.min.js" integrity="sha512-v0/YVjBcbjLN6scjmmJN+h86koeB7JhY4/2YeyA5l+rTdtKLv0VbDBNJ32rxJpsaW1QGMd1Z16lsLOSGI38Rbg==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
I have an observable that fetches an array of items (32 each time) from an API and emits a new response until there are no items left to fetch.
Okay, I assume that is _dataService.getItemsObservable()?
I want to process said list of items
What does this mean? Process how? Lets assume you have some function called processItemById that processes an itemId and returns the processed item.
one by one as soon as i get the first batch until im done with ALL items fetched.
Sounds like you're turning an Observable<T[]> into an Observable<T>. You can use mergeMap (don't care about order) or concatMap (maintain order) to do this. Since you're just flattening an inner array, they'll be the same in this case.
_dataService.getItemsObservable().pipe(
mergeMap(v => v),
map(item => processItemById(item.itemId)),
// Repeat here doesn't call `getItemsObservable()` again,
// instead it re-subscribes to the observable that was returned.
// Hopefully that's what you're counting on. It's not clear to me
repeat()
).subscribe(processedItemOutput => {
// Do something with the output?
});
Any idea on what can i do?
From your explanation and code, it's not clear what you're trying to do. Maybe this helps.
Right now what happens is it will loop over the first batch of items and ignore the rest
This could happen for a number of reasons.
Tip 1
Using higher-order mapping operators with RxJS::of is a code smell. Just use a regular map instead.
for example:
concatMap(v => of(fn(v)))
// or
switchMap(v => of(fn(v)))
are the same as:
map(v => fn(v))
Tip 2
I have no idea if this would help you but you can generate a new observable on each subscribe by using the delay operator.
For example:
defer(() => _dataService.getItemsObservable()).pipe(
mergeMap(v => v),
map(item => processItemById(item.itemId)),
repeat()
).subscribe(processedItemOutput => {
// Do something with the output?
});
It looks like you want to get all records from an API that is paginated but won't tell you how many pages are there which sounds like you're looking for expand() operator which is great for recursive calls.
import { of, EMPTY, expand, range, toArray, mergeMap, concat, map, takeLast } from 'rxjs';
const MAX_PAGES = 3;
const PER_PAGE = 32;
const fetchList = (offset: number) => {
return Math.ceil(offset / PER_PAGE) >= MAX_PAGES ? of([]) : range(offset, 32).pipe(toArray());
};
const fetchDetail = (id: number) => {
return of(`response for ${id}`);
};
of([]) // Seed for `expand()`.
.pipe(
expand((acc, index) => fetchList(acc.length).pipe(
mergeMap(list => { // Process each response array
console.log(`Response #${index}`, list);
// When the reponse is an empty we can stop recursive calls.
if (list.length === 0) {
return EMPTY;
}
// Process the response and make `fetchDetail()` call for each item.
// `concat()` will guarantee order and process items one by one.
return concat(...list.map(id => fetchDetail(id)))
.pipe(
toArray(),
map(details => [...acc, ...details]), // Append the processed response to a single large array.
);
}),
)),
takeLast(1), // Only take the final array after all pages have been fetched.
)
.subscribe(console.log);
Working demo: https://stackblitz.com/edit/rxjs-yqd4kx?devtoolsheight=60&file=index.ts
First time with RxJS. Basically, I'm trying to make a twitter scraper which retrieves tweets from a query string. The search url allows specifying of a min_position parameter which can be the last id of the previous search to sort of paginate.
The process kind of looks like this (where it loops back at the end):
get page -> next() each scraped tweet -> set min_position -> get page (until !has_more_items)
Requesting the page returns a promise and so I somehow have to wait until this is completed until I can proceed. I was hoping to pass an async function to Observable.create() but that doesn't seem to work, it's only called a single time.
EDIT
I've had a play around after reading your resources as best as I could. I came up with the following abstraction of my problem.
import { from, Observable } from 'rxjs'
import { concatMap, map, switchMap } from 'rxjs/operators'
let pageNumber = 0
const PAGE_SIZE = 3, MAX_PAGES = 3
async function nextPage() {
if (pageNumber >= MAX_PAGES) {
throw new Error('No more pages available')
}
await new Promise(res => setTimeout(res, 500)) // delay 500ms
const output = []
const base = pageNumber++ * PAGE_SIZE
for (let i = 0; i < PAGE_SIZE; i++) {
output.push(base + i)
}
return output
}
function parseTweet(tweet: number): string {
// simply prepend 'tweet' to the tweet
return 'tweet ' + tweet
}
const getTweets = (): Observable<string> => {
return from(nextPage()) // gets _html_ of next page
.pipe(
concatMap(page => page), // spreads out tweet strings in page
map(tweet => parseTweet(tweet)), // parses each tweet's html
switchMap(() => getTweets()) // concat to next page's tweets
// stop/finish observable when getTweets() observable returns an error
)
}
getTweets()
.subscribe(val => console.log(val))
It's quite close to working but now whenever nextPage() returns a rejected promise, the entire observable breaks (nothing logged to the console).
I've attempted inserting a catchError after the pipe to finish the observable instead of running through and throwing an error but I can't get it to work.
Also this implementation is recursive which I was hoping to avoid because it's not scalable. I don't know how many tweets/pages will be processed in the observable in future. It also seems that tweets from all 3 pages must be processed before the observable starts emitting values which of course is not how it should work.
Thanks for your help! :)
We need to loadTwits until some condition and somehow work with Promise? Take a look at example:
function loadTwits(id) {
// Observable that replay last value and have default one
twitId$ = new BehaviorSubject(id);
return twitId$.pipe(
// concatMap - outside stream emit in order inner do
// from - convert Promise to Observable
concatMap(id => from(fetchTwits(id))),
map(parseTwits),
// load more twits or comlete
tap(twits => getLastTwitId(twits) ? twitId$.next(getLastTwitId(twits)) : twitId$.complete())
)
}
I figured it out after looking further into expand and realising it was recursion that I needed in my observable. This is the code that creates the observable:
const nextPage$f = () => from(nextPage()) // gets _html_ of next page
.pipe(
concatMap(page => page), // spreads out tweet strings in page
map(tweet => parseTweet(tweet)) // parses each tweet's html
)
const tweets$ = nextPage$f()
.pipe(
expand(() => morePages() ? nextPage$f() : empty())
)
I have a function that does an http request based on a parameter. And I want to add some kind of "debounce" functionality. So if the function gets called multiple times in a set time window, I want to combine the parameters into one request instead of making multiple requests.
I want to achieve this with Observables and in Angular. This does not sound that complicated, however I'm not able to get it running, maybe I'm missing something.
For now let's just skip the combining in a single request as this can be done with an aggregated debounce or a Oberservable.buffer. I have trouble combining the single Observables.
Here's what I've tried so far.
I tried using a Subject, as this seemed to be the proper object for this case (https://stackblitz.com/edit/angular-hcn41v?file=src%2Fapp%2Fapp.component.ts).
constructor(private http: HttpClient) {
this.makeRequest('1').subscribe(x => console.log(x))
this.makeRequest('2').subscribe(console.log)
setTimeout(() => {
this.makeRequest('3').subscribe(console.log)
}, 1000)
}
private makeRequest(id: string) {
this.observable = this.observable.pipe(
merge(Observable.of(id).pipe(delay(1)))
)
return this.aggregateDebounce(this.observable)
}
private getUrl(value) {
console.log('getUrl Call', value);
return 'https://jsonplaceholder.typicode.com/posts/1';
}
private aggregateDebounce(ob$) {
const shared$ = ob$.publishReplay(1).refCount()
return shared$.buffer(shared$.debounceTime(75))
}
I expect to have one 'getUrl Call' log for each function call and one result log. However I only get results if I add more than 1 calls to this.makeRequest() and the result is also weird. All previous values are always returned as well. I think I don't fully understand how Subject works in this case.
Another approach (taken from here RXJS: Aggregated debounce) was to create some sort of aggregate debounce (https://stackblitz.com/edit/angular-mx232d?file=src/app/app.component.ts)
constructor(private http: HttpClient) {
this.makeRequest('1').subscribe(x => console.log(x))
this.makeRequest('2').subscribe(console.log)
setTimeout(() => {
this.makeRequest('3').subscribe(console.log)
}, 1000)
}
private makeRequest(id: string) {
this.observable = this.observable.pipe(
merge(Observable.of(id).pipe(delay(1)))
)
return this.aggregateDebounce(this.observable)
}
private getUrl(value) {
console.log('getUrl Call', value);
return 'https://jsonplaceholder.typicode.com/posts/1';
}
private aggregateDebounce(ob$) {
const shared$ = ob$.publishReplay(1).refCount()
return shared$.buffer(shared$.debounceTime(75))
}
In this scenario I have the problem I'm also getting all previous values as well.
In theory (at least to me) both variants sounded plausible, however it seems like I'm missing something. Any wink in the right direction is highly appreciated.
Edit:
As requested I added the final real-world goal.
Imagine a service that requests information from an API. Within 50-75ms you call the service with a certain id. I want to group those ids together to a single request instead of doing 3. And if 100ms later another call to the service is made, a new request will be done
this.makeRequest(1).subscribe();
private makeRequest(number: number) {
this.values.next(number);
return this.idObservable.pipe(
You emit the value before you subscribe -> The value gets lost.
private values: Subject = new Subject();
private idObservable = this.values.pipe(
private makeRequest(number: number) {
this.values.next(number);
return this.idObservable.pipe(
Every call creates a new observable based on the subject. Whenever you emit a value, all subscribers receive the value.
A possible solution could look something like this (I'm using the new rxjs syntax here):
subject: Subject<String> = null;
observable = null;
window = 100;
constructor() {
this.subject = null;
this.window = 100;
this.makeRequest('1').subscribe(console.log)
this.makeRequest('2').subscribe(console.log)
setTimeout(() => {
this.makeRequest('3').subscribe(console.log)
}, 1000)
}
private makeRequest(id: string) {
if (!this.subject) {
this.subject = new ReplaySubject()
this.observable = this.subject.pipe(
takeUntil(timer(this.window).pipe(take(1))),
reduce((url, id, index) => this.combine(url, id), baseUrl),
flatMap(url => this.request(url)),
tap(() => this.subject = null),
share()
)
}
this.subject.next(id);
return this.observable;
}
Where combine creates the url and request makes the actual request.
Rxjs is quite good at handling this kind of case. You'll need two different Subjects:
One will be used to collect and combine all requests
The second will be used for subscribing to results
When a request is made, the value will be pushed onto the first subject but the second will be returned, abstracting away the internal logic of combining requests.
private values: Subject = new Subject();
private results: Subject = new Subject();
private makeRequest(number: number) {
this.values.next(number);
return this.results;
}
The pipeline for merging the requests could be a buffer and debounceTime as indicated in the question or other logic, as required. When a response is recieved, it just needs to be pushed onto the results Subject:
constructor(private http: HttpClient) {
this.values
.pipe(
buffer(this.values.pipe(debounceTime(1000))),
switchMap(values => this.getUrl(values)),
map(response => this.results.next(response)))
.subscribe();
}
I've used a switchMap to simulate an asynchronous request before pushing the response onto the results.
Full example here: https://angular-8yyvku.stackblitz.io
I'm having some difficulty getting RxJS' retry operator to work in combination with the map operator.
Essentially what I'm trying to do is take a stream of values (in this case GitHub users for testing purposes), and for each value do some operation on that value.
However I want to be able to handle errors in a way that allows me to set a finite number of retries for that operation. I'm also assuming RxJS handles such a thing asynchronously.
Rx.Observable.fromPromise(jQuery.getJSON("https://api.github.com/users"))
.concatAll(x => Rx.Observable.of(x))
.map(x => {
if(x.id % 5 == 0) {
console.log("error");
return Rx.Observable.throw("error");
}
return x;
})
.retry(3)
.subscribe(x => console.log(x), e => console.log(e));
This is what I've got so far, however where it should retry the same value 3 times it only prints "error" once which isn't the behaviour I'm looking for.
Am I missing something trivial or if this sort of thing simply not possible with RxJS? In short I would like to execute operations on values asynchronously, retying said operation a finite number of times but allowing other valid operations to continue in parallel. I have achieved something similar using only promises but the code is rather messy and I was hoping RxJS would allow me to make things a bit nicer to read.
Edit - removing earlier failed solutions for clarity (some comments may no longer be relevent)
To retry individual failures, you need to break to individual requests and retry those.
To allow proper handling of errors, split the streams into sub-streams for each expected outcome.
I've also added a bit of code to allow one of the retries to suceed.
const Observable = Rx.Observable
// Define tests here, allow requestAgain() to pass one
const testUser = (user) => user.id % 5 !== 0
const testUserAllowOneSuccess = (user) => user.id === 5 || user.id % 5 !== 0
const requestAgain = (login) => Observable.of(login)
.switchMap(login => jQuery.getJSON("https://api.github.com/users/" + login ))
.map(x => {
if(!testUserAllowOneSuccess(x)) {
console.log("logging error retry attempt", x.id);
throw new Error('Invalid user: ' + x.id)
}
return x;
})
.retry(3)
.catch(error => Observable.of(error))
const userStream = Observable.fromPromise(jQuery.getJSON("https://api.github.com/users"))
.concatAll()
const passed = userStream.filter(x => testUser(x))
const failed = userStream.filter(x => !testUser(x))
.flatMap(x => requestAgain(x.login))
const retryPassed = failed.filter(x => !(x instanceof Error))
const retryFailed = failed.filter(x => (x instanceof Error))
.toArray()
.map(errors => { throw errors })
const output = passed.concat(retryPassed, retryFailed)
output.subscribe(
x=> console.log('subscribe next', x.id ? x.id : x),
e => console.log('subscribe error', e)
);
Working example CodePen
I see a couple issues that are probably causing you problems:
1) In your map statement, you're doing the throw by returning an observable. This would be appropriate if the success case was also returning an observable which you were then flattening (with concat, switch, or merge), but instead your success case just emits a value and never flattens. So when that Throw observable is returned, it never gets flattened, and thus just gets passed through and the observable object is logged.
Instead, you can just do a plain javascript throw 'error' (though consider throwing an error object instead of a string)
2) When retry gets an error, it will resubscribe. If you're using observables to do your fetching, this will mean that it will make the fetch again. However, you did the fetching with a promise, and then only created an observable from that promise. So when you resubscribe to the observable, it will end up using the very same promise object that it did the first time, and that promise is already in a resolved state. No additional fetch will be made.
To fix this, you'll need to have an observable version of doing the fetch. If you need to make this from scratch, it could look like this:
Rx.Observable.create(observer => {
jQuery.getJSON("https://api.github.com/users")
.then(result => {
observer.next(result);
observer.complete();
}, err => {
observer.error(err);
observer.complete();
});
})
Though if you intend to do this often, i recommend creating a function that does that wrapping for you.
EDIT:
My end goal with this is to essentially be able to take an array of api urls, and initiate get requests with each URL. If a get request fails for whatever reason, it should try a total 3 times to reach that endpoint again. However, the failing of such a get request should not inhibit the other requests from taking place, they should be happening in parallel. Furthermore if a get request fails 3 times, it should just be marked as failed and it should not stop the process as a whole. So far I've not found a way of doing so using RxJS.
Then i would do something like this:
function getJson (url) {
return Rx.Observable.create(observer) => {
jQuery.getJSON(url)
.then(result => {
observer.next(result);
observer.complete();
}, err => {
observer.error(err);
observer.complete();
});
}
}
const endpoints = ['someUrl', 'someOtherUrl'];
const observables = endpoints.map(endpoint =>
return getJson(url)
.retry(3)
.catch(err => Rx.Observable.of('error'));
});
Rx.Observable.forkJoin(...observables)
.subscribe(resultArray => {
// do whatever you need to with the results. If any of them
// errored, they will be represented by just the string 'error'
});
Your map function returns Rx.Observable.throw("error");. It is wrong, because the following retry() will then receive it wrapped (i.e. as Observable<Observable<T>> instead of Observable<T>. You can use flatMap to correct that, but then you also need to wrap the value via Rx.Observable.of(x).
.flatMap(x => {
if(x % 5 == 0) {
console.log("error");
return Rx.Observable.throw("error");
}
return Rx.Observable.of(x);
})
.retry(3)
.subscribe(x => console.log(x), e => console.log(e));
Another option is to throw new Error() instead of trying to return Rx.Observable.throw("error");
Make sure you retry on the right thing
Rx.Observable.fromPromise(jQuery.getJSON("https://api.github.com/users"))
.retry(3)
.concatAll(x => Rx.Observable.of(x))
...
Then convert promise with from is not repeatable
Example: https://stackblitz.com/edit/rxjs-dp3md8
But, with Observable.create works well
If somebody have solution to make promise repeatable, please post comment.
I know, it is a newbie question:
I created a mobile application reading an information feed from a bluetooth serial device.
Data are retrieved using a promise in this way:
myServiceClass.getRemoteValue(valueId).then(reply: number) {
...
}
I need to read multiple parameters coming from this feed and I have to wait the previous call to finish before requesting the new value.
If I run:
let requiredValues = [1, 2, 3, 4, ..., n];
for (let i=0; i<requiredValues.length; i++) {
myServiceClass.getRemoteValue(valueId).then(reply: number) {
...
}
}
In this way request will run in parallel, but I need them to run in sequence one after the other. Is there any solution to subsequentially chain an array of promises someway?
In other words I need to run the n-th promise only after the previous promise has been resolved.
Thank you very much for your time.
Well, you can use a recursive method to achieve that... Please take a look at this plunker (when running the plunker, please notice that the values are being printed in the console)
I'm just using some fake data, but I guess it's enough to give you the overall idea:
public start(): void {
this.getSeveralRemoteValues([1,2,3,4,5,6,7,8,9]);
}
private getSeveralRemoteValues(array): Promise<boolean> {
if(array && array.length) {
return this.getRemoteValueFromService(array[0]).then(() => {
array.shift(); // remove the first item of the array
this.getSeveralRemoteValues(array); // call the same method with the items left
})
} else {
this.logEnd();
}
}
private logEnd(): void {
alert('All promises are done!');
}
private getRemoteValueFromService(value: number): Promise<boolean> {
// this simulates the call to the service
return new Promise((resolve, reject) => {
setTimeout(() => {
console.log(`Promise: ${value}`);
resolve(true);
}, 1000);
});
}