I'm having some difficulty getting RxJS' retry operator to work in combination with the map operator.
Essentially what I'm trying to do is take a stream of values (in this case GitHub users for testing purposes), and for each value do some operation on that value.
However I want to be able to handle errors in a way that allows me to set a finite number of retries for that operation. I'm also assuming RxJS handles such a thing asynchronously.
Rx.Observable.fromPromise(jQuery.getJSON("https://api.github.com/users"))
.concatAll(x => Rx.Observable.of(x))
.map(x => {
if(x.id % 5 == 0) {
console.log("error");
return Rx.Observable.throw("error");
}
return x;
})
.retry(3)
.subscribe(x => console.log(x), e => console.log(e));
This is what I've got so far, however where it should retry the same value 3 times it only prints "error" once which isn't the behaviour I'm looking for.
Am I missing something trivial or if this sort of thing simply not possible with RxJS? In short I would like to execute operations on values asynchronously, retying said operation a finite number of times but allowing other valid operations to continue in parallel. I have achieved something similar using only promises but the code is rather messy and I was hoping RxJS would allow me to make things a bit nicer to read.
Edit - removing earlier failed solutions for clarity (some comments may no longer be relevent)
To retry individual failures, you need to break to individual requests and retry those.
To allow proper handling of errors, split the streams into sub-streams for each expected outcome.
I've also added a bit of code to allow one of the retries to suceed.
const Observable = Rx.Observable
// Define tests here, allow requestAgain() to pass one
const testUser = (user) => user.id % 5 !== 0
const testUserAllowOneSuccess = (user) => user.id === 5 || user.id % 5 !== 0
const requestAgain = (login) => Observable.of(login)
.switchMap(login => jQuery.getJSON("https://api.github.com/users/" + login ))
.map(x => {
if(!testUserAllowOneSuccess(x)) {
console.log("logging error retry attempt", x.id);
throw new Error('Invalid user: ' + x.id)
}
return x;
})
.retry(3)
.catch(error => Observable.of(error))
const userStream = Observable.fromPromise(jQuery.getJSON("https://api.github.com/users"))
.concatAll()
const passed = userStream.filter(x => testUser(x))
const failed = userStream.filter(x => !testUser(x))
.flatMap(x => requestAgain(x.login))
const retryPassed = failed.filter(x => !(x instanceof Error))
const retryFailed = failed.filter(x => (x instanceof Error))
.toArray()
.map(errors => { throw errors })
const output = passed.concat(retryPassed, retryFailed)
output.subscribe(
x=> console.log('subscribe next', x.id ? x.id : x),
e => console.log('subscribe error', e)
);
Working example CodePen
I see a couple issues that are probably causing you problems:
1) In your map statement, you're doing the throw by returning an observable. This would be appropriate if the success case was also returning an observable which you were then flattening (with concat, switch, or merge), but instead your success case just emits a value and never flattens. So when that Throw observable is returned, it never gets flattened, and thus just gets passed through and the observable object is logged.
Instead, you can just do a plain javascript throw 'error' (though consider throwing an error object instead of a string)
2) When retry gets an error, it will resubscribe. If you're using observables to do your fetching, this will mean that it will make the fetch again. However, you did the fetching with a promise, and then only created an observable from that promise. So when you resubscribe to the observable, it will end up using the very same promise object that it did the first time, and that promise is already in a resolved state. No additional fetch will be made.
To fix this, you'll need to have an observable version of doing the fetch. If you need to make this from scratch, it could look like this:
Rx.Observable.create(observer => {
jQuery.getJSON("https://api.github.com/users")
.then(result => {
observer.next(result);
observer.complete();
}, err => {
observer.error(err);
observer.complete();
});
})
Though if you intend to do this often, i recommend creating a function that does that wrapping for you.
EDIT:
My end goal with this is to essentially be able to take an array of api urls, and initiate get requests with each URL. If a get request fails for whatever reason, it should try a total 3 times to reach that endpoint again. However, the failing of such a get request should not inhibit the other requests from taking place, they should be happening in parallel. Furthermore if a get request fails 3 times, it should just be marked as failed and it should not stop the process as a whole. So far I've not found a way of doing so using RxJS.
Then i would do something like this:
function getJson (url) {
return Rx.Observable.create(observer) => {
jQuery.getJSON(url)
.then(result => {
observer.next(result);
observer.complete();
}, err => {
observer.error(err);
observer.complete();
});
}
}
const endpoints = ['someUrl', 'someOtherUrl'];
const observables = endpoints.map(endpoint =>
return getJson(url)
.retry(3)
.catch(err => Rx.Observable.of('error'));
});
Rx.Observable.forkJoin(...observables)
.subscribe(resultArray => {
// do whatever you need to with the results. If any of them
// errored, they will be represented by just the string 'error'
});
Your map function returns Rx.Observable.throw("error");. It is wrong, because the following retry() will then receive it wrapped (i.e. as Observable<Observable<T>> instead of Observable<T>. You can use flatMap to correct that, but then you also need to wrap the value via Rx.Observable.of(x).
.flatMap(x => {
if(x % 5 == 0) {
console.log("error");
return Rx.Observable.throw("error");
}
return Rx.Observable.of(x);
})
.retry(3)
.subscribe(x => console.log(x), e => console.log(e));
Another option is to throw new Error() instead of trying to return Rx.Observable.throw("error");
Make sure you retry on the right thing
Rx.Observable.fromPromise(jQuery.getJSON("https://api.github.com/users"))
.retry(3)
.concatAll(x => Rx.Observable.of(x))
...
Then convert promise with from is not repeatable
Example: https://stackblitz.com/edit/rxjs-dp3md8
But, with Observable.create works well
If somebody have solution to make promise repeatable, please post comment.
Related
I have an observable that fetches an array of items (32 each time) from an API and emits a new response until there are no items left to fetch.
I want to process said list of items one by one as soon as i get the first batch until im done with ALL items fetched.
When i'm done with the complete list, i want to repeat the process indefinitely.
Here's what i have so far:
_dataService
.getItemsObservable()
.pipe(
switchMap((items) => {
const itemList = items.map((i) => i.itemId);
return of(itemList);
}),
concatMap((item) =>
from(item).pipe(
concatMap((item) => {
// do something here
}
)
)
),
repeat()
).subscribe()
Any idea on what can i do? Right now what happens is it will loop over the first batch of items and ignore the rest
Replay wont call the service again, it will reuse the original values. Try switchMap from a behaviour subject and make it emit after you have processed the values. Really not sure why you would turn each item into an observable to concatMap. Just process the items after they are emitted.
const { of, BehaviorSubject, switchMap, delay } = rxjs;
const _dataService = {
getItemsObservable: () => of(Array.from({ length: 32 }, () => Math.random()))
};
const bs$ = new BehaviorSubject();
bs$.pipe(
delay(1000), // Behavior subject are synchronous will cause a stack overflow
switchMap(() => _dataService.getItemsObservable())
).subscribe(values => {
values.forEach(val => {
console.log('Doing stuff with ' + val);
});
bs$.next();
});
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/7.8.0/rxjs.umd.min.js" integrity="sha512-v0/YVjBcbjLN6scjmmJN+h86koeB7JhY4/2YeyA5l+rTdtKLv0VbDBNJ32rxJpsaW1QGMd1Z16lsLOSGI38Rbg==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
I have an observable that fetches an array of items (32 each time) from an API and emits a new response until there are no items left to fetch.
Okay, I assume that is _dataService.getItemsObservable()?
I want to process said list of items
What does this mean? Process how? Lets assume you have some function called processItemById that processes an itemId and returns the processed item.
one by one as soon as i get the first batch until im done with ALL items fetched.
Sounds like you're turning an Observable<T[]> into an Observable<T>. You can use mergeMap (don't care about order) or concatMap (maintain order) to do this. Since you're just flattening an inner array, they'll be the same in this case.
_dataService.getItemsObservable().pipe(
mergeMap(v => v),
map(item => processItemById(item.itemId)),
// Repeat here doesn't call `getItemsObservable()` again,
// instead it re-subscribes to the observable that was returned.
// Hopefully that's what you're counting on. It's not clear to me
repeat()
).subscribe(processedItemOutput => {
// Do something with the output?
});
Any idea on what can i do?
From your explanation and code, it's not clear what you're trying to do. Maybe this helps.
Right now what happens is it will loop over the first batch of items and ignore the rest
This could happen for a number of reasons.
Tip 1
Using higher-order mapping operators with RxJS::of is a code smell. Just use a regular map instead.
for example:
concatMap(v => of(fn(v)))
// or
switchMap(v => of(fn(v)))
are the same as:
map(v => fn(v))
Tip 2
I have no idea if this would help you but you can generate a new observable on each subscribe by using the delay operator.
For example:
defer(() => _dataService.getItemsObservable()).pipe(
mergeMap(v => v),
map(item => processItemById(item.itemId)),
repeat()
).subscribe(processedItemOutput => {
// Do something with the output?
});
It looks like you want to get all records from an API that is paginated but won't tell you how many pages are there which sounds like you're looking for expand() operator which is great for recursive calls.
import { of, EMPTY, expand, range, toArray, mergeMap, concat, map, takeLast } from 'rxjs';
const MAX_PAGES = 3;
const PER_PAGE = 32;
const fetchList = (offset: number) => {
return Math.ceil(offset / PER_PAGE) >= MAX_PAGES ? of([]) : range(offset, 32).pipe(toArray());
};
const fetchDetail = (id: number) => {
return of(`response for ${id}`);
};
of([]) // Seed for `expand()`.
.pipe(
expand((acc, index) => fetchList(acc.length).pipe(
mergeMap(list => { // Process each response array
console.log(`Response #${index}`, list);
// When the reponse is an empty we can stop recursive calls.
if (list.length === 0) {
return EMPTY;
}
// Process the response and make `fetchDetail()` call for each item.
// `concat()` will guarantee order and process items one by one.
return concat(...list.map(id => fetchDetail(id)))
.pipe(
toArray(),
map(details => [...acc, ...details]), // Append the processed response to a single large array.
);
}),
)),
takeLast(1), // Only take the final array after all pages have been fetched.
)
.subscribe(console.log);
Working demo: https://stackblitz.com/edit/rxjs-yqd4kx?devtoolsheight=60&file=index.ts
I am implementing abortable fetch calls.
There are basically two reasons for aborting the fetch on my page:
the user decides he/she does not want to wait for the AJAX data anymore and clicks a button; in this case the UI shows a message "call /whatever interrupted"
the user has moved to another part of the page and the data being fetched are no longer needed; in this case I don't want the UI to show anything, as it'd just confuse the user
In order to discriminate the two cases I was planning to use the reason parameter of the AbortController.abort method, but the .catch clause in my fetch call always receives a DOMException('The user aborted a request', ABORT_ERROR).
I have tried to provide a different DOMException as reason for the abort in case 2, but the difference is lost.
Has anyone found how to send information to the fetch .catch clause with regards to the reason to abort?
In the example below, I demonstrate how to determine the reason for an abortion of a fetch request. I provide inline comments for explanation. Feel free to comment if anything is unclear.
Re-run the code snippet to see a (potentially different) random result
'use strict';
function delay (ms, value) {
return new Promise(res => setTimeout(() => res(value), ms));
}
function getRandomInt (min = 0, max = 1) {
return Math.floor(Math.random() * (max - min + 1)) + min;
}
// Forward the AbortSignal to fetch:
// https://docs.github.com/en/rest/repos/repos#list-public-repositories
function fetchPublicGHRepos (signal) {
const headers = new Headers([['accept', 'application/vnd.github+json']]);
return fetch('https://api.github.com/repositories', {headers, signal});
}
function example () {
const ac = new AbortController();
const {signal} = ac;
const abortWithReason = (reason) => delay(getRandomInt(1, 5))
.then(() => {
console.log(`Aborting ${signal.aborted ? 'again ' : ''}(reason: ${reason})`);
ac.abort(reason);
});
// Unless GitHub invests HEAVILY into our internet infrastructure,
// one of these promises will resolve before the fetch request
abortWithReason('Reason A');
abortWithReason('Reason B');
fetchPublicGHRepos(signal)
.then(res => console.log(`Fetch succeeded with status: ${res.status}`))
.catch(ex => {
// This is how you can determine if the exception was due to abortion
if (signal.aborted) {
// This is set by the promise which resolved first
// and caused the fetch to abort
const {reason} = signal;
// Use it to guide your logic...
console.log(`Fetch aborted with reason: ${reason}`);
}
else console.log(`Fetch failed with exception: ${ex}`);
});
delay(10).then(() => console.log(`Signal reason: ${signal.reason}`));
}
example();
Here I have a function that takes an array of string that contains the user names of github accounts. And this function is going to return an array of user data after resolving. There should be one fetch request per user. and requests shouldn’t wait for each other. So that the data arrives as soon as possible. If there’s no such user, the function should return null in the resulting array.
An example for the input would be ["iliakan", "remy", "no.such.users"], and the expected returned promise after resolving would give us [null, Object, Object], Object being the data that contained info about a user.
Here is my attempt to solve this question.
function getUsers(names) {
return new Promise(resolve => {
const array = [];
const url = "https://api.github.com/users/";
const requests = names.map(name => {
const endpoint = `${url}${name}`;
return fetch(endpoint);
});
Promise.all(requests).then(reponses => {
reponses.forEach(response => {
if (response.status === 200) {
response.json().then(data => {
array.push(data);
});
} else {
array.push(null);
}
});
resolve(array);
});
});
}
It does work, i.e. returning an array [null, Object, Object]. And I thought it fulfilled the requirements I stated above. However, after looking at it closely, I felt like I couldn't fully make sense of it.
My question is, look at where we resolve this array, it resolved immediately after the forEach loop. One thing I don't understand is, why does it contain all three items when some of the items are pushed into it asynchronously after the json() is finished. what I mean is, in the case where response.status === 200, the array is pushed with the data resolved from json(), and I would assume this json() operation should take some time. Since we didn't resolve the array after json() operation is finished, how come we still ended up with all data resolved from json()?
Promise.all(requests).then(reponses => {
reponses.forEach(response => {
if (response.status === 200) {
response.json().then(data => {
array.push(data); <--- this should take some time
});
} else {
array.push(null);
}
});
resolve(array); <--- resolve the array immediately after the `forEach` loop
});
});
It looks to me like the array we get should only have one null in it since at the time it is revolved, the .json() should not be finished
You're right, the result is pushed later into the array.
Try to execute this:
const test = await getUsers(['Guerric-P']);
console.log(test.length);
You'll notice it displays 0. Before the result is pushed into the array, its length is 0. You probably think it works because you click on the array in the console, after the result has arrived.
You should do something like this:
function getUsers(names) {
const array = [];
const url = "https://api.github.com/users/";
const requests = names.map(name => {
const endpoint = `${url}${name}`;
return fetch(endpoint);
});
return Promise.all(requests).then(responses => Promise.all(responses.map(x => x.status === 200 ? x.json() : null)));
};
You should avoid using the Promise constructor directly. Here, we don't need to use it at all.
const url = "https://api.github.com/users/";
const getUsers = names =>
Promise.all(names.map(name =>
fetch(url + name).then(response =>
response.status === 200 ? response.json() : null)));
getUsers(["iliakan", "remy", "no.such.users"]).then(console.log);
The Promise constructor should only be used when you're creating new kinds of asynchronous tasks. In this case, you don't need to use the Promise constructor because fetch already returns a promise.
You also don't need to maintain an array and push to it because Promise.all resolves to an array. Finally, you don't need to map over the result of Promise.all. You can transform the promises returned by fetch.
The thing is that because json() operation is really quick, especially if response data is small in size it just has the time to execute. Second of all as objects in JavaScript passed by reference and not by value and Array is a object in JavaScript, independently of execution time it'll still push that data to the array even after it was resolved.
I want to implement something similar below(they are easy if using promises)
async doSomething(sID){
let student = await service.getStudent(sID);
let teacher = await service.getTeacher(student.TeacherID);
if(!teacher.Active){
return;
}
await service.teacherSomething(teacher);
await service.studentSomething(student);
}
I have no clue on how to do this if I am using observables instead of promises but this is the one I tried so far
doSomething(sID){
let student;
let teacher;
service.getStudent(sID).pipe(
switchMap(studentR=>{
student = studentR;
return service.getTeacher(student.TeacherID);
}),
switchMap(teacherR=>{
teacher = teacherR;
if(!teacher.Active){
return of(null);
}else{
return service.teacherSomething(teacher);
}
}),
swicthMap(teacherSomethingResponse=>{
if(teacherSomethingResponse==null){
return of(null);
}else{
return service.studentSomething(student);
}
})
}).subscribe();
}
as you can see, my rxjs version seems TOO LONG compared to the promise version and I feel like i am not doing it the right way.
Here is how you can convert your current code to Rx style. takeWhile will complete your observable if condition not met
function doSomething(sID) {
return from(service.getStudent(sID)).pipe(
switchMap(student =>
service.getTeacher(student.TeacherID).pipe(
takeWhile(teacher => teacher.Active),
switchMap(teacher => service.teacherSomething(teacher).pipe(takeWhile(res => res))),
switchMap(() => service.studentSomething(student))
)
))
}
async/await was developed as mainly a readability feature, so it is natural to be quite succinct visually.
Using old-style Promise syntax, you would get a much longer function.
So, in short - you are using observables fine, and it is longer due to expected syntax differences.
In your case, you can avoid saving the values of the teacher and student, since propagating them on the pipeline I think is pretty correct in your use case.
For that, after requesting the teacher, I would map the response and return both data student and teacher as a Tuple.
If the teacher is not active, throwing an error could be an elegant solution if you want to do something on error, if not you could also return EMPTY, that is an observable that does not emit and simply completes.
So, This is my solution, considering that the requests 'teacherSomething' and 'studentSomething' can be done in parallel since does not seem to depend on to each other
doSomething(sID){
service.getStudent(sID).pipe(
switchMap(studentR =>
service.getTeacher(student.TeacherID).pipe(map((teacherR) => [studentR, teacherR]))),
switchMap(([studentR, teacherR]) => {
if(!teacherR.Active){
throw new Error('Teacher is not active'); // or return EMPTY
}
// I think this two request may have been done in parallel, if so, this is correct.
return zip(service.teacherSomething(teacher), service.studentSomething(student));
})
).subscribe(
([teacherSomethingR, studentSomethingR]) => {/* do something with responses */},
(error) => { /* Do something if teacher not active, or some request has been error...*/ }
);
}
If the requests can't be done in parallel, I would do just the same as done before (switchMap) and return the tuple of responses in order to do something if needed. If no needed, you can avoid that last step:
doSomething(sID){
service.getStudent(sID).pipe(
switchMap(studentR =>
service.getTeacher(student.TeacherID).pipe(map((teacherR) => [studentR, teacherR]))),
switchMap(([studentR, teacherR]) => {
if(!teacherR.Active){
throw new Error('Teacher is not active'); // or return EMPTY
}
// Both request teacher something and student something done in 'serie'
return service.teacherSomething(teacher)
.pipe(switchMap((teacherSomethingR) =>
service.studentSomething(student)
.pipe(map((studentSomethingR) => [teacherSomethingR, studentSomethingR]))
))
})
).subscribe(
([teacherSomethingR, studentSomethingR]) => {/* do something with responses */},
(error) => { /* Do something if teacher not active, or some request has been error...*/ }
);
}
Hope this helps!
I have an array, the type of the values is irrelevant. What I would like to do is emitting one value every x seconds, call a function with that value, and if that function failed for some reason, retry it after y seconds (can be a simple constant, no need for any incremental thing here).
What I have so far
Rx.Observable
.interval(500)
.take(arr.length)
.map(idx => arr[idx])
.flatMap(dt => randomFunc(dt))
.catch(e => conosle.log(e))
.retry(5)
.subscribe();
function randomFunc(dt) {
return Rx.Observable.create(observer => {
if (dt === 'random') {
return observer.error(`error`);
} else {
return observer.next();
}
});
}
2 problems here though:
1: When randomFunc returns an error it seems that the whole chain starts over. I only need the failed one to retry.
2: catch never actually logs any error, even though it seems to retry on error.
For the first problem I've tried switchMap instead of flatMap like this:
Rx.Observable
.interval(500)
.take(arr.length)
.map(idx => arr[idx])
.switchMap(dt => randomFunc(dt)
.catch(e => conosle.log(e))
.retry(5)
)
.subscribe();
This way it seemed that it retried the failed ones only, but still didn't log any error and I'm not even sure switchMap is good here (I'm really an Rx noob).
Any help would be appreciated, thanks!
There're a couple of things to be aware of. The retry() operator just resubscribes to its source so if you don't want to start the entire iteration again you can merge/concat the async function into the chain.
Rx.Observable.from(arr)
.concatMap(val => {
let attempts = 0;
return Rx.Observable.of(val)
.delay(500)
.concatMap(val => randomFunc(val)
.catch((err, caught) => {
console.log('log error');
if (attempts++ === 1) {
return Rx.Observable.of(err);
} else {
return caught;
}
})
);
})
.subscribe(val => console.log(val));
function randomFunc(dt) {
return Rx.Observable.create(observer => {
if (dt === 'random') {
observer.error(`error received ${dt}`);
} else {
observer.next(dt);
observer.complete();
}
});
}
See live demo: https://jsbin.com/qacamab/7/edit?js,console
This prints to console:
1
2
3
4
log error
log error
error received random
6
7
8
9
10
The catch() operator is the most important part. Its selector function takes two arguments:
err - The error that occurred
caught - The original Observable.
If we return caught from the selector function we'll just resubscribe to the source Observable (which is the same as retry(1)). Since you want to log each error message we have to use catch() instead of just retry(). By returning Rx.Observable.of(err) we propagate the error further and in turn it'll be received by the subscriber as next notification. We could also return just Observable.empty() to simply ignore the error.
When randomFunc returns an error it seems that the whole chain starts
over. I only need the failed one to retry.`
In RxJs when combining Observables together the errors will also propagate and uncaught errors will cause unsubscription.
Your idea to use catch inside the switchMap is correct. Though switchMap will only flatten one Observable at a time, when the next value is mapped the previous Observable will be unsubscribed (it is switched out)
// Observable from array
Rx.Observable.from(arr)
.concatMap(value =>
// Put a 500 ms delay between each value
Rx.Observable.timer(500).map(_ => value)
)
.flatMap(dt =>
randomFunc(dt)
.retryWhen(errs =>
errs
.do(err => console.error(err))
// Retry at most 5 times
.take(5)
// Retry after 500ms
.delay(500)
)
)
.subscribe();
catch never actually logs any error, even though it seems to retry on
error.
The function passed to catch should return an Observable e.g:
Observable.throw(new Error())
.catch(e =>
(console.error(e), Observable.of('backup value'))
)
.subscribe();
http://reactivex.io/rxjs/class/es6/Observable.js~Observable.html#instance-method-catch