Develop Reference

Unable to set interval up to max 2 min or until condition met using angular

here what my scenario is i have 2 api's apiOne and apiTwo and when ever i call the apiOne is should give response and if the response is success then i have to send this repsonse to apiTwo as param then apiTwo will give another response in that i may get like "created" ,"in_progress" . here the issue is
How can i call the apitwo using interval for every 3 seconds until i get the response as "in_progress" and if i didnt get the response as like above then i need to poll the apiTwo till max 2 min and cancel the call. if i get the response as in_progress then i need to stop the interval or max 2 min cancel the interval or subcription.
I already wrote the code in nested way but it is not efficient .
below is my code
initiate() {
this.showProgress = true;
const data = {
id: this.id,
info: this.Values.info,
};
// First Api call
this.userServ.start(data).subscribe(res => {
this.Ids = res['Id'];
if (this.Ids) {
// Second Api call
this.Service.getStatus(this.Ids).subscribe(resp => {
if (resp) {
this.Status = res['Status'];
// if resp is In_Progress
if (this.Status === 'In_Progress') {
this.Start();
} else {
// if resp is not In_Progress then i get the response i am calling the api
this.intervalTimer = interval(3000).subscribe(x => {
this.Service.Status(this.Ids).subscribe(ress => {
this.Status = ress['Status'];
if (this.Status === 'In_Progress') {
this.delayStart();
this.intervalTimer.unsubscribe();
}
});
});
}
}
}, err => {
console.log(err);
});
}
}, error => {
console.log(error);
});
}

You may consider using the below approach See Code on Stackblitz
id = 1;
Values = { info: true };
get data() { return { id: this.id,info: this.Values.info}}
showProgressSubject$ = new BehaviorSubject(true);
showProgressAction$ = this.showProgressSubject$.asObservable();
currentStatusSubject$ = new Subject<string>();
currentStatus$ = this.currentStatusSubject$.asObservable()
stoppedSubject$ = new Subject();
stopped$ = this.stoppedSubject$.asObservable();
startedSubject$ = new Subject();
started$ = this.startedSubject$.asObservable();
interval = 500; // Change to 3000 for 3s
maxTrialTime = 6000;// Change to 120000 for 2min
timer$ = timer(0, this.interval).pipe(
tap((i) => {
if(this.maxTrialTime/this.interval < i) { this.stoppedSubject$.next()}
}),
takeUntil(this.stopped$),
repeatWhen(() => this.started$)
)
apiOneCall$ = this.userServ.start(this.data);
apiTwoCall$ = this.apiOneCall$.pipe(
switchMap(({Id}) => Id ? this.Service.getStatus(Id): throwError('No Id')),
tap((res) => this.currentStatusSubject$.next(res)),
tap(res => console.log({res})),
tap((res) => {if(res === 'created') {this.stoppedSubject$.next()}})
)
trialCallsToApiTwo$ = this.timer$.pipe(mergeMap(() => this.apiTwoCall$))
In your Html you can use the async pipe
Show Progress : {{ showProgressAction$ | async }} <br>
Timer: {{ timer$ | async }}<br>
Response: {{ trialCallsToApiTwo$ | async }}<br>
<button (click)="startedSubject$.next()">Start</button><br>
<button (click)="stoppedSubject$.next()">Stop</button><br>
Explanation
We begin by setting up the properties id, Values and data being a combination of the 2 values
id = 1;
Values = { info: true };
get data() { return { id: this.id,info: this.Values.info}}
We then create a Subject to help with tracking of the progress of the operations. I am using BehaviorSubject to set the initial value of showing Progress to true.
We will use currentStatus$ to store whether current state is 'in_progress' or 'created'
stopped$ and started will control our observable stream.
You may have a look at the below post What is the difference between Subject and BehaviorSubject?
showProgressSubject$ = new BehaviorSubject(true);
showProgressAction$ = this.showProgressSubject$.asObservable();
currentStatus$ = this.currentStatusSubject$.asObservable()
stoppedSubject$ = new Subject();
stopped$ = this.stoppedSubject$.asObservable();
startedSubject$ = new Subject();
started$ = this.startedSubject$.asObservable();
Next we define interval = 500; // Change to 3000 for 3s and maxTrialTime = 6000;// Change to 120000 for 2min
We then define a timer$ observable using the timer operator. The operator is used to generate a stream of values at regular interval
We set the delay to 0 and the interval to interval property we had earlier created
We then tap into the observable stream. The tap operator allows us perform an operation without changing the observable stream
In our tap operator, we check whether the maximum time has been reached and if it has we call the next function on stoppedSubject$. We pipe our stream to takeUntil(this.stopped$) to stop the stream and repeatWhen(() => this.started$) to restart the stream
timer$ = timer(0, this.interval).pipe(
tap((i) => {
if(this.maxTrialTime/this.interval < i) { this.stoppedSubject$.next()}
}),
takeUntil(this.stopped$),
repeatWhen(() => this.started$)
)
The Remaining part is to make a call to the apis
We will use switchMap to combine the two observables. switchMap will cancel any earlier request if a new request is made. If this is not your desired behaviour you may consider exhaustMap or the mergeMap operators
From the result of apiOneCall$ if no id, we use the throwError operator to indicate an error otherwise we return a call to apiTwo
We tap into the result of apiTwoCall$ and call the next function on currentStatusSubject$ passing in the response. This sets the value of currentStatus$ to the result of the response
The line tap((res) => {if(res === 'created') {this.stoppedSubject$.next()}}) taps into the result of apiTwoCall$ and if it is 'created' it stops the timer
apiOneCall$ = this.userServ.start(this.data);
apiTwoCall$ = this.apiOneCall$.pipe(
switchMap(({Id}) => Id ? this.Service.getStatus(Id): throwError('No Id')),
tap((res) => this.currentStatusSubject$.next(res)),
tap(res => console.log({res})),
tap((res) => {if(res === 'created') {this.stoppedSubject$.next()}})
)
Now we finally combine the timer$ and apiTwoCall$ with mergeMap operator trialCallsToApiTwo$ = this.timer$.pipe(mergeMap(() => this.apiTwoCall$))
In Our HTML we can then use the async pipe to avoid worrying about unsubscribing
{{ trialCallsToApiTwo$ | async }}

I'd use expand from rxjs, which will pass through the result of the source observable, but also let's you act according to the content of the result.
Also, avoid nesting calls to subscribe whenever possible. Consider this example code for reference:
this.userServ.start(data).pipe(
// use switchMap to not have 'nested' subscribe-calls
switchMap((result) => {
if (result['Id']) {
// if there is an ID, ask for the status
return this.Service.getStatus(result['Id']).pipe(
// use the expand operator to do additional processing, if necessary
expand((response) => response['Status'] === 'In_Progress'
// if the status is 'In_Progress', don't repeat the API call
? EMPTY
// otherwise, re-run the API call
: this.Service.getStatus(result['Id']).pipe(
// don't re-run the query immediately, instead, wait for 3s
delay(3000)
)
),
// Stop processing when a condition is met, in this case, 60s pass
takeUntil(timer(60000).pipe(
tap(() => {
// handle the timeout here
})
))
);
} else {
// if there is no ID, complete the observable and do nothing
return EMPTY;
}
}),
/**
* Since expand doesn't filter anything away, we don't want results that
* don't have the status 'In_Progress' to go further down for processing
*/
filter((response) => response['Status'] === 'In_Progress')
).subscribe(
(response) => {
this.Start();
}, (error) => {
console.log(error)
}
);

Node, wait and retry api calls that fail

So I fetch an array of urls from api with a rate limit, currently I handle this by adding a timeout to each call like this:
const calls = urls.map((url, i) =>
new Promise(resolve => setTimeout(resolve, 250 * i))
.then(() => fetch(url)
)
);
const data = await Promise.all(calls);
forcing a 250ms wait between each call. This ensures that the rate limit is never exceeded.
The thing is, this isn't really necessary. I've tried with 0ms wait time, and most of the cases I have to repeatedly reload the page four or five times before the api starts to return:
{ error: { status: 429, message: 'API rate limit exceeded' } }
and most of the times you only have to wait a second or so before you can safely reload the page and get all data.
A more reasonable approach would be to collect the calls that return 429 (if they do), wait for a set amount of time and then retry them (and perhaps redo this a set amount of times).
Problem, I'm a bit stumped as to how one would go about achieving this?
EDIT:
Just got home and will look through the answers but there seem to have been an assumption made which I don't believe is necessary: The calls does not have to be sequential, they can be fired (and returned) in any order.

The term for what you want is exponential backoff. You can modify your code so that it continues trying on a certain failure condition:
const max_wait = 2000;
async function wait(ms) {
return new Promise(resolve => {
setTimeout(resolve, ms);
});
}
const calls = urls.map(async (url) => {
let retry = 0, result;
do {
if (retry !== 0) { await wait(Math.pow(2, retry); }
result = await fetch(url);
retry++;
} while(result.status !== 429 || (Math.pow(2, retry) > max_wait))
return result;
}
Or you can try using a library to handle the backoff for you like https://github.com/MathieuTurcotte/node-backoff

If I understand the question right, your trying to:
a) Execute fetch() calls sequentially (with a possibly optional delay)
b) Retry failed requests with a backoff delay
As you likely found out, .map() does not really help with a) as it does not wait for any async stuff when iterating (which is why you create a greater and greater timeout with i*250).
I personally find it the easiest to keep things sequential by using a for of loop instead, as this will work nicely with async/await:
const fetchQueue = async (urls, delay = 0, retries = 0, maxRetries = 3) => {
const wait = (timeout = 0) => {
if (timeout) { console.log(`Waiting for ${timeout}`); }
return new Promise(resolve => {
setTimeout(resolve, timeout);
});
};
for (url of urls) {
try {
await wait(retries ? retries * Math.max(delay, 1000) : delay);
let response = await fetch(url);
let data = await (
response.headers.get('content-type').includes('json')
? response.json()
: response.text()
);
response = {
headers: [...response.headers].reduce((acc, header) => {
return {...acc, [header[0]]: header[1]};
}, {}),
status: response.status,
data: data,
};
// in reality, only do that for errors
// that make sense to retry
if ([404, 429].includes(response.status)) {
throw new Error(`Status Code ${response.status}`);
}
console.log(response.data);
} catch(err) {
console.log('Error:', err.message);
if (retries < maxRetries) {
console.log(`Retry #${retries+1} ${url}`);
await fetchQueue([url], delay, retries+1, maxRetries);
} else {
console.log(`Max retries reached for ${url}`);
}
}
}
};
// populate some real URLs urls to fetch
// index 0 will generate an inexistent URL to test error behaviour
const urls = new Array(101).fill(null).map((x, i) => `https://jsonplaceholder.typicode.com/todos/${i}`);
// fetch urls one after another (sequentially)
// and delay each request by 250ms
fetchQueue(urls, 250);
If a request fails (e.g. you get one of the errors specified in the array with error status codes), the above function will retry them a maximum of 3 times (by default) with a backoff delay that increases by a second on each retry.
As you wrote, the delay between requests is probably not necessary, so you could just remove the 250 in the function call. Because each request is executed one after the other, you're less likely to run into rate limit issues but if you do, it's very easy to add some custom delay.

Here is an example that allows to handle an array of promises sequencially, by setting a delay expressed in milliseconds and accepting a third callback determining whether the request should be retried.
In the below code, some sample requests are mocked to:
Test a successful response.
Test an error response. If the error response contains an error code and the error code is 403, true is returned and the call is retried in the next run (delayed by x milliseconds).
Test an error response without an error code.
There is a global counter below that give up the promise after N tries (in the below example 5), all of that is handled in this code:
const result = await resolveSequencially(promiseTests, 250, (err) => {
return ++errorCount, !!(err && err.error && err.error.status === 403 && errorCount <= 5);
});
Where the error count is first increased and it returns true if the error is defined, has an error property and its status is 403.
Of course, the example is just to test things out, but I think you're looking for something allowing you to have a cleverer control over the promise loop cycle, hence here is a solution doing just that.
I will add some comments below, you can run the test below to check what happens directly in the console.
// Nothing that relevant, this one is just for testing purposes!
let errorCount = 0;
// Declare the function.
const resolveSequencially = (promises, delay, onFailed, onFinished) => {
// store the results.
const results = [];
// Define a self invoking recursiveHandle function.
(recursiveHandle = (current, max) => { // current is the index of the currently looped promise, max is the maximum needed.
console.log('recursiveHandle invoked, current is, ', current ,'max is', max);
if (current === max) onFinished(results); // <-- if all the promises have been looped, resolve.
else {
// Define a method to handle the promise.
let handlePromise = () => {
console.log('about to handle promise');
const p = promises[current];
p.then((success) => {
console.log('success invoked!');
results.push(success);
// if it's successfull, push the result and invoke the next element.
recursiveHandle(current + 1, max);
}).catch((err) => {
console.log('An error was catched. Invoking callback to check whether I should retry! Error was: ', err);
// otherwise, invoke the onFailed callback.
const retry = onFailed(err);
// if retry is true, invoke again the recursive function with the same indexes.
console.log('retry is', retry);
if (retry) recursiveHandle(current, max);
else recursiveHandle(current + 1, max); // <-- otherwise, procede regularly.
});
};
if (current !== 0) setTimeout(() => { handlePromise() }, delay); // <-- if it's not the first element, invoke the promise after the desired delay.
else handlePromise(); // otherwise, invoke immediately.
}
})(0, promises.length); // Invoke the IIFE with a initial index 0, and a maximum index which is the length of the promise array.
}
const promiseTests = [
Promise.resolve(true),
Promise.reject({
error: {
status: 403
}
}),
Promise.resolve(true),
Promise.reject(null)
];
const test = () => {
console.log('about to invoke resolveSequencially');
resolveSequencially(promiseTests, 250, (err) => {
return ++errorCount, !!(err && err.error && err.error.status === 403 && errorCount <= 5);
}, (done) => {
console.log('finished! results are:', done);
});
};
test();

Async in RxJS Observable

First time with RxJS. Basically, I'm trying to make a twitter scraper which retrieves tweets from a query string. The search url allows specifying of a min_position parameter which can be the last id of the previous search to sort of paginate.
The process kind of looks like this (where it loops back at the end):
get page -> next() each scraped tweet -> set min_position -> get page (until !has_more_items)
Requesting the page returns a promise and so I somehow have to wait until this is completed until I can proceed. I was hoping to pass an async function to Observable.create() but that doesn't seem to work, it's only called a single time.
EDIT
I've had a play around after reading your resources as best as I could. I came up with the following abstraction of my problem.
import { from, Observable } from 'rxjs'
import { concatMap, map, switchMap } from 'rxjs/operators'
let pageNumber = 0
const PAGE_SIZE = 3, MAX_PAGES = 3
async function nextPage() {
if (pageNumber >= MAX_PAGES) {
throw new Error('No more pages available')
}
await new Promise(res => setTimeout(res, 500)) // delay 500ms
const output = []
const base = pageNumber++ * PAGE_SIZE
for (let i = 0; i < PAGE_SIZE; i++) {
output.push(base + i)
}
return output
}
function parseTweet(tweet: number): string {
// simply prepend 'tweet' to the tweet
return 'tweet ' + tweet
}
const getTweets = (): Observable<string> => {
return from(nextPage()) // gets _html_ of next page
.pipe(
concatMap(page => page), // spreads out tweet strings in page
map(tweet => parseTweet(tweet)), // parses each tweet's html
switchMap(() => getTweets()) // concat to next page's tweets
// stop/finish observable when getTweets() observable returns an error
)
}
getTweets()
.subscribe(val => console.log(val))
It's quite close to working but now whenever nextPage() returns a rejected promise, the entire observable breaks (nothing logged to the console).
I've attempted inserting a catchError after the pipe to finish the observable instead of running through and throwing an error but I can't get it to work.
Also this implementation is recursive which I was hoping to avoid because it's not scalable. I don't know how many tweets/pages will be processed in the observable in future. It also seems that tweets from all 3 pages must be processed before the observable starts emitting values which of course is not how it should work.
Thanks for your help! :)

We need to loadTwits until some condition and somehow work with Promise? Take a look at example:
function loadTwits(id) {
// Observable that replay last value and have default one
twitId$ = new BehaviorSubject(id);
return twitId$.pipe(
// concatMap - outside stream emit in order inner do
// from - convert Promise to Observable
concatMap(id => from(fetchTwits(id))),
map(parseTwits),
// load more twits or comlete
tap(twits => getLastTwitId(twits) ? twitId$.next(getLastTwitId(twits)) : twitId$.complete())
)
}

I figured it out after looking further into expand and realising it was recursion that I needed in my observable. This is the code that creates the observable:
const nextPage$f = () => from(nextPage()) // gets _html_ of next page
.pipe(
concatMap(page => page), // spreads out tweet strings in page
map(tweet => parseTweet(tweet)) // parses each tweet's html
)
const tweets$ = nextPage$f()
.pipe(
expand(() => morePages() ? nextPage$f() : empty())
)

How can I unsubscribe or cancel the filtering of a large array that is an RxJS observable?

My understanding is that an entire array is pushed to a subscriber, unlike say an interval observer that can be unsubscribed/cancelled.
For example the following cancellation works...
// emit a value every second for approx 10 seconds
let obs = Rx.Observable.interval(1000)
.take(10)
let sub = obs.subscribe(console.log);
// but cancel after approx 4 seconds
setTimeout(() => {
console.log('cancelling');
sub.unsubscribe()
}, 4000);
<script src="https://unpkg.com/rxjs#5.5.10/bundles/Rx.min.js"></script>
However, replacing the interval with an array doesn't.
// emit a range
let largeArray = [...Array(9999).keys()];
let obs = Rx.Observable.from(largeArray)
let sub = obs.subscribe(console.log);
// but cancel after approx 1ms
setTimeout(() => {
console.log('cancelling');
sub.unsubscribe()
}, 1);
// ... doesn't cancel
<script src="https://unpkg.com/rxjs#5.5.10/bundles/Rx.min.js"></script>
Does each element need to be made asynchronous somehow, for example by wrapping it in setTimeout(..., 0)? Perhaps I've been staring at this problem too long and I'm totally off course in thinking that the processing of an array can be cancelled?

When using from(...) on an array all of the values will be emitted synchronously which doesn't allow any execution time to be granted to the setTimeout that you are using to unsubscribe. Infact, it finishes emitting before the line for the setTimeout is even reached. To allow the emits to not hog the thread you could use the async scheduler (from(..., Rx.Scheduler.async)) which will schedule work using setInterval.
Here are the docs: https://github.com/ReactiveX/rxjs/blob/master/doc/scheduler.md#scheduler-types
Here is a running example. I had to up the timeout to 100 to allow more room to breath. This will slow down your execution of-course. I don't know the reason that you are attempting this. We could probably provide some better advice if you could share the exact use-case.
// emit a range
let largeArray = [...Array(9999).keys()];
let obs = Rx.Observable.from(largeArray, Rx.Scheduler.async);
let sub = obs.subscribe(console.log);
// but cancel after approx 1ms
setTimeout(() => {
console.log('cancelling');
sub.unsubscribe()
}, 100);
// ... doesn't cancel
<script src="https://unpkg.com/rxjs#5.5.10/bundles/Rx.min.js"></script>

I've marked #bygrace's answer correct. Much appreciated! As mentioned in the comment to his answer, I'm posting a custom implementation of an observable that does support such cancellation for interest ...
const observable = stream => {
let timerID;
return {
subscribe: observer => {
timerID = setInterval(() => {
if (stream.length === 0) {
observer.complete();
clearInterval(timerID);
timerID = undefined;
}
else {
observer.next(stream.shift());
}
}, 0);
return {
unsubscribe: () => {
if (timerID) {
clearInterval(timerID);
timerID = undefined;
observer.cancelled();
}
}
}
}
}
}
// will count to 9999 in the console ...
let largeArray = [...Array(9999).keys()];
let obs = observable(largeArray);
let sub = obs.subscribe({
next: a => console.log(a),
cancelled: () => console.log('cancelled')
});
// except I cancel it here
setTimeout(sub.unsubscribe, 200);

Repeat request (Angular2 - http.get) n seconds after finished

I played around with angular2 and got stuck after a while.
Using http.get works fine for a single request, but I want to poll live-data every 4 seconds, after tinkering for quite a while and reading a lot of reactivex stuff i ended up with:
Observable.timer(0,4000)
.flatMap(
() => this._http.get(this._url)
.share()
.map(this.extractData)
.catch(this.handleError)
)
.share();
Is there a simple way to start a (4 second) interval after the http.get-observable has emitted the result of the request? (Or will I end up in observable-hell?)
Timeline i want:
Time(s): 0 - - - - - 1 - - - - - 2 - - - - - 3 - - - - - 4 - - - - - 5 - - - - - 6
Action: Request - - Response - - - - - - - - - - - - - - - - - - - -Request-...
Wait: | wait for 4 seconds -------------------------> |

Update to RxJS 6
import { timer } from 'rxjs';
import { concatMap, map, expand, catchError } from 'rxjs/operators';
pollData$ = this._http.get(this._url)
.pipe(
map(this.extractData),
catchError(this.handleError)
);
pollData$.pipe(
expand(_ => timer(4000).pipe(concatMap(_ => pollData$)))
).subscribe();
I'm using RxJS 5 and I'm not sure what the RxJS 4 equivalent operators are. Anyway here is my RxJS 5 solution, hope it helps:
var pollData = this._http.get(this._url)
.map(this.extractData)
.catch(this.handleError);
pollData.expand(
() => Observable.timer(4000).concatMap(() => pollData)
).subscribe();
The expand operator will emit the data and recursively start a new Observable with each emission

I managed to do it myself, with the only downside beeing that http.get can't be repeated more easily.
pollData(): Observable<any> {
//Creating a subject
var pollSubject = new Subject<any>();
//Define the Function which subscribes our pollSubject to a new http.get observable (see _pollLiveData() below)
var subscribeToNewRequestObservable = () => {
this._pollLiveData()
.subscribe(
(res) => { pollSubject.next(res) }
);
};
//Subscribe our "subscription-function" to custom subject (observable) with 4000ms of delay added
pollSubject.delay(4000).subscribe(subscribeToNewRequestObservable);
//Call the "subscription-function" to execute the first request
subscribeToNewRequestObservable();
//Return observable of our subject
return pollSubject.asObservable();
}
private _pollLiveData() {
var url = 'http://localhost:4711/poll/';
return this._http.get(url)
.map(
(res) => { return res.json(); }
);
};
Here is why you can't use the more straight forward subscription:
var subscribeToNewRequestObservable = () => {
this._pollLiveData()
.subscribe(pollSubject);
};
The completion the http.get-observable would also complete your subject and prevent it from emitting further items.
This is still a cold observable, so unless you subscribe to it no requests will be made.
this._pollService.pollData().subscribe(
(res) => { this.count = res.count; }
);

A minor rework of the answer from Can Nguyen, in case you want polling delay to depend on previous request completion status.
var pollData = () => request() // make request
.do(handler, errorHandler) // handle response data or error
.ignoreElements() // ignore request progress notifications
.materialize(); // wrap error/complete notif-ns into Notification
pollData() // get our Observable<Notification>...
.expand( // ...and recursively map...
(n) => Rx.Observable // ...each Notification object...
.timer(n.error ? 1000 : 5000) // ...(with delay depending on previous completion status)...
.concatMap(() => pollData())) // ...to new Observable<Notification>
.subscribe();
Plunk.
Or alternatively:
var pollData = () => request() // make request
.last() // take last progress value
.catch(() => Rx.Observable.of(null)); // replace error with null-value
pollData()
.expand(
(data) => Rx.Observable
.timer(data ? 5000 : 1000) // delay depends on a value
.concatMap(() => pollData()))
.subscribe((d) => {console.log(d);}); // can subscribe to the value stream at the end
Plunk.

You can try using interval if that is more convenient. Calling subscribe gives you Subscription that lets you cancel the polling after sometime.
let observer = Observable.interval(1000 * 4);
let subscription = observer.subsscribe(x => {
this._http.get(this._url)
.share()
.map(this.extractData)
.catch(this.handleError)
});
....
// if you don't require to poll anymore..
subscription.unsubscribe();