First time with RxJS. Basically, I'm trying to make a twitter scraper which retrieves tweets from a query string. The search url allows specifying of a min_position parameter which can be the last id of the previous search to sort of paginate.
The process kind of looks like this (where it loops back at the end):
get page -> next() each scraped tweet -> set min_position -> get page (until !has_more_items)
Requesting the page returns a promise and so I somehow have to wait until this is completed until I can proceed. I was hoping to pass an async function to Observable.create() but that doesn't seem to work, it's only called a single time.
EDIT
I've had a play around after reading your resources as best as I could. I came up with the following abstraction of my problem.
import { from, Observable } from 'rxjs'
import { concatMap, map, switchMap } from 'rxjs/operators'
let pageNumber = 0
const PAGE_SIZE = 3, MAX_PAGES = 3
async function nextPage() {
if (pageNumber >= MAX_PAGES) {
throw new Error('No more pages available')
}
await new Promise(res => setTimeout(res, 500)) // delay 500ms
const output = []
const base = pageNumber++ * PAGE_SIZE
for (let i = 0; i < PAGE_SIZE; i++) {
output.push(base + i)
}
return output
}
function parseTweet(tweet: number): string {
// simply prepend 'tweet' to the tweet
return 'tweet ' + tweet
}
const getTweets = (): Observable<string> => {
return from(nextPage()) // gets _html_ of next page
.pipe(
concatMap(page => page), // spreads out tweet strings in page
map(tweet => parseTweet(tweet)), // parses each tweet's html
switchMap(() => getTweets()) // concat to next page's tweets
// stop/finish observable when getTweets() observable returns an error
)
}
getTweets()
.subscribe(val => console.log(val))
It's quite close to working but now whenever nextPage() returns a rejected promise, the entire observable breaks (nothing logged to the console).
I've attempted inserting a catchError after the pipe to finish the observable instead of running through and throwing an error but I can't get it to work.
Also this implementation is recursive which I was hoping to avoid because it's not scalable. I don't know how many tweets/pages will be processed in the observable in future. It also seems that tweets from all 3 pages must be processed before the observable starts emitting values which of course is not how it should work.
Thanks for your help! :)
We need to loadTwits until some condition and somehow work with Promise? Take a look at example:
function loadTwits(id) {
// Observable that replay last value and have default one
twitId$ = new BehaviorSubject(id);
return twitId$.pipe(
// concatMap - outside stream emit in order inner do
// from - convert Promise to Observable
concatMap(id => from(fetchTwits(id))),
map(parseTwits),
// load more twits or comlete
tap(twits => getLastTwitId(twits) ? twitId$.next(getLastTwitId(twits)) : twitId$.complete())
)
}
I figured it out after looking further into expand and realising it was recursion that I needed in my observable. This is the code that creates the observable:
const nextPage$f = () => from(nextPage()) // gets _html_ of next page
.pipe(
concatMap(page => page), // spreads out tweet strings in page
map(tweet => parseTweet(tweet)) // parses each tweet's html
)
const tweets$ = nextPage$f()
.pipe(
expand(() => morePages() ? nextPage$f() : empty())
)
Related
I have an observable that fetches an array of items (32 each time) from an API and emits a new response until there are no items left to fetch.
I want to process said list of items one by one as soon as i get the first batch until im done with ALL items fetched.
When i'm done with the complete list, i want to repeat the process indefinitely.
Here's what i have so far:
_dataService
.getItemsObservable()
.pipe(
switchMap((items) => {
const itemList = items.map((i) => i.itemId);
return of(itemList);
}),
concatMap((item) =>
from(item).pipe(
concatMap((item) => {
// do something here
}
)
)
),
repeat()
).subscribe()
Any idea on what can i do? Right now what happens is it will loop over the first batch of items and ignore the rest
Replay wont call the service again, it will reuse the original values. Try switchMap from a behaviour subject and make it emit after you have processed the values. Really not sure why you would turn each item into an observable to concatMap. Just process the items after they are emitted.
const { of, BehaviorSubject, switchMap, delay } = rxjs;
const _dataService = {
getItemsObservable: () => of(Array.from({ length: 32 }, () => Math.random()))
};
const bs$ = new BehaviorSubject();
bs$.pipe(
delay(1000), // Behavior subject are synchronous will cause a stack overflow
switchMap(() => _dataService.getItemsObservable())
).subscribe(values => {
values.forEach(val => {
console.log('Doing stuff with ' + val);
});
bs$.next();
});
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/7.8.0/rxjs.umd.min.js" integrity="sha512-v0/YVjBcbjLN6scjmmJN+h86koeB7JhY4/2YeyA5l+rTdtKLv0VbDBNJ32rxJpsaW1QGMd1Z16lsLOSGI38Rbg==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
I have an observable that fetches an array of items (32 each time) from an API and emits a new response until there are no items left to fetch.
Okay, I assume that is _dataService.getItemsObservable()?
I want to process said list of items
What does this mean? Process how? Lets assume you have some function called processItemById that processes an itemId and returns the processed item.
one by one as soon as i get the first batch until im done with ALL items fetched.
Sounds like you're turning an Observable<T[]> into an Observable<T>. You can use mergeMap (don't care about order) or concatMap (maintain order) to do this. Since you're just flattening an inner array, they'll be the same in this case.
_dataService.getItemsObservable().pipe(
mergeMap(v => v),
map(item => processItemById(item.itemId)),
// Repeat here doesn't call `getItemsObservable()` again,
// instead it re-subscribes to the observable that was returned.
// Hopefully that's what you're counting on. It's not clear to me
repeat()
).subscribe(processedItemOutput => {
// Do something with the output?
});
Any idea on what can i do?
From your explanation and code, it's not clear what you're trying to do. Maybe this helps.
Right now what happens is it will loop over the first batch of items and ignore the rest
This could happen for a number of reasons.
Tip 1
Using higher-order mapping operators with RxJS::of is a code smell. Just use a regular map instead.
for example:
concatMap(v => of(fn(v)))
// or
switchMap(v => of(fn(v)))
are the same as:
map(v => fn(v))
Tip 2
I have no idea if this would help you but you can generate a new observable on each subscribe by using the delay operator.
For example:
defer(() => _dataService.getItemsObservable()).pipe(
mergeMap(v => v),
map(item => processItemById(item.itemId)),
repeat()
).subscribe(processedItemOutput => {
// Do something with the output?
});
It looks like you want to get all records from an API that is paginated but won't tell you how many pages are there which sounds like you're looking for expand() operator which is great for recursive calls.
import { of, EMPTY, expand, range, toArray, mergeMap, concat, map, takeLast } from 'rxjs';
const MAX_PAGES = 3;
const PER_PAGE = 32;
const fetchList = (offset: number) => {
return Math.ceil(offset / PER_PAGE) >= MAX_PAGES ? of([]) : range(offset, 32).pipe(toArray());
};
const fetchDetail = (id: number) => {
return of(`response for ${id}`);
};
of([]) // Seed for `expand()`.
.pipe(
expand((acc, index) => fetchList(acc.length).pipe(
mergeMap(list => { // Process each response array
console.log(`Response #${index}`, list);
// When the reponse is an empty we can stop recursive calls.
if (list.length === 0) {
return EMPTY;
}
// Process the response and make `fetchDetail()` call for each item.
// `concat()` will guarantee order and process items one by one.
return concat(...list.map(id => fetchDetail(id)))
.pipe(
toArray(),
map(details => [...acc, ...details]), // Append the processed response to a single large array.
);
}),
)),
takeLast(1), // Only take the final array after all pages have been fetched.
)
.subscribe(console.log);
Working demo: https://stackblitz.com/edit/rxjs-yqd4kx?devtoolsheight=60&file=index.ts
I am a novice with Rxjs, I'm trying to do a request to my API with limits passed in parameters.
My problem is sometimes the returned result is empty for some reasons. The thing I need to do is retry this API call with updated parameters (skip param)
pollService.getPoll$(skip, limit).subscribe((pollList) => {
doStuff;
},
(error) => {
doStuff;
});
I read some topics about the RetryWhen RXJS function but it is about errors when the request fail and you want to retry the same one but I ve no errors and I don't want to retry the same request, I also saw topics about Replay function but it is not very clear to me.
Can someone explain me what to do here please !!
Thanks
Alex
Consider utilizing the expand operator as demonstrated below:
import { EMPTY, of } from "rxjs"
import { expand } from "rxjs/operators"
public startPolling(skip: number, limit: number): void {
of([])
.pipe(
expand(x => x.length < 2 ? pollService.getPoll$(skip--, limit) : EMPTY)
)
.subscribe(pollList => {})
}
Update:
public poll = (skip: number, limit: number): void => {
defer(() => getPoll$(1 + skip--, limit))
.pipe(
repeat(),
first(x => {
if(x.length < 2){
// update some variable in the component
return false;
}
return true;
})
)
.subscribe(pollList => { })
}
If I understand correctly, your backend is paging data using the skip and limit parameters.
So, if you have a skip value that is too high, you want to reduce it automatically.
There are many, many ways to solve this problem in RxJS:
you could insert a switchMap after getPoll$. SwitchMap would return a new observable, either wrapping the result if it's ok (with of(pollList)), or returning pollService.getPoll$(newSkipValue, newLimitValue)
you could map the result and throw an Error if the result doesn't pass validation. Then you could catchError and return the result of a new call to getPoll$
However, what I suggest is modelling the call differently. I would use a Subject as a source of requests, and switchMap to execute the requests.
// inside the component or service
interface GetPollRequest {
skip: number;
limit: number;
}
private _callSource = new Subject<GetPollRequest>();
public triggerCall(skip: number, limit: number) {
this._callSource.next({skip, limit});
}
constructor(...) {
this._callSource.pipe(
// every time _callSource emits, we call the server
switchMap(({skip, limit) => pollService.getPoll$(skip, limit).pipe(
map(pollList => ({ pollList, skip, limit }))
),
tap(({pollList, skip, limit}) => {
// update the request in any way you need. You need to make sure
// that the next automatic trigger doesn't repeat indefinitely,
// or you'll simulate a DOS attack to your backend
if (pollList.length < 2) this.triggerCall(skip - 2, limit);
})
).subscribe(pollList => // update the component status);
}
Using this pattern, you use subjects as triggers (or custom events, they are pretty much the same), and you wrap them up during constructor time.
SwitchMap is used to create an observable (in this case, performing a request) every time the source emits.
Tap is used to perform an operation (pretty much like a subscribe), embedded in the chain of transformations inside the pipe.
I have 2 methods that call some APIs:
getData(param).
getAnotherData(resp from get data).
The below method calls the first method:
this.service.getData(param).subscribe(res => {
this.variable = res;
// here based on the first response i will call the second api
this.service.getAnotherData(this.variable['data']).subscribe(res2 =>
// Here, I will get a response which will be either "started"
// or not "started". If it's not "started", then I have to call
// this API until I get "started" for up to 1 minute.
)})
How can I achieve this?
Try to avoid nested subscriptions. Instead try to use higher order mapping operators like switchMap.
You could use RxJS repeat and takeWhile operators to subscribe again to an observable until it satisfies a condition.
Try the following
this.service.getData(param).pipe(
switchMap(res =>
this.service.getAnotherData(res['data']).pipe(
repeat(),
takeWhile(res => res === 'started') // poll `getAnotherData()` until response is 'started'
)
)
).subscribe(
res => { },
err => { }
);
You may consider below approach using timer operator
private readonly _stop = new Subject<void>();
private readonly _start = new Subject<void>();
trialTime = 60 * 1000;
trialInterval = 500;
timer$ = timer(0, this.trialInterval).pipe(
takeUntil(this._stop),
takeWhile( i => (this.trialTime / this.trialInterval) > i),
repeatWhen(() => this._start),
);
start(): void { this._start.next();}
stop(): void { this._stop.next();}
responses = [];
ngOnInit() {
const param = 1;
this.timer$.pipe(
mergeMap(() => this.service.getData(param)),
switchMap(res => this.service.getAnotherData(res.data)),
tap(res => this.responses.push(res)),
tap(res => { if (res === 'started') { this.stop(); }}),
).subscribe();
}
The above approach will run for 60s and if not started, this will automatically stop.
A user can also manually stop and start the observable
See Below link on stackblitz
If I were you and had to do something similar with vanilla JS, I would use this approach:
1. Subscribe to `getData` function call result
2. In the scope of subscription handler:
2.1 Add a new method to indicate if 1 minute has elapsed with initial value to `false`
2.2 Use `setTimeout` to callback a function and update flag after a minute to `true`
2.3 Create inline function which would call `getAnotherData` method and subscribe to it's result and within its subscription scope
2.3.1 If result is started, clear timeout so that timeout event handler will be cleared
2.3.2 If one minute is not elapsed, call inline function again
2.4 call inline function
something like:
this.service.getData(param).subscribe(res => {
this.variable = res;
const minuteElapsed = false;
// based on the first response
const timeoutHandler = setTimeout(
() => { minuteElapsed = true; },
60000
);
const fetchAnotherData = () => {
this.service.getAnotherData(this.variable['data']).subscribe(res2 =>
if (res === 'started') {
clearTimeout(timeoutHandler); // clean up
// do stuff!
} else if (!minuteElapsed) {
fetchAnotherData(); // calling again!
}
});
};
fetchAnotherData(); // call inline function
})
Like what was mentioned in the other answers, avoid nested subscriptions whenever possible. I'd approach this with rxjs expand operator, which we will use to re-run the second API-request until your desired result is received.
expand will pass-through the observed value from the API request and, depending on a condition, you can have it either stop additional processing or just repeat the call.
We can then use takeUntil to implement a timeout to the request. It might look something like this (I'm borrowing the scaffolding from another answer):
this.service.getData(param).pipe(
switchMap(res =>
this.service.getAnotherData(res['data']).pipe(
expand((result) => result === 'finished' ? EMPTY : this.service.getAnotherData(res['data']).pipe(delay(500))),
takeUntil(timer(60000).pipe(
// Here, you can react to the timeout
tap(() => console.log('Timeout occurred'))
)),
)
)
).subscribe(
(res) => {
/**
Do whatever you want with the result, you need to check again for 'finished',
since everything is passed through
*/
}
err => { }
);
Have a play with it here: https://retry-api.stackblitz.io
EDIT: usage of toPromise will be depreciated, see comments on this answer
if your function containing the calls is async you could use await to wait for the result in this particular case.
const res = await this.service.getData(param).toPromise();
this.variable = res;
let anotherData;
while (anotherData !== 'started') {
anotherData = await this.service.getAnotherData(this.variable['data']).toPromise();
}
I have a rather obscure problem and it may indeed arise from my own doing so feel free to point it out if that is the case.
I have a chain of promises, which I need access to intermediate values for subsequent API calls. Once the promises are all resolved, I then execute an API call that makes use of the various intermediate return values.
Finally, based on the results of that call I then make a final call to an API. However, the results of this call are used to update a Redux State and therefore are done using Redux Thunk as the middleware to facilitate async actions.
The challenge arises in that I then need to poll a database at a set interval to check if the Redux API call has carried out the work requested of it (This worked involved placing some long running tasks on a Redis task queue and then having the queue worker update a database once the task is complete).
When the database confirms that it has indeed been updated with the completed task status, I want to clear the setIntervalId. I have simplified the code to that shown below. The issue is that the execution does not wait for the Redux async action to be completed. So it executes the action then goes straight on to carry out the 'if' check before the Redux action is cmplete. Therefore the if is never true and the polling continues indefinitely.
I do not know how to ensure the Redux Action is complete before code continues. I have trimmed the code down to get the logic in place without overcrowding this post with irrelevant code.
Any ideas how I can get this to work as intended?
buttonClickedHandler.js
callbackFn {function(){
const a = ApiCallA(...) // Returns a promise
const b = a.then(resA => {
// Some work done here
return ApiCallB(…); // Returns a promise
const c = b.then(resB => {
// Some work done here
return ApiCallC(…); // Returns a promise
return Promise.all( ([a, b, c] )
.then( ([resA, resB, resC] ) => {
// Some work done here
return ApiCallD
})
.then( resD => {
const intId = setInterval( () => {
reduxAsyncAction(resD.jobId) // <-- This makes a final API call
if (props.jobStatus){ // <-- jobStatus gets updated too late
clearInterval(intId );
}
}, 5000)
})
.catch(err => console.log(err))
}
redux action creator
export const reduxAsyncAction= (jobId) => {
return dispatch => {
dispatch(taskStatusStart()); // <- Basic sync. action creator
return databaseQuery() // Returns Promise therefore pushed to taskqueue while other function run
.then(dbData=>{
const completionStatus = dbData.status; // <- True or False
dispatch(taskStatusSuccess(completionStatus)) // <- Basic sync. action creator
},
error => {
dispatch(taskStatusFail(error)); // <- Basic sync. action creator
})
};
}
Use componentDidUpdate or useEffect
These will run whenever props.jobStatus updates in your redux store
Check for taskStatusSuccess(completionStatus) or taskStatusFail(error) that you are dispatching from reduxAsyncAction
clearinterval(intId) if the status matches
You will also need to add the intId to be within scope by declaring outside of the callbackFn such as:
this.intId in the constructor of a class based component
let intId at the top of the functional component.
Class based components:
componentDidUpdate(){
if (intId && props.jobStatus === 'completionStatus' || props.jobStatus === 'error'){
clearInterval(intId);
intId = null;
}
}
Functional components:
useEffect(()=>{
if (intId && props.jobStatus === 'completionStatus' || props.jobStatus === 'error'){
clearInterval(intId);
intId = null;
}
},[props.jobStatus])
I have an issue with Observable chain and can't found decision.
I need to wait for result from IndexedDB, push it to next request and combine two results in last Observable.
Here is example:
const first$ = Database.gelAll(store); // return Observable
first.mergeMap((data) => {
const uuids = data.map((v) => {
return v.uuid;
});
// here i need to send request with 'uuids' array and combine result with
values from first request
const second$ = database.find(store, {'uuid': uuids}); // return Observable
});
Thanks for any advices.
If i understand you correctly, you're trying to make it so the end result of your observable is an object containing both the result from Database.gelAll and also the result from database.find. To do that, you'll just need to add a .map statement after the database.find, and return it to the merge map.
Database.gelAll(store)
.mergeMap((data) => {
const uuids = data.map((v) => v.uuid);
return database.find(store, {'uuid': uuids})
.map((databaseResult) => {
// Combine with data in some fashion
return {
firstData: data,
secondData: databaseResult
}
});
})
.subscribe(objectWithBothFirstDataAndSecondData => {
console.log(objectWithBothFirstDataAndSecondData)
});
Also, you should consider whether .mergeMap is appropriate. If .gelAll only emits one value then it should be ok, but in most cases, .switchMap or .concatMap is a better choice than .mergeMap. With mergeMap you have no guarantee of the order. Switchmap ensures that only the latest is used, concatMap ensures that everything gets through, but they get through in the same order they were asked for.