Develop Reference

RXJS: Can I interleave multiple streams based on order of entries to a single output?

This is purely an academic exercise to understand operators. How can I use RXJS to interleave incoming streams
e.g. go from this:
// RxJS v6+
import { of } from 'rxjs';
//emits any number of provided values in sequence
const source = of(1, 2, 3, 4, 5);
//output: 1,2,3,4,5
const subscribe = source.subscribe(val => console.log(val));
to this:
import { of } from 'rxjs';
const source1 = of(1,3,5,7,9);
const source2 = of(2,4,6,8,10);
// a simple merge will just append source1 and source2
// how do I obtain an output = 1,2,3,4,5,6,7,8,9,10
const subscribe = merge(source1, source2).subscribe(val => console.log(val))
Constraints: This is a general question so I dont want to use predicates based on the odd/even separation of this specific example, but to create an output based purely on order of incoming elements i.e. take the first from each stream, then the second, etc.
Is that possible?

The standard way of doing this is indeed the merge function, which can receive any number of observables as arguments, and emits all of their items in sequece regardless of the order of the observables in the function parameters. But as you pointed out, it appended the results, the reason for this is not an issue with merge, but actually that of emmited the values synchronously, so they all ran in order in the same javascript loop. You can change that by using an observable that emits values assynchronously, or create an observable with some of the available async Schedulers
Assynchronous observable:
I created this asyncOffunction as a mockup to a Observable with values emmited over time, like a websocket client or user interations.
import { merge, Observable, EMPTY } from 'rxjs';
import { toArray } from 'rxjs/operators';
const asyncOf = <T = any>(args: T[], interval = 500): Observable<T> => {
let i = 0;
let intervalRef;
if(!args?.length) {
return EMPTY;
}
return new Observable(observer => {
intervalRef = setInterval(() => {
if(args[i]) {
observer.next(args[i])
}
++i;
if(!args[i]) {
observer.complete()
if(intervalRef) {
clearInterval(intervalRef)
}
}
}, interval)
})
}
const source1 = asyncOf([1, 3, 5, 7, 9]);
const source2 = asyncOf([2, 4, 6, 8, 10]);
//output: 1,2,3,4,5,6,7,8,9,10
const subscribe = merge(source1, source2)
//.pipe(toArray()) //collect all emmited values and emmit an array when the observable completes
.subscribe((val) => console.log(val));
In this case the merge works correctly, because both observables are running asynchronously.
Async schedulers
Another option is using one of the async schedulers (or create your own) to describe when each item should be emmited. This is closer to your example, and a somewhat deeper dive into rxjs:
import { merge, scheduled, asyncScheduler } from 'rxjs';
import { toArray } from 'rxjs/operators';
const source1 = scheduled([1, 3, 5, 7, 9], asyncScheduler);
const source2 = scheduled([2, 4, 6, 8, 10], asyncScheduler);
//output: 1,2,3,4,5,6,7,8,9,10
const subscribe = merge(source1, source2)
//.pipe(toArray()) //collect all emmited values and emmit an array when the observable completes
.subscribe((val) => console.log(val));
Scheduled comes on RXJS 6.5+ and deprecates the use of scheduler function in other functions like of,

You could zip the streams, and then unpack the zipped values like this:
const odd$ = of(1, 3, 5, 7, 9).pipe(delay(50));
const even$ = of(2, 4, 6, 8, 10);
zip([odd$, even$])
.pipe(switchMap(([odd, even]) => of(odd, even)))
.subscribe(console.log);
stackblitz: https://stackblitz.com/edit/rxjs-uktrkz?devtoolsheight=60&file=index.ts

Ensuring React set hook runs synchronously in order

I have a react state that contains an array of objects that look like this {ID: 7, LicenseNumber: 'Testing123', LicenseCreator: 7, added: true}
To manipulate the individual values of this object, I have prepared a function that looks like this:
const updateLicenseInfo = (index, licenseInfoKey, licenseInfoVal) => {
const foundLicenseInfo = { ...licenseInfo[index] };
const licenseInfoItems = [...licenseInfo];
foundLicenseInfo[licenseInfoKey] = licenseInfoVal;
licenseInfoItems[index] = foundLicenseInfo;
setLicenseInfo([...licenseInfoItems]);
};
My problem is when I run this function twice in a row, I believe the asynchronous nature of hooks only causes the last function to run.
For example, when I make an API call and then try to update the ID, then the added field, ID will be null but added will have changed. Here is that example:
postData('/api/licenseInfo/createLicenseInfoAndCreateRelationship', {
LicenseNumber,
RequestLineItemID: procurementLine.ID
})
.then((res) => res.json())
.then((r) => {
updateLicenseInfo(licenseIndex, 'ID', r.ID);
updateLicenseInfo(licenseIndex, 'added', true);
})
How Can I ensure that this function runs and then the next function runs synchronously

How about refactoring your initial function to take an array of objects as parameters, something with the structure {key: "", val: ""} and then iterating over that array in your function, adding all the values to the paired keys and calling setState only once, with all the new changes
const updateLicenseInfo = (index, pairs) => {
const foundLicenseInfo = { ...licenseInfo[index] };
const licenseInfoItems = [...licenseInfo];
pairs.forEach((pair)=>{
foundLicenseInfo[pair.key] = pair.value;
});
licenseInfoItems[index] = foundLicenseInfo;
setLicenseInfo([...licenseInfoItems]);
};
and called like this
updateLicenseInfo(licenseIndex, [{key:'ID', value:r.ID},
{key:'added', value:true}]);

double combineLatest doesn't emit update

In my project there are activities that people have created, joined, bookmarked or organized. I've read a lot of these question already. But most of the code was less complex or people forgot to subscribe...
I would like to get all the activities in a certain time period and then add creator information (name, image, etc) and add booleans if the user retrieving these activities has joined/bookmarked/organized this activity. The code I used before would provide live updates (ex. I join an activity, by adding my userId to the participants array and the activity.joined would update to true).
Previous code:
public getActivities(user: UserProfileModel): Observable<Array<ActivityModel>> {
const now: number = moment().startOf('day').unix();
const later: number = moment().startOf('day').add(30, 'day').unix();
return this.afs.collection<ActivityModel>(`cities/${user.city.id}/activities`, ref => ref
.where('datetime', '>=', now)
.where('datetime', '<=', later))
.valueChanges({ idField: 'id' })
.pipe(
map(activities => activities.map(activity => {
const bookmarked = activity.bookmarkers ? activity.bookmarkers.includes(user.uid) : false;
const joined = activity.participants ? activity.participants.includes(user.uid) : false;
const organized = activity.organizers ? activity.organizers.includes(user.uid) : false;
return { bookmarked, joined, organized, ...activity } as ActivityModel;
}))
);
}
The I wanted to add the creator as an observable object, so their latest changes in name or profile picture would be shown. But with this code change, my getActivities doesn't emit any updates anymore...
My new code:
public getActivities(user: UserProfileModel): Observable<Array<CombinedActivityCreatorModel>> {
const now: number = moment().startOf('day').unix();
const later: number = moment().startOf('day').add(30, 'day').unix();
return this.afs.collection<ActivityModel>(`cities/${user.city.id}/activities`, ref => ref
.where('datetime', '>=', now)
.where('datetime', '<=', later))
.valueChanges({ idField: 'id' })
.pipe(
concatMap(activities => {
const completeActivityData = activities.map(activity => {
const activityCreator: Observable<UserProfileModel> = this.getCreator(activity.creator);
const bookmarked = activity.bookmarkers ? activity.bookmarkers.includes(user.uid) : false;
const joined = activity.participants ? activity.participants.includes(user.uid) : false;
const organized = activity.organizers ? activity.organizers.includes(user.uid) : false;
return combineLatest([
of({ bookmarked, joined, organized, ...activity }),
activityCreator
]).pipe(
map(([activityData, creatorObject]: [ActivityModel, UserProfileModel]) => {
return {
...activityData,
creatorObject: creatorObject
} as CombinedActivityCreatorModel;
})
);
});
return combineLatest(completeActivityData);
})
);
}
The code has become a bit complex, that I don't see the solution myself. Anybody that can offer some assistance?

Looks like one of activityCreator doesn't emit a value, combineLatest requires all observables to emit at least once.
I would recommend you to debug how activityCreator behaves.
If it's fine that it doesn't emit you have 2 options: startWith to set a value for an initial emit, or defaultIfEmpty, it emits in case if stream is going to be closed without any emit.
activityCreator = this.getCreator(activity.creator).pipe(
// startWith(null), // for example if you want to trigger combineLatest.
// defaultIfEmpty(null), // in case of an empty stream.
);
another thing is concatMap it requires an observable to complete, only then it switches to the next one, parhaps mergeMap or switchMap fits here better.
Try the code below and add its output to the comments. Thanks.
const activityCreator: Observable<UserProfileModel> = this.getCreator(activity.creator).pipe(
tap(
() => console.log('getCreator:emit'),
() => console.log('getCreator:error'),
() => console.log('getCreator:completed'),
),
);

In rxjs, how do I chain mapping through arrays of data received from different API's?

I'm calling an API and receiving an array of results, I'm checking for pagination and if more pages exist I call the next page, repeat until no more pages.
For each array of results, I call another endpoint and do the exact same thing: I receive an array of results, check for another page and call endpoint again. Wash, rinse repeat.
For instance:
I want to grab a list of countries that might be a paginated response, then for each country I want to grab a list of cities, which might also be paginated. And for each city I execute a set of transformations and then store in a database.
I already tried this, but got stuck:
const grabCountries = Observable.create(async (observer) => {
const url = 'http://api.com/countries'
let cursor = url
do {
const results = fetch(cursor)
// results = {
// data: [ 'Canada', 'France', 'Spain' ],
// next: '47asd8f76358df8f4058898fd8fab'
// }
results.data.forEach(country => { observer.next(country) })
cursor = results.next ? `${url}/${results.next}` : undefined
} while(cursor)
})
const getCities = {
next: (country) => {
const url = 'http://api.com/cities'
let cursor = url
do {
const results = fetch(cursor)
// results = {
// data: [
// 'Montreal', 'Toronto',
// 'Paris', 'Marseilles',
// 'Barcelona', 'Madrid'
// ],
// next: '89ghjg98nd8g8sdfg98gs9h868hfoig'
// }
results.data.forEach(city => {
`**** What do I do here?? ****`
})
cursor = results.next ? `${url}/${results.next}` : undefined
} while(cursor)
}
}
I tried a few approaches:
Making a subject (sometimes I'll need to do parallel processed base on the results of 'grabCountries'. For example I may want to store the countries in a DB in parallel with grabbing the Cities.)
const intermediateSubject = new Subject()
intermediateSubject.subscribe(storeCountriesInDatabase)
intermediateSubject.subscribe(getCities)
I also tried piping and mapping, but it seems like it's basically the same thing.
As I was writing this I thought of this solution and it seems to be working fine, I would just like to know if I'm making this too complicated. There might be cases where I need to make more that just a few API calls in a row. (Imagine, Countries => States => Cities => Bakeries => Reviews => Comments => Replies) So this weird mapping over another observer callback pattern might get nasty.
So this is what I have now basically:
// grabCountries stays the same as above, but the rest is as follows:
const grabCities = (country) =>
Observable.create(async (observer) => {
const url = `http://api.com/${country}/cities`
let cursor = url
do {
const results = fetch(cursor)
// results = {
// data: [
// 'Montreal', 'Toronto',
// 'Paris', 'Marseilles',
// 'Barcelona', 'Madrid'
// ],
// next: '89ghjg98nd8g8sdfg98gs9h868hfoig'
// }
results.data.forEach(city => {
observer.next(city)
})
cursor = results.next ? `${url}/${results.next}` : undefined
} while (cursor)
})
const multiCaster = new Subject()
grabCountries.subscribe(multiCaster)
multiCaster.pipe(map((country) => {
grabCities(country).pipe(map(saveCityToDB)).subscribe()
})).subscribe()
multiCaster.pipe(map(saveCountryToDB)).subscribe()
tl;dr - I call an API that receives a paginated set of results in an array and I need to map through each item and call another api that receives another paginated set of results, each set also in an array.
Is nesting one observable inside another and mapping through the results via 'callApiForCountries.pipe(map(forEachCountryCallApiForCities))' the best method or do you have any other recommendations?

Here's the code that should work with sequential crawling of next url.
You start with a {next:url} until res.next is not available.
of({next:http://api.com/cities}).pipe(
expand(res=>results.next ? `${url}/${results.next}` : undefined
takeWhile(res=>res.next!==undefined)
).subscribe()

OK, so I have spent a lot of brain power on this and have come up with two solutions that seem to be working.
const nestedFlow = () => {
fetchAccountIDs.pipe(map(accountIds => {
getAccountPostIDs(accountIds) // Has the do loop for paging inside
.pipe(
map(fetchPostDetails),
map(mapToDBFormat),
map(storeInDB)
).subscribe()
})).subscribe()
}
const expandedflow = () => {
fetchAccountIDs.subscribe((accountId) => {
// accountId { accountId: '345367geg55sy'}
getAccountPostIDs(accountId).pipe(
expand((results) => {
/*
results : {
postIDs: [
131424234,
247345345,
],
cursor: '374fg8v0ggfgt94',
}
*/
const { postIDs, cursor } = results
if (cursor) return getAccountPostIDs({...accountId, cursor})
return { postIDs, cursor }
}),
takeWhile(hasCursor, true), // recurs until cursor is undefined
concatMap(data => data.postIDs),
map(data => ({ post_id: data })),
map(fetchPostDetails),
map(mapToDBFormat),
map(storeInDB)
).subscribe()
})
}
Both seem to be working with similar performance. I read some where that leaving the data flow is a bad practice and you should pipe everything, but I don't know how to eliminate the first exit in the 'expandedFlow' because the 'expand' needs to call back an observable, but maybe it can be done.
Now I just have to solve the race condition issues from the time the 'complete' is called in getAccountPostIDs the the last record is stored in the DB. Currently in my test, the observer.complete is finishing before 3 of the upsert actions.
Any comments are appreciated and I hope this helps someone out in the future.

What you need is the expand operator. It behaves recursively so it fits the idea of having paginated results.

Difference between two observables

Let's say I have two observables.
The first observable is an array of certain listings:
[
{id: 'zzz', other props here...},
{id: 'aaa', ...},
{id: '007', ...}
... and more over time
]
The second observable is an array of ignored listings:
[
{id: '007'}, // only id, no other props
{id: 'zzz'}
... and more over time
]
The result should be a new observable of listings (first observable) but must not have any of the ignored listings:
[
{id: 'aaa', other props here...}
... and more over time
]
This is what I have now before posting:
obs2.pipe(withLatestFrom(obs1, ? => ?, filter(?));

I didn't test it out, but I think it should be ok:
combineLatest(values$, excluded$).pipe(
map(([values, excluded]) => {
// put all the excluded IDs into a map for better perfs
const excludedIds: Map<string, undefined> = excluded.reduce(
(acc: Map<string, undefined>, item) => {
acc.set(item.id, undefined)
return acc;
},
new Map()
);
// filter the array, by looking up if the current
// item.id is in the excluded list or not
return values.filter(item => !excludedIds.has(item.id))
})
)
Explanation:
Using combineLatest you'll always be warned no matter where you get the update from. If you use withLatestFrom as in your example, it'll trigger an update only if the values$ observable is updated. But if the excluded$ changes, it wouldn't trigger an update in your case.
Then get all the excluded IDs into a map instead of an array as we'll need to know whether a given ID should be excluded or not. Looking into a map is wayyyyy faster than looking into an array.
Then just filter the values array.

If I'm understanding correctly, what you'll want to do is
Aggregate the incoming items over time
Aggregate the ids that are to be ignored over time
Finally, as both of the above streams emit over time, emit a resulting list of items that don't include the ignored ids.
Given the above, below is a rough example you could try. As noted towards the bottom, you'll get different results depending on the cadence of the first two streams because, well, thats's what happens with async. To show that, I'm simulating a random delay in the emission of things over time.
Hope this helps!
P.S.: The below is Typescript, assuming rxjs#^6.
import { BehaviorSubject, combineLatest, of, Observable } from "rxjs";
import { delay, map, scan, concatMap } from "rxjs/operators";
/**
* Data sources
*/
// Just for showcase purposes... Simulates items emitted over time
const simulatedEmitOverTime = <T>() => (source: Observable<T>) =>
source.pipe(
concatMap(thing => of(thing).pipe(delay(Math.random() * 1000)))
);
interface Thing {
id: string;
}
// Stream of things over time
const thingsOverTime$ = of(
{ id: "zzz" },
{ id: "aaa" },
{ id: "007" }
).pipe(
simulatedEmitOverTime()
);
// Stream of ignored things over time
const ignoredThingsOverTime$ = of(
{ id: "007" },
{ id: "zzz" }
).pipe(
simulatedEmitOverTime()
);
/**
* Somewhere in your app
*/
// Aggregate incoming things
// `scan` takes a reducer-type function
const aggregatedThings$ = thingsOverTime$.pipe(
scan(
(aggregatedThings: Thing[], incomingThing: Thing) =>
aggregatedThings.concat(incomingThing),
[]
)
);
// Create a Set from incoming ignored thing ids
// A Set will allow for easy filtering over time
const ignoredIds$ = ignoredThingsOverTime$.pipe(
scan(
(excludedIdSet, incomingThing: Thing) =>
excludedIdSet.add(incomingThing.id),
new Set<string>()
)
);
// Combine stream and then filter out ignored ids
const sanitizedThings$ = combineLatest(aggregatedThings$, ignoredIds$)
.pipe(
map(([things, ignored]) => things.filter(({ id }) => !ignored.has(id)))
);
// Subscribe where needed
// Note: End result will vary depending on the timing of items coming in
// over time (which is being simulated here-ish)
sanitizedThings$.subscribe(console.log);