Develop Reference

Add streams dynamically to combined stream (eg forkJoin)

My function (lets call it myFunction) is getting an array of streams (myFunction(streams: Observable<number>[])). Each of those streams produces values from 1 to 100, which acts as a progress indicator. When it hits 100 it is done and completed. Now, when all of those observables are done I want to emit a value. I could do it this way:
public myFunction(streams: Observable<number>[]) {
forkJoin(streams).subscribe(_values => this.done$.emit());
}
This works fine, but imagine following case:
myFunction gets called with 2 streams
one of those streams is done, second one is still progressing
myFunction gets called (again) with 3 more streams (2nd one from previous call is still progressing)
I'd like to somehow add those new streams from 3rd bullet to the "queue", which would result in having 5 streams in forkJoin (1 completed, 4 progressing).
I've tried multiple approaches but can't get it working anyhow... My latest approach was this:
private currentProgressObs: Observable<any> | null = null;
private currentProgressSub: Subscription | null = null;
public myFunction(progressStreams: Observable<number>[]) {
const isUploading = this.cumulativeUploadProgressSub && !this.cumulativeUploadProgressSub.closed;
const currentConcatObs = this.currentProgressObs?.pipe(concatAll());
const currentStream = isUploading && this.currentProgressObs ? this.currentProgressObs : of([100]);
if (this.currentProgressSub) {
this.currentProgressSub.unsubscribe();
this.currentProgressSub = null;
}
this.currentProgressObs = forkJoin([currentStream, ...progressStreams]);
this.currentProgressSub = this.currentProgressObs.subscribe(
_lastProgresses => {
this._isUploading$.next(false); // <----- this is the event I want to emit when all progress is completed
this.currentProgressSub?.unsubscribe();
this.currentProgressSub = null;
this.currentProgressObs = null;
},
);
}
Above code only works for the first time. Second call to the myFunction will never emit the event.
I also tried other ways. I've tried recursion with one global stream array, in which I can add streams while the subscription is still avctive but... I failed. How can I achieve this? Which operator and in what oreder should I use? Why it will or won't work?

Here is my suggestion for your issue.
We will have two subjects, one to count the number of request being processed (requestsInProgress) and one more to mange the requests that are being processed (requestMerger)
So the thing that will do is whenever we want to add new request we will pass it to the requestMerger Subject.
Whenever we receive new request for processing in the requestMerger stream we will first increment the requestInProgress counter and after that we will merge the request itself in the source observable. While merging the new request/observable to the source we will also add the finalize operator in order to track when the request has been completed (reached 100), and when we hit the completion criteria we will decrement the request counter with the decrementCounter function.
In order to emit result e.g. to notify someone else in the app for the state of the pending requests we can subscribe to the requestsInProgress Subject.
You can test it out either here or in this stackBlitz
let {
interval,
Subject,
BehaviorSubject
} = rxjs
let {
mergeMap,
map,
takeWhile,
finalize,
first,
distinctUntilChanged
} = rxjs.operators
// Imagine next lines as a service
// Subject responsible for managing strems
let requestMerger = new Subject();
// Subject responsible for tracking streams in progress
let requestsInProgress = new BehaviorSubject(0);
function incrementCounter() {
requestsInProgress.pipe(first()).subscribe(x => {
requestsInProgress.next(x + 1);
});
}
function decrementCounter() {
requestsInProgress.pipe(first()).subscribe(x => {
requestsInProgress.next(x - 1);
});
}
// Adds request to the request being processed
function addRequest(req) {
// The take while is used to complete the request when we have `value === 100` , if you are dealing with http-request `takeWhile` might be redudant, because http request complete by themseves (e.g. the finalize method of the stream will be called even without the `takeWhile` which will decrement the requestInProgress counter)
requestMerger.next(req.pipe(takeWhile(x => x < 100)));
}
// By subscribing to this stream you can determine if all request are processed or if there are any still pending
requestsInProgress
.pipe(
map(x => (x === 0 ? "Loaded" : "Loading")),
distinctUntilChanged()
)
.subscribe(x => {
console.log(x);
document.getElementById("loadingState").innerHTML = x;
});
// This Subject is taking care to store or request that are in progress
requestMerger
.pipe(
mergeMap(x => {
// when new request is added (recieved from the requestMerger Subject) increment the requrest being processed counter
incrementCounter();
return x.pipe(
finalize(() => {
// when new request has been completed decrement the requrest being processed counter
decrementCounter();
})
);
})
)
.subscribe(x => {
console.log(x);
});
// End of fictional service
// Button that adds request to be processed
document.getElementById("add-stream").addEventListener("click", () => {
addRequest(interval(1000).pipe(map(x => x * 25)));
});
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/6.6.6/rxjs.umd.min.js"></script>
<div style="display:flex">
<button id="add-stream">Add stream</button>
<h5>Loading State: <span id="loadingState">false</span> </h5>
</div>

Your problem is that each time your call your function, you are creating a new observable. Your life would be much easier if all calls of your function pushed all upload jobs through the same stream.
You can achieve this using a Subject.
I would suggest you push single "Upload Jobs" though a simple subject and design an observable that emits the state of all upload jobs whenever anything changes: A simple class that offers a createJob() method to submit jobs, and a jobs$ observable to reference the state:
class UploadService {
private jobs = new Subject<UploadJob>();
public jobs$ = this.jobs.pipe(
mergeMap(job => this.processJob(job)),
scan((collection, job) => collection.set(job.id, job), new Map<string, UploadJob>()),
map(jobsMap => Array.from(jobsMap.values()))
);
constructor() {
this.jobs$.subscribe();
}
public createJob(id: string) {
this.jobs.next({ id, progress: 0 });
}
private processJob(job: UploadJob) {
// do work and return observable that
// emits updated status of UploadJob
}
}
Let's break it down:
jobs is a simple subject, that we can push "jobs" through
createJob simply calls jobs.next() to push the new job through the stream
jobs$ is where all the magic happens. It receives each UploadJob and uses:
mergeMap to execute whatever function actually does the work (I called it processJob() for this example) and emits its values into the stream
scan is used to accumulate these UploadJob emissions into a Map (for ease of inserting or updating)
map is used to convert the map into an array (Map<string, UploadJob> => UploadJob[])
this.jobs$.subscribe() is called in the constructor of the class so that jobs will be processed
Now, we can easily derive your isUploading and cumulativeProgress from this jobs$ observable like so:
public isUploading$ = this.jobs$.pipe(
map(jobs => jobs.some(j => j.progress !== 100)),
distinctUntilChanged()
);
public progress$ = this.jobs$.pipe(
map(jobs => {
const current = jobs.reduce((sum, j) => sum + j.progress, 0) / 100;
const total = jobs.length ?? current;
return current / total;
})
);
Here's a working StackBlitz demo.

RxJS operator that waits for quiet period in event stream, but in case of busy event stream does not wait for ever

The scenario:
I have a stream of events, each event should result in the updated display of information (event stream is from websockets and the display is in a highcharts chart, but that is not important)
For performance reasons, I do not want to trigger a UI update for each event.
I would rather do the following:
When I receive an event I want to do the UI update only it is more than X milliseconds since the last update
But every Y milliseconds (Y > X) I want to do an update anyway, if there have been any incoming events
So I am looking for some kind of (combination of) RxJS operator that will rate-limit the event stream to emit an event only when a quiet period occurs (or the wait-for-quiet-period-maximum-time has been exceeded).
I.e. I want to wait for quiet periods, but not for ever.
How can I implement what I describe above?
I have looked at:
https://rxjs-dev.firebaseapp.com/api/operators/sampleTime
https://rxjs-dev.firebaseapp.com/api/operators/debounceTime
... and some other rxjs time/rate-limiting operators

You can write an operator to do what you want by using debounce and employing two timers in the composition of the notifier observable:
a timer that emits X milliseconds after the source emits a value; and
a timer that emits Y milliseconds after the observable returned by the operator emits a value.
See the snippet below. Comments within should explain how it works.
const {
ConnectableObservable,
merge,
MonoTypeOperatorFunction,
Observable,
of,
Subject,
Subscription,
timer
} = rxjs;
const {
concatMap,
debounce,
mapTo,
publish,
startWith,
switchMap
} = rxjs.operators;
// The pipeable operator:
function waitUntilQuietButNotTooLong(
quietDuration,
tooLongDuration
) {
return source => new Observable(observer => {
let tooLongTimer;
// Debounce the source using a notifier that emits after `quietDuration`
// milliseconds since the last source emission or `tooLongDuration`
// milliseconds since the observable returned by the operator last
// emitted.
const debounced = source.pipe(
debounce(() => merge(
timer(quietDuration),
tooLongTimer
))
);
// Each time the source emits, `debounce` will subscribe to the notifier.
// Use `publish` to create a `ConnectableObservable` so that the too-long
// timer will continue independently of the subscription from `debounce`
// implementation.
tooLongTimer = debounced.pipe(
startWith(undefined),
switchMap(() => timer(tooLongDuration)),
publish()
);
// Connect the `tooLongTimer` observable and subscribe the observer to
// the `debounced` observable. Compose a subscription so that
// unsubscribing from the observable returned by the operator will
// disconnect from `tooLongTimer` and unsubscribe from `debounced`.
const subscription = new Subscription();
subscription.add(tooLongTimer.connect());
subscription.add(debounced.subscribe(observer));
return subscription;
});
}
// For a harness, create a subject and apply the operator:
const since = Date.now();
const source = new Subject();
source.pipe(
waitUntilQuietButNotTooLong(100, 500)
).subscribe(value => console.log(`received ${value} # ${Date.now() - since} ms`));
// And create an observable that emits at a particular time and subscribe
// the subject to it:
const emissions = of(0, 50, 100, 300, 350, 400, 450, 500, 550, 600, 650, 700, 750, 800, 850, 900, 950, 1000, 1050, 1100, 1150);
emissions.pipe(
concatMap((value, index) => timer(new Date(since + value)).pipe(
mapTo(index)
))
).subscribe(source);
.as-console-wrapper { max-height: 100% !important; top: 0; }
<script src="https://unpkg.com/rxjs#6/bundles/rxjs.umd.min.js"></script>

You can combine a timer with a debounceTime and use it to sample the original event stream:
let $tick = Rx.Observable.timer(100, 100);
let $updates = $events
.sample($tick.merge($events.debounceTime(30))
.distinctUntilChanged();
This will take an event every 100 milliseconds, but also if an event occurs before a 30 millisecond gap.
With sample, the values in the sampling stream are ignored. So this technique creates a sampling stream that includes both the time-based requirement and also the debouncing. Whenever either of these happens, it will take the latest value from the original stream.
Using distinctUntilChanged prevents events repeating consecutively with the same value if nothing changed. You may need to add a comparison function as an argument to distinctUntilChanged if your data is structured or otherwise can't be compared with ===.

MobX action schedule / execution order isn't preserved due to race condition

Here's a running example of what I've got so far:
https://codesandbox.io/s/github/BruceL33t/mobx-action-synchronous-execution-order/tree/master/
store.js:
import { observable, action } from "mobx";
import Sensor from "../models/Sensor";
export default class RootStore {
#observable sensors = new Map();
constructor() {
let self = this;
const sensorIds = [
"sensor1",
"sensor2",
"sensor3",
"sensor4",
"sensor5",
"sensor6",
"sensor7",
"sensor8",
"sensor9",
"sensor10"
];
for (let sensor of sensorIds) {
self.sensors.set(sensor, new Sensor(5));
}
// setInterval simulates some incoming data (originally from SignalR, and roughly each second)
setInterval(function() {
let out = {};
const x = +new Date(); // unix timestamp
for (let sensor of sensorIds) {
const y = Math.floor(Math.random() * 10000) + 1;
const m = { x: x, y: y };
out[sensor] = m;
}
self.addMeasurement(out); // the problem starts here.
}, 1000);
}
// the problem!
#action
addMeasurement(sensorMeasurementMap) {
let self = this;
// this timeout is to try and simulate a race condition
// since each measurement is incoming each second,
// here some of them will take as long as 6 seconds to add,
// due to the timeout.
// the point is that they should always be added,
// in the order they were called in.
// so if the first measurement takes 20 seconds to be added,
// the next measurements that were received on 2, 3, 4, 5..., 19th second etc,
// should all "wait" for the prev measurement, so they're added
// in the right order (order can be checked by timestamp, x)
setTimeout(() => {
const keys = self.sensors.keys();
if (keys.length === 0) {
// never really gonna happen, since we already set them above
} else {
for (const key in sensorMeasurementMap) {
if (self.sensors.keys().indexOf(key) > -1) {
self.sensors.get(key).add(sensorMeasurementMap[key]);
} else {
// also not gonna happen in this example
}
}
}
}, Math.floor(Math.random() * 20 + 1) * 1000);
}
}
Sensor.js:
import Queue from './Queue';
import {observable, action} from 'mobx';
export default class Sensor {
#observable queue;
constructor(n) {
this.n = n;
this.queue = new Queue(this.n);
}
#action add(measurement) {
this.queue.add(measurement);
}
}
Queue.js:
import {observable, action} from 'mobx';
export default class Queue {
#observable data;
constructor(maxSize) {
this.maxSize = maxSize;
this.size = 0;
this.data = [];
}
#action add(measurement) {
let removedItem = undefined;
if(this.size >= this.maxSize) {
let temp = this.data[0];
removedItem = temp && temp.y ? temp.y+'' : undefined;
this.data.shift();
}
this.data.push(measurement);
if (removedItem === undefined && this.size < this.maxSize) {
this.size++;
}
return removedItem;
}
}
There's a few comments in the code but you absolutely need to see the output https://codesandbox.io/s/github/BruceL33t/mobx-action-synchronous-execution-order/tree/master/ to understand it.
Let me also try to explain it here, what this is all about.
This is basically a overly simplified version of a part of a real application, where setInterval is just used instead to simulate a SignalR event handler to indicate incoming data each second. The incoming data is what we create inside the setInterval func above the addMeasurement action.
So given some incoming data is received each second, we want to add this to the observable map sensors on the store. Since this data is used for drawing charts in the real application, we need to make sure it is indeed added in the order which the actions are invoked in - no matter how long the action takes to complete.
In the real application I saw some inconsistency in the order of how the data were pushed to the MobX state, so I isolated it and extracted the relevant parts into this example and tried to exaggerate it a bit by using the setTimeout func inside the addMeasurement action.
Since each data is fetched each second, but some measurements could take up to 20 seconds to fetch (not realisticly, but to clearly show race condition problem), as the code is right now, it often happens that we end up with something like:
[
{"x":1519637083193,"y":4411},
{"x":1519637080192,"y":7562},
{"x":1519637084193,"y":1269},
{"x":1519637085192,"y":8916},
{"x":1519637081192,"y":7365}
]
Which really should never happen, since 1519637083193 is greater/later than 1519637080192.
This is a real problem when drawing charts from this data and ordering it afterwards is way too expensive, so I'm looking for a way to improve this code so we can trust each addMeasurement is only fired once the previous action has completely finished. Or at least a way to update the MobX state in the right order
Hope it makes sense.

should all "wait" for the prev measurement, so they're added in the right order (order can be checked by timestamp, x).
Could you elaborate on that? How could one ever know that no timestamp larger than the current one will be received in the future, and hence wait indefinitely? Isn't what you are looking for just a sorted insertion to the array of measurements (instead of waiting)?
If sorted insertion doesn't solve the problem, I would probably do the following (untested):
lastAddition = Promise.resolve() // start with already finishied addition
addMeasurement(sensorMeasurementMap) {
this.lastAddition = this.lastAddition.then(() => {
return new Promise((resolve, reject) => {
setTimeout(action(() => {
const keys = self.sensors.keys();
if (keys.length === 0) {
// never really gonna happen, since we already set them above
} else {
for (const key in sensorMeasurementMap) {
if (self.sensors.keys().indexOf(key) > -1) {
self.sensors.get(key).add(sensorMeasurementMap[key]);
} else {
// also not gonna happen in this example
}
}
}
resolve()
}), Math.floor(Math.random() * 20 + 1) * 1000);
})
})
}
}
N.B.: Note that I moved action inside, as you need it at the place where you are actually modifying the state, not where the scheduling happens

ReactiveX filtering on observable multiple times and merging

I have a problem creating the following observable.
I want it to receive a predefined array of values
And I want to filter by different things, and be able to work with these as individual observables.
And then when it comes time to merge these filtered observables, I want to preserve the order from the original one
//Not sure the share is necessary, just thought it would tie it all together
const input$ = Observable.from([0,1,0,1]).share();
const ones$ = input$.filter(n => n == 1);
const zeroes$ = input$.filter(n => n == 0);
const zeroesChanged$ = zeroes$.mapTo(2);
const onesChanged$ = ones$.mapTo(3);
const allValues$ = Observable.merge(onesChanged$,zeroesChanged$);
allValues$.subscribe(n => console.log(n));
//Outputs 3,3,2,2
//Expected output 3,2,3,2
EDIT: I am sorry I was not specific enough in my question.
I am using a library called cycleJS, which separates sideeffects into drivers.
So what I am doing in my cycle is this
export function socketCycle({ SOCKETIO }) {
const serverConnect$ = SOCKETIO.get('connect').map(serverDidConnect);
const serverDisconnect$ = SOCKETIO.get('disconnect').map(serverDidDisconnect);
const serverFailedToConnect$ = SOCKETIO.get('connect_failed').map(serverFailedToConnect);
return { ACTION: Observable.merge(serverConnect$, serverDisconnect$, serverFailedToConnect$) };
}
Now my problem arose when I wanted to write a test for it. I tried with the following which worked in the wrong matter(using jest)
const inputConnect$ = Observable.from(['connect', 'disconnect', 'connect', 'disconnect']).share();
const expectedOutput$ = Observable.from([
serverDidConnect(),
serverDidDisconnect(),
serverDidConnect(),
serverDidDisconnect(),
]);
const socketIOMock = {
get: (evt) => {
if (evt === 'connect') {
return inputConnect$.filter(s => s === 'connect');
} else if (evt === 'disconnect') {
return inputConnect$.filter(s => s === 'disconnect');
}
return Observable.empty();
},
};
const { ACTION } = socketCycle({ SOCKETIO: socketIOMock });
Observable.zip(ACTION, expectedOutput$).subscribe(
([output, expectedOutput]) => { expect(output).toEqual(expectedOutput); },
(error) => { expect(true).toBe(false) },
() => { done(); },
);
Maybe there is another way I can go about testing it?

When stream is partitioned, the timing guarantees between elements in different daughter streams is actually destroyed. In particular, even if connect events always come before disconnect events at the event source, the events of the connect Observable won't always come before their corresponding events items in the disconnect Observable. At normal timescales, this race condition probably quite rare but dangerous nonetheless, and this test shows the worst case.
The good news is that your function as shown is just a mapper, between events and results from handlers. If you can continue this model generally over event types, then you can even encode the mapping in a plain data structure, which benefits expressiveness:
const event_handlers = new Map({
'connect': serverDidConnect,
'disconnect': serverDidDisconnect,
'connect_failed': serverFailedToConnect
});
const ACTION = input$.map(event_handlers.get.bind(event_handlers));
Caveat: if you were reducing over the daughter streams (or otherwise considering previous values, like with debounceTime), the refactor is not so straightforward, and would also depend on a new definition of "preserve order". Much of the time, it would still be feasible to reproduce with reduce + a more complicated accumulator.

Below code might be able to give you the desire result, but it's no need to use rxjs to operate array IMHO
Rx.Observable.combineLatest(
Rx.Observable.from([0,0,0]),
Rx.Observable.from([1,1,1])
).flatMap(value=>Rx.Observable.from(value))
.subscribe(console.log)

RXJS Observable stretch

I have a Rx.Observable.webSocket Subject. My server endpoint can not handle messages receiving the same time (<25ms). Now I need a way to stretch the next() calls of my websocket subject.
I have created another Subject requestSubject and subscribe to this.
Then calling next of the websocket inside the subscription.
requestSubject.delay(1000).subscribe((request) => {
console.log(`SENDING: ${JSON.stringify(request)}`);
socketServer.next(JSON.stringify(request));
});
Using delay shifts each next call the same delay time, then all next calls emit the same time later ... thats not what I want.
I tried delay, throttle, debounce but it does not fit.
The following should illustrate my problem
Stream 1 | ---1-------2-3-4-5---------6----
after some operation ...
Stream 2 | ---1-------2----3----4----5----6-

Had to tinker a bit, its not as easy as it looks:
//example source stream
const source = Rx.Observable.from([100,500,1500,1501,1502,1503])
.mergeMap(i => Rx.Observable.of(i).delay(i))
.share();
stretchEmissions(source, 1000)
.subscribe(val => console.log(val));
function stretchEmissions(source, spacingDelayMs) {
return source
.timestamp()
.scan((acc, curr) => {
// calculate delay needed to offset next emission
let delay = 0;
if (acc !== null) {
const timeDelta = curr.timestamp - acc.timestamp;
delay = timeDelta > spacingDelayMs ? 0 : (spacingDelayMs - timeDelta);
}
return {
timestamp: curr.timestamp,
delay: delay,
value: curr.value
};
}, null)
.mergeMap(i => Rx.Observable.of(i.value).delay(i.delay), undefined, 1);
}
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/5.4.2/Rx.js"></script>
Basically we need to calculate the needed delay between emissions so we can space them. We do this using timestamp() of original emissions and the mergeMap overload with a concurrency of 1 to only subscribe to the next delayed value when the previous is emitted. This is a pure Rx solution without further side effects.

Here are two solutions using a custom stream and using only rxjs-operators - since it looks quite complicated I would not advice you to use this solution, but to use a custom stream (see 1st example below):
Custom stream (MUCH easier to read and maintain, probably with better performance as well):
const click$ = Rx.Observable
.fromEvent(document.getElementById("btn"), "click")
.map((click, i) => i);
const spreadDelay = 1000;
let prevEmitTime = 0;
click$
.concatMap(i => { // in this case you could also use "flatMap" or "mergeMap" instead of "concatMap"
const now = Date.now();
if (now - prevEmitTime > spreadDelay) {
prevEmitTime = now;
return Rx.Observable.of(i); // emit immediately
} else {
prevEmitTime += spreadDelay;
return Rx.Observable.of(i).delay(prevEmitTime - now); // emit somewhere in the future
}
})
.subscribe((request) => {
console.log(`SENDING: ${request}`);
});
<script src="https://unpkg.com/rxjs/bundles/Rx.min.js"></script>
<button id="btn">Click me!</button>
Using only RxJS Operators (contains issues, probably shouldn't use):
const click$ = Rx.Observable
.fromEvent(document.getElementById("btn"), "click")
.map((click, i) => i);
click$
// window will create a new substream whenever no click happened for 1001ms (with the spread out
.window(click$
.concatMap(i => Rx.Observable.of(i).delay(1000))
.debounceTime(1001)
)
.mergeMap(win$ => Rx.Observable.merge(
win$.take(1).merge(), // emitting the "first" click immediately
win$.skip(1)
.merge()
.concatMap(i => Rx.Observable.of(i).delay(1000)) // each emission after the "first" one will be spread out to 1 seconds
))
.subscribe((request) => {
console.log(`SENDING: ${request}`);
});
<script src="https://unpkg.com/rxjs/bundles/Rx.min.js"></script>
<button id="btn">Click me!</button>

Mark van Straten's solution didn't work completely accurately for me. I found a much more simple and accurate solution based from here.
const source = from([100,500,1500,1501,1502,1503]).pipe(
mergeMap(i => of(i).pipe(delay(i)))
);
const delayMs = 500;
const stretchedSource = source.pipe(
concatMap(e => concat(of(e), EMPTY.pipe(delay(delayMs))))
);