Develop Reference

Add streams dynamically to combined stream (eg forkJoin)

My function (lets call it myFunction) is getting an array of streams (myFunction(streams: Observable<number>[])). Each of those streams produces values from 1 to 100, which acts as a progress indicator. When it hits 100 it is done and completed. Now, when all of those observables are done I want to emit a value. I could do it this way:
public myFunction(streams: Observable<number>[]) {
forkJoin(streams).subscribe(_values => this.done$.emit());
}
This works fine, but imagine following case:
myFunction gets called with 2 streams
one of those streams is done, second one is still progressing
myFunction gets called (again) with 3 more streams (2nd one from previous call is still progressing)
I'd like to somehow add those new streams from 3rd bullet to the "queue", which would result in having 5 streams in forkJoin (1 completed, 4 progressing).
I've tried multiple approaches but can't get it working anyhow... My latest approach was this:
private currentProgressObs: Observable<any> | null = null;
private currentProgressSub: Subscription | null = null;
public myFunction(progressStreams: Observable<number>[]) {
const isUploading = this.cumulativeUploadProgressSub && !this.cumulativeUploadProgressSub.closed;
const currentConcatObs = this.currentProgressObs?.pipe(concatAll());
const currentStream = isUploading && this.currentProgressObs ? this.currentProgressObs : of([100]);
if (this.currentProgressSub) {
this.currentProgressSub.unsubscribe();
this.currentProgressSub = null;
}
this.currentProgressObs = forkJoin([currentStream, ...progressStreams]);
this.currentProgressSub = this.currentProgressObs.subscribe(
_lastProgresses => {
this._isUploading$.next(false); // <----- this is the event I want to emit when all progress is completed
this.currentProgressSub?.unsubscribe();
this.currentProgressSub = null;
this.currentProgressObs = null;
},
);
}
Above code only works for the first time. Second call to the myFunction will never emit the event.
I also tried other ways. I've tried recursion with one global stream array, in which I can add streams while the subscription is still avctive but... I failed. How can I achieve this? Which operator and in what oreder should I use? Why it will or won't work?

Here is my suggestion for your issue.
We will have two subjects, one to count the number of request being processed (requestsInProgress) and one more to mange the requests that are being processed (requestMerger)
So the thing that will do is whenever we want to add new request we will pass it to the requestMerger Subject.
Whenever we receive new request for processing in the requestMerger stream we will first increment the requestInProgress counter and after that we will merge the request itself in the source observable. While merging the new request/observable to the source we will also add the finalize operator in order to track when the request has been completed (reached 100), and when we hit the completion criteria we will decrement the request counter with the decrementCounter function.
In order to emit result e.g. to notify someone else in the app for the state of the pending requests we can subscribe to the requestsInProgress Subject.
You can test it out either here or in this stackBlitz
let {
interval,
Subject,
BehaviorSubject
} = rxjs
let {
mergeMap,
map,
takeWhile,
finalize,
first,
distinctUntilChanged
} = rxjs.operators
// Imagine next lines as a service
// Subject responsible for managing strems
let requestMerger = new Subject();
// Subject responsible for tracking streams in progress
let requestsInProgress = new BehaviorSubject(0);
function incrementCounter() {
requestsInProgress.pipe(first()).subscribe(x => {
requestsInProgress.next(x + 1);
});
}
function decrementCounter() {
requestsInProgress.pipe(first()).subscribe(x => {
requestsInProgress.next(x - 1);
});
}
// Adds request to the request being processed
function addRequest(req) {
// The take while is used to complete the request when we have `value === 100` , if you are dealing with http-request `takeWhile` might be redudant, because http request complete by themseves (e.g. the finalize method of the stream will be called even without the `takeWhile` which will decrement the requestInProgress counter)
requestMerger.next(req.pipe(takeWhile(x => x < 100)));
}
// By subscribing to this stream you can determine if all request are processed or if there are any still pending
requestsInProgress
.pipe(
map(x => (x === 0 ? "Loaded" : "Loading")),
distinctUntilChanged()
)
.subscribe(x => {
console.log(x);
document.getElementById("loadingState").innerHTML = x;
});
// This Subject is taking care to store or request that are in progress
requestMerger
.pipe(
mergeMap(x => {
// when new request is added (recieved from the requestMerger Subject) increment the requrest being processed counter
incrementCounter();
return x.pipe(
finalize(() => {
// when new request has been completed decrement the requrest being processed counter
decrementCounter();
})
);
})
)
.subscribe(x => {
console.log(x);
});
// End of fictional service
// Button that adds request to be processed
document.getElementById("add-stream").addEventListener("click", () => {
addRequest(interval(1000).pipe(map(x => x * 25)));
});
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/6.6.6/rxjs.umd.min.js"></script>
<div style="display:flex">
<button id="add-stream">Add stream</button>
<h5>Loading State: <span id="loadingState">false</span> </h5>
</div>

Your problem is that each time your call your function, you are creating a new observable. Your life would be much easier if all calls of your function pushed all upload jobs through the same stream.
You can achieve this using a Subject.
I would suggest you push single "Upload Jobs" though a simple subject and design an observable that emits the state of all upload jobs whenever anything changes: A simple class that offers a createJob() method to submit jobs, and a jobs$ observable to reference the state:
class UploadService {
private jobs = new Subject<UploadJob>();
public jobs$ = this.jobs.pipe(
mergeMap(job => this.processJob(job)),
scan((collection, job) => collection.set(job.id, job), new Map<string, UploadJob>()),
map(jobsMap => Array.from(jobsMap.values()))
);
constructor() {
this.jobs$.subscribe();
}
public createJob(id: string) {
this.jobs.next({ id, progress: 0 });
}
private processJob(job: UploadJob) {
// do work and return observable that
// emits updated status of UploadJob
}
}
Let's break it down:
jobs is a simple subject, that we can push "jobs" through
createJob simply calls jobs.next() to push the new job through the stream
jobs$ is where all the magic happens. It receives each UploadJob and uses:
mergeMap to execute whatever function actually does the work (I called it processJob() for this example) and emits its values into the stream
scan is used to accumulate these UploadJob emissions into a Map (for ease of inserting or updating)
map is used to convert the map into an array (Map<string, UploadJob> => UploadJob[])
this.jobs$.subscribe() is called in the constructor of the class so that jobs will be processed
Now, we can easily derive your isUploading and cumulativeProgress from this jobs$ observable like so:
public isUploading$ = this.jobs$.pipe(
map(jobs => jobs.some(j => j.progress !== 100)),
distinctUntilChanged()
);
public progress$ = this.jobs$.pipe(
map(jobs => {
const current = jobs.reduce((sum, j) => sum + j.progress, 0) / 100;
const total = jobs.length ?? current;
return current / total;
})
);
Here's a working StackBlitz demo.

Resubscribing happens only after triggering the source observable the second time

I'm trying to compose a stream with a send event and an undo event. after sending the message, there is a 3s delay while you can undo the message and return the sent message into the text field. if you started to compose a new message, the sent message should be prepended.
so far I've managed to create the delayed send and undo functionality. the problem occurs, when I send a message, undo it, and then send it again without touching the input, I need to change the value of the input to be able to re-send the message, but cannot resend the restored message.
tried a few workarounds, like dispatching an input event on the textarea, or calling next on the message observable, both in the restore function. none of them worked.
textarea.addEventListener('input', event => message$.next(event.target.value))
send.addEventListener('click', () => textarea.value = '')
const sendEvent$ = fromEvent(send, 'click')
const undoEvent$ = fromEvent(undo, 'click')
const message$ = new Subject()
let cache = []
sendEvent$
.pipe(
withLatestFrom(message$, (_, m) => m),
tap(m => cache.push(m)),
delay(3000),
takeUntil(undoEvent$.pipe(
tap(restore)
)),
repeatWhen(complete => complete)
)
.subscribe(x => {
console.log(x)
cache = []
})
function restore() {
if (!textarea.value) {
const message = cache.join('\n')
textarea.value = message
cache = []
}
}
link the example: https://stackblitz.com/edit/rxjs-undo-message

The problem is pretty much there in the restore function. When you restore the "onInput" event doesn't get triggered. So your message queue basically is not enqueued with the restored item. The suggestion given by #webber where you can pass the message$.next(message) is pretty much right and that's what you need to do.
But the problem is how exactly you set it. You can set the value through a setTimeout interval in restore() so that your takeUntil() completes and then the value is enqueued in the Subject
function restore() {
if (!textarea.value) {
const message = cache.join('\n')
textarea.value = message
cache = []
setTimeout(function(){
message$.next(message)
},3000)
}
}
(or)
You can remove the
textarea.addEventListener('input', event => message$.next(event.target.value))
and change your send event handler to the following.
send.addEventListener('click', () => {
message$.next(textarea.value);
textarea.value =''
})

The subscriber works fine, it's just that your message$ doesn't get updated when the undoEvent$ triggers. However the value gets set to an empty string.
If you undo, then type and then send again, you will see that it works in the first time as well.
What you have to do is set message$ to the value of your textarea and then it works.

RXJS Scan - withLatestFrom another observable

I'm trying to work out how to use scan to derive a new state whenever my input observable emits a new value, but I can't seem to get it working.
I want to output a new State every time the input$ observable emits a new value, but it should be derived from the current value of state$.
Can anyone suggest how I can fix this? I have a feeling I've got the wrong idea altogether :-)
My code looks something like this:
const stateReducer = (state$: Observable<State>, input$: Observable<Input>) => {
state$ = state$.pipe( startWith(DEFAULT_STATE) );
const foo$: Observable<State> = input$.pipe(
filter((input) => isFoo(input)),
withLatestFrom(state$),
scan((acc, ([input, state]) => {
//returns derived state
});
const bar$: Observable<State> = input$.pipe(
filter((input) => isBar(input)),
withLatestFrom(state$),
scan((acc, ([input, state]) => {
//returns derived state
});
return merge(
foo$,
bar$
);
}

Since you want to use the result of an Observable as your seed value, switchMap will help.
switchMap docs
const bar$: Observable<State> = state$.pipe(
switchMap((state) => {
return input$.pipe(
filter((input) => isBar(input)),
scan((curState, input) => {
// do some logic here
return {...curState, prop: 'new value'}
}, state);
)
})
I've made a sample CodePen for a more complete solution at https://codepen.io/askmattcairns/pen/LYjEoZz?editors=0010.
More Details
switchMap means to switch to the stream in here. So this code is saying, once state$ emits a value, store it (as state), then wait for input$ to emit.
When we call scan, its seed value (the second property of scan) is now the result of state$'s emitted value.
This will emit a new value any time input$ receives a new value.
Update to Code Sandbox
After digging in to your Code Sandbox, I better understand what the problem is. You can see my final output at https://codesandbox.io/s/elegant-chaum-2b794?file=/src/index.tsx.
When you initialize your 2 inner streams foo$ and bar$, they both reference state$ using withLatestFrom. Each time input$ emits, it still references the original value of state, using 0 as its starting total.

RxJS not all Subscribers receive all events

I'm working on an exercise about RxJS.
And there's something very strange happening:
typoStream.subscribe(x => console.log('wont get executed'));
wordCompletedStream.subscribe(nextStream);
typoStream.subscribe(x => console.log('will get executed'));
When the application runs the first console.log won't get printed and the second one will.
Regardless of what the streams are and how they interact - this should never happen, right? Why is it important when I subscribe to an observable - shouldn't it emit the event to every subscriber anyways?
If you want to try it:
http://embed.plnkr.co/xb8Yimo5RcYGPtgClYgY/
Type the displayed word correctly and you can see the "error" in action. But it doesn't happen every time - only most of the time.
Here's the stream flow: https://photos.app.goo.gl/Z4cpKzekAIuKzMF93

I had a play with the code you posted, and the key fix is to properly multicast the checkWord observable. You can do this with .publish().refCount() like you did for wordStream, or you can use the shortcut method that does the same thing, .share().
const checkStream = wordStream.combineLatest(inputStream).share();
The reason this works is that without it, multiple subscriptions to checkStream or any streams derived from it, such as typoStream and wordCompletedStream will each trigger a new subscription to the wordStream observable (which is correctly multicast, so no new request gets made) and the inputStream observable, which will register new event listeners on the input.
With the .share() operator, it doesn't matter how many subscriptions are made to checkStream or derived observables, only the first one will trigger a subscription to inputStream.
Note that after this fix, neither of the two subscribers to typoStream will fire for a correctly entered word. Which is what I would expect from an observable called typoStream. Both will fire when an incorrect character is entered.
Forked Plunkr here
Or see the snippet below:
(() => {
// --- UI Stuff, NO NEED TO TOUCH THESE --- //
const wordField = $('#TotDWord');
const inputField = $('#TotDInput');
// ----------------------------------------- //
// A stream of the users string inputs
const inputFieldStream = Rx.Observable.fromEvent(inputField, 'keyup')
.map(x => x.target.value).distinctUntilChanged();
// This stream is used to represent the users unput - we don't use the
// inputFieldStream directly because we want to manually send values aswell
const inputStream = new Rx.Subject();
// Feed the stream from the field into our inputStream
inputFieldStream.subscribe(inputStream);
// A stream that allows us to manually trigger that we need a new word
const nextStream = new Rx.Subject();
// When we want the next word we need to reset the users input
nextStream.subscribe(() => {
inputField.val('');
inputStream.onNext('');
});
// This stream calls a server for a new random word every time the nextStream emits an event. We startWith a value to trigger the first word
const wordStream = nextStream.startWith('')
.flatMapLatest(getRandomWord)
// publish & refCount cache the result - otherwise every .map on wordStream would cause a new HTTP request
.publish().refCount();
// When there is a new word, we display it
wordStream.subscribe(word => {
wordField.empty();
wordField.append(word);
});
// Checkstream combines the latest word with the latest userinput. It emits an array, like this ['the word', 'the user input'];
const checkStream = wordStream.combineLatest(inputStream).share();
// Emits an event if the user input is not correct
const typoStream = checkStream.filter(tuple => {
const word = tuple[0];
const input = tuple[1];
return !word.startsWith(input);
});
// When there is a typo we need a new word
typoStream.subscribe(nextStream);
// Emits an event when the user has entered the entire word correctly
const wordCompletedStream = checkStream.filter(tuple => {
const word = tuple[0];
const input = tuple[1];
return word == input;
});
/**
* THIS WILL (MOST OF THE TIME) NOT FIRE WHEN YOU COMPLETE A WORD
*/
typoStream.subscribe(x => console.log('wont get executed'));
// Whenever the word is completed, request a new word
wordCompletedStream.subscribe(nextStream);
/**
* THIS WILL FIRE WHEN YOU COMPLETE A WORD
*/
typoStream.subscribe(x => console.log('will get executed'));
// Calls a server for a random word
// returns a promise
function getRandomWord() {
return $.ajax({
// Change the URL to cause a 404 error
url: 'https://setgetgo.com/randomword/get.php'
}).promise();
}
})();
<script data-require="jquery" data-semver="3.1.1" src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/4.0.6/rx.all.js"></script>
<div>
<h1>Exercise: Typing of the Dead</h1>
<div>
Type the given word correctly and watch the console. Most of the time 1 of the 2 subscriptions on the typoStream will fire (when there should fire none).
</div>
<br />
<div id="TotDWord"></div>
<input type="text" name="" id="TotDInput" value="" /><span>Highscore: </span><span id="TotDHighscore"></span>
<div id="TotDScore"></div>
</div>
<script>
console.clear();
</script>

What is the correct way to manipulate RxJS streams and publish an Observable of the results?

I have a websocket connection that is generating internal message events with a ReplaySubect. I process these events and add a delay to certain messages. Internally I use publish().refCount() twice, once on the internal ReplaySubject and again on the published output stream.
Should the internal subject have both 'publish' and 'refCount' called on it? I use 'publish' because I have multiple subscribers but I'm not entirely sure when to use 'refCount'.
Is it okay to just dispose of the internal subject? Will that clean up everything else?
Whoever subscribes to 'eventStream' should get the latest revision but the connection shouldn't wait for any subscribers
Example code:
function Connection(...) {
var messageSubject = new Rx.ReplaySubject(1);
var messageStream = messageSubject.publish().refCount();
// please ignore that we're not using rxdom's websocket.
var ws = new WebSocket(...);
ws.onmessage = function(messageEvent) {
var message = JSON.parse(messageEvent.data);
messageSubject.onNext(message);
}
ws.onclose = function(closeEvent) {
messageSubject.dispose(); // is this all I need to dispose?
}
var immediateRevisions = messageStream
.filter((e) => e[0] === "immediate")
.map((e) => ["revision", e[1]]);
var delayedRevisions = messageStream
.filter((e) => e[0] === "delayed")
.map((e) => ["revision", e[1]]).delay(1000);
var eventStream = Rx.Observable.merge(immediateRevisions, delayedRevisions).publish().refCount();
Object.defineProperties(this, {
"eventStream": { get: function() { return eventStream; }},
});
}
// using the eventStream
var cxn = new Connection(...)
cxn.eventStream.subscribe((e) => {
if (e[0] === "revision") {
// ...
}
});

publish and refCounting is basically what shareReplay does in RxJS4. Honestly though, you should just let your observable be "warm" and then use a ReplaySubject as a subscriber if you really want to guarantee that the last message gets pushed to new subscribers even if subscription count falls below one. e.g:
const wsStream = Observable.create(observer => {
ws.onmessage = message => observer.next(message);
ws.onclose = () => observer.complete();
});
const latestWsMessages = new ReplaySubject(1);
wsStream.subscribe(latestWsMessages);
Make sure you review how Observables work: after creating an observable, normally, each subscriber will call the subscription (cold), but in this case, you probably want a hot observable so that you have multiple subscribers sharing a subscription. See Andre's video here and the RxJS docs on creating observables for some more info.
Also, as useful as classes can be, looks like in this case you just want a function of makeWebsocketObservable(WebsocketConfig): Observable<WebsocketEvent>