Say we have a big array and processing of each element in that array takes a long time (5s). We want to add some delay (2s) before processing of next element.
I've somewhat managed to achieve that behavior like that:
let arr = [1, 2, 3]
let i = 0
Rx.Observable.of(null)
.map(val => arr[i++])
.do(val => {
console.log(val + ' preloop')
let time = +new Date() + 5000; while (+new Date() < time) {}
console.log(val + ' afterloop')
})
.delay(2000)
.flatMap(val => (i < arr.length) ? Rx.Observable.throw(null) : Rx.Observable.empty())
.retry()
.subscribe(console.log, console.log, () => console.log('completed'))
The output is as expected:
1 preloop
delay 5s
1 afterloop
delay 2s
2 preloop
...
completed
But this code is ugly, not reusable and buggy and doesn't comply with the philosophy of rx. What is the better way?
Note that the array (or it might even be not an array at all) is big and https://stackoverflow.com/a/21663671/2277240 won't work here.
The question is hypothetycal though I can think of some use cases.
I'm not sure why the fact that the array is big important here, and you can solve your issue using the same method from the link you provided.
For your long operation a better practice would probably be a promise. I used an async sleep function to simulate a 5 second operation, you can replace it with your promise.
The trick for the extra delay, is to concat a dummy element, delay it and then ignore it.
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
Rx.Observable.from([1, 2, 3])
.concatMap(item =>
Rx.Observable.defer(() => sleep(5000))
.mapTo(item)
.concat(Rx.Observable.of(null).delay(2000).ignoreElements())
)
.subscribe(console.log, console.log, () => console.log('completed'))
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/5.5.2/Rx.js"></script>
An alternative for delay which does not delay the first emit,
source
.first()
.merge(
source.skip(1).delay(2000)
)
Related
Given:
An NgRx effect that handles a request to "zoom in" a display. It gets a notification every time users click on an appropriate button.
public readonly zoomIn$ = createEffect(
() =>
this.actions$.pipe(
ofType(zoomIn),
tap(() => {
scale();
}),
),
{ dispatch: false },
);
Note: The zoomIn action couldn't and doesn't contain any payload. Consider it only as a trigger
Problem:
The redrawing costs resources and in some cases occupy a few seconds to get a new scale. So if you want to scale up several times in a row you'll be compelled to wait.
Solution:
By using the debounceTime operator postpone the call of the scale() function and wait until users make several clicks. Sounds good. The only problem is debounceTime notifying us of the latest value. And what we need is a number of values (user's clicks) silenced by the debounceTime operator.
In a more general view, the task sounds like: how to calculate the count of values emitted by the source stream and silenced by the debounceTime operator?
My solution is to create a custom pipable operator that achieves the aim.
What do we need? Of course, we need a debounceTime operator.
.pipe(
debounceTime(300),
)
Then we should calculate the number of values has been emitted. There is a scan operator that pretty much looks like a well-known reduce function. We'll give it an initial value and will increase a counter on every received value from the source stream. We have to place it before the debounceTime operator. It looks now like a stream of indices.
.pipe(
scan(acc => acc + 1, 0),
debounceTime(300),
)
When debounceTime notifies us of the latest index, how can we know the number of muted values? We have to compare it with the previous index that has been emitted. The previous value can be received by using a pairwise operator. And then get a difference between them using the map operator.
.pipe(
scan(acc => acc + 1, 0),
debounceTime(300),
pairwise(),
map(([previous, current]) => current - previous),
)
If you try this in the current state you notice that something is wrong, that it doesn't work for the first time. The problem lies in the pairwise operator. It emits pairs of values (previous and current), so it waits until it has at least two values before starting the emission of pairs. Is it fair? Yes, it is? That's why we need to cheat it a little and provide a first value (that is 0), with the use of the startWith operator.
The final implementation
/**
* Emits a notification from the source Observable only after a particular time span has passed without another source emission,
* with an exact number of values were emitted during that time.
*
* #param dueTime the timeout duration in milliseconds for the window of time required to wait for emission silence before emitting the most recent source value.
* #returns MonoTypeOperatorFunction
*/
export const debounceCounter =
(dueTime: number): MonoTypeOperatorFunction<number> =>
(source: Observable<unknown>): Observable<number> =>
new Observable(observer =>
source
.pipe(
scan(acc => acc + 1, 0),
debounceTime(dueTime),
startWith(0),
pairwise(),
map(([previous, current]) => current - previous),
)
.subscribe({
next: x => {
observer.next(x);
},
error: err => {
observer.error(err);
},
complete: () => {
observer.complete();
},
}),
);
Usage example
public readonly zoomIn$ = createEffect(
() =>
this.actions$.pipe(
ofType(zoomIn),
debounceCounter(300),
tap(times => {
// scale n-times
}),
),
{ dispatch: false },
);
I need some help from RxJS professionals :)
I try to recursively load data from a REST API via http request.
Recursive calls are working fine, however when I susbscribe to the final Observable (returned by GetTemperatures), no data is returned within subscribe.
Seems like no data is passed back in the call chain.
Whats going wrong here?
GetTemperatures().subscribe((data: MeasureData) => {
// add data to a chart, etc...
})
GetTemperatures(): Observable<MeasureData> {
const l_startDate = new Date(2019, 0, 1);
var l_httpParams = new HttpParams()
.set('device_id', this._deviceId)
.set('module_id', this._moduleId)
.set('scale', '1hour')
.set('type', 'Temperature')
.set('date_begin', Math.floor(l_startDate.getTime() / 1000).toString())
.set('real_time', 'true')
.set('optimize', 'true');
return this._http.post<MeasureDataInternal>(this._getMeasureUrl, l_httpParams)
.pipe(
map((data: MeasureDataInternal): MeasureData => this.transformMeasureData(data)),
flatMap((data: MeasureData) => {
return this.recursiveLoadData(data);
})
);
}
recursiveLoadData(data: MeasureData): Observable<MeasureData> {
// search until now minus 1,5 hours
const endDate = new Date(Date.now() - (1.5 * 60 * 60 * 1000));
console.error('RECURSIVE begin: ' + data.value[0].date + ' end: ' + data.value[data.value.length - 1].date);
// check if complete
if (data.value[data.value.length - 1].date.getTime() >= endDate.getTime()) {
console.error('recursive ENDs here');
return EMPTY;
}
var l_httpParams = new HttpParams()
.set('device_id', this._deviceId)
.set('module_id', this._moduleId)
.set('scale', '1hour')
.set('type', 'Temperature')
.set('date_begin', Math.floor(data.value[data.value.length - 1].date.getTime() / 1000).toString())
.set('real_time', 'true')
.set('optimize', 'true');
return this._http.post<MeasureDataInternal>(this._getMeasureUrl, l_httpParams)
.pipe(
map((data2: MeasureDataInternal): MeasureData => this.transformMeasureData(data2)),
flatMap((data2: MeasureData) => {
return this.recursiveLoadData(data2);
})
)
}
I have no idea what you're really trying to accomplish, but each new step in your recursion doesn't do anything other than bringing you to the next step. So you'll want to include what you're hoping each step does.
This isn't specific to streams, this is also true of general recursion.
General Recursion
This really isn't any different from how a regular recursive function works. Say you're recursively adding up the numbers in an array, you need to add the tail of the array to the first value. If you just keep recursing on a smaller array without adding up the numbers you've popped off, you'd get the base-case value back.
This returns the last value of the array (The last value of the array is the base-case):
recursiveAdd(array){
if(array.length === 1) return array[0];
return recursiveAdd(array.shift());
}
This adds the array:
recursiveAdd(array){
if(array.length === 1) return array[0];
return array[0] + recursiveAdd(array.shift());
}
In this simple case, the + operand is doing the work at each step of the recursion. Without it, the array isn't summed up. And, of course, I could do anything. Subtract the array from 1000, average the numbers in the array, build an object from the values. Anything.
Before you make a recursive call, you have to do something. Unless what you're after is the value of the base-case (In your case, an empty stream)
Recursion with Streams
When you mergeMap a value into a stream, you don't also pass forward that value.
from([69,70,71]).pipe(
mergeMap(val => from([
String.fromCharCode(val),
String.fromCharCode(val),
String.fromCharCode(val)
]))
).subscribe(console.log);
output
e e e f f f g g g
Notice how the output doesn't include any numbers? When you mergeMap, you map values into streams. If you want the values you're mapping to be part of the stream, you must include them somehow. This is the same as with general recursion.
So, here are two examples that both include your data in the returned stream. They're very basic, but hopefully, you can take some understanding from them and apply that.
This transforms the returned steam to include your data as its first value (recursively, of course)
return this._http.post<MeasureDataInternal>(this._getMeasureUrl, l_httpParams)
.pipe(
map((data: MeasureDataInternal): MeasureData =>
this.transformMeasureData(data)
),
mergeMap((data: MeasureData) =>
this.recursiveLoadData(data).pipe(
startWith(data)
)
)
);
This creates a stream of your data, a stream of your recursive call, and merges the two streams together.
return this._http.post<MeasureDataInternal>(this._getMeasureUrl, l_httpParams)
.pipe(
map((data: MeasureDataInternal): MeasureData =>
this.transformMeasureData(data)
),
mergeMap((data: MeasureData) =>
merge (
of(data),
this.recursiveLoadData(data)
)
)
);
Imagine I have the following code:
let a = Rx.Observable.of(1, 2, 3)
let b = Observable.zip(a, a, (a, b) => a + b)
b.forEach(t => console.log(t))
This immediately outputs the results. Now, how do I put a timed delay between each message as a way of back-pressure (note that I don't want a buffer; instead, I want a and b to become Cold Observables), like:
b.takeEvery(1000).forEach(t => console.log(t))
And have the exact same answer:
<wait 1s>
2
<wait 1s>
4
<wait 1s>
6
Alternative: If backpressure (ou pull mechanisms for some observables) is something not supported in RxJS, then how could one create an infinite generator without running out of resources?
Alternative 2: Other JS frameworks that support both pull and push mechanisms?
In case of RxJS 5.x back pressure is not support, but there is for example pausable operator in 4.x version. It works only with hot observables. More info on back pressure in case of 4.x and here (especially take a loot at the bottom and RxJS related description).
This Erik Meijer's tweet may be bit controversial but relevant: https://twitter.com/headinthebox/status/774635475071934464
For your own implementation of back pressure mechanism you need to have 2-way communication channel, which can be fairly easily created with 2 subjects - one for each end. Basically use next for sending messages and .subscribe for listing to the other end.
Creating a generator is doable as well - again using a subject to bridge between push- and pull-based worlds. Below an exemplary implementation for generating Fibonacci numbers.
const fib = () => {
const n = new Rx.Subject()
const f = n
.scan(c => ({ a: c.b, b: c.b + c.a }), { a: 0, b: 1 })
.map(c => c.a)
return {
$: f,
next: () => n.next()
}
}
const f = fib()
f.$.subscribe(n => document.querySelector('#r').innerHTML = n)
Rx.Observable.fromEvent(document.querySelector('#f'), 'click')
.do(f.next)
.subscribe()
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/5.5.6/Rx.js"></script>
<button id='f'>NEXT FIBONACCI</button>
<div id='r'>_?_<div>
Another js library which may be of interest for you is https://github.com/ubolonton/js-csp - did not use it, so not sure how it deals with back pressure.
the idea is to queue the time wait one after the other when the previous one finishes execution Fiddle
let a = Rx.Observable.of(1, 2, 3);
let b = Rx.Observable.zip(a, a, (a, b) => a + b);
// getting values into array
var x = [];
b.forEach(t => x.push(t));
var takeEvery = function(msec,items,action,index=0){
if(typeof(action) == "function")
if(index<items.length)
setTimeout(
function(item,ind){
action(item);
takeEvery(msec,items,action,ind);
},msec, items[index],++index);
};
// queueing over time
takeEvery(1000,x, function(item){
console.log(item);
});
I have a array of about 18 000 elements. I'm creating a map application where I want to add the elements when the user zooms in to a certain level.
So when the user zooms in under 9 I loop tru the array looking for elements that is in the view.
However, it does take some time looping thru the elements, causing the map application lag each time the user zooms out and in of "level 9". Even if there are no elements to add or not, so the bottleneck is the looping I guess.
I've tried to solve it by asyncing it like:
function SearchElements(elementArr) {
var ret = new Promise(resolve => {
var arr = [];
for (var i in elementArr) {
var distanceFromCenter = getDistanceFromLatLonInKm(view.center.latitude, view.center.longitude, dynamicsEntities[i].pss_latitude, dynamicsEntities[i].pss_longitude);
var viewWidthInKm = getSceneWidthInKm(view, true);
if (distanceFromCenter > viewWidthInKm) continue;
arr.push(elementArr[i]);
}
resolve(arr);
});
return ret;
}
SearchElements(myElementsArray).Then(arr => {
// ...
});
But its still not async, this method hangs while the for loop runs.
Because you still have a tight loop that loops through all the elements in one loop, you'll always have the responsiveness issues
One way to tackle the issue is to works on chunks of the data
Note: I'm assuming elementArr is a javascript Array
function SearchElements(elementArr) {
var sliceLength = 100; // how many elements to work on at a time
var totalLength = elementArr.length;
var slices = ((totalLength + sliceLength - 1) / sliceLength) | 0; // integer
return Array.from({length:slices})
.reduce((promise, unused, outerIndex) =>
promise.then(results =>
Promise.resolve(elementArr.slice(outerIndex * sliceLength, sliceLength).map((item, innerIndex) => {
const i = outerIndex * sliceLength + innerIndex;
const distanceFromCenter = getDistanceFromLatLonInKm(view.center.latitude, view.center.longitude, dynamicsEntities[i].pss_latitude, dynamicsEntities[i].pss_longitude);
const viewWidthInKm = getSceneWidthInKm(view, true);
if (distanceFromCenter <= viewWidthInKm) {
return item; // this is like your `push`
}
// if distanceFromCenter > viewWidthInKm, return value will be `undefined`, filtered out later - this is like your `continue`
})).then(batch => results.concat(batch)) // concatenate to results
), Promise.resolve([]))
.then(results => results.filter(v => v !== undefined)); // filter out the "undefined"
}
use:
SearchElements(yourDataArray).then(results => {
// all results available here
});
My other suggestion in the comment was Web Workers (I originally called it worker threads, not sure where I got that term from) - I'm not familiar enough with Web Workers to offer a solution, however https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers and https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API should get you going
To be honest, I think this sort of heavy task would be better suited to Web Workers
I'm using rxjs 5.0:
How can I set a timeout, on this buffer. So that it will clear the bufferCount (11) when no keyup events happen for 5 seconds?
var keys = Rx.Observable.fromEvent(document, 'keyup');
var buffered = keys.bufferCount(11,1);
buffered.subscribe(x => console.log(x));
You can append a timeoutWith, which could return a fresh buffered after a certain timeout (5seconds in your case).
const keys$ = Rx.Observable.fromEvent(document, "keyup")
.map(ev => ev.keyCode|| ev.which); // this is just to have a readable output here in the SO-console
const buffered$ = keys$
.bufferCount(3,1) // replaced your 11 with 3 for easy demonstration
.timeoutWith(2000, Rx.Observable.defer(() => { // replaced 5 with 2 seconds (easier to test here)
console.log("New Buffer!");
return buffered$;
}));
buffered$.subscribe(console.log);
<script src="https://unpkg.com/rxjs/bundles/Rx.min.js"></script>
As an improvement, this could be even enhanced to only start the stream on the first stroke, otherwise we would have a constant timeout running (not critical, but could still be prevented).
const keys$ = Rx.Observable.fromEvent(document, "keyup")
.map(ev => ev.keyCode|| ev.which); // this is just to have a readable output here in the SO-console
const buffered$ = keys$
.take(1)
.switchMap(firstKey => {
console.log("New Buffer!");
return keys$
.startWith(firstKey)
.bufferCount(3,1) // replaced your 11 with 3 for easy demonstration
.timeoutWith(2000, Rx.Observable.defer(() => buffered$)); // replaced 5 with 2 seconds (easier to test here)
});
buffered$.subscribe(console.log);
<script src="https://unpkg.com/rxjs/bundles/Rx.min.js"></script>
I have another (and probably easier to understand) solution using window and switchMap():
var keys = Rx.Observable.fromEvent(document.getElementById('myinput'), 'keyup')
.map(event => event.keyCode)
.share();
var buffered = keys
.window(keys.debounceTime(5000))
.switchMap(observable => observable.bufferCount(5, 1))
.filter(buffer => buffer.length === 5);
buffered.subscribe(x => console.log(x));
See demo: https://jsbin.com/cakoru/17/edit?js,console,output
When you don't type for at least 5s the window() operator creates a new Observable that is subscribed internally in switchMap() and chained with a new .bufferCount() operator.
Here's how I'd do it:
const keys$ = Rx.Observable.fromEvent(document, 'keyup').map(ev => ev.keyCode|| ev.which);
keys$
.debounceTime(5000)
.startWith({})
.switchMap(x => keys$.bufferCount(11, 1))
.subscribe(x => console.log(x));
Here we've got a stream that yields a value each time typing stops for five seconds (kicked off with a dummy value) that switchMaps into a bufferCount.