I'm looking for a way to process events from ReplaySubject.subscribe() in a synchronous fashion.
let onSomeEvent = new ReplaySubject();
onSomeEvent.subscribe(async (event) => {
return await this.saveEventToDb(event);
});
In this example, saveEventToDb() first checks the database whether an event with the same ID was already stored. If not, it stores it.
The problem is I need to account for duplicate events firing from the subject.
In this example, when 2 duplicate event fire back-to-back, both get added to the database because saveEventToDb() gets called twice immediately without waiting for the previous call to finish.
How can I queue these up using Rxjs?
The following worked to process the events synchronously:
onSomeEvent
.map(event => {
return Observable.defer(() => {
return this.saveEventToDb(event);
});
})
.concatAll()
.subscribe();
ConcatAll(): Collect observables and subscribe to next when previous completes.
Related
I recently dived into subscriptions of Subject/BehaviorSubject/ etc and I am looking for the goto approach when used in combinations with Promises.
Given is the example code below:
firebase.user.subscribe((user: any | null) => {
fs.readFile('path/to/file')
.then((buf: Buffer) => {
this.modifySomeData = buf;
});
});
I subscribe to a Subject that triggers whenever the user logs in or out of their service. Whenever this happens, I read a file from disk. This readFile event could potentially take longer than the next "login/logout" event. Of course, I am in JS and in an asynchronous environment. This means, my user code is not multithreaded, but still, the 2nd user event and 2nd readFile could theoretically be faster than the first readFile.
First user event fired
First readFile is executed
Second user event is fired
Second readFile is executed
Second readFile is resolved <---
First readFile is resolved <---
The order is mixed up. The silliest approach I could think of is to create a uuid before reading the file and check inside the promise if this is still the same. If it's not I discard the data.
Is there a better solution?
If i have a process where older requests can be discarded i often keep a variable in scope to track the latest request and compare, similar to your UUID idea:
let lastRead: Promise<Buffer> | null = null;
firebase.user.subscribe((user: any | null) => {
const read = lastRead = fs.readFile('path/to/file');
read.then((buf: Buffer) => {
if (read != lastRead)
return;
this.modifySomeData = buf;
});
});
In this specific case, readFile also supports an abort signal. So you might also be able to abort the last request instead; you will still need to track it though.
The first approach is to see if your event generation logic could handle waiting for event handling. For example, you can use a promise to wait for the event OR generate another event, say doneReadFile and only then send the next event. Usually, this is not the case for a generic (distributed) environment.
If event generation does not care about how long it took to handle events, you can still use the above approach but check for the intermediate event doneReadFile in the next event handler (login/logout). This can be achieved by implementing some kind of polling or busy-wait/sleep
TL;DR: is it possible to say "call that function once all awaits in the current context finish" in JS/node?
A very simplified example:
a frontend-facing service creates a new user, then does another async task ([1]) and returns
a user service validates & saves the new user, then fires an event ([2]) that can trigger some other logic (unrelated to the current request)
goal: [1] should always finish before handlers for [2] start running
class Service1 {
// say this is called from a controller / express route handler
async createUser(userData: NewUserData): Promise<UserWithAdditionalData> {
const user = await this.userSerivce.validateAndSaveUser(userData);
// [1] this is also async, and should finish before [2] handlers start running
const userWithData = await this.getSomeAdditionalData(user);
return userWithData;
}
}
class UserService {
async validateAndSaveUser(userData: NewUserData): Promise<User> {
const validatedData = await this.validateNewUserData(userData);
await this.dbService.saveNew(validatedData)
// [2] trigger an event/hook to be executed later
this.eventBus.onUserCreated();
}
}
The question: is it possible to implement/replace [2] in a way to achieve the mentioned goal? Preferably in a better way than scheduling the event a few seconds in the future :D
In my current implementation, I'm using an event bus library, that calls registered event consumers when an event is triggered. However, since it's probably just pushing the callback onto the event loop under the hood, it's likely to be executed before [1] because both will just be queued onto the event loop. For the same reason, the same happens if I wrap the handlers in setTimeout(..., 0) or setImmediate. What I want to achieve, is that the event handlers should be fired after all the awaits from the caller are finished.
// please let's avoid discussing if the pattern above is a good design -- like most things, it can be used both in good and bad ways ;)
I have few Observables like this one in my code.
this.server.doRequest().subscribe(response => console.log(response)
error => console.log(error),
() => {
console.log('completed');
});
There could be any number of these Observables,
so I need to write a function that checks if each Observable is done otherwise waits till each is finished.
I'm assuming I can create an array push every new Observable there and when it's completed remove it by index. But is it good solution?
Where I want to use it. For example I have a page where user upload photos any amount asynchronously and then he press Finish button. Once he pressed Finish button I need to wait till ALL dynamically created Observables are completed.
you should use higher order observables for this, your exact use case will dictate the exact operator, but forkJoin seems a good candidate:
forkJoin(
this.server.doRequest1(),
this.server.doRequest2(),
this.server.doRequest3(),
this.server.doRequest4()
).subscribe(vals => console.log('all values', vals));
forkJoin won't emit till all innter observables have completed. making it the operator of choice for waiting for multiple observables to complete. You can also feed it an array of observables. There are multiple other operators that may fulfill your case too, such as concat, merge, combineLatest or a few others.
edit based on more details:
in the use case described in your update, you'll still want to use a higher order observable, but forkjoin is not what you want. you'll want to use a local subject to accomplish the goal as wanting to kick off each observable as it is selected and waiting for them all to be done complicates things a little (but not too much):
suppose you had a template like:
<button (click)="addPhoto()">Add Photo</button>
<button (click)="finish()">Finish</button>
where the add photo button gets the users photo and all that, and finish is your completion, you could have a component like this:
private addPhoto$ = new Subject();
constructor() {
this.addPhoto$.pipe(
mergeMap(() => this.uploadPhoto()),
).subscribe(
(resp) => console.log('resp', resp),
(err) => console.log('err', err),
() => console.log('complete')
);
}
private uploadPhoto() {
// stub to simulate upload
return timer(3000);
}
addPhoto() {
this.addPhoto$.next();
}
finish() {
this.addPhoto$.complete();
}
if you run this code, you'll see that the photo adds will emit in the subscribe handler as they complete, but complete will only fire once all the photo uploads have completed and the user has clicked finish.
here is a stackblitz demonstrating the functionality:
https://stackblitz.com/edit/angular-bsn6pz
I'd create a dictionary (in javascript that would be a JSON with observable names as boolean properties) where you push each observable on "create" and a method which should execute on completion of each observable, which will iterate through that dictionary and if all completed do something.
That will ensure parallelism and final execution after all completed.
var requests = {
doRequest1: false,
doRequest2: false,
doRequest3: false
};
var checkIfCAllCompleted = name => {
requests[name] = true;
for (var property in requests) {
if (object.hasOwnProperty(property)) {
if (!property) {
return;
}
}
}
// all properties are true - do something here
console.log("here");
}
this.server.doRequest1().then(() => checkIfCAllCompleted("doRequest1"));
this.server.doRequest2().then(() => checkIfCAllCompleted("doRequest2"));
this.server.doRequest3().then(() => checkIfCAllCompleted("doRequest3"));
I have a service with some methods, most of them require a certain callback to be completed before it can do its stuff. With Promises, in pseudo, it is very easy to do this:
ready = http.get(stuff); // Returns a promise, resolves after a while
methodOne() { // methods sometimes called before promise resolves
this.ready.then(_ => {
// doStuff
});
}
methodTwo() {
return this.ready.then(d => {
// doOtherStuff
});
}
Basically I need to do the stuff, only when i'm sure the service is ready.
I actually only need to check if it's ready (what methodOne is doing, just illustrating with methodTwo, that it's easy to more stuff as well).
I want to try and go all in on Observables, but for this specific case, I find it really hard to compete with a similar solution for Observables.
Promises will remember the value and know if it got resolved. An Observable is somewhat more complex and it seems that creating this same flow is troublesome. I need whatever is subscribing to the Observable, to known when it's ready. Some times the method is called early - before the Observable emits and sometimes late, after the Observable already emitted.
I have this right now, but it doesn't seem to work:
this.ready$ = someObservable // Will fire after a litle while but never finish - i only need the first to check though.
.publishReplay(1).refCount(); // Trying to replay if subscription comes after emit.
this.ready$.subscribe(_ => {
// This will be called
});
methodOne() {
this.ready$.subscribe(_ => {
// Not called
});
};
Perhaps i misunderstood the use of publishReplay and refCount?
I think what you're looking for is AsyncSubject. It mimics the promises behavior very well. Here is the description:
The AsyncSubject is a variant where only the last value of the
Observable execution is sent to its observers, and only when the
execution completes.
Here is how it can be used in your case:
subject = new AsyncSubject();
ready = streamOfData(stuff).first().subscribe(subject);
methodOne() {
return this.subject.asObservable();
}
The subject subscribes to the underlying observable returned by the first operator and waits until it's complete. It collects all the subscribers but doesn't send any values to them. As soon as the underlying observable completes it remembers the value and sends it to the collected subscribers. All new future subscribers will be immediately passed this stored resolved value.
Here is the simple example that demonstrates that you can subscribe before or after the observable completes:
const subject = new AsyncSubject();
const o = subject.asObservable();
o.subscribe((v) => {
console.log(v);
});
interval(500).first().subscribe(subject);
setTimeout(() => {
o.subscribe((v) => {
console.log(v);
});
}, 2000);
I am looking for an effective way to reply to a message sent to a child process. Currently, I am using the following code:
const { fork } = require('child_process');
const child = fork(path.join(__dirname, 'sub.js'));
async function run() {
console.log('Requesting status....');
child.send('status');
const status = await awaitMessage(child);
console.log(status);
}
function awaitMessage(childProcess) {
return new Promise((resolve) => {
childProcess.on('message', (m) => {
resolve(m);
});
});
}
The problem of this code is that it creates a new event listener every single time the awaitMessage() function is called, which is prone to memory leaks. Is there an elegant way of receiving a reply from the child process?
This isn't really "prone to memory leaks" in that a leak is something that is supposed to get freed (per the rules of the garbage collector), but isn't. In this case, you've left a promise hooked up to an event handler that can still get called so the system simply can't know that you intend for it to be freed.
So, the system is retaining exactly what your code told it to retain. It's the consequence of how your code works that it is retaining every promise ever created in awaitMessage() and also firing a bunch of extra event handlers too. Because you keep the event listener, the garbage collector sees that the promise is still "reachable" by that listener and thus cannot and should not remove the promise even if there are no outside references to it (per the rules of the Javascript garbage collector).
If you're going to add an event listener inside a promise, then you have to remove that event listener when the promise resolves so that the promise can eventually be freed. A promise is no magic object in Javascript, it's just a regular object so as long as you have an object that can be referenced by a live event listener, that object can't be garbage collected.
In addition, this is subject to race conditions if you ever call awaitMessage() twice in a row as both promises will then respond to the next message that comes. In general, this is just not a good design approach. If you want to wait for a message, then you have to somehow tag your messages so you know which message response is the actual one you're waiting for to avoid your race conditions and you have to remove the event listener after you get your message.
To avoid the memory build-up because of the accumulation of listeners, you can do this:
function awaitMessage(childProcess) {
return new Promise((resolve) => {
function handleMsg(m) {
childProcess.removeListener(handleMsg);
resolve(m);
}
childProcess.on('message', handleMsg);
});
}