I want to simulate multiple slow subscriptions. The client subscribes to two or more publications at the same time, and the result arrive later.
The goal is to be able to see how network latencies and randomness can affect my application (it bugs because I expected a publication to be ready before another, ...).
Using the following short setup for the publications:
// server/foo.js
Meteor.publish('foo', function() {
console.log('publishing foo');
Meteor._sleepForMs(2000);
console.log('waking up foo');
this.ready();
});
// server/bar.js is the same with a different name
Meteor.publish('bar', function() {
console.log('publishing bar');
Meteor._sleepForMs(2000);
console.log('waking up bar');
this.ready();
});
Both publications are slowed down thanks to Meteor._sleepForMs as seen in this amazing answer.
The client then subscribes to each publication:
Meteor.subscribe('bar'); // /client/bar.js
Meteor.subscribe('foo'); // /client/foo.js
From there I expected to see both 'publishing' logs first, then both 'waking up'.
However, this appears in the console:
15:37:45? publishing bar
15:37:47? waking up bar
15:37:47? publishing foo
15:37:49? waking up foo
(I removed some irrelevant fluff like the day)
So obviously it runs in a synchronous fashion. I thought that two things can cause that: the server waitForMs which would entirely block the server (fairly weird), or the client subscription design.
To make sure that it wasn't the server I added a simple heartbeat:
Meteor.setInterval(function() { console.log('beep'); }, 500);
And it did not stop beeping, so the server isn't fully blocked.
I thus suspect that the issue lies within the client subscription model, which maybe waits for the subscription to be ready before calling another..?
Thus, two questions:
Why doesn't my experiment run the way I wanted it to?
How should I modify it to achieve my desired goal (multiple slow publications) ?
Meteor processes DDP messages (which include subscriptions) in a sequence. This ensures that you can perform some action like deleting an object and then inserting it back in the correct order, and not run into any errors.
There is support for getting around this in Meteor.methods using this.unblock() to allow the next available DDP message to process without waiting for the previous one to finish executing. Unfortunately this is not available for Meteor.publish in the Meteor core. You can see discussion (and some workarounds) about this issue here: https://github.com/meteor/meteor/issues/853
There is also a package that adds this functionality to publications:
https://github.com/meteorhacks/unblock/
Why doesn't my experiment run the way I wanted it to?
Meteor._sleepForMs is blocking from the way it is implemented:
Meteor._sleepForMs = function (ms) {
var fiber = Fiber.current;
setTimeout(function() {
fiber.run();
}, ms);
Fiber.yield();
};
Calling it prevents the next line from executing inside the fiber until the duration passes. However, this does not block the Node server from handling other events (i.e. executing another publication) due to the way fiber works.
Here is a talk about Fibers in Meteor: https://www.youtube.com/watch?v=AWJ8LIzQMHY
How should I modify it to achieve my desired goal (multiple slow publications) ?
Try using Meteor.setTimeout to simulate latency asynchronously.
Meteor.publish('foo', function() {
console.log('publishing foo');
var self = this;
Meteor.setTimeout(function () {
console.log('waking up foo');
self.ready();
}, 2000);
});
I believe it's because the publications are blocking.
You can use meteorhacks:unblock to unblock publications:
https://atmospherejs.com/meteorhacks/unblock
It could be a good idea to use this.unblock() at the start of every publication (once you've added meteorhacks:unblock).
Related
I am using node.js 14 and currently have a loop that is made by a recursive function and a setTimeout, something like this:
this.timer = null;
async recursiveLoop() {
//Do Stuff
this.timer = setTimeout(this.recursiveLoop.bind(this), rerun_time);
}
But sometimes this loop gets stuck and I want it to automatically notice it, clean up and restart. So I tried doing something like this:
this.timer = null;
async recursiveLoop() {
this.long_timer = setTimeout(() => throw new Error('Taking too long!'), tooLong);
//Do Stuff
this.timer = setTimeout(this.recursiveLoop.bind(this), rerun_time);
}
main() {
//Do other asynchronous stuff
recursiveLoop()
.then()
.catch((e) => {
console.log(e.message);
cleanUp();
recursiveLoop();
}
}
I can't quite debug where it gets stuck, because it seems quite random and the program runs on a virtual machine. I still couldn't reproduce it locally.
This makeshift solution, instead of working, keeps crashing the whole node.js aplication, and now I am the one stuck. I have the constraint of working with node.js 14, without using microservices, and I never used child process before. I am a complete beginner. Please help me!
If you have a black box of code (which is all you've given us) with no way to detect errors on it and you just want to know when it is no longer generating results, you can put it in a child_process and ask the code in the child process to send you a message every time it runs an iteration. Then, in your main process, you can set a timer that resets itself every time it gets one of these "health" messages from the child. If the timer fires without getting a health message, then the child must be "stuck" because you haven't heard from it within your timeout time. You can then kill the child process at that point and restart it.
But, that is a giant hack. You should FIX the code that gets stuck or at least understand what's going on. Probably you're either leaking memory, file handles, database handles, running code that uses locks and messes up or there are unhandled errors happening. All are indications of code that should be fixed.
As I understand it, there are three somewhat distinct reasons to put multiple IndexedDB operations in a single transaction rather than using a unique transaction for each operation:
Performance. If you’re doing a lot of writes to an object store, it’s much faster if they happen in one transaction.
Ensuring data is written before proceeding. Waiting for the “oncomplete” event is the only way to be sure that a subsequent IndexedDB query won’t return stale data.
Performing an atomic set of DB operations. Basically, “do all of these things, but if one of them fails, roll it all back”.
#1 is fine, most databases have the same characteristic.
#2 is a little more unique, and it causes issues when considered in conjunction with #3. Let’s say I have some simple function that writes something to the database and runs a callback when it's over:
function putWhatever(obj, cb) {
var tx = db.transaction("whatever", "readwrite");
tx.objectStore("whatever").put(obj);
tx.oncomplete = function () { cb(); };
}
That works fine. But now if you want to call that function as a part of a group of operations you want to atomically commit or fail, it's impossible. You'd have to do something like this:
function putWhatever(tx, obj, cb) {
tx.objectStore("whatever").put(obj).onsuccess = function () { cb(); };
}
This second version of the function is very different than the first, because the callback runs before the data is guaranteed to be written to the database. If you try to read back the object you just wrote, you might get a stale value.
Basically, the problem is that you can only take advantage of one of #2 or #3. Sometimes the choice is clear, but sometimes not. This has led me to write horrible code like:
function putWhatever(tx, obj, cb) {
if (tx === undefined) {
tx = db.transaction("whatever", "readwrite");
tx.objectStore("whatever").put(obj);
tx.oncomplete = function () { cb(); };
} else {
tx.objectStore("whatever").put(obj).onsuccess = function () { cb(); };
}
}
However even that still is not a general solution and could fail in some scenarios.
Has anyone else run into this problem? How do you deal with it? Or am I simply misunderstanding things somehow?
The following is just opinion as this doesn't seem like a 'one right answer' question.
First, performance is an irrelevant consideration. Avoid this factor entirely, unless later profiling suggests a material problem. Chances of perf issues are ridiculously low.
Second, I prefer to organize requests into transactions solely to maintain integrity. Integrity is paramount. Integrity as I define it here simply means that the database at any one point in time does not contain conflicting or erratic data. Essentially the database is never able to enter into a 'bad' state. For example, to impose a rule that cross-store object references point to valid and existing objects in other stores (a.k.a. referential integrity), or to prevent duplicated requests such as a double add/put/delete. Obviously, if the app were something like a bank app that credits/debits accounts, or a heart-attack monitor app, things could go horribly wrong.
My own experience has led me to believe that code involving indexedDB is not prone to the traditional facade pattern. I found that what worked best, in terms of organizing requests into different wrapping functions, was to design functions around transactions. I found that quite often there are very few DRY violations because every request is nearly always unique to its transactional context. In other words, while a similar 'put object' request might appear in more than one transaction, it is so distinct in its behavior given its separate context that it merits violating DRY.
If you go the function per request route, I am not sure why you are checking if the transaction parameter is undefined. Have the caller create the function and then pass it to the requests in turn. Expect the tx to always be defined and do not over-zealously guard against it. If it is ever not defined there is either a serious bug in indexedDB or in your calling function.
Explicitly, something like:
function doTransaction1(db, onComplete) {
var tx = db.transaction(...);
tx.onComplete = onComplete;
doRequest1(tx);
doRequest2(tx);
doRequest3(tx);
}
function doRequest1(tx) {
var store = tx.objectStore(...);
// ...
}
// ...
If the requests should not execute in parallel, and must run in a series, then this indicates a larger and more difficult design issue.
I'd like to run a task on a Meteor client which is resource hungry in the background and keep the interface responsive for the user in the meantime. The task does some math (for example finding prime numbers like described here: https://stackoverflow.com/a/22930538/2543628 ).
I've tried to follow the tips from https://stackoverflow.com/a/21351966 but still the interface always "freezes" until the task is complete.
setTimeout, setInterval and those packages like in my current approach also didn't help:
var taskQueue = new PowerQueue();
taskQueue.add(function(done) {
doSomeMath();
// It's still blocking/freezing the interface here until done() is reached
done();
});
Can I do something to make the interface responsive during doSomeMath() is running or am I doing something wrong (also it doesn't look like there is much you could do wrong in PowerQueue)?
JavaScript libraries which solve the problem of asynchronous queuing, assume that the tasks being queued are running in a concurrent but single-threaded environment like node.js or your browser. However, in your case you need more than just concurrency - you need multi-threaded execution in order to move your CPU-intensive computation out of your UI thread. This can be achieved with web workers. Note that web workers are only supported in modern browsers, so keep reading if you don't care about IE9.
The above article should be enough to get you started, however it's worth mentioning that the worker script will need to be kept outside of your application tree so it doesn't get bundled. An easy way to do this is to put it inside of the public directory.
Here is a quick example where my worker computes a Fibonacci sequence (inefficiently):
public/fib.js
var fib = function(n) {
if (n < 2) {
return 1;
} else {
return fib(n - 2) + fib(n - 1);
}
};
self.addEventListener('message', (function(e) {
var n = e.data;
var result = fib(n);
self.postMessage(result);
self.close();
}), false);
client/app.js
Meteor.startup(function () {
var worker = new Worker('/fib.js');
worker.postMessage(40);
worker.addEventListener('message', function(e) {
console.log(e.data);
}, false);
});
When the client starts, it loads the worker and asks it to compute the 40th number in the sequence. This takes a few seconds to complete but your UI should remain responsive. After the value is returned, it should print 165580141 to the console.
I'm creating some algorithms that are very performance heavy, e.g. evolutionary and artificial intelligence. What matters to me is that my update function gets called often (precision), and I just can't get setInterval to update faster than once per millisecond.
Initially I wanted to just use a while loop, but I'm not sure that those kinds of blocking loops are a viable solution in the Node.js environment. Will Socket.io's socket.on("id", cb) work if I run into an "infinite" loop? Does my code somehow need to return to Node.js to let it check for all the events, or is that done automatically?
And last (but not least), if while loops will indeed block my code, what is another solution to getting really low delta-times between my update functions? I think threads could help, but I doubt that they're possible, my Socket.io server and other classes need to somehow communicate, and by "other classes" I mean the main World class, which has an update method that needs to get called and does the heavy lifting, and a getInfo method that is used by my server. I feel like most of the time the program is just sitting there, waiting for the interval to fire, wasting time instead of doing calculations...
Also, I'd like to know if Node.js is even suited for these sorts of tasks.
You can execute havy algorithms in separate thread using child_process.fork and wait results in main thread via child.on('message', function (message) { });
app.js
var child_process = require('child_process');
var child = child_process.fork('./heavy.js', [ 'some', 'argv', 'params' ]);
child.on('message', function(message) {
// heavy results here
});
heavy.js
while (true) {
if (Math.random() < 0.001) {
process.send({ result: 'wow!' });
}
}
I am trying to customize the behavior of Selenium's click command, (via user-extentions.js), by intercepting calls to doClick(locator). Basically I need to delay click actions whenever our application's "busy indicator" is being displayed.
(Now the standard answer for this kind of thing is to insert a waitFor into the script for those situations. Indeed, we currently have zillions of them throughout our scripts. I'm trying to eliminate those.)
Detecting the page element is the trivial part. The tricky part is getting the script to actually wait. My promising looking, but failed attempt looks like this:
var nativeClick = Selenium.prototype.doClick;
Selenium.prototype.doClick = function(locator) {
this.doWaitForCondition("!selenium.browserbot.findElementOrNull('busy-indicator')", 5000);
return nativeClick.call(this, locator);
}
The doWaitForCondition gets called before every click, but it does not wait when the condition evaluates to false. nativeClick always gets called immediately, and so no delay is introduced. I suspect that the doWaitForCondition function doesn't actually do any waiting per se, but rather establishes the conditions for it within the command execution loop. And in this case the click command is already in play, and I'm trying to run a command within a command.
Can somebody shed some light on how Selenium command execution and waitFor works, or offer suggestions on how this might be done?
I have finally solved this. And with an approach that is much better than trying to intercept click processing in its various forms. My refined goal is: to delay execution of script command completion when our application is "busy".
How Selenium command processing works:
Upon completion, each selenium command returns an ActionResult object, (see ActionHandler.prototype.execute). The terminationCondition attribute on this object is a function that determines when it is okay for selenium to proceed to the next command, (TestLoop.prototype.continueTestWhenConditionIsTrue). Basically, selenium repeatedly executes the condition function until it yields true. The result object it quite trivial:
function ActionResult(terminationCondition) {
this.terminationCondition = terminationCondition;
}
Customizing it:
I want to delay execution any time myAppIsBusy() returns true. Of course all of the standard delays need to remain in place as well, like waiting for page loads, and explicit waitFor conditions as scripted. The solution is to redefine the selenium result object in my user-extensions.js, as follows:
function ActionResult(terminationCondition) {
this.terminationCondition = function() {
// a null terminationCondition means okay to continue
return (!terminationCondition || terminationCondition()) && !myAppIsBusy();
}
}
The great thing is that this is at a low enough level that it works for the IDE, as well as for RC.
Note that this does not affect Accessor or Assert command types, which return different result objects. But that should be fine, because those commands don't effect the state of the application.
Well, a look at the java drivers com.thoughtworks.selenium.Wait class reveals this:
public void wait(String message, long timeoutInMilliseconds, long intervalInMilliseconds) {
long start = System.currentTimeMillis();
long end = start + timeoutInMilliseconds;
while (System.currentTimeMillis() < end) {
if (until()) return;
try {
Thread.sleep(intervalInMilliseconds);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
throw new WaitTimedOutException(message);
}
I am not to deep into selenium but I excpect that every waitXXX Method points to this.
So, Selenium is working with Thread.sleep(). While this might not look like an ideal solution it shows at least that you cant make it worse by using Thread.sleep() on your own if neccessary. ;-)