Develop Reference

How to initialize a child process with passed in functions in Node.js

Context
I'm building a general purpose game playing A.I. framework/library that uses the Monte Carlo Tree Search algorithm. The idea is quite simple, the framework provides the skeleton of the algorithm, the four main steps: Selection, Expansion, Simulation and Backpropagation. All the user needs to do is plug in four simple(ish) game related functions of his making:
a function that takes in a game state and returns all possible legal moves to be played
a function that takes in a game state and an action and returns a new game state after applying the action
a function that takes in a game state and determines if the game is over and returns a boolean and
a function that takes in a state and a player ID and returns a value based on wether the player has won, lost or the game is a draw. With that, the algorithm has all it needs to run and select a move to make.
What I'd like to do
I would love to make use of parallel programming to increase the strength of the algorithm and reduce the time it needs to run each game turn. The problem I'm running into is that, when using Child Processes in NodeJS, you can't pass functions to the child process and my framework is entirely built on using functions passed by the user.
Possible solution
I have looked at this answer but I am not sure this would be the correct implementation for my needs. I don't need to be continually passing functions through messages to the child process, I just need to initialize it with functions that are passed in by my framework's user, when it initializes the framework.
I thought about one way to do it, but it seems so inelegant, on top of probably not being the most secure, that I find myself searching for other solutions. I could, when the user initializes the framework and passes his four functions to it, get a script to write those functions to a new js file (let's call it my-funcs.js) that would look something like:
const func1 = {... function implementation...}
const func2 = {... function implementation...}
const func3 = {... function implementation...}
const func4 = {... function implementation...}
module.exports = {func1, func2, func3, func4}
Then, in the child process worker file, I guess I would have to find a way to lazy load require my-funcs.js. Or maybe I wouldn't, I guess it depends how and when Node.js loads the worker file into memory. This all seems very convoluted.
Can you describe other ways to get the result I want?

child_process is less about running a user's function and more about starting a new thread to exec a file or process.
Node is inherently a single-threaded system, so for I/O-bound things, the Node Event Loop is really good at switching between requests, getting each one a little farther. See https://nodejs.org/en/docs/guides/event-loop-timers-and-nexttick/
What it looks like you're doing is trying to get JavaScript to run multiple threads simultaniously. Short answer: can't ... or rather it's really hard. See is it possible to achieve multithreading in nodejs?
So how would we do it anyway? You're on the right track: child_process.fork(). But it needs a hard-coded function to run. So how do we get user-generated code into place?
I envision a datastore where you can take userFn.ToString() and save it to a queue. Then fork the process, and let it pick up the next unhandled thing in the queue, marking that it did so. Then write to another queue the results, and this "GUI" thread then polls against that queue, returning the calculated results back to the user. At this point, you've got multi-threading ... and race conditions.
Another idea: create a REST service that accepts the userFn.ToString() content and execs it. Then in this module, you call out to the other "thread" (service), await the results, and return them.
Security: Yeah, we just flung this out the window. Whether you're executing the user's function directly, calling child_process#fork to do it, or shimming it through a service, you're trusting untrusted code. Sadly, there's really no way around this.

Assuming that security isn't an issue you could do something like this.
// Client side
<input class="func1"> // For example user inputs '(gamestate)=>{return 1}'
<input class="func2">
<input class="func3">
<input class="func4">
<script>
socket.on('syntax_error',function(err){alert(err)});
submit_funcs_strs(){
// Get function strings from user input and then put into array
socket.emit('functions',[document.getElementById('func1').value,document.getElementById('func2').value,...
}
</script>
// Server side
// Socket listener is async
socket.on('functions',(funcs_strs)=>{
let funcs = []
for (let i = 0; i < funcs_str.length;i++){
try {
funcs.push(eval(funcs_strs));
} catch (e) {
if (e instanceof SyntaxError) {
socket.emit('syntax_error',e.message);
return;
}
}
}
// Run algorithm here
}

Angular 4 weird error: query doesn't end when -not- showing a message toast

Here this is, my weirdest error in my whole programming career. I've been struggling through this, yet I can't find what's going on in this code. It just seems not to make any sense in any way.
I'm using the following tools:
Ionic 3
Angular 4
Typescript / ES6
I'm trying to do a method, "assignChat(user)", which assigns a chat to a user. It has to use several APIs, geolocation... it's a big method, actually. That's why I've split it in two parts connected by promises, and used them after, so my method looks pretty much like this:
assignChat(user){
const getLocationName = () => {
return new Promise((resolve,reject) => {
// 30 lines of code
});
}
const assignOrCreateChat= (area) => {
return new Promise((resolve,reject) => {
// 40 lines of code
});
}
const getLocationName = () => {
return new Promise((resolve,reject) => {
// 30 lines of code
});
}
// then I use the inner functions here and write an extra 60-70 lines of code
}
Ok! This works neat. Didn't have much problems with this algorithm after some several testing, although is quite heavy and takes ~0.5s to properly execute, finish it's queries, and show the result.
Thing is... I had some toasts displaying some information, like where you're located. I wanted to remove them, and started by this one, in the inner function getLocationName(). This is the code I want to talk you about:
const getLocationName = () => {
return new Promise( (resolve, reject) => {
const ADDRESS_LEVEL = 2;
this.reverseGeocode(ADDRESS_LEVEL).then( address => {
---> this.toastify("You have been located at: "+address, 1500);
let query = new Parse.Query("PoliticalArea");
// more code
The line I marked with an arrow, is the line which is giving me problems. I mean, you probably think the code fails because of the line, but it's totally the oposite! If I remove that line, the algorithm suddenly stops working and fails to display any result.
The "toastify" method is a quick way I did for myself for displaying toasts. It works well, actually! This is the implementation:
toastify(message, duration){
this.toastCtrl.create({
message: message,
duration: duration
}).present();
}
Not like the most dangerous method. Well, in fact, it seems that the code won't work without it. If I comment the line, or erase it, I never get any result, or any error, from the big algorithm I showed you before. I've got every possible exception catched, although the API connectors don't have timeout, but it's like it gets stuck every time it doesn't display the toast.
I just don't understand what's going on. Seems like a very serious thing the Angular team should look into, in my very honest opinion.
Any idea of what kind of black magic is going there?
UPDATE:
Some further info: when I navigate through the "bugged" view (without the toastify line, and therefore not displaying the chat result), and per example, click in another chat (which pushes a view into the Navigation Controller), it somehow starts showing the chat result I expected. When I pop the new view from the navCtrl, and get back to the page, the expected result is now visible.
Is this some problem with angular watches?

Ok, the solution was not obvious.
It seems that the view was being rendered before the task completed. It was a tough task, so maybe that's the reason why Angular didn't work properly. Tried executing it both in the constructor and in ionViewDidEnter(), though nothing worked.
My final solution was to force component's re-rendering, through ApplicationRef, using the .tick() method at the dead end of my method.
That fixed it all!

How do I simulate multiple simultaneous slow Meteor publications?

I want to simulate multiple slow subscriptions. The client subscribes to two or more publications at the same time, and the result arrive later.
The goal is to be able to see how network latencies and randomness can affect my application (it bugs because I expected a publication to be ready before another, ...).
Using the following short setup for the publications:
// server/foo.js
Meteor.publish('foo', function() {
console.log('publishing foo');
Meteor._sleepForMs(2000);
console.log('waking up foo');
this.ready();
});
// server/bar.js is the same with a different name
Meteor.publish('bar', function() {
console.log('publishing bar');
Meteor._sleepForMs(2000);
console.log('waking up bar');
this.ready();
});
Both publications are slowed down thanks to Meteor._sleepForMs as seen in this amazing answer.
The client then subscribes to each publication:
Meteor.subscribe('bar'); // /client/bar.js
Meteor.subscribe('foo'); // /client/foo.js
From there I expected to see both 'publishing' logs first, then both 'waking up'.
However, this appears in the console:
15:37:45? publishing bar
15:37:47? waking up bar
15:37:47? publishing foo
15:37:49? waking up foo
(I removed some irrelevant fluff like the day)
So obviously it runs in a synchronous fashion. I thought that two things can cause that: the server waitForMs which would entirely block the server (fairly weird), or the client subscription design.
To make sure that it wasn't the server I added a simple heartbeat:
Meteor.setInterval(function() { console.log('beep'); }, 500);
And it did not stop beeping, so the server isn't fully blocked.
I thus suspect that the issue lies within the client subscription model, which maybe waits for the subscription to be ready before calling another..?
Thus, two questions:
Why doesn't my experiment run the way I wanted it to?
How should I modify it to achieve my desired goal (multiple slow publications) ?

Meteor processes DDP messages (which include subscriptions) in a sequence. This ensures that you can perform some action like deleting an object and then inserting it back in the correct order, and not run into any errors.
There is support for getting around this in Meteor.methods using this.unblock() to allow the next available DDP message to process without waiting for the previous one to finish executing. Unfortunately this is not available for Meteor.publish in the Meteor core. You can see discussion (and some workarounds) about this issue here: https://github.com/meteor/meteor/issues/853
There is also a package that adds this functionality to publications:
https://github.com/meteorhacks/unblock/

Why doesn't my experiment run the way I wanted it to?
Meteor._sleepForMs is blocking from the way it is implemented:
Meteor._sleepForMs = function (ms) {
var fiber = Fiber.current;
setTimeout(function() {
fiber.run();
}, ms);
Fiber.yield();
};
Calling it prevents the next line from executing inside the fiber until the duration passes. However, this does not block the Node server from handling other events (i.e. executing another publication) due to the way fiber works.
Here is a talk about Fibers in Meteor: https://www.youtube.com/watch?v=AWJ8LIzQMHY
How should I modify it to achieve my desired goal (multiple slow publications) ?
Try using Meteor.setTimeout to simulate latency asynchronously.
Meteor.publish('foo', function() {
console.log('publishing foo');
var self = this;
Meteor.setTimeout(function () {
console.log('waking up foo');
self.ready();
}, 2000);
});

I believe it's because the publications are blocking.
You can use meteorhacks:unblock to unblock publications:
https://atmospherejs.com/meteorhacks/unblock
It could be a good idea to use this.unblock() at the start of every publication (once you've added meteorhacks:unblock).

Performance heavy algorithms on Node.js

I'm creating some algorithms that are very performance heavy, e.g. evolutionary and artificial intelligence. What matters to me is that my update function gets called often (precision), and I just can't get setInterval to update faster than once per millisecond.
Initially I wanted to just use a while loop, but I'm not sure that those kinds of blocking loops are a viable solution in the Node.js environment. Will Socket.io's socket.on("id", cb) work if I run into an "infinite" loop? Does my code somehow need to return to Node.js to let it check for all the events, or is that done automatically?
And last (but not least), if while loops will indeed block my code, what is another solution to getting really low delta-times between my update functions? I think threads could help, but I doubt that they're possible, my Socket.io server and other classes need to somehow communicate, and by "other classes" I mean the main World class, which has an update method that needs to get called and does the heavy lifting, and a getInfo method that is used by my server. I feel like most of the time the program is just sitting there, waiting for the interval to fire, wasting time instead of doing calculations...
Also, I'd like to know if Node.js is even suited for these sorts of tasks.

You can execute havy algorithms in separate thread using child_process.fork and wait results in main thread via child.on('message', function (message) { });
app.js
var child_process = require('child_process');
var child = child_process.fork('./heavy.js', [ 'some', 'argv', 'params' ]);
child.on('message', function(message) {
// heavy results here
});
heavy.js
while (true) {
if (Math.random() < 0.001) {
process.send({ result: 'wow!' });
}
}

What are the inner workings of the Selenium waitFor mechanism?

I am trying to customize the behavior of Selenium's click command, (via user-extentions.js), by intercepting calls to doClick(locator). Basically I need to delay click actions whenever our application's "busy indicator" is being displayed.
(Now the standard answer for this kind of thing is to insert a waitFor into the script for those situations. Indeed, we currently have zillions of them throughout our scripts. I'm trying to eliminate those.)
Detecting the page element is the trivial part. The tricky part is getting the script to actually wait. My promising looking, but failed attempt looks like this:
var nativeClick = Selenium.prototype.doClick;
Selenium.prototype.doClick = function(locator) {
this.doWaitForCondition("!selenium.browserbot.findElementOrNull('busy-indicator')", 5000);
return nativeClick.call(this, locator);
}
The doWaitForCondition gets called before every click, but it does not wait when the condition evaluates to false. nativeClick always gets called immediately, and so no delay is introduced. I suspect that the doWaitForCondition function doesn't actually do any waiting per se, but rather establishes the conditions for it within the command execution loop. And in this case the click command is already in play, and I'm trying to run a command within a command.
Can somebody shed some light on how Selenium command execution and waitFor works, or offer suggestions on how this might be done?

I have finally solved this. And with an approach that is much better than trying to intercept click processing in its various forms. My refined goal is: to delay execution of script command completion when our application is "busy".
How Selenium command processing works:
Upon completion, each selenium command returns an ActionResult object, (see ActionHandler.prototype.execute). The terminationCondition attribute on this object is a function that determines when it is okay for selenium to proceed to the next command, (TestLoop.prototype.continueTestWhenConditionIsTrue). Basically, selenium repeatedly executes the condition function until it yields true. The result object it quite trivial:
function ActionResult(terminationCondition) {
this.terminationCondition = terminationCondition;
}
Customizing it:
I want to delay execution any time myAppIsBusy() returns true. Of course all of the standard delays need to remain in place as well, like waiting for page loads, and explicit waitFor conditions as scripted. The solution is to redefine the selenium result object in my user-extensions.js, as follows:
function ActionResult(terminationCondition) {
this.terminationCondition = function() {
// a null terminationCondition means okay to continue
return (!terminationCondition || terminationCondition()) && !myAppIsBusy();
}
}
The great thing is that this is at a low enough level that it works for the IDE, as well as for RC.
Note that this does not affect Accessor or Assert command types, which return different result objects. But that should be fine, because those commands don't effect the state of the application.

Well, a look at the java drivers com.thoughtworks.selenium.Wait class reveals this:
public void wait(String message, long timeoutInMilliseconds, long intervalInMilliseconds) {
long start = System.currentTimeMillis();
long end = start + timeoutInMilliseconds;
while (System.currentTimeMillis() < end) {
if (until()) return;
try {
Thread.sleep(intervalInMilliseconds);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
throw new WaitTimedOutException(message);
}
I am not to deep into selenium but I excpect that every waitXXX Method points to this.
So, Selenium is working with Thread.sleep(). While this might not look like an ideal solution it shows at least that you cant make it worse by using Thread.sleep() on your own if neccessary. ;-)