Develop Reference

How to initialize a child process with passed in functions in Node.js

Context
I'm building a general purpose game playing A.I. framework/library that uses the Monte Carlo Tree Search algorithm. The idea is quite simple, the framework provides the skeleton of the algorithm, the four main steps: Selection, Expansion, Simulation and Backpropagation. All the user needs to do is plug in four simple(ish) game related functions of his making:
a function that takes in a game state and returns all possible legal moves to be played
a function that takes in a game state and an action and returns a new game state after applying the action
a function that takes in a game state and determines if the game is over and returns a boolean and
a function that takes in a state and a player ID and returns a value based on wether the player has won, lost or the game is a draw. With that, the algorithm has all it needs to run and select a move to make.
What I'd like to do
I would love to make use of parallel programming to increase the strength of the algorithm and reduce the time it needs to run each game turn. The problem I'm running into is that, when using Child Processes in NodeJS, you can't pass functions to the child process and my framework is entirely built on using functions passed by the user.
Possible solution
I have looked at this answer but I am not sure this would be the correct implementation for my needs. I don't need to be continually passing functions through messages to the child process, I just need to initialize it with functions that are passed in by my framework's user, when it initializes the framework.
I thought about one way to do it, but it seems so inelegant, on top of probably not being the most secure, that I find myself searching for other solutions. I could, when the user initializes the framework and passes his four functions to it, get a script to write those functions to a new js file (let's call it my-funcs.js) that would look something like:
const func1 = {... function implementation...}
const func2 = {... function implementation...}
const func3 = {... function implementation...}
const func4 = {... function implementation...}
module.exports = {func1, func2, func3, func4}
Then, in the child process worker file, I guess I would have to find a way to lazy load require my-funcs.js. Or maybe I wouldn't, I guess it depends how and when Node.js loads the worker file into memory. This all seems very convoluted.
Can you describe other ways to get the result I want?

child_process is less about running a user's function and more about starting a new thread to exec a file or process.
Node is inherently a single-threaded system, so for I/O-bound things, the Node Event Loop is really good at switching between requests, getting each one a little farther. See https://nodejs.org/en/docs/guides/event-loop-timers-and-nexttick/
What it looks like you're doing is trying to get JavaScript to run multiple threads simultaniously. Short answer: can't ... or rather it's really hard. See is it possible to achieve multithreading in nodejs?
So how would we do it anyway? You're on the right track: child_process.fork(). But it needs a hard-coded function to run. So how do we get user-generated code into place?
I envision a datastore where you can take userFn.ToString() and save it to a queue. Then fork the process, and let it pick up the next unhandled thing in the queue, marking that it did so. Then write to another queue the results, and this "GUI" thread then polls against that queue, returning the calculated results back to the user. At this point, you've got multi-threading ... and race conditions.
Another idea: create a REST service that accepts the userFn.ToString() content and execs it. Then in this module, you call out to the other "thread" (service), await the results, and return them.
Security: Yeah, we just flung this out the window. Whether you're executing the user's function directly, calling child_process#fork to do it, or shimming it through a service, you're trusting untrusted code. Sadly, there's really no way around this.

Assuming that security isn't an issue you could do something like this.
// Client side
<input class="func1"> // For example user inputs '(gamestate)=>{return 1}'
<input class="func2">
<input class="func3">
<input class="func4">
<script>
socket.on('syntax_error',function(err){alert(err)});
submit_funcs_strs(){
// Get function strings from user input and then put into array
socket.emit('functions',[document.getElementById('func1').value,document.getElementById('func2').value,...
}
</script>
// Server side
// Socket listener is async
socket.on('functions',(funcs_strs)=>{
let funcs = []
for (let i = 0; i < funcs_str.length;i++){
try {
funcs.push(eval(funcs_strs));
} catch (e) {
if (e instanceof SyntaxError) {
socket.emit('syntax_error',e.message);
return;
}
}
}
// Run algorithm here
}

rxjs pausableBuffered multiple subscriptions

I'm trying to write a websocket rxjs based wrapper.
And I'm struggling with my rxjs understanding.
I have a pause stream which is supposed to pause the pausable buffered streams when an error occures and resume them once i get a "ok" form the websocket.
Somehow only the first subscription on my pauseable buffered streams are fired. From then on only the queue stacks up higher.
I have prepared a jsbin to reproduce the issue.
https://jsbin.com/mafakar/edit?js,console
There the "msg recived" stream only fires for the first subscription. And then the q and observer begin stacking up.
I somehow have the feeling this is about hot and cold obserables but I cannot grasp the issues. I would appreciate any help.
Thank you in advance!

It is not the cold/hot issue. What you do in your onMessage is subscribe, then dispose. The dispose terminates the sequence. The onMessageStream should be subscribed to only once, for example, in the constructor:
this.onmessageStream.subscribe(message => console.log('--- msg --- ', message.data));
The subscribe block, including the dispose should be removed.
Also, note that you used replaySubject without a count, this means that the queue holds all previous values. Unless this is a desired behavior, considered changing it to .replaySubject(1)
Here is a working jsbin.

As #Meir pointed out dispose in a subscribe block is a no no since its behavior is non-deterministic. In general I would avoid the use of Subjects and rely on factory methods instead. You can see a refactored version here: https://jsbin.com/popixoqafe/1/edit?js,console
A quick breakdown of the changes:
class WebSocketWrapper {
// Inject the pauser from the external state
constructor(pauser) {
// Input you need to keep as a subject
this.input$ = new Rx.Subject();
// Create the socket
this._socket = this._connect();
// Create a stream for the open event
this.open$ = Rx.Observable.fromEvent(this._socket, 'open');
// This concats the external pauser with the
// open event. The result is a pauser that won't unpause until
// the socket is open.
this.pauser$ = Rx.Observable.concat(
this.open$.take(1).map(true)
pauser || Rx.Observable.empty()
)
.startWith(false);
// subscribe and buffer the input
this.input$
.pausableBuffered(this.pauser$)
.subscribe(msg => this._socket.send(msg));
// Create a stream around the message event
this.message$ = Rx.Observable.fromEvent(this._socket, 'message')
// Buffer the messages
.pausableBuffered(this.pauser$)
// Create a shared version of the stream and always replay the last
// value to new subscribers
.shareReplay(1);
}
send(request) {
// Push to input
this.input$.onNext(request);
}
_connect() {
return new WebSocket('wss://echo.websocket.org');
}
}
As an aside you should also avoid relying on internal variables like source which are not meant for external consumption. Although RxJS 4 is relatively stable, since those are not meant for public consumption they could be changed out from under you.

How do reactive streams in JS work?

I'm novice in reactive streams and now trying to understand them. The idea looks pretty clear and simple, but on practice I can't understand what's really going on there.
For now I'm playing with most.js, trying to implement a simple dispatcher. The scan method seems to be exactly what I need for this.
My code:
var dispatch;
// expose method for pushing events to stream:
var events = require("most").create(add => dispatch = add);
// initialize stream, so callback in `create` above is actually called
events.drain();
events.observe(v => console.log("new event", v));
dispatch(1);
var scaner = events.scan(
(state, patch) => {
console.log("scaner", patch);
// update state here
return state;
},
{ foo: 0 }
);
scaner.observe(v => console.log("scaner state", v));
dispatch(2);
As I understand, the first observer should be called twice (once per event), and scaner callback and second observer – once each (because they were added after triggering first event).
On practice, however, console shows this:
new event 1
new event 2
scaner state { foo: 0 }
Scaner is never called, no matter how much events I push in stream.
But if I remove first dispatch call (before creating scaner), everything works just as I expected.
Why is this? I'm reading docs, reading articles, but so far didn't found anything even similar to this problem. Where am I wrong in my suggestions?

Most probably, you have studied examples like this from the API:
most.from(['a', 'b', 'c', 'd'])
.scan(function(string, letter) {
return string + letter;
}, '')
.forEach(console.log.bind(console));
They are suggesting a step-by-step execution like this:
Get an array ['a', 'b', 'c', 'd'] and feed its values into the stream.
The values fed are transformed by scan().
... and consumed by forEach().
But this is not entirely true. This is why your code doesn't work.
Here in the most.js source code, you see at line 1340 ff.:
exports.from = from;
function from(a) {
if(Array.isArray(a) || isArrayLike(a)) {
return fromArray(a);
}
...
So from() is forwarding to some fromArray(). Then, fromArray() (below in the code) is creating a new Stream:
...
function fromArray (a) {
return new Stream(new ArraySource(a));
}
...
If you follow through, you will come from Stream to sink.event(0, array[i]);, having 0 for timeout millis. There is no setTimeout in the code, but if you search the code further for .event = function, you will find a lot of additional code that uncovers more. Specially, around line 4692 there is the Scheduler with delay() and timestamps.
To sum it up: the array in the example above is fed into the stream asynchronously, after some time, even if the time seems to be 0 millis.
Which means you have to assume that somehow, the stream is first built, and then used. Even if the program code doesn't look that way. But hey, isn't it always the target to hide complexity :-) ?
Now you can check this with your own code. Here is a fiddle based on your snippet:
https://jsfiddle.net/aak18y0m/1/
Look at your dispatch() calls in the fiddle. I have wrapped them with setTimeout():
setTimeout( function() { dispatch( 1 /* or 2 */); }, 0);
By doing so, I force them also to be asynchronous calls, like the array values in the example actually are.
In order to run the fiddle, you need to open the browser debugger (to see the console) and then press the run button above. The console output shows that your scanner is now called three times:
doc ready
(index):61 Most loaded: [object Object]
(index):82 scanner state Object {foo: 0}
(index):75 scanner! 1
(index):82 scanner state Object {foo: 0}
(index):75 scanner! 2
(index):82 scanner state Object {foo: 0}
First for drain(), then for each event.
You can also reach a valid result (but it's not the same behind scenes) if you use dispatch() synchronously, having them added at the end, after JavaScript was able to build the whole stream. Just uncomment the lines after // Alternative solution, run again and watch the result.

Well, my question appears to be not so general as it sounds. It's just a lib-specific one.
First – approach from topic is not valid for most.js. They argue to 'take a declarative, rather than imperative, approach'.
Second – I tried Kefir.js lib, and with it code from topic works perfect. Just works. Even more, the same approach which is not supported in most.js, is explicitly recommended for Kefir.js.
So, the problem is in a particular lib implementation, not in my head.

How to trace function calls in Marionette

Somewhere in the global scope:
let App = Backbone.Marionette.Application.extend({});
window.Ext = new App();
Inside Module A:
Ext.vent.trigger('Tracked:Email:send', {
from_email: tracking_data.sender,
to_emails: tracking_data.to,
cc_emails: tracking_data.cc,
email_id: tracking_data.email_id || '',
template: tracking_data.template_id || '',
subject: tracking_data.subject
});
Inside Module B:
Ext.vent.on('all', function (evt_name, params) {
// something else...
console.log(params);
}
The object's properties (from_email, to_emails & cc_emails) are undefined when I call console.log in Module B.
I've tried to debug this using console.trace, but the console doesn't show any functions that are involved in changing the object. I've also tried using Object.observe to capture the act of changing the object but no changes are detected.
Can some please teach me some debugging techniques to trace function calls and events in Marionette.
The scenario is:
our codebase is huge.
I'm a new guy at our company so I'm not sure if there are other functions or events that are involved.
I'm the only front end developer right now.

The good news: there's nothing special about Marionette here. Everyday JavaScript debugging techniques will help you out.
The bad news: as you're aware, debugging large, event-driven applications is difficult.
You could take these steps in Chrome dev tools (YMMV with other browsers):
Split the App.trigger line to get a reference to the parameters object:
let options = { from_email: ... }
App.vent.trigger('Tracked:Email:send', options);
Set a breakpoint on Ext.vent.trigger and run your code.
When execution pauses at the breakpoint, check that options contains the expected values. If not, the problem is somewhere before App.vent.trigger. Look up the call stack for any functions that affect tracking_data. You may need to check the Async checkbox if it is populated by asynchronous code.
If options contains the values you expect, carry on...
Whilst execution is paused use the console to save a global reference to your parameters object:
> OPTIONS = options
Add OPTIONS as a watch expression. Expand the item so you can watch changes to its properties.
Step through (including Marionette and Backbone code) until you see the OPTIONS properties change. If you don't see Marionette code when you step into App.vent.trigger you may need to remove Marionette from your black boxed libraries.

array reference javascript angular

i'm trying to reference one item in an array, and i have no idea why this is not working,
console.log($scope.Times);
console.log($scope.Times[0]);
these two lines of code are EXACTLY after eachother, but the output i get from the console is the following..
Output from the console
any ideas why this is not working? the commands are exactly after each other as I mentioned before and in the same function, the variable is global in my controller.
i can add more code if you think it can help, but i don't really understand how..
some more code:
$scope.Times = [];
$scope.getStatus = function(timer){
$http.post('getStatus.php', {timer : timer})
.success(function(response){
$scope.response = response;
if ($scope.response.Running === "0"){
$scope.model = { ItemNumber : $scope.response.Part };
$scope.loadTiming($scope.response.Part);
console.log($scope.Times);
console.log($scope.Times[0]);
}
});
};
$scope.loadTiming = function(itemNumber) {
$http.post('getPartTimings.php', {itemNumber : itemNumber})
.success(function(response){
$scope.selectedTiming = response;
$scope.Times.splice(0);
var i = 0;
angular.forEach($scope.selectedTiming, function(value) {
if (value !== 0)
$scope.Times.push({
"Process" : $scope.procedures[i],
"Duration" : value*60
});
i++;
});
});
};
<?php
$postData = file_get_contents("php://input");
$request = json_decode($postData);
require "conf/config.php";
mysqli_report(MYSQLI_REPORT_STRICT);
try {
$con=mysqli_connect(DBSERVER,DBUSER,DBPASS,DBNAME);
} catch (Exception $exp) {
echo "<label style='font-weight:bold; color:red'>MySQL Server Connection Failed. </label>";
exit;
}
$query = 'SELECT *,
TIME_TO_SEC(TIMEDIFF(NOW(),Timestamp))
FROM live_Timers
WHERE Timer='.$request->timer;
$result = mysqli_query($con, $query);
$data = mysqli_fetch_assoc($result);
echo JSON_ENCODE($data);
thanks for your help.

OK, so more code does help. It looks like you have asynchronous logic happening here. loadTiming is fired, which does a POST and then a splice on the Times array. One console.log could be firing before this POST and the other after. There's no easy way to tell.
One possible fix would be to only log these once the loadTiming async process runs. Return a promise from the loadTiming function and then in the then callback of the promise, log your array.
$scope.getStatus = function(timer){
$http.post('getStatus.php', {timer : timer})
.success(function(response){
$scope.response = response;
if ($scope.response.Running === "0"){
$scope.model = { ItemNumber : $scope.response.Part };
$scope.loadTiming($scope.response.Part).then(function () {
console.log($scope.Times);
console.log($scope.Times[0]);
});
}
});
};
$scope.loadTiming = function(itemNumber) {
return $http.post('getPartTimings.php', {itemNumber : itemNumber})
.success(function(response){
$scope.selectedTiming = response;
$scope.Times.splice(0);
var i = 0;
angular.forEach($scope.selectedTiming, function(value) {
if (value !== 0)
$scope.Times.push({
"Process" : $scope.procedures[i],
"Duration" : value*60
});
i++;
});
});
};

I think your issue is a $scope reference issue.
I would try this:
$scope.vm = {};
$scope.vm.Times = [];
Adding the "." is Angular best practice when attaching to $scope. This is best described here Understanding Scopes

I have experienced a similar situation a while ago, related with this issue.
Since then, I've encountered related issues a bunch of times (AngularJS, due to its cyclic nature seems prone to produce this behaviour).
In your case, using JSON.stringify($scope.Times) might "fix" this.
Context
Usually this happens in this context:
An async call or a expensive DOM manipulation is made.
You make 2 (or more) calls to console.log in between.
The state of the DOM or object is changed
The output shows inconsistent (and strange) results
How
Take this example:
console.log(someObject);
console.log(someObject.property);
After digging a lot (and talking to Webkit developers) this is what I've found:
The second call to console.log is "resolved" first.
Why?
In your case, this has to do how Console handles objects and "expressions" in a different way:
An "expression" is resolved in the time of call, while with objects, a reference to said object is stored instead
Note that expression is used loosely here. You can observe this behaviour in this fiddle
More in depth analysis
Regarding display discrepancies, the behaviour posted above is not the only gotcha with Console. In fact, it is related in how Console works.
Console is an external tool
First you must realize that Console is an external tool and not part of the ECMAScript spec. Implementations differ between browsers and it shouldn't be used in production. It certainly won't work the same for every user.
Console is a non-standard external tool and is not on a standards track.
Console is dynamic
Console as a very dynamic tool. With console you can make assertions (test), time and profile your code, group log entries, remote connect to your server and debug Server Side Code. You can even change code itself, at runtime. So..
Console is not just a static log displayer... Its dynamic nature is one its most features
Console has a slight delay
Being an external dynamic tool, Console works as a watcher process attached to the javascript engine.
This is useful in debugging and among other things prevents Console to inadvertently block the execution of the script. A simple and crude way of thinking about this is picturing console.log as a kind of non-blocking async call. This means that:
With Console, there's a slight delay between 1)call, 2)processing and 3)output.
However, calling Console is not "instant" per se. In fact, by itself, can delay script execution. If you mix this with complex DOM manipulations and events, it can cause weird behaviours.
I've encountered an issue with Chrome, when using MutationObserver and console.log. This happened because the DOM Painting was delaying the update of the DOM object but the event triggered by that DOM change was fired nevertheless. This meant the event callback was executed and finished before the DOM Object was fully updated, resulting in an invalid reference to the DOM object.
Using console.log in the observer caused a brief delay in the callback execution, that, in most of the times, was enough to let the DOM Object update first. This proves that console.log delays code execution.
But even when an invalid reference error occurred, console.log ALWAYS showed a valid object. Since the object couldn't have been changed by code itself, this proves there is a delay delay between the call of console.log and the processing.
Console log order matches the code path
Console log entries order is unaffected by entries update status. In other words,
The order of the log entries reflect the order in which they are called, not their "freshness"
So, if an object is updated, it does not move to the end of the log. (makes sense to me)
Counterintuitive behaviour
This can lead to a number of possible counterintuitive behaviours because one might expect a console.log to be some kind of snapshot of the object, not a reference to it.
For instance, in your case, the object is changed between the the call to console.log and the end of the script.
At the time of calling, $scope.Times is empty, so $scope.Times[0] is undefined.
However, the $scope.Time object is updated posteriorly.
When the Console report is displayed, it shows an updated version of the object.
Fix
In your case, transforming the object in an "expression" can solve the "issue". For instance, you can use JSON.stringify($scope.Times).
Debate
It is debatable if the way console handles objects is a Bug or a Feature. Some propose that, when called with an object, console.log should clone that object making a kind of snapshot. Some argue that storing a reference to the object is preferable, since you can easily create a snapshot yourself if you wish to do so.