Develop Reference

Saving data from spawned process into variables in Javascript

I am having issues saving the results from a spawned python process. After converting data into json, I push the data to an array defined within the function before the spawn process is called, but the array keeps returning undefined. I can console.log and show the data correctly, but the array that is returned from the function is undefined. Any input would be greatly appreciated. Thanks in advance.
function sonar_projects(){
const projects = [];
let obj;
let str = '';
const projects_py = spawn('python', ['sonar.py', 'projects']);
let test = projects_py.stdout.on('data', function(data){
let projects = [];
let json = Buffer.from(data).toString()
str += json
let json2 = json.replace(/'/g, '"')
obj = JSON.parse(json2)
console.log(json2)
for(var dat in obj){
var project = new all_sonar_projects(obj[dat].key, obj[dat].name, obj[dat].qualifier, obj[dat].visibility, obj[dat].lastAnalysisDate);
projects.push(project);
}
for (var i= 0; i < projects.length; i++){
console.log(projects[i].key + ' ' + projects[i].name + ' ' + projects[i].qualifier + ' ' + projects[i].visibility + ' ' + projects[i].lastAnalysisDate)
}
console.log(projects)
return projects;
});
}

First of all, going through the NodeJS documentation, we have
Child Process
[child_process.spawn(command, args][, options])
Child Process Class
Stream
Stream Readable Event "data"
Even though projects_py.stdout.on(event_name, callback) accepts an callback, it returns either the EventEmitter-like object, where the events are registered (in this case, stdout that had it's method on called), or the parent element (the ChildProcess named projects_py).
It's because the callback function will be called every time the "data" event occurs. So, if the assign of the event returned the same as the callback function, it'd return only one single time, and then every next happening of the "data" event would be processed by the function, but not would be done.
In this kind of situation, we need a way to collect and compile the data of the projects_py.stdout.on("data", callback) event after it's done.
You already have the collecting part. Now see the other:
Right before you create the on "data" event, we create a promise to encapsulate the process:
// A promise says "we promise" to have something in the future,
// but it can end not happening
var promise = new Promise((resolve, reject) => {
// First of all, we collect only the string data
// as things can come in parts
projects_py.stdout.on('data', function(data){
let json = Buffer.from(data).toString()
str += json
});
// When the stream data is all read,
// we say we get what "we promised", and give it to "be resolved"
projects_py.stdout.on("end", () => resolve(str));
// When something bad occurs,
// we say what went wrong
projects_py.stdout.on("error", e => reject(e));
// With every data collected,
// we parse it (it's most your code now)
}).then(str => {
let json2 = str.replace(/'/g, '"')
// I changed obj to arr 'cause it seems to be an array
let arr = JSON.parse(json2)
//console.log(json2)
const projects = []
// With for-of, it's easier to get elements of
// an object / an array / any iterable
for(var dat of arr){
var project = new all_sonar_projects(
dat.key, dat.name, dat.qualifier,
dat.visibility, dat.lastAnalysisDate
);
projects.push(project);
}
// Template strings `a${variable or expression}-/b`
// are easier to compile things into a big string, yet still fast
for(var i = 0; i < projects.length; i++)
console.log(
`${projects[i].key} ${projects[i].name} ` +
`${projects[i].qualifier} ${projects[i].visibility} ` +
projects[i].lastAnalysisDate
)
console.log(projects)
// Your projects array, now full of data
return projects;
// Finally, we catch any error that might have happened,
// and show it on the console
}).catch(e => console.error(e));
}
Now, if you want to do anything with your array of projects, there are two main options:
Promise (then / catch) way
// Your function
function sonar_projects(){
// The new promise
var promise = ...
// As the working to get the projects array
// is already all set up, you just use it, but in an inner scope
promise.then(projects => {
...
});
}
Also, you can just return the promise variable and do the promise-things with it out of sonar_projects (with then / catch and callbacks).
async / await way
// First of all, you need to convert your function into an async one:
async function sonar_projects(){
// As before, we get the promise
var promise = ...
// We tell the function to 'wait' for it's data
var projects = await promise;
// Do whatever you would do with the projects array
...
}

JavaScript how to determine when AJAX calls are done in a loop

I've spent the last few days trying to tackle this issue and have read all sorts of solutions on StackOverflow and other sites.
I'm building a site that grabs XML data from an outside source and then gets more XML depending on the results to build a network graph. The problem is that I have to essentially wait until this loop of AJAX calls (which may loop into more AJAX calls) is finished before drawing.
I don't know if this just has an especially high cognitive load, but it really has me stumped.
My Code:
function cont(outerNodes) {
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
getXML(node["label"], node["id"]);
}
// I want the code to wait until loop is done, and then draw.
draw(nodes, edges);
}
function getXML(term, fromId) {
var url = someURL;
$.ajax({
url: url,
dataType: "xml",
success: function(result) {
var outerNodes = process(result, fromId, term);
cont(outerNodes);
}
});
}
Note: I understand I may be completely misunderstanding JavaScript synchronicity here, and I very likely am. I have used callbacks and promises successfully in the past, I just can't seem to wrap my head around this one.
If I have not been totally clear, please let me know.
I did try implementing a counter of sorts that is incremented in the process() function, like so:
if (processCount < 15) {
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
getXML(node["label"], node["id"]);
}
} else {
draw(nodes, edges);
}
However, this ended up with several draw() calls which made my performance abysmal.

There are nice new well-supported APIs and language constructs we can use. The Fetch API, await, and for...of loops.
The Fetch API uses Promises. Promises can be awaited. The for...of loop is aware of await and won't continue the loop until the await has passed.
// Loop through, one-at-a-time
for (const node of outerNodes) {
// Make the HTTP request
const res = await fetch(someUrl);
// Do something with the response here...
}
Don't forget a try/catch (which also works with await), and check res.ok.

Brad's answer changes the code to by synchronious and to me that defeats the purpose. If you are constantly waiting on all request to be finished then it could take a while, while normal browsers can handle multiple requests.
The problem you have in your original questions is with scope. Since each call to cont(outerNodes) will trigger it's own scope, it has no idea what are calls are doing. So basically if you call cont(outerNodes) twice, each call will handle it's own list of outernodes and then call draw.
The solution is to share information between the different scopes, which can be done with one, but preferably two global variables: 1 to track active processes and 1 to track errors.
var inProcess = 0;
var nrErrors = 0;
function cont(outerNodes) {
//make sure you have outerNodes before you call outerNodes.length
if (outerNodes) {
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
inProcess++; //add one more in process
getXML(node["label"], node["id"]);
}
}
//only trigger when nothing is in proces.
if (inProcess==0) {
// I want the code to wait until loop is done, and then draw.
draw(nodes, edges);
}
}
function getXML(term, fromId) {
var url = someURL;
$.ajax({
url: url,
dataType: "xml",
success: function(result) {
var outerNodes = process(result, fromId, term);
inProcess--; //one is done
cont(outerNodes);
},
error: function() {
inProcess--; //one is done
nrErrors++; //one more error
cont(null); //run without new outerNodes, to trigger a possible draw
}
});
}
Please note that I track nrErrors but dont do anything with it. You could use it to stop further processing all together or warn the user that the draw is not complete.
[important] Keep in mind that this works in javascript because at best it mimics multithreading. That means the the call to inProcess--; and then right after cont(outerNodes); is always execute directly after eachother.
If you would port this to a true multithreading environment, it could very well be that another scope/version of cont(null); would cut in between the two lines and there would still be multiple draws.

The best way to solve this question should be using either promise or callback.
If you really want to avoid promise or callback(Although i don't know why...)
You can try with a counter.
let processCount = 0;
// Increasing the processCount in getXML callback method
function getXML(term, fromId) {
var url = someURL;
$.ajax({
url: url,
dataType: "xml",
success: function(result) {
processCount++;
var outerNodes = process(result, fromId, term);
cont(outerNodes);
}
});
}
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
getXML(node["label"], node["id"]);
}
while (processCount < outerNodes.length) {
// do nothing, just wait'
}
draw(nodes, edges);

If after testing it many times, you know that it will never take more than say 5 seconds... you can use a setTimeout.
function cont(outerNodes) {
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
getXML(node["label"], node["id"]);
}
// Display a 5 second progress bar here
setTimeout(function(){ draw(nodes, edges); },5000);
}

Reading nested data from Firebase returns something else than an array

Trying to read (and expect) a nested array like this:
var array = [
0: {subArrayy: {...}, title: "Title"},
1: {subArray]: {...}, title: "Title"},
...
However after reading and (!) outputting, the result is fine. My web console shows me the array and everything seems good. BUT doing array.length returns 0. And any iteration returns undefined.
I've tried using ladosh _.toArray thing that I've seen earlier, but it does absolutely nothing.
var locations = []; // Empty array
var ref = db.ref("locations/");
ref.once("value", function(snapshot) {
snapshot.forEach(function(item) {
var itemVal = item.val();
locations.push(itemVal); // Adding new items seems to work at first
});
});
console.log(locations, locations.length);
Output:
chrome output
I expected it to be iterable, in the way I could just use array0 to navigate.

Firebase reads data asynchronously, so that the app isn't blocked while waiting for network traffic. Then once the data is loaded, it calls your callback function.
You can easily see this by placing a few log statements:
console.log("Before starting to load data");
ref.once("value", function(snapshot) {
console.log("Got data");
});
console.log("After starting to load data");
When you run this code the output is:
Before starting to load data
After starting to load data
Got data
This is probably not the order you expected, but it explains exactly why you get a zero length array when you log it. But since the main code continued on straight away, by the time you console.log(locations, locations.length) the data hasn't loaded yet, and you haven't pushed it to the array yet.
The solution is to ensure all code that needs data from the data is either inside the callback, or is called from there.
So this will work:
var locations = []; // Empty array
var ref = db.ref("locations/");
ref.once("value", function(snapshot) {
snapshot.forEach(function(item) {
var itemVal = item.val();
locations.push(itemVal);
});
console.log(locations, locations.length);
});
As will this:
function loadLocations(callback) {
var locations = []; // Empty array
var ref = db.ref("locations/");
ref.once("value", function(snapshot) {
snapshot.forEach(function(item) {
var itemVal = item.val();
locations.push(itemVal);
});
callback(locations);
});
});
loadLocations(function(locations) {
console.log(locations.length);
});
A more modern variant of that last snippet is to return a promise, instead of passing in a callback.
function loadLocations() {
return new Promise(function(resolve, reject) {
var locations = []; // Empty array
var ref = db.ref("locations/");
ref.once("value", function(snapshot) {
snapshot.forEach(function(item) {
var itemVal = item.val();
locations.push(itemVal);
});
resolve(locations);
});
})
});
And you can then call it like this:
loadLocations().then(function(locations) {
console.log(locations.length);
});
Or with modern JavaScript you can use async / await and do:
let locations = await loadLocations()
console.log(locations.length);
Just keep in mind that this last snippet still has the same asynchronous behavior, and the JavaScript runtime (or transpiler) is just hiding it from you.

Placing a sleep function for about 300ms seems to fix the problem. I think it has something to do with syncing, though not entirely sure. It just needs some time to process the query and assign everything, I suppose.

Using await on read functions also seems to help.

Promise version of a "while" loop?

I've come to the realization that since promises in ECMAScript 6 allow for synchronous coding of asynchronous functions, for every promise-laden piece of code there's a synchronous corollary. For instance:
var data = processData(JSON.parse(readFile(getFileName())));
Is the same as:
var data = getFileName()
.then(readFile)
.then(JSON.parse)
.then(processData);
Now for my current use-case I want to write code to pull data from a massive public API. The API is paginated, so in a purely synchronous world I would write something like the following:
var data = [];
var offset = 0;
var total = 10000; // For example - not actually how this would work
while( offset < total ) {
data.concat(getDataFromAPI(offset));
offset = data.length;
}
Now my question is, how would I do this with promises? I could write something like:
var data = [];
var offset = 0;
var total = 10000;
getDataFromAPI(offset)
.then(function(newData){
data.concat(newData);
return getDataFromAPI(data.length);
});
But at this point I'm forced to just chain infinite .thens -- there's no looping logic. I feel like something should be possible using recursion, but I have no idea how to do it.
I'm using BluebirdJS as my promise library, so I have access to all of their helper methods.

I feel like something should be possible using recursion
Exactly. You can name the callback so you can reference it again. As long as the condition isn't met, return a promise from the callback. Otherwise return the final result:
getDataFromAPI(offset)
.then(function next(newData){
data.concat(newData);
var newOffset = data.length;
return newOffset < total ? getDataFromAPI(newOffset).then(next) : data;
})
.then(function(data) {
console.log(data); // final result
});

Wait for all $http requests to complete in Angular JS

I have a page than can make a different number of $http requests depending on the length of a variables, and then I want to send the data to the scope only when all the requests are finished. For this project I do not want to use jQuery, so please do not include jQuery in your answer. At the moment, the data is sent to the scope as each of the requests finish, which isn't what I want to happen.
Here is part of the code I have so far.
for (var a = 0; a < subs.length; a++) {
$http.get(url).success(function (data) {
for (var i = 0; i < data.children.length; i++) {
rData[data.children.name] = data.children.age;
}
});
}
Here is the part that I am sceptical about, because something needs to be an argument for $q.all(), but it is not mentioned on the docs for Angular and I am unsure what it is meant to be.
$q.all().then(function () {
$scope.rData = rData;
});
Thanks for any help.

$http call always returns a promise which can be used with $q.all function.
var one = $http.get(...);
var two = $http.get(...);
$q.all([one, two]).then(...);
You can find more details about this behaviour in the documentation:
all(promises)
promises - An array or hash of promises.
In your case you need to create an array and push all the calls into it in the loop. This way, you can use $q.all(…) on your array the same way as in the example above:
var arr = [];
for (var a = 0; a < subs.length; ++a) {
arr.push($http.get(url));
}
$q.all(arr).then(function (ret) {
// ret[0] contains the response of the first call
// ret[1] contains the second response
// etc.
});