Develop Reference

How to load multiple files with Queue.js and D3.js?

Situation
I am trying to load multiple xml files (located on server) without the need to declare the name of the files hard coded. For this I am trying to use the d3.queue library https://github.com/d3/d3-queue.
I have implemented the xml to force layout to my own needs (https://bl.ocks.org/mbostock/1080941), but there is one crucial flaw namely I need to manually type in the name of the xml file that I want to load...
Reproduce
Given (adjusted example from http://learnjsdata.com/read_data.html) :
queue()
.defer(d3.xml, "/mappings/Customer.hbm.xml")
.defer(d3.xml, "/mappings/Actor.hbm.xml")
.await(analyze);
function analyze(error, Customer, Actor) {
if(error) { console.log(error); }
// do stuff with Customer data, do stuff with Actor data
}
And given my implementation of the processing of an xml:
d3.xml("mappings/Customer.hbm.xml","application/xml", function(error,xml){
if (error) throw error;
// do stuff with the data retrieved from Customer.hbm.xml
});
Question
How do I combine above two snippets in such a way that I dont have to write the locations of the xml hard coded and pass all the parameters to the analyze function? Any nudge in the right direction would be much appreciated.
In psuedocode I have tried to code something like the following (but I cant get it to work):
function to get all names of the xmls from the mappings folder (probably with node.js fs.readdir or fs.readdirSync methods, but I am unsure of how that would work exactly)
for each xml .defer(d3.xml, nameofxml)
pass all the found names as parameters to the analyze function
In Java I would have chosen to do this with a var...args but I dont know how to do it in JS.

There's really two parts to this question:
How do I get a list of server-side files to client-side JavaScript?
Short answer is you don't without having a server-side api that can return that list. Depending on what backend you are using, you write a method that returns a JSON array of the files in your target directory. You call this first, get the response and then process them all with queue:
d3.json('/get/list/of/xml/files', function(error, fileArray){
var q = d3.queue();
fileArray.forEach(function(d){
q = q.defer(d3.xml, d);
});
q.await(analyze);
});
How do a process a variable number of arguments in JavaScript?
This is actually very well supported in JavaScript.
function analyze(error) {
if(error) { console.log(error); }
// skip 0 it's error variable
for (i = 1; i < arguments.length; i++) {
var xml = arguments[i];
...
}
}

Querying a parse table and eagerly fetching Relations for matching

Currently, I have a table named Appointments- on appointments, I have a Relation of Clients.
In searching the parse documentation, I haven't found a ton of help on how to eagerly fetch all of the child collection of Clients when retrieving the Appointments. I have attempted a standard query, which looked like this:
var Appointment = Parse.Object.extend("Appointment");
var query = new Parse.Query(Appointment);
query.equalTo("User",Parse.User.current());
query.include('Rate'); // a pointer object
query.find().then(function(appointments){
let appointmentItems =[];
for(var i=0; i < appointments.length;i++){
var appt = appointments[i];
var clientRelation = appt.relation('Client');
clientRelation.query().find().then(function(clients){
appointmentItems.push(
{
objectId: appt.id,
startDate : appt.get("Start"),
endDate: appt.get("End"),
clients: clients, //should be a Parse object collection
rate : appt.get("Rate"),
type: appt.get("Type"),
notes : appt.get("Notes"),
scheduledDate: appt.get("ScheduledDate"),
confirmed:appt.get("Confirmed"),
parseAppointment:appt
}
);//add to appointmentitems
}); //query.find
}
});
This does not return a correct Clients collection-
I then switched over to attempt to do this in cloud code- as I was assuming the issue was on my side for whatever reason, I thought I'd create a function that did the same thing, only on their server to reduce the amount of network calls.
Here is what that function was defined as:
Parse.Cloud.define("GetAllAppointmentsWithClients",function(request,response){
var Appointment = Parse.Object.extend("Appointment");
var query = new Parse.Query(Appointment);
query.equalTo("User", request.user);
query.include('Rate');
query.find().then(function(appointments){
//for each appointment, get all client items
var apptItems = appointments.map(function(appointment){
var ClientRelation = appointment.get("Clients");
console.log(ClientRelation);
return {
objectId: appointment.id,
startDate : appointment.get("Start"),
endDate: appointment.get("End"),
clients: ClientRelation.query().find(),
rate : appointment.get("Rate"),
type: appointment.get("Type"),
notes : appointment.get("Notes"),
scheduledDate: appointment.get("ScheduledDate"),
confirmed:appointment.get("Confirmed"),
parseAppointment:appointment
};
});
console.log('apptItems Count is ' + apptItems.length);
response.success(apptItems);
})
});
and the resulting "Clients" returned look nothing like the actual object class:
clients: {_rejected: false, _rejectedCallbacks: [], _resolved: false, _resolvedCallbacks: []}
When I browse the data, I see the related objects just fine. The fact that Parse cannot eagerly fetch relational queries within the same call seems a bit odd coming from other data providers, but at this point I'd take the overhead of additional calls if the data was retrieved properly.
Any help would be beneficial, thank you.

Well, in your Cloud code example - ClientRelation.query().find() will return a Parse.Promise. So the output clients: {_rejected: false, _rejectedCallbacks: [], _resolved: false, _resolvedCallbacks: []} makes sense - that's what a promise looks like in console. The ClientRelation.query().find() will be an async call so your response.success(apptItems) is going to be happen before you're done anyway.
Your first example as far as I can see looks good though. What do you see as your clients response if you just output it like the following? Are you sure you're getting an array of Parse.Objects? Are you getting an empty []? (Meaning, do the objects with client relations you're querying actually have clients added?)
clientRelation.query().find().then(function(clients){
console.log(clients); // Check what you're actually getting here.
});
Also, one more helpful thing. Are you going to have more than 100 clients in any given appointment object? Parse.Relation is really meant for very large related collection of other objects. If you know that your appointments aren't going to have more than 100 (rule of thumb) related objects - a much easier way of doing this is to store your client objects in an Array within your Appointment objects.
With a Parse.Relation, you can't get around having to make that second query to get that related collection (client or cloud). But with a datatype Array you could do the following.
var query = new Parse.Query(Appointment);
query.equalTo("User", request.user);
query.include('Rate');
query.include('Clients'); // Assumes Client column is now an Array of Client Parse.Objects
query.find().then(function(appointments){
// You'll find Client Parse.Objects already nested and provided for you in the appointments.
console.log(appointments[0].get('Clients'));
});

I ended up solving this using "Promises in Series"
the final code looked something like this:
var Appointment = Parse.Object.extend("Appointment");
var query = new Parse.Query(Appointment);
query.equalTo("User",Parse.User.current());
query.include('Rate');
var appointmentItems = [];
query.find().then(function(appointments){
var promise = Parse.Promise.as();
_.each(appointments,function(appointment){
promise = promise.then(function(){
var clientRelation = appointment.relation('Clients');
return clientRelation.query().find().then(function(clients){
appointmentItems.push(
{
//...object details
}
);
})
});
});
return promise;
}).then(function(result){
// return/use appointmentItems with the sub-collection of clients that were fetched within the subquery.
});
You can apparently do this in parallel, but that was really not needed for me, as the query I'm using seems to return instantaniously. I got rid of the cloud code- as it didnt seem to provide any performance boost. I will say, the fact that you cannot debug cloud code seems truly limiting and I wasted a bit of time waiting for console.log statements to show themselves on the log of the cloud code panel- overall the Parse.Promise object was the key to getting this to work properly.

Changing data returned from service changes data in service too

When I store data returned from a service in my controller and then edit it, it also changes the data in the service.
JSFiddle Demo
/* The backend connection part and the actual markdown editor JS have been removed because it would just make the code huge and is irrelevant to the problem */
var myApp = angular.module('myApp', []);
// In my app, this service caches blog post data from my server on the client side and returns single posts from it
myApp.factory('PostService', function ($log, $filter, $http) {
var data;
// Just an example for data pulled from server
data = [{
id: 99999,
text: "Hello"
}];
// Function for returning a post out of the data array loaded from the server
var getData = function (id) {
if (id !== undefined) {
var arr = $filter('filter')(data, {
id: id
});
if (arr.length === 0) {
$log.error('PostService:: getData(' + id + '):: Post Not Found');
return 'not_found';
} else {
$log.debug('PostService:: getData(' + id + '):: Post returned');
return arr[0];
}
} else {
return data;
}
};
return {
getData: getData
};
});
function ctrl($log, $scope, PostService) {
var edit = this;
// Sample post id
edit.editingId = 99999;
// "Copy" (apparrently more "bind") the blog post data to edit.data
edit.data = PostService.getData(edit.editingId);
}
This is used for a markdown editor. I wanted to load the data from the service into the controller, then edit it, and give the service the new version on pressing a "Save" button.
If the aforementioned behaviour is correct in the sense of Angular's databinding, what is a better solution to achieve what I want?
Update
Based on PSL's comment and Thibaud Sowa's answer I changed the getData() function to return a copy using angular.copy(). However, it seems not to be possible to copy one object out of an array (like angular.copy(arr[0])), as it will still return the whole array. See the updated JSFiddle.
Update 2
Well, I was dumb. I corrected it in the fiddle. Thank you for your answers.

This is because you are returning an object. In javascript when you do that it's like if you are passing a pointer.
You should use angular.copy to copy the object field by field, like this:
return angular.copy(data);
See the doc here https://docs.angularjs.org/api/ng/function/angular.copy
Response to your update
Well, I edited your fiddle, to show you that you can copy an item of an array. Actually it's seems that every thing works like you want... (Or I didn't understand your need!)
The updated fiddle:
https://jsfiddle.net/thibaudsowa/aybLwa2L/3/

There is a very simple solution to your problem:
If you dont want to change the data you get from the service make a copy
There are plenty of threads on SO discussing the fastes or most elegant way to deep copy a Javascript object. A simple and rather fast solution is using json parse and stringify like this:
var copyOfA = JSON.parse(JSON.stringify(a));
Apply this to the data you get from your service and you are good to go :)

uncaught TypeError: undefined is not a function when using JSON object with name

I'm using D3js with Mongodb and AnguarlJS to display my data. All is good that it works until when I give my JSON array a name. Then angular starts complaining about stuff and I'm not sure why.
this is the original json array that works with this original code
[
{
"defaultCategory":"Happy",
"timesUsed":2704659
},
{
"defaultCategory":"Sad",
"timesUsed":4499890
},
{
"defaultCategory":"Laughter",
"timesUsed":2159981
},
{
"defaultCategory":"Smirk",
"timesUsed":3853788
},
{
"defaultCategory":"Flirty",
"timesUsed":14106543
}
]
d3.json("data.json", function(error, data) {
data.forEach(function(d) {
d.timesUsed =+ d.timesUsed;
});
but when i change the json to this format, it breaks
{
"mymood":[
{
"defaultCategory":"Happy",
"timesUsed":2704659
},
{
"defaultCategory":"Sad",
"timesUsed":4499890
},
{
"defaultCategory":"Laughter",
"timesUsed":2159981
},
{
"defaultCategory":"Smirk",
"timesUsed":3853788
},
{
"defaultCategory":"Flirty",
"timesUsed":14106543
}
]
}
d3.json("data.json", function(error, data) {
data.mymood.forEach(function(d) {
d.timesUsed =+ d.timesUsed;
});
In the chrome console the error is happening at the line data.mymood.foreach line,
but i dont understand why because its exactly returning the same json as if there was no name like this
[object, object, object,object,object]
and the parameter d in the function also returns the same object within the array
edit
Error:
Uncaught TypeError: undefined is not a function
console.log(data) -> Object {mymood: Array[5]}
console.log(data.mymood) -> [Object, Object, Object, Object, Object]
gist for those who are interested in the full code
https://gist.github.com/gwong89/e3a29b64d94ad20256bb

Like Oscar had pointed out, the forEach loop works fine. However, after look closely at your gist, if you try to use the mymood key, make sure you also change line 55 where you were using
var g = svg.selectAll('g')
.data(pie(data))
to
var g = svg.selectAll('g')
.data(pie(data.mymood))
Also on line 76 where you had
var total = d3.sum(data.map(function(d) {
to
var total = d3.sum(data.mymood.map(function(d) {
I grabbed your project repo and tried these out on my local environment, and I can render the pie chart and popup without seeing any errors.

Your code seems to be working fine (from what I've tested). I will suggest to try debugging your code or if you feel more comfortable try doing many console.log(...) as possible and keep checking your browser console from the beginning (I have nothing else to say). Try to also use Developer Tools from Chrome which is a better option.
I've tried to replicate your context by using d3.js library and uploaded a json file with your data to my Dropbox just to be able to perform an ajax request and everything is fine again (also new JSON structure is valid). Here's what I did, probably could help you to learn something new (see below).
Possible hints/suggestions to fix your issue:
According to the question title, it seems that data.mymood is undefined so you are not able to do a forEach of something that doesn't exist. Try validating things and avoid null pointers (see example).
JSON structures are valid so that's not the problem.
Syntax seems to be valid from what I've tested.
Check if there's a conflict between libraries, try to do some research about using all the libraries needed together (not sure if this is your situation but could happen).
Check your browser console and do some debugging as the first paragraph says.
Check if your request is not timing out or something else is happening, check the Network tab or log your requests in browser console using Developer Tools from Chrome (just saying).
Live Demo: http://jsfiddle.net/29f6n8L7/
d3.json('https://dl.dropboxusercontent.com/u/15208254/stackoverflow/data.json', function(data) {
// Some ways to avoid null pointers
var obj = (data || {}), // If 'data' is null, initialize as an Object
moods = (obj.mymood || []), // If 'obj.mymood' is null, initialize as an Array
time = 0,
mood;
// Same way to get 'mymood' from the object
console.log('Your obj.mymood contains: %o', obj.mymood);
console.log('Your obj["mymood"] contains: %o', obj['mymood']);
/**
* I will suggest to use a traditional loop instead of 'forEach'.
* Please read: http://stackoverflow.com/a/9329476/1178686
* for a better explanation
*/
for (var i = 0; i < moods.length; ++i) {
mood = moods[i];
if (mood) {
time += mood.timesUsed;
}
}
// Output
console.log('Sum of timesUsed is: %s', time);
});

Arrays have a built in for each in es5, objects do not.

Passing objects from NodeJS to client and then into KnockoutJS viewmodel

So thanks to SO I can pass an object from node to the client, but then getting it into a knockout view model is a bit awkward. These are the steps I have so far (I've included links to the relevant lines as they appear in my github project. Thought the context might help.):
Apply JSON.stringify and pass to the jade file
recipeJSON: JSON.stringify(recipe);
Wrap this in a function in a header script that just parses the JSON and returns the result
script
function getRecipeObject() {
var r = '!{recipeJSON}';
return JSON.parse(r);
}
Call this function and pass the result to a view model constructor
self.recipe = ko.observable(new Recipe(getRecipeObject()));
This works but is there a better way?
Question clarification (Edit): I feel step 2 shouldn't be necessary. Is there a way to directly pass the JSON from node to the Recipe() constructor, without the getRecipeObject() acting as an intermediate step? I tried passing recipeJSON in directly like so
self.recipe = ko.observable(JSON.parse('!{recipeJSON}'));
That doesn't work I think because its not a jade template and has no access to the variable.

According to the answer to this question rendering data into scripts is bad practice and I should instead make an XHR call on page load instead.

Edit
I just saw you linked a github repo! So you're already familiar with most of this...you even have an endpoint set up at /recipe/:id/view, so now I'm really confused...what isn't working out for you? Just the last step of deserialization using ko.utils.*?
Sorry about all the exposition -- I thought this was way more rudimentary than it actually was; I hope no offense taken there!
You really don't want to return a script to execute -- instead, treat this as a DTO: an object that just stores data (no behaviors). An example would be:
{
recipeID: 12,
reviewIDs: [42, 12, 55, 31],
rating: 4.2
recipeName: "A super tasty pie!"
}
This object (representation) is a projection -- a simplified version of the full data stored in the database.
The next step is to create an endpoint to access that data on the server. Let's assume you're using Express:
var app = express();
app.get('/recipes/:recipeID', function(req, res) {
var recipeID = req.params.recipeID;
// It would be cool if this existed, huh?
getRecipeAsync(recipeID, function(recipe) {
res.status(200).json(recipe);
});
});
If you send a GET request to your (hypothetical) application (let's say it's https://localhost:8080/recipes/12), you'll get json representing the (admittedly imaginary) recipe with ID 12.
You can accomplish getting the JSON with jQuery (or any other library that makes XHR nice and pretty)
var recipeID = 12;
$.ajax({
url: "/recipes/" + recipeID,
type: "GET"
}).then(function(recipe) {
console.log("Hey! I got the recipe: %O", recipe);
// Note: you might need to use ko.utils.fromJS(recipe) if the returned
// data is JSON that ISN'T deserialized into an object
var recipeObservable = ko.utils.fromJS(recipe);
});
That's about everything you need to know. Obviously, the devil's in the details, but that's basic idea; let me know if that helps!