Develop Reference

Transforming JSON in a node stream with a map or template

I'm relatively new to Javascript and Node and I like to learn by doing, but my lack of awareness of Javascript design patterns makes me wary of trying to reinvent the wheel, I'd like to know from the community if what I want to do is already present in some form or another, I'm not looking for specific code for the example below, just a nudge in the right direction and what I should be searching for.
I basically want to create my own private IFTTT/Zapier for plugging data from one API to another.
I'm using the node module request to GET data from one API and then POST to another.
request supports streaming to do neat things like this:
request.get('http://example.com/api')
.pipe(request.put('http://example.com/api2'));
In between those two requests, I'd like to pipe the JSON through a transform, cherry picking the key/value pairs that I need and changing the keys to what the destination API is expecting.
request.get('http://example.com/api')
.pipe(apiToApi2Map)
.pipe(request.put('http://example.com/api2'));
Here's a JSON sample from the source API: http://pastebin.com/iKYTJCYk
And this is what I'd like to send forward: http://pastebin.com/133RhSJT
The transformed JSON in this case takes the keys from the value of each objects "attribute" key and the value from each objects "value" key.
So my questions:
Is there a framework, library or module that will make the transform step easier?
Is streaming the way I should be approaching this? It seems like an elegant way to do it, as I've created some Javascript wrapper functions with request to easily access API methods, I just need to figure out the middle step.
Would it be possible to create "templates" or "maps" for these transforms? Say I want to change the source or destination API, it would be nice to create a new file that maps the source to destination key/values required.
Hope the community can help and I'm open to any and all suggestions! :)
This is an Open Source project I'm working on, so if anyone would like to get involved, just get in touch.

Yes you're definitely on the right track. There are two stream libs I would point you towards, through which makes it easier to define your own streams, and JSONStream which helps to convert a binary stream (like what you get from request.get) into a stream of parsed JSON documents. Here's an example using both of those to get you started:
var through = require('through');
var request = require('request');
var JSONStream = require('JSONStream');
var _ = require('underscore');
// Our function(doc) here will get called to handle each
// incoming document int he attributes array of the JSON stream
var transformer = through(function(doc) {
var steps = _.findWhere(doc.items, {
label: "Steps"
});
var activeMinutes = _.findWhere(doc.items, {
label: "Active minutes"
});
var stepsGoal = _.findWhere(doc.items, {
label: "Steps goal"
});
// Push the transformed document into the outgoing stream
this.queue({
steps: steps.value,
activeMinutes: activeMinutes.value,
stepsGoal: stepsGoal.value
});
});
request
.get('http://example.com/api')
// The attributes.* here will split the JSON stream into chunks
// where each chunk is an element of the array
.pipe(JSONStream.parse('attributes.*'))
.pipe(transformer)
.pipe(request.put('http://example.com/api2'));

As Andrew pointed out there's through or event-stream, however I made something even easier to use, scramjet. It works the same way as through, but it's API is nearly identical to Arrays, so you can use map and filter methods easily.
The code for your example would be:
DataStream
.pipeline(
request.get('http://example.com/api'),
JSONStream.parse('attributes.items.*')
)
.filter((item) => item.attibute) // filter out ones without attribute
.reduce((acc, item) => {
acc[item.attribute] = item.value;
return acc;
.then((result) => request.put('http://example.com/api2', result))
;
I guess this is a little easier to use - however in this example you do accumulate the data into an object - so if the JSON's are actually much longer than this, you may want to turn it back into a JSONStream again.

breeze.js how do I share metadata between 2 entity managers

I have 2 entity managers:
var mgr1 = new breeze.EntityManager('api/app');
var mgr2 = new breeze.EntityManager('api/app');
Right now I am getting the metadata for each one separately, although the metadata is exactly the same. I am calling the fetch method explicitly to control the timing of when the metadata is loaded.
mgr1.fetchMetadata();
mgr2.fetchMetadata();
I've read that I can share the metadata between the 2 managers but I have not found an example. From what I've read, I think I can specify the metadata in the constructor of the 2nd manager that references the 1st manager's metadata, but not sure what that would look like. So my code would look something like this:
var mgr1 = new breeze.EntityManager('api/app');
mgr1.fetchMetadata();
var mgr2 = new breeze.EntityManager({ serviceName: 'api/app', metadata: WHAT_GOES_HERE});
I know I will also have to sequnece this with promises so the 2nd manager isn't constructed before the 1st managers has it's metadata loaded.
Am I on the right path with this? My goal is to eliminate the extra bandwidth to load the metadata for the 2nd manager. thanks

You don't actually have to fetch the metadata to share the same MetadataStore. The following two statements are a pretty crisp approach:
var em1 = new breeze.EntityManager('api/app');
var em2 = em1.createEmptyCopy();
I'm not trying to be clever. My point is that a MetadataStore, which is a container of metadata, is available immediately after EntityManager creation and is well defined prior to holding any metadata whatsoever.
The createEmptyCopy() method "clones" the manager without copying its entity cache contents. The copied attributes include the manager's MetadataStore and its DataService.
Because the managers share the same MetadataStore, fetching metadata with either manager will do the trick.
Check out the Breeze API documentation for EntityManager and MetadataStore.

I have never used breeze.js before, but from what I gather from the documentation (http://www.breezejs.com/sites/all/apidocs/classes/EntityManager.html), something like this should work:
var mgr1 = new breeze.EntityManager('api/app');
mgr1.fetchMetadata();
var mgr2 = new breeze.EntityManager({
serviceName: 'api/app',
metadataStore: mgr1.metadataStore
});
Of course mgr2 should be set up after the mgr1.fetchMetadata promise is fulfilled, as you already say in your question.

Angular.js accessing and displaying nested models efficiently

I'm building a site at the moment where there are many relational links between data. As an example, users can make bookings, which will have booker and bookee, along with an array of messages which can be attached to a booking.
An example json would be...
booking = {
id: 1,
location: 'POST CDE',
desc: "Awesome stackoverflow description."
booker: {
id: 1, fname: 'Lawrence', lname: 'Jones',
},
bookee: {
id: 2, fname: 'Stack', lname: 'Overflow',
},
messages: [
{ id: 1, mssg: 'For illustration only' }
]
}
Now my question is, how would you model this data in your angular app? And, while very much related, how would you pull it from the server?
As I can see it I have a few options.
Pull everything from the server at once
Here I would rely on the server to serialize the nested data and just use the given json object. Downsides are that I don't know what users will be involved when requesting a booking or similar object, so I can't cache them and I'll therefore be pulling a large chunk of data every time I request.
Pull the booking with booker/bookee as user ids
For this I would use promises for my data models, and have the server return an object such as...
booking = {
id: 1,
location: 'POST CDE',
desc: "Awesome stackoverflow description."
booker: 1, bookee: 2,
messages: [1]
}
Which I would then pass to a Booking constructor, which would resolve the relevant (booker,bookee and message) ids into data objects via their respective factories.
The disadvantages here are that many ajax requests are used for a single booking request, though it gives me the ability to cache user/message information.
In summary, is it better practise to rely on a single ajax request to collect all the nested information at once, or rely on various requests to 'flesh out' the initial response after the fact.
I'm using Rails 4 if that helps (maybe Rails would be more suited to a single request?)

I'm going to use a system where I can hopefully have the best of both worlds, by creating a base class for all my resources that will be given a custom resolve function, that will know what fields in that particular class may require resolving. A sample resource function would look like this...
class Booking
# other methods...
resolve: ->
booking = this
User
.query(booking.booker, booking.bookee)
.then (users) ->
[booking.booker, booking.bookee] = users
Where it will pass the value of the booker and bookee fields to the User factory, which will have a constructor like so...
class User
# other methods
constructor: (data) ->
user = this
if not isNaN(id = parseInt data, 10)
User.get(data).then (data) ->
angular.extend user, data
else angular.extend this, data
If I have passed the User constructor a value that cannot be parsed into a number (so this will happily take string ids as well as numerical) then it will use the User factorys get function to retrieve the data from the server (or through a caching system, implementation is obviously inside the get function itself). If however the value is detected to be non-NaN, then I'll assume that the User has already been serialized and just extend this with the value.
So it's invisible in how it caches and is independent of how the server returns the nested objects. Allows for modular ajax requests and avoids having to redownload unnecessary data via its caching system.
Once everything is up and running I'll write some tests to see whether the application would be better served with larger, chunked ajax requests or smaller modular ones like above. Either way this lets you pass all model data through your angular factories, so you can rely on every record having inherited any prototype methods you may want to use.

Fetch Backbone Collection by Model IDs list

I have a REST API serving a few URLs:
/rest/messages
provides all messages. A message is a JSON/Backbone Model
{
title: 'foo',
body : 'bar'
}
To get a single message I have:
/rest/messages/:id
Is it possible to fetch a Backbone Collection using message IDs array? I don't want the whole message list, but just a few messages I specify by ID.
I could fetch Models one-by-one and fill up the Collection, but I'm wondering if Backbone has a cleaner way to do this.
Thanks

According to documentation, you can pass ajax options to the fetch call. So, you can pass ids as data attribute to the fetch call being done and based on it, return the respective models from the server.
For example (when doing fetch),
collection.fetch({
data : {
message_ids : [1, 3, 5] // array of the message ids you want to retrieve as models
}
})
This message_id array will be accessible as parameters (not sure of the name in your case) in the server code being executed at /rest/messages, from there you can return only specific models based on ids you receive as message_ids. The only thing you need is, client side must be aware of the ids of all the message models it needs.
You can use any data structure instead of array to send message_ids.

The url property of collection reference to the collection location on the server. When you use fetch, backbone uses that url.
The url property can be also a function that returns the url. So you can do something like that:
var ids = [1,2,3]
var messages = new MessegecCollection();
messages.url = function() {
return "/rest/messages/"+ids.join("-"); //results "/rest/messages/1-2-3"
}
messages.fetch();
You can also create a method in your collection that generated and set the url, or even fetchs a set of models.
Now all you have to do is to support this url: /rest/messages/1-2-3
Hope this helps!

Django Dynamic Drop-down List from Database

I wanted to develop a Django app and one of the functionalities I'd like to have is dynamic drop-down lists...specifically for vehicle makes and models...selecting a specific make will update the models list with only the models that fall under that make....I know this is possible in javascript or jQuery (this would be my best choice if anyone has an answer) but I don't know how to go about it.
Also, I'd want the make, model, year and series to be common then the other attributes like color, transmission etc to be variables so that one needs only enter the make, model, year, and series only for a new vehicle. Any ideas would be highly appreciated.

The 3 things you mention being common, make, model, year, would be the 3 input values. When given to the server, an object containing the details would be returned to the calling page. That page would parse the object details (using JavaScript), and update the UI to display them to the user.
From the Django side, there needs to be the facilities to take the 3 inputs, and return the output. From the client-side, there needs to be the facilities to pass the 3 inputs to the server, and then appropriately parse the server's response.
There is a REST api framework for Django that makes it rather easy to add the "api" mentioned above -- Piston. Using Piston, you'd simply need to make a URL for that resource, and then add a handler to process it. (you'll still need to skim the Piston documentation, but this should give you an idea of what it looks like)
urls.py:
vehicle_details = Resource(handler=VehicleDetails)
url(r'^vehicle/(?<make>.*)/(?<model>.*)/(?<year\d{2,4}/(?P<emitter_format>[a-z]{1,4}), vehicle_details, name='vehicle_details'),
handler.py:
class VehicleDetails(BaseHandler):
methods_allowed = ('GET',)
model = Vehicles #whatever your Django vehicle model is
def read(self, request, *args, **kwargs):
# code to query the DB and select the options
# self.model.objects.filter()...
# Build a custom object or something to return
return custom_object
This simply sets up the url www.yoursite.com/vehicle/[make]/[model]/[year]/json to return a custom data object in JSON for jquery to parse.
On the client side, you could use jquery to setup an event (bind) so that when all 3 drop downs have a value selected, it will execute a $.get() to the api URL. When it gets this result back, it passes it into the Jquery JSON parser, and gives the custom object, as a javascript object. That object could then be used to populate more drop down menus.
(Big warning, I just wrote the following off the top of my head, so it's not meant to be copy and pasted. It's just for the general idea.)
<script type="text/javascript">
// On document load
$(function() {
$('#dropdown_make').bind('change', checkForValues());
$('#dropdown_model').bind('change', checkForValues());
$('#dropdown_year').bind('change', checkForValues());
});
function checkForValues() {
if ($('#dropdown_make').val() && $('#dropdown_model').val() && $('#dropdown_year').val())
updateOptions();
}
function updateOptions() {
url = '/vehicle/';
url += $('#dropdown_make').val() + '/';
url += $('#dropdown_model').val() + '/';
url += $('#dropdown_year').val() + '/';
url += 'json/';
$.get(url, function(){
// Custom data object will be returned here
})
}
</script>

This is uncanny: Dynamic Filtered Drop-Down Choice Fields With Django
His question:
"Here is the situation: I have a database with car makes and models. When a user selects a make, I want to update the models drop-down with only the models associated with that make. ... Therefore I want to use Ajax to populate the data."
You're not the same guy? :)