I'm relatively new to Javascript and Node and I like to learn by doing, but my lack of awareness of Javascript design patterns makes me wary of trying to reinvent the wheel, I'd like to know from the community if what I want to do is already present in some form or another, I'm not looking for specific code for the example below, just a nudge in the right direction and what I should be searching for.
I basically want to create my own private IFTTT/Zapier for plugging data from one API to another.
I'm using the node module request to GET data from one API and then POST to another.
request supports streaming to do neat things like this:
request.get('http://example.com/api')
.pipe(request.put('http://example.com/api2'));
In between those two requests, I'd like to pipe the JSON through a transform, cherry picking the key/value pairs that I need and changing the keys to what the destination API is expecting.
request.get('http://example.com/api')
.pipe(apiToApi2Map)
.pipe(request.put('http://example.com/api2'));
Here's a JSON sample from the source API: http://pastebin.com/iKYTJCYk
And this is what I'd like to send forward: http://pastebin.com/133RhSJT
The transformed JSON in this case takes the keys from the value of each objects "attribute" key and the value from each objects "value" key.
So my questions:
Is there a framework, library or module that will make the transform step easier?
Is streaming the way I should be approaching this? It seems like an elegant way to do it, as I've created some Javascript wrapper functions with request to easily access API methods, I just need to figure out the middle step.
Would it be possible to create "templates" or "maps" for these transforms? Say I want to change the source or destination API, it would be nice to create a new file that maps the source to destination key/values required.
Hope the community can help and I'm open to any and all suggestions! :)
This is an Open Source project I'm working on, so if anyone would like to get involved, just get in touch.
Yes you're definitely on the right track. There are two stream libs I would point you towards, through which makes it easier to define your own streams, and JSONStream which helps to convert a binary stream (like what you get from request.get) into a stream of parsed JSON documents. Here's an example using both of those to get you started:
var through = require('through');
var request = require('request');
var JSONStream = require('JSONStream');
var _ = require('underscore');
// Our function(doc) here will get called to handle each
// incoming document int he attributes array of the JSON stream
var transformer = through(function(doc) {
var steps = _.findWhere(doc.items, {
label: "Steps"
});
var activeMinutes = _.findWhere(doc.items, {
label: "Active minutes"
});
var stepsGoal = _.findWhere(doc.items, {
label: "Steps goal"
});
// Push the transformed document into the outgoing stream
this.queue({
steps: steps.value,
activeMinutes: activeMinutes.value,
stepsGoal: stepsGoal.value
});
});
request
.get('http://example.com/api')
// The attributes.* here will split the JSON stream into chunks
// where each chunk is an element of the array
.pipe(JSONStream.parse('attributes.*'))
.pipe(transformer)
.pipe(request.put('http://example.com/api2'));
As Andrew pointed out there's through or event-stream, however I made something even easier to use, scramjet. It works the same way as through, but it's API is nearly identical to Arrays, so you can use map and filter methods easily.
The code for your example would be:
DataStream
.pipeline(
request.get('http://example.com/api'),
JSONStream.parse('attributes.items.*')
)
.filter((item) => item.attibute) // filter out ones without attribute
.reduce((acc, item) => {
acc[item.attribute] = item.value;
return acc;
.then((result) => request.put('http://example.com/api2', result))
;
I guess this is a little easier to use - however in this example you do accumulate the data into an object - so if the JSON's are actually much longer than this, you may want to turn it back into a JSONStream again.
Related
Im triying to write a JSON object to a DB with FS but it´s not working as expected. What i want to do is write the JSON data inside this: [] Example:
[
{"name":"jhon"},
{"name":"jhonathan"}
]
Here is the code:
Thanks.
The comment you provided doesn't make much sense, because the JSON data you provided in the post and in the comments is completely different. But I get the gist of it. I guess you have a JSON file containing an array and you want to push new items to it. Let's do this.
The thing is, when you call fs.appendFile, you're only writing to the end of the file. You're not following JSON format by doing so.
You need to do this:
Read file content.
Parse JSON text into an object.
Update the object in memory.
Write object in JSON format back to the file system.
I'll call the synchronous methods for simplicity's sake, but you should be able to convert it to async quite easily.
const path = __dirname + 'db.json'
// Reading items from the file system
const jsonData = fs.readFileSync(path)
const items = JSON.parse(jsonData)
// Add new item to the item list
items.push(newItem)
// Writing back to the file system
const newJsonString = JSON.stringify(items)
fs.writeFileSync(path, newJsonString)
Whenever I read all my data from a database table and receive it as JSON from my API, I get my data like this (unique_name being the primary key in this example):
[{"unique_name":"alice", "age":18, "city":"kansas"},{"unique_name":"bob", "age":20, "city":"chicago"}]
In my javascript application, however, I need my data to be formatted like so:
const myData = {alice: {age:18, city:"kansas"}, bob: {age:20, city:"chicago"}}
I guess this could be done with object mapping of some sort, but I'm afraid this would be too slow with a lot of entries. Is there any clean way to do this?
There is no real problem to map a lot of entries.
If you want more performences you should do it server side (eg: on a nodejs server)
You can simply use a .forEach()
Exemple (not complete so you just get the idea)
const myData = {};
mySQL.forEach(entry => myData[entry.unique_key] = entry)
I am optimizing an api, therefore I need to use only data that is relevant for my analysis. I have created a route that pull out of the objects, but I just need 4 of them (account_manager, fronter, closer, management_fee and sales_date)
I am currently doing this:
you can use .find() with projection to retrieve only the relevant field
try this :
const model_CancellationKPI = await CancellationKPI.find({},{
account_manager:1,
fronter:1,
closer:1,
management_fee:1,
sales_date:1
}).sort({sales_date:-1})
I'm working on tincan JavaScript API. The issue my data format is total change and TinCan have specified a why to pass data along with call. Help me to adjust my data in TinCan Api format. Here is sample data one of my call.
var data = {
"groupId": "groupId",
"groupName": "gNameEncrypt",
"tutorNames": "tutorNames",
"actorNames": "actorNames",
"otherNames": "otherNames"
};
Current what i do i simply decode this data and send it like this.
var actionList = new TinCan(
{
recordStores: [{
endpoint: "http://example.com",
username: username,
password: password,
allowFail: false
}]
});
var action = new TinCan.Agent({
"name": "insert"
});
actionList.getStatements({
'params': {
'agent': action,
'verb': {
'id': $.base64.encode(data)
}
},
'callback': function (err, data) {
console.info(data.more);
var urlref = "http://<?php echo $_SERVER['SERVER_NAME'] . ":" . $_SERVER['SERVER_PORT'] . $uriParts[0] . "?" ?>t=" + data.more.TutorToken;
window.location.href = urlref;
}
});
crypt.finish();
});
There are really two parts here:
need to get data into an xAPI (formerly Tin Can) format, and
the code itself.
In depth,
I think you need to take another look at how xAPI is used in general. Data is stored a JSON "Statement" object that has 3 required properties and various other optional ones. These properties often contain complex objects that are very extensible. It is hard to tell from what you've shown what you are really trying to capture and what the best approach would be. I suggest reading some material about the xAPI statement format. http://experienceapi.com/statements-101/ is a good starting point, and to get at least some coverage of all the possibilities continue with http://experienceapi.com/statements/ .
The code you've listed is attempting to get already stored statements based on two parameters rather than trying to store a statement. The two parameters being "agent" and "verb". In this case We can't tell what the verb is supposed to be since we don't know what data contains, I suspect this isn't going to make sense as a verb which is intended to be the action of a statement. Having said that the fact that the "actor" has a value of action is questionable, as that really sounds more like what a "verb" should contain. Getting the statements right as part of #1 should make obvious how you would retrieve those statements. As far as storing those statements, if you're using the TinCan interface object you would need to use the sendStatement method of that object. But this interface is no longer recommended, the recommended practice is to construct a TinCan.LRS object and interact directly with it, in which case you'd be using the saveStatement method.
I would recommend looking at the "Basic Usage" section of the project home page here: http://rusticisoftware.github.io/TinCanJS/ for more specifics look at the API doc: http://rusticisoftware.github.io/TinCanJS/doc/api/latest/
So thanks to SO I can pass an object from node to the client, but then getting it into a knockout view model is a bit awkward. These are the steps I have so far (I've included links to the relevant lines as they appear in my github project. Thought the context might help.):
Apply JSON.stringify and pass to the jade file
recipeJSON: JSON.stringify(recipe);
Wrap this in a function in a header script that just parses the JSON and returns the result
script
function getRecipeObject() {
var r = '!{recipeJSON}';
return JSON.parse(r);
}
Call this function and pass the result to a view model constructor
self.recipe = ko.observable(new Recipe(getRecipeObject()));
This works but is there a better way?
Question clarification (Edit): I feel step 2 shouldn't be necessary. Is there a way to directly pass the JSON from node to the Recipe() constructor, without the getRecipeObject() acting as an intermediate step? I tried passing recipeJSON in directly like so
self.recipe = ko.observable(JSON.parse('!{recipeJSON}'));
That doesn't work I think because its not a jade template and has no access to the variable.
According to the answer to this question rendering data into scripts is bad practice and I should instead make an XHR call on page load instead.
Edit
I just saw you linked a github repo! So you're already familiar with most of this...you even have an endpoint set up at /recipe/:id/view, so now I'm really confused...what isn't working out for you? Just the last step of deserialization using ko.utils.*?
Sorry about all the exposition -- I thought this was way more rudimentary than it actually was; I hope no offense taken there!
You really don't want to return a script to execute -- instead, treat this as a DTO: an object that just stores data (no behaviors). An example would be:
{
recipeID: 12,
reviewIDs: [42, 12, 55, 31],
rating: 4.2
recipeName: "A super tasty pie!"
}
This object (representation) is a projection -- a simplified version of the full data stored in the database.
The next step is to create an endpoint to access that data on the server. Let's assume you're using Express:
var app = express();
app.get('/recipes/:recipeID', function(req, res) {
var recipeID = req.params.recipeID;
// It would be cool if this existed, huh?
getRecipeAsync(recipeID, function(recipe) {
res.status(200).json(recipe);
});
});
If you send a GET request to your (hypothetical) application (let's say it's https://localhost:8080/recipes/12), you'll get json representing the (admittedly imaginary) recipe with ID 12.
You can accomplish getting the JSON with jQuery (or any other library that makes XHR nice and pretty)
var recipeID = 12;
$.ajax({
url: "/recipes/" + recipeID,
type: "GET"
}).then(function(recipe) {
console.log("Hey! I got the recipe: %O", recipe);
// Note: you might need to use ko.utils.fromJS(recipe) if the returned
// data is JSON that ISN'T deserialized into an object
var recipeObservable = ko.utils.fromJS(recipe);
});
That's about everything you need to know. Obviously, the devil's in the details, but that's basic idea; let me know if that helps!