I have had some trouble understanding the D3.JS fetch documentation:
My data source is:
20180601 000000;1.168200;1.168240;1.168140;1.168230;0;
20180601 000100;1.168220;1.168230;1.168190;1.168190;0;
20180601 000200;1.168180;1.168180;1.168080;1.168120;0;
20180601 000300;1.168130;1.168160;1.168130;1.168140;0;
where the format is:
%Y%m%d %H%M%S;number1;number2;number3;number4;number5;
My difficulties are:
Adding headers to the data
Dealing with semicolons as the delimiter instead of commas
1) From what I can work out I need to read the file without parsing it, then join a text string to the beginning of the file then finally parse the data.
d3.text(data.csv, function(error, textString){});
var headers = ["date","time","data1","data2"].join("\t");
d3.csv.parse(headers + textString);
2) I can use the dsv format and set the delimiter to semicolons?
d3.dsv(";", "text/plain")
The rough code I ended up with is:
var time_parse = d3.timeParse( '%Y%m%d %H%M%S');
var time_format = d3.timeFormat('%H%M');
d3.text(data.csv, function(error, textString){
var headers = ["time;number1;number2;number3;number4;number5;"].join("\t")
d3.csv.parse(headers + textString)
d3.dsv(";", "text/plain")
data.forEach(function(e,i){
data[i].time = time_parse(e.date);
})
})
Ideally I want the data to look like this when logged:
Time, Number1, Number2, Number3, Number4, Number5
00:00, 1.168200, 1.168240, 1.168140, 1.168230, 0
etc
What is the flaw in my thinking and can anyone offer advice on how to solve my problem and similar problems in the future?
Note: I am new to Javascript and d3 and although I have been able to work through most of the documentation involving drawing svgs, creating axis and scales, transitions etc with no problems, I am struggling to get my head around actually getting data from real sources (e.g the internet) and processing them into something workable. Please heavily critique anything I have said and offer advice, I want to learn.
It's not clear what version of d3 you are using, you reference the fetch API, but some of the code you have looks like d3v3 and v4 in the question code (which could be the problem) which doesn't use the fetch API. In any event, I'll go through v5, but also versions 4 and 3.
In all of these your thoughts look pretty close based on the code blocks you have. We need to:
we read in the dsv as text,
add headers (with an end of line \n),
and run everything through a dsv format function that will use a ; as a delimiter.
no need for d3.csv.parse though as in your question code block
In all the below I drop the date formatting for simplicity (oops, left it in the v5 demo).
Because of the use of d3-fetch module in d3v5, this approach is a bit different than the more closely related d3v3/v4 (closely related in that they both use the d3-request module, otherwise there's a fair bit of difference).
d3-fetch: d3v5
With d3v5, using the d3-fetch module the process could look like:
var dsv = d3.dsvFormat(";");
var headers = ["time;number1;number2;number3;number4;number5;\n"]
d3.text("dsv.dsv").then(function(text) {
var data = dsv.parse(headers+text);
console.log(data);
console.log(data.columns);
})
Example
d3-request: d3v4
There's a bit more flexibility with d3v4 for this.
If we look at the API docs, we see that d3.csv is equivalent to:
d3.request(url)
.mimeType("text/csv")
.response(function(xhr) { return d3.csvParse(xhr.responseText, row); });
(docs)
So if we create a new format with d3.dsvFormat we can run the content through the format and get our data, we can also tack on the headers in this process, all in one step:
var dsv = d3.dsvFormat(";");
var headers = ["time;number1;number2;number3;number4;number5;\n"]
d3.request("dsv.dsv")
.mimeType("text/plain")
.response(function(data) { return dsv.parse(headers + data.response) })
.get(function(data) {
// use data here:
console.log(data);
console.log(data.columns);
});
Example
This might be the more atypical approach, so we could emulate the way I did it with v5 above:
var psv = d3.dsvFormat(";");
var headers = ["time;number1;number2;number3;number4;number5;\n"]
d3.text("dsv.dsv", function(error,data) {
var data = psv.parse(headers + data.reponse)
console.log(data);
console.log(data.columns);
})
Example
d3-request: d3v3
As with the second option in d3v4 above and d3v5, we can parse in text and then run it through the dsv format function (here we only need to account for changes in d3 namespace between v3/v4):
var dsv = d3.dsv(";","text/plain");
var headers = ["time;number1;number2;number3;number4;number5;\n"]
d3.text("dsv.dsv", function(error,text) {
var data = dsv.parse(headers+text);
console.log(data);
// console.log(data.columns) // -> no columns property in v3
})
Example
Note
The ; at the end of each row will create an empty column as a value is expected after it before the next row.
Related
I'm developing a Zapier zap. The source is Google calendar. I collect the event description. It is a json string. I want to turn it into an object to process it in a Zap code (in JavaScript). The doc recommends to use the z.JSON.parse() function. I do it but at run time the error z is not initialized is sent. How to JSON parse a string to make an object ?
I tried to add z = new Z(), but it didn't work.
var eventDescriptorString = {
codeEvt: 'ID-LGC-02/21/2019-testoon_mail',
appCible: 'SIB',
action: 'testoon_mail',
parametre: 'ID-LGC-02/21/2019-testoon_mail-presents'
}
var eventDescriptorObject = z.JSON.parse(inputData.eventDescriptorString);
console.log('action', eventDescriptorObject.action);
output = [{
'action': eventDescriptorObject.action
}];
I expect action to equal 'testoon_mail'
David here, from the Zapier Platform team.
The docs you found are probably for the app scripting environment. While similar, it's got access to a separate set of functions (and the z object itself). Code steps (which is what you're using here) are "plain" javascript environments.
If you change z.JSON.parse -> JSON.parse it should work as expected.
I've been using fast-csv as my converter library for a while now. A problem emerged when a client actually attempted to upload a csv files that actually contained ';' as a delimiter instead of the default ','. The NPM documentation explicitly says that all methods should accept an 'option' (don't understand why not an object) to actually switch these flags. Of course I can always go into the source js file and change the delimiter manually, but I'd really like to understand this documentation since its all part of my growing as a developer, but still I can't manage to grasp it on how to actually use these options (delimiter) when parsing it on my code. If none of you guys can understand it either, maybe you have some recommendations regarding csv parsers on javascript? Maybe a manual script would be more versatile and useful?
Documentation sample from (fast-csv on npm):
All methods accept the following options
objectMode=true: Ensure that data events have an object emitted rather than the stringified version set to false to have a stringified buffer.
headers=false: Set to true if you expect the first line of your CSV to contain headers, alternatly you can specify an array of headers to use. You can also specify a sparse array to omit some of the columns.
ignoreEmpty=false: If you wish to ignore empty rows.
discardUnmappedColumns=false: If you want to discard columns that do not map to a header.
strictColumnHandling=false: If you want to consider empty lines/lines with too few fields as errors - Only to be used with headers=true
renameHeaders=false: If you want the first line of the file to be removed and replaced by the one provided in the headers option - Only to be used with headers=[String]
delimiter=',': If your data uses an alternate delimiter such as ; or \t.
Also, here is a sample code of how it works, and also how I use it (with pipe):
var stream = fs.createReadStream("my.csv");
var csvStream = csv()
.on("data", function(data){
console.log(data);
})
.on("end", function(){
console.log("done");
});
stream.pipe(csvStream);
//or
var csvStream = csv
.parse()
.on("data", function(data){
console.log(data);
})
.on("end", function(){
console.log("done");
});
stream.pipe(csvStream);
PS: I have tried it asking it elsewhere (where the package is published), but had no replies.
The NPM documentation explicitly says that all methods should accept
an 'option' (don't understand why not an object) to actually switch
these flags
The quoted text basically means that all methods accept a so-called options object as their last parameter. You can specify an alternate delimiter by setting the corresponding field in that object.
but I'd really like to understand this documentation since its all
part of my growing as a developer
I strongly recommend looking at the tests whenever you feel something's not clearly explained in the docs. There's actually a test case for the exact scenario you're describing:
it.should("support semicolon delimiters", function (next) {
var actual = [];
csv
.fromPath(path.resolve(__dirname, "./assets/test16.txt"), {headers: true, delimiter: ";"})
.on("data", function (data) {
actual.push(data);
})
.on("error", next)
.on("end", function (count) {
assert.deepEqual(actual, expected14);
assert.equal(count, actual.length);
next();
});
});
I am trying to figure out a way to fetch only the filtered values from a table if a filter is active in Office-JS API.
Right now the only way I have figured to fetch all the table data is from the table range values property:
var table = tables.getItemAt(0);
var tableRange = table.getRange();
tableRange.load("values");
ctx.sync().then(function () {
// This returns all the values from the table, and not only the visible data
var values = tableRange.values;
});
Any ideas on how I can proceed to fetch only the visible values from the table if a filter is active?
From previous experience with Office Interop I have achieved the same by looping through the different Areas of the table range, but I am unable to find the equivalent to Areas in Office-JS.
The upcoming next wave of features as part of Excel JS APIs 1.3 will include a new object "RangeView" that allows you to read only the visible values off the Range object.
Here's a link to the open spec on GitHub - https://github.com/OfficeDev/office-js-docs/tree/ExcelJs_1.3_OpenSpec/excel.
Note that this isn't available just yet, but will be in the near future.
Usage for your case off a table would look like this:
var table = tables.getItemAt(0);
var visibleView = table.getRange().getVisibleView();
ctx.load(visibleView);
ctx.sync().then(function () {
var values = visibleView.values;
});
One way to get only filtered data is through the Binding.getDataAsync method, which takes a filterType parameter.
Office.select("bindings#myTableBinding1").getDataAsync({
coercionType: "table",
filterType: "onlyVisible"
},function(asyncResult){
var values = (asyncResult.value.rows);
});
This code assumes you have already created a binding to the table. If not, you can run the following code first, which uses the table name to call Bindings.addFromNamedItemAsync:
Office.context.document.bindings.addFromNamedItemAsync("Table1","table",{
id: "myTableBinding1"
},function(asyncResult){
// handle errors and call code sample #1
});
Note that the solution above is supported as far back as Excel 2013 because it uses the shared APIs. The Excel-specific API set doesn't yet have the capability to return only unfiltered data.
-Michael Saunders, PM for Office add-ins
I'm relatively new to Javascript and Node and I like to learn by doing, but my lack of awareness of Javascript design patterns makes me wary of trying to reinvent the wheel, I'd like to know from the community if what I want to do is already present in some form or another, I'm not looking for specific code for the example below, just a nudge in the right direction and what I should be searching for.
I basically want to create my own private IFTTT/Zapier for plugging data from one API to another.
I'm using the node module request to GET data from one API and then POST to another.
request supports streaming to do neat things like this:
request.get('http://example.com/api')
.pipe(request.put('http://example.com/api2'));
In between those two requests, I'd like to pipe the JSON through a transform, cherry picking the key/value pairs that I need and changing the keys to what the destination API is expecting.
request.get('http://example.com/api')
.pipe(apiToApi2Map)
.pipe(request.put('http://example.com/api2'));
Here's a JSON sample from the source API: http://pastebin.com/iKYTJCYk
And this is what I'd like to send forward: http://pastebin.com/133RhSJT
The transformed JSON in this case takes the keys from the value of each objects "attribute" key and the value from each objects "value" key.
So my questions:
Is there a framework, library or module that will make the transform step easier?
Is streaming the way I should be approaching this? It seems like an elegant way to do it, as I've created some Javascript wrapper functions with request to easily access API methods, I just need to figure out the middle step.
Would it be possible to create "templates" or "maps" for these transforms? Say I want to change the source or destination API, it would be nice to create a new file that maps the source to destination key/values required.
Hope the community can help and I'm open to any and all suggestions! :)
This is an Open Source project I'm working on, so if anyone would like to get involved, just get in touch.
Yes you're definitely on the right track. There are two stream libs I would point you towards, through which makes it easier to define your own streams, and JSONStream which helps to convert a binary stream (like what you get from request.get) into a stream of parsed JSON documents. Here's an example using both of those to get you started:
var through = require('through');
var request = require('request');
var JSONStream = require('JSONStream');
var _ = require('underscore');
// Our function(doc) here will get called to handle each
// incoming document int he attributes array of the JSON stream
var transformer = through(function(doc) {
var steps = _.findWhere(doc.items, {
label: "Steps"
});
var activeMinutes = _.findWhere(doc.items, {
label: "Active minutes"
});
var stepsGoal = _.findWhere(doc.items, {
label: "Steps goal"
});
// Push the transformed document into the outgoing stream
this.queue({
steps: steps.value,
activeMinutes: activeMinutes.value,
stepsGoal: stepsGoal.value
});
});
request
.get('http://example.com/api')
// The attributes.* here will split the JSON stream into chunks
// where each chunk is an element of the array
.pipe(JSONStream.parse('attributes.*'))
.pipe(transformer)
.pipe(request.put('http://example.com/api2'));
As Andrew pointed out there's through or event-stream, however I made something even easier to use, scramjet. It works the same way as through, but it's API is nearly identical to Arrays, so you can use map and filter methods easily.
The code for your example would be:
DataStream
.pipeline(
request.get('http://example.com/api'),
JSONStream.parse('attributes.items.*')
)
.filter((item) => item.attibute) // filter out ones without attribute
.reduce((acc, item) => {
acc[item.attribute] = item.value;
return acc;
.then((result) => request.put('http://example.com/api2', result))
;
I guess this is a little easier to use - however in this example you do accumulate the data into an object - so if the JSON's are actually much longer than this, you may want to turn it back into a JSONStream again.
So thanks to SO I can pass an object from node to the client, but then getting it into a knockout view model is a bit awkward. These are the steps I have so far (I've included links to the relevant lines as they appear in my github project. Thought the context might help.):
Apply JSON.stringify and pass to the jade file
recipeJSON: JSON.stringify(recipe);
Wrap this in a function in a header script that just parses the JSON and returns the result
script
function getRecipeObject() {
var r = '!{recipeJSON}';
return JSON.parse(r);
}
Call this function and pass the result to a view model constructor
self.recipe = ko.observable(new Recipe(getRecipeObject()));
This works but is there a better way?
Question clarification (Edit): I feel step 2 shouldn't be necessary. Is there a way to directly pass the JSON from node to the Recipe() constructor, without the getRecipeObject() acting as an intermediate step? I tried passing recipeJSON in directly like so
self.recipe = ko.observable(JSON.parse('!{recipeJSON}'));
That doesn't work I think because its not a jade template and has no access to the variable.
According to the answer to this question rendering data into scripts is bad practice and I should instead make an XHR call on page load instead.
Edit
I just saw you linked a github repo! So you're already familiar with most of this...you even have an endpoint set up at /recipe/:id/view, so now I'm really confused...what isn't working out for you? Just the last step of deserialization using ko.utils.*?
Sorry about all the exposition -- I thought this was way more rudimentary than it actually was; I hope no offense taken there!
You really don't want to return a script to execute -- instead, treat this as a DTO: an object that just stores data (no behaviors). An example would be:
{
recipeID: 12,
reviewIDs: [42, 12, 55, 31],
rating: 4.2
recipeName: "A super tasty pie!"
}
This object (representation) is a projection -- a simplified version of the full data stored in the database.
The next step is to create an endpoint to access that data on the server. Let's assume you're using Express:
var app = express();
app.get('/recipes/:recipeID', function(req, res) {
var recipeID = req.params.recipeID;
// It would be cool if this existed, huh?
getRecipeAsync(recipeID, function(recipe) {
res.status(200).json(recipe);
});
});
If you send a GET request to your (hypothetical) application (let's say it's https://localhost:8080/recipes/12), you'll get json representing the (admittedly imaginary) recipe with ID 12.
You can accomplish getting the JSON with jQuery (or any other library that makes XHR nice and pretty)
var recipeID = 12;
$.ajax({
url: "/recipes/" + recipeID,
type: "GET"
}).then(function(recipe) {
console.log("Hey! I got the recipe: %O", recipe);
// Note: you might need to use ko.utils.fromJS(recipe) if the returned
// data is JSON that ISN'T deserialized into an object
var recipeObservable = ko.utils.fromJS(recipe);
});
That's about everything you need to know. Obviously, the devil's in the details, but that's basic idea; let me know if that helps!