So currently, I am getting data via TCP connection and we can tell if a string is a valid JSON object using something like:
let body = '';
client.on('data', (chunk) => {
body += chunk.toString();
try {
let data = JSON.parse(body);
// data is valid json
}
catch (e) {}
});
While this works, can we get all valid Objects in a chunk? Let's say we have
const chunk = // an incomplete object like { "list": [ { "id": 1 }, { "id":
So in the sample, how can we get a valid object in a string? I'm trying to get { "id": 1 } because it's the only valid object in the string.
What you need is a streaming JSON parser, such as this one: https://github.com/chrisdickinson/json-parse-stream
As you probably already know, unfinished objects are invalid and somewhat arbitrary. So, you'll have to choose how to handle these situations. That is, only if you're in control of the data and have decided how to handle these situations can you know what to do. The streaming JSON parser will emit objects and such as they're passed in, which should help you get around any incomplete objects.
Related
Given the following log object:
{
"message": "login: error {\"error\":{\"message\":\"Network Error\",\"name\":\"Error\",\"stack\":\"Error: Network Error\\n at something (somewhere)\\n at something (somewhere)\",\"config\":{\"url\":\"/a/place\",\"method\":\"get\",\"headers\":{\"Accept\":\"application/json, text/plain, */*\",\"Authorization\":\"bla blablabla\",\"X-Amzn-Trace-Id\":\"yadiyadiyadi\"},\"baseURL\":\"verygoodplace"}}}",
"level": "warning",
"sessionId": "blablabla"
}
How can I remove the message.headers.Authorization entry completely?
Since it appears inside a string, I can't (directly) use lodash unset, and I somehow need to alter the string.
I would recommend to clean message a bit so it can be parsed with JSON.parse(). It looks like discarding text before the first { will be all that's needed.
Parsing will create a JS object that is easy to manipulate, after which you can use JSON.stringify() to convert it back to a similar string as what you started with.
It may seem like a lot of steps, but doing this kind of string manipulation directly could be an even bigger pain in the you-know-where.
Working demo:
const logObj = {
"message": "login: error {\"error\":{\"message\":\"Network Error\",\"name\":\"Error\",\"stack\":\"Error: Network Error\\n at something (somewhere)\\n at something (somewhere)\",\"config\":{\"url\":\"/a/place\",\"method\":\"get\",\"headers\":{\"Accept\":\"application/json, text/plain, */*\",\"Authorization\":\"bla blablabla\",\"X-Amzn-Trace-Id\":\"yadiyadiyadi\"},\"baseURL\":\"verygoodplace\"}}}",
"level": "warning",
"sessionId": "blablabla"
}
const index = logObj.message.indexOf("{");
const jsonText = logObj.message.substring(index);
const parsed = JSON.parse(jsonText);
delete parsed.error.config.headers.Authorization; // remove unwanted node
console.log("The cleaned message:");
console.log(JSON.stringify(parsed, undefined, 2)); // print with indentation
const prefix = logObj.message.substring(0, index);
logObj.message = prefix + JSON.stringify(parsed);
console.log("The updated logObj:");
console.log(JSON.stringify(logObj, undefined, 2)); // print with indentation
Note - to make this work I had to change verygoodplace" to verygoodplace\", it looks like you made an error when preparing the log object for use in the question.
I'm making an ajax post request to a super simple python function that takes a student's name and spits out a url that corresponds to it. Currently, the Python function passes this back in json and looks like so when console.log(JSON.stringify(response)) is called:
{"readyState":4,"responseText”:”\ {\”studentURL\”: \”https://prepacademy.jackjohnson.com\”} ”,”responseJSON”: {“studentURL”:”https://prepacademy.jackjohnson.com”},”status":200,"statusText":"OK"}
I was wondering how do I take this larger chunk of information and filter it so that I would only get the https://prepacademy.jackjohnson.com part?
response is a JavaScript Object of which you can access the properties using either Dot-notation or bracket-notation, like so:
let response = {
"readyState": 4,
"responseText": "\ {\"studentURL\": \"https://prepacademy.jackjohnson.com\"} ",
"responseJSON": {
"studentURL": "https://prepacademy.jackjohnson.com"
},
"status": 200,
"statusText": "OK"
};
// dot-notation
console.log(response.responseJSON.studentURL)
// bracket-notation (allows for computed paths)
console.log(response["responseJSON"]["studentURL"])
response.responseJSON.studentURL
I am trying to append the user details from the registration form to the json file so that the user-details can be used for authentication. the problem is i am not able append to the json file in correct format.The code i have tried so far is,
var filename= "./user_login.json";
var contents = fs.readFileSync(filename);
var jsonContent = JSON.parse(contents);
//sample data
var data =[
{
"try" : "till success"
}
];
jsonContent.push(data);
fs.writeFileSync(filename,jsonContent);
I have tried different methods that i found by googling and nothing worked so far. I want the data to be stored in correct format. Most of the times i got this error like object has no push function. So what is the alternative to that?
The correct format i am looking for is ,
[
user1-details : {
//user1 details
},
user2-deatils : {
}//So on
]
Object has no push function, arrays do. Your json is invalid too, it should be an array:
[ // here
{
//user1 details
},
{
//So on
}
] // and here
Now, you can use push(). However, data is an array, if you want an array of objects in your json file, it should be a simple object:
var data = {
"try" : "till success"
};
You also have to stringify the object before writing it back to the file:
fs.writeFileSync(filename, JSON.stringify(jsonContent));
You should consider using something like node-json-db, it will take care of reading/writing the file(s) and it gives you helper functions (save(), push()...).
I'm using node-soap with a service and everything works but I need to send an array of ints and I find that I can only send the first one because I can't find the correct way to build a JS object to represent this array.
I've been looking at similar questions but I couldn't find the answer to my question.
I need to generate a XML property like the following one:
<ns1:ArrayOfInts>
<!--Zero or more repetitions:-->
<arr:int>2904</arr:int>
<arr:int>3089</arr:int>
<arr:int>4531</arr:int>
</ns1:ArrayOfInts>
by passing an object that contains the array:
soapObject = {
somefields,
"ns1:ArrayOfInts": {
Something goes here
},
};
Any idea how to create the JS object?
I had the same problem and used $xml property to add raw XML to the request and attributes to set the arr namespace:
var fields = [2904, 3089, 4531];
soapObject.arrayOfInts = {
attributes: {
'xmlns:arr': 'http://schemas.microsoft.com/2003/10/Serialization/Arrays'
},
$xml: fields.map(function(value) {
return '<arr:int>' + value + '</arr:int>';
}).join('')
};
This code will generate the following request:
<ns1:arrayOfInts xmlns:arr="http://schemas.microsoft.com/2003/10/Serialization/Arrays">
<arr:int>2904</arr:int>
<arr:int>3089</arr:int>
<arr:int>4531</arr:int>
</ns1:arrayOfInts>
I'd like to break this into smaller, tighter questions but I don't know what I don't know enough to do that yet. So hopefully a can get specific answers to help do that.
The scope of the solution requires receiving & parsing a lot of records, 2013 had ~17 million certificate(s) transactions while I'm only interested in very small subsets of the order 40,000 records.
In pseudo code:
iterate dates(thisDate)
send message to API for thisDate
receive JSONS as todaysRecords
examine todaysRecords to look for whatever criteria match inside the structure
append a subset of todaysRecords to recordsOut
save recordsOut to a SQL/CSV file.
There's a large database of Renewable Energy Certificates for the use under the Australian Government RET Scheme called the REC Registery and as well as the web interface linked to here, there is an API provided that has a simple call logic as follows
http://rec-registry.gov.au/rec-registry/app/api/public-register/certificate-actions?date=<user provided date> where:
The date part of the URL should be provided by the user
Date format should be YYYY-MM-DD (no angle brackets & 1 date limit)
A JSON is returned (with potentially 100,000s of records on each day).
The API documentation (13pp PDF) is here, but it mainly goes into explaining the elements of the returned structure which is less relevant to my question. Includes two sample JSON responses.
While I know some Javascript (mostly not in a web context) I'm not sure how send this message within a script and figure I'd need to do it server side to be able to process (filter) the returned information and then save the records I'm interested in. I'll have no issue parsing the JSON (if i can use JS) and copying the objects I wish to save I'm not sure where to even start doing this. Do I need a LAMP setup to do this (or MAMP since I'm on OS X) or is there a more light-weight JS way I can execute this. I've never known how to save file from within web-browser JS, I thought it was banned for security reasons but I guess theres ways and means.
If i can rewrite this question to be more clear and effective in soliciting an answer I'm happy for edits to question also.
I guess maybe I'm after some boilerplate code for calling a simple API like this and the stack or application context in which I need to do it. I realise there's potential several ways to execute this but looking for most straightforward for someone with JS knowledge and not much PHP/Python experience (but willing to learn what it takes).
Easy right?
Ok, to point you in the right direction.
Requirements
If the language of choice is Javascript, you'll need to install Node.js. No server whatsoever needed.
Same is valid for PHP or Python or whatever. No apache needed, just the lang int.
Running a script with node
Create a file.js somewhere. To run it, you'll just need to type (in the console) node file.js (in the directory the file lives in.
Getting the onfo from the REC Webservice
Here's an example of a GET request:
var https = require('https');
var fs = require('fs');
var options = {
host: 'rec-registry.gov.au',
port: 443,
path: '/rec-registry/app/api/public-register/certificate-actions?date=2015-06-03'
};
var jsonstr = '';
var request = https.get(options, function(response) {
process.stdout.write("downloading data...");
response.on('data', function (chunk) {
process.stdout.write(".");
jsonstr += chunk;
});
response.on('end', function () {
process.stdout.write("DONE!");
console.log(' ');
console.log('Writing to file...');
fs.writeFile("data.json", jsonstr, function(err) {
if(err) {
return console.error('Error saving file');
}
console.log('The file was saved!');
});
});
})
request.on('error', function(e) {
console.log('Error downloading file: ' + e.message);
});
Transforming a json string into an object/array
use JSON.parse
Parsing the data
examine todaysRecords to look for whatever criteria match inside the structure
Can't help you there, but should be relatively straightforward to look for the correct object properties.
NOTE: Basically, what you get from the request is a string. You then parse that string with
var foo = JSON.parse(jsonstr)
In this case foo is an object. The results "certificates" are actually inside the property result, which is an array
var results = foo.result;
In this example the array contains about 1700 records and the structure of a certificate is something like this:
"actionType": "STC created",
"completedTime": "2015-06-02T21:51:26.955Z",
"certificateRanges": [{
"certificateType": "STC",
"registeredPersonNumber": 10894,
"accreditationCode": "PVD2259359",
"generationYear": 2015,
"generationState": "QLD",
"startSerialNumber": 1,
"endSerialNumber": 72,
"fuelSource": "S.G.U. - solar (deemed)",
"ownerAccount": "Solargain PV Pty Ltd",
"ownerAccountId": 25782,
"status": "Pending audit"
}]
So, to access, for instance, the "ownerAccount" of the first "certificateRanges" of the first "certificate" you would do:
var results = JSON.parse(jsonstr).result;
var ownerAccount = results[0].certificateRanges[0].ownerAccount;
Creating a csv
The best way is to create an abstract structure (that meets your needs) and convert it to a csv.
There's a good npm library called json2csv that can help you there
Example:
var fs = require('fs');
var json2csv = require('json2csv');
var fields = ['car', 'price', 'color']; // csv titles
var myCars = [
{
"car": "Audi",
"price": 40000,
"color": "blue"
}, {
"car": "BMW",
"price": 35000,
"color": "black"
}, {
"car": "Porsche",
"price": 60000,
"color": "green"
}
];
json2csv({ data: myCars, fields: fields }, function(err, csv) {
if (err) console.log(err);
fs.writeFile('file.csv', csv, function(err) {
if (err) throw err;
console.log('file saved');
});
});
If you wish to append instead of writing to a new file you can use
fs.appendFile('file.csv', csv, function (err) { });