How to get JSON Elements from String(full with JSON object). - javascript

I have a JSON file
{ "impressions":
[
{
"impressionId": "7ad7a77fas346a7a2a1da6",
"userId": "hafsa",
"clientId": "400"
},
{
"impressionId": "7ad7a77fas346a7a2a1da7",
"userId": "asif",
"clientId": "200"
},
{
"impressionId": "7ad7a77fas346a7a2a1da8",
"userId": "zadarov",
"clientId": "300"
},
{
"impressionId": "7ad7a77fas346a7a2a1da9",
"userId": "julia",
"clientId": "100"
}
]
}
I am working on Kafka and have a NodeJS Producer and Java Consumer.I have to get each impression as a separate message in my consumer. My nodeJS code is:
console.log("Kafka is recieving JSON....");
var fs = require('fs');
var kafka = require('kafka-node');
var Producer = kafka.Producer;
var Client = kafka.Client;
var client = new Client('localhost:2181');
var producer = new Producer(client);
fs.readFile('data.json', 'utf8', function (err, data) {
if (err) throw err;
var jsonobj = JSON.parse(data);
var countMessages = jsonobj.impressions.length;
var impressionArr = [];
impressionArr = jsonobj.impressions;
payloads = [
{ topic: 'test', messages: impressionArr, partition: 0 }
];
producer.on('ready', function(){
producer.send(payloads, function(err, data){
console.log(data);
});
});
producer.on('error', function(err){
console.log("Error: "+err)
});
});
and my JAVA consumer is :
JavaInputDStream<String> messagesFrmSpecifiedOffset= KafkaUtils.createDirectStream(
sc,
String.class,
String.class,
StringDecoder.class,
StringDecoder.class,
String.class,
kafkaParams,
fromOffset,
new Function<MessageAndMetadata<String, String>, String>() {
public String call(MessageAndMetadata<String, String> msgAndMd) throws Exception {
return msgAndMd.message();
}
}
);
and I am getting impressions as separate messages as follows in JAVA consumer.
JavaDStream<Record> rdd_impressionRecords = messagesFrmSpecifiedOffset.map(
new Function<String, Record>() {
public Record call(String impressions) throws Exception {
System.out.println("impressions: " + impressions);
Record sd = new Record();
return sd;
}
});
BUT MY PROBLEM IS:
that I am getting output in object form like:
impressions: [object Object]
impressions: [object Object]
impressions: [object Object]
impressions: [object Object]
I know that I am sending JSON and recieving as String but anyone can help me to do solution in three ways:
1. Can I get key, values from [object Object] in Java class.
2. Is it possible to send each array element as String (using JSON.stringify) in NodeJS without any loop. Direct transfer as I am doing.
3. Can I rewrite method of createDirectStream that returns JSON obect directly.

In addition to Zdeněk Tisoň answer:
a) First use JSON.stringify() to create a string and send it over the net.
b) At the consumers side transform this string to whatever you want. If you want it to be Json you can use JSON.parse().

Client code could be:
var _ = require('lodash');
var payloads = _.map(JSON.parse(data).impressions, function(imp){
return JSON.stringify(imp);
});
producer.send(payloads, function(err, data){
});
Spark code could use json-simple library:
import org.json.simple.*;
JSONObject imp = (JSONObject) JSONValue.parse(impression);
String impressionId = (String) imp.get("impressionId");

You could use Java's JSONObject to get the string as a JSONObject like so:
JSONObject jsonObject = new JSONObject(impressions)
Then use the JSONObject functions to get what you need as listed here:
http://docs.oracle.com/javaee/7/api/javax/json/JsonObject.html

Use JSON.stringify before sending messages onto the Kafka bus. For example, your code changes to messages: JSON.stringify(impressionArr).

AFAIK producing object to kafka is not supported by kafka-node. If you try to do that, the object itself would be in format of [object Object], actually it's String '[object Object]'. That's why you can not de-serialize from Java.
try JSON.stringify(impressionArr) when produce in your node code and some JSON library when consume from Java side.

Related

How can I insert data from a JSON to a const variable in NodeJS for using an API?

I'm currently working on a NodeJS project, this takes data from a JSON and then use it to take weather stuff form an API, after that I want to save it to a DB, I already asked about it and that question helped me fixing some problems but now I have some others, I'm sending the data to a constant but the issue is that I don't know why am I getting an error in the JSON Parse, I want to use the lat and lon from the JSON (I have like a hundred data coords) and insert it into the const, any help would help, This is the error I'm getting
Successful connection
[]
undefined:1
^
SyntaxError: Unexpected token in JSON at position 0
at JSON.parse (<anonymous>)
here is my function that takes data from JSON and gets data from the API:
async function calcWeather() {
fs.readFile("./json/data.json","utf8", function (err, data) {
if(err) throw err;
data2 = JSON.parse(data);
console.log(typeof data2);
for (let item of data2) {
let base = `https://api.openweathermap.org/data/2.5/weather?lat=${item.latjson}&lon=${item.lonjson}&appid=${api_key}&units=metric&lang=sp`;
fetch(base)
.then((responses) => {
return responses.json();
})
.then((data) => {
var myObject = {
Id_Oficina: item.IdOficina,
Humedad: data.main.humidity,
Nubes: data.clouds.all,
Sensacion: data.main.feels_like,
Temperatura: data.main.temp,
Descripcion: data.weather.description,
};
// validation and saving data to array
if (myObject.Temperatura < 99) {
lstValid.push(myObject);
}
});
}
});
console.log(lstValid);
}
here is the JSON where I take the data:
[
{
"latjson": 1,
"lonjson": 1,
"IdOficina": "1"
},
{
"latjson": 2,
"lonjson": 2,
"IdOficina": "2"
}
]
I think the issue is in the parse, but I don't get what I am doing wrong
Since you are reading the file with fs.readFile, you are getting a string and not a JavaScript object. You would need to parse it entirely in order to be able to manipulate the content (you seem to be parsing the first character only):
const fs = require('fs')
let rawdata = fs.readFileSync('./data.json')
let data = JSON.parse(rawdata)
Personally, I think it's way easier to require it (no need to use fs):
const jsonData = require('./json/data.json')
async function calcWeather() {
for (let item of jsonData) {
// ...
}
}

Write JSON with Node.js

I'm trying to push names in a json file. I'm trying to do something like:
socket.on('create json', function(data){
var data = JSON.stringify(Data, null, 2);
fs.writeFile('participants.json', data)
console.log(data);
});
This is only outputting the data that I've send and results in:
{
"type": "Buffer",
"data": [34,69,120,97,109,112,108,101,32,110,97,109,101, 34 ]
}
When I'm writing the file it deletes everything and puts that in.
I'm looking for a way to write:
{
"names":["name1", "name2", "name3"]
}
Any ideas on how to fix and write this?
Help is very much appreciated!
you have to again read your file, parse the JSON, append your new result to the array, transform it back into a string and save it again.
var fs = require('fs')
fs.readFile('participants.json', function (err, data) {
var json = JSON.parse(data);
json.name = ["name1", "name2", "name3"];
fs.writeFile("results.json", JSON.stringify(json))
})

How to write a postman test to compare the response json against another json?

I have the below json response after running a postMan test of a Rest API:
{
"glossary": {
"title": "example glossary",
"GlossDiv": {
"title": "S",
"GlossList": {
"GlossEntry": {
"ID": "SGML",
"SortAs": "SGML",
"GlossTerm": "Standard Generalized Markup Language",
"Acronym": "SGML",
"Abbrev": "ISO 8879:1986",
"GlossDef": {
"para": "A meta-markup language, used to create markup languages such as DocBook.",
"GlossSeeAlso": ["GML", "XML"]
},
"GlossSee": "markup"
}
}
}
}
}
Now I would like to compare the above json against a predefined json. Say, its the same as above.
How can I compare two jsons via the Postman test?
I had a similar problem to solve except that my JSON also contained an array of objects. I used the following technique that can be modified to deal with the simple array of strings in your question.I created an array of global functions called "assert", which contained helper functions such as "areEqual" and "areArraysOfObjectsEqual" and saved these under the "Tests" tab at a top folder level of my tests.
assert = {
areEqual: (actual, expected, objectName) => {
pm.test(`Actual ${objectName} '` + actual + `' matches Expected ${objectName} '` + expected + `'`, () => {
pm.expect(_.isEqual(actual, expected)).to.be.true;
});
},
areArraysOfObjectsEqual: (actual, expected, objectName) => {
if (!_.isEqual(actual, expected)) {
// Arrays are not equal so report what the differences are
for (var indexItem = 0; indexItem < expected.length; indexItem++) {
assert.compareArrayObject(actual[indexItem], expected[indexItem], objectName);
}
}
else
{
// This fake test will always pass and is just here for displaying output to highlight that the array has been verified as part of the test run
pm.test(`actual '${objectName}' array matches expected '${objectName}' array`);
}
},
compareArrayObject: (actualObject, expectedObject, objectName) => {
for (var key in expectedObject) {
if (expectedObject.hasOwnProperty(key)) {
assert.areEqual(expectedObject[key], actualObject[key], objectName + " - " + key);
}
}
}
};
Your "Pre-request Script" for a test would set your expected object
const expectedResponse =
{
"id": "3726b0d7-b449-4088-8dd0-74ece139f2bf",
"array": [
{
"item": "ABC",
"value": 1
},
{
"item": "XYZ",
"value": 2
}
]
};
pm.globals.set("expectedResponse", expectedResponse);
Your Test would test each item individually or at the array level like so:
const actualResponse = JSON.parse(responseBody);
const expectedResponse = pm.globals.get("expectedResponse");
assert.areEqual(
actualResponse.id,
expectedResponse.id,
"id");
assert.areArraysOfObjectsEqual(
actualResponse.myArray,
expectedResponse.myArray,
"myArrayName");
This technique will give nice "property name actual value matches expected value" output and works with arrays of objects being part of the JSON being compared.
Update:
To test your array of strings "GlossSeeAlso", simply call the supplied global helper method in any of your tests like so:
assert.compareArrayObject(
actualResponse.glossary.GlossDiv.GlossList.GlossEntry.GlossDef.GlossSeeAlso,
expectedResponse.glossary.GlossDiv.GlossList.GlossEntry.GlossDef.GlossSeeAlso,
"glossary.GlossDiv.GlossList.GlossEntry.GlossDef.GlossSeeAlso");
Primitive types in JSON key value pairs can be tested like so:
assert.areEqual(
actualResponse.glossary.title,
expectedResponse.glossary.title,
"glossary.title");
I got it after a while. Add test into your request and use Runner to run all your requests in the collection.
Postman info: Version 7.10.0 for Mac.
Test scripts:
pm.test("Your test name", function () {
var jsonData = pm.response.json();
pm.expect(jsonData).to.eql({
"key1": "value1",
"key2": 100
});
});
You can paste this code into your collection or single request tests tab.
What this code do is to save the request into a global variable with a key for that request. You can change your enviroment and hit the same request and if the response are different the test will fail.
const responseKey = [pm.info.requestName, 'response'].join('/');
let res = '';
try {
res = JSON.stringify(pm.response.json());
} catch(e) {
res = pm.response.text();
}
if (!pm.globals.has(responseKey)) {
pm.globals.set(responseKey, res);
} else {
pm.test(responseKey, function () {
const response = pm.globals.get(responseKey);
pm.globals.unset(responseKey);
try {
const data = pm.response.json();
pm.expect(JSON.stringify(data)).to.eql(response);
} catch(e) {
const data = pm.response.text();
pm.expect(data).to.eql(response);
}
});
}
Hope this help.
You can write javascript code inside Tests tab of Postman. Just write simple code to compare and check result in Tests.
var serverData = JSON.parse(responseBody);
var JSONtoCompare = {}; //set your predefined JSON here.
tests["Body is correct"] = serverData === JSONtoCompare;
Looks like the same question asked at POSTMAN: Comparing object Environment variable with response's object which also lists a solution that works, which is to use JSON.stringify() to turn the objects into strings and then compare the strings.
Came across this issue when migrating from a legacy API to a new one and wanting to assert the new API is exactly the same as the old under different scenarios
For context this clones params of the original get request to the legacy endpoint and validates both match up
LEGACY_API_URL should be defined in the environment and the Request is going to the new API
const { Url } = require('postman-collection');
// Setup the URL for the Legacy API
const legacyRequestUrl = new Url({ host: pm.variables.replaceIn("http://{{LEGACY_API_HOST}}/blah")});
// Add All Parameters From the Source Query
legacyRequestUrl.addQueryParams(pm.request.url.query.all());
// Log out the URL For Debugging Purposes
console.log("URL", legacyRequestUrl.toString());
pm.sendRequest(legacyRequestUrl.toString(), function (err, response) {
pm.test('New API Response Matches Legacy API Response', function () {
// Log Out Responses for Debugging Purposes
console.log("New API Response", pm.response.json())
console.log("Legacy API Response", response.json())
// Assert Both Responses are Equal
pm.expect(_.isEqual(pm.response.json(), response.json())).to.be.true
});
});
Link to an example collection
https://www.getpostman.com/collections/4ff9953237c0ab1bce99
Write JavaScript code under 'Tests' section. Refer below link for more info.
Click Here

JSon File first read as an object then as a Json

So my problem is pretty simple.
I have a Json File which contain : { "message": [] }
In this File I'll add continuously messages by parsing it, pushing an item and "stringifying" it back.
Like this :
jf.readFile(file, 'utf8', function(err, json){
json = JSON.parse(json)
json.message.push({user: message.username, date: message.hour, message: message.text})
json = JSON.stringify(json)
jf.writeFile(file, json, 'utf8');
}
However when it is the first message basicJsonF is consider as an [object Object].
And as soon as I have a message in it is consider as a JSon file.
How could I (without an if condition) specify basicJsonF as a Json File?
Edit :
I've found out a solution I guess is clean if anyone need it :
typeof(json) === 'object' ? null : json = JSON.parse(json)
This Code Snippet works fine for what you wanna do. You only need to write the json into your file.
var message = { username: "TestUser", hour: new Date(), text: "HEY THERE!" };
var jsonBasicFile = '{ "message": [] }';
console.log(jsonBasicFile);
var obj = JSON.parse(jsonBasicFile);
console.log(obj);
obj.message.push({user: message.username, date: message.hour, message: message.text});
console.log(obj);
var json = JSON.stringify(obj);
console.log(json);

Not able to pass an object in writeFile method

I want to write into json file .I want to write an object that i am passing
Here is the code
path.exists(logfile_name, function(exists) {
if (!exists) {
var jsonObject={ "req": req,
"result": result ,
"fields": fields } ;
fs.writeFile(logfile_name ,jsonObject,function(err){
if(err){
console.log("error is: " + err)
}
else
console.log("no error found");
});
}
});
In logfile_name it writes
[object Object]
But i want it to write like this
{
"req": value,
"result": value ,
"fields": value}
If you don't pass a string or a buffer to writeFile, the toString function of what you pass is called. In your case it returns "[object Object]".
You have to convert it yourself :
fs.writeFile(logfile_name, JSON.stringify(jsonObject), function(err){
I would advise against naming a JavaScript object variable "jsonObject" : it might create confusion between what is an object and what is some JSON (i.e. a string holding the serialization of an object).

Categories

Resources