Merge json flowfiles in NiFi using Executescript with Javascript - javascript

I am currently trying to merge two Json files - one that is nested and one that is flat:
"ampdata": [
{
"nr": "303",
"code": "JGJGh4958GH",
"Anr": "AVAILABLE",
"ability": [ "" ],
"type": "wheeled",
"conns": [
{
"nr": "447",
"status": "",
"version": "3",
"format": "sckt",
"amp": "32",
"vol": "400",
"vpower": 22
}
]
}
[ {
"nr" : 91643421,
"Anr" : "Real",
"Title" : null,
"Comp" : null,
"Name" : "Smith",
"CompanyName" : "WhiteC"
}]
My current Approach is:
var flowFile = session.get();
if (flowFile != null) {
var StreamCallback = Java.type("org.apache.nifi.processor.io.StreamCallback")
var IOUtils = Java.type("org.apache.commons.io.IOUtils")
var StandardCharsets = Java.type("java.nio.charset.StandardCharsets")
flowFile = session.write(flowFile,
new StreamCallback(function(inputStream, outputStream) {
var text = IOUtils.buffer(inputStream)
var obj = JSON.parse(text)
var neu = [];
var neuesObjekt = {};
for (var i = 0; i < obj.ampdata.length; i++) {
var entry = obj.ampdata[i];
if(obj.ampdata[i].nr != obj2.nr) {
obj2.nr = obj.ampdate[i].nr
}
}
outputStream.write(JSON.stringify(newObj, null, '\t').getBytes(StandardCharsets.UTF_8))
}))
flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename').split('.')[0]+'_translated.json')
session.transfer(flowFile, REL_SUCCESS)
How do I parse two flowfiles that are incoming at the same time? I do like to work with both at the same time as I have to compare them at several positions. I can not figure out how I can avoid overwriting the first flowfile.
I had another Approach with using the MergeConent-Processor, but the result was just the concatenation of the both Jsons in a way that was not a valid Json anymore. Anyway I do prefer the Javascript attempt more, I just need your help in figuring out, how to do it in a proper way.

i think you can use merge content with parameters:
merge format: binary
header: [
footer: ]
demarcator: ,
by this merge of two json files into one will produce a valid json (array).
then, if you need to reformat json - you still can use ExecuteScript processor...
and you don't need to implement join files logic.
PS: to get two files from input queue use this type of code:
var flowFiles = session.get(2);
if(!flowFiles)return;
if(flowFiles.size()!=2){
session.transfer(flowFiles); //return files back to input queue
return;
}
//we have exactly two files. let's process them...
var flowFile1 = flowFiles[0];
var flowFile2 = flowFiles[1];
//read each, parse, apply logic, write result
...

Related

How to remove " from json nested value

How do I remove special char " in data and origin value below?
EDIT:
I have added comma after blue before origin, the actual issue was how to make "[]" to [] after origin.
I want from this:
{
"data" : "[{
"color": "blue",
"origin": "[{"state" : "USA"}, {"state" : "AFRICA"}]"
}]"
}
To This:
{
"data" : [{
"color": "blue",
"origin": [{"state" : "USA"}, {"state" : "AFRICA"}]
}]
}
const strArr = "[{\"foo\": \"bar\"}]";
const arr = JSON.parse(strArr);
console.log(arr);
if that objects name is say "obj" you can do:
obj.data = JSON.parse(obj.data);
this will convert a valid json string into an array
Edit:
if you are just trying to make origin an array instead of a string (assuming valid json) it is probably best you just JSON parse the whole thing then set origin to JSON.parse(origin)
let data = "[{\"color\": \"blue\", \"origin\": \"[{\\\"state\\\": \\\"USA\\\"},{\\\"state\\\": \\\"AFRICA\\\"}]\"}]";
let obj = {
data: data
};
let objData = JSON.parse(obj.data);
objData[0].origin = JSON.parse(objData[0].origin);
console.log(objData);

How to Turn a Multiple Array Object into Query String Parameters in JavaScript

I have the following object below with multiple arrays.
{
"services": [
{
"id": "100",
"name": "PIX"
},
{
"id": "200",
"name": "Rendimentos"
}
],
"channels": [
{
"id": "300",
"name": "Chat"
}
]
}
The idea is to generate query strings, something like that.
services=100&services=200&channels=300
I know you can do it with map and join, but I would know if it was with a pure object, now this format below, I'm confused
You can use URLSearchParams() API.
Iterate your data and append key/value pairs or map an entries array to pass to the constructor
I have no idea what determines the expected output you have shown from the data displayed so am using a simpler data structure for demonstration purposes.
You can combine with URL() API to create full url string as shown below also
const data = [
{name:'foo', value:10},
{name:'bar', value:20}
]
// Loop and append key/values
const params = new URLSearchParams();
data.forEach(e => params.append(e.name, e.value));
console.log('params:', params.toString());
// Alternate approach passing entries array to constructor
const params2 = new URLSearchParams(data.map(e => [e.name,e.value]));
console.log('params2:',params2.toString())
//Adding to a URL
const url = new URL('http://example.com')
url.search = params
console.log('Full url:',url)
Using the updated array data in question:
const data={services:[{id:"100",name:"PIX"},{id:"200",name:"Rendimentos"}],channels:[{id:"300",name:"Chat"}]};
const entries = [];
Object.entries(data).forEach(([k,arr])=> arr.forEach(({id}) => entries.push([k,id])));
const params = new URLSearchParams(entries);
const url = new URL('http://example.com')
url.search = params;
console.log(url)
Looks like you're hung up on trying to iterate an object with map() or join(), which you can't do directly. Instead you can use Object.entries to convert the object into an array and iterate that. Since there is a nested map() you can flat() it before join()
let obj = {
"services": [{
"id": "100",
"name": "PIX"
},
{
"id": "200",
"name": "Rendimentos"
}
],
"channels": [{
"id": "300",
"name": "Chat"
}]
}
let queryString = Object.entries(obj).map(s => s[1].map(e => `${s[0]}=${e.id}`)).flat().join('&')
console.log(queryString)

How to convert JSON string having multiple rows to single row using javascript function

I have an output of REST API in following JSON format:
I need to convert the format to flat format so it can be passed as input to another API call.
{
"result": {
"data": [
{
"data": 2.824315071105957,
"dateTime": "2019-09-10T11:32:05.220Z",
"device": { "id": "b3" },
"diagnostic": { "id": "DiagnosticAccelerationForwardBrakingId" },
"controller": "ControllerNoneId",
"version": "00000000000363b0",
"id": "a5UyPzhknSC-N2wtLBph3BA"
},
{
"data": 0,
"dateTime": "2019-09-10T11:32:05.220Z",
"device": { "id": "b3" },
"diagnostic": { "id": "DiagnosticAccelerationSideToSideId" },
"controller": "ControllerNoneId",
"version": "00000000000363b1",
"id": "a5UyPzhknSC-N2wtLBph3BQ"
},
// ... 1000's of rows like this
]
}
}
I need to convert it in below format using a java-script
Desired format:
{"result":{ "data":[{"id":"b3","dateTime":"2019-09- 10T11:32:05.220Z","DiagnosticAccelerationSideToSideId":0,"DiagnosticAccelerationForwardBrakingId ":2.824315071105957},...
The rows needs to be merged with primary key as combination of ID and dateTime attributes. Please note the diagnostic id value becomes key for the required format and data value is the value of the key.
Is there any way to convert this JSON to above flat format.
Need to convert JSON having many rows for single data entry to single row format. Need one java-script function that can accept a string of rows format and convert or merge it and return the string in desired format
function String mergeRows(String flatDataJSONString) {
...
}
If the items are ordered (meaning i and i+1 are merged) than iterate with jumps of i += 2;
If its not ordered or the amount of items to be merged can be > 2 you use an object with unique key composed of the id and date, and override its data whenever a record match this key:
function merger (jsonStr) {
// convert str to obj
const jsonObj = JSON.parse(jsonStr);
const dataObj = {};
for (let i = 0; i < jsonObj.result.length; i++) {
const item = jsonObj.result[i];
// use unique key to merge by
const itemUniqueKey = item.device.id + item.dateTime;
// take last value or create empty object if not exists
const existingItem = dataObj[itemUniqueKey] || {};
// add some logic to merge item with existingItem as you need
...
// set the result back to dataObj to be used on next merges
dataObj[itemUniqueKey] = [merge result of item and existing item];
}
// take dataObj values, you don't need the keys any more
const dataArr = Object.values(dataObj);
const finalResult = {
result: {
data: dataArr
}
}
// convert back to json
return JSON.stringify(finalResult);
}
As stated in the comment you want first to have a clean json definition in order to stringify it. Please get to the following definition of your JSON first:
const json = {
"result": [
{
"data": 2.824315071105957,
"dateTime": "2019-09-10T11:32:05.220Z",
"device": { "id": "b3" },
"diagnostic": { "id": "DiagnosticAccelerationForwardBrakingId" },
"controller": "ControllerNoneId",
"version": "00000000000363b0",
"id": "a5UyPzhknSC-N2wtLBph3BA"
},
{
"data": 0,
"dateTime": "2019-09-10T11:32:05.220Z",
"device": { "id": "b3" },
"diagnostic": { "id": "DiagnosticAccelerationSideToSideId" },
"controller": "ControllerNoneId",
"version": "00000000000363b1",
"id": "a5UyPzhknSC-N2wtLBph3BQ"
}]
};
and then you will be able to perform like hereafter :
JSON.stringify(json)
Hope this helps !

Populate a variable using jQuery based on selections in a drop down

I'm trying to build a nested array in jQuery based on a user's selection from a drop down menu. This will be used in a JSON request at a later date.
So far my code does produce (almost) the required result, however no matter order i select the options from my drop down menu, the output (which i log in the console at the end) is always the same.
$('#comboGenre').change(function () {
var values = $('#comboGenre').val();
var parsedJSON = JSON.parse($data); //Data returned from ajax request
for (var i = 0; i < values.length; i += 1) {
$genreList = parsedJSON.genre[i];
console.log($genreList);
}
});
So if i select RPG and Action from my drop down, the output gives me RPG and Driving. If i selected RPG, Driving and Action (in that order), i get what i would expect RPG, Driving and Action.
So it's just iterating through my JSON, when really it should be returning the 'selected' option.
How can i achieve this?
My JSON looks like this if it's useful:
{"genres": [{
"genre": "RPG",
"publishers": [{
"publisher": "Square",
"games": [{
"game": "FFX",
"rating": [
12, 15
]
}]
}]
},
{
"genre": "Driving",
"publishers": [{
"publisher": "Turn10",
"games": [{
"game": "Forza",
"rating": [
5
]
}]
}]
},
{
"genre": "Action",
"publishers": [{
"publisher": "EA",
"games": [{
"game": "COD",
"rating": [
18, 20
]
}]
}]
}
]}
EDIT:
I've also tried this:
$('#comboGenre').change(function () {
var parsedJSON = JSON.parse($data);
$genreList = "";
$.each(parsedJSON.genres, function(index, value){
$genreList = parsedJSON.genres[index];
console.log($genreList);
});
});
And i end up getting ALL the objects in my JSON, so from here, i'm only wanting to add the selected object to the $genreList variable.
If you broke out some of the logic and created a genre finding function and used the selected string to find the proper object you could then put the object into the variable you will use later. I do some checking to ensure that the genre that has been selected isn't already in my array which is because I am using the multiple select
JSFiddle: http://jsfiddle.net/vkTFq/
Code:
$(function(){
var selectedGenres = [];
var genres =[{"genre":"RPG","publishers":[{"publisher":"Square","games":[{"game":"FFX","rating":[12,15]}]}]},{"genre":"Driving","publishers":[{"publisher":"Turn10","games":[{"game":"Forza","rating":[5]}]}]},{"genre":"Action","publishers":[{"publisher":"EA","games":[{"game":"COD","rating":[18,20]}]}]}]
$('#comboGenre').change(function() {
$(this).find(":selected").each(function() {
var selectedGenre = findGenre($(this).val())
if (!genreAlreadySelected(selectedGenre.genre)) {
selectedGenres.push(selectedGenre);
};
});
console.log (JSON.stringify(selectedGenres));
});
function genreAlreadySelected(genre){
for(var i = 0; i < selectedGenres.length; i++){
if (genre == selectedGenres[i].genre) {
return true;
};
return false;
}
}
function findGenre(genre){
for(var i = 0; i < genres.length; i ++){
console.log(genre)
if(genre == genres[i].genre){
return genres[i];
}
}
};
});

JSON/JS Object accessing key when you don't know the name in Google Apps Script

I'm working with a response from the Webtrends API in Google apps script and I have a JSON/JS object that looks like this:
"data": [
{
"period": "Month",
"start_date": "2013-12",
"end_date": "2013-12",
"attributes": {},
"measures": {
"Visits": 500
},
"SubRows": [
{
"facebook.com": {
"attributes": {},
"measures": {
"Visits": 100
},
"SubRows": null
},
"google.co.uk": {
"attributes": {},
"measures": {
"Visits": 100
},
"SubRows": null
},
"newsnow.co.uk": {
"attributes": {},
"measures": {
"Visits": 100
},
"SubRows": null
},
"No Referrer": {
"attributes": {},
"measures": {
"Visits": 100
},
"SubRows": null
},
"t.co": {
"attributes": {},
"measures": {
"Visits": 100
},
"SubRows": null
}
}
]
}
]
What I need to access is the names i.e facebook.com etc... and visit numbers for each of the SubRows.
I'm able to get the visit numbers, but I can't work out how to get the names. Please note the names will change constantly as different sites will send different amounts of traffic each day.
Section of my code at the moment where I get the visit numbers:
for(i in dObj){
var data = dObj[i].SubRows;
var sd = dObj[i].start_date;
var ed = dObj[i].end_date;
if(sd == ed){
var timep = ""+ sd;
}
else{
var timep = ""+ sd + "-" + ed;
}
var subRows = data[0];
Logger.log(subRows);
for(i in subRows){
var row = subRows[i];
var rmeasures = row.measures;
var rvis = rmeasures.Visits;
values = [timep,"",rvis]; //Blank string for where the name of the site would go
}
}
I've tried the following links, but none of them seem to have the answer:
Getting JavaScript object key list
How to access object using dynamic key?
How to access key itself using javascript
How do I access properties of a javascript object if I don't know the names?
I'm just using vanilla google apps script as I don't have any experience with Jquery etc...
Any help would be much appreciated!
I usually use a little helper function that looks like this:
var keyVal = function(o) {
var key = Object.keys(o)[0];
return {"key": key, "val":o[key]};
} ;
This will map an object with a variable key to a key/value object {key:...., val:{}}, which is usually convenient enough to work with.
describe.only ("stack overflow answer", function(){
it ("is should create a key/value pair" , function(){
var res = keyVal( {
"facebook.com": {
"attributes": {},
"measures": {
"Visits": 100
},
"SubRows": null
}});
res.key.should.equal('facebook.com');
res.val.attributes.should.deep.equal({});
});
Within the loop, the variable i contains the current key. Replacing the empty string with i should give you what you need.
You might also want to look at some of the more functional tools built into Javascript. Some more concise code might also be more explicit:
data.map(function(datum) {
var timep = datum.start_date == datum.end_date ? datum.end_date :
(data.start_date + "-" + datum.end_date);
return datum.SubRows.map(function(subRow) {
return Object.keys(subRow).map(function(key) {
return [timep, key, subRow[key].measures.Visits];
});
});
});
would return an object something like this:
[
[
[
["2013-12", "facebook.com", 100],
["2013-12", "google.co.uk", 100],
["2013-12", "newsnow.co.uk", 100],
["2013-12", "No Referrer", 100],
["2013-12", "t.co", 100 ]
]
]
]
This just uses map and Object.keys to simplify some of what you're doing with explicit loops.

Categories

Resources