I'm trying to create a process to parse a log file into a JSON file to a future processing.
The log file has this format:
[
{
"textPayload": "Dialogues0 gRPC Response : response_id: \"fc4e2e63-509b4ae2-a7d8-c401e563aa4b\"\nquery_result {\n query_text: \"wonderful\"\naction: \"smalltalk.appraisal.good\"\n parameters {\n }\nall_required_params_present: true\n fulfillment_text: \"Glad you think so!\"\n fulfillment_messages {\n text {\n text: \"Glad you think so!\"\n }\n }\n intent_detection_confidence: 1.0\n language_code: \"en\"\nsentiment_analysis_result {\n query_text_sentiment {\n }\n}\n}\nagent_id: \"3d22af45-f603-4a8a-a7ce-a9b2af47b762\"\n",
"insertId": "1lfux63g16s1nna",
"resource": {
"type": "global",
"labels": {
"project_id": "data-analytics-digital-dev"
}
},
"timestamp": "2018-11-07T14:31:02.435Z",
"severity": "INFO",
"labels": {
"request_id": "fc4e2e63-509b-4ae2-a7d8-c401e563aa4b",
"type": "dialogflow_response",
"protocol": "V2BETA1"
},
"logName": "projects/data-analytics-digital-dev/logs/dialogflow_agent",
"trace": "7fa08c8c-df50-4d46-9f20-b1e357b844a4",
"receiveTimestamp": "2018-11-07T14:31:02.555590714Z"
}
]
My target is the content of "textPayload"
the Node.js code is:
fs = require('fs');
fs.readFile('./global_logs1.json', 'utf8', function (err,data) {
if (err) {
return console.log(err);
}
let parsedLog = JSON.parse(data);
for(let myKey in parsedLog) {
let tempJson = (parsedLog[myKey]["textPayload"]);
tempJson = (tempJson.substr(91,));
tempJson = (tempJson.substr(0, tempJson.length - 50));
console.log((tempJson));
//console.log(JSON.parse(tempJson));
}
});
and it is resulting in a JSON like string:
{
query_text: "wonderful"
action: "smalltalk.appraisal.good"
parameters {
}
all_required_params_present: true
fulfillment_text: "Glad you think so!"
fulfillment_messages {
text {
text: "Glad you think so!"
}
}
intent_detection_confidence: 1.0
language_code: "en"
sentiment_analysis_result {
query_text_sentiment {
}
}
}
However when I call the JSON.parser I receive an error:
undefined:2
query_text: "wonderful"
^
SyntaxError: Unexpected token q in JSON at position 5
at JSON.parse (<anonymous>)
at c:\Codes\Logging\test.js:15:26
at FSReqWrap.readFileAfterClose [as oncomplete]
(internal/fs/read_file_context.js:53:3)
Seems it is missing double quotes, but I'm not sure.
Any idea on how to do it?
The text in the 'textPayload' attribute is absolutely not valid JSON. As you pointed out, the attribute names are missing double quotes. You are going to need to parse it out on your own. You could experiment with regex (results and opinions may vary.) or use existing "relaxed JSON" libraries that do not make use of eval.
EDIT: Using the module 'relaxed-json' I threw together this dirty script. You obviously can sanitize it yourself instead of relying on an external module but I am lazy here, and there might even be an out of the box solution to clean this up better, but this works. You need to remove the newlines, add commas after the items, add colons to the object attributes and add double quotes to the attribute names in order for it to be valid JSON. Like I said, its a dirty script and I did some awful regex matching and replacing just for the sake of making a proof of concept, so I am prepared to be judged on how crappy it is.
var parsed = require('./payload.json');
const rjson = require('relaxed-json');
for(let key in parsed){
let tempJson = (parsed[key]["textPayload"]);
tempJson = (tempJson.substr(91,));
tempJson = (tempJson.substr(0, tempJson.length - 50));
tempJson = tempJson.replace(/\n/g,",");
tempJson = tempJson.replace(/\{,/g,"{");
tempJson = tempJson.replace(/ \{/g,":{");
let transformed = JSON.parse(rjson.transform(tempJson));
console.log(transformed);
}
The output is a true javascript object now.
{ query_text: 'wonderful',
action: 'smalltalk.appraisal.good',
parameters: {},
all_required_params_present: true,
fulfillment_text: 'Glad you think so!',
fulfillment_messages: { text: { text: 'Glad you think so!' } },
intent_detection_confidence: 1,
language_code: 'en',
sentiment_analysis_result: { query_text_sentiment: {} } }
Your tempJson is actually a javascript object.
To convert it back to json just call
JSON.stringify(tempJson)
Sometimes the two concepts get mixed because they are so easily interchangeable.
Related
I'm currently working on a NodeJS project, this takes data from a JSON and then use it to take weather stuff form an API, after that I want to save it to a DB, I already asked about it and that question helped me fixing some problems but now I have some others, I'm sending the data to a constant but the issue is that I don't know why am I getting an error in the JSON Parse, I want to use the lat and lon from the JSON (I have like a hundred data coords) and insert it into the const, any help would help, This is the error I'm getting
Successful connection
[]
undefined:1
^
SyntaxError: Unexpected token in JSON at position 0
at JSON.parse (<anonymous>)
here is my function that takes data from JSON and gets data from the API:
async function calcWeather() {
fs.readFile("./json/data.json","utf8", function (err, data) {
if(err) throw err;
data2 = JSON.parse(data);
console.log(typeof data2);
for (let item of data2) {
let base = `https://api.openweathermap.org/data/2.5/weather?lat=${item.latjson}&lon=${item.lonjson}&appid=${api_key}&units=metric&lang=sp`;
fetch(base)
.then((responses) => {
return responses.json();
})
.then((data) => {
var myObject = {
Id_Oficina: item.IdOficina,
Humedad: data.main.humidity,
Nubes: data.clouds.all,
Sensacion: data.main.feels_like,
Temperatura: data.main.temp,
Descripcion: data.weather.description,
};
// validation and saving data to array
if (myObject.Temperatura < 99) {
lstValid.push(myObject);
}
});
}
});
console.log(lstValid);
}
here is the JSON where I take the data:
[
{
"latjson": 1,
"lonjson": 1,
"IdOficina": "1"
},
{
"latjson": 2,
"lonjson": 2,
"IdOficina": "2"
}
]
I think the issue is in the parse, but I don't get what I am doing wrong
Since you are reading the file with fs.readFile, you are getting a string and not a JavaScript object. You would need to parse it entirely in order to be able to manipulate the content (you seem to be parsing the first character only):
const fs = require('fs')
let rawdata = fs.readFileSync('./data.json')
let data = JSON.parse(rawdata)
Personally, I think it's way easier to require it (no need to use fs):
const jsonData = require('./json/data.json')
async function calcWeather() {
for (let item of jsonData) {
// ...
}
}
This is the error I am seeing in my AWS Console:
{
"type": "x_content_parse_exception",
"reason": "[1:38] [bool] failed to parse field [should]",
"caused_by": {
"type": "parsing_exception",
"reason": "[match] query malformed, no start_object after query name",
"line": 1,
"col": 38
}
}
My query formation below:
const result = JSON.stringify(JSON.parse(pBody.body));
const { body } = await client.search({
index: 'first-entry',
body: {
query: {
bool: {
should: [
{
match: result
}
]
}
}
}
});
The result looks like this in the AWS console:
{ "hello": "World!" }
So it's just a normal Object? I've tried a number of things to try and get around this error but not having any luck. Does anyone see a problem with how it is formed?
The reason I was seeing this error was because, although I was correctly doing a JSON.parse(), I was stringifying the result immediately after.
The Client did not like the stringified result.
To fix this I just made sure the result was parsed and not stringified.
I am new to javascript. I am trying to replace a JSON elements with a set values passed from another function. When i parse the object I am getting an error message unexpected token at pos 0.
Thank you.
var defaultMessage = {
"MTI": "1101",
dataElements: {
"DE01": "101",
"DE02": "201",
"DE03": "301",
"DE04": "401",
"DE05": "501",
"DE06": "601",
"DE07": "701"
}
}
replaceDefaultElements: (testElements) => {
console.info(testElements)
console.info(defaultMessage);
var messageElements = JSON.parse(defaultMessage.dataElements);
Object.keys(testElements).forEach(function (key) {
messageElements[key] = testElements[key];
});
return messageElements;
}
//Calling from JEST test function:
test('Should be ', () => {
expect(functions.replaceDefaultElements('{"MTI":"1101",{"DE01":"XXX"}}')).toEqual({
"MTI": "1101",
dataElements: {
"DE01": "XXX",
"DE02": "201",
"DE03": "301",
"DE04": "401",
"DE05": "501",
"DE06": "601",
"DE07": "701"
}
})
When i parse the object I am getting an error message unexpected token at pos 0.
defaultMessage.dataElements is a JavaScript object. It isn't JSON.
Don't use JSON.parse there.
testElements, on the other hand, is JSON.
Replace JSON.parse(defaultMessage.dataElements) with defaultMessage.dataElements
Replace Object.keys(testElements) with Object.keys(JSON.parse(testElements))
Hope it will work
I am trying to convert from an introspection query to a GraphQL schema using the npm GraphQL library.
It keeps stating:
devAssert.mjs:7 Uncaught Error: Invalid or incomplete introspection result. Ensure that you are passing "data" property of introspection response and no "errors" was returned alongside: { ... }
Issue is I am getting it directly from GraphiQL Shopify and can't figure out how to validate my introspection return is correct.
Code:
var introspection = `
{
"data": {
"__schema": {
"types": [
{
"name": "Boolean"
},
{
"name": "String"
},
{
"name": "QueryRoot"
},
{
"name": "Job"
},
{
"name": "ID"
},
{
"name": "Node"
},
{
"name": "Order"
}
]
}
},
"extensions": {
"cost": {
"requestedQueryCost": 2,
"actualQueryCost": 2,
"throttleStatus": {
"maximumAvailable": 1000,
"currentlyAvailable": 980,
"restoreRate": 50
}
}
}
}`;
let schema = buildClientSchema(introspection);
//console.log(schema.printSchema());
I thought the introspection result could be a string? Or is there not enough info to build a schema? Do I need to expand the number of fields? What's the bare minimum needed to exchange an introspection result into a schema?
You should use getIntrospectionQuery to get the complete introspection query needed. If the response you're working with is a string, it should then be parsed to an object before being passed to buildClientSchema -- the function accepts an object, not a string.
Here's an example directly querying a schema using the graphql function -- you'll need to modify it based on how you're actually executing the query.
const { getIntrospectionQuery, buildClientSchema, graphql } = require('graphql')
const schema = new GraphQLSchema(...)
const source = getIntrospectionQuery()
const { data } = await graphql({ source, schema })
const clientSchema = buildClientSchema(data)
Make sure that you are only passing in the data portion of the response. The object you pass in should look like this:
{
__schema: {
// ...more properties
}
}
I'm currently working on a configuration service in my Angular 2 application, my current concern is about the utilization of eval in my code to retrieve a value in my configuration.
Here's a sample of my configuration file:
{
"application": {
"environment": "dev",
"displayMenu": false
},
"error": {
"title": "Title Error",
"message": "Error message"
}
}
I'm retrieving this JSON with a simple HTTP request, but now I would like to access an element with my get method defined like this:
get(key: any) {
return (eval("this._config." + key));
}
As you can see, there is an eval in my code that I would like to avoid. I'm forced to use eval to allow the developer to do .get('application.environment') and actually I don't find any others an easy possibility.
The only other way that I could see is to split the key on a "." and retrieve the right element in my JSON like it was a simple array. But with this solution, I would be stuck to one depth only.
You could use an array of your object keys you wish to view, then reduce that array returning the key of the object. If you wish to have a string as the object accessors you can easily use string.split('.') to create the array you can reduce.
const data = {
"application": {
"environment": "dev",
"displayMenu": false
},
"error": {
"title": "Title Error",
"message": "Error message",
"deeper": {
"evenDeeper": "You can get to any level"
}
}
}
const path = (keys, obj) => {
return keys.reduce((obj, key) => {
return typeof obj !== 'undefined'
? obj[key]
: void 0
}, obj)
}
console.log(
path(['application', 'environment'], data)
)
console.log(
path('error.message'.split('.'), data) // move the split inside the path function
)
console.log(
path(['error', 'deeper', 'evenDeeper'], data)
)
console.log(
path(['error', 'fail', 'damn'], data) // fail safe
)
<script src="http://codepen.io/synthet1c/pen/WrQapG.js"></script>