Edit content of JSON file imported to MongoDB - javascript

I'm using Node.js to:
Query an external API.
Save JSON result to file.
Import query result to MongoDB.
Example of code:
//Query the API and save file
var file = '../data/employees.json';
tl.employees.all(function(response) {
fs.writeFile(file, JSON.stringify(response, function(key, value) {
var result = value;
return result;
}, 3, 'utf8'));
//Inserts API response above to MongoDB
var insertDocument = function(db, callback) {
db.collection('employees').insert(response, function(err, result) {
assert.equal(err, null);
console.log("Step 2: Inserted Employees");
callback();
});
};
This is all working fine, and I'm getting the following JSON saving to file and importing to MongoDB:
{
"data": [
{
"full name": "Keith Richards",
"age": 21,
"userName": "keith1#keith.com",
"employeeDetails": {
"id": 102522
}
},
{
"full name": "Jim Morrison",
"age": 27,
"userName": "jim#jim.com",
"employeeDetails": {
"id": 135522
}
}
]
}
The problem is that this gets imported to MongoDB as 1 object because of the "data" part.
Like this:
{
"_id" : ObjectId("58ae5ceac10d7a5005fc8370"),
"data" : [
{ // contents as shown in JSON above ..
This makes aggregation and matching really hard (or not possible).
Is there a way of stripping out the "data" part and only importing the contents of it, so each 'employee' would become it's own object like this:
[
{
"_id" : ObjectId("58ae5ceac10d7a5005fc8370"),
"full name": "Keith Richards",
"age": 21,
"userName": "keith1#keith.com",
"employeeDetails": {
"id": 102522
}
},
{
"_id" : ObjectId("234332c10d7a5005fc8370"),
"full name": "Jim Morrison",
"age": 27,
"userName": "jim#jim.com",
"employeeDetails": {
"id": 135522
}
}
]

So you can parse the Json before sending it to file. After getting the result, use parseJSON(data) to check if it contains "data" field. if yes, get the arrays inside that and store them to file.
Basically, I have explained a bit more to what #Bertrand has said.
Hope this helps.

Related

Put items in DynamoDB without knowing the attributes

This feels like a really stupid question, but my lack of JS knowledge combined with lack of AWS knowledge has me in a tight spot! I'm just trying to get to grips with a basic AWS stack i.e. Lambda/Dynamo/API Gateway for some basic API work.
If I want a simple API endpoint to handle PUT requests e.g. https://my.endpoint.amazonaws.com/users. If I have a DynamoDB table with a composite primary key of userID and timestamp I could use the code snippet below to take the unknown data (attributes that weren't known when setting a schema), but this obviously doesn't work well
const dynamo = new AWS.DynamoDB.DocumentClient();
let requestJSON = JSON.parse(event.body);
await dynamo
.put({
TableName: "myDynamoTable",
Item: {
userID: requestJSON.userID,
timestamp: requestJSON.timestamp,
data: requestJSON.data
}
})
.promise();
I could send a PUT request with the following JSON
{
"userID": "aaaa1111",
"timestamp": 1649677057,
"data": {
"address": "Elm Street",
"name": "Glen"
}
}
but then address and name get shoved into a single DynamoDB attribute named data. How do I construct the node code to create a new attribute named address and one named name with the corresponding values, if I didn't know these attributes i.e. I want to use JSON in my request like below, but assuming I don't know this and can't use requestJSON.address
{
"userID": "aaaa1111",
"timestamp": 1649677057,
"address": "Elm Street",
"name": "Glen"
}
You can use the spread operator, for example:
const data = {
address: "Elm Street",
name: "Glen",
};
const item = {
userID: "aaaa1111",
timestamp: 1649677057,
...data,
};
console.log("item:", item);
If you insist on the API contract as
{
"userID": "aaaa1111",
"timestamp": 1649677057,
"data": {
"address": "Elm Street",
"name": "Glen"
}
}
then you can do mapping on the TypeScript level
const request: {userID: string, timestamp: number, data: any} = {
"userID": "aaaa1111",
"timestamp": 1649677057,
"data": {
"address": "Elm Street",
"name": "Glen"
}
};
const ddbObject: any = {
"userID": request.userID,
"timestamp": request.timestamp
};
Object
.keys(request.data)
.forEach(key => ddbObject[key] = request.data[key]);
Representation of ddbObject is
{
"userID": "aaaa1111",
"timestamp": 1649677057,
"address": "Elm Street",
"name": "Glen"
}

Adding data to a Json file using javascript

So I have an existing JSON file where the login data is stored. How can I add new users to this JSON file with JavaScript or JQuery?
This is my JSON-File:
{
"users": [
{
"username": "Tim",
"password": "test1"
},
{
"username": "Tom",
"password": "test2"
}
]
}
At the first put your data from login in data
var data = {
"users": [
{
"username": "Tim",
"password": "test1"
},
{
"username": "Tom",
"password": "test2"
}
]
}
And after that you shoud parse your json and put in jsonData and push your new user in it.
var jsonData = JSON.parse(data); //parse the JSON
jsonData.users .push({ //add new user
username :"Mohammad",
password :"test3",
});
At the end stringfy your json data to a string
data= JSON.stringify(jsonData);

Parse nested JSON Response Javascript

I know there are several threads on this subject but I've looked through over 30 threads without success.
I have managed to parse a JSON response so it looks like this:
{
"1": {
"id": "1",
"name": "Fruit",
.
.
.
"entities": {
"1": {
"id": "1",
"name": "blue bird",
.
.
.
"status": "1"
},
"2": {
using this code
let json = JSON.parse(body);
console.log(json);
Now I want to access the "id", "name" etc. AND the "id" and "name" for the "entities" tag.
So far I have tried:
console.log(json[0]);
console.log(json.id);
which both returns undefined
I have also tried
console.log(json[0].id);
which gives an error
Any ideas?
In this instance, your first key is 1, so you can access it with json[1].
const json = {
"1": {
"id": "1",
"name": "Fruit"
},
"2": {
"id": "2",
"name": "Veggies"
}
};
console.log(json[1]);
In this json, you can reach the id by
json.1.id
But I think that first of all your json is not correctly written, you should have something like
{
"elements": [
{ "id" : 1, "name" : "fruit" },
{ "id" : 2, "name" : "vegetable" }
]
}
like that, json.elements is a collection/array, and you can loop, count, or any other things you will not be able to do because your json looks like a super heavy list of different properties ( he doesn't know that json.1 and json.2 are the same type of objects.
const jsonData = JSON.parse(body);
for (const i in jsonData) {
for (const j in jsonData[i]) {
console.log('${i}: ${jsonData[i][j]}');
}
}

While retrieve a model by ID from a Backbone.js collection returning undefined

I'm trying to get a model from a collection by it's ID and display in view. After instantiating a model, I'm using the methods like get(), at() but it's returning undefined.
My collection:
[
{
"id": "1",
"firstname": "Abc",
"lastname": "Xyz"
},
{
"id": "2",
"firstname": "Klm",
"lastname": "Opq"
},
{
"id": "2",
"firstname": "rst",
"lastname": "Yvw"
}
]
Instantiation:
var persons = new PersonCollection();
console.log(persons.get(1)); // undefined
NOTE: I'm getting all the models in console (Not an issue). I want only to fetch a model by it's id name.
fetch is async so you need to place your code inside success callback
persons.fetch({
success: function() {
console.log('Now I have something: ', persons.get(1))
}
})
console.log('Nothing here: ', persons.get(1))
BTW to fetch single model by id you could use Model#fetch instead

Trouble parsing json with node.js

I wrote a quick script to parse two fairly large json file (~17k records) to do a comparison of the two. I have confirmed they are both valid json (via jsonlintpro) and the same format. (The source is the same so this should be a given. But, I always assume the mistake is mine. And I still do. Just somewhere else.) However, the parsed file just outputs [object, Object]. I'm wondering what the cause could possibly be?
The json format is like this small snippet (anonymized of course):
[
{
"id": "1234",
"name": "Name1",
"url": "https://localhost/Name1",
"date_created": "2013-07-05T18:47:05Z",
"date_cancelled": "",
"props": [
{
"id": "54321",
"type": "Client",
"value": "General Store"
},
{
"id": "65432",
"type": "Contact_Name",
"value": "Joe Smith"
}
]
},
{
"id": "23456",
"name": "Name2",
"url": "https://localhost/Name2",
"date_created": "2014-02-27T17:46:43Z",
"date_cancelled": "",
"props": [
{
"id": "34567",
"type": "Client",
"value": "Bait Shop"
}
]
}]
And here is the pertinent code:
var _ = require('underscore');
var recs = require('./prod.json');
printArr(recs);
console.log(recs.length);
function printArr(arr) {
arr.forEach(function(item) {
console.log(item + ", ");
});
}
Any guidance would be greatly appreciated.
UPDATE:
Ok, so apparently the issue is with my printArr function. I'm not sure what I'm doing wrong there. I'd like to figure it out because I want to expand upon that so I can print selectively.
the parsed file just outputs [object, Object].
This is the expected behavior BECAUSE you are concatenating an object with a string.
Try console.log(item) instead
console.log(item); should indeed print [object, Object], did you try to output its properties instead?
function printArr(arr) {
arr.forEach(function(item) {
console.log( item.id, item.name, item.url, item.date_created, item.date_cancelled, item.props, ';');
});
}
Just export the value from the prod.json file.
prod.json file
module.exports = [
{
"id": "1234",
"name": "Name1"
},
{
"id": "1234",
"name": "Name1"
}]
elsewhere
var recs = require('./prod.json')
console.log(recs)

Categories

Resources