Develop Reference

Is it correct to use .reduce and filter in 1 million json array in node js?

Is it correct to use .reduce and .filter in 1 million json array in node js? I am using JSONStream to parse it. However I need to find the duplicate and unique records and have to do some massage on every records and also have to comparisons between two json array of 1 millions.
What is the best way to approach this? Performance is not a constraint its a background job only. Planning to deploy this job aws ECS using scheduled task.
I am planning to do something like this below
var values = [
{ name: "a", age: 15 },
{ name: "a", age: 17 },
{ name: "a", age: 17 },
{ name: "b", age: 18 },
{ name: "c", age: 18 },
{ name: "d", age: 18 },
{ name: "c", age: 18 },
];
const lookup = values.reduce((a, e) => {
a[e.name] = ++a[e.name] || 0;
return a;
}, {});
console.log("Name count",lookup)
var unique =[];
var dup = values.filter(e => {
if(lookup[e.name]){
return lookup[e.name];
}else {
unique.push(e)
}
});
console.log("Unique:",unique);
console.log("Duplicate", dup)

If you want to know whether this is going to work with a million record array, then you will just have to run it on your system and in your environment to see. That isn't something one can just look at the code and "know". You have to experiment.
I'd have to say that I start out thinking that a million record set of data is probably best managed in a database. If you put all these in a database, then you could structure how you put the data into the database to make it easier to query for all the records with a given name or age or all the records that have a unique name or a non-unique name. This is what databases are built for.
I can see one way that might speed things up a bit. You can build lookup, unique and dup with fewer name lookups. If, while you are building the lookup data structure, you also just keep track of all the items with that particular name, then you can just see which items end up with only one item for a name. So, rather than iterate through all the data and lookup every name, you can just iterate through the lookup data structure and see which names have a cnt of 1 or greater than 1.
I also personally like using a Map object for the lookup since that's what it is built for, but since your keys are strings, your object will probably work just fine too. Anyway, here's a slightly tweaked version:
var values = [
{ name: "a", age: 15 },
{ name: "a", age: 17 },
{ name: "a", age: 17 },
{ name: "b", age: 18 },
{ name: "c", age: 18 },
{ name: "d", age: 18 },
{ name: "c", age: 18 },
];
const lookup = new Map();
const unique = [];
const dup = [];
for (let item of values) {
let array = lookup.get(item.name);
if (!array) {
// if we don't have this name yet, add it to the Map as a one element array
lookup.set(item.name, [item]);
} else {
// otherwise, just push this item into the array
array.push(item);
}
}
// To find unique and dup, we can just iterate the Map and
// look at the length of each array
for (let [name, array] of lookup) {
if (array.length === 1) {
unique.push(array[0]);
} else {
dup.push(...array);
}
}
console.log("Name count", lookup);
console.log("Unique:", unique);
console.log("Duplicate", dup);
The values data structure is not the most efficient way to store a million records that all have the same two properties. It's possible that the interpreter will recognize the internal repetition and optimize it for you, but it could probably be stored more efficiently as:
const values = [
[ "a", 15 ],
[ "a", 17 ],
[ "a", 17 ],
[ "b", 18 ],
[ "c", 18 ],
[ "d", 18 ],
[ "c", 18 ],
];
Or, even just:
const values = [
"a", 15,
"a", 17,
"a", 17,
"b", 18,
"c", 18,
"d", 18,
"c", 18,
];
where the code had some pre-built knowledge for how to get each individual item. You could consider optimizations like this if the memory usage was too high.

How can I set a jsonarray inside the jsonobject?

How can I set a jsonarray inside the jsonobject?
I want to be able to push the values ​​but I do not know exactly what to do
sample :
{
"ContactsDetails": [
{
"Prefix": "string",
"FirstName": "string",
"LastName": "string",
"Mobile": "string",
"EmojiId": "byte"
}
],
"GroupId": integer
}

Here is the sample code what you're looking for.
let empty = {}; // init a empty object.
empty["ContactsDetails"] = []; // array creation
empty["GroupId"] = 45; // number adding
// creating a new object
let c_details = {
"Prefix": "string",
"FirstName": "string",
"LastName": "string",
"Mobile": "string",
"EmojiId": "byte"
};
// adding an existing obj to array, push operation
empty["ContactsDetails"].push(c_details);
// print object
console.log(empty);
// this will convert the object to JSON.
console.log(JSON.stringify(empty))

What you call a JSONArray is simply a Javascript array like []. You can set it like any property of an Object (JSON).
In Javascript, an Object (symbolized as {}) will be represented as a JSON Object and a Array (symbolized by []) will be represented as a JSON Array.
So you can just do something like this :
var object = {};
object.foo = [];
object.foo.push("newvalue");
object.foo.push("bar");
console.log(object);
And of course, you can push an Object in your array like this :
var array = [];
array.push({ foo: "bar" });
console.log(array);

Arrays and objects are structures that can recursively contain themselves, or each other. JSON is just a format that represents these recursive data structures.
Luckily, you can define all of these in JavaScript using plain literals:
An array literal is wrapped within brackets []
An object literal is wrapped within curly brackets {}
Hence, you can build an array of objects:
[
{name: 'x', data: 0},
{name: 'y', data: 1}
]
Or an object that its parameters are arrays:
{
array0: [0, 1],
array1: [2, 3]
}
And, as I mentioned above, you can also use these structures recursively. For example:
[
{id: 0, array: [0, 1]},
{id: 1, array: [2, 3]}
]

Object Values sorting with umulate order

I have been trying to sort the json response object array in specific order like if the object contains Umlaute words/characters
object {
item: 1,
users: [
{name: "A", age: "23"},
{name: "B", age: "24"},
{name: "Ä", age: "27"}
]
}
Expected name sorting is A, Ä and B.
when trying to sort with localecompare().
object.sort(function (a, b) { return a.localeCompare(b); });
Getting the error like object.sort is not a function. Is there anyway to sort the object array.

You need to take the property of the object for sorting.
var object = { item: 1, users: [{ name: "A", age: "23" }, { name: "B", age: "24" }, { name: "Ä", age: "27" }] };
object.users.sort(function (a, b) { return a.name.localeCompare(b.name); });
console.log(object);
.as-console-wrapper { max-height: 100% !important; top: 0; }

You must apply the sort() function to the array, not the outer object. Objects don't have that method and in the general case key order is not guaranteed, anyways. For "ordered" data always use arrays, not objects!
This is an example that uses the o.users array. The output n is also an array.
NOTE: never abuse reserved keywords as variable names. You use object as a variable name, it is strongly recommended to not do so! I've replaced it with o. Albeit object is not a reserved keyword, Object (with capital O) is. And the two are easily confused.
var o = {
"item": 1,
"users": [
{ "name": "A", "age": "23" },
{ "name": "B", "age": "24" },
{ "name": "Ä", "age": "27" }
]
};
var n = o.users.sort((a, b) => a.name.localeCompare(b.name, "de"));
console.log(n);

How do I transform an array to hold data based on a value in the data?

I have seen some questions that might look similar but none is the solution in my case. I want to regroup and recreate my array the way that it is arranged or grouped based on one of my values(age). I want to have all data of the same "age" in one place. So here is my sample array:
[
{
"age": 15,
"person": {
name: 'John',
hobby: 'ski'
},
},
{
"age": 23,
"person": {
name: 'Suzi',
hobby: 'golf'
},
},
{
"age": 23,
"person": {
name: 'Joe',
hobby: 'books'
}
},{
"age": 25,
"person": {
name: 'Rosi',
hobby: 'books'
}
},{
"age": 15,
"person": {
name: 'Gary',
hobby: 'books'
}
},
{
"age": 23,
"person": {
name: 'Kane',
hobby: 'books'
}
}
]
And I need to have an array that kind of have age as a key and person as value, so each key could have multiple values meaning the value will kind of be an array itself.
I have read this and this questions and many more but they were not exactly the same.
I feel like I need to use reduce to count duplicate ages and then filter it based on that but how do I get the values of those ages?
EIDT:
Sorry for not being clear:
This is what I need:
{
23: [
{ name: 'Suzi', hoby: 'golf' },
{ name: 'Joe', hobby: 'books'}
],
15: [
{ name: 'Gary', hobby: 'books' }
] ,
.
.
.
}

You're actually going to want to reduce, not filter. Filtering an Array means to remove elements and place the kept elements into a new container. Reducing an array means to transform it into a single value in a new container. Mapping an array means to transform every value in place to a new container. Since you want to change how the data is represented that's a Reduction, from one form to another more condensed form.
Assume your Array of values is stored in let people = [...]
let peopleByAge = people.reduce(function (accumulator, value, index, array){
// The first time through accumulator is the passed extra Object after this function
// See the MDN for Array.prototype.reduce() for more information
if (accumulator[value.age] == undefined){
accumulator[value.age] = [];
}
accumulator[value.age].push(value);
return accumulator
}, {})
console.log(peopleByAge) // { 23: [{ age: 23, name: ..., hobby: ...}, ...], 13: [...], ...}
You can find the MDN article for Array#reduce() here

Thanks to #RobertMennell who patiently answered me and I voted as answer. But I just wanted to write my version which MDN had a great example of. It is a longer version assuming the people is the array name:
const groupedByvalue = 'age';
const groupedArray = people;
const groupBy = (peopleArray, value) => {
return peopleArray.reduce((acc, obj) => {
const key = obj[value];
if (!acc[key]) {
acc[key] = [];
}
acc[key].push(obj);
return acc;
}, {});
}
console.log(groupBy(groupedArray,groupedByvalue));
Update:
More polished using ternary operator:
const groupedByvalue = 'age';
const groupedArray = people;
const groupBy = (peopleArray, value) => {
return peopleArray.reduce((acc, obj) => {
const key = obj[value];
(!acc[key]) ? (acc[key] = []) : (acc[key].push(obj))
return acc;
}, {});
}
console.log(groupBy(groupedArray,groupedByvalue));

Issue on having an array in an JavaScript object

Can you please let me know if it is possible to pass an array into a JavaScript object like this?
var content = {
item1: "John",
item2: "Doe",
item3: { "item3_1", "item3_2", "item3_3" }
}
console.log(content);
Can you please let me know how I can access or update the items, or if not, can you please let me know how I can create a data format (Array of Array for example) to store these data?

The syntax for defining an array on a JavaScript object looks like this:
var content = {
item1: "John",
item2: "Doe",
item3: ["item3_1", "item3_2", "item3_3"]
};
You're not "passing" the array into the object, you're simply defining one of the properties to be an array. Any nested data inside that would take the same form that an object or another array would:
var foo = {
arr: [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
};
The important thing to remember is that you use '{}' for object syntax, and '[]' for array syntax. You can also have objects inside that array:
var bar = {
obj_arr: [
{
name: "Tim",
age: 14,
},
{
name: "Bob",
age: 36
},
{
name: "Sue",
age: 73
}
]
};
To access one of the individual elements from the array property, use this syntax:
content.item3[0] for "item3_1"
content.item3[1] for "item3_2"
or, alternatively:
content["item3"][0] for "item3_1"
content["item3"][1] for "item3_2"
although this syntax is less common when the property name is known ahead of time. To loop through the items in that array, you can use a for loop:
for (var i = 0, l = content.item3.length; i < l; i += 1) {
console.log(content.item3[i]);
}
This is how you would console.log the items out - if you wanted to do something else with them, you would need to substitute that action for the console.log call.
Here's an updated fiddle that does what I believe you're looking for: http://jsfiddle.net/qobo98yr/5/
This is the code for printing everything out in the content object to the console:
var outputString = "";
for (var prop in content) {
outputString += (content[prop] + " ");
}
console.log(outputString);

You content variable is an object that contains:
item1: String
item2: String
item3: array
var content = {
item1: "John",
item2: "Doe",
item3: { "item3_1", "item3_2", "item3_3" }
}
If item3 is an array then is should use the array notation
var content = {
item1: "John",
item2: "Doe",
item3: ["item3_1", "item3_2", "item3_3"]
}
Now you can pass this object around and call its fields like that:
function print(obj) {
var fname = obj.item1;
var lname = obj.item2;
var array = obj.item3;
alert(array.toString());
}
where call to the function will look like
print(content);