Develop Reference

Generic way for filtering properties from arbitrary and deeply nested JSON (with arrays)

Goal
I want to develop a middleware in TypeScript that filters the response of a REST API and returns only defined properties.
It should work generically, i.e. independent of specific entities. Neither their properties nor the exact depth (e.g. with any number of relations) should be necessarily known.
Example
An author has any number of articles with any number of comments.
[
{
"name": "John Doe",
"email": "john#doe.com",
"articles": [
{
"title": "Lalilu 1",
"text:": "la li lu",
"comments": [
{
"author": "Bendthatdict Cumberstone",
"text": "Great article!"
},
{
"author": "Bendthatdict Cumberstone",
"text": "Great article!"
}
]
},
{
"title": "Lalilu 1",
"text:": "la li lu",
"comments": [
{
"author": "Bendthatdict Cumberstone",
"text": "Great article!"
},
{
"author": "Bendthatdict Cumberstone",
"text": "Great article!"
}
]
}
]
},
{
"name": "Jane Doe",
"email": "jane#doe.com",
"articles": [
{
"title": "Lalilu 1",
"text:": "la li lu",
"comments": [
{
"author": "Bendthatdict Cumberstone",
"text": "Great article!"
},
{
"author": "Bendthatdict Cumberstone",
"text": "Great article!"
}
]
},
{
"title": "Lalilu 1",
"text:": "la li lu",
"comments": [
{
"author": "Bendthatdict Cumberstone",
"text": "Great article!"
},
{
"author": "Bendthatdict Cumberstone",
"text": "Great article!"
}
]
}
]
}
]
Now I want to specify that it should return everything except the "text" of each article and the "author" of each comment.
Syntax could look like this with glob notation:
select("*,!articles.text,!articles.comments.author")
Approach
For objects and nested objects it is quite simple, e.g. with pick() and omit() of "lodash", but I fail when arrays step into the game.
I did some research and came across packages such as json-mask, node-glob or glob-object but none of them exactly met my needs and I was not able to combine them for success.
Question
What is the most efficient way to generically filter an arbitrarily nested JSON with any number of further objects / arrays?
Also, how could the TypeScripts type system be used to advantage?
I would be very grateful for general coding approaches or even tips for a package that can already do this!

In short I would break this up into functions. You could create helpers that do more or less what you want with a string/filter as you show however I'd work it in reverse. Get a nice way to iterate so any post processing can be done, then build your helpers as you wish against that. Here's what I mean:
Example
export interface IComment {
author: string;
text: string;
}
export interface IArticle {
title: string;
text: string;
comments: IComment[];
}
export interface IComposer {
name: string,
email: string,
articles: IArticle[];
}
// Remove items from list for brevity sake...
const authorList = [
{
"name": "John Doe",
"email": "john#doe.com",
"articles": [
{
"title": "Lalilu 1",
"text": "la li lu",
"comments": [
{
"author": "Bendthatdict Cumberstone",
"text": "Great article!"
}
]
}
]
}
] as IComposer[];
/**
* Accepts JSON string or array of type.
*
* #param arr a JSON string containing array of type or array of type.
*/
export function selectFrom<T extends Record<string, any>>(arr: string | T[]) {
// If you want to use this route I would suggest
// also a function to validate that the JSON is
// shaped correctly.
if (typeof arr === 'string')
arr = JSON.parse(arr);
const collection = arr as T[];
const api = {
filters: [],
register,
run
};
/**
* Register a search op.
* #param fn function returning whether or not to filter the result.
*/
function register(fn: (obj: T) => Partial<T>) {
if (typeof fn === 'function')
api.filters.push(fn);
return api;
}
/**
* Run registered ops and filter results.
*/
function run() {
return collection.reduce((results, obj) => {
let result = obj;
// Don't use reducer here as you can't break
// and would unnecessarily loop through filters
// that have no need to run, use for of instead.
for (const filter of api.filters) {
// if we set the result to null
// don't continue to run filters.
if (!result) break;
// Pipe in the previous result, we start with
// original object but it's shape could change
// so we keep iterating with the previous result.
const filtered = filter(result);
// update the result.
if (filtered)
result = filtered;
}
if (result)
results.push(result);
return results;
// If changing the object you're going to
// end up with partials of the original
// shape or interface.
}, [] as Partial<T>[]);
}
return api;
}
Usage
By making this function based at the core you have a lot more flexibility. From there you could make a simple helper that maps your Glob or SQL like string to the pre-defined filter functions. Let me know if you have further questions.
const filtered =
selectFrom(authorList)
.register((composer) => {
composer.articles = composer.articles.map(article => {
const { text, ...filteredArticle } = article;
filteredArticle.comments = filteredArticle.comments.map(comment => {
const { author, ...filteredComment } = comment;
return filteredComment as typeof comment;
});
// Note setting to type of IArticle here so typescript
// doesn't complain, this is because you are removing props
// above so the shape changes so you may want to consider
// setting the props you plan to strip as optional or make
// everything a partial etc. I'll leave that to you to decide.
return filteredArticle as typeof article;
});
return composer;
})
.run();
What's Next
From here to get where you want it's about string parsing. Keep in mind Lodash does support gets down into nested values in an array. You can see this here in the docs.
Given that you could leverage Lodash using both _.get _.omit... etc along with a little parsing using dot notation.
Done this very thing with permissions. As such I feel strongly you need to start with a simple api to process then from there make your map from either Glob like or SQL string to those helpers.

Loop into Array of Objects comparing two arrays - PUG (Jade)

I am trying in Javascript, using PUG template (if possible), to compare two arrays and when I find a correspondance in IDs, display some particular elements.
// First Array : I iterate over "hearts" object
// Called in PUG : - const user
[
{
"hearts": [
"5e70c63a94b27b164c9b897f",
"5e723c75e4bfdf4f58c55e32"
],
"_id": "5e6bb1189978fd5afc98c57a",
"email": "catherine#catherine.com",
"name": "Catherine",
"photo": "0121b7fe-b2ae-4e75-979d-7dea1a432855.jpeg",
"__v": 0
},
{
"hearts": [
"5e723c75e4bfdf4f58c55e32"
],
"_id": "5e6bc41f5915e3d2980a5174",
"email": "marc#marc.com",
"name": "Marc",
"photo": "4caa7bfb-6408-4893-a78b-fa6e8e5b03e7.png",
"__v": 0
}
]
// Second array : I iterate over "author.hearts" object
// Called in PUG : - const store
[{
"product": {
"categories": [
1,
2
]
},
"_id": "5e6bcc76c4022eae00e22af6",
"date": "2222-02-20T21:22:00.000Z",
"author": {
"hearts": [
"5e723c75e4bfdf4f58c55e32",
"5e70c63a94b27b164c9b897f"
],
"_id": "5e6bb1189978fd5afc98c57a",
"__v": 0
},
"created": "2020-03-13T18:09:58.086Z",
"id": "5e6bcc76c4022eae00e22af6"
}]
I want to loop over the first array, find the first ID (here 5e70c63a94b27b164c9b897f), loop over the second array and see if this ID is present within the "author.hearts" object. If it is not, carry on with the second ID and if it is present, display all the keys (tags, photos, _id, date...) from the object where the ID was found.
In my example, I have just one object in my array, but I'll be having much more later on.
Many thanks for your help

If I'm understanding correctly you can do something like this. Loop through all your users and when you find their id in author.hearts stop the loop there and return the object the user's _id was found in.
var resultFound = undefined;
try {
user.forEach((el) => {
const id = el._id;
const result = store.find(el => el.author.hearts.includes(id));
if (result) {
resultFound = result;
throw resultFound;
}
});
} catch (e) {
if (e !== resultFound) {
throw e;
}
}

Object.assign is modifying original array object

I'm trying to modify a property value of a specific index in my state this property is the post_comments But my problem is the state is being modified even though i am only modifying the copy of it.. The code works how i want it to be but i'm modifying the state so it's probably bad how do i fix this?
socket.on('statusComment', (data) => {
const mutator = Object.assign([], this.state.getStatus);
const index = mutator.findIndex(i => i._id === data._id);
mutator[index].post_comments = data.post_comments; // Replace old post_comments with data.post_comments
console.log(mutator) // Got the post_comments
console.log(this.state.getStatus) // Also modified
// Commented out setState
// this.setState({
// getStatus: mutator
// })
});
Here is a sample data detected by socket
const data = {
post_id: "5b0689f03fb2fd1404f1854d",
post_comments: [{text: 'what'}]
}
This is what my state looks like
const data_arr = [
{
"post_img": [],
"post_date": "2018-05-24T09:46:24.948Z",
"post_comments": [
{
"comment_posted": "2018-05-24T09:46:31.015Z",
"_id": "5b0689f73fb2fd1404f1854e",
"comment_from": {
"photo_url": "png",
"_id": "5af16d60f3957c11e46500ae",
"display_name": "Lumpo"
},
"comment_text": "kaka2"
},
{
"comment_posted": "2018-05-24T09:47:42.752Z",
"_id": "5b068a3e2fdd6f141d5ba995",
"comment_from": {
"photo_url": "png",
"_id": "5af16d60f3957c11e46500ae",
"display_name": "Lumpo"
},
"comment_text": "kaka!"
}
],
"_id": "5b0689f03fb2fd1404f1854d",
"post_description": "get out\r\n",
"post_by": {
"photo_url": "png",
"_id": "5af16d60f3957c11e46500ae",
"display_name": "Lumpo"
},
"__v": 2
}
]
Spread operator is not working logs the same thing with the Object.assign method
// console.log(mutator)
[
{
"post_img": [],
"_id": "5b0694cc7925c914e4d95dda",
"post_description": "test",
"post_by": {
"_id": "5af16d60f3957c11e46500ae",
"display_name": "Lumpo",
"photo_url": "png"
},
"post_comments": [
{
"_id": "5b0694d67925c914e4d95ddb",
"comment_from": {
"photo_url": "png",
"_id": "5af16d60f3957c11e46500ae",
"display_name": "Lumpo"
},
"comment_text": "This comment should only be in the mutator ",
"comment_posted": "2018-05-24T10:32:54.937Z"
}
],
"post_date": "2018-05-24T10:32:44.613Z",
"__v": 0
}
]
// console.log(this.state.getStatus);
[
{
"post_img": [],
"_id": "5b0694cc7925c914e4d95dda",
"post_description": "test",
"post_by": {
"_id": "5af16d60f3957c11e46500ae",
"display_name": "Lumpo",
"photo_url": "png"
},
"post_comments": [
{
"_id": "5b0694d67925c914e4d95ddb",
"comment_from": {
"photo_url": "png",
"_id": "5af16d60f3957c11e46500ae",
"display_name": "Lumpo"
},
"comment_text": "This comment should only be in the mutator ",
"comment_posted": "2018-05-24T10:32:54.937Z"
}
],
"post_date": "2018-05-24T10:32:44.613Z",
"__v": 0
}
]

const mutator = Object.assign([], this.state.getStatus);
its doing shallow/reference copy of array.
So,original array is copied as it is using reference.
Use spread operator to create new copy of array and then do JSON.stringify followed by JSON.parse.U need a deep copy.
let mutator = [...this.state.getStatus];
mutator = JSON.parse(JSON.stringify(mutator));

you can copy your array something like this :
const mutator = [...this.state.getStatus];

Object.assign([], this.state.getStatus)
[] is an array, not an object. This is likely causing a problem.
Edit: See Josh’s comment, it is an object, but also an array. But the behaviour will be different to if it were an object object.

The quickest way to make a copy of an existing array without copying a reference is the following:
const mutator = this.state.getStatus.slice(0);
as described here https://developer.mozilla.org/de/docs/Web/JavaScript/Reference/Global_Objects/Array/slice

Your state is an object containing an array of objects.
First you copy state and reset the array getStatus with getStatus mapped. When the status item is found that needs to change you copy that item but set post_comments with another value (see code below).
this.setState({
...this.state,//copy state
getStatus: this.state.getStatus.map(//getStatus is a new array
(item,index)=>
(item._id===data._id)//if item._id is data._id
? {...item,post_comments:data.post_comments}//copy item but reset post_comments
: item//not the item we are looking for, return item (not changed copy)
)
})
If you need more help the please let me know.
Using JSON.parse(JSON.stringify(state))) will cause all components to re render even if their part of the state did not change (deep copy versus shallow copy). You can use shouldComponentUpdate to see if the state actually changed and tell react not to re render components where this did not happen. However; since you are deep copying everything (not only the items that changed) you cannot do this.
Here is an example of a base component that checks if the state passed to it actually changed reference and should re render:
import React from 'react';
class OnlyIfChanged extends React.Component {
shouldComponentUpdate(nextProps, nextState) {
return nextProps.state !== this.props.state;
}
}
export default OnlyIfChanged;

Change in array format

I want to create a array structure with child entities like this ->
$scope.groups = [
{
"categories": [
{
"name": "PR",
"sortOrder": 0,
"type": "category"
}
],
"name": "DEPT 1",
"sortOrder": 0,
"type": "group",
"id": "-JY_1unVDQ5XKTK87DjN",
"editing": false
}
];
from an array that dosen't have child entities but all the items are listed in one object like this->
$scope.groups = [
{
"name": "PR",
"sortOrder": 0,
"type": "category"
},
{
"name": "AD",
"sortOrder": 3,
"type": "category"
},
{
"name": "DEPT 2",
"sortOrder": 1,
"type": "group",
"id": "-JYZomQKCVseJmaZoIF9",
"editing": false,
"categories": []
},
];
Is there any possible way?

As #Eagle1 has rightly pointed out. You need to define your data model properly to define a function that does that grouping for you. That said, from what I understand you have a $scope.groups array of objects for a specific department containing some categories which you need to consolidate as a child element.
You could start by defining a function that returns an object like you mention:
var organize = function(arr){
cats = [];
dep = {};
$.each( arr, function( i, val ) {
if(val.type == "category")
cats.push(val);
else
dep = val;
});
dep.categories = cats;
return dep;
}
Ultimately, you'll have to traverse the array and look for objects of type category and dump them in an array and have that array as the categories key of the object that you intend to return. I hope it gets you started in the right direction.

of course it is.
It's doable in javascipt although to help you devise something we would need a relationship between categories.
However, that's sounds like something that should be done in your data model (a relationship between dept - category, classic reflexive relationship parent - children). angular should be receiving from the back end an array already ordered.

How to filter a JSON tree according to an attribute inside

I have to re-post this questions with more details again:
I got a JSON tree array.
The structure of JSON tree looks like this:
{
"app": {
"categories": {
"cat_222": {
"id": "555",
"deals": [{
"id": "73",
"retailer": "JeansWest"
}, {
"id": "8630",
"retailer": "Adidas"
}, {
"id": "11912",
"retailer": "Adidas"
}]
},
"cat_342": {
"id": "232",
"deals": [{
"id": "5698",
"retailer": "KFC"
}, {
"id": "5701",
"retailer": "KFC"
}, {
"id": "5699",
"retailer": "MC"
}]
}
}
}
}
now, I'd like to filter this JSON tree with var pattern="KF",
return all with retailer name contains KF with it's id.
======================update===========================
Just check my other question. It got solved.
filter multi-dimension JSON arrays

Use Array.filter or _.filter if you need to support IE < 9

Well, you can use _.filter:
var filteredArray = _.filter(arrayOfStrings, function(str) {
return str.indexOf(data) !== -1; });
... or jQuery.grep:
var filteredArray = $.grep(arrayOfStrings, function(str) {
return str.indexOf(data) !== -1; });
As you see, the approaches are quite similar - and, in fact, both use Array.filter, if it's available in the host environment.
Also note that the original array is not affected here. If you want otherwise, just assign the result of filtering to the same variable (i.e., arrayOfStrings in this example).