Develop Reference

Efficient way to search within multiple values in json

I have multiple records like this,
name: John Doe aliases: John, Doe, JD unique_id: 1 ...
My question is how do I search efficiently within the aliases & full name.
If the search query is any of those 4 (John Doe, John, Doe, JD) I would like to find the unique id (in this case 1).
What I have done: I have a very straightforward implementation that loops through the entire data until it finds. It takes a long time since the number of fields is very high.
Note: I am using javascript if it helps. Also I have the permission to change the data format (permanently), if it will make the search more efficient. Most of the search queries tend to be one of the aliases rather than full name.
Sample Code: https://jsfiddle.net/nh7yqafh/
function SearchJSON(json, query) {
var champs = json.champs;
for (var i = 0; i < champs.length; ++i) {
if (query == champs[i].name)
return champs[i].unique_id;
for (var j = 0; j < champs[i].aliases.length; ++j) {
if (query == champs[i].aliases[j])
return champs[i].unique_id;
}
}
}
//Data format is similar to what vivick said
var json_string = '{"count":5,"champs":[{"name":"Abomination","aliases":["abomination","AB","ABO"],"unique_id":1},{"name":"Black Bolt","aliases":["blackbolt","BB","BBT"],"unique_id":2},{"name":"Black Panther","aliases":["blackpanther","BP","BPR"],"unique_id":3},{"name":"Captain America","aliases":["captainamerica","CA","CAP"],"unique_id":4}]}'
var json = JSON.parse(json_string);
query="CA";
alert( "id of "+query+" is "+SearchJSON(json, query));

I guess you have a structure similar to the following one :
[
{
"name": "xxx",
"aliases": ["x", "xx", "xxx"],
"unique_id": 1,
/* [...] */
},
/* [...] */
]
You can then do something like this :
const queryParam = /*search query*/;
const arr = /* get the JSON record */;
const IDs = arr
.filter( entry =>(entry.aliases.includes(queryParam) || queryParam===entry.name) )
.map(entry=>entry.uniqueId);
This will give you an array of IDs which are potential matches.
If you need either 0 or 1 result only :
const ID = IDs[0] || null;
This will simply retrieve the first matched ID if there's one, otherwise it will set ID to null.
NB:
If you use an object of objects instead of an array of object, there's just a little bit of modifications to do (mainly using Object.entries) but it still is trivial.
PS:
I would recommend to always add the full name in the aliases, this will ease the filtering part (no || would be required).

Making a Search Filter with JQuery?

So I have a Table made from some json data...
{
"AKH":{
"name": "Amonkhet",
"code": "AKH"
"cards": [
{
"artist": "Izzy",
"cmc": 3,
"colorIdentity": [
"W"
],
"colors": [
"White"
],
"id": "df3a6e0336684c901358f3ff53ec82ff5d7cdb9d",
"imageName": "gideon of the trials",
"layout": "normal",
"loyalty": 3,
"manaCost": "{1}{W}{W}",
"multiverseid": 426716,
"name": "Gideon of the Trials",
"number": "14",
"rarity": "Mythic Rare",
"subtypes": [
"Gideon"
],
"text": "+1: Until your next turn, prevent all damage target permanent would deal.\n0: Until end of turn, Gideon of the Trials becomes a 4/4 Human Soldier creature with indestructible that's still a planeswalker. Prevent all damage that would be dealt to him this turn.\n0: You get an emblem with \"As long as you control a Gideon planeswalker, you can't lose the game and your opponents can't win the game.\"",
"type": "Planeswalker — Gideon",
"types": [
"Planeswalker"
]
},
The Table row ends up looking like this for each of the cards. at the moment I only Attach the ID, Card name, and Mana Cost to each row
<td><a href="#" onclick="showInfo(this.id)"
id="df3a6e0336684c901358f3ff53ec82ff5d7cdb9d">Gideon of the Trials</a></td>
Now I want to search through these cards. (Keep in mind there are over 17,000 different cards that will be on this list) I can get it to find the things.. But I'm having several different issues... Either it finds them all but doesn't hide the rest of the list, or it hides the whole list and only displays one of the found cards.
So question A... What am I missing to make the search work correctly?
$(document).on('change', 'input[type=checkbox]', function() {
var lis = $('.cardsRow')
$('input[type=checkbox]').filter(':checked').each(function(){
filterKeyB = $(this).attr('id')
filterKeyA = $(this).attr('name')
$.each(json, function(setCode, setListing) {
$.each(setListing.cards,function(cardNum, cardListing){
var x = Object.keys(cardListing)
var y = Object.keys(cardListing).map(function (key){
return cardListing[key]
})
for (i = 0; (i < x.length); i++) {
if(x[i] === filterKeyA){
if (y[i] instanceof Array){
var holder = y[i]
var valueArr =[]
for(var k = 0; k < holder.length; k++){
valueArr = holder.join('|').toLowerCase().split('|')
var foundIt = valueArr.includes(filterKeyB)
}
}else{
var stringy = y[i]
var stringyA= stringy.toLowerCase().replace(/\s/g, '')
if (stringyA === filterKeyB){
var foundIt = true
}
}
if(foundIt === true){
$winner = cardListing.name
for (k = 0; (k < lis.length); k++){
if (lis[k].innerText.indexOf($winner) != -1) {
$(lis[k]).show()
}
}
}
}
}
})
Question B... Since you are already here... Would it be better practice to attach the data that can be searched to the element itself? Maybe just the most searched (Like Name and Mana) and have more advanced queries go through the data again?

I don't understand why the code isn't working or even how it works, it looks like it references some functions that aren't defined in the sample. But I can share with you a really simple/intuitive way to filter stuff, I hope you find it useful.
Native filter method is so useful for what you're trying to do, it takes a callback that takes current element as an arg and returns true or false, if true, the element is included in the new array it produces.
But filter only takes one function, and you have many filters, so let's make a function that combines many filter Fns together into one fn, so you can pass them in all at once:
const combineFilters = (...fns) => val => fns.reduce((prev, curr) => prev || curr(val), false);
OK, how about storing the names of the filter functions as keys in an object so we can reference them using a string? That way we could give each checkbox an ID corresponding to the name of the filter function they are supposed to apply, and makes things really easy to implement (and read):
const filterFns = {
startsWithG(card) {
return card.name[0] === 'G';
},
//etc.
};
OK, time to get the IDs of all the checkboxes that are clicked, then map them into an array of functions.
const filters = $('input[type=checkbox]')
.filter(':checked')
.map((e, i) => $(i).attr('id'))
.get()
.map(fnName => filterFns[fnName])
(Assume the relevant data is stored in a var called...data.) We can use combineFilters combined with filters (array of Fns) to activate all of the relevant filters, then map the resulting array of matching objects into the HTML of your choosing.
const matches = data.cards
.filter(combineFilters(...filters))
.map(card => `<div>${card.name}</div>` );
Then time to update DOM with your matches!
As others have noted, if you need to do any more complicated filtering on objects or arrays, lodash library is your friend!

Fastest datastructure for filtering schema-less collections

Lets say I have a collection
var data = [
{ fieldA: 5 },
{ fieldA: 142, fieldB: 'string' },
{ fieldA: 1324, fieldC: 'string' },
{ fieldB: 'string', fieldD: 111, fieldZ: 'somestring' },
...
];
Lets assume fields are not uniform across elements but I know in advance the number of unique fields, and that the collection is not dynamic.
I want to filter it with something like _.findWhere. This is simple enough, but what if I want to prioritize speed over ease? Is there a better data structure that will always minimize the number of elements that will be checked? Perhaps some kind of tree?

Yes, there is something faster if your queries are of the type "give me all records with fieldX=valueY". However, it does have an overhead.
For each field, build an inverted index that lists all the record-ids ( = row positions in the original data) that have each value:
var indexForEachField = {
fieldA: { "5": [0], "142": [1], "1324": [2]},
...
}
When someone asks for "records where fieldX=valueY", you return
indexForEachField["fieldX"]["valueY"]; // an array with all results
Lookup time is therefore constant (and requires only 2 lookups in tables), but you do need to keep your index up to date.
This is a generalization of the strategy used by search engines to look up webpages with certain terms; in that scenario, it is called an inverted index.
Edit: what if you want to find all records with fieldX=valueX and fieldY=valueY?
You would use the following code, which requires all input arrays
to be sorted:
var a = indexForEachField["fieldX"]["valueX"];
var b = indexForEachField["fieldY"]["valueY"];
var c = []; // result array: all elements in a AND in b
for (var i=0, j=0; i<a.length && j<b.length; /**/) {
if (a[i] < b[j]) {
i++;
} else if (a[i] > b[j]) {
j++;
} else {
c.push(a[i]);
i++; j++;
}
}
You can see that, in the worst case, the total complexity is exactly a.length + b.length; and, in the best case, half of that. You can use something very similar to implement OR.

Loop Through JSON, Insert Key/Value Between Objects?

UPDATE - Thanks for all the great answers and incredibly fast response. I've learned a great deal from the suggested solutions. I ultimately chose the answer I did because the outcome was exactly as I asked, and I was able to get it working in my application with minimal effort - including the search function. This site is an invaluable resource for developers.
Probably a simple task, but I can't seem to get this working nor find anything on Google. I am a Javascript novice and complex JSON confuses the hell out of me. What I am trying to do is make a PhoneGap Application (Phone Directory) for our company. I'll try to explain my reasoning and illustrate my attempts below.
I have JSON data of all of our employees in the following format:
[
{
"id":"1",
"firstname":"John",
"lastname":"Apple",
"jobtitle":"Engineer"
},
{
"id":"2",
"firstname":"Mark",
"lastname":"Banana",
"jobtitle":"Artist"
},
... and so on
]
The mobile framework (Framework 7) that I am using offers a "Virtual List" solution which I need to take advantage of as our directory is fairly large. The virtual list requires you to know the exact height of each list item, however, you can use a function to set a dynamic height.
What I am trying to do is create "headers" for the alphabetical listing based on their last name. The JSON data would have to be restructured as such:
[
{
"title":"A"
},
{
"id":"1",
"firstname":"John",
"lastname":"Apple",
"jobtitle":"Engineer"
},
{
"title":"B"
},
{
"id":"2",
"firstname":"Mark",
"lastname":"Banana",
"jobtitle":"Artist"
},
... and so on
]
I've been able to add key/value pairs to existing objects in the data using a for loop:
var letter, newLetter;
for(var i = 0; i < data.length; i++) {
newLetter = data[i].lastname.charAt(0);
if(letter != newLetter) {
letter = newLetter
data[i].title = letter;
}
}
This solution changes the JSON, thus outputting a title bar that is connected to the list item (the virtual list only accepts ONE <li></li> so the header bar is a div inside that bar):
{
"id":"1",
"firstname":"John",
"lastname":"Apple",
"jobtitle":"Engineer",
"title":"A"
},
{
"id":"1",
"firstname":"Mike",
"lastname":"Apricot",
"jobtitle":"Engineer",
"title":""
}
This solution worked until I tried implementing a search function to the listing. When I search, it works as expected but looks broken as the header titles ("A", "B", etc...) are connected to the list items that start the particular alphabetical section. For this reason, I need to be able to separate the titles from the existing elements and use them for the dynamic height / exclude from search results.
The question: How can I do a for loop that inserts [prepends] a NEW object (title:letter) at the start of a new letter grouping? If there is a better way, please enlighten me. As I mentioned, I am a JS novice and I'd love to become more efficient programming web applications.

var items = [
{ "lastname":"Apple" },
{ "lastname":"Banana" },
{ "lastname":"Box" },
{ "lastname":"Bump" },
{ "lastname":"Can" },
{ "lastname":"Switch" }
];
var lastC = null; //holds current title
var updated = []; //where the updated array will live
for( var i=0;i<items.length;i++) {
var val = items[i]; //get current item
var firstLetter = val.lastname.substr(0,1); //grab first letter
if (firstLetter!==lastC) { //if current title does not match first letter than add new title
updated.push({title:firstLetter}); //push title
lastC = firstLetter; //update heading
}
updated.push(val); //push current index
}
console.log(updated);

Well right now you have an array of objects - prefixing the title as its own object may be a bit confusing - a better structure may be:
[
{
title: "A",
contacts: [
{
"id":"1",
"firstname":"John",
"lastname":"Apple",
"jobtitle":"Engineer",
"title":"A"
}
]
Given your current structure, you could loop and push:
var nameIndexMap = {};
var newContactStructure = [];
for (var i = 0; i < data.length; i++) {
var letter = data[i].lastname.charAt(0);
if (nameIndexMap.hasOwnProperty(letter)) {
//push to existing
newContactStructure[nameIndexMap[letter]].contacts.push(data[i])
} else {
//Create new
nameIndexMap[letter] = newContactStructure.length;
newContactStructure.push({
title: letter,
contacts: [
data[i]
]
});
}
}
newContactStructure will now contain your sorted data.
Demo: http://jsfiddle.net/7s50k104/

Simple for loop with Array.prototype.splice will do the trick:
for (var i = 0; i < data.length; i++) {
if (i == 0 || data[i-1].lastname[0] !== data[i].lastname[0]) {
data.splice(i, 0, {title: data[i].lastname[0]});
i++;
}
}
Demo. Check the demo below.
var data = [
{"lastname":"Apple"},
{"lastname":"Banana"},
{"lastname":"Bob"},
{"lastname":"Car"},
{"lastname":"Christ"},
{"lastname":"Dart"},
{"lastname":"Dog"}
];
for (var i = 0; i < data.length; i++) {
if (i == 0 || data[i-1].lastname[0] !== data[i].lastname[0]) {
data.splice(i, 0, {title: data[i].lastname[0]});
i++;
}
}
alert(JSON.stringify( data, null, 4 ));

Update multiple documents / add keys and its values in to a new array

I have collection named inventory where I have multiple documents that has values for each doc
{ "apples": 2 ,"oranges": 3, "carrots": 5 }
{ "apples": 4, "oranges": 6, "carrots": 9 }
How do I update push all fruits in to a single array on multiple documents like so:
{ "fruits": { "apples":2 ,"oranges":3 }, "carrots": 5 }

First thing to note here is that the example you give is not an array but just a sub-document for "fruits" that has different keys. An "array" in MongoDB would look like this:
{ "fruits": [{ "apples":2 } , { "orange":3 }], "carrot": 5 }
Also, aside from the term "fruits" being subjective, as with no other identifier you would have to specify a "list" of things that qualify as fruits, the other thing to consider is that there is no actual way in MongoDB at present to refer to the existing value of a field when processing an update.
What that means is you need to .find() each document to retrieve the data in order to be able to work with the sort of "re-structure" that you want. This essentially means looping the results an performing an .update() operation for each document.
The Bulk API for MongoDB 2.6 and greater can be of some help here, where at least the "write" operations to the database can be sent in batches, rather than one at a time:
var bulk = db.collection.initializeOrderedBulkOp();
var count = 0;
var fruits = ["apples","oranges"];
var unset = {};
fruits.forEach(function(fruit) {
unset[fruit] = 1;
});
db.collection.find({}).forEach(function(doc) {
var fields = [];
fruits.forEach(function(fruit) {
if ( doc.hasOwnProperty(fruit) ) {
var subDoc = {};
subDoc[fruit] = doc[fruit];
fields.push(subDoc);
}
});
bulk.find({ "_id": doc._id }).updateOne({
"$unset": unset, "$push": { "fruits": { "$each": fields } }
});
count++;
if ( count % 1000 == 0 ) {
bulk.execute();
var bulk = db.collection.initializeOrderedBulkOp();
}
});
if ( count % 1000 != 0 )
bulk.execute();
That also uses the $each modifier for $push in order to add multiple array entries at once. The $unset operator can be safely called for fields that don't exist in the document so there is no need to check for their presence in the document as is otherwise required when constructing the array of elements to $push.
Of course if you actually want a document like what you gave an example of that is not actually an array, then you construct differently with the $set operator:
var fields = {};
fruits.forEach(function(fruit) {
if ( doc.hasOwnProperty(fruit) )
fields[fruit] = doc[fruit];
});
bulk.find({ "_id": doc._id }).updateOne({
"$unset": unset, "$set": { "fruits": fields }
});
Whatever the case is you need to loop the existing collection. There is no operation that allows you to "take" an existing value in a document and "use it" in order to set a new value from a server side perspective.