Develop Reference

How to perform fast search on JSON file?

I have a json file that contains many objects and options.
Each of these kinds:
{"item": "name", "itemId": 78, "data": "Some data", ..., "option": number or string}
There are about 10,000 objects in the file.
And when part of item value("ame", "nam", "na", etc) entered , it should display all the objects and their options that match this part.
RegExp is the only thing that comes to my mind, but at 200mb+ file it starts searching for a long time(2 seconds+)
That's how I'm getting the object right now:
let reg = new RegExp(enteredName, 'gi'), //enteredName for example "nam"
data = await fetch("myFile.json"),
jsonData = await data.json();
let results = jsonData.filter(jsonObj => {
let item = jsonObj.item,
itemId = String(jsonObj.itemId);
return reg.test(item) || reg.test(itemId);
});
But that option is too slow for me.
What method is faster to perform such search using js?

Looking up items by item number should be easy enough by creating a hash table, which others have already suggested. The big problem here is searching for items by name. You could burn a ton of RAM by creating a tree, but I'm going to go out on a limb and guess that you're not necessarily looking for raw lookup speed. Instead, I'm assuming that you just want something that'll update a list on-the-fly as you type, without actually interrupting your typing, is that correct?
To that end, what you need is a search function that won't lock-up the main thread, allowing the DOM to be updated between returned results. Interval timers are one way to tackle this, as they can be set up to iterate through large, time-consuming volumes of data while allowing for other functions (such as DOM updates) to be executed between each iteration.
I've created a Fiddle that does just that:
// Create a big array containing items with names generated randomly for testing purposes
let jsonData = [];
for (i = 0; i < 10000; i++) {
var itemName = '';
jsonData.push({ item: Math.random().toString(36).substring(2, 15) + Math.random().toString(36).substring(2, 15) });
}
// Now on to the actual search part
let returnLimit = 1000; // Maximum number of results to return
let intervalItr = null; // A handle used for iterating through the array with an interval timer
function nameInput (e) {
document.getElementById('output').innerHTML = '';
if (intervalItr) clearInterval(intervalItr); // If we were iterating through a previous search, stop it.
if (e.value.length > 0) search(e.value);
}
let reg, idx
function search (enteredName) {
reg = new RegExp(enteredName, 'i');
idx = 0;
// Kick off the search by creating an interval that'll call searchNext() with a 0ms delay.
// This will prevent the search function from locking the main thread while it's working,
// allowing the DOM to be updated as you type
intervalItr = setInterval(searchNext, 0);
}
function searchNext() {
if (idx >= jsonData.length || idx > returnLimit) {
clearInterval(intervalItr);
return;
}
let item = jsonData[idx].item;
if (reg.test(item)) document.getElementById('output').innerHTML += '<br>' + item;
idx++;
}
https://jsfiddle.net/FlimFlamboyant/we4r36tp/26/
Note that this could also be handled with a WebWorker, but I'm not sure it's strictly necessary.
Additionally, this could be further optimized by utilizing a secondary array that is filled as the search takes place. When you enter an additional character and a new search is started, the new search could begin with this secondary array, switching to the original if it runs out of data.

Problems with immutable array

Recently i started working with D3.js to plot a sunburn graph. The data is provided in JSON. For some design stuff i wanted to swap some items (called childrens in D3 doc).
I know in JS arrays are objects...so something like this:
var buffer = myarray[2];
is just a reference. Therefore a buffer for swapping has no effect (?).
Thus i invented a second array (childrens_final) in my code which adopt the items while the swapping process. Its just iterating through every item, a second iteration is looking for an item with the same name to set items with same name in a row. Therefore the swap.
var childrens = response_data.data.data['children'];
var childrens_final = []
for (var child = 0; child < childrens.length; child++) {
var category = childrens[child];
var found = false;
var i = child+1
while (!(found) && (i < childrens.length)) {
if (childrens[i]['name'] == category['name']) {
var childrens = swapArrayElements(childrens, child+1, i);
var one = childrens[child];
var two = childrens[child+1]
found = true;
}
i++;
}
if (found) {
childrens_final.push(one);
childrens_final.push(two);
child++;
}
else {
childrens_final.push(childrens[child])
}
}
response.data.data['children'] = childrens_final;
return response.data.data;
The function swapArrayElements() is just using splice:
function swapArrayElements(list, x, y) {
if (list.length ==1) return list;
list.splice(x, 1, list.splice(y,1, list[x])[0]);
return list;
}
The problem is that there is still no effect from the swap in the graph. But when logging the childrens_final. There is something like that in the console:
Array [ Object, Object, Object, Object, Object, Object, Object, Object, Object ]
The objects are in the right order! But in the array there is still the old order.
Thats actually so basic, but i dont see a solution.
Btw...the code is working under AngularJS.

Found the problem. It's a D3.js problem. D3 is sorting the data itself. You have to set explicitly:
d3.layout.partition.sort(null)
Otherwise every pre sorting process has no effect.

How to achieve a sorted but random list with javascript/jquery

I currently use this code to randomly sort my list of items and it works quite well:
$(document).ready(function () {
(function ($) {
$.fn.randomize = function (childElem) {
return this.each(function () {
var $this = $(this);
var elems = $this.children(childElem);
elems.sort(function () {
return (Math.round(Math.random()) - 0.5);
});
$this.remove(childElem);
for (var i = 0; i < elems.length; i++)
$this.append(elems[i]);
});
};
})(jQuery);
(function ($) {
if (window.location.href.indexOf("st=") > -1) {
} else {
$('#itemBlock').randomize('.list_1');
}
})(jQuery);
});
However, I am stumped when it comes to figuring out a way to modify this script to randomly return results in a certain order. For instance, the list I use this with returns all of a specific type of item from all of 5 stores. So if someone is looking for an apron it will show all of the aprons we have at each of our 5 stores.
The code above surely randomizes the list, but I don't know how to take into account the store location.
So right now a search for aprons would initially return from stores A, B, C, D, and E a list like this:
BCDAEBCDDDCEABBACACABCDAA
But what we want to achieve is a return like this:
ABCDEABCDEABCDEABCDEABCDE
Not necessarily in exactly that order but someway to modify the function so that a result from each store is loaded before more results are loaded. So there wouldn't be 3 in a row from one store, or one store would be represented more than others.
Do I go about this by using the filter or sort function? Or would pure javascript handle this more efficiently?

To achieve this, you would have to follow the following steps:
Randomly sort the items from each store individually
Number the items in each randomised list
OPTIONAL - if you want the stores in order, pad the number with zeros and append an identifier for the store ([padded item number]|[store identifier] eg 00001|A)
Combine all the lists into one
Sort this new list by the ordered number (or string)

Knockout JS: Sorting and splicing an observable array of observable numbers

Background
To learn Knockout JS, I'm (slowly) building a gradebook in the browser. My lastest issue involves dropping the lowest scores in an observable array. I have a student model, which has an observable array called scores. This array consists of observable scores, which are plain numbers.
My methodology for removing the lowest grades is as follows. First, I sort each array of scores, high-low, then, for now, splice the end of the array, such that the two lowest numbers are stored into a new array called low. The low variable will be used later when I calculate the mean.
The Problem(s)
First, my current dropLowestGrades method outright removes data from the dom, which I do not desire. Second, myObservableArray.sort() does not appear to do any sorting! I'm not sure where to go here. Relevant script follows.
JSBin: http://jsbin.com/fehoq/141/edit
JS
var StudentsViewModel = (function () {
function StudentsViewModel() {
var _this = this;
...
this.dropLowestScores = function() {
var low = [];
ko.utils.arrayForEach(_this.students(), function(student){
console.log(student.fullName());
student.scores().sort();
/*
student.scores().sort(function(a, b) {
return a() == b() ? 0 : (a() > b() ? 1 : -1);
});
*/
low = student.scores.splice((student.scores().length-2),student.scores().length);
console.log('low: ' + low);
return low;
});
};
HTML
I currently just bind the function to a button. For simplicity's sake, I'm hard-coding the drop to be two scores. I will later allow the user to pass in a value. Note that the button is named "Sort" right now because I was originally going to have a sort feature, then build my dropLowestScores method on top it.
<button data-bind="click: dropLowestScores">Sort</button>
Update
I have significantly updated my method after insights from the answers. The script below still suffers from cutting the values in my scores array, which I do not wish to change.
this.dropLowestScores = function() {
ko.utils.arrayForEach(_this.students(), function(student){
console.log(student.fullName());
var tmp = student.scores().sort();
console.log(tmp);
student.lowest = tmp.splice((tmp.length-2),tmp.length);
console.log('student lowest: ' + student.lowest);
});
};
Update 2
I changed my StudentModel such that it now has property lowest, which keeps track of the lowest scores when user drops grades.
var StudentModel = (function () {
function StudentModel(fullName) {
var _this = this;
this.fullName = ko.observable(fullName);
this.scores = ko.observableArray();
this.lowest = [];
...

You need to remember that functions like sort() return a sorted list of the array, they don't actually transform the array itself.
var studentScores = student.scores().sort();
or something like -
var sortedStudentScores(function () {
return student.scores().sort();
});
And if you are sorting on a property of the score you need to use something like -
var sortFunction = // Google a JavaScript sort function
var studentScores = student.scores().sort(sortFunction);
and if you are trying to remove items splicing is correct (it transforms the array itself) otherwise you need to use something like a computed to just not add the lowest in.
Updated
var sortedStudentScores(function () {
// Set a local var equal to the sorted scores
var theseScores = student.scores().sort().splice(0);
theseScores.splice(student.scores().length-2),student.scores().length);
});

How to keep Javascript array sorted, without sorting it

I have a Node.js application where I have to very often do following things:
- check if particular array already contains certain element
- if element does exist, update it
- if element do not exist, push it to the array and then sort it using underscore _.sortBy
For checking if the element already exists in the array, I use this binary search function:
http://oli.me.uk/2013/06/08/searching-javascript-arrays-with-a-binary-search/
In this way, when the size of the array grows, the sorting becomes slower and slower.
I assume that the array size might grow to max 20 000 items per user. And eventually there will be thousands of users. The array is sorted by a key, which is quite a short string. It can be converted into integer if needed.
So, I would require a better way to keep the array sorted,
in stead of sorting it every time new element is pushed onto it.
So, my question is, how should/could I edit the binary search algorithm I use, to enable me to
get the array index where the new element should be placed, if it doesn't already exist in the array?
Or what other possibilities there would be to achieve this. Of course, I could use some kind of loop that would start from the beginning and go through the array until it would find the place for the new element.
All the data is stored in MongoDB.
In other words, I would like to keep the array sorted without sorting it every time a new element is pushed.

It's easy to modify this binaryIndexOf function to return an index of the next element when no matches found:
function binaryFind(searchElement) {
'use strict';
var minIndex = 0;
var maxIndex = this.length - 1;
var currentIndex;
var currentElement;
while (minIndex <= maxIndex) {
currentIndex = (minIndex + maxIndex) / 2 | 0; // Binary hack. Faster than Math.floor
currentElement = this[currentIndex];
if (currentElement < searchElement) {
minIndex = currentIndex + 1;
}
else if (currentElement > searchElement) {
maxIndex = currentIndex - 1;
}
else {
return { // Modification
found: true,
index: currentIndex
};
}
}
return { // Modification
found: false,
index: currentElement < searchElement ? currentIndex + 1 : currentIndex
};
}
So, now it returns objects like:
{found: false, index: 4}
where index is an index of the found element, or the next one.
So, now insertion of a new element will look like:
var res = binaryFind.call(arr, element);
if (!res.found) arr.splice(res.index, 0, element);
Now you may add binaryFind to Array.prototype along with some helper for adding new elements:
Array.prototype.binaryFind = binaryFind;
Array.prototype.addSorted = function(element) {
var res = this.binaryFind(element);
if (!res.found) this.splice(res.index, 0, element);
}

If your array is already sorted and you want to insert an element, to keep it sorted you need to insert it at a specific place in the array. Luckily arrays have a method that can do that:
Array.prototype.splice
So, once you get the index you need to insert at (you should get by a simple modification to your binary search), you can do:
myArr.splice(myIndex,0,myObj);
// myArr your sorted array
// myIndex the index of the first item larger than the one you want to insert
// myObj the item you want to insert
EDIT: The author of your binary search code has the same idea:
So if you wanted to insert a value and wanted to know where you should
put it, you could run the function and use the returned number to
splice the value into the array.
Source

I know this is an answer to an old question, but the following is very simple using javascripts array.splice().
function inOrder(arr, item) {
/* Insert item into arr keeping low to high order */
let ix = 0;
while (ix < arr.length) {
//console.log('ix',ix);
if (item < arr[ix]) { break; }
ix++;
}
//console.log(' insert:', item, 'at:',ix);
arr.splice(ix,0,item);
return arr
}
The order can be changed to high to low by inverting the test