Develop Reference

How do I loop over a VERY LARGE 2D array without causing a major performace hit?

I am attempting to iterate over a very large 2D array in JavaScript within an ionic application, but it is majorly bogging down my app.
A little background, I created custom search component with StencilJS that provides suggestions upon keyup. You feed the component with an array of strings (search suggestions). Each individual string is tokenized word by word and split into an array and lowercase
For example, "Red-Winged Blackbird" becomes
['red','winged','blackbird']
So, tokenizing an array of strings looks like this:
[['red','winged','blackbird'],['bald','eagle'], ...]
Now, I have 10,000+ of these smaller arrays within one large array.
Then, I tokenize the search terms the user inputs upon each keyup.
Afterwards, I am comparing each tokenized search term array to each tokenized suggestion array within the larger array.
Therefore, I have 2 nested for-of loops.
In addition, I am using Levenshtein distance to compare each search term to each element of each suggestion array.

I had a couple drinks so please be patient while i stumble through this.
To start id do something like a reverse index (not very informative). Its pretty close to what you are already doing but with a couple extra steps.
First go through all your results and tokenize, stem, remove stops words, decap, coalesce, ects. It looks like you've already done this but i'm adding an example for completion.
const tokenize = (string) => {
const tokens = string
.split(' ') // just split on words, but maybe rep
.filter((v) => v.trim() !== '');
return new Set(tokens);
};
Next what we want to do is generate a map that takes a word as an key and returns us a list of document indexes the word appears in.
const documents = ['12312 taco', 'taco mmm'];
const index = {
'12312': [0],
'taco': [0, 1],
'mmm': [2]
};
I think you can see where this is taking us... We can tokenize our search term and find all documents each token belongs, to work some ranking magic, take top 5, blah blah blah, and have our results. This is typically the way google and other search giants do their searches. They spend a ton of time in precomputation so that their search engines can slice down candidates by orders of magnitude and work their magic.
Below is an example snippet. This needs a ton of work(please remember, ive been drinking) but its running through a million records in >.3ms. Im cheating a bit by generate 2 letter words and phrases, only so that i can demonstrate queries that sometimes achieve collision. This really doesn't matter since the query time is on average propionate to the number of records. Please be aware that this solution gives you back records that contain all search terms. It doesn't care about context or whatever. You will have to figure out the ranking (if your care at this point) to achieve the results you want.
const tokenize = (string) => {
const tokens = string.split(' ')
.filter((v) => v.trim() !== '');
return new Set(tokens);
};
const ri = (documents) => {
const index = new Map();
for (let i = 0; i < documents.length; i++) {
const document = documents[i];
const tokens = tokenize(document);
for (let token of tokens) {
if (!index.has(token)) {
index.set(token, new Set());
}
index.get(token).add(i);
}
}
return index;
};
const intersect = (sets) => {
const [head, ...rest] = sets;
return rest.reduce((r, set) => {
return new Set([...r].filter((n) => set.has(n)))
}, new Set(head));
};
const search = (index, query) => {
const tokens = tokenize(query);
const canidates = [];
for (let token of tokens) {
const keys = index.get(token);
if (keys != null) {
canidates.push(keys);
}
}
return intersect(canidates);
}
const word = () => Math.random().toString(36).substring(2, 4);
const terms = Array.from({ length: 255 }, () => word());
const documents = Array.from({ length: 1000000 }, () => {
const sb = [];
for (let i = 0; i < 2; i++) {
sb.push(word());
}
return sb.join(' ');
});
const index = ri(documents);
const st = performance.now();
const query = 'bb iz';
const results = search(index, query);
const et = performance.now();
console.log(query, Array.from(results).slice(0, 10).map((i) => documents[i]));
console.log(et - st);
There are some improvements you can make if you want. Like... ranking! The whole purpose of this example is to show how we can cut down 1M results to maybe a hundred or so canidates. The search function has some post filtering via intersection which probably isn't what you want you want but at this point it doesn't really matter what you do since the results are so small.

Updating components based on sorted array. Angular (4.3)

Working with an array of data that we want to be able to sort for display in a component, and it doesn't seem to be sorting or updating the DOM, however I have a working code sample that properly demonstrates the concept, and it should be sorting, but in the angular app, it's simply not getting sorted.
The parent component that houses the original data stores the data on an Input parameter object called Batch, and the array we're sorting is on Batch.Invoices.Results. The event from the child component is fine, and the appropriate data is confirmed to bubble to the parent component.
The function that's supposed to sort the array looks like this:
public OnInvoiceSortChange({orderValue, orderAscending}){
console.log(`Invoice Sorting has been called. Value: ${orderValue} . Ascending? ${orderAscending}`);
console.log(`Before:`);
console.log(this.BatchViewModel.Invoices.Results.map(x => x.VendorName));
const sortingArray = [...this.BatchViewModel.Invoices.Results];
if(orderAscending){
const sorted = sortingArray.sort((a, b) => a[orderValue] > b[orderValue] ? 1 : 0);
this.BatchViewModel.Invoices.Results = sorted;
console.log('Sorted');
console.log(sorted.map(x => x.VendorName));
} else {
const sorted = sortingArray.sort((a, b) => a[orderValue] < b[orderValue] ? 1 : 0);
this.BatchViewModel.Invoices.Results = sorted;
console.log(sorted.map(x => x.VendorName));
}
console.log(`After:`);
console.log(this.BatchViewModel.Invoices.Results.map(x => x.VendorName));
}
All the console logs are for debugger visibility, and the output is this:
Where in my testing file (non-angular) looks like this:(where data is a direct copy of the array from the Angular app.
const ascendingData = [...data];
const descendingData = [...data];
const sortedDescending = descendingData.sort((a, b) => a['VendorName'] < b['VendorName']? 0 : 1)
const sortedAscending = ascendingData.sort((a, b) => a['VendorName'] > b['VendorName']? 0 : 1);
const vendorListAscending = sortedAscending.map(x => x.VendorName);
const vendorListDescending = sortedDescending.map(x => x.VendorName);
console.log(vendorListDescending);
console.log(vendorListAscending);
and the output looks like this:
So I see that the sorting should work, but it's just not happening in Angular.
How can I get the array sorted, and update the DOM as well?

The function you pass to sort is wrong. It is supposed to return a negative value for "less", a positive value for "greater" or zero for "equal". If orderValue is numeric then it's easiest to just return a[orderValue] - b[orderValue], if not then just change your 0 to -1.
(By the way, name orderKey could be a bit clearer maybe?)
I don't think angular has anything to do here, but I cannot tell now why you get different results. Anyway, your sort function is invalid (it states that a equals b, but at the same time b is greater than a), I hope fixing this function helps.

mongo/mongoid MapReduce on batch inserted documents

Im creating my batch and inserting it to collection using command i specified below
batch = []
time = 1.day.ago
(1..2000).each{ |i| a = {:name => 'invbatch2k'+i.to_s, :user_id => BSON::ObjectId.from_string('533956cd4d616323cf000000'), :out_id => 'out', :created_at => time, :updated_at => time, :random => '0.5' }; batch.push a; }
Invitation.collection.insert batch
As stated above, every single invitation record has user_id fields value set to '533956cd4d616323cf000000'
after inserting my batch with created_at: 1.day.ago i get:
2.1.1 :102 > Invitation.lte(created_at: 1.week.ago).count
=> 48
2.1.1 :103 > Invitation.lte(created_at: Date.today).count
=> 2048
also:
2.1.1 :104 > Invitation.lte(created_at: 1.week.ago).where(user_id: '533956cd4d616323cf000000').count
=> 14
2.1.1 :105 > Invitation.where(user_id: '533956cd4d616323cf000000').count
=> 2014
Also, I've got a map reduce which counts invitations sent by each unique User (both total and sent to unique out_id)
class Invitation
[...]
def self.get_user_invites_count
map = %q{
function() {
var user_id = this.user_id;
emit(user_id, {user_id : this.user_id, out_id: this.out_id, count: 1, countUnique: 1})
}
}
reduce = %q{
function(key, values) {
var result = {
user_id: key,
count: 0,
countUnique : 0
};
var values_arr = [];
values.forEach(function(value) {
values_arr.push(value.out_id);
result.count += 1
});
var unique = values_arr.filter(function(item, i, ar){ return ar.indexOf(item) === i; });
result.countUnique = unique.length;
return result;
}
}
map_reduce(map,reduce).out(inline: true).to_a.map{|d| d['value']} rescue []
end
end
The issue is:
Invitation.lte(created_at: Date.today.end_of_day).get_user_invites_count
returns
[{"user_id"=>BSON::ObjectId('533956cd4d616323cf000000'), "count"=>49.0, "countUnique"=>2.0} ...]
instead of "count" => 2014, "countUnique" => 6.0 while:
Invitation.lte(created_at: 1.week.ago).get_user_invites_count returns:
[{"user_id"=>BSON::ObjectId('533956cd4d616323cf000000'), "count"=>14.0, "countUnique"=>6.0} ...]
Data provided by query, is accurate before inserting the batch.
I cant wrap my head around whats going on here. Am i missing something?

The part that you seemed to have missed in the documentation seem to be the problem here:
MongoDB can invoke the reduce function more than once for the same key. In this case, the previous output from the reduce function for that key will become one of the input values to the next reduce function invocation for that key.
And also later:
the type of the return object must be identical to the type of the value emitted by the map function to ensure that the following operations is true:
So what you see is your reduce function is returning a signature different to the input it receives from the mapper. This is important since the reducer may not get all of the values for a given key in a single pass. Instead it gets some of them, "reduces" the result and that reduced output may be combined with other values for the key ( possibly also reduced ) in a further pass through the reduce function.
As a result of your fields not matching, subsequent reduce passes do not see those values and do not count towards your totals. So you need to align the signatures of the values:
def self.get_user_invites_count
map = %q{
function() {
var user_id = this.user_id;
emit(user_id, {out_id: this.out_id, count: 1, countUnique: 0})
}
}
reduce = %q{
function(key, values) {
var result = {
out_id: null,
count: 0,
countUnique : 0
};
var values_arr = [];
values.forEach(function(value) {
if (value.out_id != null)
values_arr.push(value.out_id);
result.count += value.count;
result.countUnique += value.countUnique;
});
var unique = values_arr.filter(function(item, i, ar){ return ar.indexOf(item) === i; });
result.countUnique += unique.length;
return result;
}
}
map_reduce(map,reduce).out(inline: true).to_a.map{|d| d['value']} rescue []
end
You also do not need user_id in the values emitted or kept as it is already the "key" value for the mapReduce. The remaining alterations consider that both "count" and "countUnique" can contain an exiting value that needs to be considered, where you were simply resetting the value to 0 on each pass.
Then of course if the "input" has already been through a "reduce" pass, then you do not need the "out_id" values to be filtered for "uniqueness" as you already have the count and that is now included. So any null values are not added to the array of things to count, which is also "added" to the total rather than replacing it.
So the reducer does get called several times. For 20 key values the input will likely not be split, which is why your sample with less input works. For pretty much anything more than that, then the "groups" of the same key values will be split up, which is how mapReduce optimizes for large data processing. As the "reduced" output will be sent back to the reducer again, you need to be mindful that you are considering the values you already sent to output in the previous pass.

rx: unfold array to multiple streams

I have a stream holding an array, each element of which has an id. I need to split this into a stream per id, which will complete when the source stream no longer carries the id.
E.g. input stream sequence with these three values
[{a:1}, {b:1}] [{a:2}, {b:2}, {c:1}] [{b:3}, {c:2}]
should return three streams
a -> 1 2 |
b -> 1 2 3
c -> 1 2
Where a has completed on the 3rd value, since its id is gone, and c has been created on the 2nd value, since its id has appeared.
I'm trying groupByUntil, a bit like
var input = foo.share();
var output = input.selectMany(function (s) {
return rx.Observable.fromArray(s);
}).groupByUntil(
function (s) { return s.keys()[0]; },
null,
function (g) { return input.filter(
function (s) { return !findkey(s, g.key); }
); }
)
So, group by the id, and dispose of the group when the input stream no longer has the id. This seems to work, but the two uses of input look odd to me, like there could a weird order dependency when using a single stream to control the input of the groupByUntil, and the disposal of the groups.
Is there a better way?
update
There is, indeed, a weird timing problem here. fromArray by default uses the currentThread scheduler, which will result in events from that array being interleaved with events from input. The dispose conditions on the group are then evaluated at the wrong time (before the groups from the previous input have been processed).
A possible workaround is to do fromArray(.., rx.Scheduler.immediate), which will keep the grouped events in sync with input.

yeah the only alternative I can think of is to manage the state yourself. I don't know that it is better though.
var d = Object.create(null);
var output = input
.flatMap(function (s) {
// end completed groups
Object
.keys(d)
.filter(function (k) { return !findKey(s, k); })
.forEach(function (k) {
d[k].onNext(1);
d[k].onCompleted();
delete d[k];
});
return Rx.Observable.fromArray(s);
})
.groupByUntil(
function (s) { return s.keys()[0]; },
null,
function (g) { return d[g.key] = new Rx.AsyncSubject(); });

Sort an array by a preferred order

I'd like to come up with a good way to have a "suggested" order for how to sort an array in javascript.
So say my first array looks something like this:
['bob','david','steve','darrel','jim']
Now all I care about, is that the sorted results starts out in this order:
['jim','steve','david']
After that, I Want the remaining values to be presented in their original order.
So I would expect the result to be:
['jim','steve','david','bob','darrel']
I have an API that I am communicating with, and I want to present the results important to me in the list at the top. After that, I'd prefer they are just returned in their original order.
If this can be easily accomplished with a javascript framework like jQuery, I'd like to hear about that too. Thanks!
Edit for clarity:
I'd like to assume that the values provided in the array that I want to sort are not guaranteed.
So in the original example, if the provided was:
['bob','steve','darrel','jim']
And I wanted to sort it by:
['jim','steve','david']
Since 'david' isn't in the provided array, I'd like the result to exclude it.
Edit2 for more clarity:
A practical example of what I'm trying to accomplish:
The API will return something looking like:
['Load Average','Memory Usage','Disk Space']
I'd like to present the user with the most important results first, but each of these fields may not always be returned. So I'd like the most important (as determined by the user in some other code), to be displayed first if they are available.

Something like this should work:
var presetOrder = ['jim','steve','david']; // needn't be hardcoded
function sortSpecial(arr) {
var result = [],
i, j;
for (i = 0; i < presetOrder.length; i++)
while (-1 != (j = $.inArray(presetOrder[i], arr)))
result.push(arr.splice(j, 1)[0]);
return result.concat(arr);
}
var sorted = sortSpecial( ['bob','david','steve','darrel','jim'] );
I've allowed for the "special" values appearing more than once in the array being processed, and assumed that duplicates should be kept as long as they're shuffled up to the front in the order defined in presetOrder.
Note: I've used jQuery's $.inArray() rather than Array.indexOf() only because that latter isn't supported by IE until IE9 and you've tagged your question with "jQuery". You could of course use .indexOf() if you don't care about old IE, or if you use a shim.

var important_results = {
// object keys are the important results, values is their order
jim: 1,
steve: 2,
david: 3
};
// results is the orig array from the api
results.sort(function(a,b) {
// If compareFunction(a, b) is less than 0, sort a to a lower index than b.
// See https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/Array/sort
var important_a = important_results[a],
important_b = important_results[b],
ret;
if (important_a && !important_b) {ret = -1}
else if (important_b && !important_a) {ret = 1}
else if (important_a && important_b) {ret = important_a - important_b}
else {ret = 0}; // keep original order if neither a or b is important
return(ret);
}
)
Use a sorting function that treats the previously known important results specially--sorts them to the head of the results if present in results.
items in important_results don't have to be in the results

Here's a simple test page:
<html>
<head>
<script language="javascript">
function test()
{
var items = ['bob', 'david', 'steve', 'darrel', 'jim'];
items.sort(function(a,b)
{
var map = {'jim':-3,'steve':-2,'david':-1};
return map[a] - map[b];
});
alert(items.join(','));
}
</script>
</head>
<body>
<button onclick="javascript:test()">Click Me</button>
</body>
</html>
It works in most browsers because javascript typically uses what is called a stable sort algorithm, the defining feature of which is that it preserves the original order of equivalent items. However, I know there have been exceptions. You guarantee stability by using the array index of each remaining item as it's a1/b1 value.

http://tinysort.sjeiti.com/
I think this might help. The $('#yrDiv').tsort({place:'start'}); will add your important list in the start.
You can also sort using this function the way you like.

Live demo ( jsfiddle seems to be down)
http://jsbin.com/eteniz/edit#javascript,html
var priorities=['jim','steve','david'];
var liveData=['bob','david','steve','darrel','jim'];
var output=[],temp=[];
for ( i=0; i<liveData.length; i++){
if( $.inArray( liveData[i], priorities) ==-1){
output.push( liveData[i]);
}else{
temp.push( liveData[i]);
}
}
var temp2=$.grep( priorities, function(name,i){
return $.inArray( name, temp) >-1;
});
output=$.merge( temp2, output);

there can be another way of sorting on order base, also values can be objects to work with
const inputs = ["bob", "david", "steve", "darrel", "jim"].map((val) => ({
val,
}));
const order = ["jim", "steve", "david"];
const vMap = new Map(inputs.map((v) => [v.val, v]));
const sorted = [];
order.forEach((o) => {
if (vMap.has(o)) {
sorted.push(vMap.get(o));
vMap.delete(o);
}
});
const result = sorted.concat(Array.from(vMap.values()));
const plainResult = result.map(({ val }) => val);

Have you considered using Underscore.js? It contains several utilities for manipulating lists like this.
In your case, you could:
Filter the results you want using filter() and store them in a collection.
var priorities = _.filter(['bob','david','steve','darrel','jim'],
function(pName){
if (pName == 'jim' || pName == 'steve' || pName == 'david') return true;
});
Get a copy of the other results using without()
var leftovers = _.without(['bob','david','steve','darrel','jim'], 'jim', 'steve', 'david');
Union the arrays from the previous steps using union()
var finalList = _.union(priorities, leftovers);