I have store with multiEntry index on tags.
{ tags: [ 'tag1', 'tag2', 'tag3' ] }
And i have query that also list of tags.
[ 'tag2', 'tag1', 'tag4' ]
I need to get all records which contain one of tag in query (Similar to SQL OR statement).
Currently I cannot find any other solution except iterate over tags in query and search by the each tag in the store.
Is there any better solution?
Thank you.
You cannot retrieve all results with one query except with iteration. You can optimize the search result by opening a index from the lowest value to the highest:
IDBKeyRange.bound ('tag1', 'tag4');
Other Indexed-Db feature you can use is to open multiple queries and combine the result when the queries complete. This way would be much faster than the iteration.
IndexedDB has only range query as Deni Mf answered.
OR query is simply union of multiple queries. That may be OK.
If you want efficient query, you have to iterate the cursor and seek the cursor position as necessary. Using my library, it will be
tags = ['tag2', 'tag1', 'tag4'];
tags.sort();
iter = new ydn.db.KeyIterator('store name', 'tags', IDBKeyRange.bound(tags[0], tags[tags.length-1]);
keys = [];
i = 0;
var req = db.open(iter, function(cursor) {
if (tags.indexOf(cursor.indexKey()) >= 0) {
// we got the result
if (keys.indexOf(cursor.key()) == -1) { // remove duplicate
keys.push(cursor.key());
}
} else {
return tags[++i]; // jump to next index position.
}
);
req.done(function() {
db.list('store name', keys).done(function(results) {
console.log(results);
}
});
Notice that the algorithm has no false positive retrieval. Key only query is performed first, so that we don't waste on de-serilization. The results is retrieved only we get all the primary keys after removing deplicates.
Related
I am looping through a collection of blog posts to firstly push the username and ID of the blog author to a new array of arrays, and then secondly, count the number of blogs from each author. The code below achieves this; however, in the new array, the username and author ID are no longer separate items in the array, but seem to be concatenated into a single string. I need to retain them as separate items as I need to use both separately; how can I amend the result to achieve this?
var countAuthors = [];
blogAuthors = await Blog.find().populate('authors');
blogAuthors.forEach(function(blogAuthor){
countAuthors.push([blogAuthor.author.username, blogAuthor.author.id]);
})
console.log(countAuthors);
// Outputs as separate array items, as expected:
// [ 'author1', 5d7eed028c298b424b3fb5f1 ],
// [ 'author2', 5dd8aa254d74b30017dbfdd3 ],
var result = {};
countAuthors.forEach(function(x) {
result[x] = (result[x] || 0) + 1;
});
console.log(result);
// Username and author ID become a single string and cannot be accessed as separate array items
// 'author1,5d7eed028c298b424b3fb5f1': 15,
// 'author2,5dd8aa254d74b30017dbfdd3': 2,
Update:
Maybe I can explain a bit further WHY on what to do this. What I am aiming for is a table which displays the blog author's name alongside the number of blogs they have written. However, I also want the author name to link to their profile page, which requires the blogAuthor.author.id to do so. Hence, I need to still be able to access the author username and ID separately after executing the count. Thanks
You could use String.split().
For example:
let result = 'author1,5d7eed028c298b424b3fb5f1'.split(',')
would set result to:
['author1' , '5d7eed028c298b424b3fb5f1']
You can then access them individually like:
result[1] //'5d7eed028c298b424b3fb5f1'
Your issue is that you weren't splitting the x up in the foreach callback, and so the whole array was being converted to a string and being used as the key when inserting into the results object.
You can use array destructuring to split the author name and blog id, and use them to optionally adding a new entry to the result object, and then update that result.
countAuthors = [
['author1', 'bookId1'],
['author2', 'bookId2'],
['author1', 'bookId3'],
['author1', 'bookId4'],
['author2', 'bookId5']
]
var result = {};
countAuthors.forEach(([author, id]) => {
if (result[author] === undefined) {
result[author] = {count: 0, blogIds: []};
}
result[author].count += 1;
result[author].blogIds.push(id);
});
console.log(result);
Consider the following object store, with the domain key set as the keyPath
var tags = [
//codes: 0 - markdown wrap tag
// 1 - HTML wrap tag
// 2 - single tag
{ domain: "youtube",
bold:["*",0],
strikethrough:["-",0],
italic:["_",0]
},
{ domain: "stackoverflow",
bold:["<strong>",1],
italic:["<em>",1],
strikethrough:["<del>",1],
superscript:["<sup>",1],
subscript:["<sub>",1],
heading1:["<h1>",1],
heading2:["<h2>",1],
heading3:["<h3>",1],
blockquote:["<blockquote>",1],
code:["<code>",1],
newline:["<br>",2],
horizontal:["<hr>",2]
}
];
The above code works fine and lets me do look-ups easily and efficiently. However, there are many cases where two objects in the store are completely identical except for their domain attribute.
For example, I want to add objects for all of the Stack Exchange sites to the store, and all of those objects would be equal to the one for StackOverflow.
So, rather than create many separate objects, I want to do something like this:
var tags = [
//codes: 0 - markdown wrap tag
// 1 - HTML wrap tag
// 2 - single tag
{ domain: ["youtube"],
bold:["*",0],
strikethrough:["-",0],
italic:["_",0]
},
{ domain: ["stackoverflow","stackexchange",...],
bold:["<strong>",1],
italic:["<em>",1],
strikethrough:["<del>",1],
superscript:["<sup>",1],
subscript:["<sub>",1],
heading1:["<h1>",1],
heading2:["<h2>",1],
heading3:["<h3>",1],
blockquote:["<blockquote>",1],
code:["<code>",1],
newline:["<br>",2],
horizontal:["<hr>",2]
}
];
Would it be possible to use a KeyGen rather than a keyPath and set up some kind of index that took a value and searched for it in the arrays pointed to by the domain key?
Or would I have to use a cursor each time I want to do a look up?
Some potentially helpful references are:
https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Basic_Concepts_Behind_IndexedDB
http://www.w3.org/TR/IndexedDB/#key-path-construct
https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Using_IndexedDB
The solution is to use an index with the multiEntry key property set to true
see this link (thanks #kyaw Tun)
Each index also has a multiEntry flag. This flag affects how the index behaves when the result of evaluating the index's key path yields an Array. If the multiEntry flag is false, then a single record whose key is an Array is added to the index. If the multiEntry flag is true, then the one record is added to the index for each item in the Array. The key for each record is the value of respective item in the Array.
Armed with this index, a specific keyPath is no longer necessary, so you can just use a keyGen for simplicity.
So, to create the database:
request.onupgradeneeded = function(event)
{
var db = event.target.result;
var objectStore = db.createObjectStore("domains", {autoIncrement: true });
objectStore.createIndex("domain", "domain", { unique: true, multiEntry: true });
for(var i in tags)
{
objectStore.add(tags[i]);
console.log("added " + tags[i]["domain"] + " to the DB");
}
};
and an example of using a domain to query for an object:
var objectStore = db.transaction("domains").objectStore("domains");
var query = objectStore.index("domain").get(queryURL);
query.onsuccess = function(event){...};
In parse I have a class named "TestScore". Each object has a key named "quizName".
I need to get an array of unique "quizName" values. I wrote the below which queries the "TestClass" and loops through the results looking for unique "quizName" values.
At first seemed to do the job. But then I realized that the maximum number of returned objects is 1000. Soon there will be more than 1000 objects stored which means that this method will not guarantee that I end up will all values.
function loadTests(){
//create an array to hold each unique test name as we find them
var uniqueEntries = [];
//query parse to return TestScore objects
var TestScore = Parse.Object.extend("TestScore");
var query = new Parse.Query(TestScore);
query.limit(1000) //added this after realizing that the default query limit is only 100
query.find({
success: function(testScore) {
$(testScore).each(function(index, score) {
//here I loop though all of the returned objects looking at the "quizName" for each
if($.inArray(score.get("quizName"), uniqueEntries) === -1) {
//if the quiz name is not already in the "uniqueEntries" array, I add it to the array
uniqueEntries.push(score.get("quizName"));
}
});
//do stuff with quiznames here...., add them as options in select boxes mostly
}
});
}
I looked at {Parse.Query} notContainedIn(key, values) which looks promising but cant figure out if I can add values to the array as I find them. It seems like I would have to have an array to start with (defeating the whole point.)
This part of the docs "{Parse.Query} Returns the query, so you can chain this call." makes me think I might be able to chain queries together to get what I need, but that doesn't seem very efficient.
How can I retrieve unique values for key "quizName" when my class has > 1000 objects?
I'm sure you're long past this by now, but only way I know of to do it is to use one query after another by using a .skip(#) value for each query. So get 1000, then query again with .skip(1000), concatenate the items from the first list and second, then query again with .skip(2000), etc...
Be aware that I think there's a limit on skip values of 10,000. Don't take my word on that, just pointing you to something that I think is right that you should confirm if you think it applies to your situation.
I eventually found a tutorial online that I was able to modify and came up with the below. This effectively sets the return limit to 10,000 instead of 1,000 and allows setting several different parameters for the query.
My changes could surely be written better, maybe as an options object or similar but it works for my needs.
You can see a working demo here
function getStuff(){
// here we will setup and call our helper functions with callbacks to handle the results
var scheme =['SOTest',true]; // return all objects with value `true` in the `SOTest` column
// var scheme =['descending','createdAt']; // return all objects with sort order applied
// var scheme =''; // or just return all objects
// see `findChunk()` below for more info
var Remark = Parse.Object.extend("Remark");
schemePromise(Remark, scheme).done(function (all) {
console.log('Found ' + all.length+' Remarks');
$.each( all, function(i, obj){
$('#test').append(obj.get('Remark') +'<br>');
});
})
.fail(function (error) {
console.log("error: " + JSON.stringify(error));
});
}
getStuff(); // call our function
// helper functions used to get around parse's 1000 query limit
// raises the limit to 10,000 by using promises
function findChunk(model, scheme, allData) {
// if `scheme` was an empty string, convert to an array
// this is the default and returns all objects in the called class
if(scheme==''){ ['scheme',''] };
// will return a promise
var limit = 1000;
var skip = allData.length;
var findPromise = $.Deferred();
var query = new Parse.Query(model);
// to get all objects from the queried Class then sort them by some column
// pass `scheme` as an array like [ sort method, column to sort ]
if (scheme[0]=='descending') query.descending(scheme[1]);
else if (scheme[0]=='ascending') query.ascending(scheme[1]);
// to limt results to objects that have a certain value in a specific column
// pass `scheme` as an array like [ column name, value ]
else query.equalTo(scheme[0], scheme[1]);
// more options can easily be built in here using `scheme`
query
.limit(limit)
.skip(skip)
.find()
.then(function (results) {
findPromise.resolve(allData.concat(results), !results.length);
}, function (results) {
findPromise.reject(error);
});
return findPromise.promise();
}
function schemePromise(model, scheme, allResults, allPromise) {
// find a scheme at a time
var promise = allPromise || $.Deferred();
findChunk(model, scheme, allResults || [])
.done(function (results, allOver) {
if (allOver) {
// we are done
promise.resolve(results);
} else {
// may be more
schemePromise(model, scheme, results, promise);
}
})
.fail(function (error) {
promise.reject(error);
});
return promise.promise();
}
Is there a way to retrieve the next unique index in a store based on a predicate on the record. For example if I have a book store full of objects like so:
{name: 'Hello Kitty', author: 'Me', pages: 5}
Would it be possible to return the next unique index on author, but base the uniqueness on the highest number of pages?
index.openKeyCursor('author', IDBCursor.nextunique).onsuccess = function(event) {
var cursor = event.target.result;
if (cursor) {
// How to filter the record by highest number of pages?
cursor.continue();
}
};
This is a bit tricky, but you can do. I will illustrate with my library https://bitbucket.org/ytkyaw/ydn-db but you can use IndexedDB API.
First you have to use compound index (only Firefox and Chrome supported) using array keyPath. Database schema for ydn-db is
var schema = {
stores: [{
name: 'book',
indexes: [{
name: 'author, pages',
keyPath: ['author', 'pages']
}]
}
};
var db = new ydn.db.Storage('db name', schema);
The index, 'author, pages' is sorted by author and then by pages. Then we prepare cursor or create iterator in ydn-db.
var iter = new ydn.db.IndexValueIterator('book', 'author, pages');
By default, order is in ascending. Here we want descending order to get highest pages value. This inadvertently make author to sort in descending order, but there is no way to avoid it.
iter = iter.reverse().unique(); // essentially 'PREV_UNIQUE'
Then, we open the iterator giving rise to cursor with descending ordering. The first cursor is what we want. On next iteration, we skip duplicate author name. This is done by using cursor.continue(next_key) method. next_key is given, such that it won't repeat what already got by giving lowest possible value with known author key.
db.open(function(cursor) {
var book = cursor.getValue();
console.log(book);
var effective_key = cursor.getKey();
var author_key = effective_key[0];
var next_key = [author_key];
return next_key; // continue to this or lower than this key.
}, iter);
Note that, we just need to iterate only unique author and no buffer memory require, and hence scalable.
In my web app , i am using jquery ui autocomplete which data are stored in the database (Mysql).
In this application, i have the administration official data And i enable the users to enter their own data. Data will be displayed in the autocomplete drop down list .
The administration data are limited and never change thus are preloaded and cached in memory, however the user data aren't (since user data grows exponentially and there is no point of using a database if all is to be cached in memory).
So i am doing multi levels caching in my web app, of which, i am caching autocomplete's user data on the client side (instead of getting them from db on each search).
The scenario is as follow:
When the app loads, the admin data (that are preloaded in memory) are loaded and cached in a JS object to be used in the autocomplete mechanism.
Thus when a user searches for a word, it will check the admin data JS object and if it can't find it, it will go to db and find it in user data.
So for the case of searching user's data in db, What i am doing right now is the following:
$( "#search" ).autocomplete({
source:function(request, response) {
var results = $.ui.autocomplete.filter(data, request.term);
if (request.term.toUpperCase() in autocomplete_cache) {
response($.map(autocomplete_cache[request.term.toUpperCase()], function(item) {
return {value: item.members.name, md5: item.members.ID}
}))
return;
}
where autocomplete_cache is a javascript object:
var autocomplete_cache = {};
// ...it gets filled everytime a new $.post gets new data
...autocomplete_cache[request.term.toUpperCase()] = data;
For the moment, it is caching the exact term searched by a user.
For example, if a user writes "ABCD" and which is not already cached :
--> go to the database (MySQL) do :"WHERE term Like %ABCD%"
--> autocomplete_cache["ABCD"] = "data parsed from mysql"
So if the same USER now retypes "ABCD" it will get it from the autocomplete_cache Object. However if the USER type "ABC" it won't get it from the cache object because of the line:
if (request.term.toUpperCase() in autocomplete_cache)
So what i would like to do is to match what the User types in the cache, so in this scenario if the User types "ABC" (which is contained in cache's "ABCD" String) , the autocomplete will build a drop down list containing "ABCD".
Is there a way of doing that? the reason for this requirement instead of finding the exact String index in the cache object , it will find a wider range and thus limit probabilities of needing to go to db to find the new matches.
Instead of looking for a specific key, test each key whether the term is a substring:
var term = request.term.toUpperCase();
for (var key in autocomplete_cache) {
if (key.indexOf(term) > -1) {
// term is contained in key
}
}
And instead of calling response on the first match, collect all matches’ items in an additional array and call response at the end:
var term = request.term.toUpperCase(),
items = [];
for (var key in autocomplete_cache) {
if (key.indexOf(term) > -1) {
items.merge($.map(autocomplete_cache[key], function(item) {
return {value: item.members.name, md5: item.members.ID}
});
}
}
if (items.length > 0) response(items);
You might also want to remove any duplicates in items and sort the resulting items before calling response:
var term = request.term.toUpperCase(),
items = [],
index = {};
for (var key in autocomplete_cache) {
if (key.indexOf(term) > -1) {
$.each(autocomplete_cache[key], function(item) {
var id = item.members.ID;
if (!index[id]) {
items.push({value: item.members.name, md5: item.members.ID});
index[id] = true;
}
});
}
}
items.sort(function(a, b) { return a.name.localeCompare(b.name); });
if (items.length > 0) response(items);