My website has a JSON file containing data that is intended to be transferred into my users' local IndexedDB. I'm looking for a way to only load this JSON file when an update is actually needed.
To clarify, I plan for my website to run almost entirely off of my users' locally stored data whenever possible, similar to an app. They should only have to download the JSON file when a new IDB update is available.
So far, I've tried accomplishing this by running the onUpgradeNeeded event as an async function.
if (!window.indexedDB) {
window.alert("Your browser doesn't support a stable version of IndexedDB, which is required for most functions of this website. For the best support, please use the latest version of Chrome, Firefox, or Safari.");
}
else {
var dbVer = 39; //IDB Version (int only)
var recipeObject; //instantiate global variable for module object import
var recipeArray = []; //instantiate global array for module import
var recipeDBver; //instantiate global variable for actual database version (TODO: implement version checking)
var upgradeNeeded = false;
var clearNeeded = false;
var openRequest = indexedDB.open('recipeDatabase', dbVer);
console.log("IDB.js running");
openRequest.onsuccess = function(e) {
console.log('Running onSuccess');
};
openRequest.onerror = function(e) {
console.log('Open Request ERROR');
console.dir(e);
};
openRequest.onupgradeneeded = async function(e) {
var db = e.target.result;
console.log('Running onUpgradeNeeded');
db.onerror = function(errorEvent) {
console.log("onUpgradeNeeded ERROR");
return;
};
importObject = await import("/resources/recipeDB.js");
//TODO: remove debugging
console.log('Module loaded');
console.log(importObject);
recipeObject = importObject.default;
console.log(recipeObject);
recipeDBver = recipeObject.recipeDBver;
console.log(recipeDBver);
recipeArray = recipeObject.recipeArray;
console.log(recipeArray);
upgradeNeeded = true;
if (!db.objectStoreNames.contains('drinks')) {
var storeOS = db.createObjectStore('drinks', {keyPath: 'id'});
storeOS.createIndex('name', 'name', { unique: false });
storeOS.createIndex('type', 'type', { unique: false });
storeOS.createIndex('subtype', 'subtype', { unique: false });
storeOS.createIndex('tags', 'tags', { unique: false });
}
else {
clearNeeded = true;
}
console.log('IDB Upgrade Needed: ' + upgradeNeeded);
console.log('IDB Clear Needed: ' + clearNeeded);
db = e.target.result;
if (clearNeeded == true) {
clearData();
}
else if (upgradeNeeded == true) {
for (var i = 0; i < recipeArray.length; i++) {
addItem(recipeArray[i]);
}
}
};
function clearData() {
var db = openRequest.result;
var transaction = db.transaction(["drinks"], "readwrite");
var objectStore = transaction.objectStore("drinks");
var objectStoreRequest = objectStore.clear();
objectStoreRequest.onerror = function(e) {
console.log('Error clearing data. ', e.target.error.name);
console.dir(e);
};
objectStoreRequest.onsuccess = function(e) {
console.log('Data cleared successfully.')
for (var i = 0; i < recipeArray.length; i++) {
addItem(recipeArray[i]);
}
};
}
function addItem(curItem) {
var db = openRequest.result;
var transaction = db.transaction(['drinks'], 'readwrite');
var store = transaction.objectStore('drinks');
var item = curItem;
var request = store.add(item);
request.onerror = function(e) {
console.log('Error', e.target.error.name);
console.dir(e);
};
request.onsuccess = function(e) {
console.log('Item added: ' + curItem.name);
};
}
}
The console returns the following:
I'm assuming the onUpgradeNeeded event is timing out before the JSON file is able to load.
Is there a way to delay the timeout? If not, does anyone know of a better way to accomplish what I'm trying to do?
The problem can be seen from what you see in the console. First we get IDB.js is running and then you go into your onupgradeneeded handler, but then, instead of having anything from that function console logged, we immediately see the onsuccess handler run. The cause of this is because you defined your onupgradeneeded handler to be async, which means that this function essentially stops execution at the await import("/resources/recipeDB.js"); line while it waits for the import to resolve. This essentially means as far as the IDB events are concerned that onupgradeneeded is done and it needs to go into the onsuccess. As Josh says above this is because onupgradeneeded needs to resolve synchronously.
What you can do to get around this, is:
Make onupgradeneeded synchronous
Update your IDB schema (create your new object store and indices)
Import the data and upon successful import insert them into your database
This is one of the difficult things about using IndexedDB: it's not promised-based so using promised-based async functions doesn't always play well with it. I usually find these short-comings require more code to handle them (such as using .then() calls so that I can have synchronous event handlers while still doing necessary asynchronous activities).
The onupgradeneeded event handler needs to complete synchronously. More accurately, requests upon/within the version change transaction that is running need to be started in the same tick of the event loop as when the version change transaction is started.
It is not clear from your question but it looks like you are making an async call to load the json and waiting for it to load, and this wait that happens is what allows the versionchange transaction to complete, and causes all requests made afterward to not occur in the same tick of the event loop.
Step 1：Use javascript package json data.
Step 2：Use importScript() in upgrade event.
Step 3：Run your indexedDB script with Worker.
Related
I used this post: IndexedDB: upgrade with promises?
And implemented the part here: https://stackoverflow.com/a/25565755/15778635
This works for what I need. the part I am having trouble with is this:
var newMigrationPromise = function (dbName, version, migration) {
return newPromise(function (deferred) {
var request = indexedDB.open(dbName, version);
// NB: caller must ensure upgrade callback always called
request.onupgradeneeded = function (event) {
var db = request.result;
newTransactionPromise(
function () {
var syncUPStore = transaction.objectStore("syncUP");
var syncCountRequest = syncUPStore.count();
syncCountRequest.oncomplete = function (event) {
if (syncCountRequest.result > 0)
deferred.reject(syncCountRequest.result + " SyncUp Records exist, database upgrade aborted, keeping at current version.");
else {
//Good, continue with update
migration(db, request.transaction);
return request.transaction;
}
}
})
.then(function () { db.close(); })
.then(deferred.resolve, deferred.reject);
};
request.onerror = function (ev) { deferred.reject(request.error); };
});
};
I have a syncUP object store that has data that needs to be sent to the server when the user goes online. In this particular case the service worker is installing (because they came online and a change was put on the server) and needs to know if syncUP records exist prior to allowing the service worker to update. If they do exist then it needs to abort the install until it is empty.
The service worker abort works fine, and the database aborting upgrade works fine if I were to throw an error where var syncCountRequest = syncUPStore.count(); is.
My question:
How can I check if there are records in the "syncUP" object store and still use the implementation I mentioned above? I had considered moving the logic to another method, but I found I was having the same issue of not knowing how to handle the reject/resolve. My Promises knowledge is ok, but not good enough yet to figure it out on my own.
a rushed example:
var request = indexedDb.open(...);
request.onupgradeneeded = function(event) {
if(conditionShouldDoMigrationFromVersionXToNowIsTrue) {
migrate(event.transaction);
}
};
function migrate(versionChangeTransaction) {
var store = versionChangeTransaction.objectStore('x');
var request = store.getAll();
request.onsuccess = function(event) {
var objects = event.target.result;
for (var object of objects) {
// do some mutation to the object
object.x++;
// write it back
store.put(object);
}
};
}
My problem is that the code does not seem to be running in order, as seen below.
This code is for my discord.js bot that I am creating.
var Discord = require("discord.js");
var bot = new Discord.Client();
var yt = require("C:/Users/username/Documents/Coding/Discord/youtubetest.js");
var youtubetest = new yt();
var fs = require('fs');
var youtubedl = require('youtube-dl');
var prefix = "!";
var vidid;
var commands = {
play: {
name: "!play ",
fnc: "Gets a Youtube video matching given tags.",
process: function(msg, query) {
youtubetest.respond(query, msg);
var vidid = youtubetest.vidid;
console.log(typeof(vidid) + " + " + vidid);
console.log("3");
}
}
};
bot.on('ready', () => {
console.log('I am ready!');
});
bot.on("message", msg => {
if(!msg.content.startsWith(prefix) || msg.author.bot || (msg.author.id === bot.user.id)) return;
var cmdraw = msg.content.split(" ")[0].substring(1).toLowerCase();
var query = msg.content.split("!")[1];
var cmd = commands[cmdraw];
if (cmd) {
var res = cmd.process(msg, query, bot);
if (res) {
msg.channel.sendMessage(res);
}
} else {
let msgs = [];
msgs.push(msg.content + " is not a valid command.");
msgs.push(" ");
msgs.push("Available commands:");
msgs.push(" ");
msg.channel.sendMessage(msgs);
msg.channel.sendMessage(commands.help.process(msg));
}
});
bot.on('error', e => { console.error(e); });
bot.login("mytoken");
The youtubetest.js file:
var youtube_node = require('youtube-node');
var ConfigFile = require("C:/Users/username/Documents/Coding/Discord/json_config.json");
var mybot = require("C:/Users/username/Documents/Coding/Discord/mybot.js");
function myyt () {
this.youtube = new youtube_node();
this.youtube.setKey(ConfigFile.youtube_api_key);
this.vidid = "";
}
myyt.prototype.respond = function(query, msg) {
this.youtube.search(query, 1, function(error, result) {
if (error) {
msg.channel.sendMessage("There was an error finding requested video.");
} else {
vidid = 'http://www.youtube.com/watch?v=' + result.items[0].id.videoId;
myyt.vidid = vidid;
console.log("1");
}
});
console.log("2");
};
module.exports = myyt;
As the code shows, i have an object for the commands that the bot will be able to process, and I have a function to run said commands when a message is received.
Throughout the code you can see that I have put three console.logs with 1, 2 and 3 showing in which order I expect the parts of the code to run. When the code is run and a query is found the output is this:
I am ready!
string +
2
3
1
This shows that the code is running in the wrong order that I expect it to.
All help is very highly appreciated :)
*Update! Thank you all very much to understand why it isn't working. I found a solution where in the main file at vidid = youtubetest.respond(query, msg) when it does that the variable is not assigned until the function is done so it goes onto the rest of my code without the variable. To fix I simply put an if statement checking if the variable if undefined and waiting until it is defined.*
Like is mentioned before, a lot of stuff in javascript runs in async, hence the callback handlers. The reason it runs in async, is to avoid the rest of your code being "blocked" by remote calls. To avoid ending up in callback hell, most of us Javascript developers are moving more and more over to Promises. So your code could then look more like this:
myyt.prototype.respond = function(query, msg) {
return new Promise(function(resolve, reject) {
this.youtube.search(query, 1, function(error, result) {
if (error) {
reject("There was an error finding requested video."); // passed down to the ".catch" statement below
} else {
vidid = 'http://www.youtube.com/watch?v=' + result.items[0].id.videoId;
myyt.vidid = vidid;
console.log("1");
resolve(2); // Resolve marks the promises as successfully completed, and passes along to the ".then" method
}
});
}).then(function(two) {
// video is now the same as myyt.vidid as above.
console.log(two);
}).catch(function(err) {
// err contains the error object from above
msg.channel.sendMessage(err);
})
};
This would naturally require a change in anything that uses this process, but creating your own prototypes seems.. odd.
This promise returns the vidid, so you'd then set vidid = youtubetest.response(query, msg);, and whenever that function gets called, you do:
vidid.then(function(id) {
// id is now the vidid.
});
Javascript runs async by design, and trying to hack your way around that leads you to dark places fast. As far as I can tell, you're also targetting nodeJS, which means that once you start running something synchronously, you'll kill off performance for other users, as everyone has to wait for that sync call to finish.
Some suggested reading:
http://callbackhell.com/
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
https://stackoverflow.com/a/11233849/3646975
I'd also suggest looking up ES6 syntax, as it shortens your code and makes life a hellofalot easier (native promises were only introduced in ES6, which NodeJS 4 and above supports (more or less))
In javascript, please remember that any callback function you pass to some other function is called asynchronously. I.e. the calls to callback function may not happen "in order". "In order" in this case means the order they appear on the source file.
The callback function is simply called on certain event:
When there is data to be processed
on error
in your case for example when the youtube search results are ready,
'ready' event is received or 'message' is received.
etc.
Lets say that we have two buttons, each on are calling the following method:
var NUMBER_OF_IMAGE_REQUEST_RETRIES = 3;
var IMAGE_REQUEST_TIMEOUT = 3000;
processImage: function(image_data) {
var main_response = $q.defer();
var hash = getImageHash(image_data);
var requestsCounter = -1;
requestImage = function() {
$http.post(apiUrl, {params: {data: hash},timeout: IMAGE_REQUEST_TIMEOUT})
.then(function(response) {
return main_response.resolve(response.data);
}, function(error) {
if (++requestsCounter < NUMBER_OF_IMAGE_REQUEST_RETRIES) {
requestLabelsImage();
} else {
return main_response.reject();
}
});
};
requestLabelsImage();
return main_response.promise;
}
The method passes an image related data to the server, the server process the data and then response. Every time a user press a different button different image_data is being send to the server.
The problem:
The user press button 1, the method is called with image_data_1, and then he/she immediately press button 2 and the method is called with image_data_2. The processImage function is called by another method, lets say doSomethingWithTheResponse which only cares about the latest user's action, but the image_data_2 is proceed faster by the servers, so the client gets image_data_2 before image_data_1, so the client believes that image_data_1 was related to the user's latest action, which is not the case. How can we ensure that the client is always getting the response that is related to the users latest action?
Note: The hash is different for the differente image_data requests.
I was thinking something like:
var oldhash = null;
processImage: function(image_data) {
var main_response = $q.defer();
var hash = getImageHash(image_data);
oldhash = hash;
var requestsCounter = -1;
requestImage = function(hash) {
if(hash === oldhash){
$http.post(apiUrl, {params: {data: hash},timeout: IMAGE_REQUEST_TIMEOUT})
.then(function(response) {
return main_response.resolve(response.data);
}, function(error) {
if (++requestsCounter < NUMBER_OF_IMAGE_REQUEST_RETRIES) {
requestLabelsImage(hash);
} else {
return main_response.reject();
}
});
}
else {
main_response.reject();
}
}
requestLabelsImage(hash);
return main_response.promise;
}
But I am not 100% sure that this is the right approach.
Simply disregard the previous requests.
You can create a repository of requests (array or dictionary implementation is okay). Call .abort() on the previous ones once another request is made -- when you add it in your storage.
If you want a dictionary, there is a good example here (tackles a different topic, though), but here is a modified snippet of his code which is related to your case:
var _pendingRequests = {};
function abortPendingRequests(key) {
if (_pendingRequests[key]) {
_pendingRequests[key].abort();
}
}
Where the key can be.. say... a category of your action. You can name constants for it, or it can be just the name of the button pressed. It can even be a URL of your request; completely up to you.
There is an excellent explanation of the whole concept here:
jquery abort() ajax request before sending another
https://stackoverflow.com/a/3313022/594992
If your UI allows for initiation multiple actions, while processing of those actions are mutually exclusive, then you should probably use promises, and track active promises.
button1.addEventListener("click", function(evt) {
startRunning( task1.start() );
});
button2.addEventListener("click", function(evt) {
startRunning( task2.start() );
});
With a task runner like:
function startRunning( promise ) {
while(runningTasks.length>0) {
cancel( runningTasks.unshift() );
});
runningTasks.push( promise );
}
Your cancel function can come from anything that can deal with promises, like Angular's service.cancelRequest, or you can write your own code that takes the promise and smartly breaks off its operation.
Of course, if you're not using promises, then you probably want to start doing so, but if you absolutely can't you can use a manager object like:
button1.addEventListener("click", function(evt) { task1(); });
button2.addEventListener("click", function(evt) { task2(); });
with
var manager = [];
function cancelAll() {
while(manager.lenght>0) {
var cancelfn = manager.unshift()
cancelfn();
}
return true;
}
function task1() {
var running = cancelAll();
manager.push(function() { running = false; });
asyncDo(something1, function(result) {
if(!running) return;
// do your real thing
});
}
function task1() {
var running = cancelAll();
manager.push(function() { running = false; });
asyncDo(something2, function(result) {
if(!running) return;
// do your real thing
});
}
And you can put cancels on as many aspects as you need. If you need to cancel running XHRs, you might be able to do so, if you have multiple steps in your result handling, cut off at each step start, etc.
This sounds like an ideal use-case for promises. Basically, whenever a new request is made, you want to cancel any existing promises. I am not versed in AngularJS, but the following ng-specific links might prove useful:
Angularjs how to cancel resource promise when switching routes
Canceling A Promise In AngularJS
I'm new to node and am having trouble understanding node's async behavior. I know this is a very frequently addressed question on SO, but I simply can't understand how to get any of the solutions I've read to work in my context.
I'm writing this module which I want to return an object containing various data.
var myModule = (function () {
var file,
fileArray,
items = [],
getBlock = function (fileArray) {
//get the data from the file that I want, return object
return block;
},
parseBlock = function (block) {
//[find various items in the block, put them into an "myItems" object, then
//take the items and do a look up against a web api as below]...
for (var i = 0, l = myItems.length; i < l; i ++) {
(function (i) {
needle.post(MY_URL, qstring, function(err, resp, body){
if (!err && resp.statusCode === 200){
myItems[i].info = body;
if (i === (myItems.length -1)) {
return myItems;
}
}
});
})(i);
}
},
getSomeOtherData = function (fileArray) {
//parse some other data from the file
}
return {
setFile: function (file) {
fileArray = fs.readFileSync(file).toString().split('\n');
},
render: function () {
var results = [];
results.someOtherData = getsomeOtherData();
var d = getBlock();
results.items = parseBlock(d);
return results;
}
}
})();
When I call this module using:
myModule.setFile('myFile.txt');
var res = myModule.render();
the variable res has the values from the someOtherData property, but not the items property. I understand that my long-running http request has not completed and that node just zooms ahead and finishes executing, but that's not what I want. I looked at a bunch of SO questions on this, and looked at using Q or queue-async, but with no success.
How do I get this module to return no data until all requests have completed? Or is that even possible in node? Is there a better way to design this to achieve my goal?
The problem in your example is your calling getBlock() but you have declared your function as getBlockData(). So you will not get a result. Try changing it to both the same.
Presuming that you have them both the same, your next problem is that your processing data from a file, so I presume that your reading the contents of the file and then parsing it.
If this is the case then there are sync reads that you can use to force sync, however I wouldn't recommend this.
You really want to structure your program based on events. Your thinking in the paradigm of 'call a function, when it returns continue'. You need to be thinking more along the lines of 'call a process and add a listener, the listener then does reply handling'.
This works very well for comms. You receive a request. You need to reply based on contents of file. So you start the read process with two possible results. It calls the completed function or the error function. Both would then call the reply function to process how to handle a reply for the request.
It's important not to block as you will be blocking the thread via which all processes are handled.
Hope that helps, if not add some comments and I will try and elaborate.
Have a look at this answer to another question to see a good example of processing a file using the standard listeners. All async calls have a listener concept for what can happen. All you need to do is pass a function name (or anon if you prefer) to them when you call them.
A quick example (based on node.js stream.Readable API:
fs.createReadStream(filename, {
'flags': 'r'
}).addListener( "data", function(chunk) {
// do your processing logic
}).addListener( "end", function(chunk) {
// do your end logic
response(...);
}).addListener( "error", function(chunk) {
// do your error logic
response(...);
}).addListener( "close",function() {
// do your close logic
});
function response(info) {
}
i have a recursive query like this (note: this is just an example):
var user = function(data)
{
this.minions = [];
this.loadMinions = function()
{
_user = this;
database.query('select * from users where owner='+data.id,function(err,result,fields)
{
for(var m in result)
{
_user.minions[result[m].id] = new user(result[m]);
_user.minions[result[m].id].loadMinions();
}
}
console.log("loaded all minions");
}
}
currentUser = new user(ID);
for (var m in currentUser.minions)
{
console.log("minion found!");
}
this don't work because the timmings are all wrong, the code don't wait for the query.
i've tried to do this:
var MyQuery = function(QueryString){
var Data;
var Done = false;
database.query(QueryString, function(err, result, fields) {
Data = result;
Done = true;
});
while(Done != true){};
return Data;
}
var user = function(data)
{
this.minions = [];
this.loadMinions = function()
{
_user = this;
result= MyQuery('select * from users where owner='+data.id);
for(var m in result)
{
_user.minions[result[m].id] = new user(result[m]);
_user.minions[result[m].id].loadMinions();
}
console.log("loaded all minions");
}
}
currentUser = new user(ID);
for (var m in currentUser.minions)
{
console.log("minion found!");
}
but he just freezes on the while, am i missing something?
The first hurdle to solving your problem is understanding that I/O in Node.js is asynchronous. Once you know how this applies to your problem the recursive part will be much easier (especially if you use a flow control library like Async or Step).
Here is an example that does some of what you're trying to do (minus the recursion). Personally, I would avoid recursively loading a possibly unknown number/depth of records like that; Instead load them on demand, like in this example:
var User = function(data) {
this.data = data
this.minions;
};
User.prototype.getMinions = function(primaryCallback) {
var that = this; // scope handle
if(this.minions) { // bypass the db query if results cached
return primaryCallback(null, this.minions);
}
// Callback invoked by database.query when it has the records
var aCallback = function(error, results, fields) {
if(error) {
return primaryCallback(error);
}
// This is where you would put your recursive minion initialization
// The problem you are going to have is callback counting, using a library
// like async or step would make this party much much easier
that.minions = results; // bypass the db query after this
primaryCallback(null, results);
}
database.query('SELECT * FROM users WHERE owner = ' + data.id, aCallback);
};
var user = new User(someData);
user.getMinions(function(error, minions) {
if(error) {
throw error;
}
// Inside the function invoked by primaryCallback(...)
minions.forEach(function(minion) {
console.log('found this minion:', minion);
});
});
The biggest thing to note in this example are the callbacks. The database.query(...) is asynchronous and you don't want to tie up the event loop waiting for it to finish. This is solved by providing a callback, aCallback, to the query, which is executed when the results are ready. Once that callback fires and after you perform whatever processing you want to do on the records you can fire the primaryCallback with the final results.
Each Node.js process is single-threaded, so the line
while(Done != true){};
takes over the thread, and the callback that would have set Done to true never gets run because the thead is blocked on an infinite loop.
You need to refactor your program so that code that depends on the results of the query is included within the callback itself. For example, make MyQuery take a callback argument:
MyQuery = function(QueryString, callback){
Then call the callback at the end of your database.query callback -- or even supply it as the database.query callback.
The freezing is unfortunately correct behaviour, as Node is single-threaded.
You need a scheduler package to fix this. Personally, I have been using Fibers-promise for this kind of issue. You might want to look at this or another promise library or at async