I'm trying to Store some application data using indexedDB
Here is my code
function _getLocalApplicationCache(_, payload) {
const indexedDB = window.indexedDB || window.mozIndexedDB || window.webkitIndexedDB || window.shimIndexedDB;
if (!indexedDB) {
if (__DEV__) {
console.error("IndexedDB could not found in this browser.");
}
}
const request = indexedDB.open("ApplicationCache", 1);
request.onerror = event => {
if (__DEV__) {
console.error("An error occurred with IndexedDB.");
console.error(event);
}
return;
};
request.onupgradeneeded = function () {
const db = request.result;
const store = db.createObjectStore("swimlane", {keyPath: "id", autoIncrement: true});
store.createIndex("keyData", ["name"], {unique: false});
};
request.onsuccess = () => {
// creating the transition
const db = request.result;
const transition = db.transaction("swimlane", "readwrite");
// Reference to our object store that holds the swimlane data;
const store = transition.objectStore("swimlane");
const swimlaneData = store.index("keyData");
payload = JSON.parse(JSON.stringify(payload));
store.put(payload);
const Query = swimlaneData.getAll(["keyData"]);
Query.onsuccess = () => {
if (__DEV__) {
console.log("Application Cache is loaded", Query.result);
}
};
transition.oncomplete = () => {
db.close();
};
};
}
If I do use different version then 1 here --> indexedDB.open("ApplicationCache", 1);
I'm getting a error like they keyPath is already exist. And other than than for version 1 I'm getting this error.
Can someone please help me where i'm doing wrong.
Review the introductory materials on using indexedDB.
If you did something like connect and create a database without a schema, or created an object store without an explicit key path, and then you stored some objects, and then you edited the upgradeneeded callback to specify the keypath, and then never triggered the upgradeneeded callback to run because you continue to use current version number instead of a newer version number, it would be one possible explanation for this error.
The upgradeneeded callback needs to have logic that checks for whether the object stores and indices already exist, and only create them if they do not exist. If the store does not exist, create it and its indices. If the store exists and the indices do not, add indices to the store. If the store exists and the indices exist, do nothing.
You need to trigger the upgradeneeded callback to run after changing your database schema by connecting with a higher version number. If you do not connect with a higher version number, the callback never runs, so you will end up connecting to the older version where your schema changes have not taken place.
Related
My Mongoose schema uses a custom _id value and the code I inherited does something like this
const sampleSchema = new mongoose.Schema({
_id: String,
key: String,
});
sampleSchema.statics.generateId = async function() {
let id;
do {
id = randomStringGenerator.generate({length: 8, charset: 'hex', capitalization: 'uppercase'});
} while (await this.exists({_id: id}));
return id;
};
let SampleModel = mongoose.model('Sample', sampleSchema);
A simple usage looks like this:
let mySample = new SampleModel({_id: await SampleModel.generateId(), key: 'a' });
await mySample.save();
There are at least three problems with this:
Every save will require at least two trips to the database, one to test for a unique id and one to save the document.
For this to work, it is necessary to manually call generateId() before each save. An ideal solution would handle that for me, like Mongoose does with ids of type ObjectId.
Most significantly, there is a potential race condition that will result in duplicate key error. Consider two clients running this code. Both coincidentally generate the same id at the same time, both look in the database and find the id absent, both try to write the record to the database. The second will fail.
An ideal solution would, on save, generate an id, save it to the database and on duplicate key error, generate a new id and retry. Do this in a loop until the document is stored successfully.
The trouble is, I don't know how to get Mongoose to let me do this.
Here's what I tried: Based on this SO Question, I found a rather old sample (using a very old mongoose version) of overriding the save function to accomplish something similar and based this attempt off it.
// First, change generateId() to force a collision
let ids = ['a', 'a', 'a', 'b'];
let index = 0;
let generateId = function() {
return ids[index++];
};
// Configure middleware to generate the id before a save
sampleSchema.pre('validate', function(next) {
if (this.isNew)
this._id = generateId();
next();
});
// Now override the save function
SampleModel.prototype.save_original = SampleModel.prototype.save;
SampleModel.prototype.save = function(options, callback) {
let self = this;
let retryOnDuplicate = function(err, savedDoc) {
if (err) {
if (err.code === 11000 && err.name === 'MongoError') {
self.save(options, retryOnDuplicate);
return;
}
}
if (callback) {
callback(err, savedDoc);
}
};
return self.save_original(options, retryOnDuplicate);
}
This gets me close but I'm leaking a promise and I'm not sure where.
let sampleA = new SampleModel({key: 'a'});
let sampleADoc = await sampleA.save();
console.log('sampleADoc', sampleADoc); // prints undefined, but should print the document
let sampleB = new SampleModel({key: 'b'});
let sampleBDoc = await sampleB.save();
console.log('sampleBDoc', sampleBDoc); // prints undefined, but should print the document
let all = await SampleModel.find();
console.log('all', all); // prints `[]`, but should be an array of two documents
Output
sampleADoc undefined
sampleBDoc undefined
all []
The documents eventually get written to the database, but not before the console.log calls are made.
Where am I leaking a promise? Is there an easier way to do this that addresses the three problems I outlined?
Edit 1:
Mongoose version: 5.11.15
I fixed the problem by changing the save override. The full solution looks like this:
const sampleSchema = new mongoose.Schema({
_id: String,
color: String,
});
let generateId = function() {
return randomStringGenerator.generate({length: 8, charset: 'hex', capitalization: 'uppercase'});
};
sampleSchema.pre('validate', function() {
if (this.isNew)
this._id = generateId();
});
let SampleModel = mongoose.model('Sample', sampleSchema);
SampleModel.prototype.save_original = SampleModel.prototype.save;
SampleModel.prototype.save = function(options, callback) {
let self = this;
let isDupKeyError = (error, field) => {
// Determine whether the error is a duplicate key error on the given field
return error?.code === 11000 && error?.name === 'MongoError' && error?.keyValue[field];
}
let saveWithRetries = (options, callback) => {
// save() returns undefined if used with callback or a Promise otherwise.
// https://mongoosejs.com/docs/api/document.html#document_Document-save
let promise = self.save_original(options, callback);
if (promise) {
return promise.catch((error) => {
if (isDupKeyError(error, '_id')) {
return saveWithRetries(options, callback);
}
throw error;
});
}
};
let retryCallback;
if (callback) {
retryCallback = (error, saved, rows) => {
if (isDupKeyError(error, '_id')) {
saveWithRetries(options, retryCallback);
} else {
callback(error, saved, rows);
}
}
}
return saveWithRetries(options, retryCallback);
}
This will generate an _id repeatedly until a successful save is called and addresses the three problems outlined in the original question:
The minimum trips to the database has been reduced from two to one. Of course, if there are collisions, more trips will occur but that's the exceptional case.
This implementation takes care of generating the id itself with no manual step to take before saving. This reduces complexity and removes the required knowledge of prerequisites for saving that are present in the original method.
The race condition has been addressed. It won't matter if two clients attempt to use the same key. One will succeed and the other will generate a new key and save again.
To improve this:
There ought to be a maximum number of save attempts for a single document followed by failure. In this case, you've perhaps used up all the available keys in whatever domain you're using.
The unique field may not be named _id or you might have multiple fields that require a unique generated value. The embedded helper function isDupKeyError() could be updated to look for multiple keys. Then on error you could add logic to regenerate just the failed key.
It's been a few hours since I began reading about IndexedDB and I am having some issues. This is code I followed from https://medium.com/#AndyHaskell2013/build-a-basic-web-app-with-indexeddb-8ab4f83f8bda and https://gist.github.com/JamesMessinger/a0d6389a5d0e3a24814b and it's just put together poorly for comprehension's sake. So what I am trying to do is store a secret key into IndexedDB so that I can use it to encrypt things with the ECIES scheme.
Relevant code (sequential order):
var secretKey = eccrypto.generatePrivate();
window.genSKey = function()
{
openDB();
console.log(secretKey);
return;
}
window.openDB = function()
{
let db;
let dbReq = indexedDB.open('ln', 1);
dbReq.onupgradeneeded = function(event) {
// Set the db variable to our database so we can use it!
db = event.target.result;
// Create an object store named notes. Object stores
// in databases are where data are stored.
let skey = db.createObjectStore('skey', {keyPath: "id"});
}
dbReq.onsuccess = function(event) {
db = event.target.result;
setSkey(db);
getSkey(db);
}
dbReq.onerror = function(event) {
alert('error opening database ' + event.target.errorCode);
}
}
window.setSkey = function(db) {
// Start a database transaction and get the notes object store
let tx = db.transaction(['skey'], 'readwrite');
let store = tx.objectStore('skey');
// Put the sticky note into the object store
store.put({id: 1, secretKey});
// Wait for the database transaction to complete
tx.oncomplete = function() { console.log('private key was saved') }
tx.onerror = function(event) {
alert('error storing private key ' + event.target.errorCode);
}
}
window.getSkey = function(db)
{
// Set up an object store and transaction
let tx = db.transaction(['skey'], 'readonly');
let store = tx.objectStore('skey');
// Set up a request to get the sticky note with the key 1
let req = store.get(1);
// We can use the note if the request succeeds, getting it in the
// onsuccess handler
req.onsuccess = function(event) {
let skey = event.target.result;
if (skey) {
console.log(skey);
} else {
console.log("skey 1 not found")
}
}
// If we get an error, like that the note wasn't in the object
// store, we handle the error in the onerror handler
req.onerror = function(event) {
alert('error getting note 1 ' + event.target.errorCode);
}
for some reason when genSKey is called in front end scripting, I get the following console error:
Uncaught DOMException: Failed to execute 'transaction' on 'IDBDatabase': One of the specified object stores was not found.
at window.setSkey (http://127.0.0.1:5500/browserify/builds/genKey.js:26763:17)
at IDBOpenDBRequest.dbReq.onsuccess (http://127.0.0.1:5500/browserify/builds/genKey.js:26752:3)
window.setSkey # genKey.js:26763
dbReq.onsuccess # genKey.js:26752
IndexedDB (async)
window.openDB # genKey.js:26740
window.genSKey # genKey.js:26601
signupData # signup.html:186
onclick # signup.html:64
the console logs more things but the other things have nothing to do with the issue at hand. I am not sure what I am doing wrong, and I have been at it all day. Some insight will mean the world to me, thank you.
You probably created the database without the object store. to trigger onupgradeneeded which will go and create the object store you have to connect to the database with a higher version number.
I am attempting to perform an update to a MongoDB document (using mongoose) by first using .findById to get the document, then updating the fields in that document with new values. I am still a bit new to this so I used a tutorial to figure out how to get it working, then I have been updating my code for my needs. Here is the tutorial: MEAN App Tutorial with Angular 4. The original code had a schema defined, but my requirement is for a generic MongoDB interface that will simply take whatever payload is sent to it and send it along to MongoDB. The original tutorial had something like this:
exports.updateTodo = async function(todo){
var id = todo.id
try{
//Find the old Todo Object by the Id
var oldTodo = await ToDo.findById(id);
}catch(e){
throw Error("Error occured while Finding the Todo")
}
// If no old Todo Object exists return false
if(!oldTodo){
return false;
}
console.log(oldTodo)
//Edit the Todo Object
oldTodo.title = todo.title
oldTodo.description = todo.description
oldTodo.status = todo.status
console.log(oldTodo)
try{
var savedTodo = await oldTodo.save()
return savedTodo;
}catch(e){
throw Error("And Error occured while updating the Todo");
}
}
However, since I don't want a schema and want to allow anything through, I don't want to assign static values to specific field names like, title, description, status, etc. So, I came up with this:
exports.updateData = async function(update){
var id = update.id
// Check the existence of the query parameters, If they don't exist then assign a default value
var dbName = update.dbName ? update.dbName : 'test'
var collection = update.collection ? update.collection : 'testing';
const Test = mongoose.model(dbName, TestSchema, collection);
try{
//Find the existing Test object by the Id
var existingData = await Test.findById(id);
}catch(e){
throw Error("Error occurred while finding the Test document - " + e)
}
// If no existing Test object exists return false
if(!existingData){
return false;
}
console.log("Existing document is " + existingData)
//Edit the Test object
existingData = JSON.parse(JSON.stringify(update))
//This was another way to overwrite existing field values, but
//performs a "shallow copy" so it's not desireable
//existingData = Object.assign({}, existingData, update)
//existingData.title = update.title
//existingData.description = update.description
//existingData.status = update.status
console.log("New data is " + existingData)
try{
var savedOutput = await existingData.save()
return savedOutput;
}catch(e){
throw Error("An error occurred while updating the Test document - " + e);
}
}
My original problem with this was that I had a lot of issues getting the new values to overwrite the old ones. Now that that's been solved, I am getting the error of "TypeError: existingData.save is not a function". I am thinking the data type changed or something, and now it is not being accepted. When I uncomment the static values that were in the old tutorial code, it works. This is further supported by my console logging before and after I join the objects, because the first one prints the actual data and the second one prints [object Object]. However, I can't seem to figure out what it's expecting. Any help would be greatly appreciated.
EDIT: I figured it out. Apparently Mongoose has its own data type of "Model" which gets changed if you do anything crazy to the underlying data by using things like JSON.stringify. I used Object.prototype.constructor to figure out the actual object type like so:
console.log("THIS IS BEFORE: " + existingData.constructor);
existingData = JSON.parse(JSON.stringify(update));
console.log("THIS IS AFTER: " + existingData.constructor);
And I got this:
THIS IS BEFORE: function model(doc, fields, skipId) {
model.hooks.execPreSync('createModel', doc);
if (!(this instanceof model)) {
return new model(doc, fields, skipId);
}
Model.call(this, doc, fields, skipId);
}
THIS IS AFTER: function Object() { [native code] }
Which showed me what was actually going on. I added this to fix it:
existingData = new Test(JSON.parse(JSON.stringify(update)));
On a related note, I should probably just use the native MongoDB driver at this point, but it's working, so I'll just put it on my to do list for now.
You've now found a solution but I would suggest using the MongoDB driver which would make your code look something along the lines of this and would make the origional issue disappear:
// MongoDB Settings
const MongoClient = require(`mongodb`).MongoClient;
const mongodb_uri = `mongodb+srv://${REPLACE_mongodb_username}:${REPLACE_mongodb_password}#url-here.gcp.mongodb.net/test`;
const db_name = `test`;
let db; // allows us to reuse the database connection once it is opened
// Open MongoDB Connection
const open_database_connection = async () => {
try {
client = await MongoClient.connect(mongodb_uri);
} catch (err) { throw new Error(err); }
db = client.db(db_name);
};
exports.updateData = async update => {
// open database connection if it isn't already open
try {
if (!db) await open_database_connection();
} catch (err) { throw new Error(err); }
// update document
let savedOutput;
try {
savedOutput = await db.collection(`testing`).updateOne( // .save() is being depreciated
{ // filter
_id: update.id // the '_id' might need to be 'id' depending on how you have set your collection up, usually it is '_id'
},
$set: { // I've assumed that you are overwriting the fields you are updating hence the '$set' operator
update // update here - this is assuming that the update object only contains fields that should be updated
}
// If you want to add a new document if the id isn't found add the below line
// ,{ upsert: true }
);
} catch (err) { throw new Error(`An error occurred while updating the Test document - ${err}`); }
if (savedOutput.matchedCount !== 1) return false; // if you add in '{ upsert: true }' above, then remove this line as it will create a new document
return savedOutput;
}
The collection testing would need to be created before this code but this is only a one-time thing and is very easy - if you are using MongoDB Atlas then you can use MongoDB Compass / go in your online admin to create the collection without a single line of code...
As far as I can see you should need to duplicate the update object. The above reduces the database calls from 2 to one and allows you to reuse the database connection, potentially anywhere else in the application which would help to speed things up. Also don't store your MongoDB credentials directly in the code.
I am trying to get the hang of using indexedDB to store data client side.
consider the following code:
function queryURL(message, sender)
{
chrome.contextMenus.removeAll();
var openRequest = indexedDB.open("Tags",1);
openRequest.onsuccess = function(event){
var queryURL = message['host'];
var db = event.target.result;
var objectStore = db.transaction("domains").objectStore("domains");
var query = objectStore.get(queryURL);
query.onsuccess = function(event){
alert(query.result);
delete query.result["domain"];
createMenuItems(query.result);
available_commands=request.result;
};
db.onerror = function(event){
console.log("an error bubbled up during a transaction.");
};
};
openRequest.onerror = function(event){
console.log("error opening DB");
};
}
I do not fully understand what should be happening in the query.
The result is the same whether or not the key that is queried for is in the database:
query.onsuccess() runs and query.result is undefined so the
code errors and exits as soon as I try to delete a key from
query.result.
If the key is not found, query.onsuccess() should not be
running, correct?
If the key is found, query.result should hold the object that
corresponds to that key, correct?
In case it helps, here is the code that I used to initialize the database:
const db_name="Tags";
var request = window.indexedDB.open(db_name, 1);
var tags = [
//codes: 0 - markdown wrap tag
// 1 - HTML wrap tag
// 2 - single tag
{ domain: "www.youtube.com",
bold:["*",0],
strikethrough:["-",0],
italic:["_",0]
},
{ domain: "www.stackoverflow.com",
bold:["<strong>",1],
italic:["<em>",1],
strikethrough:["<del>",1],
superscript:["<sup>",1],
subscript:["<sub>",1],
heading1:["<h1>",1],
heading2:["<h2>",1],
heading3:["<h3>",1],
blockquote:["<blockquote>",1],
code:["<code>",1],
newline:["<br>",2],
horizontal:["<hr>",2]
}
];
request.onerror = function(event) {
alert("Error opening the database");
};
request.onupgradeneeded = function(event) {
var db = event.target.result;
alert("I'm doing stuff!");
var objectStore = db.createObjectStore("domains", {keyPath: "domain" });
objectStore.createIndex("domain", "domain", { unique: true });
objectStore.transaction.onComplete = function(event) {
var domanStore=db.transaction("domains","readwrite").objectStore("domains");
for(var i in tags)
{
domainStore.add(tags[i]);
}
}
};
Here are some links to the resources I am using:
Using IndexedDB
IDBObjectStore
IDBRequest
Finding out that the result is empty or undefined is a successful query. So yes, you get onsuccess call with result === undefined.
onerror is only reserved for when something breaks, e.g. you supplied an invalid key.
From IDBObjectStore.get docs:
Note: This method produces the same result for: a) a record that doesn't exist in the database and b) a record that has an undefined value. To tell these situations apart, call the openCursor() method with the same key. That method provides a cursor if the record exists, and no cursor if it does not.
Yes. It is even more confusing when delete method return success, even if no record is deleted.
Since request error event is cancellable bubbling event, it is not feasible to invoke to error callback even if no record is found. If request is on error and error is not prevented, its transaction will be aborted and indexedDB.onerror will be called as well. So invoking success with undefined result is still better than invoking error.
I am creating a local IndexedDB for the first time and the browser fires onupgradeneeded request in response to window.indexedDB.open method.
I would like to create multiple tables (e.g. stores) in the onupgradeneeded event, but I'd like to do it in a transaction.
I see that transaction object supports `.objectStore', but that implies already having created the store/table.
How do I create multiple stores and wrap it in a transaction?
To create multiple object stores within onupgradeneeded:
var request = indexedDB.open(...);
request.onupgradeneeded = function(event) {
// side note: this === request === event.target === event.currentTarget
var db = this.result;
// Create 0 or more object stores here using this one instance of IDBDatabase.
db.createObjectStore(...);
db.createObjectStore(...);
...
};
The onupgradeneeded event creates an implicit transaction within the IDBRequest object that is of type VERSION_CHANGE. The transaction applies to all calls within the onupgradeneeded callback. Each of the createObjectStore calls above implicitly use the same transaction.
You can, if you want, get a reference to this transaction use this.transaction within this function. Here you are accessing the implicitly-generated transaction property of the open request, which references an IDBTransaction object that was created for you (with type set to VERSION_CHANGE), which is notably different than creating a transaction explicitly using the IDBDatabase.prototype.transaction method.
You can create multiple object store in onupgradeneeded event handler. It is already in transaction. In fact, it is global exclusive transaction on the database.
After you created required object stores and their indexes, you can create transaction on the database connection. You just need to pass list of object stores in db.transaction.
You can use the transaction onupgradeneeded, but better use only for creating object stores there. Create another transaction for reading and writing after finishing onupgradeneeded event.
Run this example of IndexedDB transaction in the console of your browser
let db;
dbName = "Jokes";
dbVersion = 5;
const request = indexedDB.open(dbName, dbVersion);
request.onupgradeneeded = e => {
db = e.target.result
console.log(db);
const jstore = db.createObjectStore("JokeStore", { keyPath: "title" });
const mstore = db.createObjectStore("MockStore", { keyPath: "title" });
alert("upgrade");
}
request.onsuccess = e => {
db = e.target.result
console.log(db);
alert("success");
}
request.onerror = e => {
alert("error" + e.target.error);
}
const tx = db.transaction("JokeStore", "readwrite");
tx.onerror = e => alert(e.target.error);
const jstoretx = tx.objectStore("JokeStore");
jstoretx.add({ title: "Knock Knock", text: "Who's there? There is a place." });
It creates an entry in the database store.