Context
I use PouchDB and have a localDB using indexedDb. I continuously replicating this local DB to a remote DB (CouchDB) with this function:
function replLocalToRemote() {
this.meta.localDB.replicate.to(this.meta.remoteDB, {
// Live: replication occurs as the changes are detected
live: true,
retry: true,
}).on('paused', function (info) {
console.log('paused', info);
// replication was paused, usually because of a lost connection
}).on('change', function (change) {
console.log('change', change);
// yo, something changed!
}).on('active', function (info) {
console.log('active', info);
// replication was resumed
}).on('error', function (err) {
alert('replication failed. Retry... ', err);
// totally unhandled error (shouldn't happen)
});
}
It is working well but problems appear when there is too much data on the device. Especially on IOS where Safari constantly open pop-up windows to ask for more data storage (beginning at 5Mo).
Question
So I would like to clean the localDB as I replicate it to the remoteDB. For instance delete a document on local as soon as it is on remote and continue replication even if there is more data on remote than on local.
Is this possible?
Deleting the local document would cause you problems, as the deletion would be replicated to the remote database so that you lose the document altogether.
Have you seen the "auto-compaction" feature in PouchDb? That minimises the size of the local database by removing any "non-leaf" revision documents (previous versions).
You can enable this feature like this:
var db = new PouchDB('mydb', {auto_compaction: true});
More information about this is on the PouchDb page.
What you can try is create a new local db every now and then (e.g. every day), and starting a new replication from there. Keep the old dbs as long as they still replicate, and remove them as soon as they are fully replicated.
In order not to lose any data, you should take extra care that all data has been replicated to the server before deleting the db.
Related
I am using Dexie to access IndexedDB on a flash card maker project. I can manipulate the database as expected, and it stays in the database if I close and reopen the browser, but often when I open up the project after not working on it for a few days, the stored data I put into the database isn't there any longer.
I did some research and discovered that I need to make the database persistent, but I can't figure out how to do that.
I have pretty much copied the recommended code from the Dexie Storage Manager page into the onLoad function for the main page, index.js. Here is the relevant code:
//When the page loads, display the decks in ul on the webpage.
window.addEventListener('load', onLoad);
async function onLoad() {
//Check how much data is available.
showEstimatedQuota();
//Make sure the storage is persistent.
isStoragePersisted().then(async isPersisted => {
if (isPersisted) {
console.log(":) Storage is successfully persisted.");
} else {
console.log(":( Storage is not persisted.");
console.log("Trying to persist..:");
if (await persist()) {
console.log(":) We successfully turned the storage to be persisted.");
} else {
console.log(":( Failed to make storage persisted");
}
}
});
}
The above onLoad function references three functions I have saved on dexie-setup.js:
//This function makes the storage persistent.
//(Copied from https://dexie.org/docs/StorageManager)
async function persist() {
return await navigator.storage && navigator.storage.persist &&
navigator.storage.persist();
}
//This function checks if the storage is persistent.
//(Copied from https://dexie.org/docs/StorageManager)
async function isStoragePersisted() {
return await navigator.storage && navigator.storage.persisted &&
navigator.storage.persisted();
}
//This function logs to console how much data is available.
//(Copied from https://dexie.org/docs/StorageManager)
async function showEstimatedQuota() {
if (navigator.storage && navigator.storage.estimate) {
const estimation = await navigator.storage.estimate();
console.log(`Quota: ${estimation.quota}`);
console.log(`Usage: ${estimation.usage}`);
} else {
console.error("StorageManager not found");
}
}
My console logs:
dexie-setup.js:56 Quota: 6358499328
dexie-setup.js:57 Usage: 25370
index.js:30 :( Storage is not persisted.
index.js:31 Trying to
persist..:
dexie-setup.js:84 Done checking dexie.
index.js:33 :) We successfully turned the storage to be persisted.
However, if I refresh the page, I get the same thing logged on my console: the database is still set to not persistent.
The showEstimatedQuota function checks the data storage and confirms that the DataStorage API is functioning, so I don't think that's the problem. (I'm only storing small objects with text in them, so I don't expect to exceed the storage limit, anyway.)
So far, the project is entirely local on my chromebook, and I am viewing it on a Chrome browser.
Please let me know how to make my database persistent. I'm pretty new to this (this is my first question on stackoverflow!), so hopefully it's an easy problem to solve! Thanks in advance.
citing the documentation of Dexie: "Even though IndexedDB is a fully functional client-side database for the web, it is not a persistent storage by default. IndexedDB without StorageManager is just a “best-effort” database that can be erased in situations of low disk space on a device. The browser may delete your database without notifying the user in case it needs to free up space for other website’s data that was used more recently than yours."
So, you can't make the database persistent. Just make a “best-effort”.
This links can be of help:
https://web.dev/persistent-storage/
Chrome.Storage.Local Persistence
I hope it will be of help to you.
The only way I have found is that if the user bookmarks the site then it enables persistent storage using the persist function:
//This function makes the storage persistent.
//(Copied from https://dexie.org/docs/StorageManager)
async function persist() {
return await navigator.storage && navigator.storage.persist &&
navigator.storage.persist();
}
So you may prompt the user to bookmark your site when it loads.
I am trying to sync a local PouchDB instance to a remote CouchDB. Things work great, but I am not sure how to deal with the following situation:
I have added a validation rule in CouchDB to prevent updating (it will deny all updates). When I run the sync function on my local PouchDB instance after modifying a document, the "denied" event fires as I would expect. However, if I run sync a second time, the "denied" event doesn't fire again, even though the local document differs from the CouchDB version.
How can I check if the local database matches the remote database? If I miss the "denied" event the first time (lets say the user closes the browser), how can I detect on the next run that the databases are not in sync? How can I force PouchDB to try and sync the modified document again so that I can see the denied event?
Thanks!
syncPouch: function(){
var opts = {};
var sync = PouchDB.sync('orders', db.remoteDB, opts);
sync.on('change', function (info) {});
sync.on('paused', function(){
});
sync.on('active', function () {});
sync.on('denied', function(err){
//This only fire once no matter how many times I call syncPouch
console.log("Denied!!!!!!!!!!!!");
debugger;
});
sync.on('complete', function (info) {
//This fires every time
console.log("complete");console.log(info);
});
sync.on('error', function(err){
debugger;
});
return sync;
},
What I have noticed with validate_doc_update functions is that PouchDb appears to treat any "denied" document as sync-ed. So even if you then remove the validate_doc_update function, the document will not sync into the remote database on future attempts even though it is not the same.
So you can be left with an "out of sync" situation that can only be fixed by editing one of the documents again.
Perhaps you are seeing the same thing? Perhaps the "denied" event does not fire because there is no attempt by PouchDb to sync the document (as it has already attempted to sync it previously)?
What I am trying to achieve:
render page with loader/spinner
if service-worker.js is registered and active, then check for updates
if no updates, then remove loader
if updatefound and new version installed, then reload the page
else register service-worker.js
when updatefound, meaning new one was installed, remove loader
I am using sw-precache module for me to generate service-worker.js and following registration code:
window.addEventListener('load', function() {
// show loader
addLoader();
navigator.serviceWorker.register('service-worker.js')
.then(function(swRegistration) {
// react to changes in `service-worker.js`
swRegistration.onupdatefound = function() {
var installingWorker = swRegistration.installing;
installingWorker.onstatechange = function() {
if(installingWorker.state === 'installed' && navigator.serviceWorker.controller){
// updated content installed
window.location.reload();
} else if (installingWorker.state === 'installed'){
// new sw registered and content cached
removeLoader();
}
};
}
if(swRegistration.active){
// here I know that `service-worker.js` was already installed
// but not sure it there are changes
// If there are no changes it is the last thing I can check
// AFAIK no events are fired afterwards
}
})
.catch(function(e) {
console.error('Error during service worker registration:', e);
});
});
After reading the spec it is clear that there are no handlers for something like updatenotfound. Looks like serviceWorker.register checks if service-worker.js changed internally by running get-newest-worker-algorithm, but I cannot see similar methods exposed via public api.
I think my options are:
wait for couple of seconds after service worker registration becomes active to see if onupdatefound is fired
fire custom events from service-worker.js code if cache was not updated
Any other suggestions?
Edit:
I've came up with some code which solves this issue by using postMessage() between SW registration and SW client (as #pate suggested)
Following demo tries to achieve checks through postMessage between SW client and SW registration, but fails as SW code is already cached DEMO
Edit:
So by now it looks like I cannot implement what I want because:
when service worker is active you cannot check for updates by evaluating some code in SW - this is still the same cached SW, no changes there
you need to wait for onupdatefound, there is nothing else that will notify of changes in SW
activation of older SW comes before onupdatefound
if there is no change, nothing fires after activation
SW registration update() is immature, keeps changing, Starting with Chrome 46, update() returns a promise that resolves with 'undefined' if the operation completed successfully or there was no update
setting timeout to postpone view rendering is suboptimal as there is no clear answer to how long should it be set to, it depends on SW size as well
The other answer, provided by Fabio, doesn't work. The Service Worker script has no access to the DOM. It's not possible to remove anything from the DOM or, for instance, manipulate any data that is handling DOM elements from inside the Service Worker. That script runs separately with no shared state.
What you can do, though, is send messages between the actual page-running JS and the Service Worker. I'm not sure if this is the best possible way to do what the OP is asking but can be used to achieve it.
Register an onmessage handler on the page
Send a message from the SW's activate or install event to the page
Act accordingly when the message is received on the page
I have myself kept SW version number in a variable inside the SW. My SW has then posted that version number to the page and the page has stored it into the localStorage of the browser. The next time the page is loaded SW posts it current version number to the page and the onmessage handler compares it to the currently stored version number. If they are not the same, then the SW has been updated to some version number that was included in the mssage. After that I've updated the localStorage copy of the version number and done this and that.
This flow could also be done in the other way around: send a message from the page to the SW and let SW answer something back, then act accordingly.
I hope I was able to explain my thoughts clearly :)
The only way that pops up in my mind is to start the loader normally and then remove it in the service-worker, in the install function. I will give this a try, in your service-worker.js:
self.addEventListener('install', function(e) {
console.log('[ServiceWorker] Install');
e.waitUntil(
caches.open(cacheName).then(function(cache) {
console.log('[ServiceWorker] Caching app shell');
return cache.addAll(filesToCache);
}).then(function() {
***removeLoader();***
return self.skipWaiting();
})
);
});
i am writing an offline web application.
i have a manifest files for my applicationCache. and i have handlers for appCache events.
i want to detect if the files being downloaded are being downloaded for the first time or being updated. because in case they are being updated, i would like prevent my application code from running, since i will refresh the browser after updating my app.
my specific problem here is that when the "checking" events gets fired, the applicationCache status is already "DOWNLOADING",
does anybody know how to get the applicationCache.status before any manifest or files gets downloaded?
thanks in advance for any answer
window.applicationCache.addEventListener('checking', function (event) {
console.log("Checking for updates.");
console.log("inside checking event, appcache status : %s", applicationCache.status);
}, false);
window.applicationCache.addEventListener('updateready', function (e) {
if (window.applicationCache.status == window.applicationCache.UPDATEREADY) {
// Browser downloaded a new version of manifest files
window.location.reload();
}
}, false);
window.applicationCache.addEventListener('downloading', function (event) {
appCommon.information("New version available", "Updating application files...", null, null);
}, false);
window.applicationCache.addEventListener('progress', function (event) {
$("#informationModal").find(".modal-body").html("Updating application files... " + event.loaded.toString() + " of " + event.total.toString());
}, false);
window.applicationCache.addEventListener('cached', function (event) {
$("#informationModal").find(".modal-body").html("Application up to date.");
setTimeout(function () { $("#informationModal").find(".close").click(); }, 1000);
}, false);
According to the specification the checking event is always the first event.
the user agent is checking for an update, or attempting to download the manifest for the first time. This is always the first event in the sequence.
I think it is a mistake to refresh the browser since that will not automatically fetch a new copy. If you look at the MDN guide for using the applicationCache you'll see that files are always loaded from the applicationCache first and then proceed to go through the cycle of events you attempted to avoid by refreshing.
Instead of refreshing you should simply make proper use of the applicationCache event life cycle in order to initialize and start your application.
i would like prevent my application code from running, since i will refresh the browser after updating my app.
You have a lot of control of when your application begins to run, if you really wanted to refresh the browser, just always start the app after updateready or noupdate events, but really instead of refreshing it seems like you should use window.applicationCache.swapCache instead.
I found this question and answer useful.
Using essentially the example from the MDN IndexedDb tutorial I can see that my test IndexedDb code is working on Chrome. When I load the app onto my Windows Phone 8 device inside of the deviceready handler, I get an AbortError in the error handler for the database open request.
The only other related SO question was solved by fixing errors in onupgradeneeded but this handler is never even called in my code.
In this simple example, you have to run the fiddle twice because apparently onsuccess is called (where I read a test value) before onupgradeneeded (where I write the value when the db is initialized). I was going to deal with this once I got this first test to work.
http://jsfiddle.net/WDUVx/2/
// In the following line, you should include the prefixes of
// implementations you want to test.
window.indexedDB = window.indexedDB || window.mozIndexedDB || window.webkitIndexedDB || window.msIndexedDB;
window.IDBTransaction = window.IDBTransaction || window.webkitIDBTransaction || window.msIDBTransaction;
window.IDBKeyRange = window.IDBKeyRange || window.webkitIDBKeyRange || window.msIDBKeyRange;
if (!window.indexedDB) {
window.alert("Your browser doesn't support a stable version of IndexedDB. Such and such feature will not be available.");
}
// open the database
var db;
var request = window.indexedDB.open("MyTestDatabase", 1);
request.onerror = function(e) {
alert("Couldn't open database: " + kc.util.getObjectString(e.target));
};
request.onsuccess = function(e) {
db = e.target.result;
var getRequest =
db.transaction("data")
.objectStore("data")
.get("firstObject")
.onsuccess = function(event) {
alert("Got: " + event.target.result.test);
};
};
request.onupgradeneeded = function(e) {
var db = event.target.result;
var objectStore = db.createObjectStore("data", {
autoincrement : false
});
objectStore.transaction.oncomplete = function(event) {
var myObjectStore = db.transaction("data", "readwrite").objectStore("data");
var addRequest = myObjectStore.add({
test : true
},
"firstObject");
addRequest.onerror = function(e) {
console.log("Error adding");
};
addRequest.onsuccess = function(e) {
console.log("Added!");
};
};
};
Questions:
What is my stupid mistake?
Are there any Windows Phone 8 examples of IndexedDb and PhoneGap? I could not find any after some searching. There were a few for an IndexedDb API Android and IOS polyfill, but none for wp8.
Is there something special that I have to do because I'm on a phone? Again, the code works in chrome.
Are there any other plugins that support wp8 storage > 5mb?
LocalStorage has a 5mb size limit
WebSQL is not supported
FileSystem plugin does not support filewriter.write(blob). This is what I am using for Android/iOS. It's strange that they say wp8 is supported by this plugin when this is the only way to actually write data, and you can't read the nothing you can write. I found that although the web api does not support it, the devices support filewriter.write(string). Windows Phone 8 is still not writing/reading things entirely correctly, but that is a separate question.
Recently, I faced a similar issue with indexed DB. My IndexedDB.open request was throwing an abort error.
After doing some search, I found suggestions to separate the requests for database creation and store creation.
Separating the code prevented the abort error. However, I noticed that the transaction to create the stores was sometimes run even before the completion of the database creation request.
This meant that my database connection was not closed from the first request, when the second request was run.
A minor fix was required to overcome this error. I moved the code to create stores to the success event for the first call.
Here is the code for reference.
function create_db(db_name)
{
var request = indexedDB.open(db_name);
request.onupgradeneeded=function(e)
{
console.log("1. creating database");
db=e.target.result;
};
request.onsuccess = function(e)
{
db = e.target.result;
console.log("1.1 database created successfully");
db.close();
add_tables(db_name);
};
request.onerror=function(e)
{
alert("error: "+ e.target.error.name + "failed creating db");
console.log("1.2 error creating db");
};
}
function add_tables(db_name)
{
var request = indexedDB.open(db_name,2);
request.onsuccess=function(e)
{
db=e.target.result;
console.log("2.2 table creation request successful");
};
request.onupgradeneeded=function(e)
{
db=e.target.result;
table = db.createObjectStore("table_name");
table.createIndex("id","id");
console.log("2.2 creating a single object store");
};
request.onerror=function(e)
{
console.log("2.3 error occured when creating tables");
};
};
Just some ideas, hope they help:
Don't use a global db variable. Do all of your work in callbacks. Using a global db variable can lead to numerous in-explainable situations, some of which include getting abort errors. Looking at your code, it actually looks like you are properly just using e.target, so I am not sure why you have a global var db.
Don't perform read/write requests on the version change transaction that occurs in the onupgradeneeded callback. Instead, perform requests when they are appropriate and let indexeddb worry about calling onupgradeneeded. In other words, don't retrieve the transaction in onupgradeneeded. Instead, just initiate some later transaction in a new connection as if the onupgradeneeded callback already completed.
openDBRequest having an abort event precedes onupgradeneeded callback
onupgradeneeded won't be called unless you make an attempt to connect to a database using a higher version
Listen for abort events. Add a callback to the open database request for onabort. Abort events sometimes occur because of things like opening two pages in the same context that try to access the same database. There could be something funky going on there.