Looking for advice/options for having persistent local storage using both Firefox and Chrome allowing me to save 50MB+ data. I would be storing dynamic terrain data for a WebGL game, so it wouldn't be necessary for the server to send the whole data every time the player connects. I could just update the old parts.
I thought about using an IndexedDB however Chrome doesn't allow you to increase the quota (unlike Firefox) so I wouldn't be able to store any large data. Chrome allows you to use the FileSystem API which would solve my issue however Firefox does not support it.
So it seems either way it wouldn't work. Is my only option to use the FileSystem API for Chrome and the IndexedDB for Firefox? Does anyone have any better ideas?
This info is what I've read from Mozilla's Blog and Google's Dev Site but that may be outdated now, so please feel free to correct me. Thanks!
I agree with the other comments about caching and the Chrome/Firefox marketplaces, and they may ultimately be better solutions for you. However, to answer your original question...
IndexedDB in Chrome is definitely not limited to 5 MB. You can store far more than 50 MB in IndexedDB in both Firefox and Chrome, assuming the user has enough hard drive space. Higher amounts of IndexedDB storage are regularly reached in this game I wrote. On my computer, I currently have over 500 MB stored in IndexedDB in Chrome for that one domain.
You did correctly link to https://developers.google.com/chrome/whitepapers/storage and it is quite confusing, but as I understand it, basically the upper limit is 10% of the free space on the hard drive. Another caveat described on that page is that IndexedDB is technically "temporary" storage that the browser might delete if space is running low, but in practice this seems to rarely happen (YMMV).
localStorage is limited to 5 MB (I think this was on Chrome) and you can request more space for every new 5 MB through dialog boxes.
Both localStorage and IndexedDb are created for interactive data. Because it looks like the data is not modified on the client-side your options are
Using the application cache (as mentioned in the comment)
Serving the data with cache forever HTTP headers (like 10 years), have unique URLs for new resource versions and let the browser re-download the data when it goes out of the cache (recommended)
Deploy your HTML5 application as a web app, downloads available from Chrome Store and Firefox Marketplace
Related
i'm working on caching objects as JSON and i saw that indexedDB is a great place to do it but i'm wondering that if it has a size limit
is there any limit for indexedDB?
if it has,how can make it unlimited?
i'm using vanilla javascript
thanks a lot
As per the documentation, Chrome will reserve 1/3 of the disk for offline storage. This means if a disk is 99GB large, Chrome will reserve 33GB of storage for offline storage. I would imagine if the usable space is less than that that Chrome adjusts however much it claims to suit.
If an origin is not used often, in cases where the disk is running out of space, Chrome will delete an entire origins' offline storage. An origin can claim a maximum of 20% of the pool that Chrome has, meaning that on the aforementioned disk with 33GB of storage allocated to Chrome, you can only use 6.6GB of storage.
It's recommended though that you don't rely on having a lot of storage to play with. You should handle cases where Chrome denies you the ability to write offline storage, just like with any other app. Chrome will let you know when you attempt to write to offline storage, that you're out of storage.
Most of the browsers provide localStorage with the storage limit of 5MB per domain.
Are there such memory limits/constraints with respect to service workers?
I know that web workers (on which service workers are based) don't have such limitations. But Web Workers are not exactly used for assets caching, instead they're used more for processing (so CPU is the main concern there).
If there's no limit on the memory size, could a badly designed website crash the browser?
Update Jan 15 2018
The StorageManager interface of Storage API is becoming a standard for all storage related api queries. As mentioned by #miguel-lattuada, the estimate API would provide an estimate of the storage used a web app the available storage. Also, note the QuotaExceededError exception which would help us in handling error scenarios.
eg code:
if ('storage' in navigator && 'estimate' in navigator.storage) {
navigator.storage.estimate().then(({usage, quota}) => {
console.log(`Using ${usage} out of ${quota} bytes.`);
}).catch(error => {
console.error('Loading storage estimate failed:');
console.log(error.stack);
});
} else {
console.error('navigator.storage.estimate API unavailable.');
}
For more info, refer the following 2 great articles:
MDN Web docs - Storage API
Google Developers: Estimating Available Storage Space
March 16 2017 (keeping it just for reference/history)
Recently I came across this article: offline-cookbook which states as below:
Your origin is given a certain amount of free space to do what it wants with. That free space is shared between all origin storage: LocalStorage, IndexedDB, Filesystem, and of course Caches.
The amount you get isn't spec'd, it will differ depending on device and storage conditions. You can find out how much you've got via:
navigator.storageQuota.queryInfo("temporary").then(function(info) {
console.log(info.quota);
// Result: <quota in bytes>
console.log(info.usage);
// Result: <used data in bytes>
});
The above code might not work in all the browsers. (for eg: in chrome<48 one might have to look for webkitPersistentStorage etc)
Other useful info/resources
As per Offline Storage for Progressive Web Apps by Addy Osmani
In Chrome and Opera: Your storage is per origin (rather than per API). Both storage mechanisms will store data until the browser quota is reached. Apps can check how much quota they’re using with the Quota Management API (as described above).
Firefox no limits, but will prompt after 50MB data stored
Mobile Safari 50MB max
Desktop Safari unlimited (prompts after 5MB)
IE10+ maxes at 250MB and prompts at 10MB
A more detailed guide on Working with quota on mobile browsers by Eiji Kitamura.
For now these are the most relevant articles/solutions found for my problem. If anyone knows some better article or specifications please share.
There's no explicit limit. All modern browsers are multi-process or similar, so a badly-designed page (or SW) won't do anything worse than crash itself.
Note that the SW spec is very explicit about the browser being able to kill and restart the SW at any time, for any reason. (If DevTools is open on a page, Chrome purposely kills SWs for the page constantly, to encourage you to adopt good practices.)
On latests browser you can use StorageManager which is an implementation of a new standard for browser storage, take a look at this mozilla article.
let _storageStats = await navigator.storage.estimate();
console.log(_storageStats);
/*
Will prompt something like this
{quota: 15946471833, usage: 682}
Which is a representation of quota/usage in bytes
As you can see I get an insane quota of almost 16 GB
*/
More info about navigator.storage.estimate() and navigator.webkitTemporaryStorage.queryUsageAndQuota() is here
https://developers.google.com/web/updates/2017/08/estimating-available-storage-space
Testing page is here https://thimbleprojects.org/startpolymer/361372/
I'm not 100% certain, but I think you're limited entirely by what's available on the client machine. As in, there is no fixed upper limit
If someone was running a beast of a machine and the browser was the only active application, then you'd most likely have plenty of storage available
However, if it was an old limited machine that's barely chugging along; you'd have very little
It depends entirely on what you're trying to do really. You should only really be using service workers to store things vital to your page/application working
I am working on a prototype where we need to support offline data modification of web application, the application is expected to sync back the data when an internet connection is restored. I have taken a look at various HTML5 in-browser storage option and indexeddb looked like the one I wanted. But I am not sure if the data will be persisted between browser close. Is that possible? One more question if I delete the cookie of the browser, will data in indexedDb will be wiped out? My initial tests shows data gets deleted on cookie clear of browser.
If indexeddb is not a viable option, are there any other alternatives to it which can persist data when internet connection is not available?
As per specification database created with indexedDB should be persistent acrross navigation and browser session.
But current implementation is like persistent cookies. So removal of cookies might remove your database too.
As per google chrome indexedDB is a type of temporary storage.
Chrome: https://developer.chrome.com/apps/offline_storage
For microsoft & firefox it is persitent :
Microsoft: https://msdn.microsoft.com/en-in/hh563494.aspx
Firefox: https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Basic_Concepts_Behind_IndexedDB
Check supported browsers before use.
http://caniuse.com/#search=indexeddb
It isn't dependent on a cookie, though if you need to know who the user is (likely) you'll probably end up using a cookie of some variety...
As for offline sychronization... I thought about this a lot previously and created a project... The documentation for it is detailed and explains why and how... It may help, or at least give you things to think about. It has very recently been updated to support IndexedDB!
http://forbesmyester.github.io/SyncIt/index.html
In this space there is also RemoteStorage ( they were/are looking at using SyncIt + other bits in the project going forward ), Hood.ie and the commercial FireBase in this field.
There is usually a limit of 5MB on localStorage on browsers, including iPhone's Safari.
Since PhoneGap has the access higher security privileges including access to other storage mechanisms on the device, in theory they should be able to eliminate the limit of 5MB.
For example, it should be able to get around the usual restrictions by storing the data on a file, etc while keeping the API compatible with localStorage javascript object.
Is this done? Or is PhoneGap limited to the same 5MB?
PhoneGap doesn't do anything out of the ordinary to extend the default limits. On Android, I get 2.5M characters in localStorage (Strings in JavaScript are UTF-16).
You can find default limits for most browsers here: http://dev-test.nemikor.com/web-storage/support-test/
This was helpful in understanding the limitations, and I used the code to create a simplified test PhoneGap app.
PhoneGap has File API that should not be affected by browser local storage limits but don't know if there exist any abstraction to make it behave as HTML5 local storage "backend".
If you want to store a large amount of data you should not do that in localStorage, there are databases and files for that kind of need. localStorage is a key-value datastore, it's use is limited and it should not be "hacked" to fit all needs.
Localstorage is something which is provided by the browser.
Localstorage is not something which is available on a device, either a mobile phone or a desktop, that is leveraged by a browser.
Since it is something which the browser provides there is no way, we can change/increase it using Phonegap since your Phonegap app runs inside the browser.
If you want more storage space, you can use a technique which Phonegap can access like a file storage or SQlLite.
If you open http://app.ft.com (the Financial Times mobile web app), you are prompted to add the app to your "home".
After doing this, when you open the app, you are prompted again to allow the localstoreage database size to be increased up to 50MB.
How can this be done? Is there some JavaScript API call? Permissions or whatever?
Is this iPad (iOS?) specific, or does it work on other Webkit browsers?
I happen to know something about this ;)
There's no API for requesting an increase in storage size for an existing database. There is one way to force an increase: write data to the database in such a size that an increase is required, prompting the user. However, this would be slow and there's no way to tell the currently allocated space, so it's not recommended.
Black Frog has part of this correct: the only neat way to do this is to request a very large database when it is opened, for example:
openDatabase('databaseName', '1.0', 'My Database', 50*1024*1024, …
… to request 50MB of space.
However, when the user first visits the site, you may not want to prompt them about a 50MB limit at once; so you might think that you could ask for 5MB at first, and then later re-open it with 50MB? Unfortunately, this doesn't work - the second open attempt, with an increased quantity, succeeds silently, not prompting for a size increase and not actually increasing the available size.
The FT app therefore starts off with a 5MB "preview" database, so that the user isn't prompted on first load. It tries not to exceed this 5MB limit, as any space assigned has to be shared across all databases.
If the user chooses to allow storage of more content, the app then tries to open a database with a different name with 40MB of space (for which the user is prompted to approve 50MB). This allows 40MB in that database, and 5MB in the original preview database, so neither should fail when inserting rows - as 50MB total is currently the limit on iOS.
All browsers currently handle database space limits differently, so if you're planning cross-platform, test carefully. Desktop Safari handles it rather nicely, allowing much larger; Chrome doesn't allow any increase at all; etc. Expect all "HTML5" implementations to differ in strange ways :)
This database is part of Web SQL Database API, which is not part of HTML5. Use the following the set the size of your database
function prepareDatabase(ready, error) {
return openDatabase('documents', '1.0', 'Offline document storage', 50*1024*1024, function (db) {
db.changeVersion('', '1.0', function (t) {
t.executeSql('CREATE TABLE docids (id, name)');
}, error);
});
}
Introducing Web SQL Databases on HTML5 Doctor has a very quick tutorial on how all of this works.
I just tested with my offline app in iPad 2( iOS 5.1.1) that we do not need to do anything specific inside the app. For e.g., my app has about 18 MB of offline data. When the browser hit URL, browser popped up the message requesting increase in size to 25 MB and I accepted it and all is fine. Thanks
It's browser specific. Most have set it to 5MB and some give the option of increasing it through a setting somewhere. Not all browsers offer this though.
Huh — Dive into HTML5 says that no browser supported this as of February 2011, so I guess this might be an iOS 4.3 thing? (iOS 4.3 shipped in March 2011.)
I can’t find any references to it from a quick Google. Apple’s own developer documentation might mention it — I’m not sure if that’s available to non-SDK subscribers though.