Since localStorage (currently) only supports strings as values, and in order to do that the objects need to be stringified (stored as JSON-string) before they can be stored, is there a defined limitation regarding the length of the values.
Does anyone know if there is a definition which applies to all browsers?
Quoting from the Wikipedia article on Web Storage:
Web storage can be viewed simplistically as an improvement on cookies, providing much greater storage capacity (10 MB per origin in Google Chrome(https://plus.google.com/u/0/+FrancoisBeaufort/posts/S5Q9HqDB8bh), Mozilla Firefox, and Opera; 10 MB per storage area in Internet Explorer) and better programmatic interfaces.
And also quoting from a John Resig article [posted January 2007]:
Storage Space
It is implied that, with DOM Storage,
you have considerably more storage
space than the typical user agent
limitations imposed upon Cookies.
However, the amount that is provided
is not defined in the specification,
nor is it meaningfully broadcast by
the user agent.
If you look at the Mozilla source code
we can see that 5120KB is the default
storage size for an entire domain.
This gives you considerably more space
to work with than a typical 2KB
cookie.
However, the size of this storage area
can be customized by the user (so a
5MB storage area is not guaranteed,
nor is it implied) and the user agent
(Opera, for example, may only provide
3MB - but only time will tell.)
Actually Opera doesn't have 5MB limit. It offers to increase limit as applications requires more. User can even choose "Unlimited storage" for a domain.
You can easily test localStorage limits/quota yourself.
Here's a straightforward script for finding out the limit:
if (localStorage && !localStorage.getItem('size')) {
var i = 0;
try {
// Test up to 10 MB
for (i = 250; i <= 10000; i += 250) {
localStorage.setItem('test', new Array((i * 1024) + 1).join('a'));
}
} catch (e) {
localStorage.removeItem('test');
localStorage.setItem('size', i - 250);
}
}
Here's the gist, JSFiddle and blog post.
The script will test setting increasingly larger strings of text until the browser throws and exception. At that point it’ll clear out the test data and set a size key in localStorage storing the size in kilobytes.
Find the maximum length of a single string that can be stored in localStorage
This snippet will find the maximum length of a String that can be stored in localStorage per domain.
//Clear localStorage
for (var item in localStorage) delete localStorage[item];
window.result = window.result || document.getElementById('result');
result.textContent = 'Test running…';
//Start test
//Defer running so DOM can be updated with "test running" message
setTimeout(function () {
//Variables
var low = 0,
high = 2e9,
half;
//Two billion may be a little low as a starting point, so increase if necessary
while (canStore(high)) high *= 2;
//Keep refining until low and high are equal
while (low !== high) {
half = Math.floor((high - low) / 2 + low);
//Check if we can't scale down any further
if (low === half || high === half) {
console.info(low, high, half);
//Set low to the maximum possible amount that can be stored
low = canStore(high) ? high : low;
high = low;
break;
}
//Check if the maximum storage is no higher than half
if (storageMaxBetween(low, half)) {
high = half;
//The only other possibility is that it's higher than half but not higher than "high"
} else {
low = half + 1;
}
}
//Show the result we found!
result.innerHTML = 'The maximum length of a string that can be stored in localStorage is <strong>' + low + '</strong> characters.';
//Functions
function canStore(strLen) {
try {
delete localStorage.foo;
localStorage.foo = Array(strLen + 1).join('A');
return true;
} catch (ex) {
return false;
}
}
function storageMaxBetween(low, high) {
return canStore(low) && !canStore(high);
}
}, 0);
<h1>LocalStorage single value max length test</h1>
<div id='result'>Please enable JavaScript</div>
Note that the length of a string is limited in JavaScript; if you want to view the maximum amount of data that can be stored in localStorage when not limited to a single string, you can use the code in this answer.
Edit: Stack Snippets don't support localStorage, so here is a link to JSFiddle.
Results
Chrome (45.0.2454.101): 5242878 characters
Firefox (40.0.1): 5242883 characters
Internet Explorer (11.0.9600.18036): 16386 122066 122070 characters
I get different results on each run in Internet Explorer.
Don't assume 5MB is available - localStorage capacity varies by browser, with 2.5MB, 5MB and unlimited being the most common values.
Source: http://dev-test.nemikor.com/web-storage/support-test/
I wrote this simple code that is testing localStorage size in bytes.
https://github.com/gkucmierz/Test-of-localStorage-limits-quota
const check = bytes => {
try {
localStorage.clear();
localStorage.setItem('a', '0'.repeat(bytes));
localStorage.clear();
return true;
} catch(e) {
localStorage.clear();
return false;
}
};
Github pages:
https://gkucmierz.github.io/Test-of-localStorage-limits-quota/
I have the same results on desktop Google chrome, opera, firefox, brave and mobile chrome which is ~10Mbytes
And half smaller result in safari ~4Mbytes
You don't want to stringify large objects into a single localStorage entry. That would be very inefficient - the whole thing would have to be parsed and re-encoded every time some slight detail changes. Also, JSON can't handle multiple cross references within an object structure and wipes out a lot of details, e.g. the constructor, non-numerical properties of arrays, what's in a sparse entry, etc.
Instead, you can use Rhaboo. It stores large objects using lots of localStorage entries so you can make small changes quickly. The restored objects are much more accurate copies of the saved ones and the API is incredibly simple. E.g.:
var store = Rhaboo.persistent('Some name');
store.write('count', store.count ? store.count+1 : 1);
store.write('somethingfancy', {
one: ['man', 'went'],
2: 'mow',
went: [ 2, { mow: ['a', 'meadow' ] }, {} ]
});
store.somethingfancy.went[1].mow.write(1, 'lawn');
BTW, I wrote it.
I've condensed a binary test into this function that I use:
function getStorageTotalSize(upperLimit/*in bytes*/) {
var store = localStorage, testkey = "$_test"; // (NOTE: Test key is part of the storage!!! It should also be an even number of characters)
var test = function (_size) { try { store.removeItem(testkey); store.setItem(testkey, new Array(_size + 1).join('0')); } catch (_ex) { return false; } return true; }
var backup = {};
for (var i = 0, n = store.length; i < n; ++i) backup[store.key(i)] = store.getItem(store.key(i));
store.clear(); // (you could iterate over the items and backup first then restore later)
var low = 0, high = 1, _upperLimit = (upperLimit || 1024 * 1024 * 1024) / 2, upperTest = true;
while ((upperTest = test(high)) && high < _upperLimit) { low = high; high *= 2; }
if (!upperTest) {
var half = ~~((high - low + 1) / 2); // (~~ is a faster Math.floor())
high -= half;
while (half > 0) high += (half = ~~(half / 2)) * (test(high) ? 1 : -1);
high = testkey.length + high;
}
if (high > _upperLimit) high = _upperLimit;
store.removeItem(testkey);
for (var p in backup) store.setItem(p, backup[p]);
return high * 2; // (*2 because of Unicode storage)
}
It also backs up the contents before testing, then restores them.
How it works: It doubles the size until the limit is reached or the test fails. It then stores half the distance between low and high and subtracts/adds a half of the half each time (subtract on failure and add on success); honing into the proper value.
upperLimit is 1GB by default, and just limits how far upwards to scan exponentially before starting the binary search. I doubt this will even need to be changed, but I'm always thinking ahead. ;)
On Chrome:
> getStorageTotalSize();
> 10485762
> 10485762/2
> 5242881
> localStorage.setItem("a", new Array(5242880).join("0")) // works
> localStorage.setItem("a", new Array(5242881).join("0")) // fails ('a' takes one spot [2 bytes])
IE11, Edge, and FireFox also report the same max size (10485762 bytes).
You can use the following code in modern browsers to efficiently check the storage quota (total & used) in real-time:
if ('storage' in navigator && 'estimate' in navigator.storage) {
navigator.storage.estimate()
.then(estimate => {
console.log("Usage (in Bytes): ", estimate.usage,
", Total Quota (in Bytes): ", estimate.quota);
});
}
I'm doing the following:
getLocalStorageSizeLimit = function () {
var maxLength = Math.pow(2,24);
var preLength = 0;
var hugeString = "0";
var testString;
var keyName = "testingLengthKey";
//2^24 = 16777216 should be enough to all browsers
testString = (new Array(Math.pow(2, 24))).join("X");
while (maxLength !== preLength) {
try {
localStorage.setItem(keyName, testString);
preLength = testString.length;
maxLength = Math.ceil(preLength + ((hugeString.length - preLength) / 2));
testString = hugeString.substr(0, maxLength);
} catch (e) {
hugeString = testString;
maxLength = Math.floor(testString.length - (testString.length - preLength) / 2);
testString = hugeString.substr(0, maxLength);
}
}
localStorage.removeItem(keyName);
// Original used this.storageObject in place of localStorage. I can only guess the goal is to check the size of the localStorage with everything but the testString given that maxLength is then added.
maxLength = JSON.stringify(localStorage).length + maxLength + keyName.length - 2;
return maxLength;
};
I really like cdmckay's answer, but it does not really look good to check the size in a real time: it is just too slow (2 seconds for me). This is the improved version, which is way faster and more exact, also with an option to choose how big the error can be (default 250,000, the smaller error is - the longer the calculation is):
function getLocalStorageMaxSize(error) {
if (localStorage) {
var max = 10 * 1024 * 1024,
i = 64,
string1024 = '',
string = '',
// generate a random key
testKey = 'size-test-' + Math.random().toString(),
minimalFound = 0,
error = error || 25e4;
// fill a string with 1024 symbols / bytes
while (i--) string1024 += 1e16;
i = max / 1024;
// fill a string with 'max' amount of symbols / bytes
while (i--) string += string1024;
i = max;
// binary search implementation
while (i > 1) {
try {
localStorage.setItem(testKey, string.substr(0, i));
localStorage.removeItem(testKey);
if (minimalFound < i - error) {
minimalFound = i;
i = i * 1.5;
}
else break;
} catch (e) {
localStorage.removeItem(testKey);
i = minimalFound + (i - minimalFound) / 2;
}
}
return minimalFound;
}
}
To test:
console.log(getLocalStorageMaxSize()); // takes .3s
console.log(getLocalStorageMaxSize(.1)); // takes 2s, but way more exact
This works dramatically faster for the standard error; also it can be much more exact when necessary.
Once I developed Chrome (desktop browser) extension and tested Local Storage real max size for this reason.
My results:
Ubuntu 18.04.1 LTS (64-bit)
Chrome 71.0.3578.98 (Official Build) (64-bit)
Local Storage content size 10240 KB (10 MB)
More than 10240 KB usage returned me the error:
Uncaught DOMException: Failed to execute 'setItem' on 'Storage': Setting the value of 'notes' exceeded the quota.
Edit on Oct 23, 2020
For a Chrome extensions available chrome.storage API. If you declare the "storage" permission in manifest.js:
{
"name": "My extension",
...
"permissions": ["storage"],
...
}
You can access it like this:
chrome.storage.local.QUOTA_BYTES // 5242880 (in bytes)
Related
I'm trying to calculate the EMA with this script.
But it does not give me the correct EMA.
This could be of many reasons, but i'm not sure what it is.
I've tried different formulas for the EMA without any better results, I'm really not a professional coder nor mathematician and thus i can't see what I'm doing wrong.
How Is the EMA value calculated over time? For the first value I calculate the first EMA using the SMA, i guess that should work - right?
My EMA value = 0.033144798412698406
Real EMA value = 0.033084
Close = last closing price
Period = 20;
Multiplier = (2 / (period + 1));
function calculateEMA() {
if (EMA == 0) {
EMA = (Close - SMA) * multiplier + SMA;
calculateEMA();
} else {
for (a = 0; a < period; a++) {
EMA = (Close - previous_ema) * multiplier + previous_ema;
console.log(EMA + " ema");
previous_ema = EMA;
}
}
}
// UPDATE Added my whole script (which can be runned)- https://pastebin.com/91GEuATM
You need Nodejs and the binance node api installed (npm install node-binance-api --save) ; Keep in mind that this is just my "test script" hence all the weird variable names etc.
//UPDATE 2 Ticks sample data https://pastebin.com/AFzf7GwQ
I've written a JavaScript program that calculates the depth of a binary tree based on the number of elements. My program has been working fine for months, but recently I've found a difference when the web page is viewed in Chrome vs Firefox.
In particular, on Firefox:
Math.log2(8) = 3
but now in Chrome:
Math.log2(8) = 2.9999999999999996
My JavaScript program was originally written to find the depth of the binary tree based on the number of elements as:
var tree_depth = Math.floor(Math.log2(n_elements)) + 1;
I made a simple modification to this formula so that it will still work correctly on Chrome:
var epsilon = 1.e-5;
var tree_depth = Math.floor(Math.log2(n_elements) + epsilon) + 1;
I have 2 questions:
Has anyone else noticed a change in the precision in Chrome recently for Math.log2?
Is there a more elegant modification than the one I made above by adding epsilon?
Note: Math.log2 hasn't actually changed since it's been implemented
in V8. Maybe you remembered incorrectly or you had included a shim that
happened to get the result correct for these special cases before Chrome
included its own implementation of Math.log2.
Also, it seems that you should be using Math.ceil(x) rather than
Math.floor(x) + 1.
How can I solve this?
To avoid relying on Math.log or Math.log2 being accurate amongst different implementations of JavaScript (the algorithm used is implementation-defined), you can use bitwise operators if you have less than 232 elements in your binary tree. This obviously isn't the fastest way of doing this (this is only O(n)), but it's a relatively simple example:
function log2floor(x) {
// match the behaviour of Math.floor(Math.log2(x)), change it if you like
if (x === 0) return -Infinity;
for (var i = 0; i < 32; ++i) {
if (x >>> i === 1) return i;
}
}
console.log(log2floor(36) + 1); // 6
How is Math.log2 currently implemented in different browsers?
The current implementation in Chrome is inaccurate as they rely on multiplying the value of Math.log(x) by Math.LOG2E, making it susceptible to rounding error (source):
// ES6 draft 09-27-13, section 20.2.2.22.
function MathLog2(x) {
return MathLog(x) * 1.442695040888963407; // log2(x) = log(x)/log(2).
}
If you are running Firefox, it either uses the native log2 function (if present), or if not (e.g. on Windows), uses a similar implementation to Chrome (source).
The only difference is that instead of multiplying, they divide by log(2) instead:
#if !HAVE_LOG2
double log2(double x)
{
return log(x) / M_LN2;
}
#endif
Multiplying or dividing: how much of a difference does it make?
To test the difference between dividing by Math.LN2 and multiplying by Math.LOG2E, we can use the following test:
function log2d(x) { return Math.log(x) / Math.LN2; }
function log2m(x) { return Math.log(x) * Math.LOG2E; }
// 2^1024 rounds to Infinity
for (var i = 0; i < 1024; ++i) {
var resultD = log2d(Math.pow(2, i));
var resultM = log2m(Math.pow(2, i));
if (resultD !== i) console.log('log2d: expected ' + i + ', actual ' + resultD);
if (resultM !== i) console.log('log2m: expected ' + i + ', actual ' + resultM);
}
Note that no matter which function you use, they still have floating point errors for certain values1. It just so happens that the floating point representation of log(2) is less than the actual value, resulting in a value higher than the actual value (while log2(e) is lower). This means that using log(2) will round down to the correct value for these special cases.
1: log(pow(2, 29)) / log(2) === 29.000000000000004
You could perhaps do this instead
// Math.log2(n_elements) to 10 decimal places
var tree_depth = Math.floor(Math.round(Math.log2(n_elements) * 10000000000) / 10000000000);
I am currently developing a site that will make use of HTML5's localStorage. I've read all about the size limitations for different browsers. However, I haven't seen anything on how to find out the current size of a localStorage instance. This question seems to indicate that JavaScript doesn't have a built in way of showing the size for a given variable. Does localStorage have a memory size property that I haven't seen? Is there an easy way to do this that I'm missing?
My site is meant to allow users to enter information in an 'offline' mode, so being able to give them a warning when the storage is almost full is very important.
Execute this snippet in JavaScript console (one line version):
var _lsTotal=0,_xLen,_x;for(_x in localStorage){ if(!localStorage.hasOwnProperty(_x)){continue;} _xLen= ((localStorage[_x].length + _x.length)* 2);_lsTotal+=_xLen; console.log(_x.substr(0,50)+" = "+ (_xLen/1024).toFixed(2)+" KB")};console.log("Total = " + (_lsTotal / 1024).toFixed(2) + " KB");
The same code in multiple lines for reading sake
var _lsTotal = 0,
_xLen, _x;
for (_x in localStorage) {
if (!localStorage.hasOwnProperty(_x)) {
continue;
}
_xLen = ((localStorage[_x].length + _x.length) * 2);
_lsTotal += _xLen;
console.log(_x.substr(0, 50) + " = " + (_xLen / 1024).toFixed(2) + " KB")
};
console.log("Total = " + (_lsTotal / 1024).toFixed(2) + " KB");
or add this text in the field 'location' of a bookmark for convenient usage
javascript: var x, xLen, log=[],total=0;for (x in localStorage){if(!localStorage.hasOwnProperty(x)){continue;} xLen = ((localStorage[x].length * 2 + x.length * 2)/1024); log.push(x.substr(0,30) + " = " + xLen.toFixed(2) + " KB"); total+= xLen}; if (total > 1024){log.unshift("Total = " + (total/1024).toFixed(2)+ " MB");}else{log.unshift("Total = " + total.toFixed(2)+ " KB");}; alert(log.join("\n"));
P.S. Snippets are updated according to request in the comment. Now the calculation includes the length of the key itself.
Each length is multiplied by 2 because the char in javascript stores as UTF-16 (occupies 2 bytes)
P.P.S. Should work both in Chrome and Firefox.
Going off of what #Shourav said above, I wrote a small function that should accurately grab all your the localStorage keys (for the current domain) and calculate the combined size so that you know exactly how much memory is taken up by your localStorage object:
var localStorageSpace = function(){
var allStrings = '';
for(var key in window.localStorage){
if(window.localStorage.hasOwnProperty(key)){
allStrings += window.localStorage[key];
}
}
return allStrings ? 3 + ((allStrings.length*16)/(8*1024)) + ' KB' : 'Empty (0 KB)';
};
Mine returned: "30.896484375 KB"
You can get the current size of the local storage data using the Blob function. This may not work in old browsers, check the support for new Blob and Object.values() at caniuse.
Example:
return new Blob(Object.values(localStorage)).size;
Object.values() turns the localStorage object to an array. Blob turns the array into raw data.
IE has a remainingSpace property of the Storage object. The other browsers have no equivalent at this time.
I believe that the default amount of space is 5MB, although I have not tested it personally.
Here is a simple example of how to do this and should work with every browser
alert(1024 * 1024 * 5 - unescape(encodeURIComponent(JSON.stringify(localStorage))).length);
Hope this help someone.
Because Jas- example on jsfiddle does not work for me I came up with this solution.
(thanks to Serge Seletskyy and Shourav for their bits I used in the code below)
Below is the function that can be used to test how much space is available for localStorage and (if any keys are already in lS) how much space is left.
It is a little brute force but it works in almost every browser... apart from Firefox.
Well in desktop FF it takes ages (4-5min) to complete, and on Android it just crashes.
Underneath the function is a short summary of tests that I have done in different browsers on different platforms. Enjoy!
function testLocalStorage() {
var timeStart = Date.now();
var timeEnd, countKey, countValue, amountLeft, itemLength;
var occupied = leftCount = 3; //Shurav's comment on initial overhead
//create localStorage entries until localStorage is totally filled and browser issues a warning.
var i = 0;
while (!error) {
try {
//length of the 'value' was picked to be a compromise between speed and accuracy,
// the longer the 'value' the quicker script and result less accurate. This one is around 2Kb
localStorage.setItem('testKey' + i, '11111111112222222222333333333344444444445555555555666661111111111222222222233333333334444444444555555555566666');
} catch (e) {
var error = e;
}
i++;
}
//if the warning was issued - localStorage is full.
if (error) {
//iterate through all keys and values to count their length
for (var i = 0; i < localStorage.length; i++) {
countKey = localStorage.key(i);
countValue = localStorage.getItem(localStorage.key(i));
itemLength = countKey.length + countValue.length;
//if the key is one of our 'test' keys count it separately
if (countKey.indexOf("testKey") !== -1) {
leftCount = leftCount + itemLength;
}
//count all keys and their values
occupied = occupied + itemLength;
}
;
//all keys + values lenght recalculated to Mb
occupied = (((occupied * 16) / (8 * 1024)) / 1024).toFixed(2);
//if there are any other keys then our 'testKeys' it will show how much localStorage is left
amountLeft = occupied - (((leftCount * 16) / (8 * 1024)) / 1024).toFixed(2);
//iterate through all localStorage keys and remove 'testKeys'
Object.keys(localStorage).forEach(function(key) {
if (key.indexOf("testKey") !== -1) {
localStorage.removeItem(key);
}
});
}
//calculate execution time
var timeEnd = Date.now();
var time = timeEnd - timeStart;
//create message
var message = 'Finished in: ' + time + 'ms \n total localStorage: ' + occupied + 'Mb \n localStorage left: ' + amountLeft + "Mb";
//put the message on the screen
document.getElementById('scene').innerText = message; //this works with Chrome,Safari, Opera, IE
//document.getElementById('scene').textContent = message; //Required for Firefox to show messages
}
And as promised above some test in different browsers:
GalaxyTab 10.1
Maxthon Pad 1.7 ~1130ms 5Mb
Firefox 20.0(Beta 20.0) crashed both
Chrome 25.0.1364.169 ~22250ms /5Mb
Native (identifies as Safari 4.0/Webkit534.30) ~995ms /5Mb
iPhone 4s iOS 6.1.3
Safari ~ 520ms /5Mb
As HomeApp ~525ms / 5Mb
iCab ~ 710ms /5mb
MacBook Pro OSX 1.8.3 (Core 2 Duo 2.66 8Gb memory)
Safari 6.0.3 ~105ms /5Mb
Chrome 26.0.1410.43 ~3400ms /5Mb
Firefox 20.0 300150ms(!) /10Mb (after complaining about script running to long)
iPad 3 iOS 6.1.3
Safari ~430ms /5Mb
iCab ~595ms /5mb
Windows 7 -64b (Core 2 Duo 2.93 6Gb memory)
Safari 5.1.7 ~80ms /5Mb
Chrome 26.0.1410.43 ~1220ms /5Mb
Firefox 20.0 228500ms(!) /10Mb (after complaining about script running to long)
IE9 ~17900ms /9.54Mb ( if any console.logs are in the code does not work until DevTools are opened)
Opera 12.15 ~4212ms /3.55Mb (this is when 5Mb is selected, but Opera asks nicely if we want increase the amount of lS, unfortunately it crashes if test conducted a few times in a row)
Win 8 (Under Parallels 8)
IE10 ~7850ms /9.54Mb
You can calculate your localstorage by following methods:
function sizeofAllStorage(){ // provide the size in bytes of the data currently stored
var size = 0;
for (i=0; i<=localStorage.length-1; i++)
{
key = localStorage.key(i);
size += lengthInUtf8Bytes(localStorage.getItem(key));
}
return size;
}
function lengthInUtf8Bytes(str) {
// Matches only the 10.. bytes that are non-initial characters in a multi-byte sequence.
var m = encodeURIComponent(str).match(/%[89ABab]/g);
return str.length + (m ? m.length : 0);
}
console.log(sizeofAllStorage());
Finally size in bytes will be logged in browser.
I would use the code of #tennisgen which get all and count the content, but I count the keys themselves:
var localStorageSpace = function(){
var allStrings = '';
for(var key in window.localStorage){
allStrings += key;
if(window.localStorage.hasOwnProperty(key)){
allStrings += window.localStorage[key];
}
}
return allStrings ? 3 + ((allStrings.length*16)/(8*1024)) + ' KB' : 'Empty (0 KB)';
};
The way I went about this problem is to create functions for finding out the used space and remaining space in Local Storage and then a function that calls those functions to determine the max storage space.
function getUsedSpaceOfLocalStorageInBytes() {
// Returns the total number of used space (in Bytes) of the Local Storage
var b = 0;
for (var key in window.localStorage) {
if (window.localStorage.hasOwnProperty(key)) {
b += key.length + localStorage.getItem(key).length;
}
}
return b;
}
function getUnusedSpaceOfLocalStorageInBytes() {
var maxByteSize = 10485760; // 10MB
var minByteSize = 0;
var tryByteSize = 0;
var testQuotaKey = 'testQuota';
var timeout = 20000;
var startTime = new Date().getTime();
var unusedSpace = 0;
do {
runtime = new Date().getTime() - startTime;
try {
tryByteSize = Math.floor((maxByteSize + minByteSize) / 2);
//localStorage.setItem(testQuotaKey, new Array(tryByteSize).join('1'));
//Recommended by #pkExec and #jrob007
localStorage.setItem(testQuotaKey, String('1').repeat(tryByteSize));
minByteSize = tryByteSize;
} catch (e) {
maxByteSize = tryByteSize - 1;
}
} while ((maxByteSize - minByteSize > 1) && runtime < timeout);
localStorage.removeItem(testQuotaKey);
if (runtime >= timeout) {
console.log("Unused space calculation may be off due to timeout.");
}
// Compensate for the byte size of the key that was used, then subtract 1 byte because the last value of the tryByteSize threw the exception
unusedSpace = tryByteSize + testQuotaKey.length - 1;
return unusedSpace;
}
function getLocalStorageQuotaInBytes() {
// Returns the total Bytes of Local Storage Space that the browser supports
var unused = getUnusedSpaceOfLocalStorageInBytes();
var used = getUsedSpaceOfLocalStorageInBytes();
var quota = unused + used;
return quota;
}
In addition to #serge's answer which is most voted here, size of the keys need to be considered. Code below will add the size of the keys stored in localStorage
var t = 0;
for (var x in localStorage) {
t += (x.length + localStorage[x].length) * 2;
}
console.log((t / 1024) + " KB");
As the spec goes, each character of a string is 16 bit.
But inspecting with chrome (Settings>Content Settings>Cookies & Site data) shows us that initiating localStorage takes 3kB (overhead size)
And stored data size follows this relation (accurate to 1kB)
3 + ((localStorage.x.length*16)/(8*1024)) kB
where localStorage.x is your storage string.
Yes, this question was asked like 10 years ago. But for those interested (like myself, as I am building an offline text editor that saves data with local storage) and suck at programming, you could use something simple like this:
var warning = 1;
var limit = 2000000; //2 million characters, not really taking in account to bytes but for tested number of characters stored
setInterval(function() {
localStorage["text"] = document.getElementById("editor").innerHTML; //gets text and saves it in local storage under "text"
if(localStorage["text"].length > limit && warning == 1){
alert("Local Storage capacity has been filled");
warning = 2; //prevent a stream of alerts
}
}, 1000);
//setInterval function saves and checks local storage
The best way to get the amount of storage filled is to view the site settings (say, if you stored an image in local storage). At least in chrome, you can see the amount of bytes used (ie: 1222 bytes). However, the best ways to see filled local storage with js have already been mentioned above, so use them.
//Memory occupy by both key and value so Updated Code.
var jsonarr=[];
var jobj=null;
for(x in sessionStorage) // Iterate through each session key
{
jobj={};
jobj[x]=sessionStorage.getItem(x); //because key will also occupy some memory
jsonarr.push(jobj);
jobj=null;
}
//https://developer.mozilla.org/en/docs/Web/JavaScript/Data_structures
//JavaScript's String type is used to represent textual data. It is a set of "elements" of 16-bit unsigned integer values.
var size=JSON.stringify(jsonarr).length*2; //16-bit that's why multiply by 2
var arr=["bytes","KB","MB","GB","TB"]; // Define Units
var sizeUnit=0;
while(size>1024){ // To get result in Proper Unit
sizeUnit++;
size/=1024;
}
alert(size.toFixed(2)+" "+arr[sizeUnit]);
window.localStorage.remainingSpace
I am creating an app using the Bespin editor and HTML5's localStorage. It stores all files locally and helps with grammar, uses JSLint and some other parsers for CSS and HTML to aid the user.
I want to calculate how much of the localStorage limit has been used and how much there actually is. Is this possible today? I was thinking for not to simply calculate the bits that are stored. But then again I'm not sure what more is there that I can't measure myself.
You may be able to get an approximate idea by using the JSON methods to turn the whole localStorage object to a JSON string:
JSON.stringify(localStorage).length
I don't know how byte-accurate it would be, especially with the few bytes of added markup if you're using additional objects - but I figure it's better than thinking you're only pushing 28K and instead doing 280K (or vice-versa).
I didn't find a universal way to get the remaining limit on the browsers I needed, but I did find out that when you do reach the limit there is an error message that pops up. This is of-course different in each browser.
To max it out I used this little script:
for (var i = 0, data = "m"; i < 40; i++) {
try {
localStorage.setItem("DATA", data);
data = data + data;
} catch(e) {
var storageSize = Math.round(JSON.stringify(localStorage).length / 1024);
console.log("LIMIT REACHED: (" + i + ") " + storageSize + "K");
console.log(e);
break;
}
}
localStorage.removeItem("DATA");
From that I got this information:
Google Chrome
DOMException:
code: 22
message: "Failed to execute 'setItem' on 'Storage': Setting the value of 'data' exceeded the quota."
name: "QuotaExceededError"
Mozilla Firefox
DOMException:
code: 1014
message: "Persistent storage maximum size reached"
name: "NS_ERROR_DOM_QUOTA_REACHED"
Safari
DOMException:
code: 22
message: "QuotaExceededError: DOM Exception 22"
name: "QuotaExceededError"
Internet Explorer, Edge (community)
DOMException:
code: 22
message: "QuotaExceededError"
name: "QuotaExceededError"
My solution
So far my solution is to add an extra call each time the user would save anything. And if the exception is caught then I would tell them that they are running out of storage capacity.
Edit: Delete the added data
I forgot to mention that for this to actually work you would need to delete the DATA item that was set originally. The change is reflected above by using the removeItem() function.
IE8 implements the remainingSpace property for this purpose:
alert(window.localStorage.remainingSpace); // should return 5000000 when empty
Unfortunately it seems that this is not available in the other browsers. However I am not sure if they implement something similar.
You can use the below line to accurately calculate this value and here is a jsfiddle for illustration of its use
alert(1024 * 1024 * 5 - escape(encodeURIComponent(JSON.stringify(localStorage))).length);
Ran into this today while testing (exceeding storage quota) and whipped up a solution. IMO, knowing what the limit is and where we are in relation is far less valuable than implementing a functional way to continue storing beyond the quota.
Thus, rather than trying to do size comparisons and capacity checks, lets react when we've hit the quota, reduce our current storage by a third, and resume storing. If said reduction fails, stop storing.
set: function( param, val ) {
try{
localStorage.setItem( param, typeof value == 'object' ? JSON.stringify(value) : value )
localStorage.setItem( 'lastStore', new Date().getTime() )
}
catch(e){
if( e.code === 22 ){
// we've hit our local storage limit! lets remove 1/3rd of the entries (hopefully chronologically)
// and try again... If we fail to remove entries, lets silently give up
console.log('Local storage capacity reached.')
var maxLength = localStorage.length
, reduceBy = ~~(maxLength / 3);
for( var i = 0; i < reduceBy; i++ ){
if( localStorage.key(0) ){
localStorage.removeItem( localStorage.key(0) );
}
else break;
}
if( localStorage.length < maxLength ){
console.log('Cache data reduced to fit new entries. (' + maxLength + ' => ' + localStorage.length + ')');
public.set( param, value );
}
else {
console.log('Could not reduce cache size. Removing session cache setting from this instance.');
public.set = function(){}
}
}
}
}
This function lives within a wrapper object, so public.set simply calls itself. Now we can add to storage and not worry what the quota is or how close we are too it. If a single store is exceeding 1/3rd the quota size is where this function will stop culling and quit storing, and at that point, you shouldn't be caching anyways, right?
To add to the browser test results:
Firefox
i=22.
Safari
Version 5.0.4 on my Mac didn't hang. Error as Chrome. i=21.
Opera
Tells the user that the website wants to store data but doesn't have enough space. The user can reject the request, up the limit to the amount required or to several other limits, or set it to unlimited. Go to opera:webstorage to say whether this message appears or not. i=20. Error thrown is same as Chrome.
IE9 standards mode
Error as Chrome. i=22.
IE9 in IE8 standards mode
Console message "Error: Not enough storage is available to complete this operation". i=22
IE9 in older modes
object error. i=22.
IE8
Don't have a copy to test, but local storage is supported (http://stackoverflow.com/questions/3452816/does-ie8-support-out-of-the-box-in-localstorage)
IE7 and below
Doesn't support local storage.
Wish I could add this in a comment - not enough rep, sorry.
I ran some perf tests - expecting JSON.stringify(localStorage).length to be an expensive op at large localStorage occupancy.
http://jsperf.com/occupied-localstorage-json-stringify-length
It is indeed so - about 50x more expensive than keeping track of what you're storing, and gets worse the fuller localStorage gets.
This function gets the exact storage available / left:
I made a suite of useful functions for localStorage *here*
http://jsfiddle.net/kzq6jgqa/3/
function getLeftStorageSize() {
var itemBackup = localStorage.getItem("");
var increase = true;
var data = "1";
var totalData = "";
var trytotalData = "";
while (true) {
try {
trytotalData = totalData + data;
localStorage.setItem("", trytotalData);
totalData = trytotalData;
if (increase) data += data;
} catch (e) {
if (data.length < 2) break;
increase = false;
data = data.substr(data.length / 2);
}
}
localStorage.setItem("", itemBackup);
return totalData.length;
}
// Examples
document.write("calculating..");
var storageLeft = getLeftStorageSize();
console.log(storageLeft);
document.write(storageLeft + "");
// to get the maximum possible *clear* the storage
localStorage.clear();
var storageMax = getLeftStorageSize();
Note, that this is not very quick, so don't use it all the time.
With this I also found out that: the Item-Name will take up as much space as its length, the Item-Value will also take up as much space as their length.
Maximum storage I got - all about 5M:
5000000 chars - Edge
5242880 chars - Chrome
5242880 chars - Firefox
5000000 chars - IE
You will find some out-commented code in the fiddle to see the progress in the console.
Took me some time to make, hope this helps ☺
You can test your browser with this web storage support test
I tested Firefox on both my android tablet and windows laptop and Chromium just on windows
results:
Firefox(windows):
localStorage: 5120k char
sessionStorage: 5120k char
globalStorage: *not supported
Firefox(android):
localStorage: 2560k char
sessionStorage: Unlimited (exactly test runs up to 10240k char == 20480k byte)
globalStorage: not supported
Chromium(windows):
localStorage: 5120k char
sessionStorage: 5120k char
globalStorage: not supported
###Update
On Google Chrome Version 52.0.2743.116 m (64-bit) limits where a little bit lower on 5101k characters. This means max available may change in versions.
I needed to actually simulate and test what my module will do when storage is full, so I needed to get a close precision on when the storage is full, rather than the accepted answer, which loses that precision at a rate of i^2.
Here's my script, which should always produce a precision of 10 on when memory cap is reached, and fairly quickly despite having some easy optimizations... EDIT: I made the script better and with an exact precision:
function fillStorage() {
var originalStr = "1010101010";
var unfold = function(str, times) {
for(var i = 0; i < times; i++)
str += str;
return str;
}
var fold = function(str, times) {
for(var i = 0; i < times; i++) {
var mid = str.length/2;
str = str.substr(0, mid);
}
return str;
}
var runningStr = originalStr;
localStorage.setItem("filler", runningStr);
while(true) {
try {
runningStr = unfold(runningStr, 1);
console.log("unfolded str: ", runningStr.length)
localStorage.setItem("filler", runningStr);
} catch (err) {
break;
}
}
runningStr = fold(runningStr, 1);
var linearFill = function (str1) {
localStorage.setItem("filler", localStorage.getItem("filler") + str1);
}
//keep linear filling until running string is no more...
while(true) {
try {
linearFill(runningStr)
} catch (err) {
runningStr = fold(runningStr, 1);
console.log("folded str: ", runningStr.length)
if(runningStr.length == 0)
break;
}
}
console.log("Final length: ", JSON.stringify(localStorage).length)
}
try {
var count = 100;
var message = "LocalStorageIsNOTFull";
for (var i = 0; i <= count; count + 250) {
message += message;
localStorage.setItem("stringData", message);
console.log(localStorage);
console.log(count);
}
}
catch (e) {
console.log("Local Storage is full, Please empty data");
// fires When localstorage gets full
// you can handle error here ot emply the local storage
}
This might help somebody. In chrome is possible to ask the user to allow to use more disk space if needed:
// Request Quota (only for File System API)
window.webkitStorageInfo.requestQuota(PERSISTENT, 1024*1024, function(grantedBytes) {
window.webkitRequestFileSystem(PERSISTENT, grantedBytes, onInitFs, errorHandler);
}, function(e) {
console.log('Error', e);
});
Visit https://developers.google.com/chrome/whitepapers/storage#asking_more for more info.
This script will return false if local storage is not available or return exact available space in local storage to the nearest character, regardless of if the browser throws an error or simply doesn't add anything to local storage when local storage is full.
var localstorageavail;
function localstoragetest(remaining) {
if (typeof(Storage) !== "undefined") {
localstorageavail = true;
var usedspace = JSON.stringify(localStorage).length;
if (remaining == true) {
var unusedspace = 0, data = "m", adddata, stored = 0;
for (adddata = "m";;) {
try {
localStorage.setItem("UN", data);
if (stored < JSON.stringify(localStorage).length) {
stored = JSON.stringify(localStorage).length;
adddata += adddata;
data += adddata;
}
else throw "toolong";
} catch(e) {
if (adddata == "m") break;
else adddata = "m";
data += adddata;
}
}
var totalspace = JSON.stringify(localStorage).length;
unusedspace = totalspace - usedspace;
localStorage.removeItem("UN");
alert("Space Used Calculated: " + usedspace + " " + "\nUnused space: " + unusedspace + "\nSpace Used according to browser: " + JSON.stringify(localStorage).length)
}
} else {
// alert("Sorry! No Web Storage support..");
localstorageavail = false;
}
if (localstorageavail == false) return localstorageavail;
else return unusedspace;
}
localstoragetest(true);
So I have ~12600 subnets:
eg. 123.123.208.0/20
and an IP.
I can use a SQLite Database or an array or whatever
There was a similar question asked about a month ago, however I am not looking for checking one IP against one subnet but a bunch of subnets (obviously the most efficient way, hopefully not O(total subnets)) :)
How can I check that the IP is one of in one of these subnets, I need true or false not the subnet if that helps optimisation.
There are similar subnets in the current list eg.:
(actual extract)
123.123.48.0/22 <-- not a typo
123.123.48.0/24 <-- not a typo
123.123.90.0/24
123.123.91.0/24
123.123.217.0/24
In total they range from 4.x.y.z to 222.x.y.z
The best approach is IMO making use of bitwise operators. For example, 123.123.48.0/22 represents (123<<24)+(123<<16)+(48<<8)+0 (=2071670784; this might be a negative number) as a 32 bit numeric IP address, and -1<<(32-22) = -1024 as a mask. With this, and likewise, your test IP address converted to a number, you can do:
(inputIP & testMask) == testIP
For example, 123.123.49.123 is in that range, as 2071671163 & -1024 is 2071670784
So, here are some tool functions:
function IPnumber(IPaddress) {
var ip = IPaddress.match(/^(\d+)\.(\d+)\.(\d+)\.(\d+)$/);
if(ip) {
return (+ip[1]<<24) + (+ip[2]<<16) + (+ip[3]<<8) + (+ip[4]);
}
// else ... ?
return null;
}
function IPmask(maskSize) {
return -1<<(32-maskSize)
}
test:
(IPnumber('123.123.49.123') & IPmask('22')) == IPnumber('123.123.48.0')
yields true.
In case your mask is in the format '255.255.252.0', then you can use the IPnumber function for the mask, too.
Try this:
var ip2long = function(ip){
var components;
if(components = ip.match(/^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/))
{
var iplong = 0;
var power = 1;
for(var i=4; i>=1; i-=1)
{
iplong += power * parseInt(components[i]);
power *= 256;
}
return iplong;
}
else return -1;
};
var inSubNet = function(ip, subnet)
{
var mask, base_ip, long_ip = ip2long(ip);
if( (mask = subnet.match(/^(.*?)\/(\d{1,2})$/)) && ((base_ip=ip2long(mask[1])) >= 0) )
{
var freedom = Math.pow(2, 32 - parseInt(mask[2]));
return (long_ip > base_ip) && (long_ip < base_ip + freedom - 1);
}
else return false;
};
Usage:
inSubNet('192.30.252.63', '192.30.252.0/22') => true
inSubNet('192.31.252.63', '192.30.252.0/22') => false
I managed to solve this by using the node netmask module.
You can check if an IP belongs to a subnet by making something like this:
import { Netmask } from 'netmask'
const block = new Netmask('123.123.208.0/20')
const ip = '123.123.208.0'
console.log(block.contains(ip))
Will here print true.
You can install it by using:
npm i --save netmask
Convert the lower ip and the upper ip in the range to integers and store the range in the db then make sure both columns are indexed.
Off the top of my head (pseudo code):
function ipmap(w,x,y,z) {
return 16777216*w + 65536*x + 256*y + z;
}
var masks = array[ipmap(128,0,0,0), ipmap(196,0,0,0), ..., ipmap(255,255,255,255)]
function lowrange(w, x, y, z, rangelength) {
return ipmap(w, x, y, z) & masks[rangelength]
}
function hirange(w, x, y, z, rangelength) {
return lowrange(w, x, y, z, ,rangelength) + ipmap(255,255,255,255) - masks[rangelength];
}
That ought to do it.
To find whether a particular ip falls in any of the ranges, convert it to an integer and do:
SELECT COUNT(*) FROM ipranges WHERE lowrange <= 1234567 AND 1234567 <= highrange
The query optimizer should be able to speed this up greatly.
Functions IPnumber and IPmask are nice, however I would rather test like:
(IPnumber('123.123.49.123') & IPmask('22')) == (IPnumber('123.123.48.0') & IPmask('22'))
Because for each address, you only need to take into account the network part of the address. Hence doing IPmask('22') will zero-out the computer part of the address and you should do the same with the network address.
Keywords: Binary searching, preprocessing, sorting
I had a similar problem and binary search appears to be very efficient if you can pre-process your subnet list and sort it. Then you can achieve an asymptotic time complexity of O(log n).
Here's my code (MIT License, original location: https://github.com/iBug/pac/blob/854289a674578d096f60241804f5893a3fa17523/code.js):
function belongsToSubnet(host, list) {
var ip = host.split(".").map(Number);
ip = 0x1000000 * ip[0] + 0x10000 * ip[1] + 0x100 * ip[2] + ip[3];
if (ip < list[0][0])
return false;
// Binary search
var x = 0, y = list.length, middle;
while (y - x > 1) {
middle = Math.floor((x + y) / 2);
if (list[middle][0] < ip)
x = middle;
else
y = middle;
}
// Match
var masked = ip & list[x][1];
return (masked ^ list[x][0]) == 0;
}
And an example usage:
function isLan(host) {
return belongsToSubnet(host, LAN);
}
var LAN = [
[0x0A000000, 0xFF000000], // 10.0.0.0/8
[0x64400000, 0xFFC00000], // 100.64.0.0/10
[0x7F000000, 0xFF000000], // 127.0.0.0/8
[0xA9FE0000, 0xFFFF0000], // 169.254.0.0/16
[0xAC100000, 0xFFF00000], // 172.16.0.0/12
[0xC0A80000, 0xFFFF0000] // 192.168.0.0/16
];
isLan("127.12.34.56"); // => true
isLan("8.8.8.8"); // => false (Google's Public DNS)
You can get a PAC script* and see how it performs (it loads a China IP list from somewhere else, sorts them and formats them appropriately) against 5000s of subnets. In practice its speed is surprisingly satisfactory.
The preprocessing code can be inspected using F12 Dev Tools on the above page. In short, you need to convert 1.2.3.4/16 to [0x01020304, 0xFFFF0000], i.e. 32-bit unsigned integer for the IP address and network mask.
* Link goes to my personal website.