Develop Reference

How can I or my app know that Chrome is discarding indexed db items?

I have an app which has recently started to randomly lose indexeddb items. By "lose" I mean they are confirmed as having been saved, but days later, are no longer present.
My hypothesis is that Chrome is discarding indexeddb items because the disk is full.
My question is specifically, are there any events that my app can listen for, or any Chrome log entries that I can refer to in order to confirm this is the case. NB. I'm not looking to fix the problem, I am only looking for ways in which I can detect it.

Each application can query how much data is stored or how much more
space is available for the app by calling queryUsageAndQuota() method
of Quota API.See
You can use periodic BackgroundSync
to estimate the used and free space allocated to temporary usage using:
// index.html
navigator.serviceWorker.ready.then(registration => {
registration.periodicSync.register('estimate-storage', {
// Minimum interval at which the sync may fire.
minInterval: 24 * 60 * 60 * 1000,
});
});
// service_worker.js
self.addEventListener('periodicsync', event => {
if (event.tag == 'estimate-storage') {
event.waitUntil(estimateAndNotify());
}
});
To estimate use the method: navigator.storage.estimate()//returns a Promise which resolves with {usage, quota} values in bytes.
When TEMPORARY storage quota is exceeded, all the data (incl.
AppCache, IndexedDB, WebSQL, File System API) stored for oldest used
origin gets deleted.
You can switch to unlimitedStorage or persistent storage.

Not sure how will it work in real life
This feature was initially still somewhat experimental.
but here https://developers.google.com/web/updates/2016/06/persistent-storage you can find that
This is still under development [...] the goal is to make users are aware of “persistent” data before clearing it [...] you can presume that “persistent” means your data won’t get cleared without the user being explicitly informed and directly in control of that deletion.
Which can answer you clarification in comments that you are looking for a way
how can a user or my app know for sure that this is happening
So, don't know about your app, but user maybe can get a notification. At least the page is of 2016 year. Something must have been done by now.

You may refer to Browser storage limits and eviction criteria article on MDN.
My hypothesis is that Chrome is discarding indexeddb items because the disk is full
According to this article your hypothesis seems to be true.
Now one of the ways to confirm this could be to save unique ids of items that are confirmed in a persistent storage - (this will have very small size compared to your actual data) and periodically compare whether all items are there in your indexeddb.
You might also want to refer to best practices.
and finally this comments
I think you need IndexedDB Observer. These links may help you if I got what you meant. Link-01 - Link-02
might be helpful.

How Service worker prevent 'fetch' from consuming too much disk space?

May I know what is the strategy that can be used to prevent addEventListener('fetch' && cache.put( from filling up user's disk space? Is there anyway to set expirary for the each request in cache?

According to the service worker spec:
The Cache objects are exactly what authors have to manage themselves.
The Cache objects do not get updated unless authors explicitly request
them to be. The Cache objects do not expire unless authors delete the
entries.
Note that the browser can throw your cache away if it wants to (and you should assume that sometimes it will), especially if it decides you're storing too much - it's not going to let you fill up the user's entire drive.
There do not appear do be any methods for examining the size of the cache either. However note that there are related specifications in the works, for example for the Quota Management API. The article by Jake Archibald linked above gives the following example of using the this API (which would give you a result for all site storage, not just your caches):
navigator.storageQuota.queryInfo("temporary").then(function(info) {
console.log(info.quota);
// Result: <quota in bytes>
console.log(info.usage);
// Result: <used data in bytes>
});
However I just tried this in Chrome Canary and storageQuota was undefined - I don't think this is implemented anywhere yet. Hopefully someone will correct me if I'm wrong.

Nodejs + Socket.IO - how to get total bytes transferred (read/write) after socket close

I have a simple socket.io client and server program, running on node.js. The client and server exchange messages between them for a few minutes, before disconnecting (like chat).
If there any function/method I can use to get the total bytes transferred (read/write), after the socket is closed?
At present I am adding up the message size for each each message sent and received by the client. But, as per my understanding, in socket.io depending on which protocol is used (websocket, xhr-polling, etc.), the size of the final payload being sent will differ due to the header/wrapper size. Hence, just adding message bytes won't give me an accurate measure of bytes transferred.
I can use monitoring tools like Wireshark to get this value, but I would prefer using a javascript utility to get this value. Search online, didn't give me any reasonable answer.
For pure websocket connections, I am being able to get this value using the functions: socket._socket.bytesRead and socket._socket.bytesWritten
Any help is appreciated!

As of socket v2.2.0 i managed to get byte data like this. Only problem these are specified when client closes browser window and reason parameter is transport error. If client uses socket.close() or .disconnect() or server uses .disconnect() then bytes are 0.
socket.on('disconnect', (reason) => {
let symbs = Object.getOwnPropertySymbols(socket.conn.transport.socket._socket);
let bytesRead = socket.conn.transport.socket._socket[symbs[3]];
let bytesWritten = socket.conn.transport.socket._socket[symbs[4]];
});

If you wanted such a feature that would work no matter what the underlying transport was below a socket.io connection, then this would have to be a fundamental feature of socket.io because only it knows the details of what it's doing with each transport and protocol.
But, socket.io does not have this feature built in for the various transports that it could use. I would conclude that if you're going to use the socket.io interface to abstract out the specific protocol and implementation on top of that protocol, then you give up the ability to know exactly how many bytes socket.io chooses to use in order to implement the connection on its chosen transport.
There are likely debug APIs (probably only available to browser extensions, not standard web pages) that can give you access to some of the page-wide info you see in the Chrome debugger so that might be an option to investigate. See the info for chrome.devtools.network if you want more info.

Publish data from browser app without writing my own server

I need users to be able to post data from a single page browser application (SPA) to me, but I can't put server-side code on the host.
Is there a web service that I can use for this? I looked at Amazon SQS (simple queue service) but I can't call their REST APIs from within the browser due to cross origin policy.
I favour ease of development over robustness right now, so even just receiving an email would be fine. I'm not sure that the site is even going to catch on. If it does, then I'll develop a server-side component and move hosts.

Not only there are Web Services, but nowadays there are robust systems that provide a way to server-side some logic on your applications. They are called BaaS or Backend as a Service providers, usually to provide some backbone to your front end applications.
Although they have multiple uses, I'm going to list the most common in my opinion:
For mobile applications - Instead of having to learn an API for each device you code to, you can use an standard platform to store logic and data for your application.
For prototyping - If you want to create a slick application, but you don't want to code all the backend logic for the data -less dealing with all the operations and system administration that represents-, through a BaaS provider you only need good Front End skills to code the simplest CRUD applications you can imagine. Some BaaS even allow you to bind some Reduce algorithms to calls your perform to their API.
For web applications - When PaaS (Platform as a Service) came to town to ease the job for Backend End developers in order to avoid the hassle of System Administration and Operations, it was just logic that the same was going to happen to the Backend. There are many clones that showcase the real power of this strategy.
All of this is amazing, but I have yet to mention any of them. I'm going to list the ones that I know the most and have actually used in projects. There are probably many, but as far as I know, this one have satisfied most of my news, whether it's any of the previously ones mentioned.
Parse.com
Parse's most outstanding features target mobile devices; however, nowadays Parse contains an incredible amount of API's that allows you to use it as full feature backend service for Javascript, Android and even Windows 8 applications (Windows 8 SDK was introduced a few months ago this year).
How does a Parse code looks in Javascript?
Parse works through classes and objects (ain't that beautiful?), so you first create a specific class (can be done through Javascript, REST or even the Data Browser manager) and then you add objects to specific classes.
First, add up Parse as a script tag in javascript:
<script type="text/javascript" src="http://www.parsecdn.com/js/parse-1.1.15.min.js"></script>
Then, through a given Application ID and a Javascript Key, initialize Parse.
Parse.initialize("APPLICATION_ID", "JAVASCRIPT_KEY");
From there, it's all object manipulation
var Person = Parse.Object.extend("Person"); //Person is a class *cof* uppercase *cof*
var personObject = new Person();
personObject.save({name: "John"}, {
success: function(object) {
console.log("The object with the data "+ JSON.stringify(object) + " was saved successfully.");
},
error: function(model, error) {
console.log("There was an error! The following model and error object were provided by the Server");
console.log(model);
console.log(error);
}
});
What about authentication and security?
Parse has a User based authentication system, which pretty much allows you to store a base of users that can manipulate the data. If map the data with User information, you can ensure that only a given user can manipulate specific data. Plus, in the settings of your Parse application, you can specify that no clients are allowed to create classes, to ensure innecesary calls are performed.
Did you REALLY used in a web application?
Yes, it was my tool of choice for a medium fidelity prototype.
Firebase.com
Firebase's main feature is the ability to provide Real Time to your application without all the hassle. You don't need a MeteorJS server in order to bring Push Notifications to your software. If you know Javascript, you are half way through to bring Real Time magic to your users.
How does a Firebase looks in Javascript?
Firebase works in a REST fashion, and I think they do an amazing job structuring the Glory of REST. As a good example, look at the following Resource structure in Firebase:
https://SampleChat.firebaseIO-demo.com/users/fred/name/first
You don't need to be a rocket scientist to know that you are retrieve the first name of the user "Fred", giving there's at least one -usually there should be a UUID instead of a name, but hey, it's an example, give me a break-.
In order to start using Firebase, as with Parse, add up their CDN Javascript
<script type='text/javascript' src='https://cdn.firebase.com/v0/firebase.js'></script>
Now, create a reference object that will allow you to consume the Firebase API
var myRootRef = new Firebase('https://myprojectname.firebaseIO-demo.com/');
From there, you can create a bunch of neat applications.
var USERS_LOCATION = 'https://SampleChat.firebaseIO-demo.com/users';
var userId = "Fred"; // Username
var usersRef = new Firebase(USERS_LOCATION);
usersRef.child(userId).once('value', function(snapshot) {
var exists = (snapshot.val() !== null);
if (exists) {
console.log("Username "+userId+" is part of our database");
} else {
console.log("We have no register of the username "+userId);
}
});
What about authentication and security?
You are in luck! Firebase released their Security API about two weeks ago! I have yet to explore it, but I'm sure it fills most of the gaps that allowed random people to use your reference to their own purpose.
Did you REALLY used in a web application?
Eeehm... ok, no. I used it in a Chrome Extension! It's still in process but it's going to be a Real Time chat inside a Chrome Extension. Ain't that cool? Fine. I find it cool. Anyway, you can browse more awesome examples for Firebase in their examples page.
What's the magic of these services? If you read your Dependency Injection and Mock Object Testing, at some point you can completely replace all of those services for your own through a REST Web Service provider.
Since these services were created to be used inside any application, they are CORS ready. As stated before, I have successfully used both of them from multiple domains without any issue (I'm even trying to use Firebase in a Chrome Extension, and I'm sure I will succeed soon).
Both Parse and Firebase have Data Browser managers, which means that you can see the data you are manipulating through a simple web browser. As a final disclaimer, I have no relationship with any of those services other than the face that James Taplin (Firebase Co-founder) was amazing enough to lend me some Beta access to Firebase.

You actually CAN use SQS from the browser, even without CORS, as long as you only need the browser to send messages, not receive them. Warning: this is a kludge that would make my CS professors cry.
When you perform a GET request via javascript, the browser will always perform the request, however, you'll only get access to the response if it was from the same origin (protocol, host, port). This is your ticket to ride, since messages can be posted to an SQS queue with just a GET, and who really cares about the response anyways?
Assuming you're using jquery, your queue is https://sqs.us-east-1.amazonaws.com/71717171/myqueue, and allows anyone to post a message, the following will post a message with the body "HITHERE" to the queue:
$.ajax({
url: 'https://sqs.us-east-1.amazonaws.com/71717171/myqueue' +
'?Action=SendMessage' +
'&Version=2012-11-05' +
'&MessageBody=HITHERE'
})
The'll be an error in the console saying that the request failed, but the message will show up in the queue anyways.

Have you considered JSONP? That is one way of calling cross-domain scripts from javascript without running into the same origin policy. You're going to have to set up some script somewhere to send you the data, though. Javascript just isn't up to the task.

Depending in what kind of data you want to send, and what you're going to do with it, one way of solving it would be to post the data to a Google Spreadsheet using Ajax. It's a bit tricky to accomplish though.Here is another stackoverflow question about it.
If presentation isn't that important you can just have an embedded Google Spreadsheet Form.

What about mailto:youremail#goeshere.com ? ihihi
Meantime, you can turn on some free hostings like Altervista or Heroku or somenthing else like them .. so you can connect to their server , if i remember these free services allows servers p2p, so you can create a sort of personal web services and push ajax requests as well, obviously their servers are slow for free accounts, but i think it's enought if you do not have so much users traffic, else you should turn on some better VPS or Hosting or Cloud solution.

Maybe CouchDB can provide what you're after. IrisCouch provides free CouchDB instances. Lock it down so that users can't view documents and have a sensible validation function and you've got yourself an easy RESTful place to stick your data in.

In node.js, how to declare a shared variable that can be initialized by master process and accessed by worker processes?

I want the following
During startup, the master process loads a large table from file and saves it into a shared variable. The table has 9 columns and 12 million rows, 432MB in size.
The worker processes run HTTP server, accepting real-time queries against the large table.
Here is my code, which obviously does not achieve my goal.
var my_shared_var;
var cluster = require('cluster');
var numCPUs = require('os').cpus().length;
if (cluster.isMaster) {
// Load a large table from file and save it into my_shared_var,
// hoping the worker processes can access to this shared variable,
// so that the worker processes do not need to reload the table from file.
// The loading typically takes 15 seconds.
my_shared_var = load('path_to_my_large_table');
// Fork worker processes
for (var i = 0; i < numCPUs; i++) {
cluster.fork();
}
} else {
// The following line of code actually outputs "undefined".
// It seems each process has its own copy of my_shared_var.
console.log(my_shared_var);
// Then perform query against my_shared_var.
// The query should be performed by worker processes,
// otherwise the master process will become bottleneck
var result = query(my_shared_var);
}
I have tried saving the large table into MongoDB so that each process can easily access to the data. But the table size is so huge that it takes MongoDB about 10 seconds to complete my query even with an index. This is too slow and not acceptable for my real-time application. I have also tried Redis, which holds data in memory. But Redis is a key-value store and my data is a table. I also wrote a C++ program to load the data into memory, and the query took less than 1 second, so I want to emulate this in node.js.

If I translate your question in a few words, you need to share data of MASTER entity with WORKER entity. It can be done very easily using events:
From Master to worker:
worker.send({json data}); // In Master part
process.on('message', yourCallbackFunc(jsonData)); // In Worker part
From Worker to Master:
process.send({json data}); // In Worker part
worker.on('message', yourCallbackFunc(jsonData)); // In Master part
I hope this way you can send and receive data bidirectionally. Please mark it as answer if you find it useful so that other users can also find the answer. Thanks

You are looking for shared memory, which node.js just does not support. You should look for alternatives, such as querying a database or using memcached.

In node.js fork works not like in C++. It's not copy current state of process, it's run new process. So, in this case variables isn't shared. Every line of code works for every process but master process have cluster.isMaster flag set to true. You need to load your data for every worker processes. Be careful if your data is really huge because every process will have its own copy. I think you need to query parts of data as soon as you need them or wait if you realy need it all in memory.

If read-only access is fine for your application, try out my own shared memory module. It uses mmap under the covers, so data is loaded as it's accessed and not all at once. The memory is shared among all processes on the machine. Using it is super easy:
const Shared = require('mmap-object')
const shared_object = new Shared.Open('table_file')
console.log(shared_object.property)
It gives you a regular object interface to a key-value store of strings or numbers. It's super fast in my applications.
There is also an experimental read-write version of the module available for testing.

You can use Redis.
Redis is an open source, BSD licensed, advanced key-value cache and store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs.
redis.io

This way works to "share a variable"; it is a bit more fancy than the way #Shivam did present. However, the module internally uses the same API. Therefore "shared memory" is a bit misleading as in cluster each process is a fork of the parent process. At fork time, process memory is duplicated in OS memory. Therefore there is no real shared memory except low-level shared memory like shm device or virtual shared memory page (Windows). I did implement a native module for Node.js which does make use of native shared memory (which is real shared memory) as using this technique both process read directly from a OS shared memory section. However, this solution doesn't really apply here well because it is limited to scalar values. You could of course JSON.stringify and share the JSON serialized data string, but the time it consumes to parse/stringify is totally non-ideal for most use cases. (Especially for larger objects parsing/stringifying of JSON with standard library implementations becomes non-linear).
Thus, this solutions seems the most promising for now:
const cluster = require('cluster');
require('cluster-shared-memory');
if (cluster.isMaster) {
for (let i = 0; i < 2; i++) {
cluster.fork();
}
} else {
const sharedMemoryController = require('cluster-shared-memory');
// Note: it must be a serializable object
const obj = {
name: 'Tom',
age: 10,
};
// Set an object
await sharedMemoryController.set('myObj', obj);
// Get an object
const myObj = await sharedMemoryController.get('myObj');
// Mutually exclusive access
await sharedMemoryController.mutex('myObj', async () => {
const newObj = await sharedMemoryController.get('myObj');
newObj.age = newObj.age + 1;
await sharedMemoryController.set('myObj', newObj);
});
}

This question was posted in 2012, exactly 10 years ago. Since no other answer has mentioned it, Node.js now supports Worker Threads that support shared memory.
Directly from the docs:
Workers (threads) are useful for performing CPU-intensive JavaScript operations.
Unlike child_process or cluster, worker_threads can share memory. They do so by transferring ArrayBuffer instances or sharing SharedArrayBuffer instances.