I need some advice.
I'm looking for a way to work with a huge amount of data (800+mb) in the browser. This data should not be persistent. Users upload a csv file which they play with (filter) in the browser before doing anything with it. The columnnames in those csv files can vary wildly.
I was wondering what method to use here. Storing the data in memory in JavaScript variables does not work, as it crashes the page. I've been looking at indexedDB, which gives me the opportunity to keep the data in a database and even gives me filtering options. However this system is actually designed to store persistent data for offline use.
Preferably I would like to delete the database when loading a new file, create a new database with it's own objectstore and fill it with new data. But I found that the database is not deleted as long as I don't refresh the page and I cannot change it.
My questions: is using indexedDB a viable approach for this specific case? If yes, does anyone any pointers how to use it in this case? Are there other possibilities that I have overlooked?
Related
I have a 40-50MB JSON object that I need to persist across to a different page.
This only needs to happen once (one transition) but I'm still way over HTML5 LocalStorage limits, what other options do I have?
Unfortunately, that is too much data to store for most browsers. Even combining sessionStorage and localStorage both will not get us even close.
There are a few options you can try though:
You can store the data on your own server. This will depend on what web server/environment you are using.
You can use someone else's server to store the data. For example, you could use Google Drive's API. This does mean that your user needs a google account. You could also pay for a service like Amazon S3 to store it.
You could create a 'container' page, which loads and displays the pages, but keeps the session going. How exactly this works depends again on your environment.
40-50m is too huge for a browser, the worse part is if mobile is involved, what you can do is split the data into chunks, keep some in sessionStorage, localStorage and the remaining on your server, so that the part on the server will be fast enough to load, You will have to join them once all is loaded and done. I wouldn't recommend this method though.
I have a research project which is focusing on understanding user's curiosity and connecting to Free WIFI including remote areas where internet is poor here.
To do this, we developed simple webpages with the first as a Visitor's form where we need a user/visitor to enter their data and let it be retrieved, saved or viewed later as user logs. We want to do this by simply having a Router and a Flash disk where the webfiles and data storing system will be...No server!
Is this very possible with Javascript, xml and or any other languages anyone has ever done this? That is except Javascript's LocalStorage which in this say, the user will be the one to have his/her own data.
This is an unusual request really. I agree a relational DB would be the best choice. Of course you can just save to text file (xml or not) and later use that file in Excel or so. That would be my choice.
You can probably store data with LocalStorage, and allow user to save data by exporting a .JSON or .CSV or something like that.
You can give the user a file to download by doing a window.open() on a Data URI containing the text of the JSON or CSV:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs
I made a webpage (todo app) that uses locale storage for saving the data in json format. I use it on my smartphone (android) so all data is stored locally on my device (ca 200 kB)
Now, I would like to backup the data somehow. I have tried different approaches (email, post form, copy-paste) but none of them has worked so far. Is there an easy straigt foreward way to do this?
You don't have control over the localStorage to that extend. But there are several ways to store data across different storages (e.g. evercookie) even though I would not use that at all (not sure how legal that is, if at all)
But the best way would probably be to backup the data on your server and periodically let your users send their localStorage to you if there were changes.
If the localStorage got wiped, then just ask the server to send back the last stored set of data for that user.
If you are using Cordova/Phonegap you might want to look into http://docs.phonegap.com/en/1.4.1/phonegap_file_file.md.html#FileWriter to create a copy of the localStorage as physical file.
My question is about IndexedDB vs. WebSQL vs. SQLite. There is no need to explain that they are different, what I would like to know is:
Do those three "Database Solutions" allow for storing all its Data
to a file?
(and of course to do the reverse, initialize all its data given a backup file?)
.
Background
Since I already have done some research, which partly answers this question, allow me to provide this background info to the question:
SQLite
(yes it does allow storage and retrievel of the Database to and from a file)
I have already done some work with SQLite. For this I know that there it actually right away starts the database via a reference to the file. Backup is simple copying the file. Restoring is rewriting the file.
IndexedDB and WebSQL
??? Are to my understanding Database Solutions which "life their life in the Browser's Javascript land" and there we do not deal much with files. Here is part of where the question lies. If I wanted to export the data from either of the two solution to a flat file or lets say a one string variable representation, would that be possible?
This are some SO question I think that relate to it:
SO Question: Exporting WebSQL Data
SO Question: Import and Export Indexeddb data
which indicate that there is no easy toString() (Store Database) method and FromSting() in IndexedDB nor in WebSQL.
It indeed true (and affirmed in an answer here) and there is not easy backup and retrieval for those Database, this would be very sad, and I think a gap. Databases without backup function, really?!
At Present, there is no way to back up and restore Browser databases. The only way you can achieve this is by continuously syncing your back-end database with the browser database and thus keep track of changes in data generated on browser.
Old guy will do in old way, whether appropriate or not. How on earth, browser database need a backup. Data in the client is just cache copy of a slice of server data. There is no need back. If you think IndexedDB (or web sql database) data is durable, you will be glad to know, IndexedDB data belong to a temporary class of browser data, meaning that UA can delete data at their discretion without prompting to user or app.
If your app treat browser data more than cache copy, you are doing wrong.
Just starting to use Highcharts. If I include data in an array within the javascript the data is available for anyone to download when they view the source. This would be the same when data is called, say, from a csv file. Is there a way of protecting the data against copying/download?
No, since HighCharts is a client-side JavaScript library, data available to it is also potentially available to the end user. There really is no way to "secure" it once the data reaches the user's browser, although you can use HTTPS, server-side authentication, etc to at least guarantee in principle that only the intended user receives the data.
If you need to visualize your data while keeping the actual raw data secure, the obvious solution is to render the data on the server and just (in the end) serve up an image or other static content to the user. But then you lose the nice, interactive charts.
You might be able to use Flash or Silverlight to retrieve the data, to make part of the process harder to reverse engineer. This is not securing anything, just making it a bit harder for a determined user.
On the other hand, a user can see the data anyway in the final chart. If they really want to download the data they could painstakingly identify each data point and create their own CSV file, right? You need to figure out what is good enough for your particular use case, and strike the appropriate balance.
Being that HighCharts is a client-side JS system, I don't believe there is a way to get data to it securely. If you just attempt an AJAX call to get data at runtime, a user can see that call and the response. As you said you cannot just populate a variable in the source, as it is visible there.
Try the render charts on server feature:
http://www.highcharts.com/docs/export-module/render-charts-serverside