I have a list of all articles in NY Times from its beginning and want have an instant access to all of them without connecting to external database, so my solution is holding it in one variable. But isn't that a bad practice in terms of efficiency?
I doubt that the memory usage is any problem - modern browser games for example probably use an order of magnitude more memory.
I would worry more about the data structure and the operations you intend to run on it.
What kind of application are you writing?
It would properbly be a better idea to create a local database and store the articles there and then create a sync task that pulls only the articles that are of a later date than the previous sync task and save them to the local db.
It all depends on what kind of application you are writing of course
I think you need to look into using HTML5 Local Storage to store this variable or it's contents. This way you're keeping things client-side as you wish whilst also storing the data properly.
Related
I'm building a nodejs app that needs access to some data. I am not sure what is the best way to store the data. If it is json or mongodb or a sql database considering the performance of the read operation.
The app will never update/ insert/ delete any of the data. That's why I wrote it is static. And the amount of data could be a total of at most a few hundreds small objects.
What is your opinion on that? Really considering the max performance of the read operation.
Since it is 'static' data and that too only a few hundreds small objects, I'd recommend that you go ahead with JSON. SQL should be preferred when operations such as data manipulation, concurrent sessions etc. are involved.
This is not opinion based.
The answer is a flat file.
Reasoning: When leveraging a database, there are defined use cases. triggers, inserts, deletes, updates, etc. All of this is managed by a database language of your choosing.
If you are not leveraging any key aspects of a database, then why do you need the overhead of it.
The best way to approach this situation would be to consolidate the access to a class you create called: StaticService or whatever fits your fancy. In this class you will read in the data and store it in memory as a property. Then have various methods in that service which will get you the data you request.
Even with a Database, you would still implement this kind of service worker, but you dont have this overhead. You can also optimize it as you see fit, but it sounds like you may be looking to display lists, or specific values which are generally o(1) access if the json is designed correctly.
I am trying to fix an old website error and it crashes because it runs out of memory. The reason it runs out of memory is that we are storing the data locally then displaying it. I want to make these statements use less memory. How do I do that?
localStorage.setItem("levels", JSON.stringify(__levels));
localStorage.setItem("rooms", JSON.stringify(__rooms));
var proj = JSON.parse(localStorage.getItem("levels"));
var rooms = JSON.parse(localStorage.getItem("rooms"));
This is what we call "the code pushing back". If you're hitting against memory issues or browser-storage limits, you're probably dealing with too much data for it to make sense for you to store in client-based storage, and the system requirements are pushing back against what you think your requirements are. You could look at other ways to have smaller serialized data (if it's storage limits); you could look at faster and more efficient serialization / deserialization (if it's memory limits). It sounds, however, like you should either be rethinking your decision to store it on the client in the way that you are or reconsidering the amount of data and trying to limit it to a subset of required data.
I have a large 40,000 words array loading from a database into a JavaScript/HTML array on every page of our web application... What would be the best way/technology to optimize it? In order to avoid this unnecessary downloads.
Somehow keep the array in a cookie and read from there?
Use ajax to load the array dynamically only parts that are needed?
What is the common practice?
On modern browsers you can use sessionStorage to have it persist during the current session, or localStorage to have it hang around between sessions.
NB: both only permit storage of strings - you'll have to serialise the array (e.g. into JSON) and deserialise it on retrieval.
If you want to actually use the word list as a local database with efficient lookup you might also want to investigate indexedDB
you can place the data in session and retrieve it, the same can be used in every page with out fetching the same every time.
Thanks & best regards.
If you need all the 40k words in all pages then you can use localStorage or sessionStorage. Just keep in mind sessionStorage will delete saved data when the tab/window is closed so the whole array will be downloaded again when the website is opened in new windows/tabs.
If you need only specific parts of the array in different pages I would tidy the array's elements into taxonomy/categories (if you are able to), so that you can download only the needed for a specific section of your application.
This depends on the composition of your array, if it is formed only by words or complex objects. This will help to avoid slow load of your website when it's visited the first time.
If the array is always the same (there is no need to update it), I'd create a js file and then I'd add it to every html page. The browser's cache would do the rest to avoid unnecessary re-loading. Something like:
big-array.js file:
var myBigArray=[...]
In each html file
<html>
... whatever you need
<script src="/my-path/big-array.js"></script>
...my other scripts here
</html>
It's a bit difficult to answer this question properly as to do so would require more information about your hosting environment and what you have access to. If you have a server side language available, such as PHP, you could look at caching which is generally the most efficient way to handle data that is used repeatedly across pages. Perhaps you could post more info about what technologies you have available to you?
Currently I'm experimenting with localStorage to store a large amount of objects of same type, and I am getting a bit confused.
One way of thinking is to store all the object in an array. But then for each read/write of a single object I need to deserialise/serialise the whole array.
The other way is to directly store each object with its key in the localStorage. This will make accessing each object much easier but I'm worried of the amount of objects that will be stored (tens of thousands). Also, getting all the objects will require iterating the whole localStorage!
I'm wondering which way will be better in your experience? Also, would it be worthwhile to try on more sophisticated client side database like PouchDB?
If you want something simple for storing a large amount of key/values, and you don't want to have to worry about the types, then I recommend LocalForage. You can store strings, numbers, arrays, objects, Blobs, whatever you want. It uses IndexedDB and WebSQL where available, so the storage limits are much higher than LocalStorage.
PouchDB works too, but the API is more complex, and it's better-suited for when you want to sync data with CouchDB on the server.
If you do not want to have a lot of keys, you can:
concat row JSONs with \n and store them as a single key
build and update an index(es) stored under separate keys, each linking some key with a particular row number.
In this case parsing rows is just .split('\n') that is ~2 orders of magnitude faster, then JSON.parse.
Please, notice, that you possibly need special effort to syncronize simultaneously opened tabs. It can be a challenge in complex cases.
localStorage has both good and bad parts.
Good parts:
syncronous;
extremely fast, both read and write are just memcpy – it‘s 100+Mb/s throughput even on weak devices (for example JSON.stringify is in general 5-20 times slower than localStorage.setItem);
thoroughly tested and reliable.
Bad news:
no transactions, so you need an engineering effort to sync tabs;
think you have not more than 2Mb (cause there exist systems with this limit);
2Mb of storage actually mean 1M chars you can save.
These points show borders of localStorage applicability as a DB. LS is good for tasks, where you need syncronicity and speed, and where you can trim you DB to fit into quota.
So localStorage is good for caches and logs. Not more.
I hadn't personally used localStorage to manage so many elements.
However, the pattern I usually use to manage data is to load the complete info database into a javascript object, manage it on memory during the proccess and saving it again to localStorage when the proccess is finished.
Of course, this pattern may not be a good approach to your needings, depending on your project specifications.
If you need to save data constantly, data access could become a problem, and thus probably using some type of small database access is a better option.
If your data volume is exceptionally high it also could be a problem to manage it on memory, however, depending on data model, you'd be able to build it to efficient structures that would allow you to load and save data just when it's needed.
I've got a web app which gets a couple dozen items at boot. All these items are JSON and are smaller then 1kb.
Now there are a number of storage options as seen in the Question.
I was thinking of just storing these objects inside a variable in the browser JS. I don't really see why I would want to use any of these browser storages?
So what would be reasons to use any of the browser-based storage instead of a variable inside JS.
Could be that from a certain data size it is preferable to use browser storage, e.g. from 100kb onwards it's better to not use a JS variable.
var myModel = {}
NOTE
Every time the user enters the app he will get fresh content from the server. The content is too realtime for caching.
`
localStorage , globalStorage and sessionStorage:
These features are ready in browsers that have implemented the "Web Storage", they all refer to a kind of HashMap, a map between string keys and string values. but the life is different. once the active page is closed sessionStorage would be cleaned but the localStorage is permanent.(MDN DOM Storage guide)
There is a point about globalStorage, which is its being obsolete since Gecko 1.9.1 (Firefox 3.5) and unsupported since Gecko 13 (Firefox 13), since then we should use localStorage. the difference between these 2 was just the HTML5 scope support(scheme + hostname + non-standard port).
These could be useful for you to:
-Share your objects between your different pages, in your site.
-Offline programming.
-Caching large object
-Or whenever you need to a local persistent storage.
IndexedDB:
IndexedDB is useful for applications that store a large amount of data (for example, a catalog of DVDs in a lending library) and applications that don't need persistent internet connectivity to work (for example, mail clients, to-do lists, and notepads)
based on this quote from MDN you can easily find your answer out, regarding using IndexedDB, if you don't know whether IndexedDB is useful for you or not, just answer these questions:
Do you store a large amount of data on client? if yes, so consider using it.
Does your app need to be offline enabled? if yes, so consider using IndexedDB.
Does your app need to a persistent internet connectivity? If yes, it stays still an option, based on the other factors.
So other than working offline as far as you don't need it, I guess, because as you said:
The content is too realtime for caching.
These have some features like sharing objects, and managing large amount of data, which you should be the one to decide.
localStorage and sessionStorage are solving a caching problem; think of them as cookies. You've said you don't want caching, so you can ignore them.
JavaScript objects behave basically like O(1) lookup tables (see How is a JavaScript hash map implemented?, and make sure you read both the top two answers, as both have something useful to say), and there is no maximum memory limit that I am aware of, or a point where another solution becomes a better choice
The only reason I can think of that you should bother with the extra step of inserting the data in an IndexedDB is if you need O(1) lookups on a field that is not the object key you are using.