I would like to come straight to the point and show you my sample data, which is around the average of 180.000 lines from a .csv file, so a lot of lines. I am reading in the .csv with papaparse. Then I am saving the data as array of objects, which looks like this:
I just used this picture as you can also see all the properties my objects have or should have. The data is from Media Transperency Data, which is open source and shows the payments between institiutions.
The array of objects is saved by using the localforage technology, which is basically an IndexedDB or WebSQL with localstorage like API. So I save the data never on a sever! Only in the client!
The Question:
So my question is now, the user can add the sourceHash and/or targetHash attributes in a client interface. So for example assume the user loaded the "Energie Steiermark Kunden GmbH" object and now adds the sourceHash -- "company" to it. So basically a tag. This is already reflected in the client and shown, however I need to get this also in the localforage and therefore rewrite the initial array of objects. So I would need to search for every object in my huge 180.000 lines array that has the name "Energie Steiermark Kunden GmbH", as there can be multiple and set the property sourceHash to "company". Then save it again in the localforage.
The first question would be how to do this most efficient? I can get the data out of localforage by using the following method and set it respectively.
Get:
localforage.getItem('data').then((value) => {
...
});
Set:
localforage.setItem('data', dataObject);
However, the question is how do I do this most efficiently? I mean if the sourceNode only starts with "E" for example we don't need to search all sourceNode's. The same goes of course for the targetNode.
Thank you in advance!
UPDATE:
Thanks for the answeres already! And how would you do it the most efficient way in Javascript? I mean is it possible to do it in few lines. If we assume I have for example the current sourceHash "company" and want to assign it to every node starting with "Energie Steiermark Kunden GmbH" that appear across all timeNode's. It could be 20151, 20152, 20153, 20154 and so on...
Localforage is only a localStorage/sessionStorage-like wrapper over the actual storage engine, and so it only offers you the key-value capabilities of localStorage. In short, there's no more efficient way to do this for arbitrary queries.
This sounds more like a case for IndexedDB, as you can define search indexes over the data, for instance for sourceNodes, and do more efficient queries that way.
Related
I need to do a query where I can show only specific data using an 'AND' statement or equivalent to it. I have taken the example which is displayed in the Firebase Documentation.
// Find all dinosaurs whose height is exactly 25 meters.
var ref = firebase.database().ref("dinosaurs");
ref.orderByChild("height").equalTo(25).on("child_added", function(snapshot) {
console.log(snapshot.key);
});
I understand this line is going to retrieve all the dinosaurs whose height is exactly 25, BUT, I need to show all dinosaurs whose height is '25' AND name is 'Dino'. Is there any way to retrieve this information?
Thanks in advance.
Actually firebase only supports filtering/ordering with one propery, but if you want to filter with more than one property like you said I want to filter with age and name, you have to use composite keys.
There is a third party library called querybase which gives you some capabilities of multy property filtering. See https://github.com/davideast/Querybase
You cannot query by multiple keys.
If you need to sort by two properties your options are:
Create a hybrid key. In reference to your example, if you wanted to get all 'Dino' and height '25' then you would create a hybrid name_age key which could look something like Dino_25. This will allow you to query and search for items with exactly the same value but you lose the ability for ordering (i.e. age less than x).
Perform one query on Firebase and the other client side. You can query by name on Firebase and then iterate through the results and keep the results that match age 25.
Without knowing much about your schema I would advise you to make sure you're flattening your data sufficiently. Often I have found that many multi-level queries can be solved by looking at how I'm storing the data. This is not always the case and sometimes you may just have to take one of the routes I have mentioned above.
I have a firebase model where each object looks like this:
done: boolean
|
tags: array
|
text: string
Each object's tag array can contain any number of strings.
How do I obtain all objects with a matching tag? For example, find all objects where the tag contains "email".
Many of the more common search scenarios, such as searching by attribute (as your tag array would contain) will be baked into Firebase as the API continues to expand.
In the mean time, it's certainly possible to grow your own. One approach, based on your question, would be to simply "index" the list of tags with a list of records that match:
/tags/$tag/record_ids...
Then to search for records containing a given tag, you just do a quick query against the tags list:
new Firebase('URL/tags/'+tagName).once('value', function(snap) {
var listOfRecordIds = snap.val();
});
This is a pretty common NoSQL mantra--put more effort into the initial write to make reads easy later. It's also a common denormalization approach (and one most SQL database use internally, on a much more sophisticated level).
Also see the post Frank mentioned as that will help you expand into more advanced search topics.
I have a multidimensional array that is something like this
[0]string
[1]-->[0]string,[1]string,[2]string
[2]string
[3]string
[4]-->[0]string,[1]string,[2]string[3]string,[4]string,[5]INFO
(I hope that makes sense)
where [1] and [4] are themselves arrays which I could access INFO like myArray[4][5].
The length of the nested arrays ([1] and [4]) can varry.
I use this method to store, calculate, and distribute data across a pretty complicated form.
Not all the data thats storred in the array makes it to an input field so its not all sent to the next page when the form's post method is called.
I would like to access the array the same way on the next page as I do on the first.
Thoughts:
Method 1:
I figure I could load all the data into hidden fields, post everything, then get those values on the second page and load themm all back into an array but that would require over a hundred hidden fields.
Method 2:
I suppose I could also use .join() to concatenate the whole array into one string, load that into one input, post it , and use .split(",") to break it back up. But If I do that im not sure how to handel the multidimensional asspect of it so that I still would be able to access INFO like myArray[4][5] on page 2.
I will be accessing the arrary with Javascript, the values that DO make it to inputs on page 1 will be accessed using php on page 2.
My question is is there a better way to acomplish what I need or how can I set up the Method 2 metioned above?
This solved my problem:
var str = JSON.stringify(fullInfoArray);
sessionStorage.fullInfoArray = str;
var newArr = JSON.parse(sessionStorage.fullInfoArray);
alert(newArr[0][2][1]);
If possible, you can use sessionStorage to store the string representation of your objects using JSON.stringify():
// store value
sessionStorage.setItem('myvalue', JSON.stringify(myObject));
// retrieve value
var myObject = JSON.parse(sessionStorage.getItem('myvalue'));
Note that sessionStorage has an upper limit to how much can be stored; I believe it's about 2.5MB, so you shouldn't hit it easily.
Keep the data in your PHP Session and whenever you submit forms update the data in session.
Every page you generate can be generated using this data.
OR
If uou are using a modern browser, make use of HTML5 localStorage.
OR
You can do continue with what you are doing :)
In an app, made with TideSDK; i assign a global variable (shocking I know) to a the JSON parse of a string stored in Titanium.App.Properties:
var workbookArray = JSON.parse(Titanium.App.Properties.getString('workbookArray'));
workbookArray is an array of objects.
And then on the unloading of a page, I assign Titanium.App.Properties string the value of workbookArray, which may have been changed by whoever has used the app:
Titanium.App.Properties.setString('workbookArray', JSON.stringify(workbookArray));
Each time I open the app, however, I'm told that JSON was unable to parse the first code snippet (initializing workbookArray).
Aside from this issue, I don't expect to use the app Properties API for my storage needs in the longterm, I wish i could use indexedDB with titanium. SQL is an option, but is a little messy when it comes to objects. Any other suggestions for a database solution?
Try getList and setList
http://docs.appcelerator.com/titanium/latest/#!/api/Titanium.App.Properties
What is stored in the list?
My Google Chrome extension uses JSON.parse(oldJSONstring) to create an object with configuration information. The "oldJSONstring" was saved from previous sessions on Chrome's localStorage.
As I sometimes add new features, after I create the new object, I manually validate that all configuration entries exist, if not, I'll set them with default values. This is done, in case it's he firs time a user loads the extension after it updated.
I was trying to think of a more automatic way of doing this, like using a JSON Schmea, but I really don't know where to start, and my first round of google searches didn't produce anything I could use.
Another approach I thought was to iterate on all my Default settings -also stored on a JSON object- and then confirming they exist on the new object... but I just realized I don't know how to iterate a JSON object for all its attributes :)
The end goal of this is that I'd like to forget about validating for new attributes, every time I create a new feature and I publish a new version... does it make any sense? does it make me lazy? :D
Thanks!
Keep the default object handy and use jQuery $.extend to merge the 2 objects.
var defaults={ color:'blue', size:'Large'}
var chromeObj= /* code to grab from storage*/
/* update chromeObj with all key/value pairs in defaults */
/* regardless if they already exist or not*/
$.extend( chromeObj, defaults}
/* code to put chromeObj back to storage*/
Refrence: http://api.jquery.com/jQuery.extend/
There is no such thing as a "JSON object,"* but it's quite easy to loop over a Javascript object's properties: How do I loop through or enumerate a JavaScript object?
* JSON is just a string format, and it's a subset of Javascript object notation