When data is loaded from a csv file, is it possible to get the order of the columns ?
E.g. the typical way to load a csv file is by calling the d3.csv function:
d3.csv("data.csv", function(error, data) {
var ageNames = d3.keys(data[0]).filter(function(key) { return key !== "State"; });
Typically, columns are stored as properties of an object, so the order cannot be retrieved anymore. Also, d3.keys() returns an array with a undefined order.
I am asking this because I want to sort the columns by the order in the csv file. Any help would be greatly appreciated.
You don't need to use the name of the column to access the data, you can use the index of the keys. The index is determined by the order in the file, so exactly what you're looking for.
To get the keys of an object, use Object.keys():
var keys = Object.keys(data[0]);
data[0][keys[0]]; // datum in the first row, first column
Coming from relational views, it seems surprising that the d3.js way of reading from a file has the opposite properties:
The relational view:
rows don't have any guaranteed order
columns have a well defined order and can be addressed by refering to their position
The d3.js view (of a csv file):
rows have a well defined order and can be addressed by refering to their row number. (they are stored in an array)
columns end up as a set of properties in an Object, thus they don't have a well defined order
If you want to access the order of columns as in the csv file, you can safely assume the "for...in.." loop over the properties or the d3.keys() returns the columns in the given order.
This order can just be destroyed if columns are added or removed. This is for most practical applications not the case.
See also https://javascriptweblog.wordpress.com/2011/01/04/exploring-javascript-for-in-loops/. The one exception described is: When column names are sole numbers, then the Chrome browser orders them according to their number order but their column order.
Related
In d3.csv("file.csv", function(error, data) instruction
d3.csv is a helper that uses d3.csv.parse internally to parse CSV
data loaded from a file into an array of objects, and then it passes this
to a callback.
So data is a callback variable that holds an array of objects
In the picture the structure of data
You can see that the headers of the original CSV have been used as the property names for the data objects.Using d3.csv in this manner requires that your CSV file has a header row.
data is array of objects
data (in orange) is a combination of :
- columns array (in red) holding csv headers that means **property names**
for the **data objects**
- an array of 6131 elements (in green) holding the values associated with
these propertie
Now that we have finished describing the desired structure:
Imagine I have an array of 6131 elements (the same as described in the
picture in green)
var dataArray =[];
for(var i=0;i<6131;i++){
ddd[i]={x:..,y:...etc};
dataArray.push(ddd[i]);
}
My question is how to construct in reverse the same identical structure described before and result with the same data like the one got from d3.csv.
var columnsArray=["NOM","PRENOM","SPECIALITE","Full_Address","VILLE","lat","lon"];
Thank you very much for help.
You can attach a columns property to your dataArray object:
dataArray.columns = columnsArray;
I have a dataset in csv and it looks like below.
country,col1,col2,col3
Germany,19979188,11233906,43.7719591
UK,3839766,1884423,50.92349378
France,1363608,796271,41.60557873
Italy,957516,557967,41.72765781
I'd like to drop col1, col2 off while keeping country and col3. If possible, I'd like to wrap it into a function where I can pass column list that I'd like to drop/keep.
Using pandas, which I'm familiar with, I can easily do it. e.g. data.drop(['col1', 'col2'], axis = 1). But I found d3 way or js way in general is based on each row so couldn't come up with an idea to drop columns.
I was thinking of d3.map() taking desirable columns only. But I was stuck to build a general function that the column list can be passed in.
Could anyone have thoughts?
D3 fetch methods, like d3.csv, will retrieve the whole CSV and will create an array of objects based on that CSV. Because of that, filtering out some columns is useless. Actually, it's worse than useless: you'll spend time and resources with an unnecessary operation.
Therefore, the only useful solution is, if you have access and own that CSV, creating a new CSV without those columns. That way you'll have a smaller file, faster to load. Otherwise, if you cannot change the CSV itself, don't bother: just load the whole thing and use the columns you want (which will be properties in the objects), ignoring the others.
Finally, if you have a lot of data manipulation it might be interesting reducing the size of the objects in the data array. If that's your case, use a row function to return only the properties you want. For instance:
d3.csv(url, function (d){
return {country: d.country, col3: d.col3}
}).then(etc...)
I have an array in my localStorage, product_categories, which holds numerous objects which contain a string and a nested array of objects (the inner objects are the products that belong to each category).
Since Appery's support couldn't help me figure out how to query an array of objects based on an attribute belonging to each object - I just made each object (product) belong to an array and categorize the products based on which array index they were.
Anyway, I am now trying to map an array of objects to my page's collapsible block, which I have done previously - but with a response from the online database.
Now, I am using a SEPARATE service and all I want it to do is grab the array of objects from the localStorage and map it to a collapsible block, or even a grid or list item, so that each index in the array auto-creates a new item.
However, it has not worked for anything I have tried. I tried to map it to every possible item that can have arrays mapped to it, and when I load the page - the item actually disappears, almost as if the array being mapped to it has a length of 0.
But, when I inspect the page in Chrome and look at the localStorage variable that is being used to hold the array of objects (and in-turn mapped to the page), the variable clearly has an array of objects in the same format as other localStorage variables being mapped to the page from storage.
If it helps, I am using a GenericService for pulling the localStorage variables and mapping them to the page. I am not using a custom implementation - all the service does is 'ON SUCCESS - MAPPING' and maps the storage to the page. However, like I said, this isn't working.
Since I have been here for a while but never actually posted or anything, I don't have the rep to post the images that may help in solving my problem; with that in mind, here is the link to the original Appery.io support page which contains corresponding images: https://getsatisfaction.com/apperyio/topics/mapping-localstorage-array-to-collapsible-wont-work-removes-collapsible-item
StackOverflow, I would really appreciate your assistance as I always seem to run into a language barrier when working with Appery.io's support.
*EDIT Your revisions are not useful for my post, as they are changing localStorage to local storage - but in the case of Appery, localStorage is correct syntax.
var products = offlineProductList(); // grab the returned JSON array of products from the function offlineProductList()
var UniqueCats = $.unique(products.map(function (d) {
return d.category // this will return every distinct category
}));
var product_categories = []; // create empty array to hold each category object and its respective products
for(var i=0;i<UniqueCats.length;i++){
// for every unique category...
var category_products = []; // create array to hold products
for(var j=0;j<products.length;j++){
// run through list of products
if(products[j].category == UniqueCats[i]){
// if the product's category is the same as the current indexed unique category, add to array
category_products.push(products[j]);
}
}
var category = {"category":UniqueCats[i],"category_products":category_products}; // create object for the category
product_categories.push(category); // add object to the categorical array
}
localStorage.setItem('product_categories', JSON.stringify(product_categories));
Above is the code used to create the 'array of objects', and below is an image showing the mapping
array of objects mapping to an appery page element
I would give you more but SO won't allow me to with my rep. So if you ask for more I might just 404 myself
So I figured it out myself - Appery actually offers the option for mapping expressions on events like 'Page Load' (for pages/panels etc) and 'Click' (for buttons and similar elements), and I was succesfully able to map localStorage arrays to list, collapsible, and grid elements from there. I guess my GenericService was set up inproperly - but for future users having issues of just mapping something from Storage to the page - just link a mapping expression from an element's event.
I am coming from MySQL and I am used to the conventional database table scheme. I am having trouble understanding IndexedDB and some of its terminology. I looked up these definitions in the documentation:
Key
A data value by which stored values are organized and retrieved in the object store.
indexName
The name of the index to create.
keyPath
The key path for the index to use.
Basically, Key is like a Primary Key in MySQL, right? Is indexName the same thing as a column? And I don't understand what a keyPath is.
Can someone explain these for me? Thank you again for you patience :).
Yes, key is like a primary key in SQL. But others seem to be missing an example explaining the main part of your question, and that is the distinction between indexName and keyPath. Per the Mozilla page on creating an index,
indexName
The name of the index to create. Note that it is possible to create an index with an empty name.
keyPath
The key path for the index to use. Note that it is possible to create an index with an empty keyPath, and also to pass in a sequence (array) as a keyPath.
The indexName is what you use to access that index. Indexes are used to search that "column" in the database.
The keyPath is the actual name of the "column." See other questions and answers for what forms a keyPath may take.
Note that "column" is not technically correct, but I'm using it because that's what you used.
For example, suppose your data has the column hours and you want to be able to search your database on that column. When creating your database you would create an index for that column:
objectStore.createIndex(indexName, keyPath, { unique: false });
where indexName can be anything you want, let's say hoursColumn, and keyPath would be hours.
objectStore.createIndex("hoursColumn", "hours", { unique: false });
unique: false just means that other rows of data may have the same value for hours.
I can write data to the objectStore as follows:
db.transaction(storeName, "readwrite").objectStore(storeName).add({hours: 20, minutes: 30});
So to search your data on the column hours, you could write:
var data = db.transaction(storeName).objectStore(storeName).index("hoursColumn").get(20)
and the result would be the first row of data where hours = 20, e.g. {hours: 20, minutes: 30}
So to summarize, indexName is just what you call the index you created that you want to search, and keyPath is the actual name of the stored data on which you want to search.
Indexes are a way to make it possible to query data in the indexeddb database. As you know objects are stored into the objectstores. These objectstore don't have a schema like you have in an normal SQL database.
An index exists out of 3 important properties:
indexName: The indexname is just a name you provide to the index. You will need this name if you want to use the index to query data.
keyPath: This defines which property of the object you want to address in your index. For example: you have an object
{ foo: "bar" }
and you want to query on the foo property, "foo" will be your keypath. The keypath can even go further. You can access nested properties
{ foo: { bar: "bla" } }
If you want to query the bar property the keypath will be "foo.bar"
key: The keys are the values inside the keypath. As you mentioned this key is unique for the index, but this doens't mean this value must be unique over all your objects in the objectstore.
The indexes in the indexeddb work like this:
When you create an index, it creates a new objectstore in which the object will be stored. Instead of storing these object bosed on primary key they are stored based on the values present in the keypath. This means that for a single key in an index you can have multiple objects. So if you start querying an index, it will filter on the keys and return the values that are present in these keys.
Hope this makes indexes clear for you.
I am updating my answer here having re-read the topic of Keys and Indexes. Coming from a MySQL background they are indeed confusing with IndexedDB.
(Please see https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Basic_Concepts_Behind_IndexedDB#gloss_outofline_key)
Indexes
Typically these occur on elements within your "row" - so for example if you had a row:
{Hello:"World",Foo:"Bar} in your database you could set an index (of any name) on say Hello. This would be like your indexes in MySQL.
Keys
So where Indexes can have multiple, Keys must (as the earlier answer mentions) be unique like MySQL Primary keys. Why this gets confusing is that you can also filter/search your data by Keys as well as Indexes. (see https://developer.mozilla.org/en-US/docs/Web/API/IDBKeyRange) The difference appears that Key searches appear to be more Ranges (like A-F or > 5 AND < 10) rather than specific values.
There is a very interesting explanation here about keys:
https://golb.hplar.ch/2017/09/A-closer-look-at-IndexedDB.html#primary-keys
Basically it explains that there are 4 types of keys within IndexedDB:
out-of-line : you provide the key yourself on each insert
out-of-line auto generated : primary key (i.e. AutoIncrement) number generated automatically
inline : the contents of field form the index (this is typical to say a MySQL database but must then be unique)
inline auto generated: hidden field (visible in console) added and is accessible through code but not related directly to the object data.
The difference between 2 and 4 is that 2 doesn't get a defined Name, whereas 4 (like a Primary Key in MySQL) does. You could then update (but not insert) on the ID field you've named in 4.
On an aside I believe its called a Keypath because you can call field.subproperty which is quite nice. Please also make sure your keys are spelt correctly! Obvious mistake.
It was also hard for me to understand the difference, but at the end I found out and why most of the time we set them the same name.
In short, the indexName is how we want to the "field" or "column" (index) will be named in our "table" (Object Store) and the keyPath is the property of the stored object that contains the value of the field. Therefore, is logical that they have the same name.
For example, you have a table with following fields:
name
age
They both are the indexName of each field, but you have to tell IndexedDB which property (keyPath) contains the value of each field.
objectStore.createIndex("name", "name", { unique: false });
objectStore.createIndex("age", "age", { unique: false });
Key = Row Item
indexName = Column name
keyPath = Column name
I have a little bit of an issue with a JavaScript function that needs to read data from a TextFile (something JS is already limited with) and then process tha TextFile data into a MultiDimensional Array (another thing that JS doesn't nativelly suport).
With that in mind, I have a text file in this format:
1, Name, Data, Serial
2, Name, Data, Serial
3, Name, Data, Serial
And so on.
So, the objective is to get that same data and put it, like that, into an array.
I suppose that, from what I've been reading, I need an Array of an Array, segmenting the first one by lines [/n] and the second one by commas [,]. However, given the "by-default" limitations, I'm very confused at this point. I do suppose I need jQuery, however.
I tried this:
var fs = require('fs');
var array = fs.readFileSync('file.txt').toString().split("\n");
for(i in array) {
var array = fs.readFileSync('file.txt').toString().split(",");
for(f in array) {
}
}
With little success, because then I don't really know how to store it, the objective being a Multidimensional Array that Replicates the Format of the text file, so latter it could be used to search by index or instance following an user input to get results.
I really appreciate any help.
At first glance it seems like you are trying to read in a CSV file. If that is indeed the case I recommend node-csv:
http://www.adaltas.com/projects/node-csv/
https://github.com/wdavidw/node-csv
This helped me reading file to JavaScript, however this example converts retrieved data to JSON. Just looking at the format of your text file, I would assume a JSON string or Javascript object would work with your data.
Example convert to JSON
With JSON and JS objects, instead of referencing a array indexes eg. array[i][x]. you would replace [x] with .propertyName
data = {
"id": 1,
"name": "Fred"
};
//access data like this
data[i].name //will return "Fred" as i =0
to create JS object, just initialize array properties without the ""(quotation marks). accessing JS and JSON properties are done in the same way, main advantage over a multidimensional array is that you can reference an actual property name, as opposed to indexes.