Develop Reference

d3/js data manipulation dropping columns

I have a dataset in csv and it looks like below.
country,col1,col2,col3
Germany,19979188,11233906,43.7719591
UK,3839766,1884423,50.92349378
France,1363608,796271,41.60557873
Italy,957516,557967,41.72765781
I'd like to drop col1, col2 off while keeping country and col3. If possible, I'd like to wrap it into a function where I can pass column list that I'd like to drop/keep.
Using pandas, which I'm familiar with, I can easily do it. e.g. data.drop(['col1', 'col2'], axis = 1). But I found d3 way or js way in general is based on each row so couldn't come up with an idea to drop columns.
I was thinking of d3.map() taking desirable columns only. But I was stuck to build a general function that the column list can be passed in.
Could anyone have thoughts?

D3 fetch methods, like d3.csv, will retrieve the whole CSV and will create an array of objects based on that CSV. Because of that, filtering out some columns is useless. Actually, it's worse than useless: you'll spend time and resources with an unnecessary operation.
Therefore, the only useful solution is, if you have access and own that CSV, creating a new CSV without those columns. That way you'll have a smaller file, faster to load. Otherwise, if you cannot change the CSV itself, don't bother: just load the whole thing and use the columns you want (which will be properties in the objects), ignoring the others.
Finally, if you have a lot of data manipulation it might be interesting reducing the size of the objects in the data array. If that's your case, use a row function to return only the properties you want. For instance:
d3.csv(url, function (d){
return {country: d.country, col3: d.col3}
}).then(etc...)

Heatmap DC.js - how to filter multiple items manually

I'm working on a heatmap chart using dc.js. During the page loads, i want to manually filter the heatmap.
For example, if it filter using this heatmap.filter("30,0"); it works when the page loads. but when i try to filter multiple values using the .filter() function, it didnt work.
I tried doing this just for testing.
var test = [];
test.push("30,0");
test.push("40,2");
heatmapChart.filter(test);
Though it works if its only one item, but if i add another array item, the chart breaks. Is there a specific way to filter multiple items manually?

Yes, but there are a few layers of weird between here and there.
First, the actual filter items are constructed using dc.filters.TwoDimensionalFilter - I think your comma separated strings are only working by accident.
Second, the chart has already overridden .filter() so if you want to get at the base version which can handle multiple items, you need to call chart._filter() instead.
Finally, and weirdest of all, the syntax for filtering on multiple items is to supply an array containing a single array of filter items.
Putting these all together,
var ff = [dc.filters.TwoDimensionalFilter([0,2008]),
dc.filters.TwoDimensionalFilter([3,1994]),
dc.filters.TwoDimensionalFilter([9,2000])];
heatmapChart._filter([ff]);
works with the heatmap filtering example. Note the array of array of filters!

How to update series data array in highcharts

I have the following snippet of code which updates my chart type from one type to another.
chart.series.forEach(function(serie){
serie.update({
type: type,
stacking: stacking,
})
})
I can pass different values in from type to stacking.
If I want my new chart to use a different array of values, the arrays of which are already defined and contain data from the database; how would I do it?
The reason I wish to do this is because I have two values, which are entered in their own arrays, say for example, 40 and 50 from a database. These are then stacked in a column chart (It's the only way I can stack).
I've created another array to store both these values in the one array when deciding to show it in a pie chart (not stacked).
I've tried adding 'data:arrayName' in as a property but this prevents my chart from rendering. Any idea guys?
Thank you.
EDIT: My problem probably only required one object to call. I don't have that much going on so the solution in the possible duplicate isn't really helpful to me. The duplicate code is trying to generate a new array on the fly using loops, I already have an array in place, I'm just figuring out how to update the array once the chart has changed to another type.

ext js store, manipulation of data (linq type queries)

I am working on my first Ext JS project and have my source data downloaded from a web server held in a store. Now I want to have various grids all show some variation of that source data. For example, grid 1 would be the source data grouped by 1 particular column. Grid 2 would be grouped by a particular column, and where column X = 'variable'. So on and so on.
I am coming from a c# background working in WPF. So normally I would use observable collections, and run LINQ on the observable collection to form a new observable collection, then bind that new collection to my grid.
My question is, how can I run LINQ type queries on stores in EXT JS?

You can run LINQ type queries on stores in ExtJS: You can easily get a Collection from a store using the query or queryBy method.
But you can't work like this for your grids, because grids can't run solely on collections, they need a store each. You wouldn't want to use LINQ type queries for these stores.
If you are using ExtJS6, you can look into ChainedStore. You can bind each grid to a ChainedStore which has your data store as the source. Each ChainedStore can be filtered, sorted and grouped independently from the source and from other stores, by using the sort(), group() and addFilter() functions on the respective store.
If you are using an earlier version, you have to use multiple stores and copy all data from the main store manually. Please be aware that store.add() takes an array of records, not a collection. Instead of store.add(collection), use store.add(collection.getRange()). The getRange() function will return an array that contains all items in the collection.
To answer your question from the comment:
what if I need to do something as simple as create a new store, from store1, group everything by column1, and sum column2?
In ExtJS6, if you want to show the grouping and the sum in a grid, that would be something along:
var store2 = Ext.create('Ext.data.ChainedStore',{
source:store1, // use the data from the other store.
grouper:{
property:'someTestField' // grouping is done inside this store.
}
});
Ext.create('Ext.grid.Panel',{
store:store2,
features:[{
ftype:'groupingsummary' // a summary row for each group is provided by this
}],
columns:[{
xtype:'numbercolumn'
dataIndex: 'someTestField'
summaryType: 'sum' // this column should have the sum in the summary row.
}]
})
Untested and without warranty. If you want to do it without the grid, just calculate the sum, that would have to be done manually like this:
var sums = {};
store1.getGroups().each(function(group,i) {
sums[i] = 0;
group.each(function(record) {
sums[i] += record.get(propertyName);
});
});

Most Efficient way of Filtering an Html Table?

I have an ajax function which call a servlet to get list of products from various webservices, the number of products can go up to 100,000. I need to show this list in a html table.
I am trying to provide users an interface to filter this list based on several criteria. Currently I am using a simple jQuery plugin to achieve this, but I found it to hog memory and time.
The Javascript that I use basically uses regex to search and filter rows matching the filtering criteria.
I was thinking of an alternate solution wherein I filter the JSON array returned by my servlet and bind the html table to it. Is there a way to achieve this, if there is, then is it more efficient than the regex approach.

Going through up to 100,000 items and checking if they meet your criteria is going to take a while, especially if the criteria might be complex (must be CONDO with 2 OR 3 bedrooms NOT in zip code 12345 and FIREPLACE but not JACUZZI).
Perhaps your servlet could cache the data for the 100,000 items and it could do the filtering, based on criteria posted by the user's browser. It could return, say, "items 1-50 of 12,456 selected from 100,000" and let the user page forward to the next 50 or so, and even select how many items to get back (25, 50, all).
If they select "all" before narrowing down the number very far, then a halfway observant user will expect it to take a while to load.
In other words, don't even TRY to manage the 100,000 items in the browser, let the server do it.

User enters filter and hits
search.
Ajax call to database, database has indexes on appropriate
columns and the database does the filtering.
Database returns result
Show result in table. (Probably want it to be paged to
only show 100-1000 rows at a time
because 100,000 rows in a table can
really slow down your browser.
Edit: Since you don't have a database, the best you're going to be able to do is run the regex over the JSON dataset and add results that match to the table. You'll want to save the JSON dataset in a variable in case they change the search. (I'm assuming that right now you're adding everything to the table and then using the jquery table plugin to filter it)

I'm assuming that by filtering you mean only displaying a subset of the data; and not sorting.
As you are populating the data into the table add classes to each row for everything in that row you want to filter by. e.g.:
<tr class="filter1 filter2 filter3">....
<tr class="filter1 filter3">....
<tr class="filter2">....
<tr class="filter3">....
Then when you want to apply a filter you can do something like:
$('TR:not(.filter1)').hide();

I agree with Berry that 100000 rows in the browser is bit of a stretch, but if there's anything that comes close to handling something of that magnitude then it's jOrder. http://github.com/danstocker/jorder
Create a jOrder table based on your JSON, and add the most necessary indexes. I mean the ones that you must at all cost filter by.
E.g. you have a "Name" field with people's names.
var table = jOrder(json)
.index('name', ['Name'], { sorted: true, ordered: true });
Then, for instance, this is how you select the records where the Name field starts with "John":
var filtered = table.where([{ Name: 'John' }], { mode: jOrder.startof, renumber: true });
Later, if you need paging in your table, just feed the table builder a filtered.slice(...).

If you're getting back xml, you could just use jQuery selection
$('.class', context) where context is your xml response.
From this selection, you could just write the xml to the page and use CSS to style it. That's where I'd start at first, at least. I'm doing something similar in one of my applications, but my dataset is smaller.

I don't know what you mean by "bind"? You can parse JSON and then use for loop (or $.each()) to populate ether straight HTML or by using grid plugin's insert/add