Develop Reference

d3/js data manipulation dropping columns

I have a dataset in csv and it looks like below.
country,col1,col2,col3
Germany,19979188,11233906,43.7719591
UK,3839766,1884423,50.92349378
France,1363608,796271,41.60557873
Italy,957516,557967,41.72765781
I'd like to drop col1, col2 off while keeping country and col3. If possible, I'd like to wrap it into a function where I can pass column list that I'd like to drop/keep.
Using pandas, which I'm familiar with, I can easily do it. e.g. data.drop(['col1', 'col2'], axis = 1). But I found d3 way or js way in general is based on each row so couldn't come up with an idea to drop columns.
I was thinking of d3.map() taking desirable columns only. But I was stuck to build a general function that the column list can be passed in.
Could anyone have thoughts?

D3 fetch methods, like d3.csv, will retrieve the whole CSV and will create an array of objects based on that CSV. Because of that, filtering out some columns is useless. Actually, it's worse than useless: you'll spend time and resources with an unnecessary operation.
Therefore, the only useful solution is, if you have access and own that CSV, creating a new CSV without those columns. That way you'll have a smaller file, faster to load. Otherwise, if you cannot change the CSV itself, don't bother: just load the whole thing and use the columns you want (which will be properties in the objects), ignoring the others.
Finally, if you have a lot of data manipulation it might be interesting reducing the size of the objects in the data array. If that's your case, use a row function to return only the properties you want. For instance:
d3.csv(url, function (d){
return {country: d.country, col3: d.col3}
}).then(etc...)

Google Scripts Simple Search

I want to search a sheet for a value in a sheet, and then return the row and the column where the value was found.
I am well-versed in VBA and have used the .Find function to accomplish this very easily. However, after searching for the last 30 minutes online, I have been stunned to discover that is is so hard to find the code for this extremely simple function. I feel like I am in the twilight zone. Does Javascript really not have anything analogous to .Find? If so, why is this language used so much when VBA appears to be able to accomplish the same tasks in a much more simple manner? Please advise.

I assume you are calling something like mysheet.getDataRange().getValues(), which returns the contents of a google sheet as an array of arrays, eg, [[row1A, row1B], [row2A, row2B]].
With JS, you can get the index of a value in an array using indexOf, which returns the index of the found item or -1 if the item is not in the array. I don't think you can directly search across the two nested arrays. Instead, you could try iterating over the outer array and searching the inner array. Something like this:
// get data from google sheet, doing something like this
var data = mysheet.getDataRange().getValues()
// define our own function
function findItem(data) {
// loop over outer array
for (var i=0; i < data.length; i++) {
var row = data[i]; // get the inner array
var idx = row.indexOf(searchValue); // search for value
// if the value is found, return [row, column]
if (idx > -1) {
return [i, idx];
}
}
// call the function
var res = findItem(data);
var row = res[0];
var col = res[1];

You're comparing apples and oranges. JavaScript and VBA are different languages with different goals. VBA was built to allow it to integrate seamlessly with the MSSQLServer. JavaScript in its native form has no relational-database functionality at all. It's meant more for manipulating web pages through the DOM. (It can do more than that, but that's its primary function.) While VBA can do some of the things Javascript can do, it's a rather clunky way (IMHO) of doing so and narrowly focused on a rather specific set of problems that are tied to very specific software and hardware infrastructures. While that functionality might be nice to have in some cases, most of the JavaScript you see on the web today isn't interested in databases at all.
It's not clear to me what source of data you're trying to connect to but if you are specifically looking for a JavaScript data solution you might want to look into something like MongoDB, and the code libraries that have been developed specifically for that. And there are tons of other JS libraries that are relational-data or database-specific, and you can search places like npm for those. Or you can combine JavaScript with languages that inherently include database functionality, and PHP is an excellent example of that.

Manipulating formatted numbers in a page

I'm working with numerical data I.e. adding, subtracting, present valuing, etc, numbers in a page. However, I format them and print them to the screen. Say I want to add a column of numbers, I have to parse for commas etc. Is there a paradigm to use the actual data, an not have to parse the DOM data? Or should I store both data in the page, but save the numbers as an attribute?

If I understand your question correctly, it sounds like you might be looking for Angular or some other form of two way data binding. Using that framework, you would be able to setup a template to reflect automatically to your "data model" (some javascript construct in memory) and have it update automatically to reflect changes in that model. You can also use "filters" when drawing it to the page to make the raw number display as currency.

create a template. and then pass variables to it, and it spits out html
--mustache (multi types, you want javascript)
--handlerbars.js
=================
MISC javascript code that might be worth while to you.
// would split things up on underscore _
// replace underscore with comma if wanted.
var something = your_something.split('_');
//some for loops
for( var i in something) {
var a = something[i];
}
for (var i = 0; i < something.length; i++) {
var a = something[i];
}
regex, .match, .replace
// are javascript string commands for....
// find(something) and replace with (something) doing.
// other words dealing with decimal points, dollar signs etc..
// and extracting numbers or like.
//dealing with objects and arrays and getting data from them.
something['name'] //= something.name = same thing in many situations.
something[i].name
something[i][x]
something.name.x
var a = '<table>';
a+= '<tr>';
a+= '<td>;
//a = <table><tr><td>
.innerhtml or .append
//used to add stuff to the dom / html stuff already in the something.html file.
json.parse() // may come up
there are way more better examples than i can do out there. but hopefully pickup some keywords that would reveal better internet searches for examples.
normally doing for me... google search "javascript .innerhtml" and open 4 to 10 results i get back and normally find enough to satisfy what i want.
jtables.com i want to say, or datatables.net i also want to say deal with sort spreadhsheet like columns and sorting data that the end user can do.
cookies, localstorage, indexeddb. localstorage most likely easier of the 3 with enough power for simple application and storing information.

dojo dgrid custom render cell contents

I'm using a dgrid but find the documentation available from the website (http://dojofoundation.org/packages/dgrid/) a bit lacking: it teaches you the basics but seems to leave you to work out how to do anything more complex by guesswork.
Specifically, I want to modify how the data I've loaded into a memory store is rendered. For example, I'd actually like to combine two columns from the store into one column in the grid with some additional text. Obviously one way would be to create a second memory store, iterate the first store and build the exact contents the dgrid should show in the second store. However, that feels clunky and like having to have two sets of the same data, just differently formated.
I can see there are renderRow and renderCell methods. Can anyone give me examples of how to use these, or point me to some documentation?
Thanks,
Simon

The renderCell function gives you the data object, so you can build the cell using any properties from the data.
var columns = [
{
label : ' ',
field : 'complexCell',
renderCell: function(object, value, node, options) {
domHtml.set(node, object[propA] + ' (' + object[propB] + ')');
}
},
...
];

Select/Order data from two tables

I have two tables holding game data for two different games. For the sake of simplicity, let's say they only share a column called Timestamp (in particular they have a different number of columns). I want to render a list holding information from both tables, simultaneously ordered by Timestamp.
What I'm currently doing works, but I'd take almost any bet that there is a much better way to do this. I'm mostly concerned about performance at some point (mobile app). This is a stub representing the structure – believe me, I know how horrible this looks right now. I just wanted to make it work first, now I'm looking for improvements. ;)
var readyA,
readyB = false;
var dataA,
dataB;
function doLoop () {
setTimeout(renderData, 100);
}
function renderData () {
if (!readyA || !readyB) {
doLoop();
return;
}
var dataAll = dataA.concat(dataB);
dataAll.sort(function (a,b) {
return a['Timestamp'] <= b['Timestamp'];
});
// pass data into a template depending on from which game it is and render it
// ...
}
// wait for both queries to finish
doLoop();
// select data from game A
myDatabaseClass.query('SELECT ... FROM GameA', function (results) {
dataA = new Array(results.rows.length);
for (var i=0; i<results.rows.length; i++) {
dataA[i] = results.rows.item(i);
}
readyA = true;
});
// select data from game B
myDatabaseClass.query('SELECT ... FROM GameB', function (results) {
dataB = new Array(results.rows.length);
for (var i=0; i<results.rows.length; i++) {
dataB[i] = results.rows.item(i);
}
readyB = true;
});
The question would now be if I can somehow simplify this by some kind of UNION or JOIN in the query. Obviously, the Timeout construction is horrible, but that will automatically collapse to a simple callback function if the querying can be done in one query (or at least one transaction – the database class can handle that).
Edit: I did found this ( Pull from two different tables and order ) but this whole NULL AS some_column feels dirty. Is there really no better alternative?

The result of a query always is a single table with a fixed number of columns, so all the SELECTs must have the same number of columns:
SELECT a1, a2, a3, Timestamp FROM GameA
UNION ALL
SELECT b1, b2, NULL, Timestamp FROM GameB
ORDER BY Timestamp
(UNION ALL is faster than UNION because it doesn't try to remove duplicates.)

Your code is pretty good. From the point of view of a SQL hacker like me you're doing the UNION and the ORDER BY on the client side. There's nothing wrong with that. You seem to be doing it almost right. Your "concat" is the client-side equivalent of UNION, and your "sort' is the equivalent of ORDER BY.
You say that the NULL as missing-column construction feels somehow dirty if you use server-side UNION operations. But, obviously to treat two different result sets as the same so you can sort them in order you have to make them conform to each other somehow. Your a['Timestamp'] <= b['Timestamp'] sort-ordering criterion in your sort function is also a scheme for conforming two result sets to each other. It may be lower-performance than using a UNION.
Don't be afraid of using NULL as missing-column to make two result sets in a UNION conform to each other. It's not dirty, and it's not expensive.
Do consider limiting your SELECT operation somehow, perhaps by a range of timestamps. That will allow your system to scale up, especially if you put an index on the column you use to limit the SELECT.
(By the way, your sort function has a mistake in it. sort functions need to return -1, 0, or +1 depending on whether the first item is less than, equal to, or greater than the second one. You're returning a true/false value. That doesn't work properly.)
(You're parallelizing the two queries to the same MySQL instance. That's clever, but probably in truth is a formula for overloading MySQL as your game scales up. Keep in mind that each user of your game has her own machine running Javascript but they all share your MySQL.)