Develop Reference

Efficient/Performant way to visualise a lot of data in javascript + D3/mapbox

I am currently looking at an efficient way to visualise a lot of data in javascript. The data is geospatial and I have approximately 2 million data points.
Now I know that I cannot give that many datapoint to the browser directly otherwise it would just crash most of the time (or the response time will be very slow anyway).
I was thinking of having a javascript window communicating with a python which would do all the operations on the data and stream json data back to the javascript app.
My idea was to have the javascript window send in real time the bounding box of the map (lat and lng of north east and south west Point) so that the python script could go through all the entries before sending the json of only viewable objects.
I just did a very simple script that could do that which basically
Reads the whole CSV and store data in a list with lat, lng, and other attributes (2 or 3)
A naive implementation to check whether points are within the bounding box sent by the javascript.
Currently, going through all the datapoints takes approximately 15 seconds... Which is way too long, since I also have to then transform them into a geojson object before streaming them to my javascript application.
Now of course, I could first of all sort my points in ascending order of lat and lng so that the function checking if a point is within the javascript sent bounding box would be an order of magnitude faster. However, the processing time would still be too slow.
But even admitting that it is not, I still have the problem that at very low zoom levels, I would get too many points. Constraining the min_zoom_level is not really an option for me. So I was thinking that I should probably try and cluster data points.
My question is therefore do you think that this approach is the right one? If so, how does one compute the clusters... It seems to me that I would have to generate a lot of possible clusters (different zoom levels, different places on the map...) and I am not sure if this is an efficient and smart way to do that.
I would very much like to have your input on that, with possible adjustments or completely different solutions if you have some.
This is almost language agnostic, but I will tag as python since currently my server is running python script and I believe that python is quite efficient for large datasets.
Final note:
I know that it is possible to pre-compute tiles that I could just feed my javascript visualization but as I want to have interactive control over what is being displayed, this is not really an option for me.
Edit:
I know that, for instance, mapbox provides the clustering of data point to facilitate displaying something like a million data point.
However, I think (and this is related to an open question here
) while I can easily display clusters of points, I cannot possibly make a data-driven style for my cluster.
For instance, if we take the now famous example of ethnicity maps, if I use mapbox to cluster data points and a cluster is giving me 50 people per cluster, I cannot make the cluster the color of the most represented ethnicity in the sample of 50 people that it gathers.
Edit 2:
Also learned about supercluster, but I am quite unsure whether this tool could support multiple million data points without crashing either.

Compare sound between source and microphone in JavaScript

I'm working about audio but I'm a newbie in this area. I would like to matching sound from microphone to my source audio(just only 1 sound) like Coke Ads from Shazam. Example Video (0.45 minute) However, I want to make it on website by JavaScript. Thank you.

Building something similar to the backend of Shazam is not an easy task. We need to:
Acquire audio from the user's microphone (easy)
Compare it to the source and identify a match (hmm... how do... )
How can we perform each step?
Aquire Audio
This one is a definite no biggy. We can use the Web Audio API for this. You can google around for good tutorials on how to use it. This link provides some good fundametal knowledge that you may want to understand when using it.
Compare Samples to Audio Source File
Clearly this piece is going to be an algorithmic challenge in a project like this. There are probably various ways to approach this part, and not enough time to describe them all here, but one feasible technique (which happens to be what Shazam actually uses), and which is also described in greater detail here, is to create and compare against a sort of fingerprint for smaller pieces of your source material, which you can generate using FFT analysis.
This works as follows:
Look at small sections of a sample no more than a few seconds long (note that this is done using a sliding window, not discrete partitioning) at a time
Calculate the Fourier Transform of the audio selection. This decomposes our selection into many signals of different frequencies. We can analyze the frequency domain of our sample to draw useful conclusions about what we are hearing.
Create a fingerprint for the selection by identifying critical values in the FFT, such as peak frequencies or magnitudes
If you want to be able to match multiple samples like Shazam does, you should maintain a dictionary of fingerprints, but since you only need to match one source material, you can just maintain them in a list. Since your keys are going to be an array of numerical values, I propose that another possible data structure to quickly query your dataset would be a k-d tree. I don't think Shazam uses one, but the more I think about it, the closer their system seems to an n-dimensional nearest neighbor search, if you can keep the amount of critical points consistent. For now though, just keep it simple, use a list.
Now we have a database of fingerprints primed and ready for use. We need to compare them against our microphone input now.
Sample our microphone input in small segments with a sliding window, the same way we did our sources.
For each segment, calculate the fingerprint, and see if it matches close to any from storage. You can look for a partial match here and there are lots of tweaks and optimizations you could try.
This is going to be a noisy and inaccurate signal so don't expect every segment to get a match. If lots of them are getting a match (you will have to figure out what lots means experimentally), then assume you have one. If there are relatively few matches, then figure you don't.
Conclusions
This is not going to be an super easy project to do well. The amount of tuning and optimization required will prove to be a challenge. Some microphones are inaccurate, and most environments have other sounds, and all of that will mess with your results, but it's also probably not as bad as it sounds. I mean, this is a system that from the outside seems unapproachably complex, and we just broke it down into some relatively simple steps.
Also as a final note, you mention Javascript several times in your post, and you may notice that I mentioned it zero times up until now in my answer, and that's because language of implementation is not an important factor. This system is complex enough that the hardest pieces to the puzzle are going to be the ones you solve on paper, so you don't need to think in terms of "how can I do X in Y", just figure out an algorithm for X, and the Y should come naturally.

What is the difference between a ND-Buffer and a G-Buffer?

I'm noob at WebGL. I read in several posts of ND-Buffers and G-Buffers as if it were a strategic choice for WebGL development.
How are ND-Buffers and G-Buffers related to rendering pipelines? Are ND-Buffers used only in forward-rendering and G-Buffers only in deferred-rendering?
A JavaScript code example how to implement both would be useful for me to understand the difference.

G-Buffers are just a set of buffers generally used in deferred rendering.
Wikipedia gives a good example of the kind of data often found in a g-buffer
Diffuse color info
World space or screen space normals
Depth buffer / Z-Buffer
The combination of those 3 buffers is referred to as a "g-buffer"
Generating those 3 buffers from geometry and material data you can then run a shader to combine them to generate the final image.
What actually goes into a g-buffer is up to the particular engine/renderer. For example one of Unity3D's deferred renders contains diffuse color, occlusion, specular color, roughness, normal, depth, stencil, emission, lighting, lightmap, reflection probs.
An ND buffer just stands for "normal depth buffer" which makes it a subset of what's usually found in a typical g-buffer.
As for a sample that's arguably too big for SO but there's an article about deferred rendering in WebGL on MDN

Choosing a rendering path is a major architectural decision for a 3D renderer, no matter what API does it use. That choice's heavily depends upon the set of features the renderer has to support and it's performance requirements.
A substantial set of said features consists of so-called screen-space effects. If means that we render some crucial data about each pixel of the screen to a set of renderbuffers and then using that data (not the geometry) to compute some new data needed for a frame. Ambient Occlusion is a great example of such an effect. Based on some spacial values of pixels we compute a "mask" which we can later use to properly shade each pixel.
Moreover, there is a rendering pass which almost exclusively relies on screen-space computations. And it is indeed Deferred Shading. And that's where G-buffer come in. All data needed to compute colour of a pixel are rendered to a G-buffer: a set of renderbuffers storing that data. The data it self (and hence meanings of G-buffer's renderbuffers) can be different: diffuse component, specular component, shininess, normal, position, depth, etc. And as part of rendering of a frame contemporary deferred shading engines use screen-space ambient occlusion (SSAO), which use data from several G-buffer's renderbuffers (usually, they are position, normal and depth).
About ND-buffers. It seems to me that it's not a widely used term (Google failed to find any relevant info on them besides this question). I believe that ND stands for Normal-Depth. They're just a specific case of a G-buffer for a particular algorithm and effect (in the thesis it's SSAO).
So using G-buffers (and ND-buffers as a subset of G-buffers) and exadepends upon shading algorithms and effects you're implementing. But all screen-space computation will require some form of G-buffer.
P.S. The thesis you've link contains an inaccuracy. Author lists an ability to implement to ND-buffers on GLES 2.0 as an advantage to the method. However it's not actually possible since GLES 2.0 doesn't have depth textures (they've been added in OES_depth_texture extension).

I would like to add some more informations to previvous answers.
I read in several posts of ND-Buffers and G-Buffers as if it were a
strategic choice for WebGL development.
One of the most important part of deferred rendering is, if given platform supports MRT (multiple render targets). If it doesn't, you are not able to share partial calculations in shaders between each rendering and it also forces you to run rendnering as many times as you have "layers" (in case of unity 3D, it might be up to 11 times?). This could slow down your program a lot.
Read more in this question
Is deferred rendering/shading possible with OpenGL ES 2.0 ?
Webgl doesn't support MRT, but it has extension:
https://www.khronos.org/registry/webgl/extensions/WEBGL_draw_buffers/
Also there is an extension for depth textures:
https://www.khronos.org/registry/webgl/extensions/WEBGL_depth_texture/
So it should be possible to use deferred rendering technique, but its speed is hard to guess.

Pathfinding: How to create path data for the pathfiding algorithm?

I realize this is not strictly related to programming problems but as SO is the best resource for programming related problems, I decided to try it out. :)
I have a project where I need to do 3D pathfinding with javascript, inside a building. Dijkstra algorithm is probably the best case for this, as it handles irregular shapes quite nicely.
However, the problem is this:
Dijkstra requires node structure for it to work. But how to create that data? Obviously some sort of conversion need to be done from the basedata, but how to create that basedata? Going through the blueprint, getting x & y values for each possible path node, calculating the distances by hand seems bit excessive... And prone for swearwords...
I was even thinking of using Google Scetchup for this. Drawing lines for each possible path, but then the problem is getting the path data out from it. :/
I can't be the first person to have this problem... Any ideas? Are there any ready-made tools for creating path data?

Could not find any ready made tools so I ended up creating the path data as lines in Google SketchUp, exporting them Collada files and writing my own converter for the Collada XML data.

This can all be done in code by constructing a 3d grid and removing cubes that intersect with 3d objects.
I would then layer multiple 3d grids (doubling in size each time) that gives a more general idea of reachability (constructed from smaller grids), then by sheer virtue of path finding algorithms you will always find the most efficient path from A-B that will automatically direct the path using the largest cells (and therefore the fewest calculation steps). Note: make the larger 3d grids have a slightly lower weighting so that it's paths are favoured.
This can be used for many applications. For example if you could only walk on the ground, then simply remove blocks in unreachable areas.

Organizing objects in html5 webgl canvas animation with three.js

I want to draw a 3D cat(with animation) which is no more than a bunch of 3D objects - ellipsoids, pyramids, spheres, etc.
And I have 2 questions:
1) Are there any ways to define your own complex geometrical 3D objects rather than standard Three.js objects such as Sphere, Cube...
2) When animation the whole cat should I define an animation function for each object? Is there any way to combine some objects together?

For question one I'd recommend reading up on parameter driven modelling, this will allow you to make consistent complex objects without reinventing the wheel every time you create one. As for creating the custom objects, much like in the way polylines are are effectively a collection of lines with iterative implementations of the standard line methods (as well as object-specific methods) you'd create a javascript object which contains a collection of the objects necessary to create your custom shape. Here's a good webgl cheat sheet to help you out a bit.
Question two is somewhat similar to the way we've described complex objects above in that while you'll write a Cat object render / animate function, you'll handle the animation on a per object basis (with the exception full object static movement, imagine a cat on an escalator). Once again constraint, or parameter driven design will be your saviour here since the fact that two or more objects are partially superposed in no way means that the objects are explicitly linked.
As an end note I'd recommend looking into clojurescript. It might not be necessary for this type of work but lisp is very popular in the CAD scripting world and you'd definitely be doin' yourself a favour in the long run by at least familiarising yourself with the coding conventions - a lot of the questions you're goin' to have whilst working on this project will be answered in a variety of programming languages but you'll likely find that many of the answers that were written by folk working on both sides of the fence (cad/programming) will be written in lisp. Here's a final general CAD forum that's a great resource for all things CAD.