Develop Reference

Summarise many nodes in Cytoscape.js

I am building a web app where the user can create a sort of bipartite graph like this one:
Sometimes the user would like to build a huge graph, for example a graph in which the topmost layer has 784 nodes, as in the next picture. My application can handle the computation, but the result is ugly and meaningless:
Do you have any idea for rendering a huge layer without just drawing all the nodes but, instead, summarising them with another, prettier representation?
Until now I have thought about putting all the nodes of a "huge" layer in an empty compound node, but of course then it is not possible to draw some edges (so the huge layer would seem as it were disconnected from the graph). Another solution would be to have all layers with more than 100 nodes have exactly 100 nodes, and put them inside a compound node with z-index greater than each node's z-index; but I haven't tried this yet.
If you have some other ideas, or if Cytoscape.js provides a way to summarise large graphs, please let me know.
Thank you.

you could group nodes when the amount in a layer exceeds a certain number (e.g. if you have 100 nodes, you combine them into groups of 25)
I'd do this by iterating over the nodes in the layer, making a new node N for each subgroup (inserting all relevant information needed), and then replacing any mention of the replaced nodes by N in all relevant edges (finding connected edges).
As I personally generate layouts/nodes in python before sending them to cytoscape for visualization, I'll refrain from posting a potentially ineffecient/incorrect javascript/cytoscape example :)

Best practice: Rendering volume (voxel) based data in WebGL

I´m searching for a (or more) best practice(s) for the following problem. I´ll try to describe it as abstract as possible, so the solution can be applied to scenarios i have not yet thought of.
Data available: Voxels (Volumetric Pixels), forming a cube, with coordinates x,y,z and a color attached.
Goal: Use OpenGL to display this data, as you move through it from different sides.
Question: Whats the best practice to render those voxels, depending on the viewpoint? How (which type of Object) can store the data?
Consider the following:
The cube of data can be considered as z layers of x y data. It should
be possible to view, in-between-layers, then the displayed color
should be interpolated from the closest matching voxels.
For my application, i have data sets of (x,y,z)=(512,512,128) and
more, containing medical data (scans of hearts, brains, ...).
What i´ve tried so far:
Evaluated different frameworks (PIXI.js, three.js) and worked through a few WebGL tutorials.
If something is not yet clear enough, please ask.

There are 2 major ways to represent / render 3D datasets. Rasterization and Ray-tracing.
One fair rasterization approach is a surface reconstruction technique by the use of algorithms such as Marching Cubes, Dual Contouring or Dual Marching Cubes.
Three.js have a Marching Cubes implementation in the examples section. You basically create polygons from your voxels for classical rasterization. It may be faster than it seems. Depending the level of detail you want to reach, the process can be fast enough to be done more than 60 times per second, for thousands of vertices.
Although, unless you want to simply represent cubes (I doubt) instead of a surface, you will also need more info associated to each of your voxels rather than only voxel positions and colors.
The other way is raycasting. Unless you find a really efficient raycasting algorithm, you will have serious performance hit with a naive implementation.
You can try to cast rays from your camera position through your data structure, find / stop marching through when you reach a surface and project your intersection point back to screen space with the desired color.
You may draw the resulting pixel in a texture buffer to map it on a full-screen quad with a simple shader.
In both cases, you need more information than just colors and cubes. For example, you need at least density values at each corners of your voxels for Marching cubes or intersection normals along voxels edges (hermite data) for Dual Contouring.
The same for ray-casting, you need at least some density information to figure out where the surface lies or not.
One of the keys is also in how you organize the data in your structure specially for out-of-core accesses.

maximum number of svg elements for the browser in a map

I am creating a map with leaflet and d3. A lot of circles will be plotted on a map. In terms of browser compatibility, there is an expected limit of how many svg elements the browser can render. In terms of user experience however, I would prefer that the user can see as many elements on the map as possible (otherwise the user might need to zoom in and out constantly and would need to wait for the ajax to return data). There will be some optimisation that I need to consider (user waiting time user vs. server query load vs. what the browser can handle).
See plot, there is a limit right now on the number of points that the server returns and thus only a portion of the map is filled.
The browser cannot handle a fully filled map here and the user would need to wait too long for the server response as well.
I suppose the problem that I am faced with needs to be solved by answering two questions:
Is there a standard in terms of what the average browser can handle in terms of number of simple svg shapes (circles) on a map?
What is the best technique to show as many shapes on the map as possible?
I'm considering the following points but I am unsure if it will help;
use squares instead of circles
use the leaflet API instead of the D3

Speaking in general terms, neither of the points you're considering will help. In both cases, the amount of processing to be done / information to display by the browser will be approximately the same.
Regarding your first question, not that I'm aware of. There are huge variations between browsers and platforms (especially if you consider mobile devices as well) and an average would be almost meaningless. Furthermore, this is changing constantly. I've found that up to about 1000 simple shapes are usually not a problem.
To show as many shapes as possible on the map, I would pre-render them into bitmap tiles and then use either the leaflet API or something like d3.geo.tile (example here) to overlay it on the actual map. This way you can easily scale to millions of points.

Although you can only render ~2-5k SVG elements before you start to see noticeable slowdown (depending on size, use of gradient fills, etc.), you can store and manipulate much larger datasets client-side. You can often handle tens or hundreds of thousands of data points efficiently in SVG: the trick is to be very selective about what you actually render, and to use techniques like debouncing to redraw only when necessary.
(For very large datasets: yes, you'll need to either aggregate/subsample points or pre-render.)
With this in mind, one technique I've used for d3 maps in particular is to use d3.geom.quadtree() to dynamically cull points as the user pans/zooms. More specifically, I avoid drawing points that are either
outside the current map bounding box (since these aren't visible at all), or
too close to other points (since these add visual clutter and are hard to interact with anyways).
In JS-ish pseudocode, this would look roughly like:
function getIndicesToDraw(data, r, bbox) {
var indicesToDraw = [];
var Q = d3.geom.quadtree();
// set bounds in pixel space
for (var i = 0; i < data.length; i++) {
var d = data[i];
var p = getPointForDatum(d);
if (isInsideBoundingBox(bbox, p) && !hasPointWithinRadius(Q, r, p)) {
Q.add(p);
indicesToDraw.push(i);
}
}
return indicesToDraw;
}
function redraw(svg, data, r, bbox) {
var indicesToDraw = getIndicesToDraw(data, r, bbox);
var points = svg.selectAll('.data-point')
.data(indicesToDraw, function(i) { return i; });
// draw new points for points.enter()
points.exit().remove();
// update positions of points (or SVG transforms, etc.)
}

This is a question of cartography as much as it is of technology. Just because you can put thousands of points on a map doesn't mean you should. You should ask yourself if your user needs to see as many points as possible at once, and will understand the data better as a result of it. Will seeing this many points confuse and overwhelm the user, or will it help them accomplish their desired task? Is there any way you could filter the data so that all points do not need to be drawn at once?
The most common solution to this problem is to use clusters, usually with something like Leaflet MarkerCluster. In the past while using D3 and Leaflet I have created a sort of heat map by creating an arbitrary grid, assigning each point to a bin, and applying a color ramp to the grid. However, getting back to your original concern, it is rather computationally intensive.
Depending on which platforms you want to support, be aware that phones and tablets do not always handle SVGs very well, especially while drawing thousands of features.
Another place for potential performance gain is in the delivery of the point geometry. I'm not sure how you are currently loading them, but using a spatially indexed PostGIS table and selecting by bounding box is often quite quick, but because you are drawing points and not polygons, you could even get away with loading them into the browser via CSV, which is substantially smaller than a GeoJSON or Topojson.

I would load all the data at once but only draw circles in the view port that are large enough. On zoom or pan, remove the circles that shouldn't be shown and check if previously hidden circles should be added.

You may use canvas + three.js or webgl where may join another map and 10k animation created 3d models or native code svg some elements, and intercative live time animation in one scene very good. I`m tested for fun this methode.Sry, my bad engl. Another puths - i think may used dlsl shader, opengl+ wasm and so on

Classify lon/lat coordinate into geojson polygon using Javascript

I have a geojson object defining Neighborhoods in Los Angeles using lon/lat polygons. In my web application, the client has to process a live stream of spatial events, basically a list of lon/lat coordinates. How can I classify these coordinates into neighborhoods using Javascript on the client (in the browser)?
I am willing to assume neighborhoods are exclusive. So once a coordinate as been classified as neighborhood X, there is no need to further test it for other neighborhoods.

There's a great set of answers here on how to solve the general problem of determining whether a point is contained by a polygon. The two options there that sound the most interesting in your case:
As #Bubbles mentioned, do a bounding box check first. This is very fast, and I believe should work fine with either projected or unprotected coordinates. If you have SVG paths for the neighborhoods, you can use the native .getBBox() method to quickly get the bounding box.
the next thing I'd try for complex polygons, especially if you can use D3 v3, is rendering to an off-screen canvas and checking pixel color. D3 v3 offers a geo path helper that can produce canvas paths as well as SVG paths, and I suspect if you can pre-render the neighborhoods this could be very fast indeed.
Update: I thought this was an interesting problem, so I came up with a generalized raster-based plugin here: http://bl.ocks.org/4246925
This works with D3 and a canvas element to do raster-based geocoding. Once the features are drawn to the canvas, the actual geocoding is O(1), so it should be very fast - a quick in-browser test could geocode 1000 points in ~0.5 sec. If you were using this in practice, you'd need to deal with edge-cases better than I do here.
If you're not working in a browser, you may still be able to do this with node-canvas.

I've seen a few libraries out there that do this, but most of them are canvas libraries that may rely on approximations more than you'd want, and might be hard to adapt to a project which has no direct need to rely on them for intersections.
The only other half-decent option I can think of is implementing ray casting in javascript. This algorithm isn't technically perfect since it's for Euclidean geometry and lat/long coordinates are not (as they denote points on a curved surface), but for areas as small as a neighbourhood in a city I doubt this will matter.
Here's a google maps extension that essentially does this algorithm. You'd have to adapt it a bit, but the principles are quite similar. The big thing is you'd have to preprocess your coordinates into paths of just two coordinates, but that should be doable.*
This is by no means cheap - for every point you have to classify, you must test every line segment in the neighborhood polygons. If you expect a user to be reusing the same coordinates over and over between sessions, I'd be tempted to store their neighborhood as part of it's data. Otherwise, if you are testing against many, many neighborhoods, there are a few simple timesavers you can implement. For example, you can preprocess every neighborhoods extreme coordinates (get their northmost, eastmost, southmost, and westmost points), and use these to define a rectangle that inscribes the town. Then, you can first check the points for candidate neighborhoods by checking if it lies inside the rectangle, then run the full ray casting algorithm.
*If you decide to go this route and have any trouble adapting this code, I'd be happy to help

Algorithm to draw connections between nodes without overlapping nodes

I have a series of nodes in a graph. The nodes are placed by the user in specific spots. The nodes are guaranteed to not overlap and, in fact, to have a buffer of space between them. These nodes are connected and each edge joins to a node at a specific point. I need to draw the edges between the nodes such that the edges:
(required) do not overlap the parent nodes
(ideally) would not overlap any node
I am not worried about edge crossings. Bonus points if there's an implementation of this in Javascript. I am unable to use any libraries outside of Javascript.

One solution could be using Bézier Curves:
"A Bézier curve is defined by a set of control points P0 through Pn,
where n is called its order (n = 1 for linear, 2 for quadratic, etc.).
The first and last control points are always the end points of the
curve; however, the intermediate control points (if any) generally do
not lie on the curve."
So the basic idea is to use parent node(s) as intermediate control points. You may also use points of the edges as intermediate control points to avoid edges overlapping.
In the wiki article you can find nice animations explaining it.
For javascript implementation I took a look at the followings libs:
jsdraw2d (LGPL license) with a nice demo and well referenced. Implemented it also using HTML5 and SVG for performance (jsdraw2dX).
jsbezier on google code
But if you google "javascript bezier library" you can find more.

If you are familiar with C# and .NET you can explore Microsoft.GLEE library (description is here and here) via ILSpy, or even theoretically save this sources to .csproj, modify and recompile it with Script# to JavaScript.