Develop Reference

Large quantity of data real time visualization

I'm building a dashboard receiving tons of data from MQTT in JSON format (data frequency ± 4Hz). I'm trying to visualize this data and initially picked Highcharts (http://www.highcharts.com/) but quite quickly realized that browser can't simply handle that amount of information.
I'm quite new to visualizing such amounts of data so would appreciate any advice on how to handle that amounts of data (currently arduino's post data to given topics and I receive them on the server, store in mongo and send to the browser to be updated). There's a chance my entire approach is completely wrong so please guide me to a path!

Your problem is likely that the SVG that Highcharts is creating is too large for the browser to handle in a timely manner. If you have too many data points, you'll need to do 1 of 2 things.
Switch to a charting library that uses an HTML5 Canvas instead of an SVG
Pare down your data using a best fit or some other curve interpolation.
If you switch to a Canvas based chart, you'll lose some of the cool features that SVGs have, but every data point will make it to the page. I've used jqChart for this in the past. It's not free though.
If you pare down your data, not every data point will make it to the page, but the nice features that SVGs allow will still be there. You'll still probably be able to use highcharts.

I think you need to have a look at crossfilter (filtering lib), dc.js (helper library for d3 and crossfilter) they can handle large amount of data well

After using Highcharts for more than 3 years I can recommend it even for the questioned use case. To follow recommendations to use other charting libraries might be not a good choice if you already picked Highcharts (after comparison with others) and invested quite some time in it. If you like it, stick with it.
To address this edge-case with thousands of data points the team announced the boost.js module.
It originated from this issue on Github. The discussion reveals lots of background information about the problem domain. And it demonstrates how well the Highcharts team is doing customer-/community-driven development.
Looks like you can do it with Highcharts - if you still want. :-)
Last but not least I can recommend the Highcharts Cookbook from Packt

I use grafana to visualise my time series data. I don't believe it supports mongodb at the moment. I use influxdb as the data back end.

Graph visualization frameworks

I want to visualize a Neo4j database in a graphical application. (I'm doing this primarily as a learning exercise).
I want it to behave almost identically to the visualization in Neo4j's WebAdmin: When the graphic appears, I want to see the nodes moving around as they settle into their final locations, and I want the user to be able to click and drag node icons around the screen and have them stick where they are dropped, with the rest of the graph adjusting to account for the change.
I have two questions.
What are the technical terms for the three requirements crudely described in the paragraph above?
I'm a Java programmer with no experience in JavaScript, but I'm thinking that, with this project, it's time to get some - sigma.js in particular seems very powerful and relatively simple. However, ultimately this needs to be done within a Java Swing application, and while I know there are various containers I can use to embed JS/HTML content into Swing, still it seems that a pure Java solution would be ideal, and I'm curious about any frameworks that might make this possible. I'm looking into Gephi now, but it doesn't seem anywhere near as powerful or as simple to use as sigma.js. Am I wrong in this assumption?
To summarize the second question: Which would be the better investment of my time: Learning sigma.js and the various JS frameworks (which will require me to get comfortable with JS in general), or learning to use Gephi in Java? (assuming I have a general and long-term interest in graph visualization)

Java visualization
Gephi does provide a toolkit for Java and Python which doesn't contains the UI modules, so unless you want to use Gephi itself to visualise your network you cannot embed its UI in your Java/Swing application as long as I know.
The toolkit contains all the algorithms and plugin infrastructure so if you want to use some of their analysis tool embedded in your app you can use that, otherwise I don't think it's your case.
There are other Java options if you want for graph visualisation here: Java graph library for dynamic visualisation
About Gephi and sigmaJS
IMHO Gephi is far more powerful than sigma.js: it has full SNA tools, multiple layouts, a data editor, exporters, plugin-systems...
One important factor is also the size of the dataset you want to visualise: while sigmaJS can handle up to 500 nodes in the view, Gephi can layout easily 50000 nodes with GPU support as well.
Gephi is a scientific tool made for people that don't want to code to visualise a network and it does it's job. SigmaJS is one of many JS solutions to visualise graphs on a web page.
You can always export to sigmaJS from Gephi with a plugin.
What's better?
At the end of the story I guess the tool you are going to use will depends on:
do you have to visualise a big dataset? -> Java solutions will win for big ones
do you need to distribute it easily? -> Web app / SigmaJS wins for distributed
do you want to have a full integration with your app? -> Java wins

The best one I've found is Keylines, but its not cheap.
Gephi is OK, but the last time I looked it did not support Neo4j 2.0
If your a programmer Neo4j is open source, so you can fork the repository on Git and use that (you will have to check the licencing model before you commercialise any product though)

I'm new to d3, is there a simple way?

I'm completely new to d3. I've been developing for a few years, but I managed to track down d3 after seeing this great little tool in action.
http://gia.guim.co.uk/2013/03/video-game-violence/index.html
The problem is, from what I've found so far, d3 definitely doesn't have the same following as jquery. The documentation and the tutorials just aren't there as much.
I'm unfamiliar with SVG, but it looks pretty decently usable.
I wanted to reach out and see if anyone has some a hello world for this specific example?
I'd like to start with two sets of data and show the connections between them, but this feels like a decently large operation to just jump right into.

You can check Scott Murray's tutorials here.
They were even converted to a book "Interactive Data Visualization".

Render graph in Python and Javascript

I am looking for libraries that can render graphs, PRETTY graphs.
I need to both render static images, preferably with python, and also to render them dynamically in the browser. A Canvas solution would be great, flash is acceptable too.
I googled around a little and found python-graph, but the images look quite ugly.
To clarify: I'm talking about graphs (http://en.wikipedia.org/wiki/Graph_theory), not charts.

Check out following:
canviz - Graphviz graphs in JS
pydot - Graphviz for Python
yapgvb - Another alternative for Python
You should be able to use the same graph definition both in Python and JS. I hope those help. :)

Processing.js, a port of the Processing language, is a very powerful JavaScript visualization library that can been used to generate graphs.

You can check out Pydot, though the quality of the graphics might not be up to snuff. What about invoking raphael in your webapp? Not sure if people have done mathematical graphs, but the output certainly looks nice.

We use flot for the Internet Archives Open Library project and it works great. You can see a few of the graphs it's generated on the OpenLibrary front page.

A good JS library that creates graphs, utilizes SVG/VML and uses JSON as a dataset

Any suggestions? I was looking at gRaphael but it was more of a ground up approach. I'd like to have basic or complex examples to work with.

Google Visualizations

I've been using gRaphael to create all sorts of graphs over the past two months and can't say enough good things about it.
That being said, if you want to try something else, http://highcharts.com/ may be a good resource.

I have used dojox.charting in my projects, which uses JSON as a data set, and it has worked well for me. In particular, I used it to create the charts here.