Develop Reference

JavaScript for visualizing, manipulating, and exporting data points?

I have been using Tkinter and Pygame for a GUI to visualize data points.
With Tkinter, files are chosen from a list, and then a Pygame window is open where the data points are graphed.
Pygame is not meant for data visualization, and I would like to use an alternative.
I want to use JavaScript to visualize the data, but I don't know how I would bring the large amounts of point data from my files into a browser window, and then have the user be able to manipulate the data (moving, adding, or deleting points), and then saving that data back out.
What tools do I need to bring the data into JavaScript, generate graphics for graphing and some other basic shapes (text, lines, dots), and then export the data out?

plotly.js is a great JS library that will allow you to visualize any data on the web with ease. You can check out the link and search for some tutorials on the internet, such as this one, but I don't think that is the problem.
The issue is with getting the data from a file.
When it comes to the web, you will need some sort of web server that can serve and receive files.
The web browser allows the user to input the file (this can be done with html tag input), then it sends a request containing the file data to your web server (this is where your JS code is), where you can then do whatever you like with the data and send a response back to be shown by the browser.
Although web server might be easy to get started in, with many different languages you can do it in (including JavaScript), I'm assuming it is not in your interest to build web servers as of yet.
If you insist on building this with JavaScript, you will have to find another method to get the data you want to plot, or actually build some kind of server that can handle these files you want the user to input. If this is really for you, here is a starting tutorial for building web server with Node.js (JavaScript), and here is another one for building web server with django (Python)
But if you don't like this idea and don't mind going back to Python, you can use matplotlib.
Python has a 3rd party module called matplotlib, which allows you to very easily plot points and graph them with many different customizations.
So, after you extract your points from your files, you can then remove whatever Pygame code you were using and instead very simply do:
import matplotlib.pyplot as plt
... # extracting your points from the files
plt.plot(xpoints, ypoints)
plt.show()
Where xpoints and ypoints are the points from the files
This will create a line graph. You can customize this graph by passing in an optional third argument to the plot function, for e.g:
plt.plot(xpoints, ypoints, "x")
This will only plot the points instead of drawing a line. You can also change "x" to "ro", "bo", "r+" and so many more.
You can refer to the link above to take a look at the matplotlib documentation and choose whether you would like to try it out or really still stick to JS. However IMO I think you will have to find another way to get your data in that case as building web servers takes time and understanding, and may be very confusing at first :)

Architecturally correct design pattern for Tornado/Bokeh interactive dashboard & workflow creation application with other javascript added on

I've been trying to build my own data analytics platform with the following features/workflow -
backend to generate/retrieve data from external systems (for eg. time series data using mqtt, websockets)
Tabbed Web application to -
A. Define workflows visually (i.e interactive directed graph/flowchart)
I would like to have an area on the screen where the user can see a selection of shapes (eg. rectangle) (i.e the transforms) join the shapes together to create a chain transforms that is to be run on the data
B. show results of transforms (also other charts from the original data) in a grid layout (i.e an interactive dashboard
I am new to most of this stuff, although I have worked on Django for my earlier projects. This time I would like to have an async, non-blocking architecture.
I have built my system up to a place where the MQTT based pub/sub system is complete, & the backend can send data if a request is placed for it
Here's my problem/confusion -
I have read up on the tornado docs, bokeh docs, examples (eg. the server_embed example) and also many many posts/questions/articles on how to go about doing this stuff
When i attempt to find a solution without using javascript, I have not come across any simple method/example of creating things like a good responsive grid layout, interactive flowchart or visual editor etc. etc. with pure python. Python seems to be best with the other stuff I need for my app, eg. pandas, numpy, blaze, odo, scikit, tornado.
However, whenever I look at javascript there is this huge universe of nice & beautifull stuff like jquery, JointJS, AngularJS & what-not that just seems so cool & easy to use
Plus I read an answer somewhere (quora?) on Jinja2 vs. AngularJS, which was along the lines of "You should not mix both because Jinja2=server side HTML generation vs. AngularJS=browser side HTML generation, which don't mix well)
All of this has led me deeper into confusion, because I'm new to most of this
So my questions are -
What is an architecturally correct pattern, example or some reference(s) on using
a. tornado for the server/webserver
b. Use bokeh for interactive charts for the original/transformed data
c. Use some nice CSS (or maybe SASS?)
d. Use some existing HTML5 framework such as Foundation for the layouts * responsive grids
e. Use something like JointJS in the same browser page somewhere to join blocks together to define the transforms
If I use javascript, vs. I don't, what does this architecturally mean for the tornado templating system
Can I, or should I use Jinja2 in this mix (i saw an example that explains how integration with Jinja2 is a better option for templating on tornado)

Generate charts without HTML / Browser

I have a node.js application that accesses data of a MySQL database and I want to generate charts from that data dynamically, to pipe it as image (PNG/JPG) through a HTTP server to the user. I've search for a little while now, but I was unable to find a library to do this without the HTML/canvas approach. Since I want to let the server do the rendering to use the graph in emails or also dynamically rendered PDF files, this isn't an option here.
What could be the approach here (with or without a library) ?
Edit to provide proof of own work: I found https://plot.ly, which is in terms of usage and result really close to fulfil my needs, but requires an internet connection, since it calls a remote API.

You could also try asking this in the Software Recs Stack Overflow. But one option that seems like it could fit your needs is ZingChart.
They have a phantomJS build for rendering static versions of charts. In addition, licensed users can access a Node.js build for server side charting.
Full disclosure, I'm on the ZingChart team so please let me know if I can help you review all your options.

There is a node wrapper for the chart library chartist. You could use this library to generate a SVG, and a svg2png lib to convert to PNG.

Well there are multiple options, I would use phantomjs (headless Webkit browser) to render the charts.
Why would this option be good? One of the reasons is that you don't need duplicated code for the server and client side coding, you could simply reuse the code. On server side create a html file with required scripts and data and render it.

Dynamic handling of mongodb data

My requirement is to generate charts using fusion charts.
I'm supposed to provide data to fusion charts from MongoDB which is a no SQL database.
MongoDB stores data into collections which is in the json format.and fusion charts accepts the data either in json format or in Xml format.
how should i provide the data to the fusion charts?
Currently I have written a java class to read the collection and put it into the .json file and I am passing this json file to the fusion charts(ie. a program written in html embedded with javascript)separately.I am stuck with the issue that how to call a java function from javascript?
Hence is there any other better alternative for this ??

Well, it isn't an easy task. You must make a Java Web Application with a Servlet, REST service or whatever you want. Then you must deploy the application in an application server like Tomcat or JBoss, so your HTML+Javascript application is able to do request to your Java service.
You can't call a standalone Java class from an HTML+Javascript page.
I think you're trying to do something that you don't know anything about... and must read a lot about the subject.

how should i provide the data to the fusion charts?
FusionCharts has specific data formats for different charts. So if you have got the data from your MongoDB you need to parse and create FusionCharts compatible JSON data. So any JSON data wont work.
I hope I am wrong in assuming that, you have already done so. Otherwise you really need to learn a lot about simple things about reading the basics of any library, you intend to work.

Framework for a Business Dashboard - MVC, browser accessible

We are attempting to create a few dashboards for one of our Businesses, fetching data directly from the SQL server. We know what kind of business logic & display we'll like to do - having already built the prototypes from SQL data exported in Excel, and now - having prototyped their wireframes as well. The dashboard will include dynamic timeline based line & bargraphs & geographical map overlays.
We are trying to identify the best platform to do it. Here are the constraints:
The database is SQL based
The frontend should be browser accessible - hence a pure desktop application is ruled out.
While for the first usecase, the database is MSSQL - we'll like the platform to be say PHP based (or Python etc, but not ASP.Net)
Prefer to have the platform giving the option to host it on our systems (since our database server may not be online, but behind a company LAN - and hence not accessible from any public server, without port opening, etc), as opposed to it being only on online SAS.
Now, we are trying to architecture pieces. One approach is to have a technology stack consisting of:
Database - which we can live with what we have (our data isn't large enough to require anything beyond SQL).
An application logic framework (preferably MVC), along with a good set of libraries for Analytic applications (eg processing time series, statistical analysis - or some glue logic for calling, say R).
Frontend JS / HTML libraries. For this, we have looked at GoogleCharts, HighCharts, Paper.js, Raphael, Angular.js, D3.js, and new Razorflow. We haven't yet been able to figure out the best options amongst these.
Any suggestions / comments for the JS libraries (point 3) to use? And if in PHP - suggestions for good application framework + analytics library (point 2) to use above? Or if not, another language along with a good supporting framework for this?
Another alternative we're exploring is moving to a completely UX based dashboarding - eg qlikview, Tableau, or klipfolio. kibana - on the same lines looks interesting, but seems more suited for logs & text data, not exactly our use case.
Within the Microsoft world (we are trying to avoid), devExpress seems like a complete suite - with frontend JS to the backend database integration.
Any suggestions / comments on the right platform to use?
Arvind

I am the co-founder of RazorFlow. Are you still looking for a dashboard framework to use among the options listed? Our PHP Bindings are quite powerful allowing you to build your entire application in pure PHP and not worrying about the HTML/CSS parts of it.
Let me know if you want to discuss further by reaching out to our support email at support -at- razorflow.com