I have been using Tkinter and Pygame for a GUI to visualize data points.
With Tkinter, files are chosen from a list, and then a Pygame window is open where the data points are graphed.
Pygame is not meant for data visualization, and I would like to use an alternative.
I want to use JavaScript to visualize the data, but I don't know how I would bring the large amounts of point data from my files into a browser window, and then have the user be able to manipulate the data (moving, adding, or deleting points), and then saving that data back out.
What tools do I need to bring the data into JavaScript, generate graphics for graphing and some other basic shapes (text, lines, dots), and then export the data out?
plotly.js is a great JS library that will allow you to visualize any data on the web with ease. You can check out the link and search for some tutorials on the internet, such as this one, but I don't think that is the problem.
The issue is with getting the data from a file.
When it comes to the web, you will need some sort of web server that can serve and receive files.
The web browser allows the user to input the file (this can be done with html tag input), then it sends a request containing the file data to your web server (this is where your JS code is), where you can then do whatever you like with the data and send a response back to be shown by the browser.
Although web server might be easy to get started in, with many different languages you can do it in (including JavaScript), I'm assuming it is not in your interest to build web servers as of yet.
If you insist on building this with JavaScript, you will have to find another method to get the data you want to plot, or actually build some kind of server that can handle these files you want the user to input. If this is really for you, here is a starting tutorial for building web server with Node.js (JavaScript), and here is another one for building web server with django (Python)
But if you don't like this idea and don't mind going back to Python, you can use matplotlib.
Python has a 3rd party module called matplotlib, which allows you to very easily plot points and graph them with many different customizations.
So, after you extract your points from your files, you can then remove whatever Pygame code you were using and instead very simply do:
import matplotlib.pyplot as plt
... # extracting your points from the files
plt.plot(xpoints, ypoints)
plt.show()
Where xpoints and ypoints are the points from the files
This will create a line graph. You can customize this graph by passing in an optional third argument to the plot function, for e.g:
plt.plot(xpoints, ypoints, "x")
This will only plot the points instead of drawing a line. You can also change "x" to "ro", "bo", "r+" and so many more.
You can refer to the link above to take a look at the matplotlib documentation and choose whether you would like to try it out or really still stick to JS. However IMO I think you will have to find another way to get your data in that case as building web servers takes time and understanding, and may be very confusing at first :)
Related
Just a quick one. Someone I know is using google sheets as a Read database. Their data needs are very small so that's what they want to use. I have built an App in Adobe Air and this person wants to be able to display the sheet data inside this app. I know I am going to be using some form of Javascript to do this but I am not sure how to pull the cell data into the dynamic text fields. Any advice would be most welcomed.
Way №1. The simplest way is to get the cell data as a CSV or TSV file (whatever is more suitable for your purposes) via special download link: https://docs.google.com/spreadsheets/d/{KEY}/export?gid={GID}&format=csv
Obviously, you are to replace {KEY} and {GID} with valid document key and sheet identifier respectively. The format parameter takes csv or tsv value.
You read it with old good URLLoader and get a plain text which you split into lines and then every line into values. There shouldn't be any problems.
IMPORTANT: The document should be shared as "anyone with link", so there are apparent security concerns, yet if security is not an issue, go for it as it is, as I said above, the simplest way.
Way №2. If security IS a concern, then you need to use the gspread library. It does not have an AS3 port, but it has JS one so I assume it is the one that acquaintance of yours uses: https://developers.google.com/sheets/api/quickstart/js
You don't run your app in the browser, JS wouldn't really be your ally here, you will need a server with PHP proxy (that will handle auth and communication with Google Sheets) or a local Python app maybe (it is not difficult to build and run a simple HTTP serving application), it is up to you to decide which one of the available gspread ports will be simpler for you to use and serves your needs best.
I am currently undertaking what might be an overly ambitious project for my skill level, but I think I could learn a lot form this exercise.
I work for an aerial surveying company and each time a project is flown, the LiDAR software creates a log file. From this log file, I have written a script that extracts the data and outputs a KML file with data such as flight line numbers, location, height etc.
What I now want to do is write a program that can upload these KML's to a database and show them in a window similar to Google Earth that the user can sort through the flight numbers, locations and dates etc. This would be an incredibly handy piece of software at work and a lot of people would use it, not to mention massive brownie points for my trying to secure a Jnr Dev position in the company going forward.
I know I could mostly use Google Earth to accomplish this task, but for me, it's kind of not the point.
So far, I have looked into Folium and LeafletJS.
https://leafletjs.com
https://pypi.org/project/folium/
Folium can manipulate data and Leaflet can present it, but I am not sure I have the best solution for the job and if it can do exactly what I need to. Especially considering that I only know a little Javascript. I am also not sure if Folium
Ideally, this app would be either a standalone piece of software, or something that my existing script writes to an online server and then can be opened and shown through a browser window. I am biting off more than I can chew with this, but I'm prepared for it to take a little time and be a good learning project. Preferably written in Python.
Any hints or suggestions to what packages I might use would be greatly appreciated. Or, if I am going about this in completely the wrong direction, please let me know.
Thanks a lot.
Welcome to SO.
Here are a few elements that might be useful to you :
leaflet can natively load geojson files. So you could either convert the kml files to geojson using python, or use a leaflet plugin (leaflet-omnivore for example) to read kml files with leaflet. As a first step, I would start with that : a web page that shows one of your flight path. The path data being just a geojson file that is loaded by your app.
then if you want your app to filter flight paths on some criterias, and if you have many of those files, you're going to need a backend. That means a server, with a database, potentially with geo-capabilities (postGIS, mongoDB,... there are many) and a backend application. So your front app (with the js and leaflet) is going to send some requests to your server (send me the file for the flight XYZ please), the backend receives that request, performs a database query, get the file and send it back to the client. The client displays the data on the map.
If you want to write the backend in Python, you can have a look at frameworks like Django or Flask.
That's a lot of tools, but if you start with the js + leaflet part, you will be happy to see a map quickly and will have something to show pretty soon. Then you can start having a look at the backend part. If you manage to get this project done...you definitely deserve that junior dev position.
Why would you do such thing??: because in some particular cases I need to hide the source values of a graph generated by dygraph. Since in the case of "not hidden values" I am using dygraphs js libray, using this particular feature would save me a lot of time.
This way I would have the same look and feel, and I would not have to modify the data generation process.
I would like to know if this road is feasible or utter madness:
Steps: (using Yii-PHP over an Apache server)
Generate javascript page on the server side.
Get image as png (somehow), still on the server side.
Send the image to the client as the only thing he can receive.
There's a ton of graphing libraries for PHP, why not use one of those? (e.g. JpGraph, pChart...) Pretty much every one of them can produce an image in PNG.
Feasible steps:
Invoke a PHP graphing library on server side
Render it as PNG and send to the client
If you insist on using a JavaScript library, then I guess PhantomJS is your only option, but it is not exactly optimal.
tl;dr: My vote goes to "utter madness".
Some dygraphs users have built a tool for exporting PNGs: demo and discussion.
Looking for a solution to an arguably strange problem. Ok, so we are using d3.js to plot charts and graphs. However our data sets can be very small, to intensely massive. Right now most of what we are doing is internal and just prototyping. However, we do show clients these charts and draw them in real time for them, quite often and rapidly change their inputs.
Doing this in D3 looks great, but can be slow as expected. I'm more interested in what the possibilities are for this process. Go to our website, loging, and show an instance of our dashboard being rendered remotely on the server. Our server cluster is a super demon beast so I'm not worried about it doing any heavy lifting. It can do these processes about 100x faster than our best pc so it seems if we could setup our website to create instances on the fly of our dashboard, BUT only have access to that user accounts data.
This is getting a bit convoluted so let me explain. We have a database, full of millions of data points. We have about 10 user accounts. Each have access to different pieces of this data. One has access to all of it, the other some of it. All of this is not the issue we are looking for a solution on. We are more interested in the ability of our server to create multiple instances of our site, through a window essentially, that the user is remotely controlling. Like a Remote Desktop in a way. We could even start with the user login form being part of the remote render. Where our system is fully hosted and operates on the server itself, and the we page is essentially a KVM on the server in a way. However it needs to handle multiple users at the same time.
We are using Centos 6.4 lots of python for the back end stuff, php HTML and a mixture of Postgres and SQLite, but I doubt any of this is important. Just want to cover my bases.
It seems unlikely to me that you'd be able meaningfully display millions of datapoints on a single screen without grouping and summarizing them in some way. Do the processing and summarize the data on the server and ship the resulting smaller datasets to the client, which will then plot your graphs and charts from that. It's likely you'll have more than one set of data now, but it should result in much better client performance. e.g.
{millions of points} -> transform on server -> data for bar chart to client
{millions of points} -> transform on server -> data for XY-scatter chart
etc.
What you've proposed is not really a programming issue, and isn't going to scale very well.
I have a HTML page that uses javascript to generate dynamic images using a graph handler on a different server. The images will contain the same data for 1 week but will change when the 1 week window expires.
I am trying to come up with a way to automatically save the contents of the page to either a local file on the server or write to a PDF file.
I tried to use a 'web downloader' like HTTTrack, but it does not get the dynamic images...
I am running the html page off IIS.
I have no experience with IIS or ASP.
Thanks!
I'm not sure that I see any way to do this directly off the front end in an automatic manner. The challenge is that any "screen scraper" you have go out and grab the site with would need to be running javascript to get the tables, which isn't how I see many such systems operating. It's partially why you see strangeness on Archive.org when you have a site that's heavily augmented with javascript or flash.
An untested concept you might attempt was posted in this Stack Question
I could see some sort of a system that you rig together with another computer that schedules an browser load then prints to .pdf in some fashion. I've been unable to find any specific software that would automate that process, so you'd be left cobbling such a system together on your own.
Clearly you have the data available to make your dynamic images. The most feature-rich way I could think of would be to use a system like Jasper Reports or Crystal Reports, which you could feed your data, replicate the report, and easily output via pdf, a built-in export in both systems.
Perhaps its worth questioning your end purpose. To me, creating a "snapshot" of the relevant data in another table and using another system to render your graphs from that snapshot data seems far more valuable than just a print of the screen. You can then go back and adjust data as needed, or use it for other reporting purposes, exporting in any number of tools that are even as simple as Access. Heck, 10 years down the road you may want the data to look better than the graph system you're currently using, and you'd have the data to render it any way you want. When the VP of marketing comes looking for his numbers, a simple click would output those numbers that could be manipulated as needed from there.
I was able to accomplish what I wanted to do using wkhtmltopdf to convert my HTML page with Javascript to PDF. I ran the job via a task scheduler to supply my website url and output file name as parameters.
I then used a windows batch file to check if the file was created and then rename/email it to interested parties.
This of course requires that you have the ability to install wkhtmltopdf on your server.