So this is a slightly insane idea that I've been kicking around. I'm thinking of building a classifier for (a specialized subset of) some web content, and then supplying it to users in the form of a chrome extension. (So: user goes to web page, clicks button on chrome extension, extension applies pre-trained model to generate prediction, presents result to user.)
But I'm cheap, and I don't want to run a server to receive data from the chrome extension, generate a prediction, and send it back to the user. It occurs to me: why not just find some way to serialize the trained model or something, and do it all on the client side.
So if I end up fitting a very simple model, like a logistic regression or something, then it would be easy to transfer to the client-side. Ultimately, a trained logistic regression is just expressible as an ordinary mathematical function mapping features to predictions, it's like one line of code. And I guess that's true of some fancier models as well (linear SVM). Under those circumstances, the chrome extension could do all the data cleaning and such, and then generating predictions is just a matter of calculating a dot product. But the same isn't true if I end up using, say, random forests or something.
Tragically, there doesn't appear to be a PMML library or anything like that for javascript.
So, I turn to the wisdom of the internet. Are there any other methods for productionizing ML models that might be usable on client-side javascript? Or are the options really just (1) run a server, (2) write a PMML library for JS, (3) there is no third option?
Thanks!
Do you have your models available in PMML data format already?
If so, then:
Translate PMML to Java source code. This is easy using JPMML libraries.
Translate Java source code to JavaScript source code. This is even easier using Google Web Toolkit.
As a result, you would have a self-contained JavaScript model object that exposes model schema query and scoring API similar to the JPMML-Evaluator library.
Related
I am using sklearn in python to build/train some models (Random Forest Regressor, Kmeans, SVM, ...), and I would like to use those trained models in a webapp Javascript/html. Is there a way to do so?
I've seen that tensorflow.js allows to do such things with keras model. But it seems to me that TF/keras in python are limited to neural network.
I've also seen sklearn-porter, but it seems to be limited to some specific models (classification mostly). If someone managed to use it, can he tells me more?
Thanks in advance for any help
As an option you can simply build RESTfull API in Python (for instance using Flask). After it would be easy just request a prediction from Javascript.
There are few benefits for such approach:
except the model, you still maybe need data preparation functions;
you have to separated projects;
with REST API you can easy add new version of the model;
I am trying to make sure that the classification, training process and the model is hidden while used in browser. I have found that wasm is kind of binary format for web browser to execute. Can I use wasm?
Is there a way in tensorflow or in js to hide my training or classification algorithm and pre-trained model?
Is there any other framework supporting it?
There is currently not yet a good way to simultaneously run your TFJS model in a users' browser, and also keep your model secret. If model privacy is a design constraint, you will need to either do some sever-side computation, or invent some other solution.
https://github.com/tensorflow/tfjs/issues/239
An update for those who still wonder about this issue: Tensorflow.js now utilizes WebAssembly under the hood. As noted above, this does not mean your code is inaccessible. Cloistered, obfuscated & unfriendly yes, yet the determined user can still capture, de-compile, & analyze that code. Granted, few users have both the capability & motivation to do so, but it's a big world out there...
I would like to write a 3D application using Golang, my favorite programming language. However, I would prefer not to use native OpenGL directly as the graphical frontend as this would entail a larger learning curve than I'm willing to tolerate. Additionally, I don't really want to use a Golang game engine like go:ngine.
After doing some research I found out about the amazing WebGL Javascript library three.js and I was so intrigued that I completed an introductory Javascript course in a few hours. I'm not really a web developer at all, so I'm wondering how practical it would be for me to write my application primarily in Go but with a three.js frontend.
Some specific questions:
Is it worthwhile / practical to use Javascript/three.js as a frontend to a Golang server like this?
If it is worthwhile, where can I look to learn how to integrate the two cleanly?
If it isn't really worthwhile, which alternatives do I have?
Thanks, any response appreciated.
Edit:
Do you plan to make operations on geometry in Go, possibly in realtime and communicate with your Javascript layer using Websockets?
Yes, this is what I'm thinking now. I'm intending my (Golang) program to generate streams of vertices and triangles based on a few parameters from the user. Each "structure" is generated all at once after the users supply their parameters (which I see as slider bars + input boxes on a Javascript frontend).
Here's an example of something the Go program might generate, plotted in GNUPlot:
(source: alexozer.com)
If this is the case you underestimate Javascript a bit. You should write most of your application in Javascript and use Go as a classical backend service layer like user accounts, persisting state, etc.
I'd be fine with doing that, except for these reasons:
The core generating process fundamentally depends on Goroutines and channels
I know close to nothing about web development, including frontend and backend
So I suppose I'm asking about the implementation details of one of these three possibilities:
Writing most of the program, including the generator, in JS, and a minimal backend in Go
Writing most of the program in Go and using JS just as a graphical frontend
Something else?
I recommend leaving out the whole Websocket aspect because it's only a possible optimization (and maybe not even that).
If you need to depend on Go's concurrency model then go for it, write your component in Go, then a Webserver in Go that takes parameters coming from an HTML request, uses them to compute the result and sends back the data in a JSON format.
On the frontend you will then only focus on sending this request when the user changed a parameter and for displaying of the JSON data you can use ThreeJS right away.
You'll still have to learn a bit of web development and Javascript though. But hey, they say the web is the future ;-)
I think it's the way to go for your case because your application operates in a quite strict request-result way.
The answer to your question depends a lot on where exactly you plan to draw the line between your Go component and the Javascript component. From the way you describe the problem I get the feeling that you want to write most of your application in Go and use JS only as a display layer? I wouldn't recommend doing that.
To rephrase my question: Do you plan to make operations on geometry in Go, possibly in realtime and communicate with your Javascript layer using Websockets? If this is the case you underestimate Javascript a bit. You should write most of your application in Javascript and use Go as a classical backend service layer like user accounts, persisting state, etc.
I might be able to give a more specific answer if I know what exactly you plan to do.
We are attempting to create a few dashboards for one of our Businesses, fetching data directly from the SQL server. We know what kind of business logic & display we'll like to do - having already built the prototypes from SQL data exported in Excel, and now - having prototyped their wireframes as well. The dashboard will include dynamic timeline based line & bargraphs & geographical map overlays.
We are trying to identify the best platform to do it. Here are the constraints:
The database is SQL based
The frontend should be browser accessible - hence a pure desktop application is ruled out.
While for the first usecase, the database is MSSQL - we'll like the platform to be say PHP based (or Python etc, but not ASP.Net)
Prefer to have the platform giving the option to host it on our systems (since our database server may not be online, but behind a company LAN - and hence not accessible from any public server, without port opening, etc), as opposed to it being only on online SAS.
Now, we are trying to architecture pieces. One approach is to have a technology stack consisting of:
Database - which we can live with what we have (our data isn't large enough to require anything beyond SQL).
An application logic framework (preferably MVC), along with a good set of libraries for Analytic applications (eg processing time series, statistical analysis - or some glue logic for calling, say R).
Frontend JS / HTML libraries. For this, we have looked at GoogleCharts, HighCharts, Paper.js, Raphael, Angular.js, D3.js, and new Razorflow. We haven't yet been able to figure out the best options amongst these.
Any suggestions / comments for the JS libraries (point 3) to use? And if in PHP - suggestions for good application framework + analytics library (point 2) to use above? Or if not, another language along with a good supporting framework for this?
Another alternative we're exploring is moving to a completely UX based dashboarding - eg qlikview, Tableau, or klipfolio. kibana - on the same lines looks interesting, but seems more suited for logs & text data, not exactly our use case.
Within the Microsoft world (we are trying to avoid), devExpress seems like a complete suite - with frontend JS to the backend database integration.
Any suggestions / comments on the right platform to use?
Arvind
I am the co-founder of RazorFlow. Are you still looking for a dashboard framework to use among the options listed? Our PHP Bindings are quite powerful allowing you to build your entire application in pure PHP and not worrying about the HTML/CSS parts of it.
Let me know if you want to discuss further by reaching out to our support email at support -at- razorflow.com
Preamble
So, this question has already been answered, but as it was my first question for this project, I'm going to continue to reference it in other questions I ask for this project.
For anyone who came from another question, here is the basic idea: Create a web app that can make it much easier to create other web applications or websites. To do this, you would basically create a modular site with "widgets" and then combine them into the final display pages. Each widget would likely have its own set of functions combined in a Class if you use Prototype or .prototype.fn otherwise.
Currently
I am working on getting the basics down: editing CSS, creating user JavaScript functions and dynamically finding their names/inputs, and other critical technical aspects of the project. Soon I will create a rough timeline of the features I wish to create. Soon after I do this, I intent to create a Blog of sorts to keep everyone informed of the project's status.
Original Question
Hello all, I am currently trying to formalize an idea I have for a personal project (which may turn into a professional one later on). The concept is a reflective web application. In other words, a web application that can build other web applications and is actively used to build and improve itself. Think of it as sort of a webapp IDE for creating webapps.
So before I start explaining it further, my question to all of you is this: What do you think would be some of the hardest challenges along the way and where would be the best place to start?
Now let me try to explain some of the aspects of this concept briefly here. I want this application to be as close to a WYSIWYG as possible, in that you have a display area which shows all or part of the website as it would appear. You should be free to browse it to get to the areas you want to work on and use a JavaScript debugger/console to ask "what would happen if...?" questions.
I intend for the webapps to be built up via components. In other words, the result would be a very modular webapp so that you can tweak things on a small or large scale with a fair amount of ease (generally it should be better than hand coding everything in <insert editor of choice>).
Once the website/webapp is done, this webapp should be able to produce all the code necessary to install and run the created website/webapp (so CSS, JavaScript, PHP, and PHP installer for the database).
Here are the few major challenges I've come up with so far:
Changing CSS on the fly
Implementing reflection in JavaScript
Accurate and brief DOM tree viewer
Allowing users to choose JavaScript libraries (i.e. Prototype, jQuery, Dojo, extJS, etc.)
Any other comments and suggestions are also welcome.
Edit 1: I really like the idea of AppJet and I will check it out in detail when I get the time this weekend. However, my only concern is that this is supposed to create code that can go onto others webservers, so while AppJet might be a great way for me to develop this app more rapidly, I still think I will have to generate PHP code for my users to put on their servers.
Also, when I feel this is ready for beta testers, I will certainly release it for free for everyone on this site. But I was thinking that out of beta I should follow a scheme similar to that of git: Free for open source apps, costs money for private/proprietary apps.
Conceptually, you would be building widgets, a widget factory, and a factory making factory.
So, you would have to find all the different types of interactions that could be possible in making a widget, between widgets, within a factory, and between multiple widget making factories to get an idea.
Something to keep on top of how far would be too far to abstract?
**I think you would need to be able to abstract a few layers completely for the application space itself. Then you'd have to build some management tool for it all. **
- Presentation, Workflow and the Data tier.
Presentation: You are either receiving feedback, or putting in input. Usually as a result of clicking, or entering something. A simple example is making dynamic web forms in a database. What would you have to store in a database about where it comes/goes from? This would probably make up the presentation layer. This would probably be the best exercise to start with to get a feel for what you may need to go with.
Workflow: it would be wise to build a simple workflow engine. I built one modeled on Windows Workflow that I had up and running in 2 days. It could set the initial event that should be run, etc. From a designer perspective, I would imagine a visio type program to link these events. The events in the workflow would then drive the presentation tier.
Data: You would have to store the data about the application as much as the data in the application. So, form, event, data structures could possibly be done by storing xml docs depending on whether you need to work with any of the data in the forms or not. The data of the application could also be stored in empty xml templates that you fill in, or in actual tables. At that point you'd have to create a table creation routine that would maintain a table for an app to the spec. Google has something like this with their google DB online.
Hope that helps. Share what you end up coming up with.
Why use PHP?
Appjet does something really similar using 100% Javascript on the client and server side with rhino.
This makes it easier for programmers to use your service, and easier for you to deploy. In fact even their data storage technique uses Javascript (simple native objects), which is a really powerful idea.