Develop Reference

Architecturally correct design pattern for Tornado/Bokeh interactive dashboard & workflow creation application with other javascript added on

I've been trying to build my own data analytics platform with the following features/workflow -
backend to generate/retrieve data from external systems (for eg. time series data using mqtt, websockets)
Tabbed Web application to -
A. Define workflows visually (i.e interactive directed graph/flowchart)
I would like to have an area on the screen where the user can see a selection of shapes (eg. rectangle) (i.e the transforms) join the shapes together to create a chain transforms that is to be run on the data
B. show results of transforms (also other charts from the original data) in a grid layout (i.e an interactive dashboard
I am new to most of this stuff, although I have worked on Django for my earlier projects. This time I would like to have an async, non-blocking architecture.
I have built my system up to a place where the MQTT based pub/sub system is complete, & the backend can send data if a request is placed for it
Here's my problem/confusion -
I have read up on the tornado docs, bokeh docs, examples (eg. the server_embed example) and also many many posts/questions/articles on how to go about doing this stuff
When i attempt to find a solution without using javascript, I have not come across any simple method/example of creating things like a good responsive grid layout, interactive flowchart or visual editor etc. etc. with pure python. Python seems to be best with the other stuff I need for my app, eg. pandas, numpy, blaze, odo, scikit, tornado.
However, whenever I look at javascript there is this huge universe of nice & beautifull stuff like jquery, JointJS, AngularJS & what-not that just seems so cool & easy to use
Plus I read an answer somewhere (quora?) on Jinja2 vs. AngularJS, which was along the lines of "You should not mix both because Jinja2=server side HTML generation vs. AngularJS=browser side HTML generation, which don't mix well)
All of this has led me deeper into confusion, because I'm new to most of this
So my questions are -
What is an architecturally correct pattern, example or some reference(s) on using
a. tornado for the server/webserver
b. Use bokeh for interactive charts for the original/transformed data
c. Use some nice CSS (or maybe SASS?)
d. Use some existing HTML5 framework such as Foundation for the layouts * responsive grids
e. Use something like JointJS in the same browser page somewhere to join blocks together to define the transforms
If I use javascript, vs. I don't, what does this architecturally mean for the tornado templating system
Can I, or should I use Jinja2 in this mix (i saw an example that explains how integration with Jinja2 is a better option for templating on tornado)

Use/productionize trained machine learning model in client-side javascript?

So this is a slightly insane idea that I've been kicking around. I'm thinking of building a classifier for (a specialized subset of) some web content, and then supplying it to users in the form of a chrome extension. (So: user goes to web page, clicks button on chrome extension, extension applies pre-trained model to generate prediction, presents result to user.)
But I'm cheap, and I don't want to run a server to receive data from the chrome extension, generate a prediction, and send it back to the user. It occurs to me: why not just find some way to serialize the trained model or something, and do it all on the client side.
So if I end up fitting a very simple model, like a logistic regression or something, then it would be easy to transfer to the client-side. Ultimately, a trained logistic regression is just expressible as an ordinary mathematical function mapping features to predictions, it's like one line of code. And I guess that's true of some fancier models as well (linear SVM). Under those circumstances, the chrome extension could do all the data cleaning and such, and then generating predictions is just a matter of calculating a dot product. But the same isn't true if I end up using, say, random forests or something.
Tragically, there doesn't appear to be a PMML library or anything like that for javascript.
So, I turn to the wisdom of the internet. Are there any other methods for productionizing ML models that might be usable on client-side javascript? Or are the options really just (1) run a server, (2) write a PMML library for JS, (3) there is no third option?
Thanks!

Do you have your models available in PMML data format already?
If so, then:
Translate PMML to Java source code. This is easy using JPMML libraries.
Translate Java source code to JavaScript source code. This is even easier using Google Web Toolkit.
As a result, you would have a self-contained JavaScript model object that exposes model schema query and scoring API similar to the JPMML-Evaluator library.

What helps store the inputted data on a backend page in HTML?

So I am new to web development and all I've learnt so far is how to write HTML and CSS to make web pages and forms.
I'm specifically looking for a language that will help me store the data that is input into the form onto a database for easy access later.
I think PHP does that, but I was looking for anything JAVA related, and somewhere I could learn how to do it.
So far,
I've looked into JavaScript tutorial on W3Schools, but it seems like JS only helps make the front end more dynamic, but doesn't help store the input data anywhere.
I've also looked into the AngularJS tutorial on CodeSchool and it looks like Angular helps present the forms in a better manner and again, not in storing data anywhere.
Once again,
1) What do you use to collect the data input into a HTML form and store it somewhere?
2) I don't need help making the actual form itself.

Me personally, i use PHP, it allows you to take the data that was inputted and store it in a database. You can also use JavaScript/JQuery with PHP via Ajax that will dynamically fetch and store data.

Any backend lang can do it (python, php, java, ruby, js with node), but each one has it own ways to do it. Even JS at client side can do it, using LocalStorage objects, but it not solve all problems at data storage yet.

Angular is just the client-side. You will need a server, which will handle the requests and store them into a database. I would work with Spring Boot (JAVA) on the server side, and Angular 2 on the client side. Use REST for the communication bewtween them. Its really not hard.

Node.js is a fast Javascript runtime combined with a low-level API similar to the standard libraries of many programming languages (file system access, buffers, streams, i/o, etc.).
Angular is a model-view-controller framework for client-side JS development. It can be used with Node.js as a backend, or anything else. Its main feature is 2-way data-binding, and addresses most of the concerns of a single-page web app within the framework.
React is often compared to Angular, because it is a front-end library, but it is not a framework. It is simply the view layer, with a large ecosystem of open-source projects supporting it. The big conceptual difference is a uni-directional data flow, rather than 2-way data binding. You need a lot more besides React to make a full application, but React handles it's use case exceptionally well. React is amazing on the client side, but it’s ability to be rendered on the server side makes it truly special. This is because React uses a virtual DOM instead of the real one, and allows us to render our components to markup. Node.js makes a great backend for React as well, but again, it can work with any backend.
The MEAN stack is a popular web development stack made up of MongoDB, Express, AngularJS, and Node.js. MEAN has gained popularity because it allows developers to program in JavaScript on both the client and the server. The MEAN stack enables a perfect harmony of JavaScript Object Notation (JSON) development: MongoDB stores data in a JSON-like format, Express and Node.js facilitate easy JSON query creation, and AngularJS allows the client to seamlessly send and receive JSON documents.
MEAN is generally used to create browser-based web applications because AngularJS (client-side) and Express (server-side) are both frameworks for web apps. Another compelling use case for MEAN is the development of RESTful API servers. Creating RESTful API servers has become an increasingly important and common development task, as applications increasingly need to gracefully support a variety of end-user devices, such as mobile phones and tablets.
This was the overview of all the new booming technologies.. Based on this you can decide what you need and what you want to learn.. Thanks, Hope this overview helps you to decide.

My favorite for Rest Api is Flask(python micro framework) it is build for create Rest Api. and for php falcon has it own micro frame work. if you use nodejs it is easy to communicate between backend and frontend and good with not sql dB like mongo db.

The right answer should be the database or localStorage/sessionStorage.
The decision in between which answer chose, depends if you want to share the data between Browsers/Computers or if you just want to temporarily store the data for the user so he won't need to fill the form once again.
For temporarily storage chose localStorage/sessionStorage (javascript).
For other cases chose to store the data in a Database.

Client-side or server-side framework?

My project would be a kind of craiglist, a site where users could post anouncements (evereday-life objects, cars, flat etc.). So, authentication, profile page, content creation, display the for-sale objects etc.
I have developed a very large part of the backend: I have a RESTful API in three-tier architecture developed in java. It makes the link with the db, to provide me with different urls and send me the relevant JSON.
URLs example:
http://api.mywebsite.fr/user?userid=1 sends me back:
{"user": {"username": "jdoe1234", "email", "jdoe1234#gmail.com"}}
I have urls for all actions performed on the entire site (anouncement creation, last data updates ... everything, and I've carefully declared them POST, GET, UPDATE, DELETE, etc.). There is also oAuth to protect the API from queries that are not allowed for the token.
That's all for the "server" aspect, I think that there is no problem with that.
But if all the actions are managed by the webservice, I do not see the interest that could bring me a big server-side framework like Symfony/cakePHP, Zend, etc., to make HTTP requests on my different entry points, retrieve JSON and populate the HTML.
So I looked at client framework, like Angular, Ember and so on. At first, it seemed very suitable for my case: possibility of http requests, manage what to do in case of success or error, directly exploit the resulting JSON to populate the view etc.
I didn't even manage to make my choice between angularjs and Ember, both being very similar, but with the release of Angular v2, I fear the maintainability of v1 (if I choose Angular, it will be v1 , because the majority of tutorials and questions relate to Angular 1.X).
I don't know if I'm doing the right thing by choosing client-side framework, I am afraid that they 'brident' (not sure of that word, sorry) me. Plus, it's fully instantiated in the browser, so the user can change absolutely all code and data I provide. That seems weird to me.
I want to be absolutely sure of the technology that I use in case I make this application available to the public for example. I want to do things properly, in order to avoid maintainability or security problems.
Summary: With the things I already have (webservice / api), is it a good idea to use a client framework like Angular or should I stay on big server-side framework like Symfony/Zend etc? Knowing that I position myself in the context in which this platform would be massively used (craiglist comparable traffic).

I'd say - depends whether you want to be more frontend guy or backend guy in future. If you want to be full stack developer then it doesn't apply.
In my opinion, both Symfony/Zend or other big server-side frameworks aren't so exciting as dynamic frontend JavaScript frameworks like Ember/Angular/React.
Also, if you have already RESTful API and OAuth authentication implemented in backend part I'd go with Ember. Why? Ember Data is great tool for talking to backend API. It's mature, it lazily loads records when they're needed and it's very customizable.
it's fully instantiated in the browser,so the user can change
absolutely all code and data I provide...
Ember has built in security like sanitizing data which is rendered in it's templating language - HTMLBars. Also, there's CORS and content security policy (CSP) standard which is implemented in Ember.
I want to be absolutely sure of the technology that I use in case I
make this application available to the public for example. I want to
do things properly, in order to avoid maintainability or security
problems .
In Ember you can create mature, secure, production-ready applications, but you need to comfortable with your Ember skills to some degree to build such ambitious web application, but it's part of building every application.
With the things that i already have(webservice / api), is it a good
idea to use a client framework like Angular?
Yes, it's very popular solution to use MEAN stack or go with Ember + RESTful API.
Why should I choose Ember instead of Angular (which have a larger
community/tutorials/answered questions) ?
Angular has larger community/tutorials/answered questions, but when I started some side project with Angular to learn its possible advantages over Ember, I was surprised how there was no consensus in it's community for doing one thing. So, instead of fast search how to declare and use directives (I think it was the thing that confused me) I have to do another research which way is the best. Also, there are lots of ways to setup project (where to put custom directives, different Angular objects) and you have to do another research which one to choose. I ended up using repo healthy-gulp-angular as my template, but you can see it hasn't been updated for 8 months, but I think during these 8 months Angular had a lot of changes and I'm not sure if this repo is the best choice.
In Ember you have Ember CLI tool which is built with Convention over Configuration principle. You have also Ember Data which utilizes JSON API standard - if you don't have JSON API compliant server side right now, you can write custom adapter to normalize server responses or change how backend replies. In Ember you don't have all that headache and different best solutions to do 1 basic thing depending who you ask.
What means "Single page application" ?
Single-page application is basically a page which doesn't have to reload all assets and HTML when you navigate. It's advantage over PHP - when user moves to another location he downloads only new data for that route. More info here.
Does those frameworks allow me to create real routes ? (
www.myapp/profil/userid etc )
Yes, of course. You don't even need # in your URL. With simple rewrite rule and small amount of logic for profile route and specified path profile/:userid, when user will open URL www.myapp/profile/userid he will be automatically taken to profile route, and userid would be interpreted as route parameter, so you can take this userid and find user record from the store in model hook.

Client = speed, Server = stability
JS frameworks updates once per week
Non-Js back-end once per year
Client side depends to behavior depending on browser
Back is related only on machine but not on environment
I chose FE coz I tired to debug code by writing variables values to database to actually see what is going on in controllers -_-

Reflective Web Application (WebIDE)

Preamble
So, this question has already been answered, but as it was my first question for this project, I'm going to continue to reference it in other questions I ask for this project.
For anyone who came from another question, here is the basic idea: Create a web app that can make it much easier to create other web applications or websites. To do this, you would basically create a modular site with "widgets" and then combine them into the final display pages. Each widget would likely have its own set of functions combined in a Class if you use Prototype or .prototype.fn otherwise.
Currently
I am working on getting the basics down: editing CSS, creating user JavaScript functions and dynamically finding their names/inputs, and other critical technical aspects of the project. Soon I will create a rough timeline of the features I wish to create. Soon after I do this, I intent to create a Blog of sorts to keep everyone informed of the project's status.
Original Question
Hello all, I am currently trying to formalize an idea I have for a personal project (which may turn into a professional one later on). The concept is a reflective web application. In other words, a web application that can build other web applications and is actively used to build and improve itself. Think of it as sort of a webapp IDE for creating webapps.
So before I start explaining it further, my question to all of you is this: What do you think would be some of the hardest challenges along the way and where would be the best place to start?
Now let me try to explain some of the aspects of this concept briefly here. I want this application to be as close to a WYSIWYG as possible, in that you have a display area which shows all or part of the website as it would appear. You should be free to browse it to get to the areas you want to work on and use a JavaScript debugger/console to ask "what would happen if...?" questions.
I intend for the webapps to be built up via components. In other words, the result would be a very modular webapp so that you can tweak things on a small or large scale with a fair amount of ease (generally it should be better than hand coding everything in <insert editor of choice>).
Once the website/webapp is done, this webapp should be able to produce all the code necessary to install and run the created website/webapp (so CSS, JavaScript, PHP, and PHP installer for the database).
Here are the few major challenges I've come up with so far:
Changing CSS on the fly
Implementing reflection in JavaScript
Accurate and brief DOM tree viewer
Allowing users to choose JavaScript libraries (i.e. Prototype, jQuery, Dojo, extJS, etc.)
Any other comments and suggestions are also welcome.
Edit 1: I really like the idea of AppJet and I will check it out in detail when I get the time this weekend. However, my only concern is that this is supposed to create code that can go onto others webservers, so while AppJet might be a great way for me to develop this app more rapidly, I still think I will have to generate PHP code for my users to put on their servers.
Also, when I feel this is ready for beta testers, I will certainly release it for free for everyone on this site. But I was thinking that out of beta I should follow a scheme similar to that of git: Free for open source apps, costs money for private/proprietary apps.

Conceptually, you would be building widgets, a widget factory, and a factory making factory.
So, you would have to find all the different types of interactions that could be possible in making a widget, between widgets, within a factory, and between multiple widget making factories to get an idea.
Something to keep on top of how far would be too far to abstract?
**I think you would need to be able to abstract a few layers completely for the application space itself. Then you'd have to build some management tool for it all. **
- Presentation, Workflow and the Data tier.
Presentation: You are either receiving feedback, or putting in input. Usually as a result of clicking, or entering something. A simple example is making dynamic web forms in a database. What would you have to store in a database about where it comes/goes from? This would probably make up the presentation layer. This would probably be the best exercise to start with to get a feel for what you may need to go with.
Workflow: it would be wise to build a simple workflow engine. I built one modeled on Windows Workflow that I had up and running in 2 days. It could set the initial event that should be run, etc. From a designer perspective, I would imagine a visio type program to link these events. The events in the workflow would then drive the presentation tier.
Data: You would have to store the data about the application as much as the data in the application. So, form, event, data structures could possibly be done by storing xml docs depending on whether you need to work with any of the data in the forms or not. The data of the application could also be stored in empty xml templates that you fill in, or in actual tables. At that point you'd have to create a table creation routine that would maintain a table for an app to the spec. Google has something like this with their google DB online.
Hope that helps. Share what you end up coming up with.

Why use PHP?
Appjet does something really similar using 100% Javascript on the client and server side with rhino.
This makes it easier for programmers to use your service, and easier for you to deploy. In fact even their data storage technique uses Javascript (simple native objects), which is a really powerful idea.