I'm currently working on a ML project with geo data...
In my web I let the models parameters of the machine learning algorithm to the user, then I send those to an Apache server where PHP gets the parameters... In js I use Ajax to make the POST request.
My ML algorithm is made with Python, right now is working with the library argparse to read the parameters that PHP sends (after of a verification) as command trough the function exec()
I have 2 problems with this:
If the ML model takes time to calculate the results, the exec function does't wait for them and returns null after some time, but if it's fast everything is ok. I already have the function set_limit_time(0); in my PHP file.
In my local machine it doesn't take so much time to calculate results like on the server, but the server has better hardware, so I don't know what is going on there.
PHP 7.0.15
Python 2.7
Server Apache/2.4.18 (Ubuntu 16.04.1 LTS)
Also, is there a better way to do this?.
I'd like to suggest another approach.
Whenever you call your algorithm from the command line, you have a bootstrapping time: importing libraries (such as numpy), loading datasets, etc. Then you perform your calculations, return a response and clear the memory. So the next time you need another result, you have to go through all the process again.
I'd suggest embedding that algorithm inside a small Flask application (you already know Python and know how to use PHP for web, it shouldn't be that hard.
Since your python web server will have all the libraries and the datasets already loaded, it would be much faster in answering your questions.
And you can access it from PHP doing an HTTP request with curl (it has timeout!).
I think this would be much easier and scalable.
Just my two cents!
I ended up opening a Python Socket and connecting PHP trough it, now the process it's still slower than my local machine but the timeout is not longer a problem.
Related
I have an idea to make something similar to Workflowy but with some new features.
(Workflowy is basically a note-taking app which beautifully organises all your notes as an endless tree)
At first, I implemented the logic in Python. It works in a terminal by printing notes line-by-line and then waiting for the command.
Is this a good idea to keep all the logic at the server and use JS only to render items and to send commands to the server?
For instance, if I want to move the entire folder into another folder, there are two ways of doing this:
Way 1: With Python which receives a command from JS 'move folder x to folder y', processes it and sends back a result to render.
Way 2: With JS which then has to understand all the folder structure and logic. In this case, the app will use a server only for storing data.
I have a feeling that way 2 (using JS to understand all the logic and Python only for saving data) is more appropriate, but this means that I have to rewrite everything from scratch.
Is the way 1 also reasonable?
Many thanks in advance!
It depends on the application you are making.
Like if you want to display thousands of data in html file, and data are stored in a json file. If you send html file and json file to the client from the server, then on the client side, you run a script that reads json file and displays it in html, then it will be slower, because client device may not be that powerful as the server is.
So for performance, use heavy tasks on server side, this may cause little more internet usage because as the client has no data in formatted manner, whenever new task on data is to be performed, you have to request the server again.
But for opposite case, you can save internet and little low performance. Here you can do some heavy tasks on client side.
It also depends on which device is used at client side.
The Goal
I have two processes running:
Node.js server which handles external communication (to other computers)
C code which runs in an infinite loop doing real-time processing
The node.js server will receive data from an external computer, send it to the C code (which is already running), wait for a reply, and send that reply back to an external computer.
HOWEVER, the C code cannot wait for input from the node.js server. It needs to be continuously running using the most recent data from the js code (updating those recent values each time new data is received). In other words, the C code needs non-blocking input from the node.js server. The node.js server should get an event (bound to a callback) each time the C code periodically sends it information back.
What I have done
The node.js code already handles external IO correctly (it's fairly complete)
The C code in isolation does the task I want, just without getting data from the node.js code
I am now trying to combine them so that the C code gets its sensor data from the node.js code (as I described above).
How I think I can do it
My current best guess of how to do this is using sockets (I think they're non-blocking?). I've never done this before, though, so before I spend a bunch of time learning about how sockets work, and writing a bunch of code, I want to make sure that this is a method which might actually work. So, I want to see if anyone has any suggestions on the easiest way to achieve what I described. If it does turn out to be sockets than at least I know I'm not wasting my time. Thanks for any suggestions!
So- I have some python scripts running permanently on my VPS and I want to be able to send them data to chew on via a simple web interface.
I do not mind how it gets the data. It seems to me that the simplest is probably to write the data to a file, and the python script automatically reads from this file every X seconds for changes. Then the solution need not involve python in any way.
From reading elsewhere, I believe the simplest solution is a simple php script. The problem here is that I'm running a pm2 server on the VPS. Pm2 does not support php I believe- and I can't install LAMP as this would presumably interfere with pm2.
I've looked into trying to do this with JS but that saves files client side and I need it server side.
I've also looked into cgi (I have zero experience with this) but pm2 doesn't seem to support this either?
Is there some fancy solution using node js?
What is the best way to tackle this problem!?
I'm developing a web application with Symfony2. I need to create a push notifications sysmte (like Facebook). When an user publish something, I need some of another users receive a notification.
I saw that Node.js it's the easiest manner to do this. But, I did some simple examples and all works fine, but I don't know how can I integrate this node.js application with my Symfony2 application, or really with a PHP application.
Anybody can help me?
Thanks in advance!
Please note that you've not given enough details, so I will respond as a front-end developer and not as a mobile developer
Integrating NodeJS and PHP (in general) is not a good way since you need to launch both servers separatly, create the websocket server in JS while your application is in PHP and finally create a request (GET or POST) from your PHP to you JS server. Well, a big mess, so I'll expose my solution here under.
Quick insight for mobile apps. Well, technically, there's no easy way. You can use the Push "protocol" (http://www.wikiwand.com/en/Push_technology) with NotificationPusher (https://github.com/Ph3nol/NotificationPusher). I didn't used it before so I can't help you with it.
In general.
Most of the time when people thinks of Push, long polling will do the trick. For starters it means that the request is made client-side and the server don't send data & close connection until there's new data.
How do you implement this ?!?
Basically, you change the max_execution_time using ini_set or set_time_limit to a very long time for the current script and launch a loop (like a do..while) with a sleep and the check to your data inside. From your Javascript just make an Ajax call, for example with jquery: $.get. Just remember to remove the timeout and stay in asynchronous mode.
The only drawback of this solution is that you will always have a connection opened to your server which will consume a bit more of battery on a mobile device. If you have multiple types of data to receive do not hesitate to merge the calls and publish a type in your response data, since most of the browsers allows only 2 or 3 simultaneous connections to the same server.
I sounds like your describing WebSockets.
Take a look at Socket.io, its a module for node.js.
Also there is a example at GitHub https://github.com/Automattic/socket.io/tree/master/examples/chat
Interesting files for you should be the index.js and the public/main.js.
You can see the example live at http://socket.io/demos/chat/
I just started learning Node.js and as I was learning about the fs.watchFile() method, I was wondering if a chat website could be efficiently built with it (and fs.writeFile()), against for example Socket.IO which is stable, but I believe not 100% stable (several fallbacks, including flash).
Using fs.watchFile could perhaps also be used to keep histories of the chats quite simply (as JSON would be used on the spot).
The chat files could be formatted in JSON in such a way that only the last chatter's message is brought up to the DOM (or whatever to make it efficient to 'fetch' messages when the file gets updated).
I haven't tried it yet as I still need to learn more about Node, and even more to be able to compare it with Socket.IO, but what's your opinion about it? Could it be an efficient/stable way of doing chats?
fs.watchFile() can be used to watch changes to the file in the local filesystem (on the server). This will not solve your need to update all clients chat messages in their browsers. You'll still need web sockets, AJAX or Flash for that (or socket.io, which handles all of those).
What you could typically do in the client is to try to use Web Sockets. If browser does not support them, try to use XMLHttpRequest. If that fails, fallback to Flash. It's a lot of programming to do, and it has to be handled by node.js server as well. Socket.io does that for you.
Also, socket.io is pretty stable. Fallback to Flash is not due to it's instability but due to lack of browser support for better solutions (like Web Sockets).
Storing chat files in flatfile JSON is not a good idea, because if you are going to manipulating the files, you would have to parse and serialize entire JSON objects, which would become very slow as the size of the JSON object increased. The watch methods for the filesystem module also don't work on all operating systems.
You also can't compare Node.js to Socket.IO because they are entirely different things. Socket.IO is a Node module for realtime transport between the browser and the server. What you need is dependent on what you're doing. If you need chat history, then you should be using a database such as MongoDB or MySQL. Watching files for changes is not an efficient way and you should just send messages as they received.
In conclusion no, using fs.watchFile() and fs.writeFile() is a very bad idea, because race conditions would occur due to concurrent file writes, besides that fs.watchFile() uses polling to check if a file has changed. You should instead use Socket.IO and push messages to other clients / store them in a database as they are received.
You can use long pooling method using javascript setTimeout and setInterval
long pooling
basically long pooling working on Ajax reqest and server responce time.
server will respond after a certain time (like after 50 seconds ) if there is not notification or message else it will respond with data and from client side when client gets response client javascript makes another request for new update and wait till response this process is endless until server is running