I've made a very simple Twitter-like web app to teach myself how to work with React, and how to actually get something working in practice. I've deployed it on a digital ocean droplet and it works just fine.
Now, I am currently studying distributed systems and I've become very interested in them (i could see myself working with them after I graduate). So I figured how to make my app "distributed" by running it on several servers simultaneously and have clients connect to different ones based on some arbitrary criteria. Obviously, this is all very overkill for what my app does, this is purely out of interest and because it sounds like a challenge.
The problem is that I don't really know how to start. I guess I would have to set up some sort of redirection service/replica manager that clients connect to at first, before their requests are sent to the chosen replica?
Any hints/tips for starting out would be greatly appreciated!
You could use a load balancer like the one digital ocean has. If you want to create your own:
I would create a server which would act as the balancer end. This server would keep track of the other servers and redirect clients to them. All the requests would go to that server and it could pass them on with some kind of logic. For example, you could pass the requests in a circle, one to each server and from the start again. You could also account for the response time of each server and give weights to them.
You can find many tutorials in google too:
How to Build a Load Balancer with Express
Related
I am working on an app that uses machine learning, and recently run into a problem. I found out that Expo and Python do not really work together.
I thought that a solution would be to host and run my machine learning model (Python) on a server, and then make requests from my JS Expo app, and I thought this would also have the hidden bonus of faster processing compared to just on my laptop.
So, can people confirm whether firstly this is possible (to call python code running on a server, from a JS Expo app) and if so, then provide some recommendations as to where I can host this?
This is for a university project, so I am hoping to find something either free or cheap, thanks.
I thought this would also have the hidden bonus of faster processing compared to just on my laptop.
If you want to do it for free then lowest tier instances on any cloud will have 1 or 2 cores plus 1-2 GB of RAM, it won't be stronger than your local device and it won't have any hardware acceleration, but if you planned to run that on your phone then it might be enough
So, can people confirm whether firstly this is possible (to call python code running on a server, from a JS Expo app)
Short answer yes, but
The way you formulated this question does suggest that you don't know how this communication would work. You would need to
send http request from expo app to some endpoint on your server
implement server code that could accept that request, in your case it's probably best to do that in python(with sth like django or flask)
in your server code execute tensorflow code
if tensorflow code executes quickly you can send results in the response if not you will need to have second endpoint to check results
This is for a university project, so I am hoping to find something either free or cheap, thanks.
for local development you will need to host it locally either way, for university project it should be enough, but if you want to have it on public server you can get free tier or some starting credits on almost any cloud service e.g. digital ocean gives 100$ credit for students
Alternatively you can use tensorflow in react-native app, you just can't do it in python, https://blog.tensorflow.org/2020/02/tensorflowjs-for-react-native-is-here.html?m=1
Currently I have a Flask server that runs a small web frontend as well as a command line interface to that same server. The basic idea looks like this:
<Top section: allows file upload>
* list of files from database
<Second section: allows file manipulation/ upload>
* list of files from database
<Third section: Not files, but still allows for database changes>
* list of things made from database
Now this works well from the front end, but currently if the CLI or another client makes a change to the database, it doesn't update other clients. I have it somewhat working with JS polling and rewriting the list of files every 10s, but that seems both inefficient and also would look very messy if I had to do it for every section. I saw websockets mentioned in various forums, but I've never used them and am unsure if it would be a pain to add. I'm not trying to rewrite the whole thing for a single feature.
Final takeaway: How to update all clients better than polling/ how to do polling efficiently?
Yes you are correct. You need sockets. There are bunch of articles over the internet but I would like to give a summary and try to explain why sockets will be the best fit to your requirements.
Sockets are way of achieving two way communication between client and server without the need of polling.
There is a package called Flask-SocketIO
Flask-SocketIO gives Flask applications access to low latency
bi-directional communications between the clients and the server.
Then for the scenario where you would like to send changes to all the connected client when one client does some work to your database or something similar, you will need to use broadcasting. When a message is sent with the broadcast option enabled, all clients connected to the namespace receive it, including the sender. Here you can find details of the broadcasting using Flask-SocketIO.
So, I've web application which structure is based on this file structure: https://scotch.io/tutorials/setting-up-a-mean-stack-single-page-application .
My app also has a connection to mongoDB on Mlab.
what my app does:
allows users to login/signup;
retrieves data from mlab;
retrieved data can be rated by users;
retrieved data can be deleted by admin;
users can add data to db (data is training plans);
Now I need to make my app horizontally scalable, but I am a bit lost here:
•Sine I assume there i no real-time activities I shoudn't need something like socket.io?
•Should I add some sort of MQ (rabbitMQ, ZMQ, etc.): If so, perhaps any pointers on how to, because most of the examples just use simple text messages.
•I am quite sure I would need some load balancer. Nginx, HaProxy... I probably should change my express server setup to listen to multiple ports first, is that right?
Or am I completely wrong about this?
P.S.: Hope this isn't too broad question.
Different needs require different approaches :)
These can vary according to your needs. Not every scalable application has to have them.If you want the application to be asynchronous, you can take all the requests in a queue and return to the client instantly.You may then need a push mechanism to notify the client that the operation is over. (Socket.io, RabbitMQ etc)
Of course you will need a reverse proxy to distribute requests to different servers load balanced or workload basis (HAProxy etc.)
The first thing you need to pay attention to when you want to scale the application is to have a stateless structure.Or get them out of the process.(For example session, cache, file server)The second thing you need to be aware of is the authentication phase.A client that logged in from ServerA may encounter "unauthorized" on ServerB on subsequent requests.You should also think about the resources used by the application.While these resources serve a single server, they will begin to respond to millions of requests from five to ten servers simultaneously.There are things like monitoring instances.And a lot of things like that.
These are the things you should really think about :)
I am working on a home automation hub -- a Raspberry Pi running locally that displays weather info, controls my lights, etc. It is "networked" (and I use that term loosely) to a website via a shared MongoDB. Both the site and the hub are running Node.js/Express servers.
Essentially, I am looking to be able to enter text into a field on my website and then display it on my hub.
I'm struggling to figure out how to pass data between them. I can think of a couple ways that might get it done, but the only way I know I could get working is to implement some sort of Mongo watcher/listener to watch for changes on a specific collection. Essentially, you enter the text into the site, that updates the document in Mongo, the watcher informs the locally-running hub, which then fetches and displays the new content.
This seems hacky. Is there a better way? Is this something socket.io could manage? Maybe I'm overthinking it? Help!
You can use Socket.io, WebSocket or TCP socket to connect the two servers together and communicate that way. Or you can use a queue system like ZeroMQ or RabbitMQ and communicate that way. Or you can even make an HTTP request from one server to the other one every time you want it to grab new data - or you could even sent that data right in the request.
It would be much easier if you used Redis that supports pub/sub, see:
https://redis.io/topics/pubsub
or CouchDB that supports the changes feed:
http://docs.couchdb.org/en/2.0.0/api/database/changes.html
or RethinkDB that supports changefeeds:
https://rethinkdb.com/docs/changefeeds/javascript/
I don't think Mongo supports anything like that.
I'm creating a personal project, an Android application, where user sign ins via a server are essentially. Also users will be able to share important "updates" in both a data stream that all users can have access to, and potentially down the line a data stream that will be specific to local users.
Unfortunately I have basically no idea of how to implement the server for this application. I'm confident that I'll be able to create a solution though, I just need to be pointed in the right direction. Are there any existing servers/api's that I could access that will allow me to handle my "connected" tasks? If I were to make a server myself for this application where should I start?
If you have absolutely no experience with server side coding, you should start with PHP and MySQL to create a simple API for your app.
For getting started download the XAMPP bundle that has everything you need.
SQL and PHP are fairly easy for beginners. On the client(Android) side use Volley as it is simple to understand.
However if you want to get started with Node.js, your server can be up in a few minutes with express.js