I have an Apache server A set up that currently hosts a webpage of a bar chart (using Chart.js). This data is currently pulled from a local SQLite database every couple seconds, and the web chart is updated.
I now want to use a separate server B on a Raspberry Pi to send data to the server to be used for the chart, rather than using the database on server A.
So one server sends a file to another server, which somehow realises this and accepts it and processes it.
The data can either be sent and placed into the current SQLite database, or bypass the database and have the chart update directly from the Pi's sent information.
I have come across HTTP Post requests, but not sure if that's what I need or quite how to implement it.
I have managed to get the Pi to simply host a json file (viewable from the external ip address) and pull the data from that with a simple requests.get('ip_address/json_file') in Python, but this doesn't seem like the most robust or secure solution.
Any help with what I should be using much appreciated, thanks!
Maybe I didn't quite understand your request but this is the solution I imagined:
You create a Frontend with WebSocket support that connects to Server A
Server B (the one running on the raspberry) sends a POST request
with the JSON to Server A
Server A accepts the JSON and sends it to all clients connected with the WebSocket protocol
Server B ----> Server A <----> Frontend
This way you do not expose your Raspberry directly and every request made by the Frontend goes only to Server A.
To provide a better user experience you could also create a GET endpoint on Server A to retrieve the latest received JSON, so that when the user loads the Frontend for the first time it calls that endpoint and even if the Raspberry has yet to update the data at least the user can have an insight of the latest available data.
Related
I'm coding a private webapp that can pick some data from a source (yfinance, panda, ...) and generates a time series. Basically, I want to be able to click on any of the Dow Jones stocks with an HTML btn or whatever, which will send a request to my backend server to generate such chart using python, R or any alternative.
Two possibilities:
I generate the chart and figures on the client's side, by sending the data from the server to the client on a JSON format (which seems really unproductive).
Or, I make all computations on backend, following an HTTP request from the client, generate the chart and send it to the client (my computer).
The question is, can I send such chart, from the server to the client without uploading it as a png and transmit the link through AJAX etc. May I get the output of a python function and send it ?
To summarize: Can a .png file might be sent from a server to an online webapp, any ideas ?
Thanks for your comments, this is a technical question and this might be useful for future requests. Also, I plan to code everything using Django framework.
Planning everything so nothing has been built yet.
so currently i am studying web development but the course was a little bit confusing, the teacher starts explain promise object and fetch then axios, after that he starts to talk about the "express" package to build server side, and am asking myself what is the different between using API objects & building server side using express?
Both of those things work together to create a website/app. axios, etc runs in the browser (client-side code), and is used to send requests to the server (express/other server-side code), allowing the server to fetch data, or modify the database, and return a response back to the browser side code, which continues operating from there.
imagine, client-side code does not have access to your server's database, as your database is on the server, and the browser is on the end user's computer. The two sets of code send messages back and forth to each other to create an app that has a centrally stored data store that is "shared" between users.
I recently learned about webhooks and am trying to implement one for a full-stack app that I'm building. However, I haven't been able to find information on how to create my own webhook. This is how my app is expected to work:
My client side is a website built with React, and when it loads, it will make a GET request to my server (built with Node and Express). The response will provide data that will be displayed on the website. The user will click a button on the website, which will send a (POST? GET?) request to my server, which will make its own POST request to send some of the data to an external API. When that external API data completes work on the data, it will send a POST request to my webhook URL (which I assume is part of my server), and the data from the external API needs to be sent to the client, which will display it alongside the original data. Additionally, when work on the first batch of data is completed, my server sends another POST request (with some more of the original data) to the external API, which will then make a POST request to my webhook/server with new data that then gets passed to the client again, and this repeats until all the original data from my server has been passed to and processed by the external API, and the all the processed data gets sent to my server and then to the client.
From my understanding, the webhook will allow my client to make a single request to my server, and then, my server can make multiple response to my client: one response for each POST request received from the external API. However, I have not been able to find information on how to create a webhook. Everything that I have found has been about how to use webhooks with pre-existing apps/websites like Dischord or GitHub or Stripe. Or using 3rd-party services to use their webhooks to connect pre-existing app/websites.
How do I go about making a webhook for my little project that will update my website every time that my server receives a POST request from the external API? Or am I misunderstanding how a webhook works? Can they only be created by 3rd-party services and/or only used on pre-existing apps?
For a simple use case, it just means another endpoint on your server that will receive a request from the external API when it's done with the task. It's a common way REST APIs work to let you know they are done with something, or communicate anything back to you in general. You have one endpoint that will receive a request from the React client, and send information off to the external API (as you said), and another endpoint (this "webhook") that will receive requests from the external API. That's it.
You described the client-server communications for this problem just fine with the exception that the server cannot send requests to the client over HTTP. If the client needs anything from the server, it needs to send a request and ask for it. A common way to do for this would be to store the information on the server once the external API hits your "webhook", and the client can ask the server if the data is back every X seconds (long-polling).
Example (payment flow):
Client -> Server: Heres my credit card number
Server -> Client: Thanks! Let me process that.
Server -> External API: Please process this credit card for the amount of $X and ping me at this webhook when you're done.
External API -> Server: OK, will let you know when I'm done.
Client -> Server: You done yet?
Server -> Client: Nope!
External API -> Server (to webhook): Hey, I'm done, here's the receipt.
Server -> External API: Thanks!
Client -> Server: You done yet?
Server -> Client: Yes! Here's your receipt.
More advanced ways to solve the problem would be using a long-lived bidirectional communication protocol such as websockets. And as others have mentioned, all of the endpoints don't have to be on the same server. You can have one server for each endpoint, and you can have turn them on only when they get a request and then go back to sleep (that's a Lambda). Pros and cons to this approach. That's more advanced stuff and if you're just learning I wouldn't really worry about it now.
AFAIK, webhooks are just normal web APIs. Properties that make an API a webhook:
The client sending requests to your server, in this case, is another server
This client will send you a request when there's a triggered event
In your case, your server POSTing data to the external API. What makes webhook different is that the external API's server doesn't have to respond right away, instead, it sends back you, for example, an HTTP 200 OK to acknowledge that it received the request and has started to process the data. When the external API's server finishes processing the data (which is an event), it will call your server's endpoint to inform you about the process along with the processed data (if succeeded).
Your server, correspondingly, will then have to inform your client (the user) about the processed data. I would suggest implementing some kinds of persistent connection (WebSocket or SSE), or simply just make the client poll request every some second.
"(which I assume is part of my server)"-- The webhook is part of the third party API that points back to either your server or maybe some serverless function/lambda(that you set up) that in turn can parse and perform computations on the data and then send it back to your server.
It really depends on the use case and architecture of your API. If you don't want your main API server to get overloaded, if say you need to perform heavy I/O, CPU/GPU processing, you could create Serverless Lambda Functions to handle this data processing and be the endpoint you supply to the third party's webhook post endpoint.
If you're not concerned with it, you could just create another endpoint in your server to handle this webhook.
Just keep in mind, when you're trying to scale up, you want your systems to be as distributed as possible-- within reason-- so your API can handle more requests and not be overloaded with a bunch of processes.
What am I working on?
I am trying to establish a communication between a PHP app and a telephony system's REST API.
Since I my site is all written in PHP, I decided to build the communication using PHP by making cURL calls to the API.
to bring you up to speed, there are 2 types of communication between the user and the API and I like to put them into two different categories
Send Once / Receive Once Example of this would, be a user attempt to dial a new phone number "dial 800-123-4567." The API takes the request and return back an interaction id to allow the user to control the call (i.e. disconnect, mute, put on hold.... )
Send Once / Receive Every Second In this communication, I will create "persistent" connection between the user's session and the API. Then every second, I will check the API for new messages. After the message from the API is received, I must update the user's cache, read the latest user's cache, and finally send the browser the cache data.
Problem? HTTP is stateless.
Every request the user send to the web server, it generates a new TCP connection. The issue with this is that every second I query the API for new messages I will have a new TCP connection. On average about 200 TCP connections are needed at any giving time per user. So if I have 300 users using the app/server, then that is about 60,000 TCP connection open for the web server. As you can clearly see the solution does not scale well here and it is a matter of time before the server blow up in my face... :(
Another issue is that PHP is not asynchronous which cause problem if the communication to the API took longer or return errors.
FWIW, I have tried to user JavaScript SharedWorker to eliminate some of the overhead. I every tried Server-sent-events but a user still generated too many TCP connections to the server. nothing first the problem I was only able to reduce the connection a little.
Can Nodejs help?
I was advised by couple of people to use Nodejs instead of PHP for this task. Of course, I am not going to change my PHP application into Nodejs as this would be insane since my app is huge.
I would like to consider running a nodejs server as a middle man between the PHP server and the API service. The idea is to have a WebSocket running on the node server. Then, the client will pass any communication to the websocket and the websocket will then send the communication to the server. It does not sound bad at a high level but once a dig deeper, it seems to be getting trickier.
Nodejs Challenge
When a user logs into my PHP App, I validate their credentials and once they are in then I create a session which is stored into MySQL database. A session to be valid the following must be correct
IP Address must match the Ip which created the session
The agent data must also match (I can live without it for nodejs)
The idle time of the session must be less that 900 seconds.
In order for Nodejs to start communication it must first create a new connection to the API. After the connection is accepted, nodejs must keep track of the following data "received by the API"
CSFR token
Session Id
Http Cookie
In order for Nodejs to make a connection to the API it must pass a username, password, server name, port, and a station name. I have all the needed info stored into MySQL database and I can easily get that using PHP.
The challenge is that NodeJS have to take the PHP session, validates it, pull the API needed info from the database then establish connection to the API.
Questions
Can nodejs use the PHP session to validate the user? If so how?
How will can nodejs use the TCP connection to prevent me from overloading the server?
This is your arrangement:
[User browser] -> [PHP] -> [Node.js] -> [API]
When your user's browser sends a request to your PHP server the request includes a cookie - one of the values of this cookie is the session id which PHP then uses to look up the session. The session id acts like a password, PHP will assume that if you've got the session id then you are the original user that was issued that id.
When your PHP script communicates with Node it needs to pass along that session id as part of the request. In Node you just then need to do a lookup of your sessions table in MySQL for the corresponding session.
PHP session data is stored as the $_SESSION array serialised. To extract data from it you will need to unserialise it first. There are a number of libraries out that can provide this functionality (e.g. https://github.com/naholyr/js-php-unserialize, https://github.com/kvz/phpjs/blob/master/functions/var/unserialize.js). However if the session data is simple and conforms to a known format you could 'hand parse' the data.
I'm currently planning to develop a HTML5 app. The basic concept is the following:
A user should be able to create a profile with username and password. The Server should be implemented in Ruby on Rails providing a JSONP Api (for Cross-Domain issues).
So the App will send Ajax requests to the Server and get responses from it.
My idea was now to transmit a session_key (generated by server) on the first response back to the client. Then the client has to authenticate himself with this token.
But now i have some issues.
How can i secure the first call of the client (when he is transmitting user and password)?
How can i protect the Session-key from beeing spyed out?
I am a complety noob in security aspects. Therefore it would be great if i could get some hints where to look at.
Secure your connection with SSL. This should require no changes in your code apart from putting 's' after 'http' ;-).
I used add a checksum to the ajax parameters (calculated using the submitted data), and then to crypt the hole ajax request into one string.
Somthing like sRequest=459fdjnfdw4r908vn....
sRequests holds my data (sUser=user&sPass=pass&iCheck=34564).
Edit: My client code was not public, compiled to an app.