I am using Meteor for a web application project and am trying to send a file using the HTTP library and get back a JSON object using a get request. I need to send a POST request to the echonest server where the file is sent as a binary data object. From there, I need to make two GET requests: one to get the url and one to get the data at that url. The last two steps are not a problem, but I'm finding it very difficult to send the raw binary data. Here's what I've found so far:
The Meteor HTTP library cannot send binary data in a POST request because it calls JSON.stringify on the data being sent.
An XHR cannot be used to send the data from the client because the server I am trying to communicate with blocks client-side connections to their service.
So what I want to know is if there is another library/package that can be used with Meteor to send the binary data?
Alternatively, is there a way to get an XHR request to work on Meteor server-side?
Related
I recently learned about webhooks and am trying to implement one for a full-stack app that I'm building. However, I haven't been able to find information on how to create my own webhook. This is how my app is expected to work:
My client side is a website built with React, and when it loads, it will make a GET request to my server (built with Node and Express). The response will provide data that will be displayed on the website. The user will click a button on the website, which will send a (POST? GET?) request to my server, which will make its own POST request to send some of the data to an external API. When that external API data completes work on the data, it will send a POST request to my webhook URL (which I assume is part of my server), and the data from the external API needs to be sent to the client, which will display it alongside the original data. Additionally, when work on the first batch of data is completed, my server sends another POST request (with some more of the original data) to the external API, which will then make a POST request to my webhook/server with new data that then gets passed to the client again, and this repeats until all the original data from my server has been passed to and processed by the external API, and the all the processed data gets sent to my server and then to the client.
From my understanding, the webhook will allow my client to make a single request to my server, and then, my server can make multiple response to my client: one response for each POST request received from the external API. However, I have not been able to find information on how to create a webhook. Everything that I have found has been about how to use webhooks with pre-existing apps/websites like Dischord or GitHub or Stripe. Or using 3rd-party services to use their webhooks to connect pre-existing app/websites.
How do I go about making a webhook for my little project that will update my website every time that my server receives a POST request from the external API? Or am I misunderstanding how a webhook works? Can they only be created by 3rd-party services and/or only used on pre-existing apps?
For a simple use case, it just means another endpoint on your server that will receive a request from the external API when it's done with the task. It's a common way REST APIs work to let you know they are done with something, or communicate anything back to you in general. You have one endpoint that will receive a request from the React client, and send information off to the external API (as you said), and another endpoint (this "webhook") that will receive requests from the external API. That's it.
You described the client-server communications for this problem just fine with the exception that the server cannot send requests to the client over HTTP. If the client needs anything from the server, it needs to send a request and ask for it. A common way to do for this would be to store the information on the server once the external API hits your "webhook", and the client can ask the server if the data is back every X seconds (long-polling).
Example (payment flow):
Client -> Server: Heres my credit card number
Server -> Client: Thanks! Let me process that.
Server -> External API: Please process this credit card for the amount of $X and ping me at this webhook when you're done.
External API -> Server: OK, will let you know when I'm done.
Client -> Server: You done yet?
Server -> Client: Nope!
External API -> Server (to webhook): Hey, I'm done, here's the receipt.
Server -> External API: Thanks!
Client -> Server: You done yet?
Server -> Client: Yes! Here's your receipt.
More advanced ways to solve the problem would be using a long-lived bidirectional communication protocol such as websockets. And as others have mentioned, all of the endpoints don't have to be on the same server. You can have one server for each endpoint, and you can have turn them on only when they get a request and then go back to sleep (that's a Lambda). Pros and cons to this approach. That's more advanced stuff and if you're just learning I wouldn't really worry about it now.
AFAIK, webhooks are just normal web APIs. Properties that make an API a webhook:
The client sending requests to your server, in this case, is another server
This client will send you a request when there's a triggered event
In your case, your server POSTing data to the external API. What makes webhook different is that the external API's server doesn't have to respond right away, instead, it sends back you, for example, an HTTP 200 OK to acknowledge that it received the request and has started to process the data. When the external API's server finishes processing the data (which is an event), it will call your server's endpoint to inform you about the process along with the processed data (if succeeded).
Your server, correspondingly, will then have to inform your client (the user) about the processed data. I would suggest implementing some kinds of persistent connection (WebSocket or SSE), or simply just make the client poll request every some second.
"(which I assume is part of my server)"-- The webhook is part of the third party API that points back to either your server or maybe some serverless function/lambda(that you set up) that in turn can parse and perform computations on the data and then send it back to your server.
It really depends on the use case and architecture of your API. If you don't want your main API server to get overloaded, if say you need to perform heavy I/O, CPU/GPU processing, you could create Serverless Lambda Functions to handle this data processing and be the endpoint you supply to the third party's webhook post endpoint.
If you're not concerned with it, you could just create another endpoint in your server to handle this webhook.
Just keep in mind, when you're trying to scale up, you want your systems to be as distributed as possible-- within reason-- so your API can handle more requests and not be overloaded with a bunch of processes.
I have an Apache server A set up that currently hosts a webpage of a bar chart (using Chart.js). This data is currently pulled from a local SQLite database every couple seconds, and the web chart is updated.
I now want to use a separate server B on a Raspberry Pi to send data to the server to be used for the chart, rather than using the database on server A.
So one server sends a file to another server, which somehow realises this and accepts it and processes it.
The data can either be sent and placed into the current SQLite database, or bypass the database and have the chart update directly from the Pi's sent information.
I have come across HTTP Post requests, but not sure if that's what I need or quite how to implement it.
I have managed to get the Pi to simply host a json file (viewable from the external ip address) and pull the data from that with a simple requests.get('ip_address/json_file') in Python, but this doesn't seem like the most robust or secure solution.
Any help with what I should be using much appreciated, thanks!
Maybe I didn't quite understand your request but this is the solution I imagined:
You create a Frontend with WebSocket support that connects to Server A
Server B (the one running on the raspberry) sends a POST request
with the JSON to Server A
Server A accepts the JSON and sends it to all clients connected with the WebSocket protocol
Server B ----> Server A <----> Frontend
This way you do not expose your Raspberry directly and every request made by the Frontend goes only to Server A.
To provide a better user experience you could also create a GET endpoint on Server A to retrieve the latest received JSON, so that when the user loads the Frontend for the first time it calls that endpoint and even if the Raspberry has yet to update the data at least the user can have an insight of the latest available data.
I have a "monitored" application. Every minute the web client do a POST to my services sending a JSON with some data. The json is big (circa 20Kb each time).
Is there a way to send a "compressed" version of the data?
Do I need to do it applicatively?
I'm using jQuery client side, and can send custom headers from the server.
Well you can also compress data in client side with the library https://github.com/cscott/compressjs. There are multiple methods the choice should be made based on your input data and the resources available in client side.
Besides using the POST request you may try the Websocket connection which should be better for sending larger input data.
It might be that the full json post is not required, if it possible to only post the data that has changed sinch that last post then that would greatly reduce the amount of data sent.
if that not possible you could reduce the size of the data posted using JSONH
From what I've read using web sockets might not use dramatically less bandwidth then http post, but there is a good article here on optimising websockets
I have a REST API built using node and express. Now i need to send the following data in one http request:
jSON Data
One Audio File to be playable on client
On the client side,i have a simple audio player that requires audio file path as input to play the file. Now i don't understand the whole flow. How will i send the file from API and how the client consume it?
The file is located in file system of the server. Point me in the right direction !!
Express doesn't appear to support multipart responses. I'd instead recommend returning JSON that includes a URL to the audio file to play. Different routes on your Express server can send the JSON and the audio files. This approach will require two different HTTP requests from your client, but it will also be far more compatible with different browsers, since not all of them deal with HTTP multipart responses the same.
It is very different type of data you trying to deliver to client.
Much better and scalable will be to have two separate requests. One for JSON data that will contain details over where Audio is located (file name?). RESTful dont have to answer with only JSON or XML data, but it is highly recommended though.
Then another request to node, that will respond with streaming audio data, please check this good question and answers.
If you need just to send audio file without live streaming, then read this: Nodejs send file in response
Right now, i found out that if i were to put this url to my browser, a .csv file will be downloaded to my Downloads folder on My Computer, where the word YHOO is the stock symbol of Yahoo
http://ichart.finance.yahoo.com/table.csv?s=YHOO&d=0&e=28&f=2010&g=d&a=3&b=12&c=1996&ignore=.csv
Is there any way where I can implement the above action on my own personal-use RESTful website where
on a client-side GUI, the user inputs symbol
on client-side, a request URL is constructed with the symbol
client-side sends the request URL to the browser (similar to the action i described above)
browser downloads a file to a location on a server (dropbox / EC2)
.csv file is converted to json object on server
json object is returned to client
How can i do this, and which framework is recommended for high performance. i am thinking of NodeJS and Mongoose.
I'm interpreting your question like this:
on a browser or GUI, the user inputs symbol
on server-side, a request URL is constructed with the symbol
server-side sends the request URL to the above url (similar to the action i described above)
server-side downloads a file to a location on the server or remote repository(dropbox / EC2)
.csv file is converted to a json object on the server
the server then returns the json object is returned to the browser or GUI
If so, then the answer to your question is yes, this is possible albeit a bit tricky if you want to to work properly.
This doesn't have to be Node.js specific. You don't need Node.js nor Mongoose for this since you're not saving the CSV to the harddrive but just going to be an interrim transporter.
Basically you can in your "servlet" issue a POST call to another url and fetch the CSV file.
Then convert this and write the file on the response and serve it back to the client.