I am trying to create an input form that allows for multiple files to be uploaded to my server. The way I currently have the system set up is shown in this diagram:
Essentially:
The website is running locally on each client's machine. The website makes XMLHttpRequests to the node.js webhost. If the request requires data from the Teradata database, it will in turn make a request to a local server though the node module http. The local server will then send back the data requested, then the webhost will pass that data to the client website.
The issue that am currently running into is when trying to make a POST XMLHttpRequest to the Webhost from the Website. When I do a small amount of data, it works without issue. But when I try to pass a larger amount of data such as the binary of an image in string form, it will either end the connection or somehow lose the end of the request message, I don't really know how to tell which is going on. Since I want to be able to push the image from the Website all the way to the Local Server and insert it into the database, I need to find out how to make sure I get all of it passed to the webhost correctly. I ran into this problem going the other direction, from the Local Server to the Website, but I fixed that by "chunking" the data and sending it in smaller packages. I can't seem to find a way to do that with XMLHttpRequests.
A few things to note: I cannot change the type of database, I cannot change the overall structure of the network.
If anyone has any insights in either how to troubleshoot this or suggestions for methods other than XMLHttpRequest for being able to send larger amounts of data, I would be much appreciative.
Related
I'm coding a private webapp that can pick some data from a source (yfinance, panda, ...) and generates a time series. Basically, I want to be able to click on any of the Dow Jones stocks with an HTML btn or whatever, which will send a request to my backend server to generate such chart using python, R or any alternative.
Two possibilities:
I generate the chart and figures on the client's side, by sending the data from the server to the client on a JSON format (which seems really unproductive).
Or, I make all computations on backend, following an HTTP request from the client, generate the chart and send it to the client (my computer).
The question is, can I send such chart, from the server to the client without uploading it as a png and transmit the link through AJAX etc. May I get the output of a python function and send it ?
To summarize: Can a .png file might be sent from a server to an online webapp, any ideas ?
Thanks for your comments, this is a technical question and this might be useful for future requests. Also, I plan to code everything using Django framework.
Planning everything so nothing has been built yet.
I have a sensor recording data every minute and updating an excel/csv file with the results, done in python. I then use requests.post to create an HTTP request sending the file to a Weebly website. I believe I need to use JavaScript on the website to receive the excel file and embed it into the website. I have been stuck on this part for a few days since I have never used JavaScript before. The question is: How can I create a JavaScript code to receive the file and embed it into the Weebly website?
From the research I have done thus far it appears that I need to use an XMLHttpRequest GET to request the data being sent from python, I am a little confused though because it requires a url. Through more research it looks like I may need to set up a server to send it to and request it from although I am not sure if that is necessary or how to do it.
import requests
with open('/home/pi/Images_for_Python/2/100001.csv','rb') as f:
r = requests.post("http://www.greeniethegenie123.weebly.com", files={'/home/pi/Images_for_Python/2/100001.csv':f})
print(r.text)
If anyone else is having the same problem I have found an alternative solution through Amazon Web Services (AWS). They have a setup that works quite well for this called AWS IoT Analytics. It made this a trivial problem due to its python module that connects directly to the data server!
I have an Apache server A set up that currently hosts a webpage of a bar chart (using Chart.js). This data is currently pulled from a local SQLite database every couple seconds, and the web chart is updated.
I now want to use a separate server B on a Raspberry Pi to send data to the server to be used for the chart, rather than using the database on server A.
So one server sends a file to another server, which somehow realises this and accepts it and processes it.
The data can either be sent and placed into the current SQLite database, or bypass the database and have the chart update directly from the Pi's sent information.
I have come across HTTP Post requests, but not sure if that's what I need or quite how to implement it.
I have managed to get the Pi to simply host a json file (viewable from the external ip address) and pull the data from that with a simple requests.get('ip_address/json_file') in Python, but this doesn't seem like the most robust or secure solution.
Any help with what I should be using much appreciated, thanks!
Maybe I didn't quite understand your request but this is the solution I imagined:
You create a Frontend with WebSocket support that connects to Server A
Server B (the one running on the raspberry) sends a POST request
with the JSON to Server A
Server A accepts the JSON and sends it to all clients connected with the WebSocket protocol
Server B ----> Server A <----> Frontend
This way you do not expose your Raspberry directly and every request made by the Frontend goes only to Server A.
To provide a better user experience you could also create a GET endpoint on Server A to retrieve the latest received JSON, so that when the user loads the Frontend for the first time it calls that endpoint and even if the Raspberry has yet to update the data at least the user can have an insight of the latest available data.
I have a chat application, backend is written with help of Socket.io (based on Node.js server), and client is pure vanilla JavaScript, with JWT token generation from PHP, chat loads all data on load from REST API written on PHP also, and i am stuck with file upload.
I have two variants, first is REST API, just send data with help of XMLHTTPRequest, and i already implemented this, it's a simple FormData POST to REST,
it works fine, but i am not sure of "rightness" of this solution, because for now REST have only GET routes for simple users (there is admin routes also) and by design of my project, it's set to get data for user chat, and its looks not right for me to make POST route for them, but what worries me most is what i already have another mechanism/solution where user sending their messages, it is socket.io-node.js server, with all permission checks (does this user have permission to join this chat, send information to it, and so on), so REST will take part of Socket.io task. Also beside role distribution problem, REST solution is also a logic duplication, where all permission checks written on Node.js side, must be duplicated on PHP side. As a result i already started to rewrite file sending to Node.js via Socket.io. Socket.io from version 1 is accepting binary data, so it's was pretty easy, i just sending a File and Node.js gets Buffer, which is easily stored to filesystem like this
require('fs').writeFileSync(filePath, Buffer);
but here i have problem what i don't have upload progress, and most importantly despite all projects part are now perfectly distributed by their roles (client is sending data to socket.io, and reads from rest), and i have no role duplication, i am still not 100% sure i solved this problem right.
So question is, how and where would you send your files, in case of your chat application.
So, let me explain it better, I'm pretty sure the title doesn't says much.
I'm in charge of doing some improvements on the company system. Now the system saves the current image, as a file, and insert into the database the path of the image.
Then, what's the problem if it works? Well, if the server gets a lot of images being inserted all the time, the server will need some improvements, so my first thought was, why don't I save the image data to the database and load it through ajax call? That's nice, it won't make the server overload from lots of gigabytes from the images, won't have problems with duplicates and such, but when I call the ajax method from jQuery, the data sent to the server side is via query string and if the data is higher than 6000 characters, like most are when I use base64 encoding, I can't call the method.
My other thought was, send the image to the server side and then generate the data to save to the database, and when the user wants to see the image, the server transform the data to an image and sends to the client side. But the problem is, there are lots of users for this system, and sending/receiving lots of images will make the server get memory problems due to the high amount of requisitions.
So to make long story short, what is the best way to send/receive images from client to server and the opposite without overloading the server capacity.
P.S.: Sorry for the long post, I wanted to make a clear question.
Thanks in advance.
The server will overload even more storing the images in the DB. Most web servers serve static files (like images) very efficiently.
You have to think more about the problem before trying to solve it:
Is your problem bandwidth overload? Try resizing the images client side before upload (you can use tools like PlUpload). If you can't, at least resize them server side to serve them smaller and save on download bandwidth
Is your problem harddrive space overload? Again, resize the images client o server side.
Is your problem CPU overload? Try to determine the root code that is the cause of the spikes. Be aware that lots of requests will cause high CPU, and you might need a more beefy server (or another webserver)