I'm quite new to nodejs and I'm working on a backend for an Angular 4 application. The problem is that the backend is quite slow to produce the whole data for the response and I'd like to send data over time as soon as it's available. I was reading about RxJS but I can't really figure out how to use it in node, can you please help me?
Maybe you are looking for a way to stream the data
Express
Normally you respond with res.send(data), it can be called only once.
If you are reading and sending a large file, you can stream the file data while being read with res.write(chunk) and on the 'end' event of the file reading, you call res.end() to end the response.
EDIT : As you state, what you want is to stream as soon as the chunk is available, so you can use the res.flush() command between writes ( just flush after res.write(chunk)).
It would be much faster in your case but the overall compression will be much less efficient.
Related
I have created an API in Ruby on Rails. Rather than sending the response body back with the API response, I am broadcasting it to an external service, which then takes care of the realtime distribution to all connected clients (i.e. WebSockets).
Today I have hit a snag where I realized that our 3rd party provider only allows data packets of up to 25kb. One of our responses has started giving problems as the data grew to be more than this limit and the 3rd party service has started blocking calls. As a side note, data packets will seldom grow to be greater than 25kb.
I did some research and was contemplating what the best idea would be. One idea I was thinking of, was to compress the response using ZLib and then to decompress it on the JS side. The article that led to this was this StackOverflow question.
I managed to get the deflation & Base64 encoding right, but could not decode on the JS side. I also tested the Base 64 string generated but services like this one, flags the base64 string as invalid.
My code looks like this:
In a Rails Controller
...
compressed_data = Zlib::Deflate.deflate(json_data.to_s)
encoded_data = Base64.encode64(compressed_data)
broadcast encoded_data
...
In JS that receives the broadcast:
import pako from 'pako';
import { decode } from 'js-base64';
...
const decoded = decode(payload.data);
const decompressed = pako.inflate(decoded);
...
When I execute the broadcast, I get the error: unknown compression method. I understand that this might have something to do with pako, but I have also tried other approaches with no success. Does anybody have any ideas or perhaps even a better way to approach this problem?
UPDATE:
The Base64 string generated in rails looks like this:
eJxlU2Fr2zAQ/SuHP25JkJW2abMPY6OllJWkNGFftmEU+RKLypKRTknD6H/f\nyQ5ru0EQutPdy3vvzr8LUxfz8nJ2NSrIkMViXkj4GjyGCNc+tBjgZxJCXsAq\nbbfePsF3NNa4XTEqaow6mI6Md9xWSgGPqm0RPkC+r8gjeW9yLDn+5vktRDPA\nYWrtJ4uwPBUIka9wr/qiCTze3N6t1o9f1nfLBTzcLK7vFref4cGiiggBdyYS\nU9scQRHkJEE5axjEO6AGoVZH2ODWB+xDlXRmOYGl0wgHhEbtM4xGs8cajj6F\nE2hQuXA0pGokZWyEg7GW4SCiIyDfQwa0uFccW4aI5PUTqF1+PzR+aNDekdKU\noXKT9m1nkQZCeyRiE6ELXmNkvWzniWRlvVYnT+/9gVUuQ4euVjyc16JIKlBV\nK+onJmQ5FuVYynU5nQvBv4kQ4qOQfHvTxCinFpesfc3TscswK2NZANdvTF0z\nuwqfCV0cqDj/JuSSriL1XIUeTXCcjjy3qgvYmtT2qRq3KlkmiZ2PhvqcTiGg\n00cGXKgF4+iADFFXigYdYlzKsTxbl2I+vZpPy4mUs786Ule/K+5Flyya32Uu\nvijL1+KIocrbPcv0gnK66cOzy1ER2fMsPMeDFSy5Mo7ZtGxBtVc2YSzmP0q2\ncSTzMc3HeT5yTvwaFU2/q9kG/oLOR0XLQzeaV7Hq0saa2FTkn9D9a7bl4VKq\n/xuC9W73/kHFNs+5H7HnFcCaZTlFKeTMATf5z/rvMO/VYEtuffkDW0lDVA==\n
Your data starts off with a zlib header, but the compressed data is corrupted for some reason.
I am using socket.io to send/receive messages between client/server. The server has a Redis instance that stores data and responds with the data within milliseconds.
Some of the sockets take very long to return the data (stringified large JSON object) although it is sent from the server side almost immediately. I am therefore looking for suggestion that address the following concerns:
Is it normal for socket.io to take this long to emit a long string ?
How can I know which method or socket is doing the emit that takes long ?
Any further suggestions on how to improve performance?
Help really appreciated
UPDATE:
I tried using Webdis to provide the response to the client without having to go to the server to get the Redis results. However although the response appears in a console.log in about 1 second (which is the same as the DOMLoadedContent below), the websocket still takes about 20s and shows 0 bytes as shown below:
It seems like you've hit issues that others have seen as well with large file uploads using socket.io:
Node.JS, Socket.IO & large XML files: extreme performance loss?
One possible course of action is to try file streaming in socket.io:
https://gist.github.com/companje/eea17988257a10dcbf04
As for
How can I know which method or socket is doing the emit that takes
long ?
You can always pass in dateTime's into the client's data received from socket.io and then compute the time difference, printing out the methods that it called.
The project that I am working on is to receive a request where in the main and/or most part of that request consists of data coming from a database. Upon receiving, my system proceeds with its function which is to parse all the data and ultimately concatenates the needed information to form a query, then insert those data using the mentioned query into my local database.
It is working fine and no issue at all. Except for the fact that it takes too long to process when the request has over 6,000,000 characters and over 200,000 lines (or maybe less but still with large numbers).
I have this tested with my system being used as a server (the supposed setup in production), and with Postman as well, but both drops the connection before the final response is built and sent. I have already tested and seen that although the connection drops, my system still proceeds with processing the data even up to the query, and even until it sends its supposed response. But since the request dropped somewhere in the middle of the processing, the response is ignored.
Is this about connection timeout in nodejs?
Or limit in 'app.use(bodyParser.json({limit: '10mb'}))'?
I really only see 1 way around this. I have done similar in the past. Allow the client to send as much as you need/want. However, instead of trying to have the client wait around for some undetermined amount of time (at which point the client may timeout), instead send an immediate response that is basically "we got your request and we're processing it".
Now the not so great part but it's the only way I've ever solved this type of issue. In your "processing" response, send back some sort of id. Now the client can check once in a while to see if it's request has been finished by sending you that id. On the server end you store the result for the client by the id you gave them. You'll have to make a few decisions about things like how long a response id is kept around and if it can be requested more than once, things like that.
I just started watching some node tutorials and I wanted help understanding the response and request streams that I get from http.createServer(). Response & Request are streams, so does that mean than Node.js sends and recieves data in chunks?
For example, if I called
res.write("test1");
res.write("test2");
res.end();
would it only write both those things when I call end() or would it flush to the stream and send to the client making the request as and when I call write()?
Another example to elaborate on my question is if I had a txt file with a lot of plaintext data, then I setup a read stream that pipes data from that file to the res object would it pipe that data in chunks or do it once everything is in the buffer.
I guess my question also applies to the request object. For instance, is the body of the request built up packet by packet and streamed to the server or is it all sent at once, and node just chooses to make us use a stream to access it.
Thanks alot!
The first time response.write() is called, it will send the buffered header information and the first chunk of the body to the client. The second time response.write() is called, Node.js assumes data will be streamed, and sends the new data separately. That is, the response is buffered up to the first chunk of the body.
full foc
So basically, if you .write() a small piece of data, it may be buffered until theres a complete chunk or .end() is called. If .write() already has the size of a chunk, it will be transmitted immeadiately.
So i'm trying to wrap my head around developing full applications with the MEAN stack (mongodb, express, angular, node.js).
I understand that using express and node i can create a rest api with endpoints to grab data for my app. I also understand that angular is for FRONT END only. So my question is this... when you have something like an upload form and you want to upload an image to the server, would you want to create an api endpoint called something like "/api/upload/" and have all your logic for uploading the image inside that endpoint, or would you want all that upload logic somewhere else and then simply provide the file name to the "/api/upload/" endpoint with a post request?
I'm not 100% sure what you mean by upload logic - there isn't much needed, but I would put everything (including the file itself) in a POST to /api/upload/, then save it however you wish to within that function.
It is always a better approach to put your business logic at server side and I suggest you to follow this approach. If you are following this approach you can easily manipulate images if required. for e.g while uploading an logo or avatar sometime we need to crop,re-size etc operations on image like same image is used for thumbnail and profile picture. Here this approach is very meaning full for us. we can give response to the user and create a new process for image manipulation without waiting or notifying end user. Most of the apps following this approach for better experience
You can apply a separation of concerns, first of you can delegate the file upload to the client to be more "user friendly", crop it, resize it and then post the resulting file as "a file" or base64 to the server for storing it either in the database or to the file system.
I'd recommend a combination of these two libraries for the client:
Angular File Upload and ngImgCrop
then you can post the image and use a body parser to "catch" the image in express I'd recommend you to use busboy and it could be part of an endpoint as you mentioned like api/file/upload for example
// your controller
exports.uploadDocument = function(req,res, next){
req.pipe(req.busboy);
req.busboy.on('file',function(fieldname, file, filename, encoding, contentType){
// implementation
});
//updating req.body with busboy parameters
req.busboy.on('field',function(fieldname,val){
req.body[fieldname] = val;
});
req.busboy.on('finish', function() {
// implementation
next();
});
};
I hope that helps.
Short answer: express should do the upload, but it doesn't have to.
Long answer: It's a design question. If it was an image, you could have angular post the image to the imgur api and send that returned url to your server, but that's much more unreliable than doing it server-side. People could be using your site from phones, tablets, etc and though image uploading is fairly fast and consistent, the server will always do it the fastest of all because it doesn't depend on the client's wireless connection strength. Angular and Express are equally good at validating the image file.
Most importantly, if something goes wrong, you'll want to log the attempt (and maybe the image) are mostly unable to do that on the client side.
I would strongly advice you move your logic to the backend and do it with express,
More secure, as the code is not expose on the client browser
Fast uploading (though can be fast on the client side too, but if the client browser is slow, it would relatively affect the uploading)
Reduce the code exposed to the client browser