Sorry in advance, I have a couple of questions on createReadStream() here.
Basically what I'm doing is dynamically building a file and streaming it to the user using fs once it is finished. I'm using .pipe() to make sure I'm throttling correctly (stop reading if buffer's full, start again once it's not, etc.) Here's a sample of my code I have so far.
http.createServer(function(req, res) {
var stream = fs.createReadStream('<filepath>/example.pdf', {bufferSize: 64 * 1024})
stream.pipe(res);
}).listen(3002, function() {
console.log('Server listening on port 3002')
})
I've read in another StackOverflow question (sorry, lost it) that if you're using the regular res.send() and res.end() that .pipe() works great, as it calls the .send and .end and adds throttling.
That works fine for most cases, except I'm wanting to remove the file once the stream is complete and not using .pipe() means I'm going to have to handle throttling myself just to get a callback.
So I'm guessing that I'll want to create my own fake "res" object that has a .send() and .end() method that does what the res usually does, however on the .end() I'll put additional code to clean up the generated file. My question is basically how would I pull that off?
Help with this would be much appreciated, thanks!
The first part about downloading can be answered by Download file from NodeJS Server.
As for removing the file after it has all been sent, you can just add your own event handler to remove the file once everything has been sent.
var stream = fs.createReadStream('<filepath>/example.pdf', {bufferSize: 64 * 1024})
stream.pipe(res);
var had_error = false;
stream.on('error', function(err){
had_error = true;
});
stream.on('close', function(){
if (!had_error) fs.unlink('<filepath>/example.pdf');
});
The error handler isn't 100% needed, but then you don't delete the file if there was an error while you were trying to send it.
Related
Good Day.
Been wracking the 'ol noggin for a way to solve this.
In a nutshell, I have a form that has a number of text inputs as well as an input file element to upload said file to AWS S3 (via lepozepo:s3 ver5.1.4 package). The nice thing about this package is that it does not need the server, thus keeping resources in check.
This S3 package uploads the file to my configured bucket and returns the URL to access the image among a few other data points.
So, back to the form. I need to put the AWS URL returned into the database along with the other form data. HOWEVER, the S3 call takes more time than what the app waits for since it is async, thus the field within my post to Meteor.call() is undefined only because it hasn't waited long enough to get the AWS URL.
I could solve this by putting the Meteor.call() right into the callback of the S3 call. However, I was hoping to avoid that as I'd much rather have the S3 upload be its own Module or helper function or even a function outside of any helpers as it could be reused in other areas of the app for file uploads.
Psudo-code:
Template.contacts.events({
'submit #updateContact': function(e,template){
s3.upload({file:inputFile, path:client},function(error,result){
if(error){
// throw error
}else{
var uploadInfo = result;
}
});
formInfo = {name:$('[name=name]').val(),file:uploadInfo}; // <= file is undefined because S3 hasn't finished yet
Meteor.call('serverMethod',formInfo, function(e,r){
if(e){
// throw error message
}else{
// show success message
}
});
});
I could put the formInfo and the Meteor.call() in the s3 callback, but that would result in more complex code and less code reuse where IMO this is a perfect place for code reuse.
I've tried wrapping the s3 in it's own function with and without a callback. I've tried using reactiveVars. I would think that updating the db another time with just the s3 file info would make the s3 abstraction more complex as it'd need to know the _id and such...
Any ideas?
Thanks.
If you are using javascript it's best to embrace callbacks!
What is it about using callbacks like this that you do not like, or believe is modular or reusable?
As shown below the uploader function does nothing but wrap s3.upload. But you mention this is psudeocode, so I presume that you left out logic you want included in the modular call to s3.upload (include it here), but decouple the logic around handling the response (pass in a callback).
uploader = function(s3_options, cb) {
s3.upload(s3_options, function(error,result){
if(error){
cb(error);
}else{
cb(null, result);
}
});
};
Template.contacts.events({
'submit #updateContact': function(e,template){
cb = function(error, uploadInfo) {
formInfo = {name:$('[name=name]').val(),file:uploadInfo};
Meteor.call('serverMethod',formInfo, function(e,r){
if(e){
// throw error message
}else{
// show success message
}
});
uploader({file:inputFile, path:client}, cb); // you don't show where `inputFile` or `client` come from
}
});
connection = new WebSocket("ws://localhost:1050/join?username=test")
connection.onopen = function(){
alert('Connection open!');
}
connection.onmessage = function(e){
var server_message = e.data;
alert(server_message);
}
connection.onclose = function() {
alert("websocket closing")
}
The connection to the server is established and an alert is displayed for Connection open! However immediately afterwards the connection closes. The server does not call close and there seem to be no other errors in the console. This is happening in both chrome and firefox.
I looked at a bunch of different similar examples on the web but to no avail.
to Keep Websocket Opened prevent handler from returning by return false; in connection.onmessage
like this :
connection.onmessage = function(e){
var server_message = e.data;
alert(server_message);
return false;
}
I believe I've stumbled across the solution that OP found but failed miserably to explain. I don't have enough reputation to comment, otherwise I'd be responding to all of the confused comments begging for clarification on OP's response.
The short version is that I think OP was referring to his server-side connection handler when he said "All I had to do was block the handler from returning before the websocket connection closes".
It turns out my server was closing the webSocket automatically because I didn't understand how a certain webSocket function worked. Specifically, I was using a Python server script with asyncio/websockets and the following code:
async def receiveCommandsLoop(player):
while True:
msg = await player.websocket.recv()
print(command)
async def handleClient(websocket, path):
username = await websocket.recv()
player = players[username]
...
#Start task to listen for commands from player
asyncio.get_event_loop().create_task(receiveCommandsLoop(player))
start_server = websockets.serve(handleClient, '', 8765)
The idea was that websockets.serve would use handleClient to begin the connection and do some setup, then create a new task with receiveCommandsLoop that would take over the job of communication.
But it turns out: when you call websockets.serve, Python expects that when your handler (in this case, handleClient) returns, you must be done with the socket, and it closes it automatically.
Thus, by the time receiveCommandsLoop was run, handleClient had returned, and the webSocket had been automatically closed.
I was able to fix this by simply modifying my handleClient function to directly run the loop originally contained in receiveCommandsLoop. Hope this helps someone out there.
This also could be the case when you're trying to send binary data over a websocket connection, but some side (client or server) is trying to interpret it as a text - many libraries and frameworks do it unless you explicitly specify you do want binary data.
It could also be a login problem. The websocket will automatically close the website required authentication but no authentication information was provided.
Piecing together hints from this post and others, I found a solution that works when using the python websocket server example found everywhere that includes something like:
async def handler(websocket, path):
data = await websocket.recv()
reply = f"Data recieved as: {data}!"
await websocket.send(reply)
To those of us new to websocket, I think the assumption is that the handler function will be called each time the client sends a message, which turns out not to be the case. As others mention, the connection closes as soon as the handler function returns once. The solution I found is to change it to:
async def handler(websocket, path):
async for data in websocket:
reply = f"Data recieved as: {data}!"
print(data)
await websocket.send(reply)
My client-side javascript code is equivalent to the OP's and I didn't have to change anything for this to work.
Unfortunately I can't explain why async for data in websocket: makes it actually wait forever and spontaneously run the inner code block each time a message is received, but it does for me and I get all the expected log messages both on the python server side and the client javascript console.
If anyone more knowledgeable on this topic can comment on whether this is a good-for-general-use solution or if there's a gotcha to look out for here, it would be much appreciated.
Fixed it!
All I had to do was block the handler from returning before the websocket connection closes
I am facing a strange issue with calling socket.on methods from the Javascript client. Consider below code:
for(var i=0;i<2;i++) {
var socket = io.connect('http://localhost:5000/');
socket.emit('getLoad');
socket.on('cpuUsage',function(data) {
document.write(data);
});
}
Here basically I am calling a cpuUsage event which is emitted by socket server, but for each iteration I am getting the same value. This is the output:
0.03549148310035006
0.03549148310035006
0.03549148310035006
0.03549148310035006
Edit: Server side code, basically I am using node-usage library to calculate CPU usage:
socket.on('getLoad', function (data) {
usage.lookup(pid, function(err, result) {
cpuUsage = result.cpu;
memUsage = result.memory;
console.log("Cpu Usage1: " + cpuUsage);
console.log("Cpu Usage2: " + memUsage);
/*socket.emit('cpuUsage',result.cpu);
socket.emit('memUsage',result.memory);*/
socket.emit('cpuUsage',cpuUsage);
socket.emit('memUsage',memUsage);
});
});
Where as in the server side, I am getting different values for each emit and socket.on. I am very much feeling strange why this is happening. I tried setting data = null after each socket.on call, but still it prints the same value. I don't know what phrase to search, so I posted. Can anyone please guide me?
Please note: I am basically Java developer and have a less experience in Javascript side.
You are making the assumption that when you use .emit(), a subsequent .on() will wait for a reply, but that's not how socket.io works.
Your code basically does this:
it emits two getLoad messages directly after each other (which is probably why the returning value is the same);
it installs two handlers for a returning cpuUsage message being sent by the server;
This also means that each time you run your loop, you're installing more and more handlers for the same message.
Now I'm not sure what exactly it is you want. If you want to periodically request the CPU load, use setInterval or setTimeout. If you want to send a message to the server and want to 'wait' for a response, you may want to use acknowledgement functions (not very well documented, but see this blog post).
But you should assume that for each type of message, you should only call socket.on('MESSAGETYPE', ) once during the runtime of your code.
EDIT: here's an example client-side setup for a periodic poll of the data:
var socket = io.connect(...);
socket.on('connect', function() {
// Handle the server response:
socket.on('cpuUsage', function(data) {
document.write(data);
});
// Start an interval to query the server for the load every 30 seconds:
setInterval(function() {
socket.emit('getLoad');
}, 30 * 1000); // milliseconds
});
Use this line instead:
var socket = io.connect('iptoserver', {'force new connection': true});
Replace iptoserver with the actual ip to the server of course, in this case localhost.
Edit.
That is, if you want to create multiple clients.
Else you have to place your initiation of the socket variable before the for loop.
I suspected the call returns average CPU usage at the time of startup, which seems to be the case here. Checking the node-usage documentation page (average-cpu-usage-vs-current-cpu-usage) I found:
By default CPU Percentage provided is an average from the starting
time of the process. It does not correctly reflect the current CPU
usage. (this is also a problem with linux ps utility)
But If you call usage.lookup() continuously for a given pid, you can
turn on keepHistory flag and you'll get the CPU usage since last time
you track the usage. This reflects the current CPU usage.
Also given the example how to use it.
var pid = process.pid;
var options = { keepHistory: true }
usage.lookup(pid, options, function(err, result) {
});
I'm working on an online, turned based game in order to teach myself Node.js and Socket.IO. Some aspects of the game are resolved serverside. At one point during one of these functions, the server may require input from the clients. Is there a way I can "pause" the resolution of the server's function in order to wait for the clients to respond (via a var x = window.prompt)?
Here's an idea of the code I'm working with:
Server:
for (some loop){
if (some condition){
request input via io.sockets.socket(userSocket[i]).emit('requestInput', data)
}
}
Client:
socket.on('requestInput', function (data) {
var input = window.prompt('What is your input regarding ' + data + '?');
//send input back to the server
socket.emit('refresh', input)
});
Any thoughts?
I don't think that is possible.
for (some loop){
if (some condition){
request input via io.sockets.socket(userSocket[i]).emit('requestInput', data)
/* Even if you were able to pause the execution here, there is no way to resume it when client emits the 'refresh' event with user input */
}
}
What you can do instead is emit all 'requestInput' events without pausing and save all responses you will get in socket.on('refresh',function(){}) event in an array, then you can process this array later. I don't know what your exact requirement is but let me know if that works.
Since you are emitting socket.emit('refresh', input) on the client side, you just need to set up a socket event listener on the server side as well. For example:
io.sockets.on('connection', function (socket) {
socket.on('refresh', function (data) {
console.log(data) //input
});
})
I will also point out, so that you don't run into trouble down the line, that indefinite loops are a big nono in node. Nodejs runs on a single thread so you are actually blocking ALL clients as long as your loop is running.
I am trying to write a simple POP3 client for gmail in node.js. I initially wrote one where I enter the POP3 commands and they get sent to the server, and the server responds. This works fine because I can wait for the server's response before entering the next command.
But I want to make an automated conversation, where the program itself sends the commands and awaits its own responses. Simply doing:
stream.write('USER ***&#gmail.com');
stream.write('PASS *****');
stream.write('LIST');
doesn't work, because of the asynchronous nature of streams in node. I have to wait for the stream's 'data' event before sending the next message, otherwise, nothing happens at all. So tried it like this:
var tls =require ('tls');
var stream = tls.connect(995,'pop.gmail.com', function() {
console.log ('Conexion establecida');
});
stream.on('data', function(data)
{
var str = data.toString();
console.log(str.substr(0,14));
if(str.substr(0,14)=="+OK Gpop ready")
{
console.log('Sending username...');
process.nextTick(function() {
stream.write ('USER ***********#gmail.com');
});
}
if(str.substr(0,14)=="+OK send PASS")
{
console.log('Recieving list of email...');
process.nextTick(function(){
stream.write('PASS *********');
});
}
});
The idea is that the 'data' event listener handles the sending of the next command depending on the last reply recieved. Again, doing stream.write alone did not seem to work, so I wrapped the write() calls in process.nextTick() calls. It seems that the USER command is sent, but never the PASS command. Can someone tell me why? Or if there is a more effective way to do this? Thanks for any information.
Your commands are being sent, however they are not being recognized, as they are missing the newline character, which denotes the end of a command. Adding a newline character to the end of your commands should get it working.
Additionally, you can simplify your commands by implementing a queue like so:
var tls = require('tls');
var commands = [];
commands.push('USER ***********#gmail.com\n');
commands.push('PASS *********\n');
var stream = tls.connect(995, 'pop.gmail.com', function () {
console.log('Conexion establecida');
});
stream.on('data', function (data) {
if (data.toString().indexOf('OK') > -1 && commands.length) {
stream.write(commands.shift());
}
});
Without seeing all the code, it is a bit difficult to answer.
However I noticed that in your first example, you write
stream.write('USER *&#gmail.com');
with an ampersand "&"
However in the second example there is no ampersand - could that be the source of the issue?