It seems for big enough stdout output node does not fire data events:
Example bash script:
#!/usr/bin/env sh
for i in {1..100000}; do
echo $i
done
Running the above script from bash should prints all numbers from 1 to 100k one number per line
However when proxying it via node's exec:
#!/usr/bin/env node --harmony --harmony_destructuring --harmony_default_parameters
const {exec} = require('child_process');
let stdout = '';
const child = exec('./tmp.sh');
child.stdout.on('data', data => {
stdout += data;
});
child.stdout.on('close', code => {
if (code) process.exit(code);
console.log(stdout.split('\n').slice(0, 3).join('\n'));
console.log('...');
console.log(stdout.split('\n').slice(-3).join('\n'));
});
For the above script, I'd expect to get:
1
2
3
...
99998
99999
100000
However when run It returns:
1
2
3
...
35984
35985
About a third of the output is truncated. Logging inside the data handler shows that only 1 data event is fired with 1 chunk
I've tried listening to the child.on('exit') event instead with the same result (also stdout.on('exit'))
It works the same when replacing exec with spawn
node --version # v5.4.1
osx 10.10
child_process.exec() buffers the standard output into a fixed size buffer. You are likely filling the buffer and thus you don't get the rest of the output.
There is a maxBuffer option you can pass to .exec() to make the buffer larger or you can use .spawn() and receive an unlimited stream of output.
See Stdout buffer issue using node child_process for related info.
Also worth reading: Difference between spawn and exec of Node.js child_process and Stdout of Node.js child_process exec is cut short.
Related
I am developing a lib for docker command line in nodejs, I am still in starting face, I just tried basic docker run command using spawn in node js - everything works fine but it's not working for complex cases like the one below.
I want to run docker run --rm -it julia:0.3.6 julia -E "[x^2 for x in 1:100]" in nodejs, but I am gettting below error -
the input device is not a TTY
Docker Shell existed with status = 1
Below Code -
const
spawn = require('child_process').spawn,
dockerDeamon = spawn("docker", ["run","--rm", "-it", "julia:0.3.6", "-E", "\" [x^2 for x in 1:100]\""] );
dockerDeamon.stdout.on('data', data => {
console.log(`${data}`);
});
dockerDeamon.stderr.on('data', data => {
console.log(`${data}`);
});
dockerDeamon.on('close', code => {
console.log(`Docker Shell existed with status = ${code}`);
});
Is there any better way to execute the above script ?
You're passing the -t (--tty) flag to Docker, which tells it that it should expect the input and output to be attached to a terminal (TTY). However, when you're using spawn, you're instead attaching it to a Node.js stream in your program. Docker notices this and therefore gives the error Input device is not a TTY. Therefore, you shouldn't be using the -t flag in this case.
Also, note that you don't need nested quotes in your last argument, "\" [x^2 for x in 1:100]\"". The purpose of the quotes is to preserve the spaces and other special characters in the argument when running in a shell, but when you use spawn you're not using a shell.
So your statement should be something like:
dockerDeamon = spawn("docker", ["run","--rm", "-i", "julia:0.3.6", "julia", "-E", "[x^2 for x in 1:100]"] );
I have a Javascript file being executed by Nodejs as a script in a Unix shell pipeline. The script needs to read (a few thousand lines) of text from STDIN, parse it for some bits, then return a json stream on STDOUT with the results of its work.
var readline = require('readline');
var rl = readline.createInterface({
input: process.stdin,
output: process.stdout,
terminal: false
});
Then I have some functions to process the data, then I setup an event listener that gets called on each line of input and does all the processing I need and appends it to an array:
function extract_references (data) {
...
references.push(item);
}
rl.on('line', extract_references);
So far so good, my data array ends up populated with all the components I want. The trouble comes when I try to write it back out to STDOUT. I've tried a couple things being triggered by the close event on the input stream:
function output_references () {
var output = JSON.stringify(references, null, ' ');
process.stdout.write(output);
}
rl.on('close', output_references);
This works fine if the input stream is relatively short (a couple hundred lines) but when I crank it up to my full data set I start getting truncated output and an error such as:
write error: Resource temporarily unavailable
...from the next command in my pipeline.
I've also tried using rl.write(), but that doesn't seem to give me anything at all on STDOUT. What am I doing wrong and what is the correct way to handle a stream like this?
In perl if you need to run a batch file it can be done by following statement.
system "tagger.bat < input.txt > output.txt";
Here, tagger.bat is a batch file, input.txt is the input file and output.txt is the output file.
I like to know whether it is possible to do it be done in Node.js or not? If yes, how?
You will need to create a child process. Unline Python, node.js is asynchronous meaning it doesn't wait on the script.bat to finish. Instead, it calls functions you define when script.bat prints data or exists:
// Child process is required to spawn any kind of asynchronous process
var childProcess = require("child_process");
// This line initiates bash
var script_process = childProcess.spawn('/bin/bash',["test.sh"],{env: process.env});
// Echoes any command output
script_process.stdout.on('data', function (data) {
console.log('stdout: ' + data);
});
// Error output
script_process.stderr.on('data', function (data) {
console.log('stderr: ' + data);
});
// Process exit
script_process.on('close', function (code) {
console.log('child process exited with code ' + code);
});
Apart from assigning events to the process, you can connect streams stdin and stdout to other streams. This means other processes, HTTP connections or files, as shown below:
// Pipe input and output to files
var fs = require("fs");
var output = fs.createWriteStream("output.txt");
var input = fs.createReadStream("input.txt");
// Connect process output to file input stream
script_process.stdout.pipe(output);
// Connect data from file to process input
input.pipe(script_process.stdin);
Then we just make a test bash script test.sh:
#!/bin/bash
input=`cat -`
echo "Input: $input"
And test text input input.txt:
Hello world.
After running the node test.js we get this in console:
stdout: Input: Hello world.
child process exited with code 0
And this in output.txt:
Input: Hello world.
Procedure on windows will be similar, I just think you can call batch file directly:
var script_process = childProcess.spawn('test.bat',[],{env: process.env});
I'm using Grunt to kick off a unit-test framework (Intern), which ultimately pipes another node.js process that I'm then using Charm to output results to the screen. I'm having to pass in the terminal size information from a Grunt config option, but it's a bit messy and I'd like to try and get the terminal size from within the piped process, but the standard process.stdout.cols/getWindowSize are simply unavailable as the piped process doesn't register as TTY (although Charm works fine with it all).
Any suggestions?
EDIT Just to be clear here ... the Grunt JavaScript file is running in the main node.js process, but the file I'm attempting to retrieve this info from (and where I'm therefore running people's suggested commands) is in a spawned child process.
Try these:
tput cols tells you the number of columns.
tput lines tells you the number of rows.
echo -e "lines\ncols"|tput -S to get both the lines and cols
There's stty, from coreutils:
$ stty size #60 120 <= sample output
While running the below code in terminal prints the cols:
var sys = require('sys')
var exec = require('child_process').exec;
function puts(error, stdout, stderr) { sys.puts(stdout) }
exec("tput cols", puts);
The pty.js module can make a child act like a regular terminal.
var pty = require('pty.js');
var term = pty.spawn('bash', [], {
name: 'xterm-color',
cwd: process.env.HOME,
env: process.env
});
term.on('data', function(data) {
console.log(data);
});
term.write('ls\r');
term.resize(100, 40);
term.write('ls /\r');
console.log(term.process);
I have this code:
var convert = spawn("gm", ["convert"].concat(parameters));
// output stream
obj.outStream = new BufferList();
convert.stderr.pipe(process.stderr, { end: false });
// put the output in the stream TOO
convert.stdout.pipe(obj.outStream);
// send the image to the input
obj.stream.pipe(throttle).pipe(convert.stdin);
How can I suppress the output of the "convert" process, without suppressing the input and the output to obj.outStream too?
The reason is because I don't want to output that to the user, as it does now.
What you're probably seeing in the output is convert.stderr because you are piping it to process.stderr, which is the error output of your child process' master. When you spawn a child process, by default, no stdio is handled.
var spawn = require('child_process').spawn'
var child = spawn('gm', ['convert']);
The code that you've shown is that you're directly piping the child's stderr to the main process' stderr and piping stdout to your outStream. That means the only possible output you can see is convert.stderr.
To fix this, ignore stderr.
var obj = {
outStream: new BufferList()
};
var spawn = require('child_process').spawn'
var child = spawn('gm', ['convert'], {
stdio: ['pipe', obj.outStream, 'ignore']
});
With this, you have the stdin stream as it normally is, stdout piped to obj.outStreamm and stderr ignored.
See the docs you can use listeners or use the 'ignore' property.