using child_process spawn on aws lambda for a python script - javascript

I'm trying to run a python script using my javascript file through the child_process.spawn system, but it seems to never run on the aws lambda.
The relevant code is :
getEntities: function (){
var spawn = require('child_process').spawn;
var py = spawn('python', ['mainPythonFile.py']);
var outputString = "starting string";
console.log("BEFORE ANY INPUT");
py.stdout.on('data', function (data) {
console.log("----Getting information from the python script!---");
outputString += data.toString();
console.log(outputString);
});
py.stdout.on('end', function (){
console.log("===hello from the end call in python files===");
console.log("My output : " + outputString);
});
console.log("NO INPUT CHANGED??");
return outputString;
}
These files are in the same level of the folder structure (surface level).
The python file being run is quite simple and only contains a few print statements:
MainPythonFile:
import sys;
print("Hello There");
print("My name is Waffles");
print("Potato Waffles");
sys.stdout.flush()
The output I get from the aws service is this :
BEFORE ANY INPUT
NO INPUT CHANGED??
starting string
I've tried different paths on trying to access the python file, such as
*mainPythonFile.py ./mainPythonFile.py etc.
I think the code seems to be fine as this works on my local machine, but there's a subtlety of trying to make it run on AWS that I cannot understand.
I can provide any other info if need be.
NOTE: the "getEntities" function is being called by another node.js file, but I moved the code to the calling function, I get the same result.

Due to the asynchronous nature of JS, as explained by Chris, the function reaches the "return" statement before the "end" in the spawned thread is actually called.
This means the code never got a chance to actually set the correct output text.
I changed my function calls to take in a callback, that would then respond when the program had replied with the information.
My new function is slightly changed to this (without the prints):
getEntities: function(callbackFunction, that){
var spawn = require('child_process').spawn;
var py = spawn('python', ['mainPythonFile.py']);
var outputString = "starting string";
py.stdout.on('data', function (data) {
outputString += data.toString();
});
// that = "this == alexa" that's passed in as input.
py.stdout.on('end', function (){
callbackFunction(outputString, that);
});
The function that called this function is now as follows :
HelperFunctions.getEntities(function(returnString,that){
that.response.speak(returnString);
that.emit(':responseReady');
}, this);
I'm sure there's a prettier way to do this, but this seems to be working for now. Thanks to ChrisG

Related

Invoke pre-parsed command line with Node.js

I need to invoke the following command, where password is user input. However, I am worried about the possibility of an attack, such as "; rm -rf / ;" being the input given by the user.
var checkPassword = exec('echo "'+password+ '"| cracklib-check\n', function(err, stdout, stderr) {
...
...
}
is there a way to invoke the command with pre-parsed arguments (preferably native to nodejs/ javascript), kind of like prepared statements which are used to avoid SQL injection?
I could probably avoid the problem by blacklisting certain characters, but that seems much less reliable, and I'd like to avoid it is possible.
As you point out, building a command line with user provided input is a security issue. Typically you would write a wrapper that verifies that each user-provided parameter meets a white-list before invoking the command.
In your case however there is a simpler solution: you are constructing a command line that simply sends the password to the stdin of cracklib-check. Instead of using child_process.exec you can switch to child_process.spawn which allows you to write directly to stdin, avoiding the need to build a command line with user-provided input.
The following sample code avoids the security problem:
const spawn = require('child_process').spawn;
// Read password from argument to nodejs invocation
var password = process.argv[2];
// Spawn cracklib-check
var cracklib_check = spawn("/usr/sbin/cracklib-check");
// Send password to cracklib-check STDIN
cracklib_check.stdin.write(password);
cracklib_check.stdin.end();
// Process results of cracklib-check
cracklib_check.stdout.on('data', function (data) {
console.log("[*] " + data);
});
cracklib_check.stderr.on('data', function (data) {
console.log("[-] " + data);
});
#Ilmora's answered me started, but I still had to handle encoding.
const spawn = require('child_process').spawn;
// Read password from argument to nodejs invocation
var password = process.argv[2];
var cracklib_check = spawn('/usr/sbin/cracklib-check');
cracklib_check.stdin.setEncoding = 'utf-8';
cracklib_check.stdin.write(password);
cracklib_check.stdin.end();
// Process results of cracklib-check
cracklib_check.stdout.on('data', function (data) {
console.log("[*] " + data.toString());
});
cracklib_check.stderr.on('data', function (data) {
console.log("[-] " + data.toString());
});

how to run a batch file in Node.js with input and get an output

In perl if you need to run a batch file it can be done by following statement.
system "tagger.bat < input.txt > output.txt";
Here, tagger.bat is a batch file, input.txt is the input file and output.txt is the output file.
I like to know whether it is possible to do it be done in Node.js or not? If yes, how?
You will need to create a child process. Unline Python, node.js is asynchronous meaning it doesn't wait on the script.bat to finish. Instead, it calls functions you define when script.bat prints data or exists:
// Child process is required to spawn any kind of asynchronous process
var childProcess = require("child_process");
// This line initiates bash
var script_process = childProcess.spawn('/bin/bash',["test.sh"],{env: process.env});
// Echoes any command output
script_process.stdout.on('data', function (data) {
console.log('stdout: ' + data);
});
// Error output
script_process.stderr.on('data', function (data) {
console.log('stderr: ' + data);
});
// Process exit
script_process.on('close', function (code) {
console.log('child process exited with code ' + code);
});
Apart from assigning events to the process, you can connect streams stdin and stdout to other streams. This means other processes, HTTP connections or files, as shown below:
// Pipe input and output to files
var fs = require("fs");
var output = fs.createWriteStream("output.txt");
var input = fs.createReadStream("input.txt");
// Connect process output to file input stream
script_process.stdout.pipe(output);
// Connect data from file to process input
input.pipe(script_process.stdin);
Then we just make a test bash script test.sh:
#!/bin/bash
input=`cat -`
echo "Input: $input"
And test text input input.txt:
Hello world.
After running the node test.js we get this in console:
stdout: Input: Hello world.
child process exited with code 0
And this in output.txt:
Input: Hello world.
Procedure on windows will be similar, I just think you can call batch file directly:
var script_process = childProcess.spawn('test.bat',[],{env: process.env});

Running multiple threads in parallel using Webworker-Threads for Node.js

I am VERY new to using Node.js and WebWorker-Threads for Node.js (https://github.com/audreyt/node-webworker-threads). The WebWorker-Threads module is based on Thread-A-Gogo (https://github.com/xk/node-threads-a-gogo).
Here is a summary of my problem:
I have a file called "readFile.js", which contains code to read a csv
file that is passed in, and converts the csv data into a 2D array.
I want the function ""exports.parseCSV" within "readFile.js" to be
loaded and executed in multiple worker threads, with each thread
running in parallel.
I have a file, "test1.js", which attempts to do point 2 by using a
thread pool:
var Threads = require('webworker-threads');
var parser = require('./readFile.js');
var numThreads = 3;
var threadPool = Threads.createPool(numThreads);
threadPool.all.eval(parser.parseCSV);
for (var i = 1; i <=3; i++) {
(function(i) {
threadPool.any.eval(parser.parseCSV("csvFile.csv"), function (err, val) {
if (i == 3) {
console.log("bye!");
threadPool.destroy();
}
});
})(i);
}
I have another file, "test2.js", which attempts to do point 2 by
manually creating threads:
function cb (err, result) {
if (err) {
console.log("\n" + " ERROR! ERROR! ERROR!");
throw err;
}
console.log(" NO ERROR!");
thread.destroy();
}
var Threads = require('webworker-threads');
var parser = require('./readFile.js');
var thread = Threads.create();
thread.eval(parser.parseCSV);
thread.eval(parser.parseCSV("csvFile.csv"), cb);
thread.eval(parser.parseCSV);
thread.eval(parser.parseCSV("csvFile.csv"), cb);
thread.eval(parser.parseCSV);
thread.eval(parser.parseCSV("csvFile.csv"), cb);
When I run a "test[x].js" file using the command node test[x].js in the command prompt, the threads seem to be working but it appears as if they are being executed sequentially, and not in parallel. I say this based on the outputs of the time each thread starts, which I print out to command prompt window.
How do I execute multiple Webworker-Threads which run in parallel in the background? I want the threads to start at the same time and end at the same (if that is possible). I have looked at the API for the WebWorker-Threads but I haven't been able to solve this problem. Once this works, I would like to use the same methodology to not only read the csv files, but also store their contents into a database.
Any help would be greatly appreciated! Thank you very much :)

NodeJS exec with binary from and to the process

I'm trying to write a function, that would use native openssl to do some RSA heavy-lifting for me, rather than using a js RSA library. The target is to
Read binary data from a file
Do some processing in the node process, using JS, resulting in a Buffer containing binary data
Write the buffer to the stdin stream of the exec command
RSA encrypt/decrypt the data and write it to the stdout stream
Get the input data back to a Buffer in the JS-process for further processing
The child process module in Node has an exec command, but I fail to see how I can pipe the input to the process and pipe it back to my process. Basically I'd like to execute the following type of command, but without having to rely on writing things to files (didn't check the exact syntax of openssl)
cat the_binary_file.data | openssl -encrypt -inkey key_file.pem -certin > the_output_stream
I could do this by writing a temp file, but I'd like to avoid it, if possible. Spawning a child process allows me access to stdin/out but haven't found this functionality for exec.
Is there a clean way to do this in the way I drafted here? Is there some alternative way of using openssl for this, e.g. some native bindings for openssl lib, that would allow me to do this without relying on the command line?
You've mentioned spawn but seem to think you can't use it. Possibly showing my ignorance here, but it seems like it should be just what you're looking for: Launch openssl via spawn, then write to child.stdin and read from child.stdout. Something very roughly like this completely untested code:
var util = require('util'),
spawn = require('child_process').spawn;
function sslencrypt(buffer_to_encrypt, callback) {
var ssl = spawn('openssl', ['-encrypt', '-inkey', ',key_file.pem', '-certin']),
result = new Buffer(SOME_APPROPRIATE_SIZE),
resultSize = 0;
ssl.stdout.on('data', function (data) {
// Save up the result (or perhaps just call the callback repeatedly
// with it as it comes, whatever)
if (data.length + resultSize > result.length) {
// Too much data, our SOME_APPROPRIATE_SIZE above wasn't big enough
}
else {
// Append to our buffer
resultSize += data.length;
data.copy(result);
}
});
ssl.stderr.on('data', function (data) {
// Handle error output
});
ssl.on('exit', function (code) {
// Done, trigger your callback (perhaps check `code` here)
callback(result, resultSize);
});
// Write the buffer
ssl.stdin.write(buffer_to_encrypt);
}
You should be able to set encoding to binary when you make a call to exec, like..
exec("openssl output_something_in_binary", {encoding: 'binary'}, function(err, out, err) {
//do something with out - which is in the binary format
});
If you want to write out the content of "out" in binary, make sure to set the encoding to binary again, like..
fs.writeFile("out.bin", out, {encoding: 'binary'});
I hope this helps!

WIX: Where and how should my CustomAction create and read a temporary file?

I have a script CustomAction (Yes, I know all about the opinions that say don't use script CustomActions. I have a different opinion.)
I'd like to run a command, and capture the output. I can do this using the WScript.Shell COM object, then invoking shell.Exec(). But, this flashes a visible console window for the executed command.
To avoid that, I understand I can use the shell.Run() call, and specify "hidden" for the window appearance. But .Run() doesn't give me access to the StdOut of the executed process, so that means I'd need to create a temporary file and redirect the exe output to the temp file, then later read that temp file in script.
Some questions:
is this gonna work?
How do I generate a name for the temporary file? In .NET I could use a static method in the System.IO namespace, but I am using script here. I need to insure that the use has RW access, and also that no anti-virus program is going to puke on this.
Better ideas? I am trying very hard to avoid C/C++.
I could avoid all this if there were a way to query websites in IIS7 from script, without resorting to the IIS6 Compatibility pack, without using .NET (Microsoft.Web.Administration.ServerManager), and without execing a process (appcmd list sites).
I already asked a separate question on that topic; any suggestions on that would also be appreciated.
Answering my own question...
yes, this is going to work.
Use the Scripting.FileSystemObject thing within Javascript. There's a GetTempName() method that produces a file name suitable for temporary use, and a GetSpecialFolder() method that gets the location of the temp folder. There's even a BuildPath() method to combine them.
so far I don't have any better ideas.
Here's the code I used:
function GetWebSites_IIS7_B()
{
var ParseOneLine = function(oneLine) {
...regex parsing of output...
};
LogMessage("GetWebSites_IIS7_B() ENTER");
var shell = new ActiveXObject("WScript.Shell");
var fso = new ActiveXObject("Scripting.FileSystemObject");
var tmpdir = fso.GetSpecialFolder(SpecialFolders.TemporaryFolder);
var tmpFileName = fso.BuildPath(tmpdir, fso.GetTempName());
var windir = fso.GetSpecialFolder(SpecialFolders.WindowsFolder);
var appcmd = fso.BuildPath(windir,"system32\\inetsrv\\appcmd.exe") + " list sites";
// use cmd.exe to redirect the output
var rc = shell.Run("%comspec% /c " + appcmd + "> " + tmpFileName, WindowStyle.Hidden, true);
// WindowStyle.Hidden == 0
var ts = fso.OpenTextFile(tmpFileName, OpenMode.ForReading);
var sites = [];
// Read from the file and parse the results.
while (!ts.AtEndOfStream) {
var oneLine = ts.ReadLine();
var line = ParseOneLine(oneLine);
LogMessage(" site: " + line.name);
sites.push(line);
}
ts.Close();
fso.DeleteFile(tmpFileName);
return sites;
}

Categories

Resources