Develop Reference

Decoding a device payload sent to the Helium Console

So I have defined an integration in the Helium Console (i.e. - an IoT LoRaWAN networking service). When I test out sending data from my IoT device to my Helium Console integration endpoint, I can see the payload. And I can manually decode it back to the JSON request body that's correct. But am having trouble defining a custom function in the Helium Console that will automate this for the integration. My IoT device is serializing a MsgPack payload of environmental data that's then sent to the Helium Console integration endpoint. And the Helium Console custom function allows Javascript, although just bare-bones without any requires, imports, etc.
Below is the Python script that can manually decode the payload. If anyone is familiar with this area I'd be interested as to suggestions. There are pre-defined decoders available for Helium, although none that match exactly my hardware.
If anyone has created custom decoding functions for Helium Console integrations I'd be curious as to how I could knock this out. Here is some info about this --> https://docs.helium.com/use-the-network/console/functions/.
import ssl, sys, json, base64
import msgpack
payload = "haJiVspAozMzo3RfQxilZWNfbVMWonBoB6NvcnAC"
# decode the payload from base64 and unpack
b64 = base64.b64decode(payload)
mp_dict = msgpack.unpackb(b64)
print(mp_dict) # output is {'bV': 5.099999904632568, 't_C': 24, 'orp': 2, 'ph': 7, 'ec_mS': 22}

Rather than reinvent the wheel, I modified the Arduino code running on my device. Pulling in the CayenneLPP library I manually encode what I need, so that the Helium Console's canned CayenneLPP decoding function can correctly parse the data.
Arduino code snippet below.
#include <CayenneLPP.h>
...
/*CayenneLPP variable*/
CayenneLPP lpp(51);
...
static void prepareTxFrame( uint8_t port )
{
appDataSize = lpp.getSize();
uint8_t lppBuffer[128];
memcpy(lppBuffer, lpp.getBuffer(), 128);
memcpy(appData, lppBuffer, lpp.getSize());
}
...
// The loop function is called in an endless loop
void loop()
{
lpp.reset();
lpp.addTemperature(1, 22.5);
lpp.addBarometricPressure(2, 1073.21);
lpp.addGPS(3, 52.37365, 4.88650, 2);
...
LoRaWAN.displaySending();
prepareTxFrame( appPort );
LoRaWAN.send(loraWanClass);
...
}

https://github.com/helium/console-decoders/blob/master/debugDecodeBase64.js
// LoRaWan devices automatically base64 encode data sent
// Sometimes nice just to view what original payload was sent
//
// This can help when writing your own decoder
//
//
function Decoder(bytes, port, uplink_info) {
var decoded={};
try{
var result = String.fromCharCode.apply(null, bytes);
decoded.value = result;
return decoded;
} catch (err) {
return 'Decoder: ' + err.name + " : " + err.message;;
}
}
// And in the integration section of the Helium console use this JSON
// For an Adafruit.io integration which is expecting JSON like { "value" : "56" }
//
// { "value": {{decoded.payload.value}} }
Create account on https://app.datacake.de/
Create integration to Datacake in here: https://console.helium.com/integrations
integrations
Link the flow in here:
https://console.helium.com/flows
flows
Create project and add device to DataCake, as decoder function use modified above (as is - it will receive a string)

Unpack a C struct on browser?

EDIT
I found this:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Typed_arrays
Basically if I have something like this:
struct someStruct {
unsigned long id;
char username[16];
float amountDue;
};
on client side I can do:
var ws = new WebSocket("ws://URI");
ws.binaryType = "arraybuffer";
ws.onmessage = function (e) {
var buffer = e.data;
var data_view = new DataView(buffer);
// ... read the data into the buffer ...
var idView = data_view.getUint32(0);
var usernameView = data_view.getUint32(4);
var amountDueView = data_view.getFloat32(20);
};
The problem is that I want to convert them to normal Javascript objects (numbers, strings etc).
Original question
I would send data via websocket packed using a C struct, and unpack on browser using Javascript.
I know modules exists for node.js, but I can't find nothing client-side.

If you're familiar with Python struct, then you may like structjs. It's my attempt at porting Python struct to javascript. As it is, it's for Node, but a client port should be easy.
You won't have issues with alignment or padding (you can specify those explicitly though) for that structure, but you may need to indicate little-endian (by the '<' in the format string) if that's your flavour. You might do like so (I haven't tested this example in any way):
let struct = require("./struct") // Node specific, you need to wrap it.
let someStruct = struct('<I16sf') // This is your struct definition
let ws = new WebSocket("ws://URI");
ws.binaryType = "arraybuffer";
ws.onmessage = e => {
// Unpack using the structure definition. Unpack takes an ArrayBuffer.
let [id, username, amountDue] = someStruct.unpack(e.data);
// Use data...
};

Ok,
https://www.npmjs.com/package/c-struct looks like what you want.
Good luck!

Ok, after some researching, I finally decided this could not be a good idea:
https://justin.harmonize.fm/development/2013/04/28/a-slower-js-msgpack.html
Shortly: javascript is slow in decoding.
Probably it's just more simple to use JSON and Content-Encoding: gzip, if it does not slow down your web app.

IPC on Terminal Server with C/C++ an nw.js/node.js/node-native-module (C++)?

I have a Win32-DLL (C++) which is loaded as a plugin in another application. The DLL starts a nw.js instance (ShellExecuteEx and SEE_MASK_NOCLOSEPROCESS) and ends it at DLL unloading (by the hInstance of ShellExecuteEx). I need a way to send a string (plain ansi) to the nw-process and retrieve an answer (also string). The old way was a simple http-request with the response in the body. But the environment changes during the development, the "package" app-dll-nw runs multiple times by the same user and multiple users run on the same machine (terminal server). So port listing is "impossible" (yeah random ports or singleton nw, but no).
I found different ways:
socket - port listing problem
wm_copydata/wm_... - need a custom nw-plugin with hidden window (no native nw way); no request-response-system
RPC - port listing problem
DDE - no native javascript way (found a module, which uses .net); In my old delphi days DDE was a not so simple task and it failed multiple times with no logic.
shared memory - no experience; expectations: asynchronous, trigger?, no native javascript way
shared file - no experience; expectations: asynchronous, trigger (watcher on file change) but problems with synchronization, native js way possible
named pipe - no experience; expectations: win32-api and like a chat system (in-pipe [send broadcast] and out-pipe [receive broadcast], or both in one)? If yes, I can use one name about all instances and use unique identifiers and wait for the right answer.
What is a nice and simple way to communicate like the http-way but w/o networking?
Update 1: The node module "net" is able to create a server for a named pipe. The first test, sending a string from the dll to nw, was successful.
var server = net.createServer(function(stream) {
stream.on('data', function(c) {
console.log('data:', c.toString());
});
stream.on('end', function() {
//server.close();
});
});
server.listen('\\\\.\\pipe\\MyAppDynamicGUID');
Update 2 - My Solution
With named pipe and a simplified version of https://msdn.microsoft.com/en-us/library/windows/desktop/aa365592(v=vs.85).aspx I found a working methode.
Server in nw.js:
var server = net.createServer(function(req) {
req.on('data', function(c) {
console.log(c.toString());
req.write('123|Hello World', 'ascii');
});
});
server.listen('\\\\.\\pipe\\MyAppDynamicGUID');
The client in C++ (no permanent connection, strange string handling, simplified error handling):
static std::string PipenameA = "\\\\.\\pipe\\MyAppDynamicGUID";
#define BUFSIZE 512
std::string SendPipeRequestA(std::string sRequest) {
DWORD dwToWrite, dwWritten, dwRead;
BOOL bSuccess;
char chBuf[BUFSIZE];
std::vector<char> buffer;
HANDLE hPipe = CreateFileA(PipenameA.c_str(), GENERIC_READ | GENERIC_WRITE, 0, NULL, OPEN_EXISTING, 0, NULL);
if (hPipe == INVALID_HANDLE_VALUE)
return "-1|Pipe-Error 1 (connect)";
dwToWrite = (lstrlenA(sRequest.c_str())+1)*sizeof(char);
bSuccess = WriteFile(hPipe, sRequest.c_str(), dwToWrite, &dwWritten, NULL);
if (!bSuccess)
return "-1|Pipe-Error 2 (write)";
do {
bSuccess = ReadFile(hPipe, chBuf, BUFSIZE*sizeof(char), &dwRead, NULL);
if (!bSuccess && GetLastError() != ERROR_MORE_DATA)
break;
buffer.insert(buffer.end(), chBuf, chBuf + dwRead);
} while (!bSuccess);
std::string sResponse(&buffer[0]);
CloseHandle(hPipe);
return sResponse.c_str();
}
// Jonny

The answers you will get will be opinion based, be aware of that.
you can inject the data into the JS module as command line argument
for example
start nw.js MyData
and get it insinde the JS with process.argv.
now, sending the data back to the C++ executables/DLLs is a bit tricky.
if you shell-execute the process, you can have the handle to it.
you can print the data into the stdout from the JS part , and read it in the native app by getting the STDOUT handle from the process handle.

Register your nw.js app with a custom url should be an elegant way.
Such as "github://", "thunder://", "twitter://"
On windows you may have a look at:
https://msdn.microsoft.com/en-us/library/aa767914(v=vs.85).aspx
With custom url you can take simple arguments to nw.js at single-instance mode. See:
https://github.com/nwjs/nw.js/wiki/Handling-files-and-arguments#open-file-with-existing-app
If more data required maybe base64 can help, or even more by LZ-String compress method.

Deconstruct/decode Websocket frames like Google Chrome does

I'm connecting to a website via websocket connection (client to server), I know how to encode the data and write it to the server (using the net module in node.js) but when I'm reading the data back I get odd characters in front of the important data, like if I'm suppose to get:
// Data needed on the left and data I'm receiving from websocket on the right
'inited\r\n' -> '�inited\r\n'
'n:2\r\n' -> '�n:2\r\n'
This is how I am getting the data from the server
Klass.prototype.connect = function(){
// this.port is equal to 8080 and the exact server varys, but it's not that important anyways since the problem is decoding the data properly.
var that = this;
var buffer = "";
this.socket = new net.createConnection(this.port, this.server);
this.socket
.on("connect", function(){
that.sendHandshake(); // just sends a standard client to server handshake
})
.on("data", function(recv){
// .split('\r\n\r\n').join('\r\n') needed to separate the server handshake from the data I am trying to parse
buffer += recv.toString('utf-8').split('\r\n\r\n').join('\r\n');
while (buffer){
var offset = buffer.indexOf('\r\n');
if (offset < 0)
return;
var msg = buffer.slice(0, offset);
// parseMsg(msg)
buffer = buffer.slice(offset + 3);
}
});
};
I am probably doing a lot of things improperly in the code above, but I'm not quite sure how to do it exactly so that is the best I got for now.
Problem is I don't know how to remove the mystery/special characters. Sometimes there is only 1 mystery/special character, but other times there is multiple ones depending on the data but they are never after the important data I need to check.
When I use Google Chrome and view the data on through tools->JavaScript console->network tab and find the websocket stream I'm looking for Google parses it correctly. I know it's possible since Google Chrome shows the correct frames, how do I deconstruct/decode the data so I can view the correct frames on the terminal?
I don't really need it in a particular language as long as it works I should be able to port it, but I would prefer examples/answers in node.js since that is the programming language I am using to connect to the server.

NodeJS exec with binary from and to the process

I'm trying to write a function, that would use native openssl to do some RSA heavy-lifting for me, rather than using a js RSA library. The target is to
Read binary data from a file
Do some processing in the node process, using JS, resulting in a Buffer containing binary data
Write the buffer to the stdin stream of the exec command
RSA encrypt/decrypt the data and write it to the stdout stream
Get the input data back to a Buffer in the JS-process for further processing
The child process module in Node has an exec command, but I fail to see how I can pipe the input to the process and pipe it back to my process. Basically I'd like to execute the following type of command, but without having to rely on writing things to files (didn't check the exact syntax of openssl)
cat the_binary_file.data | openssl -encrypt -inkey key_file.pem -certin > the_output_stream
I could do this by writing a temp file, but I'd like to avoid it, if possible. Spawning a child process allows me access to stdin/out but haven't found this functionality for exec.
Is there a clean way to do this in the way I drafted here? Is there some alternative way of using openssl for this, e.g. some native bindings for openssl lib, that would allow me to do this without relying on the command line?

You've mentioned spawn but seem to think you can't use it. Possibly showing my ignorance here, but it seems like it should be just what you're looking for: Launch openssl via spawn, then write to child.stdin and read from child.stdout. Something very roughly like this completely untested code:
var util = require('util'),
spawn = require('child_process').spawn;
function sslencrypt(buffer_to_encrypt, callback) {
var ssl = spawn('openssl', ['-encrypt', '-inkey', ',key_file.pem', '-certin']),
result = new Buffer(SOME_APPROPRIATE_SIZE),
resultSize = 0;
ssl.stdout.on('data', function (data) {
// Save up the result (or perhaps just call the callback repeatedly
// with it as it comes, whatever)
if (data.length + resultSize > result.length) {
// Too much data, our SOME_APPROPRIATE_SIZE above wasn't big enough
}
else {
// Append to our buffer
resultSize += data.length;
data.copy(result);
}
});
ssl.stderr.on('data', function (data) {
// Handle error output
});
ssl.on('exit', function (code) {
// Done, trigger your callback (perhaps check `code` here)
callback(result, resultSize);
});
// Write the buffer
ssl.stdin.write(buffer_to_encrypt);
}

You should be able to set encoding to binary when you make a call to exec, like..
exec("openssl output_something_in_binary", {encoding: 'binary'}, function(err, out, err) {
//do something with out - which is in the binary format
});
If you want to write out the content of "out" in binary, make sure to set the encoding to binary again, like..
fs.writeFile("out.bin", out, {encoding: 'binary'});
I hope this helps!