I'm connecting to a website via websocket connection (client to server), I know how to encode the data and write it to the server (using the net module in node.js) but when I'm reading the data back I get odd characters in front of the important data, like if I'm suppose to get:
// Data needed on the left and data I'm receiving from websocket on the right
'inited\r\n' -> '�inited\r\n'
'n:2\r\n' -> '�n:2\r\n'
This is how I am getting the data from the server
Klass.prototype.connect = function(){
// this.port is equal to 8080 and the exact server varys, but it's not that important anyways since the problem is decoding the data properly.
var that = this;
var buffer = "";
this.socket = new net.createConnection(this.port, this.server);
this.socket
.on("connect", function(){
that.sendHandshake(); // just sends a standard client to server handshake
})
.on("data", function(recv){
// .split('\r\n\r\n').join('\r\n') needed to separate the server handshake from the data I am trying to parse
buffer += recv.toString('utf-8').split('\r\n\r\n').join('\r\n');
while (buffer){
var offset = buffer.indexOf('\r\n');
if (offset < 0)
return;
var msg = buffer.slice(0, offset);
// parseMsg(msg)
buffer = buffer.slice(offset + 3);
}
});
};
I am probably doing a lot of things improperly in the code above, but I'm not quite sure how to do it exactly so that is the best I got for now.
Problem is I don't know how to remove the mystery/special characters. Sometimes there is only 1 mystery/special character, but other times there is multiple ones depending on the data but they are never after the important data I need to check.
When I use Google Chrome and view the data on through tools->JavaScript console->network tab and find the websocket stream I'm looking for Google parses it correctly. I know it's possible since Google Chrome shows the correct frames, how do I deconstruct/decode the data so I can view the correct frames on the terminal?
I don't really need it in a particular language as long as it works I should be able to port it, but I would prefer examples/answers in node.js since that is the programming language I am using to connect to the server.
Related
Diving into my first chrome extension, and trying to figure out how to modify some data in http requests.
I'm using the documentation here: https://developer.chrome.com/extensions/webRequest
I was able to setup the extension to listen for requests, but am not able to access the data I want.
When I'm in the chrome dev tools, on the Network tab, I right click the particular request I'm trying to modify and copy as cURL. The data I want to modify shows up after --data. I want to access this and change an integer value one of the parameters is set to.
I'm not sure what the equivalent is with these http requests, but I've tried the following:
chrome.webRequest.onBeforeRequest.addListener(
function(details) {
var bkg = chrome.extension.getBackgroundPage();
bkg.console.log("onBeforeRequest");
bkg.console.log(JSON.stringify(details));
blockingResponse = {};
return blockingResponse;
},
{urls: [/*URL*/]},
['requestBody','blocking']
);
I can find the request with the url that I am looking at in the Network tab of the dev tools, so I'll be able to parse that and make sure I'm only modifying the requests that I want to, but printing the details doesn't show the data that I actually want to modify. Any idea how to obtain the HTTP request equivalent of the --data argument of a cURL request? And, well, modify it.
Edit: Here's the progress I've made.
When I log those details, I get ..."requestBody":{"raw":[{"bytes":{}}]},...
However, if I change onBeforeRequest to:
chrome.webRequest.onBeforeRequest.addListener(
function(details) {
var bkg = chrome.extension.getBackgroundPage();
bkg.console.log("onBeforeRequest");
bkg.console.log(JSON.stringify(details));
var encodedData = getBase64FromArrayBuffer(details.requestBody.raw[0].bytes);
bkg.console.log("unencoded data: " + details.requestBody.raw[0].bytes);
bkg.console.log("encodedData: " + encodedData);
blockingResponse = {};
return blockingResponse;
},
{urls: ["*://*.facebook.com/*"], types: ["xmlhttprequest"]},
['requestBody','blocking']
);
function getBase64FromArrayBuffer(responseData) {
var uInt8Array = new Uint8Array(responseData);
var i = uInt8Array.length;
var binaryString = new Array(i);
while (i--)
{
binaryString[i] = String.fromCharCode(uInt8Array[i]);
}
var data = binaryString.join('');
var base64 = window.btoa(data);
return base64;
}
The encoded data exists, showing a long string of chars, though it's gibberish. Does this mean that I won't be able to access this data and modify it? Or is there a way to decode this data?
The chrome.webRequest API does allow you to access POST data. It does not, however, allow you to modify the POST data.
You are able to modify some of the header info, but not the POST data.
It appears the ability to modify POST data was intended, but a dev at Google who was working on it got moved to something else, and sat on the bug/feature request for two years afterwards, and just released it so someone else could pick it up a few months ago. If this is a feature that interests you, head to https://bugs.chromium.org/p/chromium/issues/detail?id=91191# and star this bug (requires gmail account), and perhaps some renewed interest will lead to someone completing the functionality.
I want to download and play m3u8 file which is on server machine. I am using following code to read and send m3u8 file to web server.
Browser is displaying contents of file instead of downloading it.
So please let me know that, how to download it.
if ((exportHandle = fopen(v3FileName, "a+")) != NULL) {
long end = 0, start = 0, pos = 0;
char* m3u8FileDataBuff = NULL;
fseek(exportHandle, 0, SEEK_END);
end = ftell(exportHandle);
fseek(exportHandle, 0, SEEK_SET);
start = ftell(exportHandle);
pos = end - start;
m3u8FileDataBuff = (char *) malloc(pos);
end = 0;
start = 0;
fread(m3u8FileDataBuff, 1, pos, exportHandle);
pClienCommunication->writeBuffer(m3u8FileDataBuff, pos);
free(m3u8FileDataBuff);
fclose(exportHandle);
}
Client's web browser is displaying the content, because the MIME type of the response is either nil, or something like "text/plain". Set up the http response header properly to indicate mime type of m3u8 file (application/x-mpegURL or vnd.apple.mpegURL).
The piece of code you provided does not seem to set anything around response header, just content.
Check available API of pClienCommunication->, or place where that originates, what are your options to adjust response header.
Or maybe it's possible to work-around this also by some rule set up in the web server serving the response, to set the MIME type for certain URLs, or based on the response content (but applying such rules on web server level is usually more costly then adjusting the response while being created in the C++ part).
And why is this tagged C++, when the code itself is C-like with all the problems of it. In modern C++ you never do things like "fclose(..)", because that is done in the destructor of the file wrapper class, so you don't risk the fclose will be skipped in case of some exception raised in fread, etc.
So in modern C++ these things should look somewhat like this:
{
SomeFileClass exportFile(v3FileName, "a+");
if (exportFile.isOK()) {
SomeFileContentBuffer data = exportFile.read();
pClienCommunication->writeBuffer(data.asCharPtr(), data.size());
}
}
So you can't forget to release any file handle, or buffer memory (as the destructors of particular helper classes will handle that).
I have the typical code to start a webworker in node:
var Threads = require('webworker-threads');
var worker = new Threads.Worker(__dirname + '/workers/myworker.js');
worker.onmessage = function (event) {
// 1.
// ... create and execute cypher query ...
};
// Start the worker.
worker.postMessage({
'data' : data
});
At 1. I send small pieces of processed data to a Neo4J db.
For small data this works perfectly fine, but when the data gets slightly bigger node/the worker starts to struggle.
The actual data I want to process is a csv I parsed with BabyParse resulting in an object with 149000 properties where each has another 17 properties. (149000 rows by 17 columns = 2533000 properties). The file is 17MB.
When doing this node will allocate a lot of memory and eventually crash around 53% memory allocation. The machine has 4GB.
The worker looks roughly like this:
self.onmessage = function (event) {
process(event.data.data);
};
function process(data) {
for (var i = 0; i < data.length; i++) {
self.postMessage({
'properties' : data[i]
});
}
}
I tried to chunk the data and process it chunkwise within the worker which also works okay. But I want to generate a graph and to process the edges I need the complete data, because I need to check every row (vertex) against all others.
Is there a way to stream the data into the worker? Or does anyone have an idea why node allocates so much memory with 17MB of data being send?
Instead of parsing the data in the main thread you can also pass the filename as a message to the worker and have the worker load it from disk. Otherwise you will have all the data in memory twice, once in the host and once in the worker.
A different option would be to use the csv npm package with the streaming parser. postMessage the lines as they come in and buffer them up till the final result in the worker.
Why your solution tries to allocate those enormous amounts of memory I don't know. I do know postMessage is intended to pass small messages.
I have a Win32-DLL (C++) which is loaded as a plugin in another application. The DLL starts a nw.js instance (ShellExecuteEx and SEE_MASK_NOCLOSEPROCESS) and ends it at DLL unloading (by the hInstance of ShellExecuteEx). I need a way to send a string (plain ansi) to the nw-process and retrieve an answer (also string). The old way was a simple http-request with the response in the body. But the environment changes during the development, the "package" app-dll-nw runs multiple times by the same user and multiple users run on the same machine (terminal server). So port listing is "impossible" (yeah random ports or singleton nw, but no).
I found different ways:
socket - port listing problem
wm_copydata/wm_... - need a custom nw-plugin with hidden window (no native nw way); no request-response-system
RPC - port listing problem
DDE - no native javascript way (found a module, which uses .net); In my old delphi days DDE was a not so simple task and it failed multiple times with no logic.
shared memory - no experience; expectations: asynchronous, trigger?, no native javascript way
shared file - no experience; expectations: asynchronous, trigger (watcher on file change) but problems with synchronization, native js way possible
named pipe - no experience; expectations: win32-api and like a chat system (in-pipe [send broadcast] and out-pipe [receive broadcast], or both in one)? If yes, I can use one name about all instances and use unique identifiers and wait for the right answer.
What is a nice and simple way to communicate like the http-way but w/o networking?
Update 1: The node module "net" is able to create a server for a named pipe. The first test, sending a string from the dll to nw, was successful.
var server = net.createServer(function(stream) {
stream.on('data', function(c) {
console.log('data:', c.toString());
});
stream.on('end', function() {
//server.close();
});
});
server.listen('\\\\.\\pipe\\MyAppDynamicGUID');
Update 2 - My Solution
With named pipe and a simplified version of https://msdn.microsoft.com/en-us/library/windows/desktop/aa365592(v=vs.85).aspx I found a working methode.
Server in nw.js:
var server = net.createServer(function(req) {
req.on('data', function(c) {
console.log(c.toString());
req.write('123|Hello World', 'ascii');
});
});
server.listen('\\\\.\\pipe\\MyAppDynamicGUID');
The client in C++ (no permanent connection, strange string handling, simplified error handling):
static std::string PipenameA = "\\\\.\\pipe\\MyAppDynamicGUID";
#define BUFSIZE 512
std::string SendPipeRequestA(std::string sRequest) {
DWORD dwToWrite, dwWritten, dwRead;
BOOL bSuccess;
char chBuf[BUFSIZE];
std::vector<char> buffer;
HANDLE hPipe = CreateFileA(PipenameA.c_str(), GENERIC_READ | GENERIC_WRITE, 0, NULL, OPEN_EXISTING, 0, NULL);
if (hPipe == INVALID_HANDLE_VALUE)
return "-1|Pipe-Error 1 (connect)";
dwToWrite = (lstrlenA(sRequest.c_str())+1)*sizeof(char);
bSuccess = WriteFile(hPipe, sRequest.c_str(), dwToWrite, &dwWritten, NULL);
if (!bSuccess)
return "-1|Pipe-Error 2 (write)";
do {
bSuccess = ReadFile(hPipe, chBuf, BUFSIZE*sizeof(char), &dwRead, NULL);
if (!bSuccess && GetLastError() != ERROR_MORE_DATA)
break;
buffer.insert(buffer.end(), chBuf, chBuf + dwRead);
} while (!bSuccess);
std::string sResponse(&buffer[0]);
CloseHandle(hPipe);
return sResponse.c_str();
}
// Jonny
The answers you will get will be opinion based, be aware of that.
you can inject the data into the JS module as command line argument
for example
start nw.js MyData
and get it insinde the JS with process.argv.
now, sending the data back to the C++ executables/DLLs is a bit tricky.
if you shell-execute the process, you can have the handle to it.
you can print the data into the stdout from the JS part , and read it in the native app by getting the STDOUT handle from the process handle.
Register your nw.js app with a custom url should be an elegant way.
Such as "github://", "thunder://", "twitter://"
On windows you may have a look at:
https://msdn.microsoft.com/en-us/library/aa767914(v=vs.85).aspx
With custom url you can take simple arguments to nw.js at single-instance mode. See:
https://github.com/nwjs/nw.js/wiki/Handling-files-and-arguments#open-file-with-existing-app
If more data required maybe base64 can help, or even more by LZ-String compress method.
While trying to create a tool(with nodejs, socket.io) which runs an external program (e.g. a C program or a Python program), the program stops responding when it encounters the user input (it sends the line asking input to browser though).
On Server:
var chunk = '';
python.stdout.on('data', function(data){
python.stdout.pipe(python.stdin);
chunk += data;
console.log(chunk);
socket.emit('newdata', chunk);
} );
On Client:
socket.on('newdata', function(d){
var output = document.getElementById('output');
output.innerHTML = d;
})
Everything is working fine in fact, the only problem is that the program you launched requires a user input and your node.js code is not giving it.
Tbh I don't know if you can give an input when it asks for it... but can't you pass arguments at the start of the program to avoid this user input? It's how I solved it on the same system done with Java/J2EE