I need to receive a zip file from a server, extract it and pipe its contents elsewhere.
However when trying to extract it with createInflate from the zlib builtin package, I get the error Error: incorrect header check
(I have tried with createUnzip and createGunzip too)
Downloading the file with cUrl and extracting it with the unzip linux command works correctly.
$ unzip report.zip
Archive: report.zip
inflating: report.csv
$ zipinfo -v report.zip
[...]
file system or operating system of origin: MS-DOS, OS/2 or NT FAT
version of encoding software: 2.0
minimum file system compatibility required: MS-DOS, OS/2 or NT FAT
minimum software version required to extract: 2.0
compression method: deflated
compression sub-type (deflation): normal
file security status: not encrypted
extended local header: yes
[...]
Code used to extract the already downloaded file:
const pipeline = promisify(stream.pipeline);
(async () => {
const unzipper = createInflate();
const sourceStream = fs.createReadStream('report.zip');
const destStream = fs.createWriteStream('report.csv');
await pipeline(sourceStream, unzipper, destStream);
})();
Note that the error is the same between piping the response directly and piping the result of createReadStream
Full zipinfo -v:
$ zipinfo -v report.zip
Archive: report.zip
There is no zipfile comment.
End-of-central-directory record:
-------------------------------
Zip archive file size: 527 (000000000000020Fh)
Actual end-cent-dir record offset: 505 (00000000000001F9h)
Expected end-cent-dir record offset: 505 (00000000000001F9h)
(based on the length of the central directory and its expected offset)
This zipfile constitutes the sole disk of a single-part archive; its
central directory contains 1 entry.
The central directory is 78 (000000000000004Eh) bytes long,
and its (expected) offset in bytes from the beginning of the zipfile
is 427 (00000000000001ABh).
Central directory entry #1:
---------------------------
report_SMS_1f7c2069_20200730.csv
offset of local header from start of archive: 0
(0000000000000000h) bytes
file system or operating system of origin: MS-DOS, OS/2 or NT FAT
version of encoding software: 2.0
minimum file system compatibility required: MS-DOS, OS/2 or NT FAT
minimum software version required to extract: 2.0
compression method: deflated
compression sub-type (deflation): normal
file security status: not encrypted
extended local header: yes
file last modified on (DOS date/time): 2020 Jul 30 11:05:48
32-bit CRC value (hex): 5abe6238
compressed size: 349 bytes
uncompressed size: 934 bytes
length of filename: 32 characters
length of extra field: 0 bytes
length of file comment: 0 characters
disk number on which file begins: disk 1
apparent file type: binary
non-MSDOS external file attributes: 000000 hex
MS-DOS file attributes (00 hex): none
There is no file comment.
zlib is not zip. zip is not zlib. They are two different formats. gzip is yet another. (The use of the word "unzip" in the node.js zlib interface is misleading.)
You need something that unzips zip files. Take a look at this.
Related
I'm trying to save an image by spawning a python process from a node.js script. The image is passed as a binary data file from a Node.JS script to a Python process.
The Script index.js spawns a python script my_py.py, then passes it a binary image data file. The python script my_py.py captures the binary image data and saves it to the directory assets/media.
The problem is the image isn't saved to the directory and I get the error error spawn ENAMETOOLONG.
Could you please help me spot the problem and to fix the code?
Thanks in advance!
index.js:
const spawn = require("child_process").spawn;
const fs = require('fs');
let params = {
"image":readDataset()
}
// Launch python app
const pythonProcess = spawn('py',['my_py.py', JSON.stringify(params)]);
// Print result
pythonProcess.stdout.on("data", (data) =>{
console.log(data.toString());
});
// Read errors
pythonProcess.stderr.on("data", (data) =>{
console.log(data.toString());
});
function readDataset() {
try {
return fs.readFileSync('color.png', 'binary');
}
catch (err) {
return err;
}
}
my_py.py:
import sys, json
params = json.loads(sys.argv[1]) # Load passed arguments
import numpy as np
import cv2
import os
fileName = "image.png" # file name
fileData = params['image'] # Capture file
# Convert Image to Numpy as array
img = np.array(fileData)
# Save file to local directory
cv2.imwrite(os.path.join('assets/media/', f'{fileName}'), img)
cv2.waitKey(0)
# Results must return valid JSON Object
print("File saved!")
sys.stdout.flush()
The error says it all: the command you are constructing is too long.
The thing you need to be aware of is that operating systems limits how long a command can be. For Linux this is around 4096 bytes though you can modify this value. For Windows this is 8191 bytes and for Mac OS this is around 250k bytes.
Part of the reason for this is because these OSes were written in C/C++ and in C/C++ code, not enforcing buffer size limit is an invitation to buffer overrun (the infamous stack overflow or underflow bug!!). Truly making input size unlimited result in slow code because you will not be using simple arrays for input buffer in that case but more complicated data structures.
Additionally, not having any limit on command length is a vector for DOS attacks. If you run an OS that does not have command size limit and I know for sure you have 32GB of RAM all I need to do to crash your system is construct a 32GB command!!
TLDR
The correct way to pass large data between processes is what you've been doing over the internet - upload/download the data! The simplest implementation is to just pass the data via the stdin of the python process you are connected to:
const spawn = require("child_process").spawn;
const fs = require('fs');
let params = {
"image":readDataset()
}
// Launch python app
const pythonProcess = spawn('py',['my_py.py']);
// Pass image data
pythonProcess.stdin.write(JSON.stringify(params) + '\n');
// Print result
pythonProcess.stdout.on("data", (data) =>{
console.log(data.toString());
});
// Read errors
pythonProcess.stderr.on("data", (data) =>{
console.log(data.toString());
});
function readDataset() {
try {
return fs.readFileSync('color.png', 'base64');
}
catch (err) {
return err;
}
}
In the code above I end the JSON "packet" with a newline so that the python code can read until end of line for a single packet. You can use any convention you like. For example I also often use the nul character (0x00) to mark end of packet and HTTP use two newlines ('\n\n') to mark end of header etc.
In any case I read the image file as base64 because binary data is invalid in JSON. In your python code you can do a base64 decode to get the image back. Additionally base64 does not include newlines ('\n') in its character set so converting the image to base64 ensure you don't get a newline inside your JSON data.
To get the image just read from stdin:
import sys, json, base64
params = json.loads(sys.stdin.readline())
import numpy as np
import cv2
import os
fileName = "image.png" # file name
fileData = base64.b64decode(params['image']) # Capture file
# Convert Image to Numpy as array
img = np.array(fileData)
# Save file to local directory
cv2.imwrite(os.path.join('assets/media/', f'{fileName}'), img)
cv2.waitKey(0)
# Results must return valid JSON Object
print("File saved!")
sys.stdout.flush()
We are experiencing an issue while writing files using Node.js in our web application. Node.js is installed using IISNode.
Everything is working fine on our dev server, but same code is giving problem on our production server (Windows 7, IIS7, Windows Server 2008 R2 Standard, Service pack 1).
Basically, on a button click, multiple HTML files are created by node and content is written in them. The problem is that content is not written to all the created files. All files are created but some of the files are left blank with no content.
Here is an excerpt of code that is being used to create and write to files:
const $ = cheerio.load(cssData + videoHtml + pageData.content);
let indexFile = fs.openSync(file, 'w+')
fs.writeFileSync(indexFile, $.html(), { encoding: 'utf8' })
fs.closeSync(indexFile)
const $ = cheerio.load(cssData + videoHtml + pageData.content);
fs.writeFileSync(file, $.html(), { encoding: 'utf8' , mode : 0755})
fs.closeSync(indexFile)
Try adding mode in options
I have a problem with the source code generated by the Swagger codegen.
I want to upload a file with react. For this I have created a Dropzone and get a path to the file. If I use the generated client as it is in the documentation, it will not work. Unfortunately, the file is not sent. Only the file name. The debug console also does not show that binary data has been sent.
The request is not executed correctly. The file will not be uploaded. The parameter "file" is just the file name, instead of the binary data.
Swagger-codegen version
openapi-generator-cli-3.3.4.jar
Swagger declaration file content
Swagger .yaml:
/orders/upload:
post:
operationId: "orderUploadPart"
summary: "upload a textual or audio part of an order"
tags:
- "orders"
description: "This funktion uploads a textual or audio part of an order to the sense.ai.tion cloud system.
The result is the resource identifier, that must be used in the order request."
consumes:
- multipart/form-data
parameters:
- in: "formData"
name: "file"
type: "file"
required: true
description: "the file to upload"
- in: "formData"
name: "media"
type: "string"
enum:
- "text"
- "wav"
- "mp3"
required: true
description: "the media type of the the upload, can be ***text***, ***wav*** or ***mp3***"
Code:
var apiInstance = new SenseaitionApi.OrdersApi();
var file = "/path/to/file"; // File | the file to upload
var media = "media_example"; // String | the media type of the the upload, can be ***text***, ***wav*** or ***mp3***
var callback = function(error, data, response) {
if (error) {
console.error(error);
} else {
console.log('API called successfully. Returned data: ' + data);
}
};
apiInstance.orderUploadPart(file, media, callback);
It's like in: https://github.com/swagger-api/swagger-codegen/blob/master/samples/client/petstore/javascript/docs/PetApi.md#uploadFile
Screenshot Chrome DevTools
Command line used for generation
java -jar ${GOPATH}/bin/openapi-generator-cli.jar generate -i service_js_api.yaml -g javascript -o clients/javascript/senseaition-api-js -Dio.swagger.parser.util.RemoteUrl.trustAll=true
I found the mistake. The documentation of generated Javascript code is wrong. For uploading a file (Javascript object) must be passed, not the Path.
This line is wrong:
var file = "/path/to/file"; // File | the file to upload
Python 3.6.4
Windows10
I am trying to load a json file from url
However, when I try
d3.json("http://ip_addr/field/parameters", function(error, data) {
console.log(data)
}
I have an Access-Control-Allow-Origin error.
I tried a solution
click here to see more detail
But when I typed
C:\Program Files (x86)\Google\Chrome\Application>python -m http.server 8888 &
I have error message:
Traceback (most recent call last):
File "C:\Users\xieda\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "C:\Users\xieda\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\xieda\AppData\Local\Programs\Python\Python36\lib\http\server.py", line 1211, in <module>
test(HandlerClass=handler_class, port=args.port, bind=args.bind)
File "C:\Users\xieda\AppData\Local\Programs\Python\Python36\lib\http\server.py", line 1185, in test
with ServerClass(server_address, HandlerClass) as httpd:
File "C:\Users\xieda\AppData\Local\Programs\Python\Python36\lib\socketserver.py", line 453, in __init__
self.server_bind()
File "C:\Users\xieda\AppData\Local\Programs\Python\Python36\lib\http\server.py", line 138, in server_bind
self.server_name = socket.getfqdn(host)
File "C:\Users\xieda\AppData\Local\Programs\Python\Python36\lib\socket.py", line 673, in getfqdn
hostname, aliases, ipaddrs = gethostbyaddr(name)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 3: invalid start byte
How can I fix the problem? Or is there any solutions to my Access-Control-Allow-Origin error?
I am in the process of debugging a memory leak in our node application. I am however unable to use llnode to show any javascript objects. I've followed Brendan instructions.
I have tested on different versions of Ubuntu 16.04 and 17.04, as well as using node 7 and 8.
I went back and tried a simple case:
var http = require('http');
var host = "127.0.0.1"
var port = 1338
server = http.createServer(function myRequestListener(req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('Hello World\n');
res.not_a_function()
}).listen(port, host);
console.log(`Server process ${process.pid} running at http://${host}:${port}/`);
I then would send a seg fault signal (the above code is put into server.js):
kill -s SIGSEGV `ps -ax | grep server.js | cut -d " " -f 1 | head -n 1`
I then would run:
lldb-4.0 -c /cores/core.dump -f /usr/bin/node
Inside llnode when I queried for javascript objects I got back and empty list.
(lldb) v8 bt
* thread #1: tid = 28763, 0x00007f8947ff7929, name = 'node', stop reason = signal SIGSEGV
* frame #0: 0x00007f8947ff7929
frame #1: 0x00005585e8df035b node`v8_inspector::V8FunctionCall::callWithoutExceptionHandling() + 11
frame #2: 0x00005585e8ddf26c node`v8_inspector::V8Debugger::captureStackTrace(bool) + 92
frame #3: 0x00005585e8bc1493 node`node::inspector::InspectorSocketServer::SocketConnectedCallback(uv_stream_s*, int) + 163
frame #4: 0x00005585e8bbb821 node`node::inspector::InspectorConsoleCall(v8::FunctionCallbackInfo<v8::Value> const&) + 817
frame #5: 0x00007f8947f153f1
(lldb) v8 findjsobjects
Instances Total Size Name
---------- ---------- ----
(lldb)
Where I would expect to find javascript objects. I suspect it is a PEBKAC problem, but if anyone has any pointers to where I might have gone astray.
This is some kind of necroposting, but we had same problem with electron build - v8 findjsobjects returns no output.
All things below for MacOS builds.
So, the problem is in 'visibility' linker flag, which is forced to be 'hidden' in electron builds. That's why v8dbg_ constants, needed for llnode, not attached by linker even with debug ninja config, and llonde can't find any objects.
To beat this we forced default visibility with pragma in python-generated debug-output.cc file and finally was able to view js objects. to follow this path append
#pragma GCC visibility push(default)
before footer declaration in file electron-gn/src/v8/tools/gen-postmortem-metadata.py and rebuild all electron from scratch. Also build can be really huge, make sure you have ~100GB of free space.
I think same thing will works for node/v8 builds alongside electron.
Hope this helps.