I am using websockets for file transfer, while i am downloading a file i am recieving the data as it is, but when I open an image file it was corrupted. Data files are downloading fine, the code goes as follows.
try {
fileEntry = fs.root.getFile(filename, { create : creat_file });
var byteArray = new Uint8Array(data.data.length);
for (var i = 0; i < data.data.length; i++) {
byteArray[i] = data.data.charCodeAt(i) & 0xff;
}
BlobBuilderObj = new WebKitBlobBuilder();
BlobBuilderObj.append(byteArray.buffer);
if (!writer) {
writer = fileEntry.createWriter();
pos = 0;
}
//self.postMessage(writer.position);
writer.seek(pos);
writer.write(BlobBuilderObj.getBlob());
pos += 4096;
}
catch (e) {
errorHandler(e);
}
It looks like you are reading data from a WebSocket as a string, converting it to a Blob, and then writing this to a file.
If you have control of the WebSocket server then the best thing would be to send the data as binary frames instead of UTF-8 text data. If you can get the server to send the data as binary frames then you can just tell the WebSocket to deliver the data as Blobs:
ws.binaryType = "blob";
ws.onmessage = function (event) {
if (event.data instanceof Blob) {
// event.data is a Blob
} else {
// event.data is a string
}
}
If that is not an option and you can only send text frames from the server, then you will need to encode the binary data to text before sending it from the server and then decode the text on the other end. If you try and send binary data directly as text frames over WebSockets then doing charCodeAt(x) && 0xff will result in corrupt data.
For example you could base64 encode the data at the server and then base64 decode the data in the client:
ws.onmessage = function (event) {
raw = window.atob(event.data);
}
Update:
There is a very well performing pure Javascript base64 decode/encode contained in websockify. It decodes to an an array of numbers from 0-255 but could be easily modified to return a string instead if that is what you require (Disclaimer: I made websockify).
Related
How can the plain text response of an XMLHTTPRequest be converted to a dataUrl on client side?
Image data is being send from the server to an Iframe and the only option to retrieve the data is the default encoded data from a GET request.*
I do not have any control over the server. I can not specify overrideMimeType nor the responseType of the request.
I tried to utf8 encode the returned data:
const utf8 = new TextEncoder();
const bytes = utf8.encode(imageDataAsText);
//Convert to data url
const fr = new FileReader();
const blob = new Blob([bytes], { type: attachment.metadata.mediaType });
fr.onload = (e) => {
setImageData(fr.result as string);
};
fr.readAsDataURL(blob);
Converting via the char code didn't work either:
https://stackoverflow.com/a/6226756/3244464
let bytesv2 = []; // char codes
for (var i = 0; i < imageDataAsString.length; ++i) {
var code = imageDataAsString.charCodeAt(i);
bytesv2 = bytesv2.concat([code & 0xff, code / 256 >>> 0]);
}
Raw data as it is displayed by console out. What is the actual default encoding of the data I am working with here?
Context:
* The data in question is recieved inside an iframe inside the atlassian (jira/confuence) ecosystem. They do not support piping binary data from the parent frame to the iframe, nor can I make my own request due to the authorization flow which requires cookies stored on the parent page. All other options mention to override some encoding, or changing it on the server side do not apply in this specific case.
When you are using XMLHttpRequest to get binary data then don't return it as a string. Use xhr.responseType = 'blob' (or arrayBuffer if you intend to read/modify it)
var xhr = new XMLHttpRequest()
xhr.responseType = 'blob'
xhr.onload = () => {
// Convert xhr blob to data url
const fr = new FileReader()
fr.onload = () => {
setImageData(fr.result)
}
fr.readAsDataURL(xhr.response)
}
Better yet use the fetch api instead of the old XMLHttpRequest to get the binary
It's more popular among web workers and servers. It's also simpler and based on promises
fetch(url)
.then(res => res.blob())
.then(blob => { ... })
And why do you need it to be a base64 url? if it's just to show a preview of an <img> then it's a waste of time, cpu and memory.
It would be better just to do:
img.src = URL.createObjectURL(blob)
I have a PDF file which I want to read into memory using NodeJS. Ideally I'd like to encode it using base64 for transferring it. But somehow the read function does not seem to read the full PDF file, which makes no sense to me. The original PDF was generated using pdfKit, and is ok and viewable using a PDF reader program.
The original file test.pdf has 90kB on disk. But if I read and write it back to disk there are just 82kB and the new PDF test-out.pdf is not ok. The pdf viewer says:
Unable to open document. The pdf document is damaged.
The base64 encoding therefore also does not work correctly. I tested it using this webservice. Does someone know why and what is happening here? And how to resolve it.
I found this post already.
fs = require('fs');
let buf = fs.readFileSync('test.pdf'); // returns raw buffer binary data
// buf = fs.readFileSync('test.pdf', {encoding:'base64'}); // for the base64 encoded data
// ...transfer the base64 data...
fs.writeFileSync('test-out.pdf', buf); // should be pdf again
EDIT MCVE:
const fs = require('fs');
const PDFDocument = require('pdfkit');
let filepath = 'output.pdf';
class PDF {
constructor() {
this.doc = new PDFDocument();
this.setupdocument();
this.doc.pipe(fs.createWriteStream(filepath));
}
setupdocument() {
var pageNumber = 1;
this.doc.on('pageAdded', () => {
this.doc.text(++pageNumber, 0.5 * (this.doc.page.width - 100), 40, {width: 100, align: 'center'});
}
);
this.doc.moveDown();
// draw some headline text
this.doc.fontSize(25).text('Some Headline');
this.doc.fontSize(15).text('Generated: ' + new Date().toUTCString());
this.doc.moveDown();
this.doc.font('Times-Roman', 11);
}
report(object) {
this.doc.moveDown();
this.doc
.text(object.location+' '+object.table+' '+Date.now())
.font('Times-Roman', 11)
.moveDown()
.text(object.name)
.font('Times-Roman', 11);
this.doc.end();
let report = fs.readFileSync(filepath);
return report;
}
}
let pdf = new PDF();
let buf = pdf.report({location: 'athome', table:'wood', name:'Bob'});
fs.writeFileSync('outfile1.pdf', buf);
The encoding option for fs.readFileSync() is for you to tell the readFile function what encoding the file already is so the code reading the file knows how to interpret the data it reads. It does not convert it into that encoding.
In this case, your PDF is binary - it's not base64 so you are telling it to try to convert it from base64 into binary which causes it to mess up the data.
You should not be passing the encoding option at all and you will then get the RAW binary buffer (which is what a PDF file is - raw binary). If you then want to convert that to base64 for some reason, you can then do buf.toString('base64') on it. But, that is not its native format and if you write that converted data back out to disk, it won't be a legal PDF file.
To just read and write the same file out to a different filename, leave off the encoding option entirely:
const fs = require('fs');
let buf = fs.readFileSync('test.pdf'); // get raw buffer binary data
fs.writeFileSync('test-out.pdf', buf); // write out raw buffer binary data
After a lot of searching I found this Github issue. The problem in my question seems to be the call of doc.end() which for some reason doesn't wait for the stream to finish (finish event of write stream). Therefore as suggested in the Github issue, the following approaches work:
callback based:
doc = new PDFDocument();
writeStream = fs.createWriteStream('filename.pdf');
doc.pipe(writeStream);
doc.end()
writeStream.on('finish', function () {
// do stuff with the PDF file
});
or promise based:
const stream = fs.createWriteStream(localFilePath);
doc.pipe(stream);
.....
doc.end();
await new Promise<void>(resolve => {
stream.on("finish", function() {
resolve();
});
});
or even nicer, instead of calling doc.end() direcly, call the function savePdfToFile below:
function savePdfToFile(pdf : PDFKit.PDFDocument, fileName : string) : Promise<void> {
return new Promise<void>((resolve, reject) => {
// To determine when the PDF has finished being written sucessfully
// we need to confirm the following 2 conditions:
//
// 1. The write stream has been closed
// 2. PDFDocument.end() was called syncronously without an error being thrown
let pendingStepCount = 2;
const stepFinished = () => {
if (--pendingStepCount == 0) {
resolve();
}
};
const writeStream = fs.createWriteStream(fileName);
writeStream.on('close', stepFinished);
pdf.pipe(writeStream);
pdf.end();
stepFinished();
});
}
This function should correctly handle the following situations:
PDF generated successfully
Error is thrown inside pdf.end() before write stream is closed
Error is thrown inside pdf.end() after write stream has been closed
I have HTML form with 2 controls - file input control #resume and submit button #action_cv.
Most of files I can (and want) upload to server are binary.
There is reading and packing (encoding) this data at front side (Javascript):
function readFile(file, callback) {
let reader = new FileReader();
reader.onload = callback;
reader.readAsArrayBuffer(file);
}
$(document).ready(function () {
$('#action_cv').on('submit', function (event) {
let input_data = {
resume: null,
resume_data: null
};
let resume = null;
if (($("#resume"))[0].files.length > 0) {
resume = ($("#resume"))[0].files[0];
input_data['resume'] = resume.name;
readFile(resume, function (evt) {
let data = evt.target.result;
let bs =
String.fromCharCode.apply(null, new Uint8Array(data));
input_data['resume_data'] = bs;
});
}
// AJAX call with input_data skipped here...
})
});
Here data contains raw binary data from file, and bs - already packed for AJAX submission to server written by PHP. My question is very simple - how to unpack this encoded (packed) binary in PHP to get original file at server side? (No need to provide writing file operation - it seems evident)
I'm working on an app where I need to record audio using a microphone and send it to a backend app (tomcat server).
It seems that sending too big streams drives angular crazy and freezes my browser.
To record my audio file, I use the native function RecorderWorkerFactory.getUserMedia() which allow me to get a RecordBlob object.
After that, still in Angular, I extract the audio content in base64 enconding, and I send it to the backend app using $resource.
The backend app correctly receives the data and process it, but the callback of this call is never executed, as Firefox detects an infinite loop and freezes.
However, if I keep running the program, after a very long time the page refresh will pass.
This is the code where I extract the audio content into base64 String, to send it:
var blob = $scope.audio.recordBlob;
if (blob) {
var reader = new FileReader();
reader.readAsDataURL(blob);
reader.onloadend = function() {
$scope.audioContent = reader.result;
$scope.sendMessage();
}
}
$scope.sendMessage = function(){
var outputStream = {
"audio": $scope.audioContent
};
$scope.sendIM(outputStream);
}
Here I send outputStream via POST to the back, and in callback I launch loadData() function that reload my view.
services.FileCreation= $resource(URI_SERVICE_CREATION, {}, {
'post' : urlEncodedFormPost
});
$scope.sendIM = function(fluxSortie) {
$services.FileCreation.post(angular.toJson(outputStream)).$promise.then(function(result){
$scope.loadData();
});
}
And this is the Java code for the creation of the audio file:
private void createAudioFile(File file, byte[] content) throws IOException {
FileOutputStream stream = null;
try {
stream = new FileOutputStream(file.getPath());
IOUtils.write(content, stream);
} catch (IOException e) {
LOGGER.debug("creation failed");
} finally {
if (stream != null) {
stream.flush();
stream.close();
}
}
}
Where content is the conversion of the base64 string sent.
After research I found that the infinite loop is in a native Angular function named shallowClearAndCopy() that occured after the Java execution but just before the callback. In this function the code apparently transforms each character of the audio string (base64 encoded) into an object property and do a loop on these to delete them. But this lead to a very long treatment that Firefox consider as an infinite loop.
function shallowClearAndCopy(src, dst) {
dst = dst || {};
angular.forEach(dst, function(value, key) { // This is where it freezes, as dst contains all my base64 encoded data and iterate over each character of it (which is veeeeeery long !)
delete dst[key];
});
for (var key in src) {
if (src.hasOwnProperty(key) && !(key.charAt(0) === '$' && key.charAt(1) === '$')) {
dst[key] = src[key];
}
}
return dst;
}
Is it because of angularjs performance (and there is nothing else to be done) ?
Or am I missing something that creates an infinite loop ? Or is something wrong in my callback definition ?
Cheers !
I found, the problem!
It was the angular.toJson(outputStream) that transformed the object without need.
I'm getting gzipped data from a business api i'm working with, but I can't manage to decompress it into something readable in JS, though I managed with C#.
My question is - how do I unzip the received gzipped input to a string or json?
The following code works well for me in C#:
using (HttpWebResponse response = (HttpWebResponse)WebRequest.Create(url).GetResponse())
{
using (GZipStream decompress = new GZipStream(response.GetResponseStream(), CompressionMode.Decompress))
{
using (StreamReader reader = new StreamReader(decompress))
{
responseFromServer = reader.ReadToEnd();
}
}
}
I've read various answers and tried some libraries but still can't manage to decompress it in JS (using same URL).
This is where the code should be in JS:
var requestData = {
url: url,
headers: {
"Allow-Encoding": "gzip"
}
}
request.get(requestData, function(error, response, body) {
// compressed data is in body
});
I've tried pako, zlib but I am probably not using them correctly.
[edit]
Some of my tries:
// Decode base64 (convert ascii to binary)
var strData = new Buffer(body).toString('base64');
// Convert binary string to character-number array
var charData = strData.split('').map(function (x) { return x.charCodeAt(0); });
// Turn number array into byte-array
var binData = new Uint8Array(charData);
// Pako magic
var data = pako.inflate(binData);
// Convert gunzipped byteArray back to ascii string:
var strData2 = String.fromCharCode.apply(null, new Uint8Array(data));
This code is running in a NodeJS application, and i'm using request package
Thanks