I'm trying to download a pdf file from an s3 bucket, however for security reasons the file must be streamed back to the client (an angular web app). My plan was to request a download link from s3 and stream the document from that link back to the client using NestJS StreamableFile.
My Controller:
#Post('download')
#ApiOkResponse({
schema: {
type: 'string',
format: 'binary'
}
})
#ApiProduces('application/pdf')
async documentDownload(#Body() body: DownloadURLRequest) {
const result = await this.service.getS3DownloadURL(body);
const got = require('got');
return new StreamableFile(got.stream(result.downloadUrl));
}
My client receives a Blob that is exactly the length of the original file, however the file fails to load. Upon closer inspection it turns out some of (roughly 2/3) the bytes are different.
After a bit of debugging I know for sure the got.stream is generating the correct bytes, so it would appear that StreamableFile is changing or misinterpreting some of the bytes for some reason. Anyone with experience using StreamableFile know why this might be? Or if there is a better way to handle this with NestJS or otherwise I'm open to suggestions.
Edit: Further testing -- printing the stream from the StreamableFile object on the API side shows correct bytes, however the file returned to swagger has incorrect bytes.
Related
I am trying to send over HTTP a ZIP file, to achieve that I encode/decode it in Base64. That is not working unfortunately.
I have figured out the issue is actually in the encode/decode itself and was able to isolate and reproduce it.
Consider a simple code which:
Reads a file from the filesystem.
Base64 encodes that file.
Base64 decodes the previously computed base64 string into a binary stream and save it into another file (same identical as original).
const fs = require("fs");
const buffer = fs.readFileSync("C:/users/Public/myzip.zip"); // 1. read
const base64data = buffer.toString("base64"); // 2. encode
fs.writeFileSync("C:/users/Public/myzip2.zip",
new Buffer(base64data, "base64"),
"base64"); // 3. decode + save
The code runs fine (I am on Windows 10), no errors. It successfully reads and writes files. However, file myzip2.zip is written but it cannot be opened: Windows complains it is invalid :(
A bit more context
The reason for this question is the following. I am using Base64 encoding in order to successfully send over a ZIP file from a client to a server.
This code isolates the problem I am having by leaving the networking complexity out of the equation. I need to figure out how to properly encode/decode a file using Base64. Once I can make it work on a single machine, it will work when sending the file.
Why is this basic set of commands not working?
I have an issue similar to this one where I am successfully downloading a blob generated from a backend via HTTP GET but the file is being saved to browser memory before the download begins.
There's no problem when downloading small files but it doesn't immediately download 100mb+ files.
Subscribing to the GET itself is causing the delay of saving the large files.
I'm using Angular 6 with an object store backend. Here's the download function:
finalDownload(url: string) {
let headers = new HttpHeaders();
headers = headers.append('X-Auth-Token', token);
return this.http.get(url, { headers, responseType: 'blob' })
.subscribe(response => {
saveAs(response);
})
}
Here's the process:
User hits the download button
GET request with headers is fired to back end
As soon as I subscribe for the response, the blob is stored in browser memory.
When the blob is completely stored in browser, the saveAs/download begins
Step 3 is where the issue is.
This devtools screenshot with 108 MB transferred accumulates to the file size (I downloaded a 100 mb file) before the download itself to filesystem begins.
You can try to use URL.createObjectURL:
URL.createObjectURL() can be used to construct and parse URLs. URL.createObjectURL() specifically, can be used to create a reference to a File or a Blob. As opposed to a base64-encoded data URL, it doesn’t contain the actual data of the object – instead it holds a reference.
The nice thing about this is that it’s really fast. Previously, we’ve had to instantiate a FileReader instance and read the whole file as a base64 data URL, which takes time and a lot of memory. With createObjectURL(), the result is available straight away, allowing us to do things like reading image data to a canvas.
Use the following code as reference
const blob = new Blob([data], { type: 'application/octet-stream' });
this.fileUrl = this.sanitizer.bypassSecurityTrustResourceUrl(window.URL.createObjectURL(blob));
Using a FileStreamResult from C# in a SPA website (.NET Core 2, SPA React template), I request a file from my endpoint, which triggers this response in C#:
var file = await _docService.GetFileAsync(token.UserName, instCode.Trim()
.ToUpper(), fileSeqNo);
string contentType = MimeUtility.GetMimeMapping(file.FileName);
var result = new FileStreamResult(file.File, contentType);
var contentDisposition = new ContentDispositionHeaderValue("attachment");
Response.Headers[HeaderNames.ContentDisposition] =
contentDisposition.ToString();
return result;
The returned response is handled using msSaveBlob (spesificly for MS, but this is a problem even though I use createObjectURL and different browser (Yes, I have tried multiple solutions to this, but none of them seems to work). This is the code I use to send the request, and receive the PDF FileStreamResult from the server.
if (window.navigator.msSaveBlob) {
axios.get(url).then(response => {
window.navigator.msSaveOrOpenBlob(
new Blob([response.data], {type: "application/pdf"}),
filename);
});
The problem is that the returned PDF file that I get has a wrong encoding on it somehow. So the PDF will not open.
I have tried adding encoding to the end of type: {type: "application/pdf; encoding=UTF-8"} which was suggested in different posts, however, it makes no difference.
Comparing a PDF file that I have fetched in a different way, I can clearly see that the encoding is wrong. Most of the special characters are not correct. Indicated by the response header, the PDF file should be in UTF-8, but I have no idea how to actually find out and check.
Without knowing axios it seems though from its readme page that it uses JSON as default responseType. This may potentially alter the content as it is now treated as text (axios will probably bail out when it cannot convert to an actual JSON object and keep the string/text source for response data).
A PDF should be loaded as binary data even though it can be both, either 8-bit binary content or 7-bit ASCII - both should in any case be treated as a byte stream, from Adobe PDF reference sec. 2.2.1:
PDF files are represented as sequences of 8-bit binary bytes.
A PDF file is designed to be portable across all platforms and
operating systems. The binary rep resentation is intended to be
generated, transported, and consumed directly, without translation
between native character sets, end-of-line representations, or other
conventions used on various platforms. [...].
Any PDF file can also be represented in a form that uses only 7-bit
ASCII [...] character codes. This is useful for the purpose of
exposition, as in this book. However, this representation is not
recommended for actual use, since it is less efficient than the normal
binary representation. Regardless of which representation is
used, PDF files must be transported and stored as binary files,
not as text files. [...]
So to solve the conversion that happens I would suggest trying specifying the configuration entry responseType when doing the request:
axios.get(url, {responseType: "arraybuffer"}) ...
or in this form:
axios({
method: 'get',
url: url,
responseType:'arraybuffer'
})
.then( ... )
You can also go directly to response-type blob if you are sure the mime-type is preserved in the process.
Users customize an SVG, and when they're satisfied, I want to give them the option of sharing the image on Facebook.
I know Facebook doesn't accept SVGs, so, after reading this question, I came up with the following using canvg:
var svgText = "<svg>" + $("#canvas2").html() + "</svg>";
canvg(document.getElementById('drawingCanvas'), svgText);
var img = document.getElementById('drawingCanvas').toDataURL("image/png");
Despite the naming of .toDataURL, img is not then a URL (at least, I don't think it is), but a >10,000-character string that beings with data:image/png;base64,....
If I then try to allow the user to share the PNG on Facebook,
window.FB.ui({
href: "https://dna.land",
method: 'feed',
name: 'name',
link: "dna.land",
picture: img,
caption: 'mywebsite.com',
description: "description",
message: "message"
}, function(response) {});
the error message I get is "the parameter href" is required (which I'm clearly supplying... maybe img is too long and the JSON is being truncated?), though I have also seen that I'm not supplying a valid URL for picture.
How can I share the image data on Facebook?
If there's no way to do this with the raw, base 64 encoding of the PNG, I'm not averse to temporarily storing the PNG file on a server. But, ideally, I'd only store for 24 hours, or a few months, or some minor span of time, and then be able to delete it without the user's post disappearing. Is that possible?
Despite the naming of .toDataURL, img is not then a URL (at least, I don't think it is), but a >10,000-character string that beings with data:image/png;base64,....
This is what exactly Data URI means! and the function name toDataURL() is not misleading.
In your case what you need to do is that you need to convert the Base64 encoded image (represented by the Data URI) into a file and upload it somewhere (possibly on your own server or any Cloud Storage provider like Amazon S3 or Google Cloud Storage), then share that url on Facebook.
On your own server:
If you are uploading it to your own server then you can simply POST the Data URL via ajax, convert it to an image and return the URL of that image. To convert Base64 image you need to look for specific ways on how to do it with your backend language/system. For example, to convert Base64 encoded image to file in
PHP: look at this answer
Python: look at this answer
NodeJS: look at this answer
For any other backend just do a Google Search and you should find ways on how to do it.
Uploading to Third-party Cloud Storage services:
If you want to upload the image to any Third-party service, you can get a Blob object from the canvas by using Canvas.toBlob() function and upload the resulting blob to the provider (possibly by using a Javascript / REST API provided by them). AFAIK, many popular Cloud Storage providers will allow you to upload a blob with appropriate MIME type, though you should always check for support.
At work we are trying to upload files from a web page to a web service using html 5/javascript in the browser end and C# in the web service. But have some trouble with encoding of some sort.
As for the javascript we get the file's binary data with help from a FileReader.
var file = ... // gets the file from an input
var fileReader = new FileReader();
fileReader.onload = dataRecieved;
fileReader.readAsBinaryString(file);
function dataRecieved() {
// Here we do a normal jquery ajax post with the file data (fileReader.result).
}
Wy we are posting the data manually and not with help from XmlHttpRequest (or similar) is for easier overall posting to our web service from different parts of the web page (it's wrapped in a function). But that doesn't seem to be the problem.
The code in the Web Service looks like this
[WebMethod]
public string SaveFileValueFieldValue(string value)
{
System.Text.UnicodeEncoding encoder = new UnicodeEncoding();
byte[] bytes = encoder.GetBytes(value);
// Saves file from bytes here...
}
All works well, and the data seems to be normal, but when trying to open a file (an image as example) it cannot be opened. Very basic text files seems to turn out okay. But if I upload a "binary" file like an image and then open both the original and the uploaded version in a normal text editor as notepad to see what differs, it seems to be wrong with only a few "invisible" characters and something that displays as a new line a few bytes in from from the start.
So basicly, the file seems to encode just a few bytes wrong somewhere in the conversions.
I've also tried to create an int array in javascript from the data, and then again transformed to a byte[] in the web service, with the exact same problem. If I try to convert with anything else than unicode (like UTF-8), the data turns out completly different from the original, so I think om on the right track here, but with something slightly wrong.
The request itself is text, so binary data is lost if you send the wrong enc-type.
What you can do is encode the binary to base64 and decode it on the other side.
To change the enc-type to multi-part/mixed and set boundaries (just like an e-mail or something) you'd have to assemble the request yourself.