I am trying to download and save a large zip file. The zip file is possibly larger than the heap, so I want to use a stream to avoid java.lang.OutOfMemoryError: Java heap space error.
Also, the large zip file is generated on request, so I would like to delete the file after downloading it.
My current code is
#POST
#Path("/downloadLargeZip")
public Response largeZip() throws FileNotFoundException {
File file = generateZipFile(); // generates zip file successfully
FileInputStream input = new FileInputStream(file);
StreamingOutput so = os -> {
try {
int n;
byte[] buffer = new byte[1024];
while ((n = input.read(buffer)) >= 0) {
os.write(buffer, 0, n);
}
os.flush();
os.close();
} catch (Exception e) {
throw new WebApplicationException(e);
}
};
return Response.ok(so).build();
}
My current client-side code is
import { saveAs } from 'browser-filesaver/FileSaver.js';
save() {
this.http.post<any>('url', '', { observe: 'response', responseType: 'blob'})
.subscribe(res => {
this.downloadFile(res);
});
}
downloadFile(response: any) {
const contentDisposition = 'attachment; filename="KNOWN_FILE_NAME"'; // response.headers('content-disposition'); - response object has no headers
// Retrieve file name from content-disposition
let fileName = contentDisposition.substr(contentDisposition.indexOf('filename=') + 9);
fileName = fileName.replace(/\"/g, '');
const contentType = 'application/zip'; // response.headers('content-type');
const blob = new Blob([response.data], { type: contentType });
saveAs(blob, fileName);
}
I have a few problems with my code:
Using dev tools to check the response, it has no headers (normalizedNames is a map with no entries) or data.
Checking the saved zip file, I can't open it using WinRAR. The error is The archive is either in unknown format or damaged.
Trying to open the zip file with Notepad++, the content is the text undefined.
The JSON representation of the response is
{
"headers":{
"normalizedNames":{
},
"lazyUpdate":null
},
"status":200,
"statusText":"OK",
"url":"URL",
"ok":true,
"type":4,
"body":{
}
}
Although the body does contain data {size: 2501157, type: "application/json"}.
Please ignore the number (I am guessing it's the zip file size in bytes, the actual file will be much larger).
What am I doing wrong? How can I read the stream and save the generated zip file?
I think the issue is in my downloadFile function, but I don't know what to change there.
Any help would be appreciated.
I needed to completely change the way I approached the issue.
The server will now generate the file and return a URI for the client. The client will then download the file via given URI.
Server code
#POST
#Path("/create")
public Response createLogs(String data) {
String fileName = generateFileAndReturnName(data);
if (fileName != null) {
return Response.created(URI.create(manipulateUri(fileName))).build();
}
return Response.status(500).build();
}
Client code
save() {
this.http.post<any>(this.baseUrl + '/create', this.data, { observe: 'response'}).subscribe(postResponse => {
if (postResponse.status !== 201) {
this.logger.error('Failed.');
return;
}
postResponse.headers.keys(); // lazy init headers
const uri = postResponse.headers.get('location');
if (!uri) {
this.logger.error('URI not present.');
return;
}
const link = document.createElement('a');
link.href = uri;
link.setAttribute('download', 'fileName.fileExtension');
document.body.appendChild(link);
link.click();
if (link.parentNode) {
link.parentNode.removeChild(link);
}
});
}
Working fine now with 40GB file (32GB RAM, so file is definitely bigger than any allocated heap).
Related
I'm trying to zip varying amounts of files so that one zip folder can be served to the user instead of them having to click multiple anchor tags. I am using the System.IO.Compression namespace in asp.net core 3.1 to create the zip folder.
Here is the code I'm using to create the Zip folder.
public IActionResult DownloadPartFiles(string[] fileLocations, string[] fileNames)
{
List<InMemoryFile> files = new List<InMemoryFile>();
for (int i = 0; i < fileNames.Length; i++)
{
InMemoryFile inMemoryFile = GetInMemoryFile(fileLocations[i], fileNames[i]).Result;
files.Add(inMemoryFile);
}
byte[] archiveFile;
using (MemoryStream archiveStream = new MemoryStream())
{
using (ZipArchive archive = new ZipArchive(archiveStream, ZipArchiveMode.Create, true))
{
foreach (InMemoryFile file in files)
{
ZipArchiveEntry zipArchiveEntry = archive.CreateEntry(file.FileName, CompressionLevel.Fastest);
using (Stream zipStream = zipArchiveEntry.Open())
{
zipStream.Write(file.Content, 0, file.Content.Length);
zipStream.Close();
}
}
archiveStream.Position = 0;
}
archiveFile = archiveStream.ToArray();
}
return File(archiveFile, "application/octet-stream");
}
The files I am trying to zip are stored remotely so I grab them with this block of code. The InMemoryFile is a class to group the file name and file bytes together.
private async Task<InMemoryFile> GetInMemoryFile(string fileLocation, string fileName)
{
InMemoryFile file;
using (HttpClient client = new HttpClient())
using (HttpResponseMessage response = await client.GetAsync(fileLocation))
{
byte[] fileContent = await response.Content.ReadAsByteArrayAsync();
file = new InMemoryFile(fileName, fileContent);
}
return file;
}
The DownloadPartFiles method is called using Ajax. I grab the remote paths to the files and their respective names using javascript and pass them into the Ajax call.
function downloadAllFiles() {
let partTable = document.getElementById("partTable");
let linkElements = partTable.getElementsByTagName('a');
let urls = [];
for (let i = 0; i < linkElements.length; i++) {
urls.push(linkElements[i].href);
}
if (urls.length != 0) {
var fileNames = [];
for (let i = 0; i < linkElements.length; i++) {
fileNames.push(linkElements[i].innerText);
}
$.ajax({
type: "POST",
url: "/WebOrder/DownloadPartFiles/",
data: { 'fileLocations': urls, 'fileNames': fileNames },
success: function (response) {
var blob = new Blob([response], { type: "application/zip" });
var link = document.createElement('a');
link.href = window.URL.createObjectURL(blob);
link.download = "PartFiles.zip";
link.click();
window.URL.revokeObjectURL(blob);
},
failure: function (response) {
alert(response.responseText);
},
error: function (response) {
alert(response.responseText);
}
});
}
}
Now the issue I keep running into is that I can't open the zip folder within Windows 10. Every time I try to open the zip folder using Windows or 7-zip I get an error message that the folder can't be opened or the folder is invalid. I've tried looking at various similar issues on stackoverflow, ie Invalid zip file after creating it with System.IO.Compression, but still can't figure out why this is.
Could it be the encoding? I found that Ajax expects its responses to be encoded UTF-8 and when I view the zip file using notepad++ with UTF-8 I see that there are � characters indicating corruption.
Any thoughts on this would be helpful. Let me know if more information is needed.
If one of the corrupt zip files is needed I can provide that as well.
Edit:
I have since changed my method of receiving the byte array in javascript. I am using a XMLHttpRequest to receive the byte array.
var parameters = {};
parameters.FileLocations = urls;
parameters.FileNames = fileNames;
var xmlhttp = new XMLHttpRequest();
xmlhttp.open("POST", "/WebOrder/DownloadPartFiles/", true);
xmlhttp.setRequestHeader("Content-Type", "application/json");
xmlhttp.responseType = "arraybuffer";
xmlhttp.onload = function (oEvent) {
var arrayBuffer = xmlhttp.response;
if (arrayBuffer) {
var byteArray = new Uint8Array(arrayBuffer);
var blob = new Blob([byteArray], { type: "application/zip" });
var link = document.createElement('a');
link.href = window.URL.createObjectURL(blob);
link.download = "PartFiles.zip";
link.click();
window.URL.revokeObjectURL(blob);
}
}
xmlhttp.send(JSON.stringify(parameters));
From what I read, Ajax is not the best for receiving byte arrays and binary data. With this method I was able to open one of the zip file with 7-zip, but not Windows, however, one of the files within the archive was showing as a size of 0KB and couldn't be opened. The other three files in the archive were fine. Other zip folders with different files could not be opened at all though.
After some time I found a post that was able to fix my issue, Create zip file from byte[]
From that post this is the revised method I'm using to create a zip folder with files in it.
public IActionResult DownloadPartFiles([FromBody] FileRequestParameters parameters)
{
List<InMemoryFile> files = new List<InMemoryFile>();
for (int i = 0; i < parameters.FileNames.Length; i++)
{
InMemoryFile inMemoryFile = GetInMemoryFile(parameters.FileLocations[i], parameters.FileNames[i]).Result;
files.Add(inMemoryFile);
}
byte[] archiveFile = null;
using (MemoryStream archiveStream = new MemoryStream())
{
using (ZipArchive archive = new ZipArchive(archiveStream, ZipArchiveMode.Create, true))
{
foreach (InMemoryFile file in files)
{
ZipArchiveEntry zipArchiveEntry = archive.CreateEntry(file.FileName, CompressionLevel.Optimal);
using (MemoryStream originalFileStream = new MemoryStream(file.Content))
using (Stream zipStream = zipArchiveEntry.Open())
{
originalFileStream.CopyTo(zipStream);
}
}
}
archiveFile = archiveStream.ToArray();
}
return File(archiveFile, "application/octet-stream");
}
I still don't know why the previous method was having issues so if anyone knows the answer to that in the future I'd love to know.
I created a service to download a PDF file.
On my server-side(Java) the PDF is generated successfully. But I am unable to download that on the UI side (Using Jquery Ajax call).
Could anyone please help me with this?
$(document).on('click', '.orderView', function(event){
orderId = $(this).attr('data');
$.ajax({
type : 'GET',
contentType : 'application/json',
url : '../service/purchase/generateInventoryPurchasePdf/'+orderId,
success : function(response) {
console.log("Success");
},
error : function(response) {
console.log("Error :" + response);
}
});
});
Java Code:
#RequestMapping(value = "/generateInventoryPurchasePdf/{purchaseId}", method = RequestMethod.GET)
public ResponseEntity<ByteArrayResource> generateInventoryPurchasePdf(HttpServletResponse response,#PathVariable("purchaseId") Long purchaseId) throws Exception {
PurchaseOrder purchaseOrder = null;
purchaseOrder = purchaseService.findByPurchaseOrderId(purchaseId);
// generate the PDF
Map<Object,Object> pdfMap = new HashMap<>();
pdfMap.put("purchaseOrder", purchaseOrder);
pdfMap.put("purchaseOrderDetail", purchaseOrder.getPurchaseOrderDetail());
pdfMap.put("vendorName", purchaseOrder.getInvVendor().getName());
pdfMap.put("vendorAddrs", purchaseOrder.getInvVendor().getVenAddress().get(0));
File file = util.generatePdf("email/purchasepdf", pdfMap);
MediaType mediaType = MediaTypeUtils.getMediaTypeForFileName(this.servletContext, file.getName());
System.out.println("fileName: " + file.getName());
System.out.println("mediaType: " + mediaType);
//Path path = Paths.get(file.getAbsolutePath() + "/" + file.getName());
Path path = Paths.get(file.getAbsolutePath());
byte[] data = Files.readAllBytes(path);
ByteArrayResource resource = new ByteArrayResource(data);
return ResponseEntity.ok()
// Content-Disposition
.header(HttpHeaders.CONTENT_DISPOSITION, "attachment;filename=" + path.getFileName().toString())
// Content-Type
.contentType(mediaType) //
// Content-Lengh
.contentLength(data.length) //
.body(resource);
}
mediaUtil class:
public class MediaTypeUtils {
public static MediaType getMediaTypeForFileName(ServletContext servletContext, String fileName) {
// application/pdf
// application/xml
// image/gif, ...
String mineType = servletContext.getMimeType(fileName);
try {
MediaType mediaType = MediaType.parseMediaType(mineType);
return mediaType;
} catch (Exception e) {
return MediaType.APPLICATION_OCTET_STREAM;
}
}
}
PDF Generation code:
public File generatePdf(String templateName, Map<Object, Object> map) throws Exception {
Assert.notNull(templateName, "The templateName can not be null");
Context ctx = new Context();
if (map != null) {
Iterator<Entry<Object, Object>> itMap = map.entrySet().iterator();
while (itMap.hasNext()) {
Map.Entry<Object, Object> pair = itMap.next();
ctx.setVariable(pair.getKey().toString(), pair.getValue());
}
}
String processedHtml = templateEngine.process(templateName, ctx);
FileOutputStream os = null;
String fileName = "POLIST";
try {
final File outputFile = File.createTempFile(fileName, ".pdf",new File(servletContext.getRealPath("/")));
outputFile.mkdir();
os = new FileOutputStream(outputFile);
ITextRenderer renderer = new ITextRenderer();
renderer.setDocumentFromString(processedHtml);
renderer.layout();
renderer.createPDF(os, false);
renderer.finishPDF();
System.out.println("PDF created successfully");
return outputFile;
} finally {
if (os != null) {
try {
os.close();
} catch (IOException e) {
}
}
}
}
I'm not getting any error, PDF generate successfully in the server side. But In UI side not working.
Downloading files via AJAX isn't really a logical thing to do. When you make an AJAX call, the data returned from the server is returned into your page's JavaScript code (in the response callback value), rather than being returned to the browser itself to decide what to do. Therefore the browser has no way to initiate a download, because the browser is not directly in control of the response - your JavaScript code is in control instead.
As you've indicated in your comment below the question, there are workarounds you can use, but really the best approach is simply to use a regular non-AJAX request to download
For instance you could replace your jQuery code with something like
$(document).on('click', '.orderView', function(event){
orderId = $(this).attr('data');
window.open('../service/purchase/generateInventoryPurchasePdf/'+orderId);
});
This will download the document from a new tab without navigating away from the current page.
I want to save a PNG image received from Backend (Java Project) to a folder inside my Angular Project. So far I can only save the image under Downloads/ folder of the PC and you can see how the file is downloaded. What I want is to silently download the image in my project (when I check the folder to see the new image stored).
Backend:
#GET
#Produces(MediaType.TEXT_PLAIN)
#Path("getImage")
public Response getImage() {
File dir = new File(Utilities.IMAGE_DIRECTORY);
File[] directoryListing = dir.listFiles();
String encodedImages = null;
// Get the first image stored in Backend project folder
try {
if (directoryListing != null) {
// Encode the image in Base64 and save it in a string
encodedImages = Base64
.getEncoder()
.withoutPadding()
.encodeToString(
Files.readAllBytes(directoryListing[0].toPath()));
}
} catch (IOException e) {
e.printStackTrace();
...
}
// Send the base64 string to Frontend
return Response
.status(Response.Status.OK)
.entity(encodedImages)
.build();
}
Frontend:
/* Extract Image */
getImage() {
this
.http
.get(this.baseUrl + "getImage", { responseType:
'text' })
.subscribe((res) => {
console.log("I received the image: \n" + res);
// Decode from base64 to PNG
var decodedImage = atob(res);
var blob = new Blob([decodedImage], { type: 'image/png' });
//this method saves the image in Downloads/ and it is not silent
saveAs(blob, 'imageFileName.png');
});
}
Since your user will not need the image it self why to download it? You don't need to download the image just for using it.
Only if your user need to download the image you could go with and "Save File" system.
I'm using ASP.NET Core and React JS. I'm newbie to this both platforms. I have used Axios for requesting data and getting response from server. But I have not requested images or any kind of file from server. This time I'm working on Download feature where user will click on button and can download desired file which is of .png, .jpg, .pdf format. I'm not understanding how can server will send data? I read, I needed to send base64 data which is converted from blob format. But not understanding how to request data from client and how server will serve desired file. In DB, I have stored only address of file e.g. /images/img1.jpg. This file actually resides in wwwroot/images folder. I have used downloadjs for downloading which is working correctly but after downloading, that image is not readable as it does not have any content.
Please anyone help me to implement this feature.
First you need API to download data something like this
public async Task<IActionResult> Download(string filename)
{
if (filename == null)
return Content("filename not present");
var path = Path.Combine(
Directory.GetCurrentDirectory(),
"wwwroot", filename);
var memory = new MemoryStream();
using (var stream = new FileStream(path, FileMode.Open))
{
await stream.CopyToAsync(memory);
}
memory.Position = 0;
return File(memory, GetContentType(path), Path.GetFileName(path));
}
private string GetContentType(string path)
{
var types = GetMimeTypes();
var ext = Path.GetExtension(path).ToLowerInvariant();
return types[ext];
}
private Dictionary<string, string> GetMimeTypes()
{
return new Dictionary<string, string>
{
{".txt", "text/plain"},
{".pdf", "application/pdf"},
{".doc", "application/vnd.ms-word"},
{".docx", "application/vnd.ms-word"},
{".xls", "application/vnd.ms-excel"},
{".xlsx", "application/vnd.openxmlformats
officedocument.spreadsheetml.sheet"},
{".png", "image/png"},
{".jpg", "image/jpeg"},
{".jpeg", "image/jpeg"},
{".gif", "image/gif"},
{".csv", "text/csv"}
};
}
Then download file like this
axios({
url: 'your url',
method: 'POST', // Worked using POST or PUT. Prefer POST
responseType: 'blob', // important
}).then((response) => {
const url = window.URL.createObjectURL(new Blob([response.data]));
const link = document.createElement('a');
link.href = url;
link.setAttribute('download', 'file.pdf');
document.body.appendChild(link);
link.click();
});
Ref link
I have this working but I want to share this out to see if I missed anything obvious and to solve a mystery as to why my file chunk size has to be a multiple of 2049. The main requirements are:
Files uploaded from website must be stored in SQL server, not as files
Website must be able to download and display file data as a file (opened in a separate window.
Website is angularjs/javascript SPA, no server side code, no MVC
API is Web API 2 (again not MVC)
I'm just going to focus on the download part here. Basically what I'm doing is:
Read a chunk of data from SQL server varbinary field
Web API 2 api returns file name, mime type and byte data as a base64 string. NOTE - tried returning byte array but Web API just serializes it into base64 string anyway.
concatenate the chunks, convert the chunks to a blob and display
VB library function that returns a dataset with the chunk (I have to use this library which handles the database connection but doesn't support parameter queries)
Public Function GetWebApplicationAttachment(ByVal intId As Integer, ByVal intChunkNumber As Integer, ByVal intChunkSize As Integer) As DataSet
' the starting number is NOT 0 based
Dim intStart As Integer = 1
If intChunkNumber > 1 Then intStart = ((intChunkNumber - 1) * intChunkSize) + 1
Dim strQuery As String = ""
strQuery += "SELECT FileName, "
strQuery += "SUBSTRING(ByteData," & intStart.ToString & "," & intChunkSize.ToString & ") AS ByteData "
strQuery += "FROM FileAttachments WHERE Id = " + intId.ToString + " "
Try
Return Query(strQuery)
Catch ex As Exception
...
End Try
End Function
Web API business rules bit that creates the file object from the dataset
...
result.FileName = ds.Tables[0].Rows[0]["FileName"].ToString();
// NOTE: Web API converts a byte array to base 64 string so the result is the same either way
// the result of this is that the returned data will be about 30% bigger than the chunk size requested
result.StringData = Convert.ToBase64String((byte[])ds.Tables[0].Rows[0]["ByteData"]);
//result.ByteData = (byte[])ds.Tables[0].Rows[0]["ByteData"];
... some code to get the mime type
result.MIMEType = ...
Web API controller (simplified - all security and error handling removed)
public IHttpActionResult GetFileAttachment([FromUri] int id, int chunkSize, int chunkNumber) {
brs = new Files(...);
fileResult file = brs.GetFileAttachment(appID, chunkNumber, chunkSize);
return Ok(file);
}
angularjs Service that gets the chunks recurively and puts them together
function getFileAttachment2(id, chunkSize, chunkNumber, def, fileData, mimeType) {
var deferred = def || $q.defer();
$http.get(webServicesPath + "api/files/get-file-attachment?id=" + id + "&chunkSize=" + chunkSize + "&chunkNumber=" + chunkNumber).then(
function (response) {
// when completed string data will be empty
if (response.data.StringData === "") {
response.data.MIMEType = mimeType;
response.data.StringData = fileData;
deferred.resolve(response.data);
} else {
if (chunkNumber === 1) {
// only the first chunk computes the mime type
mimeType = response.data.MIMEType;
}
fileData += response.data.StringData;
chunkNumber += 1;
getFileAttachment2(appID, detailID, orgID, GUID, type, chunkSize, chunkNumber, deferred, fileData, mimeType);
}
},
function (response) {
... error stuff
}
);
return deferred.promise;
}
angular controller method that makes the calls.
function viewFile(id) {
sharedInfo.getWebPortalSetting("FileChunkSize").then(function (result) {
// chunk size must be a multiple of 2049 ???
var chunkSize = 0;
if (result !== null) chunkSize = parseInt(result);
fileHelper.getFileAttachment2(id, chunkSize, 1, null, "", "").then(function (result) {
if (result.error === null) {
if (!fileHelper.viewAsFile(result.StringData, result.FileName, result.MIMEType)) {
... error
}
result = {};
} else {
... error;
}
});
});
}
And finally the bit of javascript that displays the file as a download
function viewAsFile(fileData, fileName, fileType) {
try {
fileData = window.atob(fileData);
var ab = new ArrayBuffer(fileData.length);
var ia = new Uint8Array(ab); // ia provides window into array buffer
for (var i = 0; i < fileData.length; i++) {
ia[i] = fileData.charCodeAt(i);
}
var file = new Blob([ab], { type: fileType });
fileData = "";
if (window.navigator.msSaveOrOpenBlob) // IE10+
window.navigator.msSaveOrOpenBlob(file, fileName);
else { // Others
var a = document.createElement("a"),
url = URL.createObjectURL(file);
a.href = url;
a.download = fileName;
document.body.appendChild(a);
a.click();
setTimeout(function () {
document.body.removeChild(a);
window.URL.revokeObjectURL(url);
}, 0);
}
return true;
} catch (e) {
... error stuff
}
}
I see already that a more RESTful approach would be to use headers to indicate chunk range and to separate the file meta data from the file chunks. Also I could try returning a data stream rather than Base64 encoded string. If anyone has tips on that let me know.
Well that was entirely the wrong way to go about that. In case it helps here's what I ended up doing.
Dynamically create the href address of an anchor tag to return a file (security token and parameters in query string)
get byte array from database
web api call return response message (see code below)
This is much faster and more reliable, but provides less in the way of progress monitoring.
business rule method uses...
...
file.ByteData = (byte[])ds.Tables[0].Rows[0]["ByteData"];
...
web api controller
public HttpResponseMessage ViewFileAttachment([FromUri] int id, string token) {
HttpResponseMessage response = new HttpResponseMessage();
... security stuff
fileInfoClass file = ... code to get file info
response.Content = new ByteArrayContent(file.ByteData);
response.Content.Headers.ContentDisposition =
new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment") {
FileName = file.FileName
};
response.Content.Headers.ContentType = new MediaTypeHeaderValue(file.MIMEType);
return response;
This could even be improved with streaming