Develop Reference

Downloading files from Azure Blob Storage using Azure Function returns different file size

We are trying to serve files hosted in Azure blob storage using an Azure Function. When fetching them we get the following error:
Uncaught (in promise) Error: Length in header does not match actual data length: X != Y
I'm not very well-versed in these kinds of things but our devs are wracking their brains and I'm coming up empty as well.
The snippet in our Azure Function that we believe is the culprit is this:
const streamToBuffer = async (readableStream) => {
return new Promise((resolve, reject) => {
const chunks = [];
readableStream.on('data', (data) => {
chunks.push(data instanceof Buffer ? data : Buffer.from(data));
});
readableStream.on('end', () => {
resolve(Buffer.concat(chunks));
});
readableStream.on('error', reject);
});
};
EDIT:
const blobClient = containerClient.getBlobClient(path);
if (await blobClient.exists()) {
const downloadBlockBlobResponse = await blobClient.download();
const content = (
await streamToBuffer(downloadBlockBlobResponse.readableStreamBody)
).toString();
EDIT2:
Errors in the trace:
Uncaught (in promise) Error: Can not parse environment file at god3.js:16
Uncaught (in promise) Error: Length in header does not match actual data length: X != Y
Error while trying to use the following icon from the Manifest: $url (Download error or resource isn't a valid image)
We are only encountering this issue while serving the data through an Azure Function. We have tried hosting the logic on a different platform and it works when pulling the files.
Additional part of the function:
const containerClient = blobServiceClient.getContainerClient('site');
const blobClient = containerClient.getBlobClient(path);
if (await blobClient.exists()) {
const downloadBlockBlobResponse = await blobClient.download();
const content = (
await streamToBuffer(downloadBlockBlobResponse.readableStreamBody)
).toString();
context.res = {
headers: {
'Content-Type': downloadBlockBlobResponse.contentType,
},
body: content,
};
} else {
context.res = {
status: 404,
};
}
} else {
context.res = {
status: 401,
headers: { 'WWW-Authenticate': 'Basic realm=Access to the staging site' },
body: { success: false },
};
}
};
Any pointers would be very appreciated.
Thank you

We found the problem.
We were stringifying binaries, which BabylonJS didn't appreciate.
The solution was to add a filetype check in the function.
const content = await streamToBuffer(
downloadBlockBlobResponse.readableStreamBody,
);
const fileType = path.split('.').slice(-1)[0];
context.res = {
headers: {
'Content-Type': downloadBlockBlobResponse.contentType,
},
body: textPlainType.has(fileType) ? content : content.toString(),
};

how to send zip file as body in a post request in react?

Let's say I have a zip file uploaded by user, how do I pass it to the server for process?
I have front end function to handle zip file. I can verify that fileUploaded holds the zip file.
const onImport = async evt => {
const fileUploaded = evt.target.files[0];
const response = await extractImport('/extractimport', fileUploaded);
};
Helper function used above
export const extractImport = async (url, fileUploaded) => {
return await fetch(url, {
method: 'POST',
headers: {'accept': 'application/zip'},
body: fileUploaded
});
};
router for post call.
And here, req body is empty. I don't know what went wrong here.
router.post('/extractimport',
asyncWrapper(async (req, res) => {
//req.body is empty here, I dont understand why.
try {
const result = await extractImportPost(req.body);
res.status(200).json(result).end();
} catch (err) {
res.status(500).json({errors: [{error: err.message}]}).end();
}
})
);
extractImportPost utility used above is as below
const extractImportPost= async (zipFile) => {
// zipFile is undefined here.
// how do I pass the zip file to this place for process
}
Thank you for your time.

File Streams Wtih Axios

I need to use Axios to download files from a stream. I don't think it is a problem with the server because it works if i use the http package. This is the code i have:
export function downloadRequest (savePath, reqURL, currentFile, serverPath) {
const file = fs.createWriteStream(savePath[0] + `/${currentFile}`)
axios({
method: 'get',
url: `${storedIP}${reqURL}`,
responseType: 'stream',
headers: {
Authorization: `Bearer ${token}`
}
}).then(function (reponse) {
reponse.data.pipe(file)
hashCheck(currentFile, savePath, serverPath)
})
however this returns these two errors:
the provided value 'stream' is not a valid enum value of type XMLHttpRequestResponseType.
Uncaught (in promise) TypeError: reponse.data.pipe is not a function
I tried doing what is said in this post: Type error this.httpClient.get(...).pipe is not a function
But that made a windows popup saying something like this:
windows script host
Script: c:\npm\node_modules\webpack\bin\webpack.js
Line: 1
Char: 1
Error: Invalid character
Code: 800A03F6
Source: Microsoft JScript compilation error
I went into that js file and Line 1 is #!/usr/bin/env node
I also tried uninstalling webpack and reinstalling but that didn't help.
Note: I am using Axios with quasar if that matters.

If you're using electron, you should set the adapter to http instead of XMLHttpRequest
axios.defaults.adapter = require('axios/lib/adapters/http');
If you're getting response.data.pipe is not a function it appears that you can only get response.data as a stream in the main thread:
https://github.com/axios/axios/issues/1474#issuecomment-380594110
ipcRenderer.send("downloadRequest"); // when downloadRequest is called
Main thread
app.on("ready", async () => {
// other electron setup
ipcMain.on("downloadRequest", (event, arg) => {
// axios download code here
});
});

In electron version 9.0.4, no need of below line
axios.defaults.adapter = require('axios/lib/adapters/http');
In Renderers
ipcRenderer.send("download_file");
In your Main use below code
ipcMain.on("download_file", (event, args) => {
var url = "YourFileUrl";
// console.log(data)
require("axios").get(url, { responseType: 'stream' })
.then(res => {
var fileName = res.headers['content-disposition'].split("=")[1].split("\"")[0];
const writer = require('fs').createWriteStream("Location to save the file");
new Promise((resolve, reject) => {
res.data.pipe(writer);
let error = null;
writer.on('error', err => {
error = err;
writer.close();
reject(err);
});
writer.on('close', () => {
if (!error) {
resolve(true);
}
//no need to call the reject here, as it will have been called in the
//'error' stream;
});
});
}).catch((error) => {
console.log(error)
});
}
);
Reference :
node.js axios download file and writeFile

node.js axios download file stream and writeFile

i want download a pdf file with axios and save on disk (server side) with fs.writeFile, i have tried:
axios.get('https://xxx/my.pdf', {responseType: 'blob'}).then(response => {
fs.writeFile('/temp/my.pdf', response.data, (err) => {
if (err) throw err;
console.log('The file has been saved!');
});
});
the file is saved but the content is broken...
how do I correctly save the file?

Actually, I believe the previously accepted answer has some flaws, as it will not handle the writestream properly, so if you call "then()" after Axios has given you the response, you will end up having a partially downloaded file.
This is a more appropriate solution when downloading slightly larger files:
export async function downloadFile(fileUrl: string, outputLocationPath: string) {
const writer = createWriteStream(outputLocationPath);
return Axios({
method: 'get',
url: fileUrl,
responseType: 'stream',
}).then(response => {
//ensure that the user can call `then()` only when the file has
//been downloaded entirely.
return new Promise((resolve, reject) => {
response.data.pipe(writer);
let error = null;
writer.on('error', err => {
error = err;
writer.close();
reject(err);
});
writer.on('close', () => {
if (!error) {
resolve(true);
}
//no need to call the reject here, as it will have been called in the
//'error' stream;
});
});
});
}
This way, you can call downloadFile(), call then() on the returned promise, and making sure that the downloaded file will have completed processing.
Or, if you use a more modern version of NodeJS, you can try this instead:
import * as stream from 'stream';
import { promisify } from 'util';
const finished = promisify(stream.finished);
export async function downloadFile(fileUrl: string, outputLocationPath: string): Promise<any> {
const writer = createWriteStream(outputLocationPath);
return Axios({
method: 'get',
url: fileUrl,
responseType: 'stream',
}).then(response => {
response.data.pipe(writer);
return finished(writer); //this is a Promise
});
}

You can simply use response.data.pipe and fs.createWriteStream to pipe response to file
axios({
method: "get",
url: "https://xxx/my.pdf",
responseType: "stream"
}).then(function (response) {
response.data.pipe(fs.createWriteStream("/temp/my.pdf"));
});

The problem with broken file is because of backpressuring in node streams. You may find this link useful to read: https://nodejs.org/es/docs/guides/backpressuring-in-streams/
I'm not really a fan of using Promise base declarative objects in JS codes as I feel it pollutes the actual core logic & makes the code hard to read. On top of it, you have to provision event handlers & listeners to make sure the code is completed.
A more cleaner approach on the same logic which the accepted answer proposes is given below. It uses the concepts of stream pipelines.
const util = require('util');
const stream = require('stream');
const pipeline = util.promisify(stream.pipeline);
const downloadFile = async () => {
try {
const request = await axios.get('https://xxx/my.pdf', {
responseType: 'stream',
});
await pipeline(request.data, fs.createWriteStream('/temp/my.pdf'));
console.log('download pdf pipeline successful');
} catch (error) {
console.error('download pdf pipeline failed', error);
}
}
exports.downloadFile = downloadFile
I hope you find this useful.

// This works perfectly well!
const axios = require('axios');
axios.get('http://www.sclance.com/pngs/png-file-download/png_file_download_1057991.png', {responseType: "stream"} )
.then(response => {
// Saving file to working directory
response.data.pipe(fs.createWriteStream("todays_picture.png"));
})
.catch(error => {
console.log(error);
});

The following code taken from https://gist.github.com/senthilmpro/072f5e69bdef4baffc8442c7e696f4eb?permalink_comment_id=3620639#gistcomment-3620639 worked for me
const res = await axios.get(url, { responseType: 'arraybuffer' });
fs.writeFileSync(downloadDestination, res.data);

node fileSystem writeFile encodes data by default to UTF8. which could be a problem in your case.
Try setting your encoding to null and skip encoding the received data:
fs.writeFile('/temp/my.pdf', response.data, {encoding: null}, (err) => {...}
you can also decalre encoding as a string (instead of options object) if you only declare encoding and no other options. string will be handled as encoding value. as such:
fs.writeFile('/temp/my.pdf', response.data, 'null', (err) => {...}
more read in fileSystem API write_file

I have tried, and I'm sure that using response.data.pipe and fs.createWriteStream can work.
Besides, I want to add my situation and solution
Situation:
using koa to develop a node.js server
using axios to get a pdf via url
using pdf-parse to parse the pdf
extract some information of pdf and return it as json to browser
Solution:
const Koa = require('koa');
const app = new Koa();
const axios = require('axios')
const fs = require("fs")
const pdf = require('pdf-parse');
const utils = require('./utils')
app.listen(process.env.PORT || 3000)
app.use(async (ctx, next) => {
let url = 'https://path/name.pdf'
let resp = await axios({
url: encodeURI(url),
responseType: 'arraybuffer'
})
let data = await pdf(resp.data)
ctx.body = {
phone: utils.getPhone(data.text),
email: utils.getEmail(data.text),
}
})
In this solution, it doesn't need to write file and read file, it's more efficient.

This is what worked for me and it also creates a temporary file for the image file in case the output file path is not specified:
const fs = require('fs')
const axios = require('axios').default
const tmp = require('tmp');
const downloadFile = async (fileUrl, outputLocationPath) => {
if(!outputLocationPath) {
outputLocationPath = tmp.fileSync({ mode: 0o644, prefix: 'kuzzle-listener-', postfix: '.jpg' });
}
let path = typeof outputLocationPath === 'object' ? outputLocationPath.name : outputLocationPath
const writer = fs.createWriteStream(path)
const response = await axios.get(fileUrl, { responseType: 'arraybuffer' })
return new Promise((resolve, reject) => {
if(response.data instanceof Buffer) {
writer.write(response.data)
resolve(outputLocationPath.name)
} else {
response.data.pipe(writer)
let error = null
writer.on('error', err => {
error = err
writer.close()
reject(err)
})
writer.on('close', () => {
if (!error) {
resolve(outputLocationPath.name)
}
})
}
})
}
Here is a very simple Jest test:
it('when downloadFile should downloaded', () => {
downloadFile('https://i.ytimg.com/vi/HhpbzPMCKDc/hq720.jpg').then((file) => {
console.log('file', file)
expect(file).toBeTruthy()
expect(file.length).toBeGreaterThan(10)
})
})

Lorenzo's response is probably the best answer as it's using the axios built-in.
Here's a simple way to do it if you just want the buffer:
const downloadFile = url => axios({ url, responseType: 'stream' })
.then(({ data }) => {
const buff = []
data.on('data', chunk => buff.push(chunk))
return new Promise((resolve, reject) => {
data.on('error', reject)
data.on('close', () => resolve(Buffer.concat(buff)))
})
})
// better
const downloadFile = url => axios({ url, responseType: 'arraybuffer' }).then(res => res.data)
const res = await downloadFile(url)
fs.writeFileSync(downloadDestination, res)
I'd still probably use the 'arraybuffer' responseType

There is a much simpler way that can be accomplished in a couple of lines:
const fileResponse = await axios({
url: fileUrl,
method: "GET",
responseType: "stream",
});
// Write file to disk (here I use fs.promise but you can use writeFileSync it's equal
await fsPromises.writeFile(filePath, fileResponse.data);
Axios has internal capacity of handling streams and you don't need to necessarily meddle with low-level Node APIs for that.
Check out https://axios-http.com/docs/req_config (find the responseTypepart in the docs for all the types you can use).

If you just want the file use this
const media_data =await axios({url: url, method: "get", responseType: "arraybuffer"})
writeFile("./image.jpg", Buffer.from(media_data.data), {encoding: "binary"}, console.log)

import download from "downloadjs";
export const downloadFile = async (fileName) => {
axios({
method: "get",
url: `/api/v1/users/resume/${fileName}`,
responseType: "blob",
}).then(function (response) {
download(response.data, fileName);
});
};
it's work fine to me

This is my example code run with node js
There is a synctax error
should be writeFile not WriteFile
const axios = require('axios');
const fs = require('fs');
axios.get('http://www.africau.edu/images/default/sample.pdf', {responseType: 'blob'}).then(response => {
fs.writeFile('./my.pdf', response.data, (err) => {
if (err) throw err;
console.log('The file has been saved!');
});
});
After the file is saved it might look like in a text editor, but the file was saved properly
%PDF-1.3
%����
1 0 obj
<<
/Type /Catalog
/Outlines 2 0 R
/Pages 3 0 R
>>
endobj
2 0 obj
<<
/Type /Outlines
/Count 0
>>
endobj
3 0 obj
<<
/Type /Pages
/Count 2
/Kids [ 4 0 R 6 0 R ]
>>
endobj

Download and upload image without saving to disk

Using Node.js, I am trying to get an image from a URL and upload that image to another service without saving image to disk. I have the following code that works when saving the file to disk and using fs to create a readablestream. But as I am doing this as a cron job on a read-only file system (webtask.io) I'd want to achieve the same result without saving the file to disk temporarily. Shouldn't that be possible?
request(image.Url)
.pipe(
fs
.createWriteStream(image.Id)
.on('finish', () => {
client.assets
.upload('image', fs.createReadStream(image.Id))
.then(imageAsset => {
resolve(imageAsset)
})
})
)
Do you have any suggestions of how to achieve this without saving the file to disk? The upload client will take the following
client.asset.upload(type: 'file' | image', body: File | Blob | Buffer | NodeStream, options = {}): Promise<AssetDocument>
Thanks!

How about passing the buffer down to the upload function? Since as per your statement it'll accept a buffer.
As a side note... This will keep it in memory for the duration of the method execution, so if you call this numerous times you might run out of resources.
request.get(url, function (res) {
var data = [];
res.on('data', function(chunk) {
data.push(chunk);
}).on('end', function() {
var buffer = Buffer.concat(data);
// Pass the buffer
client.asset.upload(type: 'buffer', body: buffer);
});
});

I tried some various libraries and it turns out that node-fetch provides a way to return a buffer. So this code works:
fetch(image.Url)
.then(res => res.buffer())
.then(buffer => client.assets
.upload('image', buffer, {filename: image.Id}))
.then(imageAsset => {
resolve(imageAsset)
})

well I know it has been a few years since the question was originally asked, but I have encountered this problem now, and since I didn't find an answer with a comprehensive example I made one myself.
i'm assuming that the file path is a valid URL and that the end of it is the file name, I need to pass an apikey to this API endpoint, and a successful upload sends me back a response with a token.
I'm using node-fetch and form-data as dependencies.
const fetch = require('node-fetch');
const FormData = require('form-data');
const secretKey = 'secretKey';
const downloadAndUploadFile = async (filePath) => {
const fileName = new URL(filePath).pathname.split("/").pop();
const endpoint = `the-upload-endpoint-url`;
const formData = new FormData();
let jsonResponse = null;
try {
const download = await fetch(filePath);
const buffer = await download.buffer();
if (!buffer) {
console.log('file not found', filePath);
return null;
}
formData.append('file', buffer, fileName);
const response = await fetch(endpoint, {
method: 'POST', body: formData, headers: {
...formData.getHeaders(),
"Authorization": `Bearer ${secretKey}`,
},
});
jsonResponse = await response.json();
} catch (error) {
console.log('error on file upload', error);
}
return jsonResponse ? jsonResponse.token : null;
}