I am currently implementing a file download in a web application.
I was struggling a little with the pop up blocker, the download needs to start immediately after the user interaction, so there cannot be a server round trip.
Now I noticed, that Goolge Drive for example allows you to download folders. When you do that, their server first creates a compressed zip file, which takes some time.When the server finished, the download starts immediately without another user interaction.
Now I am wondering how this can be done?
I wrote a function to download a file via url.
In your case, you must use ajax request to make a zip file on server then give back url and run below function to download:
function download(url, filename){
fetch(url)
.then(resp => resp.blob())
.then(blob => {
const url = window.URL.createObjectURL(blob);
const a = document.createElement('a');
a.style.display = 'none';
a.href = url;
a.download = filename;
document.body.appendChild(a);
a.click();
window.URL.revokeObjectURL(url);
console.log('your file has downloaded!');
})
.catch(() => console.log('Download failed!'));
}
Test codepen here: download file via url
I noticed, that Google Drive for example allows you to download
folders. When you do that, their server first creates a compressed zip
file, which takes some time.
Alternatively, you could request the contents of the directory as json, then loop over the json and download each file as a blob and create a zip file, the only blocking then would be the request to the json, then you could show a status of each file downloaded etc.
Libs to do that:
FileSaver: https://github.com/eligrey/FileSaver.js/
JSZip: https://stuk.github.io/jszip/
Snippet example, using vue, s3 etc
async download(bucket) {
this.$snackbar.show({
type: 'bg-success text-white',
message: 'Building Zip, please wait...'
})
//..eek fetch all items in bucket, plop into a zip, then trigger download
// - there is some limits on final filesize, will work around it by disabling download :)
// resolve objects
const objects = async(bucket) => new Promise(async(resolve, reject) => {
let objectStream = await this.state.host.client.listObjectsV2(bucket, '', true)
let objects = []
//
objectStream.on('data', (obj) => {
if (obj && (obj.name || obj.prefix)) {
this.$snackbar.show({
type: 'bg-success text-white',
message: 'Fetching: ' + obj.name
})
objects.push(obj)
}
})
objectStream.on('end', () => resolve(objects))
objectStream.on('error', (e) => reject(e))
})
// get an objects data
const getObject = (bucket, name) => new Promise((resolve, reject) => {
this.state.host.client.getObject(bucket, name, (err, dataStream) => {
let chunks = []
dataStream.on('data', chunk => {
this.$snackbar.show({
type: 'bg-success text-white',
message: 'Downloading: ' + name
})
chunks.push(chunk)
})
dataStream.on('end', () => resolve(Buffer.concat(chunks || [])))
})
})
// fetch objects info a zip file
const makeZip = (bucket, objects) => new Promise(async(resolve, reject) => {
let zip = new JSZip()
for (let i in objects) {
this.$snackbar.show({
type: 'bg-success text-white',
message: 'Zipping: ' + objects[i].name
})
zip.file(objects[i].name, await getObject(bucket, objects[i].name));
}
zip.generateAsync({
type: "blob"
}).then(content => {
this.$snackbar.show({
type: 'bg-success text-white',
message: 'Zip Created'
})
resolve(content)
})
})
// using FileSaver, download file
const downloadZip = (content) => {
this.$snackbar.show({
type: 'bg-success text-white',
message: `Downloading: ${bucket.name}.zip`
})
FileSaver.saveAs(content, `${bucket.name}.zip`)
}
try {
downloadZip(await makeZip(bucket.name, await objects(bucket.name)))
} catch (e) {
this.$snackbar.show({
type: 'bg-danger text-white',
message: e.message
})
console.error(e)
}
},
If you want an ugly way to download a directory, fetch the json list then place a bunch of document.createElement('a') on the dom with a.setAttribute("target", "_blank"), but you will get a bunch of "Save As" dialogues open.
How to download multiple images?
Related
I'm new to JavaScript, and am trying to write some code that uses the google drive API (via the gapi client) to transform an existing slide into a pdf document, upload it to a specific folder, and return the pdf file id. This is all to be done in the browser, if possible.
I've already done this on python for another use case, and the code looks something like this:
import googleapiclient.http as client_methods
from io import BytesIO
...
data = drive.files().export(fileId=slideId, mimeType='application/pdf').execute()
body = {'name': fileName, 'mimeType': 'application/pdf', 'parents': [folderId]}
# wrapping the binary (data) file with BytesIO class
fh = io.BytesIO(data)
# creating the Media Io upload class for the file
media_body = client_methods.MediaIoBaseUpload(fh, mimetype='application/pdf')
pdfFileId = drive.files().create(body=body, media_body=media_body, supportsAllDrives=True).execute(['id'])
I've tried to replicate the same steps using JavaScript and my limited knowledge, and can successfully upload a pdf file into the desired folder, but the file shows as empty (doesn't even open in the drive).
I believe it might be due to the way I'm handling the binary data that I get from exporting the initial slide.
The last iteration of my JavaScript code is shown below (I have all the necessary permissions to use the gapi client):
async function createPdfFile() {
gapi.client.load("drive", "v3", function () {
// Set the MIME type for the exported file
const mimeType = "application/pdf";
// Set the file name for the exported PDF file
const fileName = "Trial upload.pdf";
// Export the Google Slides presentation as a PDF file
gapi.client.drive.files.export({
fileId,
mimeType
}).then(async function (response) {
// Get the binary data of the PDF file
const pdfData = await response.body;
const blob = await new Blob([pdfData], {type: 'application/pdf'})
const file = new File([blob], "presentation.pdf");
// Create a new file in the specified Google Drive folder with the PDF data
await gapi.client.drive.files.create({
name: fileName,
parents: [folderId],
mimeType: mimeType,
media: {mimeType: 'application/pdf', body: file},
supportsAllDrives: true
}).then(function (response) {
// Get the ID of the created PDF file
const pdfFileId = response.result.id;
console.log("PDF file created with ID: " + pdfFileId);
})
})
})
}
await createPdfFile()
As for the output, and as stated, it does create a pdf file, and logs the pdf file id, but the file itself is empty. I'd really appreciate it if someone could help me make sense of this (similar thread here, but can't replicate his success).
I believe your goal is as follows.
You want to convert Google Slides to PDF format using googleapis for Javascript.
Your access token can be exported and uploaded to Google Drive.
Issue and workaround:
When I tested your script, unfortunately, response.body from gapi.client.drive.files.export is binary data, and in this case, this cannot be correctly converted to the blob. And also, in the current stage, it seems that a file cannot be uploaded using gapi.client.drive.files.create. I thought that these might be the reason for your current issue.
From these situations, I would like to propose the flow for achieving your goal using fetch API. The modified script is as follows.
In this case, the access token is retrieved from the client like gapi.auth.getToken().access_token.
Modified script:
Please modify your script as follows.
From:
gapi.client.drive.files.export({
fileId,
mimeType
}).then(async function (response) {
// Get the binary data of the PDF file
const pdfData = await response.body;
const blob = await new Blob([pdfData], { type: 'application/pdf' })
const file = new File([blob], "presentation.pdf");
// Create a new file in the specified Google Drive folder with the PDF data
await gapi.client.drive.files.create({
name: fileName,
parents: [folderId],
mimeType: mimeType,
media: { mimeType: 'application/pdf', body: file },
supportsAllDrives: true
}).then(function (response) {
// Get the ID of the created PDF file
const pdfFileId = response.result.id;
console.log("PDF file created with ID: " + pdfFileId);
})
})
To:
gapi.client.drive.files.get({ fileId, fields: "exportLinks", supportsAllDrives: true }).then(function (response) {
const obj = JSON.parse(response.body);
if (Object.keys(obj).length == 0) throw new Error("This file cannot be converted to PDF format.");
const url = obj.exportLinks["application/pdf"];
if (!url) throw new Error("No exported URL.");
const accessToken = gapi.auth.getToken().access_token;
fetch(url, {
method: 'GET',
headers: { 'Authorization': 'Bearer ' + accessToken },
})
.then(res => res.blob())
.then(blob => {
const metadata = { name: fileName, parents: [folderId], mimeType };
const form = new FormData();
form.append('metadata', new Blob([JSON.stringify(metadata)], { type: 'application/json' }));
form.append('file', blob);
fetch('https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart&supportsAllDrives=true', {
method: 'POST',
headers: { 'Authorization': 'Bearer ' + accessToken },
body: form
})
.then(res => res.json())
.then(obj => console.log("PDF file created with ID: " + obj.id));
});
});
When this script is run, the export URL of PDF data is retrieved from the file ID. And, the PDF data is downloaded and uploaded to Google Drive.
Note:
In your script, fileId is not declared. Please be careful about this.
If the file size is more than 5 MB, please use the resumable upload.
Reference:
Upload file data
Added:
From your following reply,
?uploadType=multipart also returns a 404 type error
I'm worried about that in your situation, new FormData() might not be able to be used. If my understanding is correct, please test the following script. In this script, the request body of multipart/form-data is manually created.
Modified script:
gapi.client.drive.files.get({ fileId, fields: "exportLinks", supportsAllDrives: true }).then(function (response) {
const obj = JSON.parse(response.body);
if (Object.keys(obj).length == 0) throw new Error("This file cannot be converted to PDF format.");
const url = obj.exportLinks["application/pdf"];
if (!url) throw new Error("No exported URL.");
const accessToken = gapi.auth.getToken().access_token;
fetch(url, {
method: 'GET',
headers: { 'Authorization': 'Bearer ' + accessToken },
})
.then(res => res.blob())
.then(blob => {
const metadata = { name: fileName, parents: [folderId], mimeType };
const fr = new FileReader();
fr.onload = e => {
const data = e.target.result.split(",");
const req = "--xxxxxxxx\r\n" +
"Content-Type: application/json\r\n\r\n" +
JSON.stringify(metadata) + "\r\n" +
"--xxxxxxxx\r\n" +
"Content-Transfer-Encoding: base64\r\n\r\n" +
data[1] + "\r\n" +
"--xxxxxxxx--";
fetch('https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart&supportsAllDrives=true', {
method: 'POST',
headers: { 'Authorization': 'Bearer ' + accessToken, "Content-Type": "multipart/related; boundary=xxxxxxxx" },
body: req
})
.then(res => res.json())
.then(obj => {
console.log("PDF file created with ID: " + obj.id)
});
}
fr.readAsDataURL(blob);
});
});
When I tested this script, no error occurs. I confirmed that the Google Slides file could be converted to a PDF file and the PDF file was uploaded to the specific folder.
noob question, I'm just getting started with Google Drive API v3. How can I download dynamic file from google drive when I only have fileId. file can be, image, pdf, or docs.
I tried searching but I couldn't found any reference or example related to this.
This what I have so far but it only download specific file extension.
downloadFile(req, res) {
const auth = new google.auth.JWT(
client_email,
null,
private_key,
SCOPES,
);
const { fileId } = req.params;
const drive = google.drive({ version: 'v3', auth});
var dest = fs.createWriteStream('./tmp/downloads/dummy.pdf')
drive.files.get({
fileId,
alt: 'media',
}, {
responseType: 'stream'
}).then((driveResponse) => {
driveResponse.data.on('end', () => {
console.log(`downloading fileID ${fileId}`);
})
.on('error', (err) => {
console.log(err);
})
.on('data', (d) => {
console.log(d);
})
.pipe(dest)
})
.catch((err) => {
console.log(err);
})
}
Is there way to download dynamic files from google drive?
I believe your goal as follows.
You want to download the files from Google Drive using the service account and the file ID.
The files include both Google Docs files and the files except for Google Docs files.
You want to achieve this using googleapis for Node.js.
Modification points:
Unfortunately, from it only download specific file extension., I cannot understand about the detail of your situation. But I guess that the reason of your issue might be due to downloading both Google Docs files and the files except for Google Docs files.
When Google Docs files are downloaded, the files are required to be downloaded using the method of "Files: export" in Drive API.
When the files except for Google Docs files are downloaded, the files are required to be downloaded using the method of "Files: get" in Drive API.
I thought that above situation might be the reason of your issue.
In order to download both Google Docs files and the files except for Google Docs files, I propose the following flow.
Check the mimeType of the file ID.
Download the file using each method by the mimeType.
When above points are reflected to your script, it becomes as follows.
Modified script:
From:
var dest = fs.createWriteStream('./tmp/downloads/dummy.pdf')
drive.files.get({
fileId,
alt: 'media',
}, {
responseType: 'stream'
}).then((driveResponse) => {
driveResponse.data.on('end', () => {
console.log(`downloading fileID ${fileId}`);
})
.on('error', (err) => {
console.log(err);
})
.on('data', (d) => {
console.log(d);
})
.pipe(dest)
})
.catch((err) => {
console.log(err);
})
To:
drive.files.get({ fileId, fields: "*" }, async (err, { data }) => {
if (err) {
console.log(err);
return;
}
let filename = data.name;
const mimeType = data.mimeType;
let res;
if (mimeType.includes("application/vnd.google-apps")) {
const convertMimeTypes = {
"application/vnd.google-apps.document": {
type:
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
ext: ".docx",
},
"application/vnd.google-apps.spreadsheet": {
type:
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
ext: ".xlsx",
},
"application/vnd.google-apps.presentation": {
type:
"application/vnd.openxmlformats-officedocument.presentationml.presentation",
ext: ".pptx",
},
};
filename += convertMimeTypes[mimeType].ext;
res = await drive.files.export(
{
fileId,
mimeType: convertMimeTypes[mimeType].type,
},
{ responseType: "stream" }
);
} else {
res = await drive.files.get(
{
fileId,
alt: "media",
},
{ responseType: "stream" }
);
}
const dest = fs.createWriteStream(filename);
res.data
.on("end", () => console.log("Done."))
.on("error", (err) => {
console.log(err);
return process.exit();
})
.pipe(dest);
});
Note:
In this modification, I prepared 3 types of Google Docs files at convertMimeTypes. When you want to download other mimeTypes, please modify convertMimeTypes. In this case, for example, Google Docs files are downloaded as Microsoft Docs files.
References:
Download files
Files: get
Files: export
im having a website where i need to support download features in browser.
However, the video url is a stream url that returns m3u8
I dont have much knowledge in this file format, but if I understand correctly, it will break the video into multiple chunks and keep downloading the next chunk after finished
Now my question is
Is there any way we can write a script (that run in browsers) to download all chunks at once, and then concatenate them all to become a final video?
fetch(
'video link here',
)
// Retrieve its body as ReadableStream
.then(response => {
const reader = response.body.getReader();
return new ReadableStream({
start(controller) {
return pump();
function pump() {
return reader.read().then(({ done, value }) => {
// When no more data needs to be consumed, close the stream
if (done) {
controller.close();
return;
}
// Enqueue the next data chunk into our target stream
controller.enqueue(value);
return pump();
});
}
},
});
})
// Create a new response out of the stream
.then(stream => new Response(stream))
// Create an object URL for the response
.then(response => response.blob())
.then(blob => {
console.log('finish downloading blob');
const link = document.createElement('a');
link.href = URL.createObjectURL(blob);
link.download = _filename;
link.click();
})
.catch(err => console.error(err));
I have two functions in separate files to split up the workflow.
const download = function(url){
const file = fs.createWriteStream("./test.png");
const request = https.get(url, function(response) {
response.pipe(file);
});
}
This function in my fileHelper.js is supposed to take a URL with an image in it and then save it locally to test.png
function uploadFile(filePath) {
fs.readFile('credentials.json', (err, content) => {
if (err) return console.log('Error loading client secret file:', err);
// Authorize a client with credentials, then call the Google Drive API.
authorize(JSON.parse(content), function (auth) {
const drive = google.drive({version: 'v3', auth});
const fileMetadata = {
'name': 'testphoto.png'
};
const media = {
mimeType: 'image/png',
body: fs.createReadStream(filePath)
};
drive.files.create({
resource: fileMetadata,
media: media,
fields: 'id'
}, (err, file) => {
if (err) {
// Handle error
console.error(err);
} else {
console.log('File Id: ', file.id);
}
});
});
});
}
This function in my googleDriveHelper.js is supposed to take the filePath of call and then upload that stream into my google drive. These two functions work on their own but it seems that the https.get works asynchronously and if I try to call the googleDriveHelper.uploadFile(filePath) function after the download, it doesn't have time to get the full file to upload so instead a blank file will be uploaded to my drive.
I want to find a way so that when the fileHelper.download(url) is called, it automatically uploads into my drive.
I also don't know if there is a way to create a readStream directly from the download function to the upload function, so I can avoid having to save the file locally to upload it.
I believe your goal as follows.
You want to upload a file retrieving from an URL to Google Drive.
When you download the file from the URL, you want to upload it to Google Drive without creating the file.
You want to achieve this using googleapis with Node.js.
You have already been able to upload a file using Drive API.
For this, how about this answer?
Modification points:
At download function, the retrieved buffer is converted to the stream type, and the stream data is returned.
At uploadFile function, the retrieved stream data is used for uploading.
When the file ID is retrieved from the response value of Drive API, please use file.data.id instead of file.id.
By above modification, the file downloaded from the URL can be uploaded to Google Drive without creating a file.
Modified script:
When your script is modified, please modify as follows.
download()
const download = function (url) {
return new Promise(function (resolve, reject) {
request(
{
method: "GET",
url: url,
encoding: null,
},
(err, res, body) => {
if (err && res.statusCode != 200) {
reject(err);
return;
}
const stream = require("stream");
const bs = new stream.PassThrough();
bs.end(body);
resolve(bs);
}
);
});
};
uploadFile()
function uploadFile(data) { // <--- Modified
fs.readFile("drive_credentials.json", (err, content) => {
if (err) return console.log("Error loading client secret file:", err);
authorize(JSON.parse(content), function (auth) {
const drive = google.drive({ version: "v3", auth });
const fileMetadata = {
name: "testphoto.png",
};
const media = {
mimeType: "image/png",
body: data, // <--- Modified
};
drive.files.create(
{
resource: fileMetadata,
media: media,
fields: "id",
},
(err, file) => {
if (err) {
console.error(err);
} else {
console.log("File Id: ", file.data.id); // <--- Modified
}
}
);
});
});
}
For testing
For example, when above scripts are tested, how about the following script?
async function run() {
const url = "###";
const data = await fileHelper.download(url);
googleDriveHelper.uploadFile(data);
}
References:
Class: stream.PassThrough
google-api-nodejs-client
Using Node.js, I am trying to get an image from a URL and upload that image to another service without saving image to disk. I have the following code that works when saving the file to disk and using fs to create a readablestream. But as I am doing this as a cron job on a read-only file system (webtask.io) I'd want to achieve the same result without saving the file to disk temporarily. Shouldn't that be possible?
request(image.Url)
.pipe(
fs
.createWriteStream(image.Id)
.on('finish', () => {
client.assets
.upload('image', fs.createReadStream(image.Id))
.then(imageAsset => {
resolve(imageAsset)
})
})
)
Do you have any suggestions of how to achieve this without saving the file to disk? The upload client will take the following
client.asset.upload(type: 'file' | image', body: File | Blob | Buffer | NodeStream, options = {}): Promise<AssetDocument>
Thanks!
How about passing the buffer down to the upload function? Since as per your statement it'll accept a buffer.
As a side note... This will keep it in memory for the duration of the method execution, so if you call this numerous times you might run out of resources.
request.get(url, function (res) {
var data = [];
res.on('data', function(chunk) {
data.push(chunk);
}).on('end', function() {
var buffer = Buffer.concat(data);
// Pass the buffer
client.asset.upload(type: 'buffer', body: buffer);
});
});
I tried some various libraries and it turns out that node-fetch provides a way to return a buffer. So this code works:
fetch(image.Url)
.then(res => res.buffer())
.then(buffer => client.assets
.upload('image', buffer, {filename: image.Id}))
.then(imageAsset => {
resolve(imageAsset)
})
well I know it has been a few years since the question was originally asked, but I have encountered this problem now, and since I didn't find an answer with a comprehensive example I made one myself.
i'm assuming that the file path is a valid URL and that the end of it is the file name, I need to pass an apikey to this API endpoint, and a successful upload sends me back a response with a token.
I'm using node-fetch and form-data as dependencies.
const fetch = require('node-fetch');
const FormData = require('form-data');
const secretKey = 'secretKey';
const downloadAndUploadFile = async (filePath) => {
const fileName = new URL(filePath).pathname.split("/").pop();
const endpoint = `the-upload-endpoint-url`;
const formData = new FormData();
let jsonResponse = null;
try {
const download = await fetch(filePath);
const buffer = await download.buffer();
if (!buffer) {
console.log('file not found', filePath);
return null;
}
formData.append('file', buffer, fileName);
const response = await fetch(endpoint, {
method: 'POST', body: formData, headers: {
...formData.getHeaders(),
"Authorization": `Bearer ${secretKey}`,
},
});
jsonResponse = await response.json();
} catch (error) {
console.log('error on file upload', error);
}
return jsonResponse ? jsonResponse.token : null;
}