Develop Reference

How to Download Images from a List of URLs in Google Sheets using Google App Script? [duplicate]

noob question, I'm just getting started with Google Drive API v3. How can I download dynamic file from google drive when I only have fileId. file can be, image, pdf, or docs.
I tried searching but I couldn't found any reference or example related to this.
This what I have so far but it only download specific file extension.
downloadFile(req, res) {
const auth = new google.auth.JWT(
client_email,
null,
private_key,
SCOPES,
);
const { fileId } = req.params;
const drive = google.drive({ version: 'v3', auth});
var dest = fs.createWriteStream('./tmp/downloads/dummy.pdf')
drive.files.get({
fileId,
alt: 'media',
}, {
responseType: 'stream'
}).then((driveResponse) => {
driveResponse.data.on('end', () => {
console.log(`downloading fileID ${fileId}`);
})
.on('error', (err) => {
console.log(err);
})
.on('data', (d) => {
console.log(d);
})
.pipe(dest)
})
.catch((err) => {
console.log(err);
})
}
Is there way to download dynamic files from google drive?

I believe your goal as follows.
You want to download the files from Google Drive using the service account and the file ID.
The files include both Google Docs files and the files except for Google Docs files.
You want to achieve this using googleapis for Node.js.
Modification points:
Unfortunately, from it only download specific file extension., I cannot understand about the detail of your situation. But I guess that the reason of your issue might be due to downloading both Google Docs files and the files except for Google Docs files.
When Google Docs files are downloaded, the files are required to be downloaded using the method of "Files: export" in Drive API.
When the files except for Google Docs files are downloaded, the files are required to be downloaded using the method of "Files: get" in Drive API.
I thought that above situation might be the reason of your issue.
In order to download both Google Docs files and the files except for Google Docs files, I propose the following flow.
Check the mimeType of the file ID.
Download the file using each method by the mimeType.
When above points are reflected to your script, it becomes as follows.
Modified script:
From:
var dest = fs.createWriteStream('./tmp/downloads/dummy.pdf')
drive.files.get({
fileId,
alt: 'media',
}, {
responseType: 'stream'
}).then((driveResponse) => {
driveResponse.data.on('end', () => {
console.log(`downloading fileID ${fileId}`);
})
.on('error', (err) => {
console.log(err);
})
.on('data', (d) => {
console.log(d);
})
.pipe(dest)
})
.catch((err) => {
console.log(err);
})
To:
drive.files.get({ fileId, fields: "*" }, async (err, { data }) => {
if (err) {
console.log(err);
return;
}
let filename = data.name;
const mimeType = data.mimeType;
let res;
if (mimeType.includes("application/vnd.google-apps")) {
const convertMimeTypes = {
"application/vnd.google-apps.document": {
type:
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
ext: ".docx",
},
"application/vnd.google-apps.spreadsheet": {
type:
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
ext: ".xlsx",
},
"application/vnd.google-apps.presentation": {
type:
"application/vnd.openxmlformats-officedocument.presentationml.presentation",
ext: ".pptx",
},
};
filename += convertMimeTypes[mimeType].ext;
res = await drive.files.export(
{
fileId,
mimeType: convertMimeTypes[mimeType].type,
},
{ responseType: "stream" }
);
} else {
res = await drive.files.get(
{
fileId,
alt: "media",
},
{ responseType: "stream" }
);
}
const dest = fs.createWriteStream(filename);
res.data
.on("end", () => console.log("Done."))
.on("error", (err) => {
console.log(err);
return process.exit();
})
.pipe(dest);
});
Note:
In this modification, I prepared 3 types of Google Docs files at convertMimeTypes. When you want to download other mimeTypes, please modify convertMimeTypes. In this case, for example, Google Docs files are downloaded as Microsoft Docs files.
References:
Download files
Files: get
Files: export

Get contents of a specific file in google drive API with NodeJS

I have found many posts on how to retrieve the contents of a .txt file from the google drive with their API. I have tried using this:
const drive = google.drive({version: 'v3', auth});
var data = drive.files.get({
fileId: file_id,
alt: "media"
});
data.execute(function(response){
console.log(reponse)
})
My error
data.execute(function(response){
^
TypeError: data.execute is not a function
and also data.then instead of data.execute each time there is an error that I research and find no resolution to it. Could someone please give me an updated version of how to get the contents of a file from the file id? As I think the previous answers are outdated somewhat.
Sorry if this is pretty obvious. I relatively new to javascript and apis in general. So this would help me out a lot as it's the final stretch before I finish my program :)
Thanks, Mathias

When you run 'drive.files.get' for google drive API you receive a promise, and to get data you have to use then on it. This is how it works:
const filePath = `give_path_tosave_file`;
const dest = fs.createWriteStream(filePath);
let progress = 0;
drive.files.get(
{ fileId, alt: 'media' },
{ responseType: 'stream' }
).then(res => {
res.data
.on('end', () => {
console.log('Done downloading file.');
})
.on('error', err => {
console.error('Error downloading file.');
})
.on('data', d => {
d+='';
console.log(d);
//data will be here
// pipe it to write stream
}
})
.pipe(dest);
});
If above solution doesn't work, you can use this one. It is on official google website for doing the same:
var fileId = '1ZdR3L3qP4Bkq8noWLJHSr_iBau0DNT4Kli4SxNc2YEo';
var dest = fs.createWriteStream('/tmp/filename.txt');
drive.files.export({
fileId: fileId,
mimeType: 'application/txt'
})
.on('end', function () {
console.log('Done');
})
.on('error', function (err) {
console.log('Error during download', err);
})
.pipe(dest);
For more info, you should check here.
Also, the below method will return all the files which you have access to in drive.
drive.files.list({}, (err, res) => {
if (err) throw err;
const files = res.data.files;
if (files.length) {
files.map((file) => {
console.log(file);
});
} else {
console.log('No files found');
}
});

CORS Error on uploading image/video file to google cloud in react js

When user drag and drop the Image, I need to call a method of the server to get the Media_id for that particluar image/video, in the response of that I am getting this ->
MAIN RESPONSE -->>
{
"status": 1,
"media": {"media_id": 27, "media_type": 1, "media_file_name": "a9989aafcdf1482d8a0967a81b54b476_80a2d60394f15063bef4e44e1a4d83f3.png", "media_placeholder": null, "media_ext": "png"},
"upload":
{
"upload_url": "https://storage.googleapis.com/fnc-59aa2e6b-71552c9d-6441d628-951a8f6f/l.img/ori/a9989aafcdf1482d8a0967a81b54b476_80a2d60394f15063bef4e44e1a4d83f3.png?Expires=1603388214&GoogleAccessId=12345678-compute%40developer.gserviceaccount.com&Signature=UNt8nS3%2BJYiS4AuYdZ7Z2fvfDZ0fAKf8bSZbeRlHyhqxb5i6xjpqnqgR7JYp9Q3FgJItcYr%2BHDL90WiUpbMQi%2B4s0XNW683CaSoUChkRMjj1AvkH%2Be0u8%2Fw5VVIMF9j52bTFePWISTLvwQ1RlEdNPNkrpbcamTsJFyBVi89%2BIpXArsVlhvDzK55Zvj%2Fvzh00GgdNrH%2BRog8Q%2BkGITE8bW%2FxRpQ30OdMZLjpLtp%2FNg5KVotHrx6Bet7vidKymiJQ9BbwCxTRGzBdAITr2rsKTMGZJzfvEKnIczsoiY91Zmc3hjGzUD9OxHGR%2BiRdN%2F2FbotOIVR48RE%2BoAdIGIEfKlw%3D%3D",
"file_name": "a9989aafcdf1482d8a0967a81b54b476_80a2d60394f15063bef4e44e1a4d83f3.png",
"content_type": "image/png", "exp": "2020-10-22 17:36:54.447484"
}}
So, I need to hit this upload url which is coming from the response.Below is my file where I am hitting this as soon as user drop the image ->
UploadImage.js
await this.props.getFirstMediaId(postdata).then(res => {
if (res.value && res.value.status === 1) {
let media_idArr = this.state.media_id.concat(res.value.media.media_id)
this.setState({ media_id: media_idArr, mediaUrl: res.value.upload })
customStatus = 'done';
}
}) //First call to the server to get Media_id and the cloud **upload URL**
***** FOR THIS API RESPONSE, PLEASE SEE THE ABOVE MAIN RESPONSE *****
const getUploadParams = () => {
console.log(this.state.mediaUrl, ' -->>> this.state.mediaUrl')
if (this.state.mediaUrl !== null) {
console.log(' in get upload param.')
return this.props.postImageToCloud(this.state.mediaUrl).then(res => {
console.log(res, '===>> here is cloud res.')
})
.catch(err => {
console.log(' here is error cloud -->>> ', err)
})
}
}
Below is the file where the method actually call API ->
service.js
export const getFirstMediaId = (data) => {
return {
type: GET_FIRST_LISTING_MEDIA,
async payload() {
let response = await callAxios.post(SUBMIT_LISTING_FIRST_MEDIA, data);
return objectPath.get(response, 'data', []);
}
}
}
export const postImageToCloud = (url) => {
return {
type: PUT_MEDIA_TO_CLOUD,
async payload() {
let response = await axios.put(url.upload_url, {}, {
headers: {
'Content-Type': `${url.content_type}`
}
})
return objectPath.get(response, 'data', []);
}
}
}
So, the first call is success and I got the above MAIN RESPONSE but as soon as it completes, I call the cloud PUT request and got this CORS error ->
Access to XMLHttpRequest at 'https://storage.googleapis.com/fnc-59aa2e6b-71552c9d-6441d628-951a8f6f/l.img/ori/a9989aafcdf1482d8a0967a81b54b476_80a2d60394f15063bef4e44e1a4d83f3.png?Expires=1603388214&GoogleAccessId=123456789-compute%40developer.gserviceaccount.com&Signature=UNt8nS3%2BJYiS4AuYdZ7Z2fvfDZ0fAKf8bSZbeRlHyhqxb5i6xjpqnqgR7JYp9Q3FgJItcYr%2BHDL90WiUpbMQi%2B4s0XNW683CaSoUChkRMjj1AvkH%2Be0u8%2Fw5VVIMF9j52bTFePWISTLvwQ1RlEdNPNkrpbcamTsJFyBVi89%2BIpXArsVlhvDzK55Zvj%2Fvzh00GgdNrH%2BRog8Q%2BkGITE8bW%2FxRpQ30OdMZLjpLtp%2FNg5KVotHrx6Bet7vidKymiJQ9BbwCxTRGzBdAITr2rsKTMGZJzfvEKnIczsoiY91Zmc3hjGzUD9OxHGR%2BiRdN%2F2FbotOIVR48RE%2BoAdIGIEfKlw%3D%3D' from origin 'http://localhost:8000' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Please suggest me anything for make it working.
Thanks.

Finally after lot of efforts I came to know that I have to pass the file in the body of PUT request, here ->
let response = await axios.put(url.upload_url, { **file here** }, {
headers: {
'Content-Type': `${url.content_type}`
}
})
But I tried passing the image file object simple the html file object using formData and passing as it is, still getting the same error. Then i started using
react-dropzone and converted the image file into the string buffer as one of the example in React-dropzone. I am going to paste here that example maybe it can help anyone. See below ->
import React, {useCallback} from 'react'
import {useDropzone} from 'react-dropzone'
function MyDropzone() {
const onDrop = useCallback((acceptedFiles) => {
acceptedFiles.forEach((file) => {
const reader = new FileReader()
reader.onabort = () => console.log('file reading was aborted')
reader.onerror = () => console.log('file reading has failed')
reader.onload = () => {
// Do whatever you want with the file contents
const binaryStr = reader.result
console.log(binaryStr)
*****PASS THIS (binaryStr) AS IN THE BODY OF PUT TO AXIOS****
}
reader.readAsArrayBuffer(file)
})
}, [])
const {getRootProps, getInputProps} = useDropzone({onDrop})
return (
<div {...getRootProps()}>
<input {...getInputProps()} />
<p>Drag 'n' drop some files here, or click to select files</p>
</div>
)
}
This is one of the Official examples of React-Dropzone, So I just pass that string buffer obj and finally It worked, no CORS issue nothing.

The Google Storage API does simply not accept requests initiated from a browser in another domain, so you won't be able to achieve this.
You should not call the API from a client but from your backend. Here is the list of the suggested libraries: https://cloud.google.com/storage/docs/reference/libraries?hl=fr
Note that JavaScript in a browser environment is not suggested (only Node.js is)

Upload file to google drive after http get request

I have two functions in separate files to split up the workflow.
const download = function(url){
const file = fs.createWriteStream("./test.png");
const request = https.get(url, function(response) {
response.pipe(file);
});
}
This function in my fileHelper.js is supposed to take a URL with an image in it and then save it locally to test.png
function uploadFile(filePath) {
fs.readFile('credentials.json', (err, content) => {
if (err) return console.log('Error loading client secret file:', err);
// Authorize a client with credentials, then call the Google Drive API.
authorize(JSON.parse(content), function (auth) {
const drive = google.drive({version: 'v3', auth});
const fileMetadata = {
'name': 'testphoto.png'
};
const media = {
mimeType: 'image/png',
body: fs.createReadStream(filePath)
};
drive.files.create({
resource: fileMetadata,
media: media,
fields: 'id'
}, (err, file) => {
if (err) {
// Handle error
console.error(err);
} else {
console.log('File Id: ', file.id);
}
});
});
});
}
This function in my googleDriveHelper.js is supposed to take the filePath of call and then upload that stream into my google drive. These two functions work on their own but it seems that the https.get works asynchronously and if I try to call the googleDriveHelper.uploadFile(filePath) function after the download, it doesn't have time to get the full file to upload so instead a blank file will be uploaded to my drive.
I want to find a way so that when the fileHelper.download(url) is called, it automatically uploads into my drive.
I also don't know if there is a way to create a readStream directly from the download function to the upload function, so I can avoid having to save the file locally to upload it.

I believe your goal as follows.
You want to upload a file retrieving from an URL to Google Drive.
When you download the file from the URL, you want to upload it to Google Drive without creating the file.
You want to achieve this using googleapis with Node.js.
You have already been able to upload a file using Drive API.
For this, how about this answer?
Modification points:
At download function, the retrieved buffer is converted to the stream type, and the stream data is returned.
At uploadFile function, the retrieved stream data is used for uploading.
When the file ID is retrieved from the response value of Drive API, please use file.data.id instead of file.id.
By above modification, the file downloaded from the URL can be uploaded to Google Drive without creating a file.
Modified script:
When your script is modified, please modify as follows.
download()
const download = function (url) {
return new Promise(function (resolve, reject) {
request(
{
method: "GET",
url: url,
encoding: null,
},
(err, res, body) => {
if (err && res.statusCode != 200) {
reject(err);
return;
}
const stream = require("stream");
const bs = new stream.PassThrough();
bs.end(body);
resolve(bs);
}
);
});
};
uploadFile()
function uploadFile(data) { // <--- Modified
fs.readFile("drive_credentials.json", (err, content) => {
if (err) return console.log("Error loading client secret file:", err);
authorize(JSON.parse(content), function (auth) {
const drive = google.drive({ version: "v3", auth });
const fileMetadata = {
name: "testphoto.png",
};
const media = {
mimeType: "image/png",
body: data, // <--- Modified
};
drive.files.create(
{
resource: fileMetadata,
media: media,
fields: "id",
},
(err, file) => {
if (err) {
console.error(err);
} else {
console.log("File Id: ", file.data.id); // <--- Modified
}
}
);
});
});
}
For testing
For example, when above scripts are tested, how about the following script?
async function run() {
const url = "###";
const data = await fileHelper.download(url);
googleDriveHelper.uploadFile(data);
}
References:
Class: stream.PassThrough
google-api-nodejs-client

Download and upload image without saving to disk

Using Node.js, I am trying to get an image from a URL and upload that image to another service without saving image to disk. I have the following code that works when saving the file to disk and using fs to create a readablestream. But as I am doing this as a cron job on a read-only file system (webtask.io) I'd want to achieve the same result without saving the file to disk temporarily. Shouldn't that be possible?
request(image.Url)
.pipe(
fs
.createWriteStream(image.Id)
.on('finish', () => {
client.assets
.upload('image', fs.createReadStream(image.Id))
.then(imageAsset => {
resolve(imageAsset)
})
})
)
Do you have any suggestions of how to achieve this without saving the file to disk? The upload client will take the following
client.asset.upload(type: 'file' | image', body: File | Blob | Buffer | NodeStream, options = {}): Promise<AssetDocument>
Thanks!

How about passing the buffer down to the upload function? Since as per your statement it'll accept a buffer.
As a side note... This will keep it in memory for the duration of the method execution, so if you call this numerous times you might run out of resources.
request.get(url, function (res) {
var data = [];
res.on('data', function(chunk) {
data.push(chunk);
}).on('end', function() {
var buffer = Buffer.concat(data);
// Pass the buffer
client.asset.upload(type: 'buffer', body: buffer);
});
});

I tried some various libraries and it turns out that node-fetch provides a way to return a buffer. So this code works:
fetch(image.Url)
.then(res => res.buffer())
.then(buffer => client.assets
.upload('image', buffer, {filename: image.Id}))
.then(imageAsset => {
resolve(imageAsset)
})

well I know it has been a few years since the question was originally asked, but I have encountered this problem now, and since I didn't find an answer with a comprehensive example I made one myself.
i'm assuming that the file path is a valid URL and that the end of it is the file name, I need to pass an apikey to this API endpoint, and a successful upload sends me back a response with a token.
I'm using node-fetch and form-data as dependencies.
const fetch = require('node-fetch');
const FormData = require('form-data');
const secretKey = 'secretKey';
const downloadAndUploadFile = async (filePath) => {
const fileName = new URL(filePath).pathname.split("/").pop();
const endpoint = `the-upload-endpoint-url`;
const formData = new FormData();
let jsonResponse = null;
try {
const download = await fetch(filePath);
const buffer = await download.buffer();
if (!buffer) {
console.log('file not found', filePath);
return null;
}
formData.append('file', buffer, fileName);
const response = await fetch(endpoint, {
method: 'POST', body: formData, headers: {
...formData.getHeaders(),
"Authorization": `Bearer ${secretKey}`,
},
});
jsonResponse = await response.json();
} catch (error) {
console.log('error on file upload', error);
}
return jsonResponse ? jsonResponse.token : null;
}