Develop Reference

How to use a Google Cloud Task Queue and get a json response?

I would like to use Google Cloud Task Queue in my firebase cloud function to call another function and return the work. Is this something we can do with Google Cloud Task?
I am trying to do something similar such as Redis queue where once the work is done I can get the results and continue with my app.
Here I am trying to show what I want to do with a simple addition function. In reality, I would like to call another Http endpoint and use my Queue to rate limit request. For now, this example gets the point across however I cant seem to send the data back.
exports.addNumbersFunc = functions.https.onRequest(async (data, res) => {
const payload = data.body;
console.log("Received: ", payload);
// Work done
const answer = payload["numbers"]["a"] + payload["numbers"]["b"];
// Can I send back the answer here?
res.json({ "answer": answer });
}
);
exports.exampleFunction1 = functions.https.onCall(async (data) => {
// create json object to send to server with data
const jsonPayload = JSON.stringify({
"numbers": { "a": 1, "b": 2 }
});
const url =
`https://${location}-${project}.cloudfunctions.net/addNumbersFunc`;
const task = {
httpRequest: {
httpMethod: 'POST',
url: url,
headers: {
'Content-Type': 'application/json'
},
body: Buffer.from(jsonPayload).toString('base64')
}, oidcToken: {
serviceAccountEmail
}
};
const request = {
parent: parent,
task: task,
};
// Send create task request.
console.log('Sending task:');
const [response] = await client.createTask(request);
console.log(`Response: ${response}`);
console.log(`Created task ${response.name}`);
// Would the response be here??
console.log(`Response httpReques: ${response.httpRequest.body}`);
return response;
});

Cloud Task is async. You cann't get the HTTP response of the task; –
guillaume blaquiere

Batch Delete Request with Google REST api getting Bad Request error

I am trying to delete multiple files at once from my google drive using the REST API. I found this SO Question-bulk deletion Google Drive API, I have tried this solution without the npm package using fetch.
This is what I have at the moment.
const boundary = 'END_OF_PART';
const separation = `\n--'${boundary}'\n`;
const ending = `\n--'${boundary}'--`;
const requestBody = files.reduce((accum, current) => {
accum += separation +
'Content-Type: application/http\n\n' +
`DELETE https://www.googleapis.com/drive/v2/files/${current.id}
\nAuthorization: Bearer ${this.getToken()}`
return accum
}, '') + ending
const _multiPart = [];
files.forEach((file) => {
const obj = {
"Content-Type": "application/http",
body: `DELETE https://www.googleapis.com/drive/v2/files/${file.id}\n`,
};
_multiPart.push(obj);
});
const url = new URL('https://www.googleapis.com/batch/drive/v3');
const requestOptions = {
method: 'POST',
headers: {
"Content-Type": "multipart/mixed",
'Authorization' : `Bearer ${this.getToken()}`
},
body: requestBody,
multipart: _multiPart,
};
await fetch(url, requestOptions);
When I run the deletion I get the id's of the files but the end result is Error 400 bad request. Documentation is scarce for this process. What is it that I am doing wrong here and how can I fix it to get this to work?
End result that I am looking for is deleting a large amount of files from my google drive.
Can anyone point me in the right direction on this ?.
EDIT: I just tried a solution at this SO Question: Bulk delete files on Google Drive with raw XMLHttpRequest
And it did delete 100 files, but the response I got back was Code 500, Internal error.
Why am I getting this response and again why is my previous code not working at all ?

I believe your goal is as follows.
In your script, files is an array including the file IDs like ["fileId1", "fileId2",,,].
In your goal, you want to delete files with the batch requests using the array files by the fetch API of Javascript.
Your access token can be used for deleting files and you have permission for deleting those files.
In this case, how about the following sample script? In order to use the batch requests with Javascript, I have created a library. Ref In this answer, I would like to use this library.
Sample script:
<script src="https://cdn.jsdelivr.net/gh/tanaikech/BatchRequest_js#master/batchrequest_js.min.js"></script>
<script>
const accessToken = "###"; // Please set your access token.
const files = ["fileId1", "fileId2",,,]; // Please set your file IDs.
const object = {
accessToken,
batchPath: "batch/drive/v3",
requests: files.map(id => ({ method: "DELETE", endpoint: "https://www.googleapis.com/drive/v3/files/" + id })),
};
Do(object)
.then((e) => console.log(e))
.catch((err) => console.log(err));
</script>
References:
Batch requests of official document
BatchRequest_js (Author: me)

CORS Error on uploading image/video file to google cloud in react js

When user drag and drop the Image, I need to call a method of the server to get the Media_id for that particluar image/video, in the response of that I am getting this ->
MAIN RESPONSE -->>
{
"status": 1,
"media": {"media_id": 27, "media_type": 1, "media_file_name": "a9989aafcdf1482d8a0967a81b54b476_80a2d60394f15063bef4e44e1a4d83f3.png", "media_placeholder": null, "media_ext": "png"},
"upload":
{
"upload_url": "https://storage.googleapis.com/fnc-59aa2e6b-71552c9d-6441d628-951a8f6f/l.img/ori/a9989aafcdf1482d8a0967a81b54b476_80a2d60394f15063bef4e44e1a4d83f3.png?Expires=1603388214&GoogleAccessId=12345678-compute%40developer.gserviceaccount.com&Signature=UNt8nS3%2BJYiS4AuYdZ7Z2fvfDZ0fAKf8bSZbeRlHyhqxb5i6xjpqnqgR7JYp9Q3FgJItcYr%2BHDL90WiUpbMQi%2B4s0XNW683CaSoUChkRMjj1AvkH%2Be0u8%2Fw5VVIMF9j52bTFePWISTLvwQ1RlEdNPNkrpbcamTsJFyBVi89%2BIpXArsVlhvDzK55Zvj%2Fvzh00GgdNrH%2BRog8Q%2BkGITE8bW%2FxRpQ30OdMZLjpLtp%2FNg5KVotHrx6Bet7vidKymiJQ9BbwCxTRGzBdAITr2rsKTMGZJzfvEKnIczsoiY91Zmc3hjGzUD9OxHGR%2BiRdN%2F2FbotOIVR48RE%2BoAdIGIEfKlw%3D%3D",
"file_name": "a9989aafcdf1482d8a0967a81b54b476_80a2d60394f15063bef4e44e1a4d83f3.png",
"content_type": "image/png", "exp": "2020-10-22 17:36:54.447484"
}}
So, I need to hit this upload url which is coming from the response.Below is my file where I am hitting this as soon as user drop the image ->
UploadImage.js
await this.props.getFirstMediaId(postdata).then(res => {
if (res.value && res.value.status === 1) {
let media_idArr = this.state.media_id.concat(res.value.media.media_id)
this.setState({ media_id: media_idArr, mediaUrl: res.value.upload })
customStatus = 'done';
}
}) //First call to the server to get Media_id and the cloud **upload URL**
***** FOR THIS API RESPONSE, PLEASE SEE THE ABOVE MAIN RESPONSE *****
const getUploadParams = () => {
console.log(this.state.mediaUrl, ' -->>> this.state.mediaUrl')
if (this.state.mediaUrl !== null) {
console.log(' in get upload param.')
return this.props.postImageToCloud(this.state.mediaUrl).then(res => {
console.log(res, '===>> here is cloud res.')
})
.catch(err => {
console.log(' here is error cloud -->>> ', err)
})
}
}
Below is the file where the method actually call API ->
service.js
export const getFirstMediaId = (data) => {
return {
type: GET_FIRST_LISTING_MEDIA,
async payload() {
let response = await callAxios.post(SUBMIT_LISTING_FIRST_MEDIA, data);
return objectPath.get(response, 'data', []);
}
}
}
export const postImageToCloud = (url) => {
return {
type: PUT_MEDIA_TO_CLOUD,
async payload() {
let response = await axios.put(url.upload_url, {}, {
headers: {
'Content-Type': `${url.content_type}`
}
})
return objectPath.get(response, 'data', []);
}
}
}
So, the first call is success and I got the above MAIN RESPONSE but as soon as it completes, I call the cloud PUT request and got this CORS error ->
Access to XMLHttpRequest at 'https://storage.googleapis.com/fnc-59aa2e6b-71552c9d-6441d628-951a8f6f/l.img/ori/a9989aafcdf1482d8a0967a81b54b476_80a2d60394f15063bef4e44e1a4d83f3.png?Expires=1603388214&GoogleAccessId=123456789-compute%40developer.gserviceaccount.com&Signature=UNt8nS3%2BJYiS4AuYdZ7Z2fvfDZ0fAKf8bSZbeRlHyhqxb5i6xjpqnqgR7JYp9Q3FgJItcYr%2BHDL90WiUpbMQi%2B4s0XNW683CaSoUChkRMjj1AvkH%2Be0u8%2Fw5VVIMF9j52bTFePWISTLvwQ1RlEdNPNkrpbcamTsJFyBVi89%2BIpXArsVlhvDzK55Zvj%2Fvzh00GgdNrH%2BRog8Q%2BkGITE8bW%2FxRpQ30OdMZLjpLtp%2FNg5KVotHrx6Bet7vidKymiJQ9BbwCxTRGzBdAITr2rsKTMGZJzfvEKnIczsoiY91Zmc3hjGzUD9OxHGR%2BiRdN%2F2FbotOIVR48RE%2BoAdIGIEfKlw%3D%3D' from origin 'http://localhost:8000' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Please suggest me anything for make it working.
Thanks.

Finally after lot of efforts I came to know that I have to pass the file in the body of PUT request, here ->
let response = await axios.put(url.upload_url, { **file here** }, {
headers: {
'Content-Type': `${url.content_type}`
}
})
But I tried passing the image file object simple the html file object using formData and passing as it is, still getting the same error. Then i started using
react-dropzone and converted the image file into the string buffer as one of the example in React-dropzone. I am going to paste here that example maybe it can help anyone. See below ->
import React, {useCallback} from 'react'
import {useDropzone} from 'react-dropzone'
function MyDropzone() {
const onDrop = useCallback((acceptedFiles) => {
acceptedFiles.forEach((file) => {
const reader = new FileReader()
reader.onabort = () => console.log('file reading was aborted')
reader.onerror = () => console.log('file reading has failed')
reader.onload = () => {
// Do whatever you want with the file contents
const binaryStr = reader.result
console.log(binaryStr)
*****PASS THIS (binaryStr) AS IN THE BODY OF PUT TO AXIOS****
}
reader.readAsArrayBuffer(file)
})
}, [])
const {getRootProps, getInputProps} = useDropzone({onDrop})
return (
<div {...getRootProps()}>
<input {...getInputProps()} />
<p>Drag 'n' drop some files here, or click to select files</p>
</div>
)
}
This is one of the Official examples of React-Dropzone, So I just pass that string buffer obj and finally It worked, no CORS issue nothing.

The Google Storage API does simply not accept requests initiated from a browser in another domain, so you won't be able to achieve this.
You should not call the API from a client but from your backend. Here is the list of the suggested libraries: https://cloud.google.com/storage/docs/reference/libraries?hl=fr
Note that JavaScript in a browser environment is not suggested (only Node.js is)

Saving data from JSON end point

I am trying to map over the returned json and save the id into profile/profiles. However it does not seem to be mapping over the the data correctly, id: ${ profile.id } this bit needs to be changed? Any help is much appreciated.
Is their a online tool that can help with me this?
API request:
// Grabs company data from the json url
private getProfiles() {
let config = {
headers: {'Authorization':'Bearer AQVVEqNXTWVYPpPYivKNWVO8jsTx2eveV3kBg'}
}
axios
.get("https://cors-anywhere.herokuapp.com/" + "https://api.linkedin.com/v2/me", config)
.then(response =>
response.data.map(profile => ({
id: `${ profile.id }`
}))
)
.then(profiles => {
this.setState({
profiles
});
})
// We can still use the `.catch()` method since axios is promise-based
.catch(error => this.setState({ error, isLoading: false }));
}
Json data returned:
{
"localizedLastName": "King",
"id": "fm0B3D6y3I",
"localizedFirstName": "Benn"
}
When I console log the response.data

If the only data returned from your endpoint is the JSON you posted, then you don't have an array to map over.
You have a single object.
I've never used the axios library before, but looking at the source code response.data should be the JSON-parsed responseText from the XHR request:
https://github.com/axios/axios/blob/4f189ec80ce01a0275d87d24463ef12b16715d9b/lib/adapters/xhr.js#L51-L53
https://github.com/axios/axios/blob/4f189ec80ce01a0275d87d24463ef12b16715d9b/lib/defaults.js#L61
And now I see that you have posted response.data and it matches what I'd expect.
With that in mind I'd suggest handling it like this:
// Grabs company data from the json url
private getProfiles() {
let config = {
headers: {'Authorization':'Bearer AQVVEqNXTWVYPpPYivKNWVO8jsTx2eveV3kBg'}
}
axios
.get("https://cors-anywhere.herokuapp.com/" + "https://api.linkedin.com/v2/me", config)
.then(response => ({
id: profile.id
}))
.then(profiles => {
this.setState({
profiles
});
})
// We can still use the `.catch()` method since axios is promise-based
.catch(error => this.setState({ error, isLoading: false }));
}
What you're getting back is a single profile though. If you need profiles to be an array you'll need to put the response in an array.

I don't get it, what you are trying to do. In the map you have a callback function, but as I see you wrote there an object. If you are wanting to rewrite the current profile's id then write this:
response.data.map(profile => ({
profile.id = `${ profile.id }`;
}))
But if you want it to make a variable then this:
response.data.map(profile => ({
let id = `${ profile.id }`;
}))

Download and upload image without saving to disk

Using Node.js, I am trying to get an image from a URL and upload that image to another service without saving image to disk. I have the following code that works when saving the file to disk and using fs to create a readablestream. But as I am doing this as a cron job on a read-only file system (webtask.io) I'd want to achieve the same result without saving the file to disk temporarily. Shouldn't that be possible?
request(image.Url)
.pipe(
fs
.createWriteStream(image.Id)
.on('finish', () => {
client.assets
.upload('image', fs.createReadStream(image.Id))
.then(imageAsset => {
resolve(imageAsset)
})
})
)
Do you have any suggestions of how to achieve this without saving the file to disk? The upload client will take the following
client.asset.upload(type: 'file' | image', body: File | Blob | Buffer | NodeStream, options = {}): Promise<AssetDocument>
Thanks!

How about passing the buffer down to the upload function? Since as per your statement it'll accept a buffer.
As a side note... This will keep it in memory for the duration of the method execution, so if you call this numerous times you might run out of resources.
request.get(url, function (res) {
var data = [];
res.on('data', function(chunk) {
data.push(chunk);
}).on('end', function() {
var buffer = Buffer.concat(data);
// Pass the buffer
client.asset.upload(type: 'buffer', body: buffer);
});
});

I tried some various libraries and it turns out that node-fetch provides a way to return a buffer. So this code works:
fetch(image.Url)
.then(res => res.buffer())
.then(buffer => client.assets
.upload('image', buffer, {filename: image.Id}))
.then(imageAsset => {
resolve(imageAsset)
})

well I know it has been a few years since the question was originally asked, but I have encountered this problem now, and since I didn't find an answer with a comprehensive example I made one myself.
i'm assuming that the file path is a valid URL and that the end of it is the file name, I need to pass an apikey to this API endpoint, and a successful upload sends me back a response with a token.
I'm using node-fetch and form-data as dependencies.
const fetch = require('node-fetch');
const FormData = require('form-data');
const secretKey = 'secretKey';
const downloadAndUploadFile = async (filePath) => {
const fileName = new URL(filePath).pathname.split("/").pop();
const endpoint = `the-upload-endpoint-url`;
const formData = new FormData();
let jsonResponse = null;
try {
const download = await fetch(filePath);
const buffer = await download.buffer();
if (!buffer) {
console.log('file not found', filePath);
return null;
}
formData.append('file', buffer, fileName);
const response = await fetch(endpoint, {
method: 'POST', body: formData, headers: {
...formData.getHeaders(),
"Authorization": `Bearer ${secretKey}`,
},
});
jsonResponse = await response.json();
} catch (error) {
console.log('error on file upload', error);
}
return jsonResponse ? jsonResponse.token : null;
}