I am trying to understand how stream work. I have a readableStream which i am pipeing to a writableStream.
However, i am getting this error in the console when i do so. Failed to execute 'pipeTo' on 'ReadableStream': Illegal invocation
I am making a call to fetch and pass the ReadableStream using PipeTo to a WritableStream
below is the relevant part of my code.
let writeStream = fs.createWriteStream(`exports/${exportParams.key}`, {flags:'a'});
writeStream.on('end', function(){
console.log('file downloaded');
});
const fetchData = await exportPopup.evaluate(async (fetchUrl, writeStream) => {
const stream = await fetch(fetchUrl);
stream.body.pipeTo(writeStream);
}, fetchUrl, writeStream);
Any help to fix this would be really great, Thanks.
Are you using node-fetch or are you on electron? I faced the same problem trying to download files from electron. It looks like streams from the browser aren't compatible with the streams used by fs. What I finally do is.
async function downloadFile(filename, url) {
const file = fs.createWriteStream(filename);
res = await fetch(url);
buffer = await res.arrayBuffer()
file.write(Buffer.from(buffer));
file.close();
}
I don't know if this is the most efficient way of doing this but it works.
Related
I have a a file name that comes from a file uploaded by the user
const fileName = "foo.doc"
I then went to pass that to an api that saves this to storage.
const mydata = await myApiToSaveFile(fileName);
// if the server throws an error even after my sanitization, I want to just pass in a UUID, which I know will pass. I don't care about the specific error, just that an error occurs, so is it wrong to use a try/catch for this?
try {
const mydata = await myApiToSaveFile(fileName);
} catch(e) {
const mydata = await myApiToSaveFile(generateUUID());
}
In my app I would like to download files (.pdf and .xls) from Firebase storage.
I can generate a download link using ref.getDownloadURL(), but that link causes the browser to open a new tab before downloading or opening the target file.
In order to avoid this behavior, I can download the the file in javascript using res = await fetch([the download URL]) and then await res.blob()
THIS WORKS FINE:
const res = await fetch([downloadURL]);
const blob = await res.blob();
const url = URL.createObjectURL(blob);
const anchor = document.createElement("a");
anchor.href = url;
anchor.download = [file_name]; // e.g. my_file.xls
anchor.click();
However, the documentation suggests that I can download the blob directly via the SDK using getBlob()
I have tried to download a list of the files in my storage bucket and to loop over these to get the blob as follows:
const storageRef = storage.ref().child("my_files");
const list = await storageRef.listAll();
const blob_array = await Promise.all(
list.items
.map(async (file) => {
const blob= await file.getBlob();
return {
name: file.name,
blob: blob,
};
})
);
however, I get an error:
TypeError: file.getBlob is not a function
The documentation for getBlob() states:
To use this functionality, you have to whitelist your app's origin in
your Cloud Storage bucket. See also
https://cloud.google.com/storage/docs/configuring-cors
In order to test this, I have followed the referenced documentation and, using gsutil, I have set the CORS policy for my storage bucket as follows:
[
{
"origin": ["*"],
"method": ["GET"],
"maxAgeSeconds": 3600
}
]
I thought that this would have the effect of 'whitelisting' my apps origin (per the getBlob() documentation), but instead I still get an error as follows:
TypeError: file.getBlob is not a function
How can I enable this getBlob() functionality?
The problem is exactly that stated by the error - you're trying to call getBlob on an object that doesn't have that method.
Firstly, take a look at the documentation for downloading files. It states that since SDK version 9.5 there is a global getBlob function. You're not using a new enough version of the SDK to get that function. You're using the old version 8, and Reference objects don't have a getBlob method (scan the API documentation v8 for Reference and you just won't see it).
If you want to use getBlob, you'll have to upgrade to the v9 SDK and rewrite all your Firebase code to conform to the new APIs. Then you will be able to use the new getBlob.
I think the problem is that you are trying to call a method on the StorageReference, when in fact you need to use the global getBlob function, with the StorageReference as the first argument in the function call.
Look at this example:
import { getBlob } from "#firebase/storage"
import type { StorageReference } from "firebase/storage"
export const downloadFile = (fromReference: StorageReference) => {
getBlob(fromReference)
.then((blob) => {
console.log('blob: ', blob)
})
.catch((error) => {
console.log('error downloading file: ', error)
})
}
class Fetch {
async getCurrent(input) {
const apiKey = "my_api_key";
// make request to url
const response = await fetch(
`https:/api.openweathermap.org/data/2.5/weather?q=${input}&appid=${apiKey}`
);
const data = await response.json();
console.log(data);
return data;
}
}
above is a snippet of code, can someone point me to the direction of the error?
I'm assuming you are running this code in Node.js, right?
If it's in the browser, you shouldn't have any issue since the fetch() API is implemented in the browser.
If you are running in Node.js you can use node-fetch which is a great implementation of fetch in Node.js.
Your code works fine. Problem can be in that how you call function.
You need to create an object, outside the class and call the function.
let f = new Fetch();
f.getCurrent("Kiev");
You can`t use variable name fetch because this name is reserved.
I'm trying to upload some images and get them back with the .getDownloadURL() method however it won't work. I works fine in a different part of the app when I update the user profile picture but won't work in the following example:
memory.files.forEach(async (file) => {
const response = await fetch(file);
const blob = await response.blob();
firebase
.storage()
.ref(`files/${file.slice(-10)}`)
.put(blob)
.then(() => {
console.log("file upladed");
})
.catch((e) => console.log('BIG ERROE'));
var pathReference = await firebase
.storage()
.ref(`files/${file.slice(-10)}`)
.getDownloadURL();
pathRefs.push(pathReference);
});
This is the error I get :
Possible Unhandled Promise Rejection (id: 8):
p {
"code_": "storage/object-not-found",
"message_": "Firebase Storage: Object 'files/img.jpg' does not exist.",
"name_": "FirebaseError",
"serverResponse_": "{
\"error\": {
\"code\": 404,
\"message\": \"Not Found. Could not get object\",
\"status\": \"GET_OBJECT\"
}
}",
}
It's not related to the security rules since it works when uploading and getting back the profile picture. I'm really struggling why I get the error since I'm following the exact same pattern as whit the profile pic upload. Please help!
The problem is that your code isn't waiting for the upload to complete before getting the download URL. Since you are using async/await, you can wait for the upload to finish using the promise returned by put(), then fetch the URL the same way you are doing now:
await firebase
.storage()
.ref(`files/${file.slice(-10)}`)
.put(blob);
console.log("file upladed");
var pathReference = await firebase
.storage()
.ref(`files/${file.slice(-10)}`)
.getDownloadURL();
pathRefs.push(pathReference);
If you need error handling, use try/catch around the whole thing. Don't mix then/catch callbacks with async/await and try/catch as you are now
Some Context
Im running a nodejs aplication that works with the Twit API.
Im using it to periodically tweet some images on tweeter, to generate the images im using processing (java API)
So im using the 'fs' lib of nodejs to execute the command:
processing-java --sketch="%cd%"\\processing\\sketch --run
Then im using a eventListener to know when processing is closed to load the generated image, to do this im using this JS promise:
File: generate-sketch-images.js
const {exec} = require('child_process');
const fs = require('fs');
module.exports = new Promise((resolve,reject) =>{
let cmd = `processing-java --sketch="%cd%"\\processing\\sketch --run`;
console.log('running processing');
exec(cmd).addListener('close',() => {
resolve(fs.readFileSync(
'processing/paints/paint.png',
{encoding:'base64'}
));
console.log('image received from processing');
});
})
Then i made this function:
File: bot.js
const generate_image = require('./utils/generate-sketch-images');
async function gerar_imagem(){
console.log('generating the image');
let img = await generate_image;
}
The problem
The 'gerar_imagens' function only works at first time
When i do something like setInterval(gerar_imagens,1000); it works just one time
I have already made a lot of console logs, and seems like the function call happens but the promise is ignored
someone knows whats happening??