Develop Reference

Getting Google Cloud Platform custom metadata " firebaseStorageDownloadToken" programmatically

I am using Firebase Cloud Storage.
I know how to get the URL of a downloaded file (tags.txt) using Firebase SDK function running on a client javascript :
storage.ref('tags.txt').getDownloadURL().then((url) => {
console.log(url);
});
I want to get the downloadURL from Node.JS . I do know this function does not exists for Node.JS
However, In Google Cloud Platform you can get the following key/value pair :
firebaseStorageDownloadTokens : 0ea9a60a-7719-4c6b-9cb5-7fcf69d7c633, the value being the token you want. From this token I can easily build the downloadURL I need.
The path to get there is:
Cloud Storage / "Bucket name" / Bucket details / tags.txt / EDIT METADATA / Custom metadata.
I try this code to access this metadata :
async function getBucketMetadata() {
const bucketName = "gs://tags.appspot.com";
try {
// Get Bucket Metadata
let metadata = await admin.storage().bucket(bucketName).file('tags.txt').getMetadata();
console.log(metadata)
}
catch (err) {
console.log(err.message)
}
}
I got keys/values (not a real project though) info such as:
bucket:'tags.appspot.com'
contentType:'text/plain'
crc32c:'Y1Sdxw=='
etag:'CI1EETD18Co9vECEAE='
generation:'162694124484794756'
id:'tags-admin.appspot.com/tags.txt/162694431484794756'
kind:'storage#object'
md5Hash:'P1YSFER4xSf5p0/KWrdQWx1z1Lyg=='
mediaLink:'https://storage.googleapis.com/download/storage/v1/b/tags-admin.appspot.com/o/tags.txt?generation=162694443184794756&alt=media'
metageneration:'1'
name:'tags.txt'
selfLink:'https://www.googleapis.com/storage/v1/b/tags-admin.appspot.com/o/tags.txt'
size:'5211247'
storageClass:'STANDARD'
timeCreated:'2021-07-22T09:01:24.862Z'
timeStorageClassUpdated:'2021-07-22T09:01:24.862Z'
updated:'2021-07-22T09:01:24.862Z'
But nothing regarding the key/value pair I want : firebaseStorageDownloadTokens : 0ea9a60a-7719-4c6b-9cb5-7fcf69d7c633
If the key/value can be seen on Google Cloud Platform , I do believe the key/value is also accessible via some code.
Your help is appreciated.

I mixed up two projects. I re-tried it, and its work pretty nicely. Using this method I can retrieve the file token and build the file URL around it on the back-end. The front-end function getDownloadURL() is not longer required.
Thanks for your help anyway.
The code become:
async function getBucketMetadata() {
const bucketName = "gs://tags.appspot.com";
try {
// Get Bucket Metadata
let metadata = await admin.storage().bucket(bucketName).file('tags.txt').getMetadata();
console.log(metadata[0].metadata.firebaseStorageDownloadTokens)
}
catch (err) {
console.log(err.message)
}
}

Amazon S3 Fetching Error: "NoSuchKey" However Key Does Exist

I am trying to fetch a S3 Object using AWS Storage
fetchAvatar = async () => {
try {
const imageData = await Storage.get("public/public/us-east-2:/1597842961073/family.jpg")
console.log(imageData)
} catch (err) {
console.log('error fetching avatar: ')
console.log(err)
}
}
When I click on the link that the imageData provides I get NoSuchKey error, however it does exist
I've made sure that the image is public and accessible by everyone, so there shouldn't be any authentication problems. I've also looked at similar issue to this and I made sure there is no spaces or a lot of special keys in my image keys. I am kind of stumped on this...

So I figured out the reason, and it has to do something with AWS S3 Management. For some reason that every time I upload an image, the folder will reset and become privet. When I remake the folders and image public manually I am able to render the image properly...So i guess it is more of AWS issue or bug that they need to fix I think

I suggest to use javascript aws sdk, you can get an object from the bucket like below:
var params = {
Bucket: "your-bucket-name",
Key: "yourFileName.jpg"
};
s3.getObject(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data);
});
UPDATE:
You can define your region when you create a s3 instance, like:
const s3 = new S3({
region: 'eu-central-1',
});

NodeJS - Return a single status code from an async function using Array.protoype.map() to save multiple files in different locations in a database

I am a little stumped on how to handle this the best way possible. I've decided to rewrite this controller, and I need to (at least I think) make use of promise.all() here.
Premise:
In this application, the Admin user must be able to bulk upload a bunch of .pdf's at once that are for multiple users. The .pdf's adhere to a specific naming convention that my backend upload controller by using a regEx, pulls out a first and last name. These .pdf's are auto-generated in a program, that always names them exactly the same, so there is no human error in misspelling names.
Each call to the database and an AWS S3 Bucket is made within an Array.prototype.map() a function that is looping through and uploading a file to an S3 bucket, and then it takes the Key name of the file returned from s3.upload() and saves that Key to a user model in Mongo DB as a reference to their file(s) within the S3 Bucket.
Example Code:
This is what I currently have (that does work somewhat). This is the block of code responsible for what I described above. employeeFiles is created further up in the controller and contains an array of objects that each have a file and id property. The file name destructuring and user matching happen further up in the controller as well, and the employeeFiles array is a result of that. The id property contains the mongo _id of the employee, and the file property contains the file to be saved. This all works perfectly, and I don't think that code is needed for context here. fileType is a variable available within the scope of the controller:
const employeeFileUploadToDb = () => {
employeeFiles.map((employee, i) => {
const { file, id } = employee;
const params = {
Bucket: S3_BUCKET_NAME,
Body: file.buffer,
Key: `${filetype}/${file.originalname}`
};
s3.upload(params, (err, data) => {
if (err) {
next(err);
}
if (data) {
//Save reference to Employee model
let dataObj = {
key: data.key,
fileName: file.originalname,
date: Date.now()
};
Employee.findOneAndUpdate(
{ _id: id },
{ $push: { [`${filetype}`]: dataObj } }
)
.then(resp => res.send(200))
.catch(err => next(err));
}
});
});
};
I am making use of next() to handle any errors within the s3.upload() and findOneAndUpdate() functions (I do realize findOneAndUpdate() is deprecated) moving forward. My idea here is that if there is an error with one of the functions, next() will send it to my error handler middleware and keep going, versus ending the process and halting all of it.
Inside of every iteration of s3.upload(), I make a call to my database so that I can save the reference to the file uploaded to the S3 Bucket. Inside of a then() method of Employee.findOneAndUpdate(), I return a (200) response to let my client know everything has been uploaded to S3 and saved in my DB. So on each iteration of this map() function, I am returning a 200. If I have 10 files, I am returning 200 10 times.
I feel that I can convert this into an async function, and make use of a promise.all() to return a single status code upon completion. Returning that many status codes seem a bit crazy to me. But I am not too sure how to approach this while using a map() function to loop and make an async call on every iteration.
Hope this makes sense, and thank you in advance for looking at this!

I would split it up into a 2-step process. Upload in bulk and then save to mongo if it all worked out.
const employeeFileUploadToDb = () => {
const uploadFiles = files => files.map((employee, i) => new Promise((resolve, reject) => {
//...
s3.upload(params, (err, data) => {
if (err) {
return reject(err);
}
resolve(data);
})
});
});
Promise.all(uploadData(employeeFiles)).then((err, data) => {
// Handle saving to mongo
})
};

error uploading to cloud storage using a cloud function

I am trying to upload files to google cloud storage using a cloud function which is triggered by HTTP. However when the cloud function sends the file to be uploaded I often (although not always) get the following error
ERROR uploading to storage: { ApiError: Anonymous caller does not have storage.objects.create access to bucket_name/folder/test.jpg.
I am not sure why this error occurs - and why only some of the time
Here is the code:
const storage = require('#google-cloud/storage')();
function uploadToStorage(filepath, folder, filename) {
const options = {
destination: bucket.file(`${folder}/${filename}`),
public: false,
resumable: false
};
storage
.bucket(BUCKET_NAME)
.upload(filepath, options)
.then(function () {
console.log(`${filename} uploaded to ${BUCKET_NAME}`);
})
.catch((err) => {
console.error('ERROR uploading to storage: ', err);
});
}
Thanks

I had the same error after adding a return statement at the end of my function that performed file deletes on storage objects. This is what I was doing:
Make a database call to get some data
Once that request comes back, delete some files out of cloud storage (GCS)
The code structurally looked like this:
deleteStuffOutStorage() {
admin.firestore().doc(`My-doc-ref`).get()
.then(snapshot => {
// Do the deleting here {Interacting with GCS}
return deleteFile(snapshot.data().path); // Deletes file
})
.then(success => {
// Worked
})
.catch(error => {
// Error = ApiError: Anonymous caller does not have storage.objects...
})
return; // This statement was creating the problems
}
When I removed the return statement, I no longer got the error. I thought in my case it may have something to do with firebase-admin object instance getting deallocated and re-allocated between asynchronous operations (steps 1 and 2 above), or at least its GCS auth token?
All FCF instances should have access to GCS via a service account that is auto-generated. You can confirm this in the GCP console : https://console.cloud.google.com/iam-admin/serviceaccounts/
From the code snippet you posted I can't see anything that would cause the same issue I was getting, but maybe have a think about any time-based events that could cause this behaviour. That may explain the inconsistent behaviour you elude to.
Hope that's some sort of help.

NeDB not loading or storing to file

I cannot get the simplest example of NeDB to run properly. My code only works in-memory, persistence to file keeps failing without any error messages.
The error callbacks for the loaddatabase and insert events always pass a null reference as error, so no information there. Oddly it seems no one else has this issue, so I guess I'm missing something here. All help is much appreciated.
Here is the code:
var Datastore = require('nedb'), db = new Datastore({ filename: 'test.db' });
db.loadDatabase(function (err) {
alert(err); // err is null, with the autoload flag no error is thrown either
});
var doc = { hello: 'world'};
db.insert(doc, function (err, newDoc) {
alert(err); // err is null here as well. Doc will be in the memory storage but no persisted to file
});

Although this question is pretty old, I'd like to share my experience for anyone facing a similar issue.
NeDB API does not allow JSON input. You have to put in a javascript object. When you use JSON input, no error is returned and nothing will be persisted.
'null' is returned as error in callback to signal that no problem occurred. When saving the first JSON document it is indexed with 'undefined' key, because NeDB calls 'key = obj[fieldname[0]]' which returns 'undefined', when the obj is just a (JSON) string. No error is returned unfortunately. Inserting a second document will cause a unique constraint violation error in the callback as the key 'undefined' has already been taken. Anyhow, nothing will be persisted.
Try
var Datastore = require('nedb'), db = new Datastore({ filename: 'test.db' });
db.loadDatabase(function (error) {
if (error) {
console.log('FATAL: local database could not be loaded. Caused by: ' + error);
throw error;
}
console.log('INFO: local database loaded successfully.');
});
// creating the object with new, just to make it clear.
// var doc = {hello: 'world'}; should work too.
function myDoc(greeting)
{
this.hello=greeting;
}
var doc = new myDoc('world');
db.insert(doc, function (error, newDoc) {
if (error) {
console.log('ERROR: saving document: ' + JSON.stringify(doc) + '. Caused by: ' + error);
throw error;
}
console.log('INFO: successfully saved document: ' + JSON.stringify(newDoc));
});
Maybe it helps someone. :)

This question is quite old but since I had very similar problem I thought that I'll write my resolution for anyone facing similar issues.
In my case I was writing Electron app using electron-webpack as an application builder. It turns out that NeDB loaded by Webpack was running in browser mode without access to file system.
To get it working I had to change import statement from:
import DataStore from 'nedb';
to this:
const DataStore = require('nedb');
Also I had to add NeDB to Webpack configuration as external module (in package.json):
"electronWebpack": {
"externals": {
"nedb": "commonjs nedb"
}
}
I have found this resolution on NeDB github page: https://github.com/louischatriot/nedb/issues/329

All I had to do to fix this was delete the .db file and let the program make one for me by running it one more time.
The other thing I did that could have fixed it was making sure my package.json had all the required information. this can be easily done with a quick "npm init" in the terminal.