I am using Azure storage and Nestjs. I am using Azure storage to store some static files. I can upload files by Nestjs storage SDK successfully.Now I need to upload a file with some custom blob metadata, I have go through the source code of Nestjs storage SDK, but seems there is no predefined way to do this. So is it possible to upload blobs with custom metadata? Or is there any workarounds?
Thanks!
I also have reviewed the source code of azureStorageService, it not provides useful methods. But the upload operation replys a storageUrl with SAS token, we could use it to make another HTTP request: set-blob-metaData to set blob metadata. This is my test code,name is the metadata in my test:
#Post('azure/upload')
#UseInterceptors(
AzureStorageFileInterceptor('file', null),
)
async UploadedFilesUsingInterceptor(
#UploadedFile()
file: UploadedFileMetadata,
) {
file = {
...file,
buffer : Buffer.from('file'),
originalname: 'somename.txt'
};
const storageUrl = await this.azureStorage.upload(file);
//call rest api to set metadata
await this.httpService.put(storageUrl + "&comp=metadata",null,{headers:{'x-ms-meta-name':'orginal name here'}})
.subscribe((response) => {
console.log(response.status);
});
{
Logger.log(storageUrl);
}}
}
Result:
Related
I have Angular running on the FrontEnd and Firebase Admin SDK for Node.js on the BackEnd.
What I want to achieve is to allow the user to select an image from his computer, using a simple <input> of type file. When I receive the user image which is of type File on the Angular side, I want to send this to my Node.js server and let him upload it to the Firebase Storage.
Here's how I'm sending the image to Node.js:
method(imageInput): void {
const image: File = imageInput.files[0];
const reader = new FileReader();
reader.addEventListener('load', (event: any) => {
const imageData = {
source: event.target.result,
file: image
}
this.myService.uploadImage(imageData.file).subscribe(
(res) => {
// image sent successfully
},
(err) => {
// error
})
});
reader.readAsDataURL(image);
}
So on the Node.js side I don't see a way to upload this image.
I'm trying:
admin.storage().bucket().upload(imageFromAngular, { --> Here's the problem
destination: "someDestination/",
contentType: "image/png",
metadata: {
contentType: "image/png"
}
}).then(() => {
// send successful response
}).catch(err => {
// send error response
});
The issue comes from the fact that the upload method only takes as a parameter the path to the image and the options. However in this case I can't pass the path to the image, rather I can pass the image itself. I read this - https://googleapis.dev/nodejs/storage/latest/ but I couldn't find anything that would suit my needs.
What would be the correct way to do this ?
Update:
Here's a more detailed explanation to the approach I took:
I'm using the arrayBuffer method of the image File inside my method. This method returns a promise of type ArrayBuffer. I get the value and send it to my Server.
The Server uses Buffer.from(ArrayBuffer, 'base64') to convert the data and then I can safely use the save API (https://googleapis.dev/nodejs/storage/latest/File.html#save).
To get the image later on I use download - (https://googleapis.dev/nodejs/storage/latest/File.html#download).
You can write a byte stream (or a Buffer) to Cloud Storage.
createWriteStream() API for streaming data to Cloud Storage: https://googleapis.dev/nodejs/storage/latest/File.html#createWriteStream
save() API for writing buffered data to Cloud Storage: https://googleapis.dev/nodejs/storage/latest/File.html#save
I am using Firebase Cloud Storage.
I know how to get the URL of a downloaded file (tags.txt) using Firebase SDK function running on a client javascript :
storage.ref('tags.txt').getDownloadURL().then((url) => {
console.log(url);
});
I want to get the downloadURL from Node.JS . I do know this function does not exists for Node.JS
However, In Google Cloud Platform you can get the following key/value pair :
firebaseStorageDownloadTokens : 0ea9a60a-7719-4c6b-9cb5-7fcf69d7c633, the value being the token you want. From this token I can easily build the downloadURL I need.
The path to get there is:
Cloud Storage / "Bucket name" / Bucket details / tags.txt / EDIT METADATA / Custom metadata.
I try this code to access this metadata :
async function getBucketMetadata() {
const bucketName = "gs://tags.appspot.com";
try {
// Get Bucket Metadata
let metadata = await admin.storage().bucket(bucketName).file('tags.txt').getMetadata();
console.log(metadata)
}
catch (err) {
console.log(err.message)
}
}
I got keys/values (not a real project though) info such as:
bucket:'tags.appspot.com'
contentType:'text/plain'
crc32c:'Y1Sdxw=='
etag:'CI1EETD18Co9vECEAE='
generation:'162694124484794756'
id:'tags-admin.appspot.com/tags.txt/162694431484794756'
kind:'storage#object'
md5Hash:'P1YSFER4xSf5p0/KWrdQWx1z1Lyg=='
mediaLink:'https://storage.googleapis.com/download/storage/v1/b/tags-admin.appspot.com/o/tags.txt?generation=162694443184794756&alt=media'
metageneration:'1'
name:'tags.txt'
selfLink:'https://www.googleapis.com/storage/v1/b/tags-admin.appspot.com/o/tags.txt'
size:'5211247'
storageClass:'STANDARD'
timeCreated:'2021-07-22T09:01:24.862Z'
timeStorageClassUpdated:'2021-07-22T09:01:24.862Z'
updated:'2021-07-22T09:01:24.862Z'
But nothing regarding the key/value pair I want : firebaseStorageDownloadTokens : 0ea9a60a-7719-4c6b-9cb5-7fcf69d7c633
If the key/value can be seen on Google Cloud Platform , I do believe the key/value is also accessible via some code.
Your help is appreciated.
I mixed up two projects. I re-tried it, and its work pretty nicely. Using this method I can retrieve the file token and build the file URL around it on the back-end. The front-end function getDownloadURL() is not longer required.
Thanks for your help anyway.
The code become:
async function getBucketMetadata() {
const bucketName = "gs://tags.appspot.com";
try {
// Get Bucket Metadata
let metadata = await admin.storage().bucket(bucketName).file('tags.txt').getMetadata();
console.log(metadata[0].metadata.firebaseStorageDownloadTokens)
}
catch (err) {
console.log(err.message)
}
}
What I am trying to do: I have a clientside (Browser, not node.js) JS App that uploads files to Blob Storage with the #azure/storage-blob package. To do so, it fetches a SAS Token from an Azure Function for a specific Blob. The Blob is created by the Function on each request and a SAS Token is returned to the client. Blob and SAS generation works and I can download it when the Blob Url with SAS is opened in a Browser.
Now what does not work is when I try to connect with the BLOB SAS (not Storage Account SAS or Connection String) to the Storage Account. The code below works when used with SAS from the whole Storage Account, but I do not want to give that much of permissions. I do not understand why a SAS token can be created for a specific Blob, if it is not possible to connect to it via Blob Service Client.
It is possible to create a Read-only SAS token for the whole storage account to get the connection up and running. But where would the Blob SAS go afterwards, so that the Blob can be accessed?
There is something fundamental that I seem to miss, so how can this be accomplished?
const url = `https://${storageName}.blob.core.windows.net/${sasToken}`; // sas for blob from the azure function
// const url = `https://${storageName}.blob.core.windows.net${containerSas}`; // sas from container
// const url = `https://${storageName}.blob.core.windows.net/${containerName}/${fileName}${sasToken}`; // does not work either
const blobService = new BlobServiceClient(url);
await this.setCors(blobService);
// get Container
const containerClient: ContainerClient = blobService.getContainerClient(containerName);
// get client
const blobClient = containerClient.getBlockBlobClient(fileName);
const exists = await blobClient.exists();
console.log('Exists', exists);
// set mimetype as determined from browser with file upload control
const options: BlockBlobParallelUploadOptions = { blobHTTPHeaders: { blobContentType: file.type } };
// upload file
await blobClient.uploadBrowserData(file, options);
EDIT:
The SAS token for the Blob:
?sv=2018-03-28&sr=b&sig=somesecret&se=2021-07-04T15%3A14%3A28Z&sp=racwl
The CORS method, though I can confirm that it works when I use the global storageaccount SAS:
private async setCors(blobService: BlobServiceClient): Promise<void> {
var props = await blobService.getProperties();
props.
cors =
[{
allowedOrigins: '*',
allowedMethods: '*',
allowedHeaders: '*',
exposedHeaders: '*',
maxAgeInSeconds: 3600
}]
;
}
Errors:
When using the Blob SAS, at the setCors/getProperties method: 403 (Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.)
When using the following service url, at the setCors/getProperties method: https://${storageName}.blob.core.windows.net/${containerName}/${fileName}${sasToken} => RestError: The resource doesn't support specified Http Verb.
When using a Storage Account SAS with only READ permissions, when accessing the blob (blob.exists()): 403 (This request is not authorized to perform this operation using this resource type.) (makes sense, but then I would like to use the Blob-specific SAS)
The reason you're running into this error is because you're trying to set CORS using a SAS token created for a blob (which is a Service SAS). CORS operation is a service level operation and for that you need to use a SAS token obtained at the account level. In other words, you will need to use an Account SAS. Any SAS token created for either blob container or blob is a Service SAS token.
Having said this, you really don't need to set CORS properties on the storage account in each request. This is something you can do at the time of account creation.
Once you remove call to setCors method from your code, it should work fine.
Considering you're working with just a Blob SAS, you can simplify your code considerably by creating an instance of BlockBlobClient directly using the SAS URL. For example, your code could be as simple as:
const url = `https://${storageName}.blob.core.windows.net/${containerName}/${fileName}${sasToken}`;
const blobClient = new BlockBlobClient(url);
//...do operations on blob using this blob client
What is a proper way to upload an external image via URL into strapi on backend-side?
I had tried to load image with node-fetch and processed it with buffer()/blob()/blob().stream() and then passed it into strapi.plugins['upload'].services.upload.upload(). Also tried to generate FormData in node.js and passed it into upload service but still didn't help.
How to convert image buffer from fetch into suitable type for upload service?
I used axios and it was on client, but you can try in on server too I think.
This worked for me:
Fetch an image and create File instance from it
async getImage(imageUrl, imageName) {
const response = await axios.get(imageUrl, { responseType: 'blob' });
const mimeType = response.headers['content-type'];
const imageFile = new File([response.data], imageName, { type: mimeType });
return imageFile;
}
GraphQL API query
{
query: `
mutation($files: [Upload!]!) {
multipleUpload(files: $files) {
id
}
}
`,
variables: {
files: [
// your files to upload
]
}
}
Then I called this mutation and it worked perfect.
Resources that I used to find this solution:
https://www.freecodecamp.org/news/how-to-manage-file-uploads-in-graphql-mutations-using-apollo-graphene-b48ed6a6498c/
Client side convert png file stream into file object
https://github.com/jaydenseric/graphql-multipart-request-spec
I am currently trying to download the file from the s3 bucket using a button from the front-end. How is it possible to do this? I don't have any idea on how to start this thing. I have tried researching and researching, but no luck -- all I have searched are about UPLOADING files to the s3 bucket but not DOWNLOADING files. Thanks in advance.
NOTE: I am applying it to ReactJS (Frontend) and NodeJS (Backend) and also, the file is uploaded using Webmerge
UPDATE: I am trying to generate a download link with this (Tried node even if I'm not a backend dev) (lol)
see images below
what I have tried so far
onClick function
If the file you are trying to download is not public then you have to create a signed url to get that file.
The solution is here Javascript to download a file from amazon s3 bucket?
for getting non public files, which revolves around creating a lambda function that will generate a signed url for you then use that url to download the file on button click
BUT if the file you are trying to download you is public then you don't need a signed url, you just need to know the path to the file, the urls are structured like: https://s3.amazonaws.com/ [file path]/[filename]
They is also aws amplify its created and maintain by AWS team.
Just follow Get started and downloading the file from your react app is simply as:
Storage.get('hello.png', {expires: 60})
.then(result => console.log(result))
.catch(err => console.log(err));
Here is my solution:
let downloadImage = url => {
let urlArray = url.split("/")
let bucket = urlArray[3]
let key = `${urlArray[4]}/${urlArray[5]}`
let s3 = new AWS.S3({ params: { Bucket: bucket }})
let params = {Bucket: bucket, Key: key}
s3.getObject(params, (err, data) => {
let blob=new Blob([data.Body], {type: data.ContentType});
let link=document.createElement('a');
link.href=window.URL.createObjectURL(blob);
link.download=url;
link.click();
})
}
The url in the argument refers to the url of the S3 file.
Just put this in the onClick method of your button. You will also need the AWS SDK