Develop Reference

I keep getting a 403 from Firebase storge when trying to read image files

I'm having a hard time understanding the whole token part of Firebase uploads.
I want to simply upload use avatars, save them to the database and then read them at the client side.
const storageRef = firebase.storage().ref();
storageRef.child(`images/user-avatars/${user.uid}`).put(imageObj);
Then, in my cloud function, I grab the new url like this:
exports.writeFileToDatabase = functions.storage.object().onFinalize(object => {
const bucket = defaultStorage.bucket();
const path = object.name as string;
const file = bucket.file(path);
return file
.getSignedUrl({
action: "read",
expires: "03-17-2100"
})
.then(results => {
const url = results[0];
const silcedPath = path.split("/");
return db
.collection("venues")
.doc(silcedPath[1])
.set({ images: FieldValue.arrayUnion(url) }, { merge: true });
});
});
I've enabled IAM in the Google APIs platform, and have added Cloud functions service agent to the App Engine default service account.
I feel like the exact same configuration has worked before, butt now it sometimes doesn't even write the new url or I get 403 trying to read it. I can't find any explanations or errors to what I'm doing wrong.
EDIT:
Forgot to add this piece of code, but FieldValue is set at the top of the document as
const FieldValue = admin.firestore.FieldValue;
EDIT:
This the exact error I get Failed to load resource: the server responded with a status of 403 ()
And I just got it when I've tried to use this link, which has been generated automatically by the function above, as the source for an image component:
https://storage.googleapis.com/frothin-weirdos.appspot.com/images/user_avatars/yElCIVY4bAY5g5LnoOBhqN6mDNv2?GoogleAccessId=frothin-weirdos%40appspot.gserviceaccount.com&Expires=1742169600&Signature=qSqPuuY4c5xmdnpvfZh39Pw3Vyu2B%2FbGMD1rQwHDBUZTAnKwP11MaOFQt%2BTV53krkIgvJgQT0Xl3UUxkngmW9785fUri75SSPoBk0z4DKyZnEBLxgTGRE8MzmXadQ%2BHDJ3rSI8IkkoomdnANpLsPN9oySshZ1h4BfOBvAmK0hQ4Gge1glH7qhxFjVWfX3tovZoL8e2smhuCRXxDsZtJh0ihbIeZUEnX8lGic%2B9IT6y4OskS2ZlrZNjvM10hcEesoPdHsT4oCvfhCNbUcJcueRKfsWlDCd9m6qmf42WVOc7UI0nE0oEvysMutWY971GVRKTLwIXRnTLSNOr6fSvJE3Q%3D%3D

NodeJS download file from AWS S3 Bucket

I'm trying to make an endpoint in NodeJS/Express for downloading content from my AWS S3 Bucket.
It works well, I can download the file in the client side but I can also see the stream preview in the Network tab which is annoying...
QUESTION
I'm wondering if what I'm doing is correct and a good practice.
Also would like to know if it's normal to see the output stream in the Network tab.
How should I properly send I file from S3 to my client application using NodeJS/Express?
I'm pretty sure other websites requests don't let you preview the content with a: "Fail to load response data".
This is what I do in my NodeJS application to get the stream file from AWS S3:
download(fileId) {
const fileObjectStream = app.s3
.getObject({
Key: fileId
})
.createReadStream();
this.res.set("Content-Type", "application/octet-stream");
this.res.set(
"Content-Disposition",
'attachment; filename="' + fileId + '"'
);
fileObjectStream.pipe(this.res);
}
And in the client side I can see this:

I think the issue is with the header :
//this line will set proper header for file and make it downloadable in client's browser
res.attachment(key);
// this will execute download
s3.getObject(bucketParams)
.createReadStream()
.pipe(res);
So code should be like this (This is what I am doing it in my project handling file as res.attachment or res.json in case of error so client can display error to end user) :
router.route("/downloadFile").get((req, res) => {
const query = req.query; //param from client
const key = query.key;//param from client
const bucketName = query.bucket//param from client
var bucketParams = {
Bucket: bucketName,
Key: key
};
//I assume you are using AWS SDK
s3 = new AWS.S3({ apiVersion: "2006-03-01" });
s3.getObject(bucketParams, function(err, data) {
if (err) {
// cannot get file, err = AWS error response,
// return json to client
return res.json({
success: false,
error: err
});
} else {
res.attachment(key); //sets correct header (fixes your issue )
//if all is fine, bucket and file exist, it will return file to client
s3.getObject(bucketParams)
.createReadStream()
.pipe(res);
}
});
});

How to get all download url from firebase storage?

I'm a newbie and I am trying to get all download url from firebase storage and display it in a table or a list .

The download URL isn't provided in the snapshot you receive when your file is uploaded to Storage. It'd be nice if Firebase provided this.
To get the download URL you use the method getDownloadURL() on your file's storage reference. There are two types of storage references, firebase.storage.ref() and firebase.storage.refFromURL(. The former uses the path to the file, which is provided in the snapshot after your file is uploaded to Storage. The latter uses either a Google Cloud Storage URI or a HTTPS URL, neither of which is provided in the snapshot. If you want to get the download URL immediately after uploading a file, you'll want to use firebase.storage.ref() and the file's path, e.g.,
firebase.storage.ref('English_Videos/Walker_Climbs_a_Tree/Walker_Climbs_a_Tree_3.mp4');
Here's the full code for uploading a file to Storage and getting the download URL:
firebase.storage().ref($scope.languageVideos + "/" + $scope.movieOrTvShow + "/" + $scope.videoSource.name).put($scope.videoSource, metadata) // upload to Firebase Storage
.then(function(snapshot) {
console.log("Uploaded file to Storage.");
firebase.storage().ref(snapshot.ref.location.path).getDownloadURL() // get downloadURL
.then(function(url) {
var $scope.downloadURL = url;
})
.catch(function(error) {
console.log(error);
});
})
.catch(function(error) {
console.log(error);
});

How to use Google Drive API to download files with Javascript

I want to download files from google drive with javascript API. I have managed to authenticate and load list of files using gapi.client.drive.files request. However, I stuck at downloading those files.
My attempt to download the file:
var request = gapi.client.drive.files.get({
fileId:id,
alt:'media'
});
request.execute(function(resp){
console.log(resp);
});
I have these errors when trying to run the above:
(403) The user has not granted the app 336423653212 read access to the file 0B0UFTVo1BFVmeURrWnpDSloxQlE.
(400) Bad Request
I recognize that the files which aren't google drive file (google doc, google slide) return the 403 error.
I am new to this. Any help and answer is really appreciated.
Update 0
From Google Drive documentation about Handling API Error, here is part of the explanation for 400 errors
This can mean that a required field or parameter has not been provided, the
value supplied is invalid, or the combination of provided fields is
invalid.
This is because I have alt:'media' in my parameter object.
I tried gapi.client.drive.files.export, but it doesn't work either and it returns (403) Insufficient Permission although my Google Drive account has the edit permission for those files. Here is my code:
var request = gapi.client.drive.files.get({
fileId:element.id,
});
request.then(function(resp){
console.log(resp.result);
type = resp.result.mimeType;
id = resp.result.id;
var request = gapi.client.drive.files.export({
fileId:id,
mimeType:type
})
request.execute(function(resp){
console.log(resp);
});
});
Update 1
Based on abielita's answer, I have tried to make an authorized HTTP request but it doesn't download the file. It actually returns the file information in response and responseText attribute in the XMLHttpRequest object.
function test() {
var accessToken = gapi.auth.getToken().access_token;
var xhr = new XMLHttpRequest();
xhr.open("GET", "https://www.googleapis.com/drive/v3/files/"+'1A1RguZpYFLyO9qEs-EnnrpikIpzAbDcZs3Gcsc7Z4nE', true);
xhr.setRequestHeader('Authorization','Bearer '+accessToken);
xhr.onload = function(){
console.log(xhr);
}
xhr.send('alt=media');
}
______________________________________________________
I found out that I can actually retrieve URLs of all those files from the folder using files' webViewLink or webViewContent attributes.
A file which is from Google Drive type (Google Doc, Google Sheet,
etc...) will have webViewLink attribute. A webViewLink will open
the file in Google Drive.
A non Google Drive type file will have webContentLink. A
webContentLink will download the file.
My code:
var request = gapi.client.drive.files.list({
q:"'0Bz9_vPIAWUcSWWo0UHptQ005cnM' in parents", //folder ID
'fields': "files(id, name, webContentLink, webViewLink)"
});
request.execute(function(resp) {
console.log(resp);
}

Based from this documentation, if you're using alt=media, you need to make an authorized HTTP GET request to the file's resource URL and include the query parameter alt=media.
GET https://www.googleapis.com/drive/v3/files/0B9jNhSvVjoIVM3dKcGRKRmVIOVU?alt=media
Authorization: Bearer ya29.AHESVbXTUv5mHMo3RYfmS1YJonjzzdTOFZwvyOAUVhrs
Check here the examples of performing a file download with our Drive API client libraries.
String fileId = "0BwwA4oUTeiV1UVNwOHItT0xfa2M";
OutputStream outputStream = new ByteArrayOutputStream();
driveService.files().get(fileId)
.executeMediaAndDownloadTo(outputStream);
For the error (403) Insufficient Permission, maybe this is a problem with your access token, not with your project configuration.
The insufficient permissions error is returned when you have not requested the scopes you need when you retrieved your access token. You can check which scopes you have requested by passing your access_token to this endpoint: https://www.googleapis.com/oauth2/v1/tokeninfo?access_token=ACCESS_TOKEN
Check these links:
google plus api: "insufficientPermissions" error
Google drive Upload returns - (403) Insufficient Permission
Remember you are uploading to the service accounts google drive account. If you want to be able to see it from your own Google drive account you are going to have to do an insert of the permissions. to give yourself access

Phu, you were so close!
Thank you for sharing your method of using the webContentLink and webViewLink. I think that is best for most purposes. But in my app, I couldn't use viewContentLink because need to be able to enter the image into a canvas, and the image google provides is not CORS ready.
So here is a method
var fileId = '<your file id>';
var accessToken = gapi.auth2.getAuthInstance().currentUser.get().getAuthResponse().access_token;// or this: gapi.auth.getToken().access_token;
var xhr = new XMLHttpRequest();
xhr.open("GET", "https://www.googleapis.com/drive/v3/files/"+fileId+'?alt=media', true);
xhr.setRequestHeader('Authorization','Bearer '+accessToken);
xhr.responseType = 'arraybuffer'
xhr.onload = function(){
//base64ArrayBuffer from https://gist.github.com/jonleighton/958841
var base64 = 'data:image/png;base64,' + base64ArrayBuffer(xhr.response);
//do something with the base64 image here
}
xhr.send();
Notice that I set the response type to arraybuffer, and moved alt=media up to the xhr.open call. Also I grabbed a function that converts the array buffer to base64 from https://gist.github.com/jonleighton/958841.

I found out that I can actually retrieve URLs of all those files from the folder using files' webViewLink or webViewContent attributes. A file which is of Google Drive type (Google Doc, Google Sheet, etc...) will have webViewLink attribute and a non Google Drive type file will have webContentLink. The webViewLink will open the file in Google Drive and the webContentLink will download the file. My code:
var request = gapi.client.drive.files.list({
q:"'0Bz9_vPIAWUcSWWo0UHptQ005cnM' in parents", //folder ID
fields: "files(id, name, webContentLink, webViewLink)"
});
request.execute(function(resp) {
console.log(resp); //access to files in this variable
}

Task: download the file and create File object;
Environment: browser;
const URL = 'https://www.googleapis.com/drive/v3/files';
const FIELDS = 'name, mimeType, modifiedTime';
const getFile = async (fileId) => {
const { gapi: { auth, client: { drive: { files } } } } = window;
const { access_token: accessToken } = auth.getToken();
const fetchOptions = { headers: { Authorization: `Bearer ${accessToken}` } };
const {
result: { name, mimeType, modifiedTime }
} = await files.get({ fileId, fields: FIELDS });
const blob = await fetch(`${URL}/${fileId}?alt=media`, fetchOptions).then(res => res.blob());
const fileOptions = {
type: mimeType,
lastModified: new Date(modifiedTime).getTime(),
};
return new File([blob], name, fileOptions);
};

I was able to download using the files.get API:
var fileId = '<your file id>';
gapi.client.drive.files.get(
{fileId: fileId, alt: 'media'}
).then(function (response) {
// response.body has the file data
}, function (reason) {
alert(`Failed to get file: ${reason}`);
});

let url = https://drive.google.com/uc?id=${file_id}&export=download;
Make sure to pass the file_id in this link.
You can get the file id from the file you want to download by getlink --> general access. Make sure the file is public.

Amazon S3 Signature Does Not Match - AWS SDK Java

I have a play application that needs to upload files to S3. We are developing in scala and using the Java AWS SDK.
I'm having trouble trying to upload files, I keep getting 403 SignatureDoesNotMatch when using presigned urls. The url is being genereated using AWS Java SDK by the following code:
def generatePresignedPutRequest(filename: String) = {
val expiration = new java.util.Date();
var msec = expiration.getTime() + 1000 * 60 * 60; // Add 1 hour.
expiration.setTime(msec);
s3 match {
case Some(s3) => s3.generatePresignedUrl(bucketname, filename, expiration, HttpMethod.PUT).toString
case None => {
Logger.warn("S3 is not availiable. Cannot generate PUT request.")
"URL not availiable"
}
}
}
For the frontend code we followed ioncannon article.
The js function that uploads the file (the same as the one used in the article)
function uploadToS3(file, url)
{
var xhr = createCORSRequest('PUT', url);
if (!xhr)
{
setProgress(0, 'CORS not supported');
}
else
{
xhr.onload = function()
{
if(xhr.status == 200)
{
setProgress(100, 'Upload completed.');
}
else
{
setProgress(0, 'Upload error: ' + xhr.status);
}
};
xhr.onerror = function()
{
setProgress(0, 'XHR error.');
};
xhr.upload.onprogress = function(e)
{
if (e.lengthComputable)
{
var percentLoaded = Math.round((e.loaded / e.total) * 100);
setProgress(percentLoaded, percentLoaded == 100 ? 'Finalizing.' : 'Uploading.');
}
};
xhr.setRequestHeader('Content-Type', 'image/png');
xhr.setRequestHeader('x-amz-acl', 'authenticated-read');
xhr.send(file);
}
}
The server's response is
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<StringToSignBytes>50 55 bla bla bla...</StringToSignBytes>
<RequestId>F7A8F1659DE5909C</RequestId>
<HostId>q+r+2T5K6mWHLKTZw0R9/jm22LyIfZFBTY8GEDznfmJwRxvaVJwPiu/hzUfuJWbW</HostId>
<StringToSign>PUT
image/png
1387565829
x-amz-acl:authenticated-read
/mybucketname/icons/f5430c16-32da-4315-837f-39a6cf9f47a1</StringToSign>
<AWSAccessKeyId>myaccesskey</AWSAccessKeyId></Error>
I have configured CORS, double checked aws credentials and tried changing request headers. I always get the same result.
Why is Amazon telling me that signatures dont match?

Doubt the OP still has a problem with this, but for anyone else who runs into this, here is the answer:
When making a signed request to S3, AWS checks to make sure that the signature exactly matches the HTTP Header information the browser sent. This is unfortunately required reading: http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html
However in the code above this is not actually the case, the Javascript is sending:
xhr.setRequestHeader('Content-Type', 'image/png');
xhr.setRequestHeader('x-amz-acl', 'authenticated-read');
But in the Java/Scala, s3.generatePresignedUrl is being called without passing in either of them. So the resulting signature is actually telling S3 to reject anything with a Content-Type or x-ams-acl header set. Oops (I fell for it too).
I've seen browsers send Content-Types automatically, so even if they're not explicitly added to the header they could still be coming into S3. So the question is, how do we add Content-Type and x-amz-acl headers into the signature?
There are several overloaded generatePresignedUrl functions in the AWS SDK, but only one of them allows us to pass in anything else besides the bucket-name, filename, expiration-date and http-method.
The solution is:
Create a GeneratePresignedUrlRequest object, with your bucket and filename.
Call setExpiration, setContentType, etc, to set all of your header info on it.
Pass that into s3.generatePresignedUrl as the only parameter.
Here's the proper function definition of GeneratePresignedUrlRequest to use:
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#generatePresignedUrl(com.amazonaws.services.s3.model.GeneratePresignedUrlRequest)
The function's code on the AWS GitHub repo was also helpful for me to see how to code up the solution. Hope this helps.

I faced a similar issue and setting the config signatureVersion: 'v4' helped solve it in my case -
In JavaScript:
var s3 = new AWS.S3({
signatureVersion: 'v4'
});
Adapted from https://github.com/aws/aws-sdk-js/issues/902#issuecomment-184872976

I just encountered this problem using the NodeJs AWS SDK.
It was due to using credentials that were valid, but without sufficient permissions.
Changing to my admin key fixed this with no code changes!

I had the same issue, but removing content-type works fine. Hereby sharing the complete code.
public class GeneratePresignedUrlAndUploadObject {
private static final String BUCKET_NAME = "<YOUR_AWS_BUCKET_NAME>";
private static final String OBJECT_KEY = "<YOUR_AWS_KEY>";
private static final String AWS_ACCESS_KEY = "<YOUR_AWS_ACCESS_KEY>";
private static final String AWS_SECRET_KEY = "<YOUR_AWS_SECRET_KEY>";
public static void main(String[] args) throws IOException {
BasicAWSCredentials awsCreds = new BasicAWSCredentials(AWS_ACCESS_KEY, AWS_SECRET_KEY);
AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withRegion(Regions.US_EAST_1)
.withCredentials(new AWSStaticCredentialsProvider(awsCreds)).build();
try {
System.out.println("Generating pre-signed URL.");
java.util.Date expiration = new java.util.Date();
long milliSeconds = expiration.getTime();
milliSeconds += 1000 * 60 * 60;
expiration.setTime(milliSeconds);
GeneratePresignedUrlRequest generatePresignedUrlRequest =
new GeneratePresignedUrlRequest(BUCKET_NAME, OBJECT_KEY);
generatePresignedUrlRequest.setMethod(HttpMethod.PUT);
generatePresignedUrlRequest.setExpiration(expiration);
URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);
UploadObject(url);
System.out.println("Pre-Signed URL = " + url.toString());
} catch (AmazonServiceException exception) {
System.out.println("Caught an AmazonServiceException, " +
"which means your request made it " +
"to Amazon S3, but was rejected with an error response " +
"for some reason.");
System.out.println("Error Message: " + exception.getMessage());
System.out.println("HTTP Code: " + exception.getStatusCode());
System.out.println("AWS Error Code:" + exception.getErrorCode());
System.out.println("Error Type: " + exception.getErrorType());
System.out.println("Request ID: " + exception.getRequestId());
} catch (AmazonClientException ace) {
System.out.println("Caught an AmazonClientException, " +
"which means the client encountered " +
"an internal error while trying to communicate" +
" with S3, " +
"such as not being able to access the network.");
System.out.println("Error Message: " + ace.getMessage());
}
}
public static void UploadObject(URL url) throws IOException
{
HttpURLConnection connection=(HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("PUT");
OutputStreamWriter out = new OutputStreamWriter(
connection.getOutputStream());
out.write("This text uploaded as object.");
out.close();
int responseCode = connection.getResponseCode();
System.out.println("Service returned response code " + responseCode);
}
}

Got a problem, the mime type on windows was setting the fileType to empty string and it didn't work. Just handle empty strings and add some file type.

I faced with SignatureDoesNotMatch error using the Java AWS SDK. In my case, SignatureDoesNotMatch error occurred after upgraded maven dependencies without changes in my code (so credentials are correct and were not changed). After upgrading dependency org.apache.httpcomponents:httpclient from version 4.5.6 to 4.5.7 (actually it was upgrade of Spring Boot from 2.1.2 to 2.1.3, and there bom has specified httpclient version), code became throw exceptions while doing some AWS SDK S3 requests like AmazonS3.getObject.
After digging into the root cause, I found that httpclient library did breaking changes with normalized URI, that affected Java AWS SDK S3. Please take a look for opened GitHub ticket org.apache.httpcomponents:httpclient:4.5.7 breaks fetching S3 objects for more details.

If your access keys and secret keys are good but it is saying "SignatureDoesNotMAtch", check your secret key, it probably has any of some special charaters, e.g +/ - / *
Go to aws and generate another access key, where the the secret key does not have those. Then try again :)