I'm implementing a direct pdf file upload from client machine to Amazon S3 via REST API using only Go langangue, All works fine but one thing is worrying me...
Here is the steps
User click on pdf button
New browser tab is open there is in html page(which says generating
your report)
On background pdf file is uploading(in process) on s3. And API return s3
url to client.
Problem
how can I check if the URL is active yet or not. If it's a 404 it doesn't redirect… waits another N seconds. Once it's a 200, then I redirect to s3 url.
How can I achieve this on javascript ?
AWS S3 ensures GET after PUT consistency for new objects. From https://aws.amazon.com/s3/faqs/
"
Q: What data consistency model does Amazon S3 employ?
Amazon S3 buckets in all Regions provide read-after-write consistency for PUTS of new objects and eventual consistency for overwrite PUTS and DELETES.
"
This ensures that once the upload is done, your object will be reachable. Now, with JS you can issue an Ajax request only if you're on same domain or you enable CORS on your S3 bucket. This is explained here: http://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html and it will allow you to check your object on S3 being there.
Otherwise, you would need a server-side component to check if the object is uploaded and call that resource from JS on the same domain.
Related
In my app, my customers can create small widgets with text fields and text. They can customize the look and feel through JS and CSS. I upload the JS and CSS in my S3 bucket and use Cloudfront for CDN.
Once the widget is created, they can embed the widget on their website using embed code.
In the embed code, I have used a 1x1 pixel image which is used to send request to php so I can increase the visit counter.
public function track(Request $request) {
// increase stored number here
header('Content-Type: image/gif');
return base64_decode('R0lGODlhAQABAJAAAP8AAAAAACH5BAUQAAAALAAAAAABAAEAAAICBAEAOw==');
}
My server is getting overload because of the visit counter. I want to now track the number of visits to embedded widgets for each customer by using S3 and Cloudfront access count.
I searched and found - Getting the download count of a specific S3 object, but it is for download.
How can I get the count of times when S3 object is accessed?
Use cloudtrail to and parse the logs. List of actions trackable by cloudtrail on s3
You can use URL shortener service to redirect to your file (like goo.gl, bit.ly), they will track number of clicks,views etc, and it wont even overload your server.
Hope it helps.
Users generate files on my node js server by pressing a button on a web page.
The server then creates a .zip file.
I want to expose this zip file so that it can be downloaded automatically on to the users' client.
Once downloaded, I want the server to detect that the download is finished and delete the zip file.
1- How do I expose the file in node js? Should the system put it in public folder? That means it will be security risk and anyone can read it.How can I link to a file & make it downloadable which is not in public folder?
2- How do I detect that the download is finished? Should I run a cron job to delete the files without worrying about the download progress?
A few remarks that should help you:
If you are creating temporary files, a good practice is to create signed URLs. Those are URLS that contain specific token that is valid for a limited amount of time. Implementation is trivial - generate the file .zip and token, set timestamp preferably in the DB and construct signed link with token. If the file is not downloaded by client in a given amount of time, it is invalid.
Zip file should have unique name, preferably some random one (if it's not a problem, you could still use header called Content-Disposition to decide on its name during download). You store it in the TEMP dir inside your project.
After user clicks previously generated signed link with token that relates to that file, you start download (streaming). After streaming is complete (refer to NodeJS streams lib), you just delete the file.
And on the client side:
You create a button that leads to endpoint on server (triggers AJAX call or whatever). After clicking, you run mentioned above logic on server.
In return, user client gets generated link (leading to ANOTHER endpoint that handles those signed links only) that has to be followed to download the file
Using any kind of DOM manipulation, you create hidden <a/> tag with href leading to this link and then you trigger automatic click of this link in the JS code. Preferably, if you support new browsers, it's a good idea to add download attribute to it.
DO NOT:
put the file in the public folder. Create endpoint that will stream its contents to the client. Create just some temp dir for it.
run the CRON job for deleting the files. Or run it only if something fails, to clean old files. File should be deleted after it's downloaded (which you will know, as when your stream is closed, you will get a proper event).
IMPLEMENTATION SUGGESTIONS
Create two endpoints on the server (using Express or whatever framework for routing). One for requesting the file (that starts generation process) and another one for downloading it.
After the generation process is finished, store the .zip inside e.g. temp catalog and create token for it.
Store set of data like this in the DB for EVERY download:
zip file name
token (e.g. generated random hash)
timestamp of the generation
Pass the new link to the client (for the second endpoint that is used for download process). Client should initialise the download automatically without human interaction, as suggested above
When link is "clicked" by the code, your server receives another request on second endpoint and then:
checks if the token is still valid (e.g. for 30 seconds).
if not: 403 or 404
if yes: start streaming the data (create file stream and stream it to the client)
when streaming back, include proper headers with response, e.g. file name that client should see (this will obscure your internal file names from temp catalog), using Content-Disposition
After streaming is complete, delete the file
Create CRON job that will run e.g. once a day, ask the DB for ALL the files that have invalid tokens (expired ones) and will try to delete them, if they exist (but this should not be a common scenario, if you delete them properly when the streaming is finished).
I'm trying to understand how tokens work in Firebase Storage.
Whenever my web app uploads an image to FS it adds a token to its public url. The problem is whenever you upload that same image file to another part of the web app, it seems like you don't get another file, but a different token for the file url that was already uploaded, thus rendering a 403 error for the previous registered image display.
Is there a way to solve this?
Example:
storageRef.put(picture.jpg);
uploadTask.snapshot.downloadURL
// returns something like https://firebasestorage.googleapis.com/v0/b/<your-app>/o/picture.jpg?alt=media&token=09cb2927-4706-4e36-95ae-2515c68b0d6e
That url is then displayed somewhere inside an img src.
<img src="https://firebasestorage.googleapis.com/v0/b/<your-app>/o/picture.jpg?alt=media&token=09cb2927-4706-4e36-95ae-2515c68b0d6e">
If the user repeats the process and uploads the same picture.jpg in another section of the app, instead of getting a brand new copy in Firebase Storage, the file is overwritten with an URL ending with a new token; say 12345.
So:
<img src="https://...picture.jpg?alt=media&token=12345"> // New upload renders fine
<img src="https://...picture.jpg?alt=media&token=09cb2927-4706..."> // But old upload breaks because of wrong url
Tokens are unique for a particular version of an upload. If you overwrite the file with new content, then a new token will be generated with a new unguessable url.
So in other words, tokens are unique for a particular blob -- they are not unique per storage location. We did this as an increased measure of security to ensure that developers and end users did not accidentally expose data they did not intend.
You can, however, translate the storage location ("gs://mybucket/myfile.png") into a download url using our js SDK. That way, you can pass around the gs uri if you wish and translate it to a full URL once you want to place it into an image.
See: https://firebase.google.com/docs/reference/js/firebase.storage.Reference.html#getDownloadURL
For public file upload: If you upload files in firebase functions you'll need to call makePublic() on the reference object in order to make it accessible without having a valid token.
I am developing a new website where users can upload files to an Amazon S3 bucket. After evaluating different upload libraries for jQuery I finally chose Dropzone JS.
I was able to integrate Dropzone into my application in order to upload files directly to an Amazon S3 bucket. Everything is working fine with the upload.
However I am having troubles reading the response from Amazon using jQuery. In particular, I'd like to get the Location header that comes as a response from Amazon. This location header has the information I need to process the uploaded file, but I am unable to get it using Dropzone. Anyone can advice how to get the XHR response headers? Checking the code I don't think this is possible, seems we can only get the response text but not the headers.
see How to get read data from response header in jquery/javascript for obtaining the response info.
Assuming you are using the AWS POST operation http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html
I suspect that the URI it returns is the one with amazon's domain: https://s3.amazonaws.com/Bucket/Object
if you are using a "web" bucket and want to use your custom domain you will have to figure that out for yourself. You already have the bucket name since you provided it in the call.
Another wrinkle could be the permissions of the file after upload. Be sure to set a policy on the paths for the uploads appropriately.
According to the creator of dropzone, The XHR object is stored in the file itself as file.xhr. So if you want to access its parameters, you would have to do console.log(file.xhr."what you want to access")
I suggest you console.log(file.xhr) to see its contents first. It would give you an idea of the values that are available.
However, the Response headers are "unsafe" and can not be viewed except you add a CORS policy to your bucket that marks them as safe.
So if you want to access the Location header for example, you would need to add
<ExposeHeader>location</ExposeHeader>
to your CORS policy.
Then you can now access it like so
console.log(file.xhr.getResponseHeader("Location"));
Sorry to resurrect an old thread
I'm using Amazon S3 with a simple project, when the user uploads a file, I first use an XMLHTTPRequest to send the details to my database, then trigger the form POST that uploads the file straight to S3.
What I can't quite work out though is how to tell when the upload is done so I can update the display properly (I'm cheating at the moment and just updating the display once it's saved in the database, you can barely tell for small uploads, but obviously a problem in the long run.)
The S3 server appears to "cancel navigation" once the upload is complete, so the page doesn't reload - are there any events that are fired that I can use to detect when the upload is complete?
EDIT: I'm using the form code from here http://s3.amazonaws.com/doc/s3-example-code/post/post_sample.html just added an onsubmit to the form that fires the XMLHTTPRequest to my own server before the form gets posted to amazon.
Amazon S3 will redirect to the url specified in the success_action_redirect field or if that's not specified, it returns an empty document with the status code specified in success_action_status.
On completion of the POST, the user is redirected to the location that you specified in the success_action_redirect field. If Amazon S3 cannot interpret the URL, it ignores the success_action_redirect field.
If success_action_redirect is not specified, Amazon S3 returns the empty document type specified in the success_action_status field.
If the POST fails, Amazon S3 displays an error and does not provide a redirect.
See the AWS docs for more details.
Amazon S3 won't trigger any kind of event, but your XMLHTTPRequest code should be able to trigger a callback when S3 returns an HTTP 200 status code.
If S3 isn't returning a successful response code, it's quite possible that S3 doesn't know that your upload is complete. Are you sending a Content-Length header as part of your request?