Users generate files on my node js server by pressing a button on a web page.
The server then creates a .zip file.
I want to expose this zip file so that it can be downloaded automatically on to the users' client.
Once downloaded, I want the server to detect that the download is finished and delete the zip file.
1- How do I expose the file in node js? Should the system put it in public folder? That means it will be security risk and anyone can read it.How can I link to a file & make it downloadable which is not in public folder?
2- How do I detect that the download is finished? Should I run a cron job to delete the files without worrying about the download progress?
A few remarks that should help you:
If you are creating temporary files, a good practice is to create signed URLs. Those are URLS that contain specific token that is valid for a limited amount of time. Implementation is trivial - generate the file .zip and token, set timestamp preferably in the DB and construct signed link with token. If the file is not downloaded by client in a given amount of time, it is invalid.
Zip file should have unique name, preferably some random one (if it's not a problem, you could still use header called Content-Disposition to decide on its name during download). You store it in the TEMP dir inside your project.
After user clicks previously generated signed link with token that relates to that file, you start download (streaming). After streaming is complete (refer to NodeJS streams lib), you just delete the file.
And on the client side:
You create a button that leads to endpoint on server (triggers AJAX call or whatever). After clicking, you run mentioned above logic on server.
In return, user client gets generated link (leading to ANOTHER endpoint that handles those signed links only) that has to be followed to download the file
Using any kind of DOM manipulation, you create hidden <a/> tag with href leading to this link and then you trigger automatic click of this link in the JS code. Preferably, if you support new browsers, it's a good idea to add download attribute to it.
DO NOT:
put the file in the public folder. Create endpoint that will stream its contents to the client. Create just some temp dir for it.
run the CRON job for deleting the files. Or run it only if something fails, to clean old files. File should be deleted after it's downloaded (which you will know, as when your stream is closed, you will get a proper event).
IMPLEMENTATION SUGGESTIONS
Create two endpoints on the server (using Express or whatever framework for routing). One for requesting the file (that starts generation process) and another one for downloading it.
After the generation process is finished, store the .zip inside e.g. temp catalog and create token for it.
Store set of data like this in the DB for EVERY download:
zip file name
token (e.g. generated random hash)
timestamp of the generation
Pass the new link to the client (for the second endpoint that is used for download process). Client should initialise the download automatically without human interaction, as suggested above
When link is "clicked" by the code, your server receives another request on second endpoint and then:
checks if the token is still valid (e.g. for 30 seconds).
if not: 403 or 404
if yes: start streaming the data (create file stream and stream it to the client)
when streaming back, include proper headers with response, e.g. file name that client should see (this will obscure your internal file names from temp catalog), using Content-Disposition
After streaming is complete, delete the file
Create CRON job that will run e.g. once a day, ask the DB for ALL the files that have invalid tokens (expired ones) and will try to delete them, if they exist (but this should not be a common scenario, if you delete them properly when the streaming is finished).
Related
I am working on a nodejs service where I need to read a PDF document from a file.
At a high level, here is the workflow. First time a user requests a PDF, I generate it (using pdfkitjs) and save it to the server. Then, when the user request the same document again, I need to read it from the server and send it back.
Is there a way that I can use PDFDocument from pdfkitjs to read the contents from the file and create a corresponding PDFDocument? Almost all searches come up with how to pipe the PDF to a stream, but now how to read from a stream.
If "the stream" is your server response then you shouldn't have to do much more than add the appropriate headers to the response first.
I'm implementing a direct pdf file upload from client machine to Amazon S3 via REST API using only Go langangue, All works fine but one thing is worrying me...
Here is the steps
User click on pdf button
New browser tab is open there is in html page(which says generating
your report)
On background pdf file is uploading(in process) on s3. And API return s3
url to client.
Problem
how can I check if the URL is active yet or not. If it's a 404 it doesn't redirect… waits another N seconds. Once it's a 200, then I redirect to s3 url.
How can I achieve this on javascript ?
AWS S3 ensures GET after PUT consistency for new objects. From https://aws.amazon.com/s3/faqs/
"
Q: What data consistency model does Amazon S3 employ?
Amazon S3 buckets in all Regions provide read-after-write consistency for PUTS of new objects and eventual consistency for overwrite PUTS and DELETES.
"
This ensures that once the upload is done, your object will be reachable. Now, with JS you can issue an Ajax request only if you're on same domain or you enable CORS on your S3 bucket. This is explained here: http://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html and it will allow you to check your object on S3 being there.
Otherwise, you would need a server-side component to check if the object is uploaded and call that resource from JS on the same domain.
I'm trying to understand how tokens work in Firebase Storage.
Whenever my web app uploads an image to FS it adds a token to its public url. The problem is whenever you upload that same image file to another part of the web app, it seems like you don't get another file, but a different token for the file url that was already uploaded, thus rendering a 403 error for the previous registered image display.
Is there a way to solve this?
Example:
storageRef.put(picture.jpg);
uploadTask.snapshot.downloadURL
// returns something like https://firebasestorage.googleapis.com/v0/b/<your-app>/o/picture.jpg?alt=media&token=09cb2927-4706-4e36-95ae-2515c68b0d6e
That url is then displayed somewhere inside an img src.
<img src="https://firebasestorage.googleapis.com/v0/b/<your-app>/o/picture.jpg?alt=media&token=09cb2927-4706-4e36-95ae-2515c68b0d6e">
If the user repeats the process and uploads the same picture.jpg in another section of the app, instead of getting a brand new copy in Firebase Storage, the file is overwritten with an URL ending with a new token; say 12345.
So:
<img src="https://...picture.jpg?alt=media&token=12345"> // New upload renders fine
<img src="https://...picture.jpg?alt=media&token=09cb2927-4706..."> // But old upload breaks because of wrong url
Tokens are unique for a particular version of an upload. If you overwrite the file with new content, then a new token will be generated with a new unguessable url.
So in other words, tokens are unique for a particular blob -- they are not unique per storage location. We did this as an increased measure of security to ensure that developers and end users did not accidentally expose data they did not intend.
You can, however, translate the storage location ("gs://mybucket/myfile.png") into a download url using our js SDK. That way, you can pass around the gs uri if you wish and translate it to a full URL once you want to place it into an image.
See: https://firebase.google.com/docs/reference/js/firebase.storage.Reference.html#getDownloadURL
For public file upload: If you upload files in firebase functions you'll need to call makePublic() on the reference object in order to make it accessible without having a valid token.
My requirement - Single page application having a File upload and many other sliders/dropdowns that fire events to server whenever there is a change.
Ideal workflow for my application is
1) User uploads file using File Upload widget-> server should retain this file and use this for further requests from the same client
2) User changes sliders or dropdown values -> Client fires requests to the server. Server makes use of the file that was uploaded initially and applies some logic and returns a response that is rendered somewhere on the page.
User can simultaneously open the app in multiple tabs on web browser and each tab should be isolated, i.e., user can upload different files in different tabs and server should retain files and respond respectively.
Please let me know what technologies/frameworks are best fit for my use case.
Any links/example pages having the similar use case would also be very helpful.
I'd suggest this short of scheme:
When you upload the file, the server stores the file somewhere, coins a unique ID for that file and then stores that unique ID such that it can associate that unique ID with the recently uploaded file - probably using a simple database to associate the two in a persistent manner.
The unique ID is then returned to the client after the upload. The client window then keeps track of that unique ID in a javascript variable for that window.
Whenever that window wishes to apply some operation to that file, it sends an Ajax request to the server with the file's unique ID and parameters for the operation. For example, an ajax request to crop a file like this: http://x.com/api/crop?fileID=19482302&x=0&y=0&h=1024&w=2048.
When the server receives the Ajax request, it uses the unique ID to find which file it is associated with, applies the operation and returns the results.
The client receives the results and renders whatever it needs to render to show the results.
To keep unused files from building up on the server, the server probably needs to "age" away files that are no longer in use. For this, you can create some sort of expiration. When a file is accessed, it's recent access time is recorded. Then, you can have some periodic timer (say every hour) that cycles through the files and remove any who haven't been accessed recently (say in the last day or 6 hours or whatever time is appropriate for your app).
Here is what I am trying to do: I am making a custom text file containing a test. This test is unique to the user and well I don't want my server to stack all those text files.
Is there a way to use Ajax/JavaScript/jQuery to find if the user has already finished the download and if they have get a return value (1 if finished) so the response can be sent back to the PHP file and it can delete that file off the server (real-time like)?
I know there are plenty of ways to do this using PHP. Sort of like run clean up upon user log out and so on but I wanted to try using the method above since it can have many other applications that might be cool to use. I tried most search engines but they have nothing close to what I need.
Why do you need to store them in a file? Just use a PHP script or such that creates the test and outputs it directly to the user. That's how it will get deleted when the download is complete.
If it's important you may want the user to return to your server with the hash of the downloaded file. If the hash matches you know two things:
1. The user downloaded the file successfully
2. It's now ok to delete the file
Well it is very simple. I don't know how to make a PHP webpage send itself to the user other than to make the PHP make a text file and force send that to the user. This creates the problem of having so many text files in a temporary folder.
Now if the test required say 15 chapters each having a text or HTML format file then the script neatly zips all those files and sends them to the user. Again falling on the same problem if the user is finished downloading I am trying to get any type of script to delete the temporary zip or text file out of the temporary directory in somewhat real time.
If I could MD5 a downloaded file using JavaScript I welcome it and it would be a hack solution to the problem but how will the JavaScript gain access to the root access of the download folder of the user? There are security issues there if I am not mistaken. Hope this helps round the question a bit more.
I have a good solution for you here using the jQuery File Download plugin I created. It allows for you to get the behavior of performing an Ajax file download (not actually possible possible) complete with Success and Failure callbacks. In a nutshell you can just use the Success callback (that indicates the file download was successful) to perform an Ajax post back to the server to delete the file. Take a look at the blog post for an example on how to use the Success callback option or a demo which uses those callbacks in the context of showing modals to inform the user of what is going on.