I'm looking for a way to select and upload multiple files to Amazon S3, something in the vein of Uploadify, but with the following constraints :
No flash or HTML 5 - but AJAX and iframe tricks are allowed.
Multiple selection must happen in a single dialog.
Files must be sent directly to Amazon (there is no intermediary server to handle them).
Also, Amazon S3 does not allow uploading multiple files in a single request, so this means every file will have to be sent with a distinct request to a distinct URL, so I need to specify what those URLs will be.
Are any components around that might do this, or any known techniques I could leverage to build my own? Thank you.
Plain HTML file uploads are limited to one file at a time.
Javascript is restricted from accessing the user's file system, and must depend on the HTML file upload mechanism.
Consequently, we are left only with the complex options such as Flash, Java applets, or browser plugins. If they are not acceptable, you will not be able to support multiple file uploads.
Related
I have an application that allows users to upload relatively large video files, which are stored on S3. We previously used flow-php-server to chunk uploads over multiple requests that are then assembled and stored on S3. Unfortunately, this method no longer works, as we recently upgraded our server architecture to a load balanced environment and chunks are being split across the multiple servers behind our load balancer.
What is the solution to this problem? I am under the impression that if we split file uploads over multiple requests, we can make no guarantees about which server it hits so uploads will fail. Does this mean I'll have to settle with single request uploads and deal with browser single request file size limits? Is there another way around this?
I'm not sure if the solution requires configuring the server/load balancer to somehow direct uploads for the same file to the same server, or if there's a different method I can implement on the front/back ends to accommodate this.
I have a pre signed url that allows you to PUT documents objects into S3.
http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadObjectPreSignedURLDotNetSDK.html
How do I put a file from the browser to S3 using javascript? I am using angular, but I am open to using any javascript library.
I believe I could POST to my server, and then PUT the object on the amazon server, but I would prefer to do it from the browser.
I have changed the CORS settings on S3 to allow PUTs.
I was planning to use angular file upload, but it is hard coded to POST not PUT.
https://github.com/danialfarid/angular-file-upload
Amazon has a guide (here) that describes how to POST-upload a file into your S3 bucket. It relies on an input <form> signed with your AWS private key. You can specify restrictions on the target directory, as well as file-size restrictions.
It's a bit annoying to use, because you have to duplicate most of the fields in both the <form> and the signed policy, but it seems to work.
After the POST, S3 will redirect the browser to a URL you specify in the form (with parameters specifying the name of the uploaded file, etc.). This isn't ideal for Angular web sites, which tend to be "applications" rather than a set of discrete pages, but you could probably work with it.
On my Angular site, I did the POST in Javascript using $http.post() and passed all the appropriate form data. Unfortunately, I was always getting "cancelled" errors, even though the uploads were actually successful. In my case, I just double-checked by downloading the file with $http.get() and comparing it to the original data... but this was only a viable solution because my files were only a couple of KB.
I am working on a file upload system which will store individual parts of large files on more than one server. So the distribution of a 1GB file will look something like this:
Server 1: 0-128MB
Server 2: 128MB-256MB
Server 2: 256MB-384MB
... etc
The intention of this is to allow for redundancy (each part will exist on more than one server), security (no one server has access to the entire file), and cost (bandwidth expenses are distributed).
I am curious if anyone has an opinion on how I might be able to "trick" web browsers into downloading the various parts all in one link.
What I had in mind was something like:
Browser is linked to Server 1, which provides a content-size of the full file
Once 128MB is served, Server 1 will intentionally close the connection
Hopefully, the browser will try to restart the download, requesting Server 1
Server 1 provides a 3XX redirect to Server 2
Browser continues downloading from Server 2
I don't know for certain that my example works, as I haven't tested it yet. I was curious if there were other solutions someone might have?
I'd like to make the whole process as easy as possible (ideally requiring no work beyond a simple download). I don't want the users to have to use another program (ie: cat'ing the files together). I'd also like to not use a proxy server, since it would incur extra bandwidth costs.
As far as I'm aware, there is no javascript solution for writing a file, if there was one, that would be great.
AFAIK this is not possible by using the HTTP protocol. You can probably use a custom browser extension but it would depend on the browser. Another alternative is to create a Java applet that would download the file from different servers. The applet can accept the URLs to the different servers as parameters.
To save the generated file:
https://stackoverflow.com/a/4551467/329062
That solution stores the file in memory though, so it won't work with very large files.
You can download the partial files into a JS variable using JSONP. That will also let you get around the same-origin policy.
Javascripts security model will only allow you to access data from the same origin where the Javascript came from - i.e. not multiple servers.
If you are going to have the file bits on multiple servers, you will need the user to load the web page, fetch the bit and then finally stick the bits together in the correct order. If you can manage to get all your users to do this (correctly), you are a better man than I.
It's possible to do in modern browsers over standard HTTP.
You can use XHR2 with CORS to download file chunks as ArrayBuffers and then merge them using Blob constructor and use createObjectURL to send merged file to the user.
However, I suspect that browsers will store these objects in RAM, so it's probably a bad idea to use it for large files.
What is the best practice for coordinating access to files in node.js?
I'm trying to write an http based file uploader for very large files (10sGB) that is resumable. I'm trying to figure out what the best approach is to handle two people trying to upload the same file at the same time... I'm also trying to think ahead to the possibility where more than one copy of the node.js http server is running behind a load balancer, which means catching duplicate uploads can't rely on just the code itself.
In python, for example, you can create a file by passing the correct flags to the open() call to force an atomic create. Not sure if the default node.js open new file is atomic.
Another option I thought of, but don't really want to pursue, is using a database with an async driver that supports atomic transactions to track this state...
In order to know if multiple users are uploading the same file, you will have to identify the files somehow. Hashing is best for this. First, hash the entire file on the client side to identify it. Tell the server the hash of the file, if there is already a file on the server with the same hash, then the file has already been uploaded or is currently being uploaded.
Since this is an http file server, you will likely want users to upload files from a browser. You can get the contents of a file with a browser using the File Reader API. Unfortunately as of now this isn't widely supported. You might have to use something like flash to to get it to work in other browsers.
As you stream the file into memory with the file reader, you will want to break it into chunks and hash the chunks. Then send the server all of the file's hashed chunks. It's important that you break the file into chunks and hash those individual chunks instead of the contents of the entire file because otherwise the client could send one hash and upload an entire different file.
After the hashes are received and compared to other files' hashes and it turns out someone else is currently uploading the same file, the server then decides which user gets to upload which chunks of the file. The server then tells the uploading clients what chunks it wants from them, and the clients upload their corresponding chunks.
As each chunk is finished uploading, it is rehashed on the server and compared with the original array of hashes to verify that the user is uploading the correct file.
I found this on HackerNews under a response to someone complaining about some of the same things in node.js. I'll put it here for completeness. This allows me to at least lock some file writes in node.js like I wanted to.
IsaacSchlueter 4 hours ago | link
You can open a file with O_EXCL if you pass in the open flags as a
number. (You can find them on require("constants"), and they need to
be binary-OR'ed together.) This isn't documented. It should be. It
should probably also be exposed in a cleaner way. Most of the rest of
what you describe is APIs that need to be polished and refined a bit.
The boundaries are well defined at this point, though. We probably
won't add another builtin module at this point, or dramatically expand
what any of them can do. (I don't consider seek() dramatic, it's just
tricky to get right given JavaScript's annoying Number problems.)
first of all my question: Is it possible to pass file names from a running Flash application, which only purpose is to enable multiple-file-selection, to a JavaScript application which handles upload of all files to the server?
I have examined various Flash upload solutions (like SWFUpload, Uploadify, etc.) and none of them meets my needs. I want an easy to implement solution (like Uploadify) which also lets me specify various parts of the HTTP request.
The reason I need this is because my upload form uses session cookies (for user authentication) and an CSRF token both passed to the server when uploading files.
Is it technically possible to pass filenames (+ paths) to a JavaScript application which then handles the upload?
Thank you,
FMD
I'm sorry but no, its not possible to pass the filenames to JavaScript from Flash, however, you could pass the session ID to Flash.
If you are using PHP (I'm not saying you are, your server side language might have similar functions), you could reestablish the session:
session_id($_POST['ses']);
session_start();
The reason why you can't pass the filenames to JavaScript, (or set it by script in the first place) is that it would be a major security issue, consider the following:
var uploader = document.getElementById('id_of_input_type_file');
uploader.value = 'c:\Users\Administrator\Documents\commonBankKeyFile.ebjkeystore';
document.getElementById('formId').submit();
...And there you go, I just got your bank credentials just by you visiting my page, no Phishing needed.