I have a mobile phone application where I allow users to take photographs and persist the picture file to S3. I am using a Node.js server to do the persistence to S3.
On the node.js server side I have exposed an API which does the following:
Constructs an in memory buffer of file data sent by the client (mobile device which took the photograph).
Persists this data in local file system.
Creates a thumbnail from original picture file in local fs (Thumbnail creation is another requirement I have). Persists thumbnail to local fs. I use easyimg for thumbnail creation.
Persists original pic and thumbnail from local fs to S3
I have few questions in the process above:
The above process works for me, however I want to avoid steps 1. and 2. (the in memory buffer creation and persistence to local file system) as node is just acting as a conduit for the data to be streamed to S3. Can someone give me pointers to code to do the same?
Is easyimg a good way to create thumbnail? What are the advantages and disadvantages of using this module? Is there something better in terms of performance and ease of use?
I am ok with some delay in thumbnail generation. So the thumbnail file need not be persisted along with the original file in S3. Does this help in any way?
Related
I have an application that allows users to upload relatively large video files, which are stored on S3. We previously used flow-php-server to chunk uploads over multiple requests that are then assembled and stored on S3. Unfortunately, this method no longer works, as we recently upgraded our server architecture to a load balanced environment and chunks are being split across the multiple servers behind our load balancer.
What is the solution to this problem? I am under the impression that if we split file uploads over multiple requests, we can make no guarantees about which server it hits so uploads will fail. Does this mean I'll have to settle with single request uploads and deal with browser single request file size limits? Is there another way around this?
I'm not sure if the solution requires configuring the server/load balancer to somehow direct uploads for the same file to the same server, or if there's a different method I can implement on the front/back ends to accommodate this.
How can I allow the client to upload an image in the browser, and then I upload it to Amazon S3? I have been looking around a lot and have found no resources explaining how to do this.
Are there any tutorials that I could follow?
Are there any libraries that I should use for this?
I am using AngularJS on the frontend and Node.js on the backend.
In short, look for two different tutorials. One for uploading from a client to a server, one for uploading from a server to S3.
StackOverflow discourages linking to specific tutorials, but there are lots of them out there, so it shouldn't be too tricky to track down.
For the client-to-server, you'll want to do a basic HTML form upload up to the server, then snag the data. You can temporarily write it to your file system (if you're on Linux, the /tmp directory is a good place to stash it).
After that, just upload from your server to S3. Amazon itself has some good documentation on that. The s3 package for Node also has good examples: https://www.npmjs.com/package/s3
It's also possible to go straight from the browser to S3, which may be better depending on your use case. Check this out: http://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/s3-example-photo-album.html
You're going to need the AWS SDK for node. Then they have a pretty comprehensive developper guide. You may have to read up on credential management, too.
the procedure would be as follows
user uploads the image from browser to your server (I'd recomend plain form upload, unless you feel ok with uploading via ajax)
then your server uses the SDK to save to S3
you display back info to the user (link to the image, upload status ?).
you could also use pre-signed posts but that seems more advanced and I haven't seen info on it for Node.
I am using the standard python app engine environment and currently looking at how one goes about uploading multiple large media files to Google Cloud Storage (Public Readable) using App Engine or the Client directly (preferred).
I currently send a bunch of smaller images (max 20 - between 30 and 100k on average), at the same time directly via a POST to the server. These images are provided by the client and put in my projects default bucket. I handle the requests images using a separate thread and write them one at a time to the cloud and then associate them with an ndb object. This is all fine and dandy when the images are small and do not cause the request to run out of memory or invoke a DeadlineExceededError.
But what is the best approach for large image files of 20mb+ a piece or video files of up to 1GB in size? Are there efficient ways to do this from the client directly, would this be possible via the Json api ,a resumable upload, for example? If so, are there any clear examples of how to do this purely in javascript on the client? I have looked at the docs but it's not intuitively obvious at least to me.
I have been looking at the possibilities for a day or two but nothing hits you with a clear linear description or approach. I notice in the Google Docs there is a way using PHP to upload via a POST direct from the client...https://cloud.google.com/appengine/docs/php/googlestorage/user_upload...Is this just relevant to using PHP on app engine or is there an equivalent to createUploadUrl for python or javascript?
Anyway, I'll keep exploring but any pointers would be greatly appreciated.
Cheers
Try BlobStore with Cloud Storage or the Image Service
I want to stream data from my Dropbox to webpage in real time, but don't know how to do it.
It's usually a bad idea because Dropbox can throttle speed, stop sharing file when using from many locations.
You can install Dropbox to your server and sync some folder with your Dropbox:
https://www.dropbox.com/install
And to stream from your local folder is easier task.
But if you really need to get files from Dropbox real-time, you can use their API. They've got libraries for many languages. For example this one is for PHP, also tutorial there:
https://www.dropbox.com/developers-v1/core/start/php
I'm looking for a front-end solution for uploading files to amazon s3 (that is, not passing them through my server.
The solution I have found is
https://code.google.com/p/swfupload/
It might do the job, but it requier flash and this is the first sentence of the project description is:
SWFUpload has not been under active development for several years.
Here are my desired features, though none of them are nessesary
No plugins
Built with Amazon S3 in mind, handling buckets authentification etc.
Some way to see file upload-progress
Support from IE8+
kgu87 is correct, this article pretty much explains the entire process to upload files directly to S3 without passing them trough your own server.
You can also check out the AWS docs related to this on:
http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingHTTPPOST.html
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html
If you're looking for an upload tool that supports HTML5 uploads directly to S3, check out Plupload
They have a great article that explains how to set it up:
https://github.com/moxiecode/plupload/wiki/Upload-to-Amazon-S3
The documentation describes a PHP service that's used to generate a policy and signature (both are required for S3 to accept your download) but you can use any language to generate those. Also, in certain use cases, you can just generate a one-time policy with a very high expiration time and hard code it into your upload form.
You could start by using this tutorial as a baseline , if you are asking about uploading from your web app -
http://aws.amazon.com/articles/1434
You could use this tool: http://aws.amazon.com/customerapps/Amazon-S3/Consumers/2069 free, and works in most browsers; doesn't require a server.
S3 Browser is a free client interface for users of the Amazon S3 service.
It provides an easy to use GUI to manage your storage buckets and allows you to:
- Browse, create, delete Amazon S3 buckets
- Upload and download files to and from Amazon S3
- Create public URLs to share the files.
- Keep your files backed up on a multiple data centers.
- Set Access Control on buckets and files.
S3 Browser is free for non-commercial use.