12. 08. 2009

Uploading files directly to Amazon S3 using FancyUpload

I'm happy to share some pieces of code I created to upload files directly to Amazon's S3 data storage, using the great flash based uploader FancyUpload.



Commonly, when using S3 as a data storage with rails, e.g. with paperclip, files are first uploaded to the webserver, potentially a post process follows and finally the files are uploaded to S3. The obvious advantage of directly uploading to S3 is that each file is only uploaded once (for files that don't require post processing). Another advantage is that the webserver is not touched at all saving its resources for serving pages.

For files that require post processing, like thumbnail generation, the uploaded files can be downloaded to the webserver from S3 and processed in a background process. This is especially efficient when using Amazon's EC2 for serving the web application, as data transfer between EC2 and S3 is very fast.

Based on the FancyUpload Attach a File sample I created the following rails helper:

def s3_uploader(options = {})
  filename = "#{RAILS_ROOT}/config/amazon_s3.yml"
  config = YAML.load_file(filename)

  bucket            = config[RAILS_ENV]['bucket_name']
  access_key_id     = config[RAILS_ENV]['access_key_id']
  secret_access_key = config[RAILS_ENV]['secret_access_key']

  key             = options[:key] || ''
  content_type    = options[:content_type] || ''
  acl             = options[:acl] || 'public-read'
  expiration_date = (options[:expiration_date] || 10.hours).from_now.utc.strftime('%Y-%m-%dT%H:%M:%S.000Z')
  max_filesize    = options[:max_filesize] || 2.megabyte

  policy = Base64.encode64(
    "{'expiration': '#{expiration_date}',
      'conditions': [
        {'bucket': '#{bucket}'},
        ['starts-with', '$key', '#{key}'],
        {'acl': '#{acl}'},
        {'success_action_status': '201'},
        ['content-length-range', 0, #{max_filesize}],
        ['starts-with', '$Filename', ''],
        ['starts-with', '#{content_type}', '']
      ]
    }").gsub(/\n|\r/, '')

  signature = Base64.encode64(
                OpenSSL::HMAC.digest(
                  OpenSSL::Digest::Digest.new('sha1'),
                  secret_access_key, policy)).gsub("\n","")

  out = ""
  out << %(
    <form action="https://#{bucket}.s3.amazonaws.com/" method="post" enctype="multipart/form-data" id="upload-form">
    <input type="hidden" name="key" value="#{key}/${filename}">
    <input type="hidden" name="AWSAccessKeyId" value="#{access_key_id}">
    <input type="hidden" name="acl" value="#{acl}">
    <input type="hidden" name="policy" value="#{policy}">
    <input type="hidden" name="signature" value="#{signature}">
    <input type="hidden" name="success_action_status" value="201">
    <input type="hidden" name="Content-Type" value="#{content_type}">
    </form>
  )

  out << "\n"
  out << link_to('Upload File(s)', '#',:id=> 'upload_link')
  out << "\n"
  out << content_tag(:ul, '', :id => 'uploader_file_list')
  out << "\n"

  out << javascript_tag("window.addEvent('domready', function() {

  /**
   * Uploader instance
   */
  var up = new FancyUpload3.Attach('uploader_file_list', '#upload_link', {
    path: 'http://your_app/javascripts/fancyupload/source/Swiff.Uploader.swf',
    url: 'https://#{bucket}.s3.amazonaws.com/',
    fieldName: 'file',
    data: $('upload-form').toQueryString(),

    fileSizeMax: 2000 * 1024 * 1024,

   // verbose: true,

    onSelectFail: function(files) {
      files.each(function(file) {
        new Element('li', {
          'class': 'file-invalid',
          events: {
            click: function() {
              this.destroy();
            }
          }
        }).adopt(
          new Element('span', {html: file.validationErrorMessage || file.validationError})
        ).inject(this.list, 'bottom');
      }, this);
    },

    onFileComplete: function(file) {
      if (file.response.code == 201 || file.response.code == 0){
        file.ui.element.highlight('#e6efc2');
        file.ui.element.children[2].setStyle('display','none');
        file.ui.element.children[3].setStyle('display','none');
      }
    },

    onFileError: function(file) {
      if (file.response.code != 201){
        file.ui.cancel.set('html', 'Retry').removeEvents().addEvent('click', function() {
          file.requeue();
          return false;
        });

        new Element('span', {
          html: file.errorMessage,
          'class': 'file-error'
        }).inject(file.ui.cancel, 'after'); }
    },

    onFileRequeue: function(file) {
      file.ui.element.getElement('.file-error').destroy();

      file.ui.cancel.set('html', 'Cancel').removeEvents().addEvent('click', function() {
        file.remove();
        return false;
      });

      this.start();
    }

    });

  });")

end

This helper generates a form with hidden fields that S3 requires for a file upload, including the base64 encoded policy and signature. This part of the code is inspired by the d2s3 plugin (http://github.com/mwilliams/d2s3/tree/master). The form fields are then passed to the uploader in the data argument. The uploader javascript is inluded in the helper - I had some trouble making it work when in a separate .js file. Please let me know if you get that working.

To make S3 not post an empty response after successful upload I needed to include the success_action_status in the form and policy and set it to 201. This is a flash issue and is described in the Amzon S3 documents (http://docs.amazonwebservices.com/AmazonS3/latest/). Also note that you need to have a public-readable crossdomain.xml in the bucket you upload to which allows your server's domain to access the bucket.

For some reason FancyUpload treats a 201 response as an error, so I changed the error handling in onFileError to exclude 201 "errors". Unfortunately this also has the effect that the onFileSuccess callback never gets called. I worked around this by handling successful uploads in the onFileComplete callback. Because onFileComplete is also called in case a file upload aborts with an error, i needed to filter for the 201 response here as well. If anyone finds a better way to handle this, please let me know. Edit: I just noticed that only when on a Mac the response code returned is 201. On windows (at least with Firefox) the response code returned after a successful upload is 0. So we also need to allow this case in the onFileComplete callback.

An example usage of the s3_uploader helper (from a haml view template):

# javascript is a helper that adds script tags for the given .js files to the head tag in the layout
- javascript( 'http://ajax.googleapis.com/ajax/libs/mootools/1.2.2/mootools', |
              'fancyupload/source/Fx.ProgressBar', |
              'fancyupload/source/Swiff.Uploader', |
              'fancyupload/source/FancyUpload3.Attach') |

= s3_uploader :key => 'test', :acl => 'public-read', :max_filesize => 800.megabytes

Note that the helper currently only supports setting the same content type for all files. If no content type is given, like in this example, it defaults to binary/octet-stream.

Here is the CSS I used to style the uploader. It is the slightly modified CSS from the Attach a File Example:

When a file upload is successul an ajax request can be triggered to initiate some action on the webserver, like creating a paperclip or similar ressource to store meta info about the uploaded file and create thumbnails or start another post process. I will write about how to do that in a follow up post.

Edit: Here you find a working example rails app: http://github.com/ncri/Rails-S3-Uploader-Example




Comments


Stacia wrote on Wed 28. Oct 2009:
I'm confused by this code. First off I can't find FancyUpload3 - the donwload site seems to be 2. Am I missing something? I don't have much more time to look at it right now to fix it, but just wanted to ask you about it.

Also the JS in this helper has an error around onFileCOmplete
Nico wrote on Wed 28. Oct 2009:
Stacia, the link I give on top of the post (http://digitarald.de/project/fancyupload/) links to Version 3. Click download on the right side and the page scrolls down to the download. I don't get any JS Error. Can you please give me some details?

Thanks,
Nico
Kia Kroas wrote on Fri 06. Nov 2009:
Are you sure this works? And with a progress bar? I'm going to try it out later today.

I believe the error Stacia mentioned was that the onFileComplete is missing the final closing } I just noticed it while retyping the code.
Nico wrote on Fri 06. Nov 2009:
Yes I am :) I'm using it...   Thanks for spotting the missing }. I will correct it.
Kia Kroas wrote on Fri 06. Nov 2009:
Also, there's no "or" in javascript, it's ||
Nico wrote on Fri 06. Nov 2009:
Ugh, thanks again, corrected. It's different in my code. Mixed ruby with js here in the post... ;-)
Kia Kroas wrote on Mon 09. Nov 2009:
You're welcome. Thanks for the guide. I did get it to work with some tweaks in my code.
Nico wrote on Mon 09. Nov 2009:
Great! That's good to hear. Any substantial tweaks you needed? Because for me this code works straight away.
stacia wrote on Tue 24. Nov 2009:
I still can't get this to work. I put everything in, but when I go to upload a file, the first time id displays the size and flashes no progress bar. After that, it will display the size with a frozen progress bar. I was looking in firebug to see if anything could be wrong but all I could see was a "GET none" request that's failing.
Unknown action
No action responded to none...
I'm not even sure wehre it's getting that none from.

I keep checking s3 but my stuff isn't being upload there and certainly the progress bar isn't working. Thanks for your help
Nico wrote on Wed 25. Nov 2009:
@stacia, and others that are interested: I added a working example rails app here: http://github.com/ncri/Rails-S3-Uploader-Example
K wrote on Wed 02. Dec 2009:
Hi Nico,

Thanks for this post, it's extremely useful. However, on a Mac under Safari I'm getting:
Error caused a send or load operation to fail (Error #2038)
And in Firefox, the progress bar just hangs with no activity, any thoughts about what I've done wrong? Or what could be causing this?
S wrote on Thu 25. Feb 2010:
Your uploader example works great! Thanks for this post!
To those who did not manage to make it work: make sure you added crossdomain.xml to your bucket. I could not make the progressbar change its position for few hours until i realized that I had not put the crossdomain.xml to the bucket.
Bernhard wrote on Tue 06. Apr 2010:
Hi! Cloned the app, added my S3 creds and the crossdomain.xml to the bucket and pushed it to heroku at bue001.heroku.com (user/passwd not changed). It just don't get it to work. Tested with FF 3.6.6 and Safari 4.0.5 on a MAC with Leopard. Any help would be very appreciated!! Also tips for debugging.
Nico wrote on Tue 06. Apr 2010:
Hi Bernhard,
did you create a folder named 'test' in your bucket? To help debugging comment in the //verbose = true in the UploadsHelper (line 67). Also make sure your crossdomain.xml is public readable.

Nico
Brian wrote on Tue 06. Apr 2010:
I am also receiving the same behavior using the sample app, although copying out the form, adding a file input field and submit button did work.
Nico wrote on Tue 06. Apr 2010:
Brian and Bernhard,
did you try setting verbose = true as suggested in my previous comment? You need to test with firefox and firebug then. What do you see in the firebug console when you try to upload a file? Did you put the crossdomain.xml in the root of the bucket? Don't put it in the test folder (which has to exist inside the bucket to make the exampe work).

Let me know if you still can't fix tghe problem. I have the example app running on heroku without any problems.
Bernhard wrote on Tue 06. Apr 2010:
Hello Nico! Thanks for the very fast feedback! I have the crossdomain.xml public and in the right folder (http://s3.amazonaws.com/bueupload/crossdomain.xml). For tests I have made the access to the test folder open (temporarly write and read for all) and put there a screenshot of the firebug log after I removed the comment in front of verbose (http://s3.amazonaws.com/bueupload/test/ff-screen.png). Nothing happens after firing the start event. Your solution would be perfect for my new app, since then uploads won't block access for other users of my Heroku app!!

Bernhard wrote on Tue 06. Apr 2010:
On thought: If you like I can test the upload with the app you depolyed at Heroku. Maybe it's a browser/platform issue?!
Nico wrote on Tue 06. Apr 2010:
Bernhard, I sent you a mail with the details of my testapp on heroku for you to test...
Bernhard wrote on Thu 08. Apr 2010:
WARNING! As you read in my postings I had technical problems. I used S3fm to manage the S3 service and the upload with fancyupload just wouldn't start. After switching to S3Hub (Mac), it immediatly worked. Using different software to check what was wrong before didn't show up the reason. Big thanks to Nico who helped me to get the application running!!
Yuval wrote on Thu 17. Jun 2010:
Hi

I am trying to use this (Fancy Uploader with Amazon S3) , but I get the same problem as others = the upload doesn't start, the progress bar appears but stays at 0%.
I did everything mentioned here, and it still does not work...any ideas?

Nico wrote on Thu 17. Jun 2010:
Hi Yuval,

did you get the example running: http://github.com/ncri/Rails-S3-Uploader-Example   ?
Aditya wrote on Sat 10. Jul 2010:
In your example you have code:
fileSizeMax: 2000 * 1024 * 1024,

Due to wich file size validation was not working.

Thanks a lot for this post... very useful..

I would also like to add that after installing ubuntu 10.04 LTS i am no more facing those hangs with firefox while uploading.

Write new Comment