25. 08. 2009

FancyUpload Amazon S3 Uploader with Paperclip

In the previous (and first) post on this Blog I wrote about how to to upload files directly to Amazon S3 using FancyUpload. This post describes how to extend the uploader so that for each file successfully uploaded to S3 a rails model with a paperclip attachment (the uploaded file) is generated. This is useful for keeping track of the uploaded files from within a rails application and for applying post processes like thumbnail generation. I chose the paperclip plugin because I find it provides the easiest way for adding attachments to a rails model.



Conveniently, paperclip already has a S3 storage module built in. However it assumes that files are first uploaded to the webserver and then to S3. So to make paperclip work with files that are already uploaded to S3 I needed to slightly modify the S3 module in paperclip.

Normally, Paperclip sets up a file attachment when assigning an uploaded file to the paperclip attribute. In this process file_name, file_size and content_type attributess are assigned and potentially the uploaded file is post processed. Because in case of uploading directly to S3 we dont have a file uploaded to the server that can be assigned to the paperclip attribute we need to find a different way of setting up the attachment object.

Creating a Paperclip Attachment for the Uploaded S3 File

First of all we specify the attachment, just like we do with a normal Paperclip model where the S3 upload is handled by paperclip:

has_attached_file :attachment,
                 :styles => { :thumb => "60x60" },
                 :storage => :s3,
                 :s3_credentials => "#{RAILS_ROOT}/config/amazon_s3.yml",
                 :path => ":style/:filename",
                 :bucket => S3_CONFIG['bucket_name']

The only thing noteworthy here is that we cannot use the :id interpolation in the attachment path as we don't have it when uploading the file to S3. So in this case I just set the path to :style/:filename. This tells paperclip that e.g. original uploads are stored in bucket_name/original/file_name and thumbs in bucket_name/thumb/file_name.

To create the paperclip attachment we simply need to assign the three paperclip attributes file_name, file_size and content_type and potentially initiate a post process. The assignment of file_name, file_size is straight forward as fancy upload conveniently provides us with those attributes. The file's content_type can be inferred from its extension. For images a thumbnail generation post process is needed. I decided to use Delayed Job (DJ) to do this task in the background, avoiding long running http requests. Here is the code that sets the file's content type and starts the post process in case the attachment is an image:

def init_from_s3_upload
  self.attachment_content_type = file_extension_content_type(self.attachment_file_name)
  acl_obj = self.attachment.s3_object.acl
  if acl_obj.grants.find { |g| g.to_s =~ /READ to AllUsers/ }
    self.acl = 'public-read'
  else
    self.acl = 'private'
  end  
end

def after_create
  self.send_later(:post_process)
end

def post_process
  self.attachment.reprocess!
end

before_post_process :image?

def image?
  return (attachment.content_type =~ /^image.*/) ? true : false
end

def file_extension_content_type filename
  types = MIME::Types.type_for(filename)
  types.empty? ? nil : types.first.content_type
end

Note that I also added an acl attribute to the model to store the S3 Access policy. This way I can differentiate between public and protected media. send_later is a DJ method that simply schedules the specified method for immediate execution in the background. Post process calls the paperclip's reprocess! function which does the actual post process. In this case the post process is only executed for images as it is thumbnail generation.

To make this work the s3_object method has to be added to the S3 storage module to access the s3 file and s3 meta info. s3_object simply returns an AWS-S3 object for the uploaded file (paperclip lately switched the S3 library from RightAWS to AWS-S3):

# paperclip/storage.rb

def s3_object style = default_style
  AWS::S3::S3Object.find(path(style), bucket_name)
end

We also need to slighly change the flush_writes method in the paperclip S3 module that writes out the files after a post_process or when the attachment is created. For thumbnail generation the original file is not modified, so it's not necessary to reupload it to S3. Also I like the thumbs use the access control settings specified in paperclip's has_attached_file (which is 'public-read' by default) but the original (and other styles) using the access settings of the uploaded file:

# paperclip/storage.rb

def flush_writes #:nodoc:
  # Added: no need to reupload original after thumb was generated
  @queued_for_write.delete(:original) if @queued_for_write[:thumb]

  @queued_for_write.each do |style, file|
    begin
      log("saving #{path(style)}")
      AWS::S3::S3Object.store(path(style),
                              file,
                              bucket_name,
                              {:content_type => instance_read(:content_type),
                               :access => (style.to_s == 'thumb') ? @s3_permissions : instance.acl, # Changed
                              }.merge(@s3_headers))
    rescue AWS::S3::ResponseError => e
      raise
    end
  end
  @queued_for_write = {}
end

Hooking into the FancyUpload Uploader

First of all the uploader needs to trigger the rails action that creates the paperclip model. In my case that model is called Medium and the action triggered is "create" (surprise :). To achieve that I modified the onFileComplete callback of the uploader:

onFileComplete: function(file) {
  if (file.response.code == 201 || file.response.code == 0){
    file.ui.element.highlight('#e6efc2');
    var req = new Request({
      method: 'post',
      url: '#{media_url}',
      data: { 'upload_element_id' : file.ui.element.id,
              'authenticity_token' : '#{form_authenticity_token}', 
              'filename' : file.name,
              'filesize' : file.size } }).send();
  }
},

As mentioned in the previous post, the callback first needs to filter for a correct upload by looking at the upload's response code. Only if it's 201 the upload was successful (Edit: This is only true when using Mac OS. In Windows the return code is 0, at least when using Firefox - don't ask me why, so we also need to allow for this). After the highlight effect from the original FancyUpload Attach a File sample the create action of the media controller is triggered. The parameters passed in are the dom id of the uploaded element (for removing it from the page after the model was successfully generated), the rails authenticity token, the filename and filesize.

The create action of the media controller looks like this:

def create
  medium = Medium.new
  
  medium.name = params[:filename]
  medium.medium_file_name = params[:filename]
  medium.medium_file_size = params[:filesize]
  medium.init_from_s3_upload

  if medium.save!  
    medium_thumb = render_to_string :partial => "media/#{medium.medium_type}_thumb", :locals => { :medium => medium }
    render :update do |page|
       page << "new Element('div', { 'html': '#{escape_javascript(medium_thumb)}' } ).inject($('media_container'))"
       page << "$('#{params[:upload_element_id]}').dispose();"
     end
   else
     render :nothing => true
   end
end

So what does this do? First the medium's name is set to the filename as well as the paperclip filename medium_file_name and the files_size which is also a paperclip attribute. After that the medium's init_from_s3_upload method described above is called. It sets the content type and acl of the file and potentially schedules a post process via DJ.

On the view side the uploader dom element for the uploaded file (progress bar etc.) is removed from the page and a thumb for the uploaded file is added to a media box. Note that I'm using MooTools here as this is the javascript library FancyUpload uses. Of cause if you don't feel comfortable having the view code in the controller you can just move it to a .js file.

For uploaded images, the thumbs that are initially added to the media box contain a loading spinner. A periodic ajax call then constantly checks whether new thumbnail images have been created by DJ and replaces the spinner with the generated thumb image. To give the periodic thumb checking action a clue when new thumbs are ready I added a new_thumb field to the Medium model. It is set after a successful post process, modifying the post_process function described above:

def post_process
  self.medium.reprocess!
  self.new_thumb = true if image?
  self.save!
end

The periodic ajax call itself is straight forward and looks like this:

def check_for_new_thumbs
  thumb_divs = []
  new_thumbs = Medium.with_new_thumb

  new_thumbs.each do |t|
    t.update_attribute(:new_thumb, false)
    thumb_divs << render_to_string( :partial => 'media/image_thumb', :locals => { :medium => t )
  end

  render :update do |page|
    new_thumbs.each_with_index do |t,i|
      page << "new Element('div', { 'html': '#{escape_javascript(thumb_divs[i])}' } ).replaces($('medium_#{t.id}'));"
    end
  end
end

with_new_thumb is a named scope looking like this:

  named_scope :with_new_thumb, :conditions => ["new_thumb = ?", true]

One thing that is missing so far is a check for whether the uploaded file's name already exists on S3. If the file exists I'd like the name to be added a suffix that makes the name unique. I implemented this already in a MediaController action called from FancyUpload's onBeforeStart callback. It works all well but I couldn't find out how to change the filename in FancyUpload before the file is uploaded. I tried setting the fileList[index].name attribute of the uploader instance for duplicate files, but it doesn't have any effect. If any of you know how to make this work, I'm happy to know!

Comments

Write new Comment