19. 11. 2010

Rails Amazon S3 Uploader with Progress Bar (2010 Update)

More than a year ago I wrote about an S3 uploader I made for rails, which is based on the great FancyUpload multiple file uploader. This uploader is still my first choice when it comes to uploading files directly to Amazon's S3 storage from within a rails application. Any other solution I found so far was lacking in some or another aspect.

Anyway, since my posts Uploading files directly to Amazon S3 using FancyUpload and FancyUpload Amazon S3 Uploader with Paperclip I encapsulated the uploader Javascript in its own class (FancyUpload3.S3Uploader) and simplified the rails helper. I also updated the uploader example app using the new code. Feel free to use it, share it and improve it! I won't go into detail here about the usage. Please check out the source code of the example app. However, if you have questions, feel free to ask me.

The uploader has been tested and works fine on the Mac in Firefox, Chrome, Safari and Opera and on Windows in Firefox, Chrome, Safari, Opera, IE7 and IE8. Some browsers seem to limit the number of parallel uploads for some reason. I could only upload 2 files in parallel in Chrome on Windows for example. It might be that there is a browser config setting to adjust this limit somewhere. If anybody finds out about it, I'd be happy to know.

Upload speeds are reasonably fast with my connection here. I managed to upload at about 500 KB/s to 1 MB/s, although my connection could do faster. The limiting factor here is S3 I think. I noticed in the past that uploads to S3 can be rather slow and also read about it on the web. How are your experiences about this? Also do you know other great and easy to integrate S3 multi file uploaders that work cross browser?

What I'd like to add to the uploader is a percentage completed and upload speed (KB/s) display. Both should be relatively easy to implement as FancyUpload offeres plenty of callbacks to hook into the upload process.

Comments

matz wrote on Wed 24. Nov 2010:

hi nico,

I tried to run the example, unfortunately my uploads won't start. The progress bar stays at 0% and there's nothing uploaded to my bucket. I played around a bit with firebug and the output, but without success.

I get a gem dependency warning when I start the server:
Gem::Dependency#version_requirements is deprecated and will be removed on or after August 2010. Use #requirement

Although I don't think that has something to do with it.
I use Mac OS 10.6, ruby 1.8.7 and for this example rails 2.3.3.
I tried Firefox, Safari and Opera, all with the same result.

Any help or recommendations would be appreciated.

Nico wrote on Wed 24. Nov 2010:

Hi Matz,

I assume you were checking the points from the readme? So you added your Amazon config to config/amazon_s3.yml and also uploaded a public readable crossdomain.xml to the s3 bucket?

One user reports in one of my older posts:

I used S3fm to manage the S3 service and the upload with fancyupload just wouldn't start. After switching to S3Hub (Mac), it immediatly worked.

Maybe that helps. I'm using the s3fox organizer firefox addon which works fine. Using Amazon's Bucket Explorer should work as well, but I haven't tested it.

You can also debug the uploader with firebug. FancyUpload is set to verbose mode per default in the example app. That means firebug will show you status info about the upload in the debug console. What do you see there?

Nico

matz wrote on Wed 24. Nov 2010:

hi nico,

thank you for the immediate response.
It works now. I thought the crossdomain.xml was fine as it was, but I manually had to set permissions. That's what you mean by "public readable" I suppose ;-)
I've been using the Amazon Bucket Explorer, but S3Hub looks way better!
Thanks again

Nico wrote on Wed 24. Nov 2010:

Great! Note that it's probably best to turn off verbose mode in production for performance reasons, by removing the parameter from the upload helper call (in views/uploads/new.haml in the example).

matz wrote on Thu 25. Nov 2010:

ok, thx for the hint.

Have you tried to use the code in a Rails 3 App yet? I will definitely give it a try when I have the time to, for now I'm just being curious.

Nico wrote on Thu 25. Nov 2010:

Nope, haven't tried it with Rails 3 yet.

Simon wrote on Thu 25. Nov 2010:

It works perfect!
Nico, but where I can find the url of uploaded file?
Should I do like in http://www.railstoolkit.com/posts/fancyupload-amazon-s3-uploader-with-paperclip or there is another variant?

Nico wrote on Fri 26. Nov 2010:

Simon, the example app uploads the files to a folder called "test" in your bucket. If you like to retrieve the url from within Rails, you need to store the filename (and maybe some meta information) of the uploaded file in the database. Yes, you can do it the way I described in the previous post you mention. You can specify a url in the upload helper's on_complete_url parameter that is called after the upload has completed. It passes in some parameters like the name of the file, so you can store it in some database model.

You probably also need to address issues with duplicate filenames. If you don't do anything, files on s3 are overwritten with new files uploaded that have the same name. The upload helper provides a callback to check your db for existing files on s3 with the same name like files in the upload queue. You can then supply a prefix for those filenames like a number or uid so that no files are overwritten. See the upload helper comments for more info.

If there is interest, I might at some point put up an example app for using the uploader with paperclip.

Simon wrote on Fri 26. Nov 2010:

I would be very grateful if you make it!)

Simon wrote on Sat 27. Nov 2010:

"You can specify a url in the upload helper's on_complete_url parameter that is called after the upload has completed. It passes in some parameters like the name of the file, so you can store it in some database model."
I set option on_complete_url like filename, but I don't know how to get out this parameter and write in db model.
Could you explain please, in witch part of ruby code should I do that and how?

Nico wrote on Sat 27. Nov 2010:

@Simon: One way of doing it is this: You can make a Rails model called S3File or Medium or whatever suits your app, with at least database column 'filename'. Setup restful routes for that model and set on_complete_url to the url to create that model like :on_complete_url => s3_files_url and :on_complete_method => 'post'. Then setup the create action of the models controller to create an instance with filename set to params[:filename]. That's it! Anything else you want to associate with that file, like a user id for example you need to pass in to the on_complete_url.

Simon wrote on Sat 27. Nov 2010:

Thanks a lot!
now it works as I want)

Philip wrote on Fri 25. Feb 2011:

Hi,

Have you looked into or do you happen to know whether it's possible for the uploader to use the multipart upload approach in S3 for uploading large size files?

Thanks.

Nico wrote on Fri 25. Feb 2011:

Hi Philip,
no, I haven't yet. But I'm also curious to know if that would work. So if you look into it, I would appreciate it if you could give some feedback here! ;-)

Nico

Josh wrote on Mon 01. Aug 2011:

Hi Nico,
I would like for the application to redirect to a specified URL once the file upload is complete. It seems that there is a way to do just this, but I'm unsure just how to get it to work. Any advice would be appreciated.

Nico wrote on Mon 01. Aug 2011:

Hi Josh,
I guess you could simply set window.location.href to the redirect url in the on_complete callback.

Josh wrote on Mon 01. Aug 2011:

Cool. Thank worked. Thanks Nico.

Sam b wrote on Wed 10. Aug 2011:

I found this post searching for "rails direct upload to s3" or something, but had some problems on Rails 3.1. Fortunately came across your PLupload version https://github.com/iwasrobbed/Rails3-S3-Uploader-Plupload and got that to work with active_admin - very nice. Just thought I'd mention it - and thanks, all the example code was a huge help.

Nico wrote on Wed 10. Aug 2011:

Glad you got it working Sam. The Plupload version you link to is not mine though.

Jon Keating wrote on Thu 25. Aug 2011:

Using Firefox6, it has stopped working for some reason. The last line when verbose is true is: "Main::fireEvent "start"" After that, it never calls fileOpen and just hangs forever.
Any idea what is broken in Firefox6?

Jon Keating wrote on Fri 26. Aug 2011:

I put it up on StackOverflow: http://stackoverflow.com/questions/7199415/flash-upload-to-s3-doesnt-work-in-firefox-6

Nico wrote on Mon 29. Aug 2011:

Hi Jon, I'm actually not the author of the plupload version of the uploader. Plupload is a completely different uploader than fancy upload, so I think your best bet is to check in some plupload forums for help. However, this made me curious, I will test the fancy uploader in Firefox 6 when I get a chance to see if that also broke.

Comments closed