File Resume Upload

[Reuben]

Hello experts,

Currently I am doing a project that require users to upload file that
is about 200mb in size, and I wonder if there is any gems or plugins
that deals with " resumable " upload.

I did a little google search and apparently there is no existing RoR
solution to " resumable upload " except a program called "
Jumperloader " which is written in java.

Any advices ?

[Da Cunha Daniel]

In such case you might wan to rely on FTP protocol, I am unable to look for it now but there must be some gem for FTP (or better sftp), you can then rely on the resume of FTP.

Sent from my iPhone

> --
> You received this message because you are subscribed to the Google Groups "Singapore Ruby Brigade" group.
> To post to this group, send email to singap...@googlegroups.com.
> To unsubscribe from this group, send email to singapore-rb...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/singapore-rb?hl=en.
>

[Yasith Fernando]

Haven't done this my self. however i'd say this functionality should be supported from the client side. And ASFIK no browser supports this. So its more of a browser/protocol limitation than that of rails(or whatever fancy web framework).

This means you will have to ask your users to use some java applet/browser plugin/custom uploader...etc.

[Yasith Fernando]

Upon further googling, i found this. Seems you can do this with some html5 magic. Needless to say this will not work on all browsers.

waterunderice: Resumable, asynchronous file uploads using WebSockets and HTML5

http://thechangelog.com/post/1986531993/waterunderice-resumable-asynchronous-file-uploads-using

[Suman Mukherjee]

Can't BG jobs handle this?

--
Suman Mukherjee
Software developer ( East Agile )

Twitter : http://twitter.com/mukherjeesuman

LinkedIn : http://www.linkedin.com/in/sumanmukherjee03

[Reuben]

Hello,

Wow, Thanks for the responses. =)

Daniel : Thanks for the direction, I will dig deeper into using SFTP
as an alternative. Never thought of using SFTP.

Yasith : The client would like to have the ability to use the upload
function across different browsers. sadly..

Thanks people! I will update once I find out a way to implement the
resume function. =)

On Jun 21, 12:22 am, Suman Mukherjee <sumanmukherje...@gmail.com>
wrote:

> Can't BG jobs handle this?
>
>
>
>
>
>
>
>
>
> On Mon, Jun 20, 2011 at 9:51 PM, Yasith Fernando <yas...@gmail.com> wrote:
> > Upon further googling, i found this. Seems you can do this with some html5
> > magic. Needless to say this will not work on all browsers.
>
> > waterunderice: Resumable, asynchronous file uploads using WebSockets
> > and HTML5
>

> >http://thechangelog.com/post/1986531993/waterunderice-resumable-async...

>
> > On Mon, Jun 20, 2011 at 10:45 PM, Yasith Fernando <yas...@gmail.com>wrote:
>
> >> Haven't done this my self. however i'd say this functionality should be
> >> supported from the client side. And ASFIK no browser supports this. So its
> >> more of a browser/protocol limitation than that of rails(or whatever fancy
> >> web framework).
>
> >> This means you will have to ask your users to use some java applet/browser
> >> plugin/custom uploader...etc.
>

> Software developer ( East Agile <http://eastagile.com/> )
> Twitter :http://twitter.com/mukherjeesuman

[Jason Ong]

Cool. Let us know your implementation. Might work great as a Srb meet up topic too!

Cheers,
Jason

----
web: http://bit.ly/jasonong

[Matthew Rudy Jacobs]

On 21 June 2011 11:01, Reuben <natn...@gmail.com> wrote:

Hello,

Wow, Thanks for the responses. =)

Daniel : Thanks for the direction, I will dig deeper into using SFTP
as an alternative. Never thought of using SFTP.

Yasith : The client would like to have the ability to use the upload
function across different browsers. sadly..

Thanks people! I will update once I find out a way to implement the
resume function. =)

 

Hi Reuben

I went on a hunch that said "I've never seen this on the web"

So I tried to upload a 200mb video to Vimeo

no resume

So I tried to upload a 200mb video to YouTube

no resume

So my guess is, if these big players aren't doing it

either its not worth doing

or the results are less than desirable.

I wonder if you really need this functionality.

200mb upload for a country with as good internet access as singapore shouldnt be a problem.

[Husein Choroomi]

You can use a Flash component to implement that, something like s3-swf-upload-plugin [1] if you're using S3.

[1] https://github.com/GreenAsJade/s3-swf-upload-plugin

Husein

http://about.me/hus

On 21 June 2011 11:01, Reuben <natn...@gmail.com> wrote:

[Ashish Puliyel]

Hi,

I've had some experience with this (at my work with DocuBuzz.com), and the sort answer is - it's do-able but probably not worth the effort. Logically, to do this you need two parts:

1. A client capable of sending parts of a file
2. A server capable of identifying segments that belong together and joining them up again.

It's further complicated by the exact scenario of your implementation (are users allowed to access each others's uploads? Are users anonymous?).

FTP:
Ftp nativly supports resume, but:

• Users need an ftp client to upload (no mainstream browser supports ftp uploads without a plugin afaik)
• Users will have the ability to overwrite other files in the same folder they can upload to (with logged-in users they can each have an upload-folder - with anonymous users this is harder)

Java applet:
If you stick with browser-based http solutions, you can use a Java applet on the client side to handle the cutting up the file, and write custom ruby on the server side to put it together again. Jumploader, JUpload, JFileUPload and postlet
are some of the options you can try. Or you can write your own.
For anonymous users, the trick is to give the applets a random token they can identify themselves to the server with, then even if they are disconnected, and have to resume, they can identify themselves to the server using that token.

Flash:
Similarly there are flash-based upload tools for the client side. But none I found supported resume - probably a limitation with flash.

HTML5:
HTML5's file api lets you read a file in, and slice a chunk out and send it. Of course you're limited to browsers which support this, but I recall the Mozilla dev blog had a demo of this recently.

Summary:
None of these are ideal, and in general none of the popular rails file-upload gems support them.

So in the end, we decided not to support resuming uploads until html5 adoption was more widespread.

Recently I discovered and started using plupload as a client-side js wrapper API (it lets you easily control and switch between HTML5, sliverlight, flash, browserplus, and gears), and gives you a relatively simple way to offer nice features (like multiple file upload and drag and drop selection) to browsers support it, and elegantly degrade for browsers which don't.

TLDR:
If you must have this feature use jumploader and write your own ruby to handle it server-side. If you can mange without it, plupload is the easiest way to users a nice upload experience where possible.

Disclaimer:
Most of my in-depth research into the java and flash options was about a year and a half ago. Things may have changed in the meantime!

Cheers,
Ashish

+65 98568742
http://www.puliyel.com

[Aen Tan]

Of all the sites that handle large files, i.e. YouTube, Yousendit, Rapidshare, etc… none of their uploaders have any fancy resume function. While nice for technical exploration. Something like this probably has too low a business impact to justify the amount of engineering needed. You should probably re-examine your use case.

Aen

[Reuben]

Aen Tan : Yes, I am re-examining the use case and will be discussing
with the client soon. Given the tight time frame, it really
difficult.

Ashish : Thanks for the detailed answer, I am going for plupload for
the time being before meeting the client. =)

Matthew : The problem is that the target market are third world
country that has slow or unstable internet connection, hence the idea
of resume function.

Thanks guys, I will gather the feedbacks back to the client.

On Jun 22, 1:20 pm, Aen Tan <he...@aentan.com> wrote:
> Of all the sites that handle large files, i.e. YouTube, Yousendit,
> Rapidshare, etc… none of their uploaders have any fancy resume function.
> While nice for technical exploration. Something like this probably has too
> low a business impact to justify the amount of engineering needed. You
> should probably re-examine your use case.
>
> Aen
>
>
>
>
>
>
>
> On Wed, Jun 22, 2011 at 12:57 PM, Ashish Puliyel <ash...@puliyel.com> wrote:
> > Hi,
>
> > I've had some experience with this (at my work with DocuBuzz.com), and the
> > sort answer is - it's do-able but probably not worth the effort. Logically,
> > to do this you need two parts:
>

> >    1. A client capable of sending parts of a file
> >    2. A server capable of identifying segments that belong together and

> >    joining them up again.
>
> > It's further complicated by the exact scenario of your implementation (are
> > users allowed to access each others's uploads? Are users anonymous?).
>

> > *FTP:*

> > Ftp nativly supports resume, but:
>

> >    - Users need an ftp client to upload (no mainstream browser supports

> >    ftp uploads without a plugin afaik)

> >    - Users will have the ability to overwrite other files in the same

> >    folder they can upload to (with logged-in users they can each have an
> >    upload-folder - with anonymous users this is harder)
>

> > *Java applet:*

> > If you stick with browser-based http solutions, you can use a Java applet
> > on the client side to handle the cutting up the file, and write custom ruby

> > on the server side to put it together again. Jumploader, JUpload<http://jupload.sourceforge.net/>,
> > JFileUPload <http://jfileupload.org/index.html> and postlet<http://postlet.com/>

> > are some of the options you can try. Or you can write your own.
> > For anonymous users, the trick is to give the applets a random token they
> > can identify themselves to the server with, then even if they are
> > disconnected, and have to resume, they can identify themselves to the server
> > using that token.
>

> > *Flash:*

> > Similarly there are flash-based upload tools for the client side. But none
> > I found supported resume - probably a limitation with flash.
>

> > *HTML5:*

> > HTML5's file api lets you read a file in, and slice a chunk out and send
> > it. Of course you're limited to browsers which support this, but I recall

> > the Mozilla dev blog <https://hacks.mozilla.org/2011/04/resumeupload/> had

> > a demo of this recently.
>

> > *Summary:*

> > None of these are ideal, and in general none of the popular rails
> > file-upload gems support them.
>
> > So in the end, we decided not to support resuming uploads until html5
> > adoption was more widespread.
>
> > Recently I discovered and started using plupload

> > <https://github.com/moxiecode/plupload>as a client-side js wrapper API (it

> > lets you easily control and switch between HTML5, sliverlight, flash,
> > browserplus, and gears), and gives you a relatively simple way to offer nice
> > features (like multiple file upload and drag and drop selection) to browsers
> > support it, and elegantly degrade for browsers which don't.
>

> > *TLDR:*
> > If you *must* have this feature use jumploader <http://jumploader.com/>and write your own ruby to handle it server-side. If you can mange without

> > it, plupload is the easiest way to users a nice upload experience where
> > possible.
>

> > *Disclaimer:
> > *Most of my in-depth research into the java and flash options was about a

> > year and a half ago. Things may have changed in the meantime!
>
> > Cheers,
> > Ashish
>

> > <http://postlet.com/>

> > On Tue, Jun 21, 2011 at 12:25 PM, Husein Choroomi <hchoro...@gmail.com>wrote:
>
> >> You can use a Flash component to implement that, something like
> >> s3-swf-upload-plugin [1] if you're using S3.
>
> >> [1]https://github.com/GreenAsJade/s3-swf-upload-plugin
>
> >> Husein
> >>http://about.me/hus
>

[Aen Tan]

Suggestion. Should consider a desktop client that has resume.

Sent from my iPhone 5

[Matthew Rudy Jacobs]

On 22 June 2011 18:25, Reuben <natn...@gmail.com> wrote:

Matthew : The problem is that the target market are third world
country that has slow or unstable internet connection, hence the idea
of resume function.

 

So maybe your answer isn't that you handle upload resumes,

but instead get your clients to resize their uploads beforehand.

if its video then teach them how to compress it

if its bmp images they're uploaded, then teach them how to make them into jpgs

...

or if necessary,

create an adobe air app that does some of this for you

and acts as a branded ftp uploader.

[Meng Weng Wong]

this doesn't sound like that hard a problem -- why not just recode all upload functionality to split into "MTU" chunks and do explicit transmission control.

[Meng Weng Wong]

On Jun 22, 2011, at 7:38 PM, Meng Weng Wong wrote:

> this doesn't sound like that hard a problem -- why not just recode all upload functionality to split into "MTU" chunks and do explicit transmission control.
>
>

that is to say, rather than conceiving the problem as a resumption after failure of a linear stream of bytes, always internally decompose the file into sectors and always send each sector separately, possibly in parallel, multithreaded; sort of like doing RAR on the fly, or a monopoly-monopsony version of Bittorrent. Game developers and streaming media developers don't always depend on TCP; they implement their own reliable transports over UDP. You can implement your own reliable transport over AJAX.

[Gideon Simons]

If this is a web application and will sit in a browser it will therefore be limited to either http or https. (ajax is also http)

A flash component can do some other protocols like rtp or rtmp but there are still limitations of the browser for number of connections etc.. because the AVM is embedded as a plugin. 

and also these kinds of protocols are not your best choice for file transfer because they allow a very high error rate.

any other solution will require a new plugin or software to be installed, and is not a good user experience.

With flash you can also use AMF which is very efficient way to transfer data in compact binary format, you can create a back end with amfphp and save the binary data back to a file in the php class.

I have used amfphp on several projects and its super fast. but i have never transferred as much data as you are proposing.

- again, this will also be transferred on http, and within the browser limitations, you may not be able to manipulate the data too much as it will probably be a violation of the security sandbox.

Gideon

--

You received this message because you are subscribed to the Google Groups "Singapore Ruby Brigade" group.
To post to this group, send email to singap...@googlegroups.com.
To unsubscribe from this group, send email to singapore-rb...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/singapore-rb?hl=en.

--
Gideon