How to monitor upload progress?

[javadev]

I am using this java code to upload files:

String destFolderStr = "http://docs.google.com/feeds/default/private/
full/";
String mimeType = DocumentListEntry.MediaType.fromFileName(file.getName
()).getMimeType();

DocumentListEntry newDocument = new DocumentListEntry();
newDocument.setFile(new File(file.getPath()), mimeType);
newDocument.setTitle(new PlainTextConstruct(file.getName()));

DocumentListEntry newEntry = (DocumentListEntry) m_Service.insert(new
URL(destFolderStr), newDocument);


Is there any way that I can monitor the reading of the byte stream
such that I can monitor the progress of the upload? I assume this
would require extending and re-implementing the "insert" method from
the gdata library? Any chance some sort of example for doing is this
is available?

Thank You

[brian karlo gutierrez]

General monitoring of network connections, better use wireshark. That really helps me a lot in determining if I really send the correct http request to the google server. - Brian --- On Tue, 12/15/09, javadev <abe.op...@gmail.com> wrote:

[abe oppenheim]

well, i want to monitor the progress of a document upload from my java code.  so that i could do things like display a progress dialog. or cancel an upload.

[Bobby]

I think you may be able to implement this by using the
MediaStreamSource class for uploading document contents:
http://code.google.com/apis/gdata/javadoc/com/google/gdata/data/media/MediaStreamSource.html

The constructor for MediaStreamSource takes in an InputStream so for
example, suppose you create a class, MonitoredInputStream, that
extends InputStream and overrides the read methods to keep track of
how many bytes have been read, then, since it's unlikely that the
GData library reads the whole stream into memory (and rather just
keeps a small buffer) this should give you a good idea of how many
bytes have been uploaded, which you can make available to your
progress bar control.

I haven't implemented this, but it sounds like it would work.

Bobby

On Dec 14, 9:10 pm, abe oppenheim <abe.oppenh...@gmail.com> wrote:
> well, i want to monitor the progress of a document upload from my java code.
>  so that i could do things like display a progress dialog. or cancel an
> upload.
>
> On Mon, Dec 14, 2009 at 9:09 PM, brian karlo gutierrez

> <brikz...@yahoo.com>wrote:

>
> > General monitoring of network connections, better use wireshark. That
> > really helps me a lot in determining if I really send the correct http
> > request to the google server.
>
> > - Brian
>

[abe oppenheim]

Thank You, I was able to implement this to monitor the number of bytes read.  However, it appears that the whole stream is in fact read into a very large memory buffer.  I'm not sure how large, but large enough that when I monitored the progress of the inputStream for a 30mb file, the entire input stream was read in a few seconds.  With this in mind, is there anyway I can monitor the upload of a file? Or, more importantly, cancel the upload upon user input?

[Eric Bidelman]

Hi all,

The current upload path (we'll call it normal upload) sends

a file over the wire in its entirety. There's no way for you

to ping the server to request the upload's progress.

This is something a resumable upload feature would take care

of, which is something we're looking into for next quarter.

Eric

[abe oppenheim]

Thank you Eric.  Would you also say there is no way to programmatically cancel an upload?

[Eric Bidelman]

I suppose you (or your app) could cut off the request before

the upload is complete (eg. before an entry is returned

by the server).

[abe oppenheim]

There is no way to "cut off the request" through the api?  My app could try to kill the thread processing the request, but that seems unpredictable. 

[Eric Bidelman]

I guess I don't understand. Your app is making the request

to the API and it's in control of cancelling that request until the 

doc has completed uploading (the server receives all 

bytes according to the content-length header).

That said, I know curl provides tools like this (progress bar, resumable upload):

http://www.cs.sunysb.edu/documentation/curl/index.html

So, it is possible. In Java it might be another story :)

Eric

[abe oppenheim]

Well, my app makes the upload request to API.  But once the API has read the entire inputstream into memory, which is does prior to upload, the API handles the upload internally, the processing of the memory buffer and socket whatnots.  Unless some sort of cancel method is exposed by the API, i'm not sure how to cancel the request. 

The only things I can think of are really bad solutions.  Like trying to kill the thread processing the request.  Or extending/re-implementing some API class to allow an upload to be canceled. 

But I'm liking missing something.  Which is why I'm asking...

[Eric Bidelman]

Again, this is something we're looking into supporting. Hopefully for next quarter.

Resumable upload will allow clients to:

-Initiate a resumable upload request

- pause an upload and resume it within 24hrs.

- query the status of an incomplete upload.

It'll look something like this:

http://code.google.com/p/gears/wiki/ResumableHttpRequestsProposal

Eric

[abe oppenheim]

Thank you for your help.  I was just confirming that in addition to monitoring progress, there is no current way to cancel an upload. 

[Bobby]

By the way, for the buffer size, see the writeTo to method:
http://code.google.com/p/gdata-java-client/source/browse/trunk/java/src/com/google/gdata/data/media/MediaSource.java

It doesn't seem to be reading the whole file into memory.

I sense that this is doable even if not explicitely supported, it's
one of those things that comes down to how much you want it. I'm going
to play around with this later today, to see if i can find a way.

Bobby

On Dec 15, 3:03 pm, abe oppenheim <abe.oppenh...@gmail.com> wrote:
> Thank you for your help.  I was just confirming that in addition to
> monitoring progress, there is no current way to cancel an upload.
>

> On Tue, Dec 15, 2009 at 2:56 PM, Eric Bidelman <api.e...@google.com> wrote:
> > Again, this is something we're looking into supporting. Hopefully for next
> > quarter.
>
> > Resumable upload will allow clients to:
> > -Initiate a resumable upload request
> > - pause an upload and resume it within 24hrs.
> > - query the status of an incomplete upload.
>
> > It'll look something like this:
> >http://code.google.com/p/gears/wiki/ResumableHttpRequestsProposal
>
> > Eric
>

> > On Tue, Dec 15, 2009 at 11:40 AM, abe oppenheim <abe.oppenh...@gmail.com>wrote:
>
> >> Well, my app makes the upload request to API.  But once the API has read
> >> the entire inputstream into memory, which is does prior to upload, the API
> >> handles the upload internally, the processing of the memory buffer and
> >> socket whatnots.  Unless some sort of cancel method is exposed by the API,
> >> i'm not sure how to cancel the request.
>
> >> The only things I can think of are really bad solutions.  Like trying to
> >> kill the thread processing the request.  Or extending/re-implementing some
> >> API class to allow an upload to be canceled.
>
> >> But I'm liking missing something.  Which is why I'm asking...
>

> >> On Tue, Dec 15, 2009 at 2:29 PM, Eric Bidelman <api.e...@google.com>wrote:
>
> >>> I guess I don't understand. Your app is making the request
> >>> to the API and it's in control of cancelling that request until the
> >>> doc has completed uploading (the server receives all
> >>> bytes according to the content-length header).
>
> >>> That said, I know curl provides tools like this (progress bar, resumable
> >>> upload):
> >>>http://www.cs.sunysb.edu/documentation/curl/index.html
>
> >>> So, it is possible. In Java it might be another story :)
>
> >>> Eric
>

> >>> On Tue, Dec 15, 2009 at 10:41 AM, abe oppenheim <abe.oppenh...@gmail.com

> >>> > wrote:
>
> >>>> There is no way to "cut off the request" through the api?  My app could
> >>>> try to kill the thread processing the request, but that seems
> >>>> unpredictable.
>

> >>>> On Tue, Dec 15, 2009 at 1:33 PM, Eric Bidelman <api.e...@google.com>wrote:
>
> >>>>> I suppose you (or your app) could cut off the request before
> >>>>> the upload is complete (eg. before an entry is returned
> >>>>> by the server).
>
> >>>>> On Tue, Dec 15, 2009 at 10:32 AM, abe oppenheim <
> >>>>> abe.oppenh...@gmail.com> wrote:
>
> >>>>>> Thank you Eric.  Would you also say there is no way to
> >>>>>> programmatically cancel an upload?
>

> >>>>>> On Tue, Dec 15, 2009 at 1:12 PM, Eric Bidelman <api.e...@google.com>wrote:
>
> >>>>>>> Hi all,
>
> >>>>>>> The current upload path (we'll call it normal upload) sends
> >>>>>>> a file over the wire in its entirety. There's no way for you
> >>>>>>> to ping the server to request the upload's progress.
>
> >>>>>>> This is something a resumable upload feature would take care
> >>>>>>> of, which is something we're looking into for next quarter.
>
> >>>>>>> Eric
>
> >>>>>>> On Tue, Dec 15, 2009 at 8:16 AM, abe oppenheim <
> >>>>>>> abe.oppenh...@gmail.com> wrote:
>
> >>>>>>>> Thank You, I was able to implement this to monitor the number of
> >>>>>>>> bytes read.  However, it appears that the whole stream is in fact read into
> >>>>>>>> a very large memory buffer.  I'm not sure how large, but large enough that
> >>>>>>>> when I monitored the progress of the inputStream for a 30mb file, the entire
> >>>>>>>> input stream was read in a few seconds.  With this in mind, is there anyway
> >>>>>>>> I can monitor the upload of a file? Or, more importantly, cancel the upload
> >>>>>>>> upon user input?
>

> >>>>>>>> On Tue, Dec 15, 2009 at 5:01 AM, Bobby <bobbysoa...@gmail.com>wrote:
>
> >>>>>>>>> I think you may be able to implement this by using the
> >>>>>>>>> MediaStreamSource class for uploading document contents:
>

> >>>>>>>>>http://code.google.com/apis/gdata/javadoc/com/google/gdata/data/media...

[Bobby]

Also, if you implemented a MonitoredInputStream class, in order to
implement cancelling you can keep a flag "boolean cancelled" which the
read methods check for. If cancelled is true, throw an IOException
this should cause the request to be interrupted.

Bobby

On Dec 15, 3:20 pm, Bobby <bobbysoa...@gmail.com> wrote:
> By the way, for the buffer size, see the writeTo to method:http://code.google.com/p/gdata-java-client/source/browse/trunk/java/s...

[abe oppenheim]

This looks very promising. However, when I was playing around with it earlier today, the behavior certainly did not indicate this was the case.  I monitored the read method for the inputstream of a 30mb file, all 30mb were almost immediately read.  But, if you do get a chance to try, please report back.

*Also, I agree.  If the upload is indeed handled in this manner, by reading then uploading small chunks of an input stream, then canceling the upload would be easy. 

[Bobby]

Abe, i'm seeing the same behavior that you described, where the whole
stream is read in at once - in fact it triggers an OutOfMemoryError
for large enough files depending on how much memory the VM is allowed
to use. The problem is that the OutputStream implementation of an
HttpUrlConnection, which GData uses, is actually a
ByteArrayOutputStream which basically buffers any data written to it.
The result is that it "eats" all of the input stream at once. The
following page describes this issue
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4212479

The GData service class exposes setChunkedMediaUpload, which calls the
HttpUrlConnection's setChunkedStreamingMode (http://java.sun.com/j2se/
1.5.0/docs/api/java/net/HttpURLConnection.html#setChunkedStreamingMode
(int)). This method causes the HTTP POST to be broken down into
chunks. I tried this with the same effect - i can tell it's not really
chunking even after invoking setChunkedStreamingMode because it still
causes an OutOfMemoryError. Is HTTP 1.1 supported?

Bobby

On Dec 15, 3:31 pm, abe oppenheim <abe.oppenh...@gmail.com> wrote:
> This looks very promising. However, when I was playing around with it
> earlier today, the behavior certainly did not indicate this was the case.  I
> monitored the read method for the inputstream of a 30mb file, all 30mb were
> almost immediately read.  But, if you do get a chance to try, please report
> back.
>
> *Also, I agree.  If the upload is indeed handled in this manner, by reading
> then uploading small chunks of an input stream, then canceling the upload
> would be easy.
>

[Eric Bidelman]

That sounds like a possible bug in the library. Would you mind

filing this here: http://code.google.com/p/gdata-java-client/issues/list

Thanks!

Eric

[Bobby]

Ok, even though in my opinion the real problem is ultimately Java's
rough implementation of HttpUrlConnection, it's probably worth
tracking since there may be alternatives available that offer more
flexibility. It's also possible that the GData Java library is not
properly using setChunkedStreamingMode or that the HTTP 1.1 is not
being successfully implemented on either side.

Bobby

On Dec 15, 7:45 pm, Eric Bidelman <api.e...@google.com> wrote:
> That sounds like a possible bug in the library. Would you mind
> filing this here:http://code.google.com/p/gdata-java-client/issues/list
>
> Thanks!
> Eric
>

> ...
>
> read more »

[Bobby]

Here's the issue entry:
http://code.google.com/p/gdata-java-client/issues/detail?id=217

Bobby

> ...
>
> read more »