Transferring streams along with meta data

[Philipp Sumi]

Hi group

A (hopefully quick) question about best practice when it comes to
streaming data. Let's say I have an object that looks like this:

public class DataObject
{
public string Foo { get; set; }
public long Bar { get; set; }

public Stream Data { get; set; }
}

What is the recommended way to transfer the information encapsulated
in this object in both directions (GET / POST) in OR? What I've been
doing so far:

Download: Added the meta data to the response header (my handler has
an IResponse injected), and read the headers on the client. The stream
is just returned as the handler's return value.

Upload: Submitted the meta data as part of the URI. As an alternative,
I was thinking whether I should switch to multipart forms (after all,
one could have tons on meta data), but I'm not sure whether this gives
me proper streaming capabilities from client to server (I don't want
to have the whole stream's content buffered on either side). What's
the story here in OpenRasta when it comes to streaming?

Thanks for your input :)
Philipp

[Sebastien Lambla]

Use IFile both ways, it already manages content-disposition and filenames, using multipart/form-data. No need to go and muck around with the http headers :)

As for streaming, it's not direct streaming on either side, but OR will put those requests to disk before passing them to your handler. Asp.net is preventing incoming streaming anyway.

OR is ready for direct streaming, but the asp.net hosting is not, so it's buffered.



> Date: Wed, 10 Feb 2010 04:42:44 -0800
> Subject: [openrasta] Transferring streams along with meta data
> From: hard...@gmail.com
> To: open...@googlegroups.com

---

Got a cool Hotmail story? Tell us now

[Philipp Sumi]

Thanks for the feedback Seb! Problem: I don't want to transfer a whole
file (for this, I applied IDownloadable file, which is great) but just
a chunk of binary data. IFile sets the file name (I have none) and
ignores my other meta data properties. I'm not convinced yet ;)

As a side note: No streaming sounds bad, not the least because I can
do this in WCF. What if a user wants to upload a DVD image
(theoretically)? Does this mean the whole data will be stored in
memory, then serialized to disk by OR before being forwarded to the
handler?

On Feb 10, 1:58 pm, Sebastien Lambla <s...@serialseb.com> wrote:
> Use IFile both ways, it already manages content-disposition and filenames, using multipart/form-data. No need to go and muck around with the http headers :)
>
> As for streaming, it's not direct streaming on either side, but OR will put those requests to disk before passing them to your handler. Asp.net is preventing incoming streaming anyway.
>
> OR is ready for direct streaming, but the asp.net hosting is not, so it's buffered.
>
>
>
> > Date: Wed, 10 Feb 2010 04:42:44 -0800
> > Subject: [openrasta] Transferring streams along with meta data

> > From: hardco...@gmail.com

> > To: open...@googlegroups.com
>
> > Hi group
>
> > A (hopefully quick) question about best practice when it comes to
> > streaming data. Let's say I have an object that looks like this:
>
> > public class DataObject
> > {
> >   public string Foo { get; set; }
> >   public long Bar { get; set; }
>
> >   public Stream Data { get; set; }
> > }
>
> > What is the recommended way to transfer the information encapsulated
> > in this object in both directions (GET / POST) in OR? What I've been
> > doing so far:
>
> > Download: Added the meta data to the response header (my handler has
> > an IResponse injected), and read the headers on the client. The stream
> > is just returned as the handler's return value.
>
> > Upload: Submitted the meta data as part of the URI. As an alternative,
> > I was thinking whether I should switch to multipart forms (after all,
> > one could have tons on meta data), but I'm not sure whether this gives
> > me proper streaming capabilities from client to server (I don't want
> > to have the whole stream's content buffered on either side). What's
> > the story here in OpenRasta when it comes to streaming?
>
> > Thanks for your input :)
> > Philipp
>

> _________________________________________________________________
> Got a cool Hotmail story? Tell us nowhttp://clk.atdmt.com/UKM/go/195013117/direct/01/

[Sebastien Lambla]

No, it doesn't mean that.

There's two levels at which caching happens. The first one is anything living on top of asp.net will be caught by asp.net loading the whole file on disk before giving you any stream. It's true of asp.net, of MVC, of monorail and, as far as i know, of WCF running on top of asp.net / IIS with asp.net compat.

There's a way out, which is to bypass asp.net for reading the stream, but this has a couple of compat consequences (need to reroute asp.net stream reading on another stream through interception, and rewire it to a local stream we handle ourselves through the worker request) which makes it a no no for our 2.0 release.

Now if the client jsut uses application/octet-stream, that's all there is, openrasta won't try and cache anything.

If you use the multipart/form-data media type, things are a bit more complicated.

If you try and use IEnumerable<IMultipartHttpEntity>, then OR won't do any caching, you can stream everything per multipart and it'll just work.

If you try and use the binder (aka passing byte[], stream, IFile or anything that needs to be built), then openrasta has to start reading the stream for you, and will automatically cache on disk the entities that are passed as part of the multipart, above a treshold. It's not all kept in memory, it's the other way around. The stream is read to disk, and you get a reference t a FileStream instead of an InMemoryStream, and it will happen trasparently.

So depending on your scenario, you have those options you can work with. But hosting in asp.net means having to deal with buffering locally before we get the hand, and there's nothing we can do for this release, unless someone sends me a patch with all regression tests passing :)

As for your metadata properties, how are you returning those? Is your scenario a ReST API or just an http POD service?

Seb


> Date: Wed, 10 Feb 2010 05:30:54 -0800
> Subject: [openrasta] Re: Transferring streams along with meta data
> From: hard...@gmail.com
> To: open...@googlegroups.com

[Philipp Sumi]

I can live with the disk caching, so all is good. Thanks for the
comprehensive feedback, btw - you're really doing a great job here :).

Regarding my meta data scenario: I'm writing a (generic) tutorial on
streaming binary data with OpenRasta and Silverlight, mainly because
there's so many people struggling with the WCF bits in that area.
OpenRasta really has the potential to shine here. Currently, I'm just
returning my meta data as header values, which can be easily done
thanks to IResponse and the DI in OR - but of course, it would be nice
to return just a single resource (comparable to MessageContracts in
WCF). For the uploads, I don't have a compelling solution yet - URIs
are a messy alternative if you have a lot of meta data.

On Feb 10, 5:08 pm, Sebastien Lambla <s...@serialseb.com> wrote:
> No, it doesn't mean that.
>
> There's two levels at which caching happens. The first one is anything living on top of asp.net will be caught by asp.net loading the whole file on disk before giving you any stream. It's true of asp.net, of MVC, of monorail and, as far as i know, of WCF running on top of asp.net / IIS with asp.net compat.
>
> There's a way out, which is to bypass asp.net for reading the stream, but this has a couple of compat consequences (need to reroute asp.net stream reading on another stream through interception, and rewire it to a local stream we handle ourselves through the worker request) which makes it a no no for our 2.0 release.
>
> Now if the client jsut uses application/octet-stream, that's all there is, openrasta won't try and cache anything.
>
> If you use the multipart/form-data media type, things are a bit more complicated.
>
> If you try and use IEnumerable<IMultipartHttpEntity>, then OR won't do any caching, you can stream everything per multipart and it'll just work.
>
> If you try and use the binder (aka passing byte[], stream, IFile or anything that needs to be built), then openrasta has to start reading the stream for you, and will automatically cache on disk the entities that are passed as part of the multipart, above a treshold. It's not all kept in memory, it's the other way around. The stream is read to disk, and you get a reference t a FileStream instead of an InMemoryStream, and it will happen trasparently.
>
> So depending on your scenario, you have those options you can work with. But hosting in asp.net means having to deal with buffering locally before we get the hand, and there's nothing we can do for this release, unless someone sends me a patch with all regression tests passing :)
>
> As for your metadata properties, how are you returning those? Is your scenario a ReST API or just an http POD service?
>
> Seb
>
>
>
> > Date: Wed, 10 Feb 2010 05:30:54 -0800
> > Subject: [openrasta] Re: Transferring streams along with meta data

> > From: hardco...@gmail.com

[SerialSeb]

Thinking of it, there is no reason why we couldn't allow handlers to
return IHttpEntity themselves, it would make much sense.

Seb