RFC: Content Distribution and Management

[Ryan McDougall]

First RFC-like submission.

I think it's far from done, but I'd like to get your input on how to go.

http://www.kyoryoku.org/wiki/index.php?title=Content_Distribution_and_Management

Cheers,

[daniel miller]

I like it, as far as I understand it.

"I claim without proof that for binary storage and retreival such an
interface is optimally scalable from a machine perspective"

I prefer claims without proofs, it saves so much time and endless fussing.

-danx0r

[Hurliman, John]

See http://portal.acm.org/citation.cfm?id=1118597 and the papers cited by that one. Maybe a more accurate wording would be "this interface provides a direct mapping to scalable methods of storage that minimize storage costs".

John

[Ewen Cheslack-Postava]

Or see any of the numerous highly scalable distributed key-value store
implementations (dynamo, DHTs like CAN, Chord, Pastry and Tapestry,
and storage systems built on top of them like CoralCDN). Daniel,
you're right that leaving out a "proof" and noting it seems like a
glaring omission, but at this point, both in the academic community
and industry, this is a generally accepted claim, mainly because
scalable systems with that basic interface have been demonstrated so
many times. Rewording it as John suggests, maybe with links to a few
examples,

Of course, there are other motivations for the interface as well,
probably most easily summed up by pointing out that it is RESTful,
which generally implies scalability. Most significantly here, its
stateless and cacheable. I'll note that in this document the only
mention of caching I see is client side, but the ability to cache is
useful at many intermediate points as well, and pointing this out
might be worthwhile.

However, while scalable, these systems actually don't always have all
the characteristics we might want. For instance, latency in a lot of
the DHT implementations is abysmal and require a lot of work on top of
the basic scalable versions to actually work well in practice. For
example, see http://portal.acm.org/citation.cfm?id=1251522.1251527
("Fixing the embarrassing slowness of OpenDHT on PlanetLab").

-Ewen

[daniel miller]

> and storage systems built on top of them like CoralCDN).  Daniel,
> you're right that leaving out a "proof" and noting it seems like a
> glaring omission, but at this point, both in the academic community
> and industry, this is a generally accepted claim, mainly because
> scalable systems with that basic interface have been demonstrated so
> many times.  Rewording it as John suggests, maybe with links to a few
> examples,

Totally agree. Bad attempt at irony.

[Ryan McDougall]

Just like to point out I know Dan well enough to have known right from
the beginning he was being funny. :)

With me going on vacation for most of July soon, and undoubtedly
everyone else doing similar at some point, discussion will naturally
hit a lull; but I hope to pick up this thread later on and see if
there isn't something we can all agree on, or maybe even standardize
on, when our hearts are back in it. :)

Anyone wanna join me in Mexico? :D

Cheers,

[Henrik Bennetsen]

On Fri, Jun 26, 2009 at 2:00 AM, Ryan McDougall <semp...@gmail.com> wrote:

Anyone wanna join me in Mexico? :D

With the irony levels in this thread already dangerously high you go and say something like this?!? :)

--
Henrik Bennetsen
Associate Director
Stanford Humanities Lab
Stanford University

Wallenberg Hall, 450 Serra Mall
Building 160, Stanford University
Stanford, CA 94305-2055, USA

benn...@gmail.com
Cell: +1 415.418.4042
Fax: +1 650.725.0192

[Ewen Cheslack-Postava]

I think this came up at some point in a previous thread, but was there
a conclusion to the streaming / range request issue? The interface
you specify only allows full transfer. If you assume convenient
ordering of data from beginning of file to end, then streaming and
range request are equivalent and could, at worst, be handled by just
killing a transfer. Of course, you wouldn't be able to later pick up
more data if you actually kill it. Stalling by not accepting more
data is also a possibility, but I also consider that a hack, and the
CDN side is likely to eventually just kill the connection anyway.

We also do know of some content where the ordering isn't convenient,
for instance textures in our current system which stores mip-map
levels from highest level of detail to lowest, so getting the first n
mipmap levels requires accessing from the end of the file to some
intermediate point.

So I guess my question is, should we be adding that to the interface?
I know a lot of the academic storage systems don't provide this,
although I don't see any reason it would be difficult to add. Does
anybody know if services like S3 support range requests? Over HTTP I
would guess that they do, but I can't find a solid answer on their
site.

Also note that supporting range requests is potentially a huge win for
caches since it could significantly reduce the amount of data they
actually need to store, allow them to start shipping out chunks they
have cached while waiting for an additional request to retrieve the
remainder, etc.

-Ewen

[Hurliman, John]

I don't want to speak on behalf of anyone, but I'm going to tentatively say that adding range requests as a requirement is a good thing. The specifics of caching should probably be left up to the implementation though. We go a different direction with caching, where if partial data is requested the cache will fetch the entire resource but only send the requested range. That is making several assumptions: the data is ordered for streaming, the client will almost always want the beginning of the resource before the middle or end, and clients that request the beginning of a resource have a high probability of requesting some or all of the remaining data shortly after. I think this describes progressive textures and meshes, but probably does not describe all use cases.

By the way, Amazon S3 does support range requests.

John

>-----Original Message-----
>From: kyor...@googlegroups.com [mailto:kyor...@googlegroups.com] On
>Behalf Of Ewen Cheslack-Postava
>Sent: Friday, June 26, 2009 12:27 PM
>To: kyor...@googlegroups.com
>Subject: [kyoryoku] Re: RFC: Content Distribution and Management
>
>

[Ryan McDougall]

Just sitting here on a Saturday, I think that I hadn't really
considered that side of things at all, and will probably will be
unlikely to have an opinion before the end of July.

So I would really be happy if anyone with strong opinions could go
ahead and make a specific proposal, or even just modify the RFC
directly, if you're confident in your answers.

Cheers,

[Tommi Laukkanen]

Hello

Ryans suggestion looks good to me. It is pretty much like WebDav
simplified. Maybe things like range request and version support should
be left to phase two? Or make two levels of the interface: simple and
advanced. Standards have tendency to become overtly complex and it
should be considered carefully what is really needed and what not.
Feasibility of range requests boils down to statistics of interrupted
transfers in the modern internet. Interruption probability is
proportional to file size. As we are not planning on average asset
size of hundreds of mega bytes, range requests may be more trouble
than they are worth. TCP/IP implements guaranteed delivery of packets
and as such transmits even large files in most situations.

-tommi

[Daniel Horn]

Range requests can help you get progressive chunks of data (i.e. part
of a mesh or part of a texture)
True if you organized your texture from least detailed to most
detailed you could work around it by terminating the connection
early, but for instance the DDS format that's fairly common in video
games stores the data from biggest to smallest, making range requests
important for downloading lower resolution texture mipmaps.

[Tommi Laukkanen]

Don't you think it is impractical to retrieve those DDS files in
pieces? Due to efficiency and simplicity of implementation I would say
the entire file ought to be downloaded at once. If files are not
considered to be atomic content units from distribution perspective
then we are going for quite complex setup with questionable benefits.

-tommi

[Tommi Laukkanen]

Hello

I learned from Daniel that range reads are part of http spec and as
such a good candidate for the interface as they wont add complexity to
the interface.

-tommi