Large file upload in CDMI

[Subhendu]

CDMI spec 1.0 says that to upload a large file into the cloud we
should set "X-CDMI-Partial" header string to upload the file in
chunks. Hence we are sending large files in chunks. Since we can not
specify the range (as we can, when we get data objects) we are
assuming that the client will send the chunks in order. But I think
when CDMI will be used for different applications like uploading
movies directly from internet using torrent then we have to use ranges
to upload a file partially.

1. Is there any way to do that using CDMI spec 1.0?
2. If not, can we use Metadata to specify the range of the chunks?
3. If yes, then should we use User metadata for that?
4. Or we should wait for CDMI spec new versions, as the
above implementation will not be of CDMI standard?

[jun zhang]

This is no update in 1.0.1 spec yet

[Ancoron]

Hi,

just stumbled across this question while searching for an answer myself.

After reading the spec (version 1.0.2) and thinking about it for some minutes I came to the following workflow:

1. create an empty (as default without a "value") object first which includes all metadata with the "X-CDMI-Partial" header set to "true" to set the completionState to "Processing"
   
2. continuously update the created object using the "?value:<range>" URL notation, with which the client specifies which bytes to "modify" (remember to put "X-CDMI-Partial: true")
   
3. finalize the upload with a final PUT call without the header "X-CDMI-Partial", which sets the completionState to "Complete"

This way you could also upload from multiple clients or in multiple threads, each getting different chunks to work on and you don't have to bother with the actual order of chunks.

However, there are still some open questions:

The first problem here from my perspective is that you cannot specify the target size of the object before it actually has been uploaded completely (or at least the chunk at the very end). Also I'm not sure how available implementations handle that approach (e.g. if you start with the last chunk first - for pre-initializing the storage in the backend or making sure that there is enough space for the object).

Also a problem might be, that some offset goes far out of the objects size, e.g. you initially uploaded a 4 MiB object and now you upload an update chunk with "?value:2199023255552-2199023779840" (offset = 2.0 TiB, chunk size = 512 KiB)? Will the object be "filled" with zeros until the offset is being reached in the backend (could be quiet slow - in this case) or does the update deny modification with an offset bigger than "cdmi_size", which effectively would mean that my approach will not work?

Also what I am missing is the "logical size" of an object. I am a bit puzzled by the definition of "cdmi_size" is determined by the storage backend and thus may not represent the actual object's size (Deduplication? Sparse files? Opaque compression?) - Section 16.3:

The number of bytes consumed by the object. This storage system metadata item is computed by the storage system, and any attempts to set or modify it will be ignored.

...whereas Section 8.4.6 Table 16 describes for "valuerange":

The cdmi_size storage system metadata of the data object shall always indicate the complete size of the object, including zero-filled gaps.

...which means something different (from my perspective). Taking the latter definition as truth, e.g. what about zeros at the end?

In my view of CDMI we miss something like "cdmi_assignedsize" for objects, as it is done for containers (Section 9.1.1).
 

Would be nice to hear some opinions about this from implementors (Scality? NetApp?).

Thanx,

    Ancoron

[David Slik]

The approach you outlined will work, with the limitations you identified (client coordination is required to ensure that all worker PUT ranges have completed before the PUT is finalized).

There is a new vendor extension developed by IBM and NetApp that addresses some of the issues you raised in your e-mail. It will be posted to the SNIA CDMI Extensions web page in the next few weeks.

http://snia.org/tech_activities/publicreview/cdmi

Thanks,

David Slik

NetApp

[David Slik]

The Partial Upload Extension is now published for public review:

http://snia.org/sites/default/files/Partial%20Upload%20CDMI%20Extension%201.0d.pdf

Thanks,

David Slik

NetApp

[Ancoron Luciferis]

Hi David,

thanx for the update on this and the link to the proposed extension.

However, some questions arise immediately:

1.) Is the "range=<byte-range>" only allowed at the first (initiating)
request?

2.) Why are "[ true | false ] | upload-id=<upload-id>" exclusive?

3.) What about relation-ship between "?value:<range>" and
"range=<byte-range>"?

4.) What about aborting a partial upload?


So, effectively this all means the following:

The client has to do things very differently if it wants to use this
extension and encounters a server that understands it. Basically even
the starting and ending phases of an upload are completely different,
not only the upload itself:

Current:
1.) "X-CDMI-Partial: true"
2.) ...upload data...
3.) "X-CDMI-Partial: false"

New:
1.) "X-CDMI-Partial: upload-id=1234; range=3145729-4194304"
2.) ...upload data...
3.) ???

So, just consider that I have a 4 MiB upload, which I want to split into
1 MiB each using 4 connections. Now who tells me that the one with the
range "3145729-4194304" will be the slowest? In other words to ensure a
consistent upload I would have to block the last chunk until all others
are finished, which is usually not what you want at a client side. Just
too much logic involved.

Now going one step further considering a client library that is a bit
more sophisticated and calculates used "?value:<range>" quite
dynamically. This scenario simply is impossible with either of the new
"completion conditions".

So why not just reusing the current start and end conditions and just
extend them:

I would rather leave the existing simple true/false approach untouched
and add a couple of headers:

X-CDMI-Partial-TX: <upload-transaction>
X-CDMI-Partial-Completion: {range=<byte-range> | count=<count> | rollback}
X-CDMI-Partial-Replace: {true | false}

All those headers are optional and the default behavior is as it is
currently with version 1.0.2.

This means the following:

When the first call to the server using "X-CDMI-Partial: true" is being
issued, the server can generate a transaction-ID value, which then can
be (but doesn't have to) be used, which in turn would lead to
transactional updates (which I think was the idea of the <upload-id>).
Additionally, as the new conditions and the replace-flag are optional,
we can set them to any request that we do, even to the one (except
"count") that finalizes the upload.

Additionally we then support aborting an upload with any request that
comes in at the server with "X-CDMI-Partial-Completion: rollback". Every
potentially long running operations should have a rollback/abort
mechanism. Also, this allows even aborting an upload without any of the
other new attributes.

Even further, this approach does not collide with the current approach,
where we start and continue a partial upload with "true" and finalize it
with "false" for the "X-CDMI-Partial" header.

I also think about the server-side implementations where a new header
can be implemented (depending on its impact to other things) in a quite
modular fashion (thinking about aspect-based approaches here, like
transactions), whereas with a single header, you'll mix up completely
separate concerns.

These are just my two cents.


Please correct me if I am wrong on any of my assumptions or conclusions.

Cheers,

Ancoron

On 02/19/2013 03:36 AM, David Slik wrote:
> The Partial Upload Extension is now published for public review:
>
> http://snia.org/sites/default/files/Partial%20Upload%20CDMI%20Extension%201.0d.pdf
>
> Thanks,
>
> David Slik
> NetApp
>
> On Friday, February 15, 2013 10:12:18 AM UTC-8, David Slik wrote:
>
> The approach you outlined will work, with the limitations you
> identified (client coordination is required to ensure that all
> worker PUT ranges have completed before the PUT is finalized).
>
> There is a new vendor extension developed by IBM and NetApp that
> addresses some of the issues you raised in your e-mail. It will be
> posted to the SNIA CDMI Extensions web page in the next few weeks.
>
> http://snia.org/tech_activities/publicreview/cdmi
> <http://snia.org/tech_activities/publicreview/cdmi>
>
> Thanks,
>
> David Slik
> NetApp
>
> On Friday, January 11, 2013 5:09:56 PM UTC-8, Ancoron wrote:
>
> Hi,
>
> just stumbled across this question while searching for an answer
> myself.
>
> After reading the spec (version 1.0.2) and thinking about it for
> some minutes I came to the following workflow:
>

> 1. create an empty (as default without a "value") object first

> which includes all metadata with the "X-CDMI-Partial" header
> set to "true" to set the completionState to "Processing"

> 2. continuously update the created object using the

> "?value:<range>" URL notation, with which the client
> specifies which bytes to "modify" (remember to put
> "X-CDMI-Partial: true")

> 3. finalize the upload with a final PUT call without the header

> --
> You received this message because you are subscribed to the Google
> Groups "SNIA Cloud" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to snia-cloud+...@googlegroups.com.
> To post to this group, send email to snia-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/snia-cloud?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[David Slik]

Good questions. Some comments/answers inline below:

On Wednesday, February 20, 2013 3:16:15 PM UTC-8, Ancoron Luciferis wrote:

Hi David,

thanx for the update on this and the link to the proposed extension.

However, some questions arise immediately:

1.) Is the "range=<byte-range>" only allowed at the first (initiating)
request?

It is allowed in any request, and may be repeated as long as it is consistent across requests.

 

2.) Why are "[ true | false ] | upload-id=<upload-id>" exclusive?

For backwards compatibility. They represent two different operating modes, and are described by separate capabilities.

 

3.) What about relation-ship between "?value:<range>" and
"range=<byte-range>"?

The range=<byte-range> statement in the X-CDMI-Partial header indicates a completion condition. Read it as, "when this byte range has been received, the object shall be finalized.

The "?value:<range>" in the URI indicates what data is being sent in the transaction.

These usually are different (the transaction range is a subset of the completion byte range.

So, for example, if you split a 4K object into four transactions, you would have:

PUT /myobject?0-999

X-CDMI-Partial: range=0-3999

PUT /myobject?1000-1999

X-CDMI-Partial: range=0-3999

PUT /myobject?2000-2999

X-CDMI-Partial: range=0-3999

PUT /myobject?3000-3999

X-CDMI-Partial: range=0-3999

These transactions could be run in any order, in parallel or serialized, and the object will only be finalized when all four of the transactions have completed successfully.

 

4.) What about aborting a partial upload?

Currently, the extension specifies that a client would abandon it, and let the server garbage collect it when the timeout expires.

 

So, effectively this all means the following:

The client has to do things very differently if it wants to use this
extension and encounters a server that understands it. Basically even
the starting and ending phases of an upload are completely different,
not only the upload itself:

Correct.

 

Current:
1.) "X-CDMI-Partial: true"
2.) ...upload data...
3.) "X-CDMI-Partial: false"

New:
1.) "X-CDMI-Partial: upload-id=1234; range=3145729-4194304"
2.) ...upload data...
3.) ???

With this extension, step three is not needed. the "range=..." indicates a completion condition, such that when the server determines that this condition is met, the object is finalized (equivalent to X-CDMI-Partial: false).

 

So, just consider that I have a 4 MiB upload, which I want to split into
1 MiB each using 4 connections. Now who tells me that the one with the
range "3145729-4194304" will be the slowest? In other words to ensure a
consistent upload I would have to block the last chunk until all others
are finished, which is usually not what you want at a client side. Just
too much logic involved.

Now going one step further considering a client library that is a bit
more sophisticated and calculates used "?value:<range>" quite
dynamically. This scenario simply is impossible with either of the new
"completion conditions".

This is exactly the scenario that the extension is designed for. By expressing the completion condition independently from the byte ranges sent in individual transactions, it gives the client to dynamically chunk without coordination.

 

So why not just reusing the current start and end conditions and just
extend them:

I would rather leave the existing simple true/false approach untouched
and add a couple of headers:

We are trying to avoid adding new headers.

[Ancoron Luciferis]

Hi David,

thanx for the clarifications on this. I initially thought the "range"
was to describe the final upload block and not the whole upload range,
which confused me, of course.

However, it all makes much more sense to me now.


Cheers,

Ancoron

> > an email to snia-cloud+...@googlegroups.com <javascript:>.

> > To post to this group, send email to snia-...@googlegroups.com

> <javascript:>.

> > Visit this group at
> http://groups.google.com/group/snia-cloud?hl=en

> <http://groups.google.com/group/snia-cloud?hl=en>.

> > For more options, visit https://groups.google.com/groups/opt_out

> <https://groups.google.com/groups/opt_out>.

[pvg pvg]

Hi,

I am using CDMI version 1.0.1 over OpenStack Swift and want to update it to 1.0.2. How can I do this?

thanks