Start bulk get in order is returning chunks in wrong order (URGENT)

[Rob Freer]

This is the response I am getting

<MasterObjectList BucketName="rob" CachedSizeInBytes="0" ChunkClientProcessingOrderGuarantee="IN_ORDER" CompletedSizeInBytes="0" JobId="06f0aa7e-0c30-435f-a4ad-4e48cd0e5fa1" OriginalSizeInBytes="111122230272" Priority="HIGH" RequestType="GET" StartDate="2016-01-27T08:35:31.000Z" Status="IN_PROGRESS" UserId="81472881-6870-4928-857b-52ce1e8efacd" UserName="rob" WriteOptimization="CAPACITY">

<Nodes>

<Node EndPoint="192.168.3.13" HttpPort="80" HttpsPort="443" Id="cf27472c-1b82-11e5-97eb-002590c117d8"/>

</Nodes>

<Objects ChunkId="792482ce-843c-4027-aef6-dcacc9b39842" ChunkNumber="0" NodeId="cf27472c-1b82-11e5-97eb-002590c117d8">

<Object InCache="true" Length="3748047872" Name="0x060a2b340101010201010f12132a48248ddaf6021374058018bfB4B52FD8BF4C" Offset="107374182400"/>

</Objects>

<Objects ChunkId="b59c5b82-bd47-4947-b9ea-6d4676553587" ChunkNumber="1" NodeId="cf27472c-1b82-11e5-97eb-002590c117d8">

<Object InCache="true" Length="107374182400" Name="0x060a2b340101010201010f12132a48248ddaf6021374058018bfB4B52FD8BF4C" Offset="0"/>

</Objects>

</MasterObjectList>

as you can see the file is out of order. Am I misunderstanding how in order is supposed to work?

Thanks

Rob

[Joe Sislow]

I have seen the same behavior. It appears that objects within a chunk (if there are more than one) are always in order when requested, but the chunks themselves are not in order.

I can understand how these chunks might come available in a different order, but since the MasterObjectList is aware that IN_ORDER is set, it would be nice if the API call returned any chunks it does get in order, even if there are gaps.

Also, with large objects, 3 chunks default is kind of a guarantee that you'll always get chunks out of order.

[Rob Freer]

So does that mean if I have very large files I should set chunkProcessingOrderGuarantee to "NONE" so I am actually guaranteed to be able to get them in order as if I specify "IN_ORDER" I might have to get them out of order?

[Ryan Moore]

This is a known bug that is fixed in the about to be released firmware.  You're understanding of 'IN_ORDER' is correct.  It should return the chunks to you in order when it is specified.

I will let you know as soon as the new firmware is available.

--
Ryan Moore

[Rob Freer]

Thanks

[Joe Sislow]

This is obviously a definition of "fixed" I am unaware of. Here's what happens in 3.0 if you do a bulk get with IN_ORDER set.

<Error>
  <Code>Conflict</Code>
  <HttpErrorCode>409</HttpErrorCode>
  <Message>Job
 has been re-chunked "in-flight" and has a
chunkClientProcessingOrderGuarantee of IN_ORDER, which this request
handler does not support (this request handler can no longer guarantee
compliance with the chunkClientProcessingOrderGuarantee configured for
this job).</Mes
sage>
  <Resource>/_rest_/job_chunk/</Resource>
  <ResourceId>2285</ResourceId>
</Error>

[Mark P]

Hi Joe,

Would it be possible for you to create an ASL (log set) for us and send it over?  That will allow us to get to the bottom of this the quickest...  Suman (in support) can (and will) provide you with a place to upload the logset.

Thanks,

Mark

[Ryan Moore]

Our understanding of IN_ORDER for 3.0.0 was incorrect.  However, we are releasing 3.0.2 at the end of April, which will fix the IN_ORDER flag such that it means chunks will return blobs ordered by their offset exactly the way we thought it should behave.

I'll follow up again once 3.0.2 has been released.

--

Ryan Moore

[jeffbr]

Note that the issue Ryan referenced above is fixed in 3.2, which is scheduled to be released this week on June 15. Thanks, Jeff