Sync Queue question

[David Chandek-Stark]

I noticed recently that the sync queue size reported by the sync tool increased dramatically.  I assume (hope!) that the sync queue size does not represent the number of files to actually add/update, but perhaps the number of files to "check" for updating?

Thanks,

David

[Bill Branan]

Hi David,

The sync queue includes both files that need to be moved to DuraCloud and files which just need to be checked for changes. Often if the queue size increases significantly in the middle of a run that means the tool picked up a change to a directory being watched, so all of the files under that directory were added back to the list to check for potential updates. Of course, if a new set of files were added to one of the watched directories, that would cause an increase in the queue size as well.

best,

Bill

--
You received this message because you are subscribed to the Google Groups "DuraCloud Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to duracloud-users+unsubscribe@googlegroups.com.
To post to this group, send email to duracloud-users@googlegroups.com.
Visit this group at https://groups.google.com/group/duracloud-users.
For more options, visit https://groups.google.com/d/optout.

[David Chandek-Stark]

This particular directory contains over 3M files (in subdirectories).  Does that mean whenever one of the files changes, all 3M+ files have to be checked?

--David

On Thursday, September 22, 2016 at 11:48:42 AM UTC-4, Bill Branan wrote:

Hi David,

The sync queue includes both files that need to be moved to DuraCloud and files which just need to be checked for changes. Often if the queue size increases significantly in the middle of a run that means the tool picked up a change to a directory being watched, so all of the files under that directory were added back to the list to check for potential updates. Of course, if a new set of files were added to one of the watched directories, that would cause an increase in the queue size as well.

best,

Bill

On Thu, Sep 22, 2016 at 11:39 AM, David Chandek-Stark <dchand...@gmail.com> wrote:

I noticed recently that the sync queue size reported by the sync tool increased dramatically.  I assume (hope!) that the sync queue size does not represent the number of files to actually add/update, but perhaps the number of files to "check" for updating?

Thanks,

David

--
You received this message because you are subscribed to the Google Groups "DuraCloud Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to duracloud-use...@googlegroups.com.
To post to this group, send email to duraclo...@googlegroups.com.

[David Chandek-Stark]

Related to this question, I think it would be helpful for the "successful syncs" status number to distinguish between actual file transfers and checks w/o transfer.

--David

On Thursday, September 22, 2016 at 11:48:42 AM UTC-4, Bill Branan wrote:

Hi David,

The sync queue includes both files that need to be moved to DuraCloud and files which just need to be checked for changes. Often if the queue size increases significantly in the middle of a run that means the tool picked up a change to a directory being watched, so all of the files under that directory were added back to the list to check for potential updates. Of course, if a new set of files were added to one of the watched directories, that would cause an increase in the queue size as well.

best,

Bill

On Thu, Sep 22, 2016 at 11:39 AM, David Chandek-Stark <dchand...@gmail.com> wrote:

I noticed recently that the sync queue size reported by the sync tool increased dramatically.  I assume (hope!) that the sync queue size does not represent the number of files to actually add/update, but perhaps the number of files to "check" for updating?

Thanks,

David

--
You received this message because you are subscribed to the Google Groups "DuraCloud Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to duracloud-use...@googlegroups.com.
To post to this group, send email to duraclo...@googlegroups.com.

[Daniel Bernstein]

Hi David, 

Per your first question:  the synctool will only add changed/new/deleted files to the queue (assuming you have sync deletes enabled). So the answer to your question is no. 

Re the second item,  that's a good idea.  Would you mind adding a JIRA for that?

Thanks, 

Danny

************************************

Daniel Bernstein

Software Engineer, Duraspace

707.874.2045 (office)

415.533.2354 (cell)

To unsubscribe from this group and stop receiving emails from it, send an email to duracloud-users+unsubscribe@googlegroups.com.
To post to this group, send email to duracloud-users@googlegroups.com.

[David Chandek-Stark]

Danny,

Create issue: https://jira.duraspace.org/browse/DURACLOUD-1073.

Thanks,

David

[Daniel Bernstein]

Thank you David!

************************************

Daniel Bernstein

Software Engineer, Duraspace

707.874.2045 (office)

415.533.2354 (cell)

To unsubscribe from this group and stop receiving emails from it, send an email to duracloud-users+unsubscribe@googlegroups.com.
To post to this group, send email to duracloud-users@googlegroups.com.