Batches, threads and versions

127 views
Skip to first unread message

Colin Stephenson

unread,
May 15, 2013, 3:33:00 PM5/15/13
to alfresco-bulk-f...@googlegroups.com
Hey All,

As part of a migration effort my team is using bulk importer.  The filesystem has been prepped for inplace ingestion.  There are some content files with zero versions, others with multiple versions, and the extreme cases 50+ versions.

How does bulk importer manage ingestion of versions if multiple threads are being used?  ie. if the first batch picks up 8 objects, but the this piece of content has 10 versions, does the 2nd thread pick up the next 2 version objects or is the first batch "version aware" and includes them?

Appreciate the amount of effort that goes into this free tool - any advice or guidance is always appreciated

Thanks,
Colin.

Peter Monks

unread,
May 15, 2013, 4:39:01 PM5/15/13
to alfresco-bulk-f...@googlegroups.com
G'day Colin,

The importer uses the concept of an importable item [1] that ends up being a single node in the repository.  So if you have a file on disk with associated metadata and version files (whether content versions, metadata versions or both), all of those files are grouped together into a single importable item object.

Batching occurs after these importable item objects are constructed, and takes the "weight" of each importable item into account.  Weight is a crude approximation of the cost of writing that importable item in the repository, and is calculated as the number of files on disk that make up that importable item (see here [2] and here [3]), regardless of whether they're a content file or a metadata file.  It's not a precise metric, but it's good enough to allow the importer to create batches of approximately equal size (which is the entire point).  It's also efficient.  ;-)

Batching, then, becomes very simple - the importer simply reads importable items off the queue until their accumulated weight is equal to or greater than the batch weight (which is 100 by default [4]).  At that point that batch gets imported into the repository in a single transaction.  Note that this means a node always gets imported once, in a single transaction - it's impossible for it to be "split" between transactions (in fact this is a critical design requirement, since otherwise it wouldn't be practical to provide the no-replacement option).

Multithreading occurs at a higher level than batching.  Basically each thread is given units of work [5], which correspond to directories on disk.  The thread is then responsible for all of the work involved in importing that one directory, including the analysis phase (the part that constructs importable items from the files on disk), batching (the part that splits those importable items up, if needed) and finally writing the batches into the repository.  The thread is also responsible for submitting new units of work for any subdirectories that that directory contains - it doesn't itself import the contents of those subdirectories however - they go onto the work queue and may be picked up by any of the threads in the pool.

Here's a worked example.  Assuming:
  • there are 2 directories, one underneath the other
  • the parent directory has 10 files in it, representing 5 nodes.  2 of those nodes have metadata and 1 of them has 3 content versions (for a total of 10 files on disk)
  • the child directory has 402 files in it, representing 2 nodes, each of which has 100 versions of both metadata and content

Here's how the importer would handle this import:
  1. User initiates the tool using the directory path of the parent directory
  2. The import tool constructs a unit of work for that directory and sticks it on the work queue
  3. One of the worker threads (nondeterminate which one) is woken up and handed the unit of work
  4. That thread performs analysis i.e. it lists the parent directory and constructs 6 importable items based on what it finds (5 nodes + 1 subdirectory)
  5. That thread performs batching i.e. it creates a single batch because the combined weight of the 6 importable items is less than the the batch size (100)
  6. That threads starts an Alfresco transaction, imports all 6 items (5 content nodes + 1 folder node) then commits the transaction
  7. That thread constructs a unit of work for the child directory and sticks it on the work queue
  8. One of the worker threads (nondeterminate which one - may or may not be the same as the previous one) is woken up and handed the unit of work
  9. This thread performs analysis i.e. it lists the child directory and constructs 2 importable items based on what it finds (2 nodes)
  10. This thread performs batching i.e. it creates two batches, because each importable item has a weight (101) greater than the batch size (100)
  11. This thread starts an Alfresco transaction, imports the first importable item (1 content node with 101 versions including the current version) then commits the transaction
  12. This thread starts a second Alfresco transaction, imports the second importable item (1 content node with 101 versions including the current version) then commits the transaction
  13. The import completes, shutting down (terminating) all threads in the process <- this is why you won't (or shouldn't) see importer threads hanging around idle when no imports are in process

And that's basically it.

Now what this means is that you can end up with some unusual corner cases.  If a single item has a high weight (particularly if its weight is greater than the batch size) then the batch containing it can be significantly weightier than batches are intended to be.  But the requirement to keep an item together and always import it in its entirety within a single transaction takes precedence over evenly distributed transaction sizes, so it's pretty unavoidable.  The tool could probably be optimised to put items like that in a transaction of their own (so that it doesn't, for example, create a single batch containing 2 importable items of weight 99), but it's such an oddity, and the downsides of uneven batch sizes so minor that I haven't bothered to date.  If you're seeing problems caused by uneven batch size distributions however, let me know - thus far they haven't been a problem but that doesn't mean that it's never going to be a problem.

In fact a more common issue that I'm concerned about (see issue #96 [6]) is the "large directory hierarchy, each with a few files" case - right now that's a worst case scenario for the tool since transactions don't ever span directory boundaries.  It's not trivial to figure out a multi-threaded solution to that problem however, since you run into order-of-import issues (i.e. you can't load batch C until batch B is complete, because batch C contains files that are children of a folder created in batch B).  Importing in breadth-first, sub-directories-first order helps, but doesn't completely eliminate the need to track dependencies between batches.

Cheers,
Peter


 


--
You received this message because you are subscribed to the Google Groups "Alfresco Bulk Filesystem Import" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alfresco-bulk-filesys...@googlegroups.com.
To post to this group, send email to alfresco-bulk-f...@googlegroups.com.
Visit this group at http://groups.google.com/group/alfresco-bulk-filesystem-import?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Peter Monks

unread,
May 15, 2013, 4:42:26 PM5/15/13
to alfresco-bulk-f...@googlegroups.com
Minor correction in bold red (hopefully Google Groups preserves formatting).

Cheers,
Peter

 


On May 15, 2013, at 1:39 PM, Peter Monks <pmo...@alfresco.com> wrote:

G'day Colin,

The importer uses the concept of an importable item [1] that ends up being a single node in the repository.  So if you have a file on disk with associated metadata and version files (whether content versions, metadata versions or both), all of those files are grouped together into a single importable item object.

Batching occurs after these importable item objects are constructed, and takes the "weight" of each importable item into account.  Weight is a crude approximation of the cost of writing that importable item in the repository, and is calculated as the number of files on disk that make up that importable item (see here [2] and here [3]), regardless of whether they're a content file or a metadata file.  It's not a precise metric, but it's good enough to allow the importer to create batches of approximately equal size (which is the entire point).  It's also efficient.  ;-)

Batching, then, becomes very simple - the importer simply reads importable items off the queue until their accumulated weight is equal to or greater than the batch weight (which is 100 by default [4]).  At that point that batch gets imported into the repository in a single transaction.  Note that this means a node always gets imported once, in a single transaction - it's impossible for it to be "split" between transactions (in fact this is a critical design requirement, since otherwise it wouldn't be practical to provide the no-replacement option).

Multithreading occurs at a higher level than batching.  Basically each thread is given units of work [5], which correspond to directories on disk.  The thread is then responsible for all of the work involved in importing that one directory, including the analysis phase (the part that constructs importable items from the files on disk), batching (the part that splits those importable items up, if needed) and finally writing the batches into the repository.  The thread is also responsible for submitting new units of work for any subdirectories that that directory contains - it doesn't itself import the contents of those subdirectories however - they go onto the work queue and may be picked up by any of the threads in the pool.

Here's a worked example.  Assuming:
  • there are 2 directories, one underneath the other
  • the parent directory has 10 files in it, representing 5 nodes.  2 of those nodes have metadata and 1 of them has 3 content versions (for a total of 10 files on disk)
  • the child directory has 402 files in it, representing 2 nodes, each of which has 100 versions of both metadata and content

Here's how the importer would handle this import:
  1. User initiates the tool using the directory path of the parent directory
  2. The import tool constructs a unit of work for that directory and sticks it on the work queue
  3. One of the worker threads (nondeterminate which one) is woken up and handed the unit of work
  4. That thread performs analysis i.e. it lists the parent directory and constructs 6 importable items based on what it finds (5 nodes + 1 subdirectory)
  5. That thread performs batching i.e. it creates a single batch because the combined weight of the 6 importable items is less than the the batch size (100)
  6. That threads starts an Alfresco transaction, imports all 6 items (5 content nodes + 1 folder node) then commits the transaction
  7. That thread constructs a unit of work for the child directory and sticks it on the work queue
  8. One of the worker threads (nondeterminate which one - may or may not be the same as the previous one) is woken up and handed the unit of work
  9. This thread performs analysis i.e. it lists the child directory and constructs 2 importable items based on what it finds (2 nodes)
  1. This thread performs batching i.e. it creates two batches, because each importable item has a weight (201) greater than the batch size (100)

Colin Stephenson

unread,
May 16, 2013, 11:30:36 AM5/16/13
to alfresco-bulk-f...@googlegroups.com
Reply all
Reply to author
Forward
Message has been deleted
0 new messages