Launching concurrent jobs in different machines. Is it possible ?

[Igor Blanco]

I've read in the documentation that the import tool avoids parallel execution of import jobs by using the JobLockService. I do understand that doing several loads at same time in the same machine is not very interesting and could probably even be a problem. After all you can configure how many threads to use so you can configure it to take most out of your hardware.

But what about clustered installations ? If you have a 2 machine Alfresco cluster you might be interested in launching the process twice, for differente directories in different machines. In some circunstances that might dramatically reduce the initial load times. Is this supported somehow ? If I'm not mistaken the JobLockService does lock the resource in a cluster wide way, so unless specific locks are used for each machine this wouldn't be possible, isn't it ?

By the way I'm doing a 1.000.000 document test load in Alfresco 4.2 Community using the latest 1.2.1 tool version. It seems to work right out of the box. I'll let you know about it when it ends.

Thanks 

[Peter Monks]

G'day Igor,

The Google Code version of the tool does not use the JobLockService, though the fork that's bundled in Alfresco Enterprise may (I'm not involved in that work so can't confirm/deny).  It does however prevent more than one import from being initiated on a single cluster node at a time, and as you say doing such a thing isn't very sensible, even if it were possible.

The import tool is not cluster aware, so provided each cluster node is importing disjoint source content sets (ideally to different target folders) it should work fine.  While this may result in a shorter elapsed time than simply importing those content sets serially, it won't be as big a win as you might expect due to things like:

• Contention on shared resources (notably the database, but potentially also the contentstore and SOLR search engine)
• Alfresco overheads (e.g. cross-cluster cache invalidation)

To be frank, I'd recommend focusing on ensuring you can do in-place imports in as many cases as possible, even if that means you're only using a single cluster node.  In-place imports outperform streaming imports by orders of magnitude.

On a well provisioned (i.e. production grade) and appropriately tuned Alfresco instance, you should be able to in-place import 1,000,000 files in around half an hour, without resorting to parallel imports on multiple cluster nodes.

Cheers,
Peter

 

--
You received this message because you are subscribed to the Google Groups "Alfresco Bulk Filesystem Import" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alfresco-bulk-filesys...@googlegroups.com.
To post to this group, send email to alfresco-bulk-f...@googlegroups.com.
Visit this group at http://groups.google.com/group/alfresco-bulk-filesystem-import?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Igor Blanco]

Ok Peter,

Thank you very much for the information. Our test machine deffinitively is not a "production grade" machine, it is a KVM virtual machine with 4 cores 8 gig RAM and some space coming from a RAID1 SATA disk pair. Mysql, indexes (Lucene not SOLR) and content in the same disk and the same machine. 

It took 3,5 hours to "in place" load 1.000.000 documents. It's very interesting knowing that the expected performed would be at least 6 times better, it gives as a goal. As a suggestion it might be a good idea to have in the projects a Wiki page where people can describe their architecture and their load results.

Our 1.000.000 test documents are all 645Kb images, so no content indexing is done but obviously metadata indexing is done. In your experience do you think that disabling indexing during load time and rebuilding it afterwards would be faster ? Is it worth the trouble ?

By the way I noticed that for those images no EXIF metadata is being extracted. I assume that due to load performance reasons the tool is disabling metadato extractors. And rules?

What about behaviours ? Will they get triggered ?

Thank you very much.

2013/3/15 Peter Monks <pmo...@alfresco.com>

--
You received this message because you are subscribed to a topic in the Google Groups "Alfresco Bulk Filesystem Import" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/alfresco-bulk-filesystem-import/WKNkfrLkqGA/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to alfresco-bulk-filesys...@googlegroups.com.

To post to this group, send email to alfresco-bulk-f...@googlegroups.com.
Visit this group at http://groups.google.com/group/alfresco-bulk-filesystem-import?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Igor Blanco González
Binovo IT Human Project
e-mail: ibl...@binovo.es
Telf. :   943 493611 - 690229375
Dirección:
                         Astigarraga Bidea 2
                          Planta 6. - Ofi. 3-2 
                    20180 Oiartzun ( Gipuzkoa )