Order in bulk importing

[sfontanel]

Hi,

first of all I would like to thank you for this great tool!

I have an OutOfMemory error during an "In place" bulk import after an hour using version alfresco-bulk-filesystem-import-1.1.amp on Alfresco ENT 3.4.x.
I loaded almost 30 GB of files (as the "status of import" webscript states). The total amount is almost 150 GB.
Perhaps my data structure reflects that described in "What steps will reproduce the problem?" in

http://code.google.com/p/alfresco-bulk-filesystem-import/issues/detail?id=109

However this issue will be fixed on version 1.2 :|

I may try to solve the error splitting data in smaller packages and manually (or programmatically) launching import more times.
In that case when I have an error I can reboot Alfresco and relaunch the last failed task and not the entire import. Moreover, I'm sure that the previously data are safely imported and I can restart import from the last failed block of data.
I could avoid this empirical split if I can assume the tool follows a predefined order
in importing subfolders. In that case, when the tools stops on error, I can simply restart Alfresco, and launch the import from the "next" subfolder
(assuming "other" folders have been imported successfully).

Any suggestions?

Thanks.
Stefano.

[Peter Monks]

G'day Stefano,

The OOME is almost certainly the same problem described in issue #109 - that's the only place where the bulk import (used to) consume memory without bound.  The fix for that issue will be in the 1.2 version of the tool, which is getting pretty close to release (roadmap is here [1]).

In the meantime, it's worth rerunning the import from scratch, with the "replace" option unchecked (no replacement).  In this mode the tool will skip over files that have previously been imported, allowing an import to be run and rerun as many times as necessary, each time picking up from where it left off.  Note that it may still take a couple of runs to get through your entire data set (the OOME may still occur, but should occur later each time).

Regarding load order, the basic answer is that it's OS and filesystem dependent - the tool simply lists the contents of each folder and imports items in (approximately) the order that the OS returns them.  The tool does not perform any sorting prior to import, however it does group together metadata and version files, and that can result in a slightly different load order than the OS/FS listing order.

Cheers,
Peter

[1] http://code.google.com/p/alfresco-bulk-filesystem-import/wiki/Roadmap

 

--
You received this message because you are subscribed to the Google Groups "Alfresco Bulk Filesystem Import" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alfresco-bulk-filesys...@googlegroups.com.
To post to this group, send email to alfresco-bulk-f...@googlegroups.com.
Visit this group at http://groups.google.com/group/alfresco-bulk-filesystem-import?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[sfontanel]

Hi Peter,

thank you!
I'm very glad to hear that the tool natively supports rerunning the import :)
I immediately try to rerun the import.
Regards,
Stefano

[sfontanel]

Hi Peter,
i followed your suggestion and tried to continue the import
relaunching the command from the webscript.
The first time (before my first post on this forum) the execution
stopped with the OOME after one hour but the tool completed
the import of 30 GB.
During this second execution (and other executions that followed) I
saw in the "import status" webscript report that
the number of nodes/second was initially 0. After a period the nodes/
second started to grow slowly.
. I think this happens when the tools encounters nodes which are
already present
in the repository (the tool imported these nodes in the first
execution) and it has to check
if they exist on repository (this check involves a call to alfresco
database and so it consumes execution time?).
Hence in this second execution the imports spent a "long" time
(almost 3 hours) to import
19 GB) and than stopped for another error. This error was not a big
problem for me because I wanted to try to continue the import.
As expected, also the third execution spent a long time to import a
relatively "small" amount of data.

So, I have 2 questions:

1) is it correct to say that the check of existence of a node is not
an operation limited to the file system and
"heavely" involves the alfresco database also for the nodes imported
in a previous execution? If so,
the time for the import of all data grows if I increase the number of
times I split and repeat the import?
Perhaps a determinist rule on import order may reduce the interaction
with the remote
alfresco database to identify file and folders already imported in a
previous execution?
2) to try to solve the OOME before the 1.2 release I would like to
build the module from source (I'm working on Alfresco 3.4.x). So, I
would like to know if the current source code on development is quite
stable (to be used in "production mode") or it needs more testing
before the ufficial 1.2 release?

Thank you very much.
Stefano

On 31 Gen, 20:15, Peter Monks <pmo...@alfresco.com> wrote:
> G'day Stefano,
>
> The OOME is almost certainly the same problem described in issue #109 - that's the only place where the bulk import (used to) consume memory without bound.  The fix for that issue will be in the 1.2 version of the tool, which is getting pretty close to release (roadmap is here [1]).
>
> In the meantime, it's worth rerunning the import from scratch, with the "replace" option unchecked (no replacement).  In this mode the tool will skip over files that have previously been imported, allowing an import to be run and rerun as many times as necessary, each time picking up from where it left off.  Note that it may still take a couple of runs to get through your entire data set (the OOME may still occur, but should occur later each time).
>
> Regarding load order, the basic answer is that it's OS and filesystem dependent - the tool simply lists the contents of each folder and imports items in (approximately) the order that the OS returns them.  The tool does not perform any sorting prior to import, however it does group together metadata and version files, and that can result in a slightly different load order than the OS/FS listing order.
>
> Cheers,
> Peter
>
> [1]http://code.google.com/p/alfresco-bulk-filesystem-import/wiki/Roadmap
>

> On Jan 31, 2013, at 1:58 AM, sfontanel <stefano.fonta...@gmail.com> wrote:
>
>
>
>
>
>
>
> > Hi,
>
> > first of all I would like to thank you for this great tool!
>
> > I have an OutOfMemory error during an "In place" bulk import after an hour using version alfresco-bulk-filesystem-import-1.1.amp on Alfresco ENT 3.4.x.
> > I loaded almost 30 GB of files (as the "status of import" webscript states). The total amount is almost 150 GB.
> > Perhaps my data structure reflects that described in "What steps will reproduce the problem?" in
>

> >http://code.google.com/p/alfresco-bulk-filesystem-import/issues/detai...

[Peter Monks]

G'day Stefano,

That's correct - the import tool has to scan the repository to determine which items on disk haven't yet been imported.  Deterministic load order doesn't help with this, since:

1. The tool is stateless - it has no concept if where it got to last time (and I'm reluctant to change that behaviour - statelessness has a lot of advantages).
2. Even if it did "remember" where it got up to, the tool can't assume that the part of the source directory it previously processed doesn't have new or updated files in it.  In fact the customer the tool was originally implemented for had precisely this requirement (they ran the tool in an infinite loop on an FTP drop folder that was constantly changing).

If you'd like to fix the OOME error yourself, I'd suggest checking out the 3.x branch and applying the patch that was provided in issue #109.  That's going to be a lot simpler than pulling HEAD and trying to merge all of the changes back to 3.x (which I will be doing leading up to the 1.2 release, but which hasn't started yet).

More generally, the issues that have already been marked fixed in the 1.2 release (see the roadmap [1] for details) have been fully tested on Alfresco Enterprise 4.1 and all tests pass (look in src/test for details on the import sets that are used for testing).  I'm in the middle of issue #121, then the 3.x merge work will commence (which will include a fix for issue #110).  The test import sets will then be retested on 4.1 and 3.4 and once they pass 1.2 will be released.  I expect to release 1.2 by the end of March or so.

[Stefano Fontanel]

Hi Peter,

thank you very much for you suggestions!

I will try to fix the bug as you suggest and than split data in small groups to reduce the probability of error..

Thanks

Stefano.

2013/2/3 Peter Monks <pmo...@alfresco.com>

[sfontanel]

Hi Peter,

thank you very much for you suggestions!

I will try to fix the bug as you suggest and than split data in small groups to reduce the probability of error..

Thanks

Stefano.

To unsubscribe from this group and stop receiving emails from it, send an email to alfresco-bulk-filesystem-import+unsubscribe@googlegroups.com.
To post to this group, send email to alfresco-bulk-filesystem-imp...@googlegroups.com.

Visit this group athttp://groups.google.com/group/alfresco-bulk-filesystem-import?hl=en.
For more options, visithttps://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "Alfresco Bulk Filesystem Import" group.

To unsubscribe from this group and stop receiving emails from it, send an email to alfresco-bulk-filesystem-import+unsubscribe@googlegroups.com.
To post to this group, send email to alfresco-bulk-filesystem-imp...@googlegroups.com.