DSpace 9 aip ingest extremely slow
27 views
Skip to first unread message
Qing Zou
Jun 2, 2025, 11:13:03 PM6/2/25
to DSpace Technical Support
Hi all,
I attempted to use AIP to ingest an entire site from version 6.4 to 9 (approximately 24 GB/ 5,000 handles) on two different test machines (both running Rocky Linux 9 and Java 17). One full ingestion took around 72 hours. I observed that the process at the very beginning was relatively quick but slowed down significantly as it progressed. I also increased the heap size from 512 MB to 12 GB, but this change had little noticeable effect on performance.
Any suggestions are highly appreciated.
By the way, I used default hibernate settings.
Best,
Jason
Michael Plate
Jun 3, 2025, 6:23:31 AM6/3/25
to dspac...@googlegroups.com
Hi Jason,
Am 03.06.25 um 05:13 schrieb 'Qing Zou' via DSpace Technical Support:
this is usual.
And congrats it did run in one attempt :) .
You can speed it up by splitting the import into multiple parts. AFAIK
this is because the ingest is done in one big database transaction and
it gets slower the more data is available for commit / rollback.
Michael
Am 03.06.25 um 05:13 schrieb 'Qing Zou' via DSpace Technical Support:
> Hi all,
>
> I attempted to use AIP to ingest an entire site from version 6.4 to 9
> (approximately 24 GB/ 5,000 handles) on two different test machines
> (both running Rocky Linux 9 and Java 17). One full ingestion took around
> 72 hours. I observed that the process at the very beginning was
> relatively quick but slowed down significantly as it progressed. I also
> increased the heap size from 512 MB to 12 GB, but this change had little
> noticeable effect on performance.
[…]
>
> I attempted to use AIP to ingest an entire site from version 6.4 to 9
> (approximately 24 GB/ 5,000 handles) on two different test machines
> (both running Rocky Linux 9 and Java 17). One full ingestion took around
> 72 hours. I observed that the process at the very beginning was
> relatively quick but slowed down significantly as it progressed. I also
> increased the heap size from 512 MB to 12 GB, but this change had little
> noticeable effect on performance.
this is usual.
And congrats it did run in one attempt :) .
You can speed it up by splitting the import into multiple parts. AFAIK
this is because the ingest is done in one big database transaction and
it gets slower the more data is available for commit / rollback.
Michael
Qing Zou
Jun 3, 2025, 11:30:24 AM6/3/25
to Michael Plate, dspac...@googlegroups.com
Thanks Michael for your confirmation that I didn't do anything wrong. I suspect that is the case with a big database transaction.
All the best,
Jason
--
All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dspace-tech/12fa65d2-cb18-4d55-b1ed-91b7df07a3d5%40bibliothek.uni-kassel.de.
Dr. Qing (Jason) Zou, MLIS, PhD,
Head, Digital Initiatives
Lakehead University Library
email: qz...@lakeheadu.caLakehead University Library
Reply all
Reply to author
Forward
0 new messages