Exception: Java heap space , when setting up ORCID on DSpace 6.2

[joem...@gmail.com]

Dear DSpace Team,

The following is error message received when running index-authority on the server, we have increased memory for java to 24 GB and we have very huge database as this is happening on production site, please advise as our system engineers beleives this error might not be memory related, if this is relating to memory how big should we increase the memory to be ably to re-index successfully. This setup was for first time done on the server and followed this documentation : https://wiki.lyrasis.org/display/DSDOC5x/ORCID+Integration 

Your help is appreciated with hints to get this problem resolved. 

bash-4.2$ ./dspace index-authority 

Retrieving all data

Initialize org.dspace.authority.indexer.DSpaceAuthorityIndexer

Exception: Java heap space

java.lang.OutOfMemoryError: Java heap space

at org.hibernate.internal.util.collections.IdentityMap.entryArray(IdentityMap.java:160)

at org.hibernate.internal.util.collections.IdentityMap.concurrentEntries(IdentityMap.java:76)

at org.hibernate.event.internal.AbstractFlushingEventListener.prepareCollectionFlushes(AbstractFlushingEventListener.java:191)

at org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:93)

at org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:61)

at org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1166)

at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1223)

at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101)

at org.dspace.core.AbstractHibernateDAO.list(AbstractHibernateDAO.java:129)

at org.dspace.handle.dao.impl.HandleDAOImpl.getHandlesByDSpaceObject(HandleDAOImpl.java:61)

at org.dspace.handle.HandleServiceImpl.getInternalHandles(HandleServiceImpl.java:371)

at org.dspace.handle.HandleServiceImpl.findHandle(HandleServiceImpl.java:274)

at org.dspace.identifier.VersionedHandleIdentifierProvider.lookup(VersionedHandleIdentifierProvider.java:330)

at org.dspace.identifier.IdentifierServiceImpl.lookup(IdentifierServiceImpl.java:165)

at org.dspace.content.DSpaceObjectServiceImpl.getIdentifiers(DSpaceObjectServiceImpl.java:75)

at org.dspace.content.ItemServiceImpl.updateLastModified(ItemServiceImpl.java:252)

at org.dspace.content.ItemServiceImpl.updateLastModified(ItemServiceImpl.java:61)

at org.dspace.content.MetadataValueServiceImpl.update(MetadataValueServiceImpl.java:88)

at org.dspace.authority.AuthorityValue.updateItem(AuthorityValue.java:184)

at org.dspace.authority.indexer.DSpaceAuthorityIndexer.prepareNextValue(DSpaceAuthorityIndexer.java:199)

at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:132)

at org.dspace.authority.indexer.DSpaceAuthorityIndexer.hasMore(DSpaceAuthorityIndexer.java:157)

at org.dspace.authority.indexer.AuthorityIndexClient.main(AuthorityIndexClient.java:60)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:229)

at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:81)

-bash-4.2$ 

Regards,

Lewatle

[Mark H. Wood]

On Thu, Mar 18, 2021 at 03:29:43AM -0700, joem...@gmail.com wrote:
> Dear DSpace Team,
>
> The following is error message received when running index-authority on the
> server, we have increased memory for java to 24 GB and we have very huge
> database as this is happening on production site, please advise as our
> system engineers beleives this error might not be memory related, if this
> is relating to memory how big should we increase the memory to be ably to
> re-index successfully. This setup was for first time done on the server and
> followed this documentation
> : https://wiki.lyrasis.org/display/DSDOC5x/ORCID+Integration
> Your help is appreciated with hints to get this problem resolved.

It would be helpful to know precisely what version of DSpace you are
running, in order to pinpoint the failure.

At about line 60 of AuthorityIndexClient in v5 the code is filling a
Map with every authority value of every Item in the database. So,
while I can't give you a number for the amount of memory required by
this operation, it is directly proportional to the number of Items you
have. That line is called out in the stack trace.

This code could be made much more scalable. In essence it fills a
giant list, clears the index, then goes through the list sequentially
and adds each entry to the index. There's no need for the list;
entries could be added to the cleared index as they are produced.

This scalability issue (and perhaps another) should be fixed
regardless, but I'd like to confirm that I know where the failure
occurred before I write up a bug ticket. Please tell us the complete
version of DSpace you're running. (Like: 5.6) 'bin/dspace version'
will give you that.

--
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu

[FILIPPOS KOLOVOS]

Dear Lewatle,

When you say that you have increased memory for Java you mean that apart from the tomcat settings, you have also increased it in the ./dspace binary file that performs the indexing? Also, how large is your DB? 24GB RAM is huge!!!

I would suggest that you check the ./dspace file of your running dspace instance editing it with your text editor. Then, in the file search and find the line:

JAVA_OPTS="-Xmx256m -Dfile.encoding=UTF-8"

and change it to something really high:

 JAVA_OPTS="-Xmx22000m -Xms20000m -Dfile.encoding=UTF-8"

This will specify that the dspace TOOL (which is the file you are running for the indexing) will use from 20000MB up to 22000MB RAM for its task and not just 256MB. However, you must have more than 22000MB RAM available on the server (i.e. 24GB which you have), or the script will fail again and it might hang your server. If you want to allocate LESS than 22GB, then adjust these values accordingly (i.e. Xmx16000m Xms14000m, etc).

The problem with the "dspace" commands is that even if your server has 32GB RAM, the dspace tool will only use up to the RAM specified in this file and there are many occasions that lead to this error.

Now, if you have done that, then there must be something REALLY inefficient in the code for large DBs that leads to OOM Error and we'll try to figure it out.

I hope that I have helped!

--
All messages to this mailing list should adhere to the Code of Conduct: https://duraspace.org/about/policies/code-of-conduct/
---
You received this message because you are subscribed to the Google Groups "DSpace Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-communi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-community/d00e1bda-8a67-477b-8a67-4b09db450737n%40googlegroups.com.

[joem...@gmail.com]

Dear Filippos,

Thank you very much for sharing this helpful information, I will try it on my side and reply.

Regards,

Lewatle 

[joem...@gmail.com]

Dear Filippos,

Thank you very much, indeed after changing memory settings in ../bin/dspace file I have managed to re-index

Much appreciated,

Lewatle

On Tuesday, 23 March 2021 at 23:20:39 UTC+2 FILIPPOS KOLOVOS wrote: