Configurations For large number of items and concurrent users

[Projects Arunendra]

Hi,

I believe there are related responses scattered through out this forum but I need a summarized idea:

1)What configurations are required when there are large number of items (i.e optimized search)?

2) What configurations are required when there are large number of concurrent users?

Will only sufficient memory allocation to to Tomcat do?

Or,

Configurations are required in the Solar indexing too?

Reqd. for DSpace 5.x and 6.x

[Hrafn Malmquist]

Hi Arunendra

That's a bit of an ask.

Here => https://wiki.lyrasis.org/display/DSDOC6x/Installing+DSpace#InstallingDSpace-HardwareRecommendations

You will find hardware recommendations split into minimum, mid and high-end.

To optimize search you should run the index-discovery job see => https://wiki.lyrasis.org/display/DSDOC6x/Discovery#Discovery-DiscoverySolrIndexMaintenance

Review the "Scheduled Tasks via Cron" for further information about maintenance => https://wiki.lyrasis.org/display/DSDOC6x/Scheduled+Tasks+via+Cron

With regards to concurrent users, the default settings should be fine, but review the database settings => https://wiki.lyrasis.org/display/DSDOC6x/Configuration+Reference#ConfigurationReference-DSpaceDatabaseConfiguration

As well as settings for the database you use in your setup, Postgres or Oracle.

Best regards, Hrafn

--
All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/ce58801a-27d7-49da-9bb7-91f0b1fac1a9n%40googlegroups.com.

[ArunendraMB]

Thanks.

I will check if it works.

For concurrent users, I had come across this post :http://dspace.2283337.n4.nabble.com/Maximum-number-of-DB-connections-in-pool-td4692476.html

The user had problem even after changing the value db.maxconnections.

The solution provided appears to require a lot of changes.

Can you please comment on that?

For large number of items, cleaning and reindexing is definitely an option. But when there are say, 1 lakh items, simply reindexing might not work, I assume.

What to do then?

In terms of hardware/memory allocation, do you have any advice on- Memory or CPU cores?

From,

Arunendra MB.

[Alan Orth]

Dear Arunendra MB,

You will find many examples of users with over 100,000 items in their DSpace repository. We have ~95,000 and it's "fine".

As for specifics, our production server has 32GB of RAM, and we use DSpace 6 with Tomcat 7 and have a heap size of 8192M. I recommend stealing the JVM settings from the Solr 4.10.x init script, as DSpace is essentially a Solr frontend (and it runs Solr version 4.10.x currently). The 32GB of RAM is surely overkill, but above and beyond DSpace itself, we of course have PostgreSQL running and any other leftover memory is automatically used by the Linux kernel as a cache (for example for Solr index data).

We only do a full Discovery re-index once a month, or less, when we do some batch metadata cleanups. That takes about three hours and is an intensive process. Otherwise there is no need to do such expensive indexing.

Also, you should make sure you're running the latest PostgreSQL possible (PostgreSQL 10 with DSpace 6). On our web server we also aggressively limit non-human clients because they are a waste of resources and cause considerable load crawling useless and infinite combinations of /discovery and /browse subject results pages.

Regards,

To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/CAO_tGr%2BxpOnnij5wrUv%3DfLC1vzbpkhLymq-T%3D%2B6Wj6TSKPmf6g%40mail.gmail.com.

--

Alan Orth
alan...@gmail.com
https://picturingjordan.com
https://englishbulgaria.net
https://mjanja.ch