Managing memory

[ealt...@gmail.com]

Looking for tips on managing memory.  We have a v3.4.1 system used by 3-8 people per day, and occasionally there are problems like reports taking forever to complete, or ingests of metadata not completing -- that sort of thing, for big jobs.

I just cleared one such jam by restarting SOLR (clearing the cache), but the system is still using about 75% of its 16GB of RAM (really only about 8GB is the Java/solr/aspace stack itself)

I increased Java heap to 4096 which may be one of the issues, but I did this to clear other issues we were having.

Curious what the rest of you are doing to manage memory needs?  I'd like to be able to set up logs to see what causes spikes, figure out if there's a way to increase bandwidth.  

I know almost nothing about memory management.  Any insight or recommendation welcome.

Thanks,

Elizabeth

[Dave Mayo]

If restarting SOLR fixed some of the problem, and you're using either the internal SOLR or have them running on the same host,  the first two things I'd suggest you look at are:

1. disk space - it's not well documented, but a solr instance needs to always have disk space free equal to or slightly exceeding the total amount of data indexed in order to work well.

2. running solr externally and moving solr to another host.

A full ASpace instance is running multiple applications even without solr in the mix - the staff interface, public interface, backend, and indexer are all separate apps, each of which gets a separate JVM instance and a separate RAM allotment in default config, I believe.  16 gigabytes doesn't go as far as you'd expect given that, and contention for resources gets worse if solr is also in the mix.

- Dave Mayo

Senior Digital Library Software Engineer

Harvard University -> HUIT -> LTS

--
You received this message because you are subscribed to the Google Groups "Non-Member ArchivesSpace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nonmemberarchives...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/nonmemberarchivesspace/a037136e-369a-4861-ae50-290383e2e36fn%40googlegroups.com.

[ealt...@gmail.com]

Thanks.  I'm not adverse to running solr on a different server, but I don't see how that fixes my memory (RAM) problem.  We are not using the front-end interface - our EAD files get built in archivesspace and then the EAD is exported to an external index on a completely different server.  The archivesspace server has 100GB of storage, 29GB of which are used (at the moment).  As I mentioned, it was the memory cache solr was using that was the problem, and I don't see how that gets solved by running it on a different server.  I can see how that might mitigate CPU spikes affecting the archivesspace application when the index has to be rebuilt, but that was not our problem yesterday.

Can you explain further?  I really need some guidance on managing java/solr throughput.

Thanks,

Elizabeth

[Dave Mayo]

So, if JVM settings on ASpace are 8gb, and you're running say ASpace with the backend and staff interface, and indexer  only running, with JVM settings for 8gb max heap, that means ASpace is going to expect up to 24gb of ram (8*3) and Solr will be another chunk sized at whatever JVM settings apply to solr.  In the internal index case, I think that ends up being another 8gb? But, in any event, contention between solr and ASpace for memory and other resources will often be pretty intense, due to most ASpace write and read operations also meaning at least one and sometimes many operations in Solr.

In ASpace 4.x, external solr is the only option, partly to decrease maintenance burden, but also because separating them allows for better performance because the apps aren't contending to perform as fast as possible with no knowledge of each other's constraints.

My recommendation is to have ASpace on one server, MySQL another (or provided via a service like Amazon's RDS) and Solr on a third - that will let you determine which is actually causing your memory pressure trivially, will likely reduce that pressure even over the same set of split resources somewhat, and will be adjustable in a fine-grained way (i.e. if Solr is under pressure you can just bump solr, etc).

Does that make sense?

- Dave

[ealt...@gmail.com]

Yes, thank you, that makes sense.  I may need more guidance setting it up with 3 servers :)  But I will give it a try.

-Elizabeth

[ealt...@gmail.com]

I have a rather naiive question - I have been maintaining a production server and a dev server so I can test security fixes, and upgrades.  If I want to build a 3-server solution, and have a test/dev space, how does this seem?

1 mysql server (one or two separate databases)

2 solr servers

2 aspace servers

Or do I really just need a dev server for the archivesspace development?  My group is now seriously planning to customize the front end, which until last week everyone agreed we would not use.

Thank you for your feedback.

-Elizabeth

[Dave Mayo]

I would probably be shyer of combining MySQL servers for dev and prod than using a single solr for both, tbh - MySQL (databases in general) hit limiting factors around number of connections faster than solr will.  Ideally I'd separate the environments completely, but setting up two solr cores is no harder than two MySQL databases, and the dev won't be taking up a significant amount of resources while idle. 

So, all told:

You absolutely need two aspace servers.

You probably ought to have two mysql servers.

Ideally have two solr servers, but if you need to combine them it is no big deal.

- Dave

--

You received this message because you are subscribed to the Google Groups "Non-Member ArchivesSpace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nonmemberarchives...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/nonmemberarchivesspace/e8524b4f-1cbf-4b2c-b420-bac15501d048n%40googlegroups.com.