Re: FW: ReDBox/Mint hardware

[Greg Pendlebury]

Hi All... and specifically Grant :)

System specs are always hard to quantify. Here's the short answer for everyone:
Those specs look ok for most use cases, although I'd err on the side of more RAM.

Some broader answers and background data that may help. Sorry for the non-techies as this may put you to sleep.

OS Choice
This is largely unimportant. The codebase is platform neutral, and so obviously is the JVM. Solaris is the situation most likely to cause problems if extra features are required (like video transcoding), and because the Solaris JVM implementation has a different internal implementation with regards to multi-threading and memory allocation... that's just the way Solaris is.

Things like video transcoding aren't an issue for ReDBox and Mint, but memory allocation is. In general, if deploying on Solaris, you might want to pad some extra RAM onto the JVM to account for the difference.

CPU Performance
Is very unimportant. ReDBox and Mint don't have any CPU intensive processes, and they aren't expected to see high volume traffic. Pretty much any CPU that is even vaguely modern is sufficient.

If (as per above) you wanted to do intensive processes like video transcoding this becomes very different, and Solaris would be unsuitable. We have extensive information if more info is required, but I say again that ReDBox and/or Mint don't do this stuff as we have them configured.

Hard Drive space
Is of course situationally dependent.

• The Solr index is reasonably tight, I've put multi-million record datasets into Solr and it still sits at around 2-3GBs on disk.
• The Storage layer depends on what you want to store and/or render. Given that both ReDBox and Mint are primarily registers the storage requirements are very low. Unless you are storing staggering amounts of data you'd could survive on less then 5GB, and I think most installs would struggle to exceed 1GB.
• Attachments will change this scenario, implementers can gauge this for themselves.
• The server itself, and all requiremed binary Java dependencies come in at around 100MB last time I looked... a pittance.

Overall I'd expect the entire server and all the stuff I mentioned above would still give you change from 10GB.

RAM
This is the most important consideration. Java tends to be a memory hungry beast, and it's hard to gauge a requirement without considering WHAT you are doing with the system. The size of the dataset, the nature of the datset, the expected activities of users as well as the volume of user traffic all play a part.

The keys are what total heap space to allocation Java, and how much perm gen space it should have. If either of these areas run out of resources the JVM dies. You can reboot and things are happy at that point, so for picking your specs you are trying to balance avoidance of a crash and availability of resources. On some servers (like dev/test/demo servers) you can save on RAM and cron a reboot nightly for example.

RAM is allocated at JVM startup like so: -XX:MaxPermSize=512m -Xmx1024m and those two numbers are typically what we are most concerned about.

Some example ballpark figures from other systems we run on the same codebase (and some other thoughts):

• We run a Media Repository at USQ with student traffic and fairly intensive video transcoding going on server-side. The above numbers 512m PermGen and 1024m total heap has been sufficient for our production server and it has never run out of RAM. This runs on Red Hat.
• We ran a demo system on Windows 2008 and the total box only had 2GB so I used -XX:MaxPermSize=256m -Xmx512m and noticed periodic outages (a few times a week). Increasing to -XX:MaxPermSize=364m -Xmx768m made all the troubles go away. This was low traffic, but very intensive processing requirements on the data.
• Our development machines tend to be more generously resourced (512/1024) and we don't see problems. This is across the board; Mac, various Linux, Solaris, Windows server and desktop.
  
• Also we make periodic efforts to find and reduce the worst offenders in the system with regards to memory usage. Oliver performed the last round, which will soon (as in next week) makes its way the ReDBox codebase: https://fascinator.usq.edu.au/trac/wiki/Fascinator/Investigations/JythonScriptCaching

Given that these numbers represent just the JVM you also need to account for running the OS and anything else on that box.

Your suggested Ubuntu box looks pretty reasonable, although be aware that Ubuntu usually ships with a non-Sun JVM out-of-the-box that is known to cause problems with many Java apps (including ours, and Solr). The Solaris boxes I'd suggest more RAM for, but you can probably squeeze by with those specs and nightly reboots.

Hope that helps,
Greg

On 7 April 2011 14:26, Duncan Dickinson <Duncan.D...@usq.edu.au> wrote:

-----Original Message-----
From: Grant Jackson [mailto:gra...@csem.flinders.edu.au]
Sent: Monday, 4 April 2011 1:54 PM
To: Duncan Dickinson
Subject: ReDBox/Mint hardware

Hi Duncan,

Thanks alot for your time speaking with me today. I found it very useful
and appreciated your time.

In order for me to attempt to organise some hardware for a pilot system (3
collections of research metadata) could you please provide me with any
available info regarding hardware and Operating System requirements/examples for ReDBox/Mint? As discussed, perhaps info related to your demo or development environments may be suitable.

Also if you know what is planned for production at any of the universities
that would also be useful.


Example 1: DEMO SYSTEM (2 hosts)

ReDBox host:
 Operating system: Solaris 10
 CPU: UltraSParc ... 2GHz
 Minimum RAM: 1GB
 Minimum hard drive space: 20GB (OS + apps but without any metadata)

The Mint host:
 Operating system: Ubuntu 10.04
 CPU: Intel ... Dual core 3GHz
 Minimum RAM: 1GB
 Minimum hard drive space: 20GB (OS + apps but without any metadata)



Example 2: DEVELOPMENT SYSTEM (1 host)

ReDBox / The Mint on same host:
 Operating system: Ubuntu 10.04 as a VMWare guest
 CPU: Intel ... Dual core 3GHz
 Minimum RAM: 2GB
 Minimum hard drive space:   40GB (OS + apps but without any metadata)


Thanks and regards,
Grant


This email (including any attached files) is confidential and is for the
intended recipient(s) only.  If you received this email by mistake,
please, as a courtesy, tell the sender, then delete this email.

The views and opinions are the originator's and do not necessarily
reflect those of the University of Southern Queensland.  Although all
reasonable precautions were taken to ensure that this email contained no
viruses at the time it was sent we accept no liability for any losses
arising from its receipt.

The University of Southern Queensland is a registered provider of
education with the Australian Government (CRICOS Institution Code No's.
QLD 00244B / NSW 02225M)