ASE 11.9.2 using too much memory on HP/UX?

[Richard Smerker]

We have an HP/UX 11 system with 5GB of RAM and 2 processors. The main
application is ASE 11.9.2 - With little else running on the box.

We recently flagged the dataserver executable to SHMEM_MAGIC using
chatr, and reconfigured the total memory parameter to 2.75GB. It
worked, and according to the ipcs report, it is using 2.75GB of RAM,
and according to swapinfo -tam the memory used is at %100.

After changing this flag, we started having problems with other
applications, such as remsh giving fork errors, unable to allocate
memory errors, and finally the inability for new users to connect to
the system. Obviously this was due to the change in memory
configuration in sybase, we backed out our changes, and the problems
went away.

My question is this : Is there a rule of thumb for figuring how much
memory to leave the O/S? We were going to try again with 2GB, thereby
leaving the 4th quadrant of shared memory alone, but we are afraid
that we would bump into the problem again. Does anyone have any
experience with this?

Thanks!

Rich Smerker