oninit processes are huge
Geezer From The Freezer
I have just had my informix configuration tweaked by an informix engineer and
now we have noticed the oninit processes are huge (they may have been huge
before
but we never noticed). Each one tends to be 500Mb in size. Thats 13 x 500Mb
oninit
processes!!
As this machine (Solaris 2.6) only has 2Gb, this is causing it to swap an awful
lot
and is likely impacting our application.
Is there anyway to deal with this other than adding more memory, which is not
possible as
this machine is fully populated?
Captain Pedantic
news:40D6A343...@Freezer.com...
> Informix V7.30UC4 on Solaris 2.6 2x450Mhz/2048Mb
>
> I have just had my informix configuration tweaked by an informix engineer
and
> now we have noticed the oninit processes are huge (they may have been huge
> before
> but we never noticed). Each one tends to be 500Mb in size. Thats 13 x
500Mb
> oninit
> processes!!
>
> As this machine (Solaris 2.6) only has 2Gb, this is causing it to swap an
awful
> lot
> and is likely impacting our application.
Which engineer?
What changes were made?
Fernando Nunes
> Informix V7.30UC4 on Solaris 2.6 2x450Mhz/2048Mb
>
> I have just had my informix configuration tweaked by an informix engineer and
> now we have noticed the oninit processes are huge (they may have been huge
> before
> but we never noticed). Each one tends to be 500Mb in size. Thats 13 x 500Mb
> oninit
> processes!!
NO!...
Informix allocates chunks of _shared_ memory. The 500MB you see are
probably shared among all the oninit processes. Please check the
"BUFFER" and SHMVIRTSIZE parameters in your $INFORMIXDIR/etc/$ONCONFIG file.
Also check the page size of your system (it's 2KB for Solaris).
So, calculate 2KB * BUFFERS + SHMVIRTSIZE
It should be around 500MB.
Take a very carefull look at the memory allocation... it may not be your
oninit processes which are eating your memory.
Geezer From The Freezer
Fernando Nunes wrote:
> NO!...
> Informix allocates chunks of _shared_ memory. The 500MB you see are
> probably shared among all the oninit processes. Please check the
> "BUFFER" and SHMVIRTSIZE parameters in your $INFORMIXDIR/etc/$ONCONFIG file.
> Also check the page size of your system (it's 2KB for Solaris).
>
> So, calculate 2KB * BUFFERS + SHMVIRTSIZE
> It should be around 500MB.
>
> Take a very carefull look at the memory allocation... it may not be your
> oninit processes which are eating your memory.
it's 4048000Kb
top gives this:-
12441 informix 2 59 -10 499M 454M sleep 308:43 0.02% oninit
12445 informix 2 59 -10 499M 450M sleep 45:43 0.02% oninit
12667 informix 1 59 -10 499M 440K sleep 0:00 0.00% oninit
12666 informix 1 59 -10 499M 416K sleep 0:00 0.00% oninit
12449 informix 1 59 -10 498M 2040K sleep 0:00 0.00% oninit
12448 informix 1 59 -10 498M 1112K sleep 0:00 0.00% oninit
12447 informix 1 59 -10 498M 1080K sleep 0:00 0.00% oninit
12454 informix 1 59 -10 498M 1080K sleep 0:00 0.00% oninit
12446 informix 1 59 -10 498M 1080K sleep 0:00 0.00% oninit
12444 informix 1 59 -10 498M 1040K sleep 0:00 0.00% oninit
12455 informix 1 59 -10 498M 1024K sleep 0:00 0.00% oninit
12456 informix 1 59 -10 498M 960K sleep 0:00 0.00% oninit
Geezer From The Freezer
I'm not going to name the engineer.
BUFFERS increased
LOCKS increased
SHMBASE changed
Logical Log size/number increased
CLEANERS increased
Fernando Nunes
> it's 4048000Kb
>
> top gives this:-
>
> 12441 informix 2 59 -10 499M 454M sleep 308:43 0.02% oninit
> 12445 informix 2 59 -10 499M 450M sleep 45:43 0.02% oninit
> 12667 informix 1 59 -10 499M 440K sleep 0:00 0.00% oninit
> 12666 informix 1 59 -10 499M 416K sleep 0:00 0.00% oninit
> 12449 informix 1 59 -10 498M 2040K sleep 0:00 0.00% oninit
> 12448 informix 1 59 -10 498M 1112K sleep 0:00 0.00% oninit
> 12447 informix 1 59 -10 498M 1080K sleep 0:00 0.00% oninit
> 12454 informix 1 59 -10 498M 1080K sleep 0:00 0.00% oninit
> 12446 informix 1 59 -10 498M 1080K sleep 0:00 0.00% oninit
> 12444 informix 1 59 -10 498M 1040K sleep 0:00 0.00% oninit
> 12455 informix 1 59 -10 498M 1024K sleep 0:00 0.00% oninit
> 12456 informix 1 59 -10 498M 960K sleep 0:00 0.00% oninit
It doesn't match...
Try "onstat -g seg" and also post your onconfig please.
But as you can see they all seem to be around 499MB. The first column
should be RSS and the second VSZ.
Are you sure about page swap?
Regards.
Geezer From The Freezer
Informix Dynamic Server Version 7.31.UC4 -- On-Line -- Up 5 days 05:41:50 --
501440 Kbytes
Segment Summary:
id key addr size ovhd class blkused blkfree
1701 1381386241 2000000 464322560 7956 R 56673 7
406 1381386243 1dad0000 49152000 1348 V 2762 3238
Total: - - 513474560 - - 59435 3245
(* segment locked in memory)
onconfig file
#**************************************************************************
#
# INFORMIX SOFTWARE, INC.
#
# Title: onconfig.std
# Description: Informix Dynamic Server Configuration Parameters
#
#**************************************************************************
# Root Dbspace Configuration
ROOTNAME rootdbs # Root dbspace name
ROOTPATH /usr/informix/data/data1
# Path for device containing root dbspace
ROOTOFFSET 0 # Offset of root dbspace into device (Kbytes)
ROOTSIZE 200000 # Size of root dbspace (Kbytes)
# Disk Mirroring Configuration Parameters
MIRROR 1 # Mirroring flag (Yes = 1, No = 0)
MIRRORPATH /informix/rootdbs1.mir
# Path for device containing mirrored root
MIRROROFFSET 0 # Offset into mirrored device (Kbytes)
# Physical Log Configuration
PHYSDBS physspace # Physical log file size (Kbytes)
PHYSFILE 32768 # Physical log file size (Kbytes)
# Logical Log Configuration
LOGFILES 81 # Number of logical log files
LOGSIZE 16384 # Logical log size (Kbytes)
# Diagnostics
MSGPATH /var/telelogic/informix/online.log
# System message log file path
CONSOLE /var/telelogic/informix/console.msh
# System console message path
ALARMPROGRAM /usr/informix/etc/log_full.sh # Alarm program path
SYSALARMPROGRAM /usr/informix/etc/evidence.sh # System Alarm program path
TBLSPACE_STATS 1
# System Archive Tape Device
#TAPEDEV /informix/backup # Tape device path
TAPEDEV /dev/null # Tape device path
TAPEBLK 256 # Tape block size (Kbytes)
TAPESIZE 20480000 # Maximum amount of data to put on tape (Kbytes)
# Log Archive Tape Device
LTAPEDEV /dev/null # Log tape device path
LTAPEBLK 256 # Log tape block size (Kbytes)
LTAPESIZE 128000 # Max amount of data to put on log tape (Kbytes)
# Optical
STAGEBLOB
# System Configuration
SERVERNUM 0 # Unique id corresponding to a Dynamic Server in
stance
DBSERVERNAME test # Name of default database server
DBSERVERALIASES # List of alternate dbservernames
DEADLOCK_TIMEOUT 60 # Max time to wait of lock in distributed env.
RESIDENT 0 # Forced residency flag (Yes = 1, No = 0)
MULTIPROCESSOR 1 # 0 for single-processor, 1 for multi-processor
NUMCPUVPS 2 # Number of user (cpu) vps
SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vps to one
NOAGE 1
AFF_SPROC 0 # Affinity start processor
AFF_NPROCS 0 # Affinity number of processors
# Shared Memory Parameters
LOCKS 550000 # Maximum number of locks
BUFFERS 200000 # Maximum number of shared buffers
NUMAIOVPS 4 # Number of IO vps
PHYSBUFF 1024 # Size of root dbspace (Kbytes)
LOGBUFF 1024 # Logical log buffer size (Kbytes)
LOGSMAX 200 # Maximum number of logical log files
CLEANERS 32 # Number of buffer cleaner processes
SHMBASE 0x2000000 # Shared memory base address
SHMVIRTSIZE 48000 # initial virtual shared memory segment size
SHMADD 8192 # Size of new shared memory segments (Kbytes)
SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited
CKPTINTVL 300 # Check point interval (in sec)
LRUS 32 # Number of LRU queues
LRU_MAX_DIRTY 2 # LRU percent dirty begin cleaning limit
LRU_MIN_DIRTY 1 # LRU percent dirty end cleaning limit
LTXHWM 50 # Long transaction high water mark percentage
LTXEHWM 60 # Long transaction high water mark (exclusive)
TXTIMEOUT 0x4b0 # Transaction timeout (in sec)
STACKSIZE 32 # Stack size (Kbytes)
# System Page Size
# BUFFSIZE - Dynamic Server no longer supports this configuration parameter.
# To determine the page size used by Dynamic Server on your platform
# see the last line of output from the command, 'onstat -b'.
# Recovery Variables
# OFF_RECVRY_THREADS:
# Number of parallel worker threads during fast recovery or an offline restore.
# ON_RECVRY_THREADS:
# Number of parallel worker threads during an online restore.
OFF_RECVRY_THREADS 10 # Default number of offline worker threads
ON_RECVRY_THREADS 1 # Default number of online worker threads
# Data Replication Variables
# DRAUTO: 0 manual, 1 retain type, 2 reverse type
DRAUTO 0 # DR automatic switchover
DRINTERVAL 30 # DR max time between DR buffer flushes (in sec)
DRTIMEOUT 30 # DR network timeout (in sec)
DRLOSTFOUND /usr/informix/etc/dr.lostfound # DR lost+found file path
# CDR Variables
CDR_LOGBUFFERS 2048 # size of log reading buffer pool (Kbytes)
CDR_EVALTHREADS 1,1 # evaluator threads (per-cpu-vp,additional)
CDR_DSLOCKWAIT 5 # DS lockwait timeout (seconds)
CDR_QUEUEMEM 4096 # Maximum amount of memory for any CDR queue (Kb
ytes)
CDR_LOGDELTA 30 # % of log space allowed in queue memory
CDR_NUMCONNECT 16 # Expected connections per server
CDR_NIFRETRY 300 # Connection retry (seconds)
CDR_NIFCOMPRESS 0 # Link level compression (-1 never, 0 none, 9 ma
x)
# Backup/Restore variables
BAR_ACT_LOG /tmp/bar_act.log
BAR_MAX_BACKUP 0
BAR_RETRY 1
BAR_NB_XPORT_COUNT 10
BAR_XFER_BUF_SIZE 31
# Informix Storage Manager variables
ISM_DATA_POOL ISMData # If the data pool name is changed, be sure to
# update $INFORMIXDIR/bin/onbar. Change to
# ism_catalog -create_bootstrap -pool <new name>
ISM_LOG_POOL ISMLogs
# Read Ahead Variables
RA_PAGES 16
RA_THRESHOLD 8
# DBSPACETEMP:
# Dynamic Server equivalent of DBTEMP for SE. This is the list of dbspaces
# that the Dynamic Server SQL Engine will use to create temp tables etc.
# If specified it must be a colon separated list of dbspaces that exist
# when the Dynamic Server system is brought online. If not specified, or if
# all dbspaces specified are invalid, various ad hoc queries will create
# temporary files in /tmp instead.
DBSPACETEMP temp1:temp2:temp3:temp4 # Default temp dbspaces
# DUMP*:
# The following parameters control the type of diagnostics information which
# is preserved when an unanticipated error condition (assertion failure) occurs
# during Dynamic Server operations.
# For DUMPSHMEM, DUMPGCORE and DUMPCORE 1 means Yes, 0 means No.
DUMPDIR /tmp # Preserve diagnostics in this directory
DUMPSHMEM 1 # Dump a copy of shared memory
DUMPGCORE 0 # Dump a core image using 'gcore'
DUMPCORE 0 # Dump a core image (Warning:this aborts Dynamic
Server)
DUMPCNT 1 # Number of shared memory or gcore dumps for
# a single user's session
FILLFACTOR 75 # Fill factor for building indexes
# method for Dynamic Server to use when determining current time
USEOSTIME 0 # 0: use internal time(fast), 1: get time from O
S(slow)
# Parallel Database Queries (pdq)
MAX_PDQPRIORITY 100 # Maximum allowed pdqpriority
DS_MAX_QUERIES # Maximum number of decision support queries
DS_TOTAL_MEMORY # Decision support memory (Kbytes)
DS_MAX_SCANS 1048576 # Maximum number of decision support scans
DATASKIP off # List of dbspaces to skip
# OPTCOMPIND
# 0 => Nested loop joins will be preferred (where
# possible) over sortmerge joins and hash joins.
# 1 => If the transaction isolation mode is not
# "repeatable read", optimizer behaves as in (2)
# below. Otherwise it behaves as in (0) above.
# 2 => Use costs regardless of the transaction isolation
# mode. Nested loop joins are not necessarily
# preferred. Optimizer bases its decision purely
# on costs.
OPTCOMPIND 0 # To hint the optimizer
ONDBSPACEDOWN 2 # Dbspace down option: 0 = CONTINUE, 1 = ABORT,
2 = WAIT
LBU_PRESERVE 0 # Preserve last log for log backup
OPCACHEMAX 128
# HETERO_COMMIT (Gateway participation in distributed transactions)
# 1 => Heterogeneous Commit is enabled
# 0 (or any other value) => Heterogeneous Commit is disabled
HETERO_COMMIT 0
# Optimization goal: -1 = ALL_ROWS(Default), 0 = FIRST_ROWS
OPT_GOAL -1
# Optimizer DIRECTIVES ON (1/Default) or OFF (0)
DIRECTIVES 1
# Status of restartable restore
RESTARTABLE_RESTORE off
# Non-template variable settings
# NETTYPE settings
NETTYPE ipcstr,1,128,CPU # Configure poll thread(s) for nettype
Fernando Nunes
> onstat -g
>
> Informix Dynamic Server Version 7.31.UC4 -- On-Line -- Up 5 days 05:41:50 --
> 501440 Kbytes
As you can see Informix is "only" using about 500MB of memory.
~= 464MB of BUFFERS, locks and other minor stuff and 48MB of Virtual
segment size (for session connections, sorts, index creation, hash
tables etc.)
This should be about 1/4 of your system memory. If it's paging it's
because other processes are consuming about 1.5GB of memory.
If I'm not mistaken Solaris tends to mount /tmp in memory... Maybe it's
worth to check this out.
>
> Segment Summary:
> id key addr size ovhd class blkused blkfree
> 1701 1381386241 2000000 464322560 7956 R 56673 7
> 406 1381386243 1dad0000 49152000 1348 V 2762 3238
> Total: - - 513474560 - - 59435 3245
>
> # Shared Memory Parameters
> LOCKS 550000 # Maximum number of locks
> BUFFERS 200000 # Maximum number of shared buffers
> NUMAIOVPS 4 # Number of IO vps
> PHYSBUFF 1024 # Size of root dbspace (Kbytes)
> LOGBUFF 1024 # Logical log buffer size (Kbytes)
> LOGSMAX 200 # Maximum number of logical log files
> CLEANERS 32 # Number of buffer cleaner processes
> SHMBASE 0x2000000 # Shared memory base address
> SHMVIRTSIZE 48000 # initial virtual shared memory segment size
200000 * 2KB + 48000KB ~= 448000KB. The difference is the lock table and
other "minor" structures.
You can check the administrator guide and reference for the details.
Please check what else is consuming your memory. Eithe the application
or the /tmp filesystem maybe. Or you may hav a memory leak...
Regards.
Geezer From The Freezer
Fernando,
Thanks for that. I find it very bizarre that Informix spawns more than
one oninit process and that each one seems to have a different amount
in resident memory.....
Fernando Nunes
Each process attachs to the shared memory segment(s).
besides this memory, each oninit uses it's own "private" memory.
Please remember that each oninit is doing different things.
Execute "onstat -g glo" and you'll see all the different virtual
processors and their class (cpu, aio, kio, msc, lio etc...)
Each one is doing specific things and each one requires different memory
for their action. All of them attach to the same segments of shared
memory because all of them require access to the shared structures.
Nothing strange or bizarre in this.
By the way, the shared segments may or may not be in real memory.
Regards.
David Williams
"Geezer From The Freezer" <Gee...@Freezer.com> wrote in message
use pmap to print out the segments for each one..you should be able to
identify the shared ones...
Richard Kofler
>
>
> I have just had my informix configuration tweaked by an informix engineer and
> now we have noticed the oninit processes are huge (they may have been huge
> before
> but we never noticed). Each one tends to be 500Mb in size. Thats 13 x 500Mb
> oninit
> processes!!
>
> As this machine (Solaris 2.6) only has 2Gb, this is causing it to swap an awful
where did you see that there is any swapping?
use vmstat (man vmstat will point you to the 2 valuable columns 'po'
and 'sr')
use mount and then iostat to monitor all I/O activity on your swap
device(s)
use RESIDENT -1 in $ONCONFIG to pinn all R and V segments into memory.
From the few lines of top output missing the first 3 lines, I cannot
tell
nothing, really.
What does top say for swap / swap in use / memory?
Is your /tmp mounted as tmpfs?
mount will tell you, if so. And if it is, watch it a bit, as
memory and swap and space on /tmp is all the same.
If it is, modify $ONCONFIG to *not* let your engine write
af dump files in /tmp.
> lot
> and is likely impacting our application.
>
> Is there anyway to deal with this other than adding more memory, which is not
> possible as
> this machine is fully populated?
pls specify 'this' in ... this machine ...
most of hardware capable to run SOL2.6 should be able to be equipped
with
4GB memory, if it is not a TADPOLE laptop.
dic_k
--
Richard Kofler
SOLID STATE EDV
Dienstleistungen GmbH
Vienna/Austria/Europe
Art S. Kagel
> 499M 454M
This means that the memory addressed by the oninit is 499MB but 454MB is
resident. Most of the resident size of oninit is shared memory as Fernando
pointed out. Run oninit -. The end of the header line reports the total size
of shared memory. This is allocated ONCE for the instance and shared. To see
the memory footprint of each oninit instance in memory, subtract that number
from the 499MB reported by top or ps. You will be surprise how small it is,
and most of that is code space which is also shared.
size -F $INFORMIXDIR/bin/oninit will report the writable and read-only memory
sizes in bytes. On my Sun IDS 7.3 reports ~380K of writable and therefore
non-shared memory.
I'd guess your server is actually using ~480MB of shared memory plus 1/4 - 1/2
MB times the number of oninit processes.
Art S. Kagel
Paul Watson
Geezer From The Freezer wrote:
>
> Informix V7.30UC4 on Solaris 2.6 2x450Mhz/2048Mb
>
> I have just had my informix configuration tweaked by an informix engineer and
> now we have noticed the oninit processes are huge (they may have been huge
> before
> but we never noticed). Each one tends to be 500Mb in size. Thats 13 x 500Mb
> oninit
> processes!!
>
> As this machine (Solaris 2.6) only has 2Gb, this is causing it to swap an awful
> lot and is likely impacting our application.
With this amount of memory, the chances of Solaris actually swapping is
virtually nil. What makes you think you are swapping, if you mean the
w column in vmstat, then IMHO you are wrong - this column is meaningless
from 2.6 onwards. If you mean the de and sr columns then this is
paging,
you can not run Solaris without any paging activity - it is impossible.
>
> Is there anyway to deal with this other than adding more memory, which is not
> possible as
> this machine is fully populated?
--
Paul Watson #
Oninit Ltd # Growing old is mandatory
Tel: +44 1436 672201 # Growing up is optional
Fax: +44 1436 678693 #
Mob: +44 7818 003457 #
www.oninit.com #
Geezer From The Freezer
Informix Dynamic Server Version 7.31.UC4 -- On-Line -- Up 7 days 00:30:25 --
501440 Kbytes
MT global info:
sessions threads vps lngspins
5 56 12 0
sched calls thread switches yield 0 yield n yield forever
total: 258468812 170122003 91483743 1227117 23136469
per sec: 18 6 0 3 0
Virtual processor summary:
class vps usercpu syscpu total
cpu 2 22429.71 6846.61 29276.32
aio 4 0.07 0.47 0.54
lio 2 0.02 0.14 0.16
pio 2 0.04 0.27 0.31
adm 1 0.07 0.47 0.54
msc 1 0.07 0.09 0.16
total 12 22429.98 6848.05 29278.03
Individual virtual processors:
vp pid class usercpu syscpu total
1 12441 cpu 19151.69 6011.15 25162.84
2 12444 adm 0.07 0.47 0.54
3 12445 cpu 3278.02 835.46 4113.48
4 12446 lio 0.01 0.09 0.10
5 12447 pio 0.03 0.17 0.20
6 12448 aio 0.02 0.15 0.17
7 12449 msc 0.07 0.09 0.16
8 12454 aio 0.01 0.16 0.17
9 12455 aio 0.03 0.07 0.10
10 12456 aio 0.01 0.09 0.10
11 12666 lio 0.01 0.05 0.06
12 12667 pio 0.01 0.10 0.11
tot 22429.98 6848.05 29278.03