Online hangs during oninit after reboot.
Mark Stannard
Folks
We are running Online Dynamic Server 7.23.UC1 on an RS6000 with AIX
4.14. Currently we reboot the machine every Saturday morning. Several
times the Online instance has hung during the Initialization phase after the
reboot. Has anyone else seen anything like this? We shut the server
down with an onmode -ky and restart it with oninit.
Any ideas would be appreciated.
Thanks,
Mark Stannard
Ameritech
220 N. Meridian
Indianapolis, IN 46204
Voice: 317-265-0613
Fax: 317-265-1535
Debbie Ingram
When you issue the onint are you sure that the Unix box is completely
up. Ex1: TCP has not been started and onint is issued from the
console Ex2: disk drives for the rootdbs are not available
I have also had trouble where the instance did not come down clean and
the /INFORMIXTMP directory with 7.2+ was not cleaned up. A file gets
created for each VP and they get deleted when you shut down the
instance. However it looks like from your online log you are shutting
down cleanly.
Regards,
Debbie
m...@appi.ci.in.ameritech.com (Mark Stannard) wrote:
>Folks,
>
>Sorry I should have provided a few more details initially.
>
>Here is the squence of events:
>
>03:00
>1. We shut down the engine with onmode -ky (probably no users present)
>2. The RS6000 box is rebooted.
>3. We restart the engine with oninit
>
>It during step 3 that the online instance hangs. Below are the lines
>from the online.log file.
>
> 02:05:40 Checkpoint Completed: duration was 0 seconds.
> 03:03:16 INFORMIX-OnLine Stopped.
> 03:13:51 VP pid=13632 priority fixed at 65, former = 114
>
> Sat Aug 23 03:13:58 1997
>
> 03:13:58 Event alarms enabled. ALARMPROG = '/usr/informix/log_full.sh'
> 03:13:58 VP pid=21060 priority fixed at 65, former = 65
> 03:13:58 VP pid=21318 priority fixed at 65, former = 65
>
>This is where is sits until I kill the VPs. If you do a onstat - the
>state will indicat Initializing. After the VPs are killed I issue a oninit
>and the engine starts up okay.
>
>This is the line from online.log indicating that I killed the VPs.
> 04:51:19 mt.c, line 9319, thread 20, proc id 13632, Unexpected virtual
> processor termination, pid = 24932, exit = 0x9
>
>
>And this is from where I issued the oninit.
> 04:54:12 VP pid=25906 priority fixed at 65, former = 120
>
> Sat Aug 23 04:54:19 1997
>
> 04:54:19 Event alarms enabled. ALARMPROG = '/usr/informix/log_full.sh'
> 04:54:19 VP pid=23610 priority fixed at 65, former = 65
> 04:54:19 VP pid=25404 priority fixed at 65, former = 65
> 04:54:23 DR: DRAUTO is 0 (Off)
> 04:54:23 AIX MP latch code enabled
> 04:54:24
> 04:54:24
> 04:54:24 INFORMIX-OnLine Initialized -- Shared Memory Initialized.
> 04:54:24 Physical Recovery Started.
> 04:54:24 Physical Recovery Complete: 0 Pages Restored.
> 04:54:24 Logical Recovery Started.
> 04:54:28 Logical Recovery Complete.
> 0 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks
>
> 04:54:28
> 04:54:28 Dataskip is now OFF for all dbspaces
> 04:54:28 On-Line Mode
> 04:54:28 Checkpoint Completed: duration was 0 seconds.
>
>Any ideas?
John
Yep, I too bounce the whole box to clear up all memory nasties / zombies
once a week.
I've got ODS fired up / closed down via an rc3.d entry. (Also scrub /tmp
out on boot etc).
For the operators, we have a 'reboot' and 'shutdown' user.
I've found that Online occasionally screws up and leaks into memory /
doesn't release memory:
ipcs -m Gives a listing of memory currently alloc.
If you see user Informix, and no Informix processes such as ODS are
running, then that's prob. it. Can't remember how you can actively scrub
out mem. I usually just bounce the machine!
==========================================
> } We are running Online Dynamic Server 7.23.UC1 on an RS6000 with AIX
> } 4.14.
Several > } times the Online instance has hung during the Initialization
phase after the
> } reboot.
> } We shut the server down with an onmode -ky and restart it with oninit.
Paul Watson
Mark Stannard wrote:
>
> Folks
>
> We are running Online Dynamic Server 7.23.UC1 on an RS6000 with AIX
> times the Online instance has hung during the Initialization phase after the
> down with an onmode -ky and restart it with oninit.
>
keeping alive the tcp threads, this can be checked with a netstat -a -
look for FIN_WAIT conditions. If this is the case then the OS will not
release the SHM until the threads close, typically 15 minutes.
BTW I have also noticed that onmonitor-mode-immediate_shutdown seems to
close the network properly whereas onmode -ky doesn't. Why I have no
idea, but I have seen it on both Dynix and Solaris systems.
--
Paul Watson
WF Software Ltd.
Tel: +44 1436 674729
Fax: +44 1436 678693
Email: pa...@wfsoftware.co.uk