System fails to boot with LED 517

[isa...@hotmail.com]

I am running AIX 5.1 that is acting up. The system is stuck in LED 517
after reboot again and again.

To debug this problem I booted from the install CD and got into
maintenance mode shell, rdisk0, access without mounting file system.

I ran fsck on /dev/hd1, hd2, hd3, hd4, hd9var and hd10opt filesystems.
It ran fine and no error were found. I am also able to mount all these
filesystems and examine any files.

The problem is when I exit out of this maintenance shell the system
should automatically mount the filesystems. Instead I get the following
errors.

mount: /dev/hd2 on /usr: No such file or directory
The mount of /dev/hd2 did not succeed.
Exiting to shell.
...

What could I be doing wrong? What other tests could I run to solve this
problem?

Thanks in advance.

Isaggu

[vlad...@gmail.com]

517 is usually the problem with jfslog, some times it's also indicates
the IPL_rootvg file doesn't exists... so here is the steps:

Boot into service mode from the installation cds.

#fsck -y /dev/hd2
#fsck -y /dev/hd3
#fsck -y /dev/hd4
#fsck -y /dev/hd9var
#fsck -y /dev/hd10opt

#logform /dev/hd8

#exit -> filesystems will be mounted for service work.

#cd /dev
Is IPL_rootvg is missing?
Is ipldevice is a block file not a character file?

If so, follow these steps.

#rm ipldevice
#mknod IPL_rootvg c 10 0
#chmod 660 IPL_rootvg
#lslv -m hd5 -> this will tell you which hdisk your bootlv is located
on.
#ln rhdisk# ipldevice (rhdisk# will be the corresponding hdisk that
hd5 is located on)

#bosboot -ad /dev/hdisk# (hdisk# being the hdisk that hd5 is located
on)

#shutdown -Fr (to boot into normal mode. Remove the cd)

-Vlad

[isa...@hotmail.com]

>517 is usually the problem with jfslog, some times it's also indicates
>the IPL_rootvg file doesn't exists... so here is the steps:

>Boot into service mode from the installation cds.

>#fsck -y /dev/hd2
>#fsck -y /dev/hd3
>#fsck -y /dev/hd4
>#fsck -y /dev/hd9var
>#fsck -y /dev/hd10opt

>#logform /dev/hd8

>#exit -> filesystems will be mounted for service work.

I followed the steps, but the after exit from the shell the filesystems
did not mount and I got the same error:

[isa...@hotmail.com]

Also Vlad,

Earlier when I was in the maintenance shell I could mount all the
filesystems, but I could not find /mnt/etc/rc.boot and /mnt/etc/inittab
files. Why is that?

Thanks for your help.

isaggu

[Jozek]

isa...@hotmail.com wrote:

> I followed the steps, but the after exit from the shell the filesystems
> did not mount and I got the same error:
>
> mount: /dev/hd2 on /usr: No such file or directory
> The mount of /dev/hd2 did not succeed.
> Exiting to shell.
> ...

When you had hd4 mounted in maintenance mode, did you check that /usr
directory is actually there? Maybe mount is failing, because it cannot
find the mount point to mount /dev/hd2 onto...
Jozek

[isa...@hotmail.com]

Jazek,

I mounted the /dev/hd4 on /mnt and /dev/hd2 on /mnt/usr and
/dev/hd3 on /mnt/tmp. Is this not correct?

Not only that, I was able to go into /mnt/usr/bin and run ./ls and
other commands.

What do you think?

isaggu

[vlad...@gmail.com]

Repairing File Systems with fsck in AIX V4 and V5 (LED 517 or 518)

This document covers the use of the fsck (file system check) command in
Maintenance mode to repair inconsistencies in file systems. The
procedure described is useful when file system corruption in the
primary root file systems is suspected or, in many cases, to correct an
IPL hang at LED value 517, 518, or LED value 555.

This document applies to AIX versions 4 and 5.
Recovery procedure

1. Boot your system into a limited function maintenance shell
(Service, or Maintenance mode) from AIX bootable media to perform file
system checks on your root file systems.

Please refer to your system user's or installation and service
guide for specific IPL procedures related to your type and model of
hardware. You can also refer to the document titled "Booting in Service
Mode," available at
http://techsupport.services.ibm.com/server/aix.srchBroker.

2. With bootable media of the same version and level as the system,
boot the system. The bootable media can be any ONE of the following:
* Bootable CD-ROM
* NON_AUTOINSTALL mksysb
* Bootable Install Tape

Follow the screen prompts to the following menu:

Welcome to Base Operating System
Installation and Maintenance

3. Choose Start Maintenance Mode for System Recovery (Option 3).

The next screen displays the Maintenance menu.

4. Choose Access a Root Volume Group (Option 1).

The next screen displays a warning that indicates you will not be
able to return to the Base OS menu without rebooting.

5. Choose 0 continue.

The next screen displays information about all volume groups on
the system.

6. Select the root volume group by number.

7. Choose Access this volume group and start a shell before mounting
file systems (Option 2).

If you get errors from the preceding option, do not continue with
the rest of this procedure. Correct the problem causing the error. If
you need assistance correcting the problem causing the error, contact
one of the following:
* local branch office
* your point of sale
* your AIX support center

If no errors occur, proceed with the following steps.

8. Run the following commands to check and repair file systems.

NOTE: The -y option gives fsck permission to repair file system
corruption when necessary. This flag can be used to avoid having to
manually answer multiple confirmation prompts, however, use of this
flag can cause permanent, unnecessary data loss in some situations.

fsck /dev/hd4
fsck /dev/hd2
fsck /dev/hd3
fsck /dev/hd9var
fsck /dev/hd1

9. To format the default jfslog for the rootvg Journaled File System
(JFS) file systems, run the following command:

/usr/sbin/logform /dev/hd8

Answer yes when asked if you want to destroy the log.

10. If your system is hanging at LED 517 or 518 during a Normal mode
boot, it is possible the /etc/filesystems file is corrupt or missing.
To temporarily replace the disk-based /etc/filesystems file, run the
following commands:

mount /dev/hd4 /mnt
mv /mnt/etc/filesystems /mnt/etc/filesystems.[MMDDYY]
cp /etc/filesystems /mnt/etc/filesystems
umount /mnt

MMDDYY represents the current two-digit representation of the
Month, Day and Year, respectively.

11. Type exit to exit from the shell. The file systems should
automatically mount after you type exit. If you receive error messages,
reboot into a limited function maintenance shell again to attempt to
address the failure causes.

12. If you have user-created file systems in the rootvg volume group,
run fsck on them now. Enter:

fsck /dev/[LVname]

LVname is the name of your user-defined logical volume.

13. If you used the preceding procedure to temporarily replace the
/etc/filesystems file, and you have user-created file systems in the
rootvg volume group, you must also run the following command:

imfs -l /dev/[LVname]

14. If you have file systems in a volume group other than rootvg, run
fsck on them now. Enter:

varyonvg [VGname]
fsck /dev/[LVname]

VGname is the name of your user-defined volume group.

15. If you used the preceding procedure to temporarily replace the
/etc/filesystems file, also run the following command:

imfs [VGname]

The preceding commands can be repeated for each user-defined
volume group on the system.

16. If your system was hanging at LED 517 or 518 and you are unable
to activate non-rootvg volume groups in Service mode, you can manually
edit the /etc/filesystems file and add the appropriate entries.

The file /etc/filesystems.MMDDYY saved in the preceding steps may
be used as a reference if it is readable. However, the imfs method is
preferred since it uses information stored in the logical volume
control block to re-populate the /etc/filesystems file.

17. If your system has a mode select key, turn it to the Normal
position.

18. Reboot the system into Normal mode using the following command:

sync;sync;sync;reboot

If you followed all of the preceding steps and the system still stops
at an LED 517 or 518 during a reboot in Normal mode, you may want to
consider reinstalling your system from a recent backup. Isolating the
cause of the hang could be excessively time-consuming and may not be
cost-effective in your operating environment. To isolate the possible
cause of the hang, would require a debug boot of the system.
Instructions for doing this are included in the document "Capturing
Boot Debug", available at
http://techsupport.services.ibm.com/server/aix.srchBroker. It is still
possible, in the end, that isolation of the problem may indicate a
restore or reinstall of AIX is necessary to correct it.

[isa...@hotmail.com]

Jozek,

Doing df -k in the maint mode on the mounted filesystems shows
/mnt/usr is 96% full. You think that has something to do with it? If so
how do I fix it?

Thanks
isaggu

[isa...@hotmail.com]

Vlad,

I followed this directions and got as far as step 11 where it says
upon exit the filesystems should automatically be mounted. Here instead
I got the following error:

mount: /dev/hd2 on /usr: No such file or directory
The mount of /dev/hd2 did not succeed.
Exiting to shell.

Thanks
isaggu

[Jozek]

Check what your /usr on hd4 contains.
In maintenance mode, this is what should be there in hd4 when hd2 isn't
mounted over /usr: /usr/lib -> /../usr/lib (I know, it appears
strange, but it is necessary so that proper libraries are found in both
stages of the boot process - before and after mounting of hd2).
Also make sure this link is present: /lib -> /usr/lib
Jozek

[Frank Fegert]

Have you checked if the /dev/hd2 node exists on the system?
Boot in maintenance mode again and do a ls -al /mnt/dev/hd2
on the mounted hd4.

Regards,

Frank

[isa...@hotmail.com]

Frank,

In the maint mode /mnt/dev/hd2 doesn't exist and neither does
/mnt/dev directory? How should I fix this?

isaggu

[Frank Fegert]

Double check, while in maintenance mode, that you have /dev/hd4
mounted to /mnt. Do a:
ls -ald /mnt/dev/
ls -al /mnt/dev/
If both fail, you have somehow managed to delete your /dev
directory. As far as i know the node entries under /dev are
recreated at system boot, so you should only have to do a:
mkdir /mnt/dev
sync;sync
umount /mnt
and reboot the machine.

Regards,

Frank

[isa...@hotmail.com]

Frank,

The /mnt/dev directory was not there. So I created it and left it
empty. After the steps sync, sync , umount, reboot didn't create the
devices in the dev directory. So I am still getting LED 517.

isaggu

[isa...@hotmail.com]

Frank,

Is it possible to run "bosboot -ad /dev/hdisk0" command from the
maint shell? I think this might create the devices in the dev
directory. What do you think?

isaggu

[Frank Fegert]

Digged a bit in /sbin/rc.boot. Apparently 517 is rather common
(see for yourself). What happens if you let the machine sit at
517 for some time (1/2-1 hour)? Might be that your syncvg is
still running, or a system core is copied over to sysdumpdir.

Regards,

Frank

[isa...@hotmail.com]

I think my boot image might be corrupted because I don't see /dev/*
files or /etc/rc.d directory. How could I recreate boot image?

All I am able to do now is boot of the CD and get into maint mode. Is
it possible to do some diagnostics from maint mode to recreate boot
image? Or replace /dev/* files and /rc2.d/* files?

In the docs I saw that it is possible if I get into single-user mode
after booting of the hard disk and run bosboot command. This will
recreate the boot image. But I am able to only boot of the CD. Help
please.

isaggu

[matt...@austin.rr.com]

In article <1126534989.2...@z14g2000cwz.googlegroups.com>,
isa...@hotmail.com wrote:

In maintenance mode:

mount /dev/hd4 /mnt
mkdir /mnt/usr
mount /dev/hd2 /mnt/usr

if that works, then:

umount /mnt/usr
umount /mnt
exit

[isa...@hotmail.com]

Thanks All,

In the end I had to do "Preservation Installation". Save most of my
/home and /opt partitions and reinstall the rest(/, /usr, /var and /tmp
partitions). Since most of the user accounts were in the /home
partition, that didn't put me back too much.

Thanks again.
isaggu