Safe remote kernel install howto (Re: [Bugme-new] [Bug 6613] New: iptables broken on 32-bit PReP (ARCH=ppc))

[Ingo Oeser]

Hi Meelis,

> Unfortunatlety, 2.6.15 does not boot on this machine so I'm locked out
> remotely at the moment.

Here it my paranoid boot setup:

1. Use "lilo -R new-kernel", to boot a kernel only
once and reboot the default kernel next time.

2. Force reboot on any panic after 10 seconds:
append="panic=10" in /etc/lilo.conf

3. Schedule automatic reboot in case of impossible login
echo "/bin/sync; /sbin/reboot -f "|at now + 15min

4. Put "sysctl -w kernel.panic_on_oops=1" as early as possible
in your boot scripts[1].

And now reboot into the new kernel, try to login and delete the reboot
cronjob. If this doesn't work, just wait 15min and have the last stable kernel
booted automatically.

This method saved me and our customers a lot of time already :-)

Regards

Ingo Oeser

[1] This should be the default and should be disabled by the init scripts
as soon as we reach the desired runlevel (S99oops_not_fatal).
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

[Meelis Roos]

>> Unfortunatlety, 2.6.15 does not boot on this machine so I'm locked out
>> remotely at the moment.
>
> Here it my paranoid boot setup:

Thanks, but it's not much use here, since the machine is a PReP powerpc
machine that can boot one kernel from disk (directly loaded from boot
partition, no fancy bootloader) or netboot via serial console for test
kernels. However, if the test kernel hangs, it hangs and I would need
remote power cycling device that I do not have.

--
Meelis Roos (mr...@linux.ee)

[Michael Tokarev]

Ingo Oeser wrote:
> Hi Meelis,
>
>> Unfortunatlety, 2.6.15 does not boot on this machine so I'm locked out
>> remotely at the moment.
>
> Here it my paranoid boot setup:
>
> 1. Use "lilo -R new-kernel", to boot a kernel only
> once and reboot the default kernel next time.
>
> 2. Force reboot on any panic after 10 seconds:
> append="panic=10" in /etc/lilo.conf
>
> 3. Schedule automatic reboot in case of impossible login
> echo "/bin/sync; /sbin/reboot -f "|at now + 15min

Instead of this, I usually use a system startup script like this:

case "$(cat /proc/cmdline)" in
*linux-test*)
(sleep 300; [ -f /var/run/noreboot ] || reboot) &
;;
esac

which means that if the kernel image is named 'linux-test', it will
be rebooted in 15 minutes after booting if no /var/run/noreboot file
exist. So if I'm able to log in, i just touch /var/run/noreboot and
be done with it.

And oh, yes, for this to work, in lilo.conf the new entry should be
labeled linux-test -- ie, install new kernel, add new entry into lilo.conf
with label=linux-test, run `lilo && lilo -R linux-test && init 6' and..
wait ;) After successeful reboot (and touching /var/run/noreboot), edit
lilo.conf, restore the proper label, set proper order of entries if needed
and re-run lilo.

/mjt

[Andi Kleen]

> 4. Put "sysctl -w kernel.panic_on_oops=1" as early as possible
> in your boot scripts[1].

You can as well boot with oops=panic

-Andi

[Ingo Oeser]

Hi Andi,

Andi Kleen wrote:
> > 4. Put "sysctl -w kernel.panic_on_oops=1" as early as possible
> > in your boot scripts[1].
>
> You can as well boot with oops=panic

Only on x86_64 as of Linux 2.6.16.
But maybe this could be put into kernel/panic.c instead :-)

Regards

Ingo Oeser

[Bill Davidsen]

Meelis Roos wrote:
>>> Unfortunatlety, 2.6.15 does not boot on this machine so I'm locked out
>>> remotely at the moment.
>>
>> Here it my paranoid boot setup:
>
> Thanks, but it's not much use here, since the machine is a PReP powerpc
> machine that can boot one kernel from disk (directly loaded from boot
> partition, no fancy bootloader) or netboot via serial console for test
> kernels. However, if the test kernel hangs, it hangs and I would need
> remote power cycling device that I do not have.
>

I did a lot of this at one time, and used lilo in just the way
described. I did have a remote reboot device, however, an operator (1st
shift), janitor (2nd shift), or security guard (3rd/wkend shift) who had
been instructed to push the clearly marked reset button on demand "when
the weird guy in New York tells you."

IBM rack units, like x345 and such, can have an "RSA" card which allows
remote hardware monitor and reboot with a separate IP address for
control. Worth its weight in gold! The latest will let you do remote
console as well.

--
Bill Davidsen <davi...@tmr.com>
Obscure bug of 2004: BASH BUFFER OVERFLOW - if bash is being run by a
normal user and is setuid root, with the "vi" line edit mode selected,
and the character set is "big5," an off-by-one errors occurs during
wildcard (glob) expansion.