RX2600 ILO reset

[k...@kayceesoftware.com]

Is there a way to reset the ILO on a rx2600 without pulling the plug?

Recently my telnet/ssh attempts to my ILO just hang every few months. I have to power cycle the rx2600. I login a few times per month for years, and I have never had any trouble with the login until the past 6 months.

So far i can get into the normal VMS login, so is there any tools from the VMS side that will allow me to reset or look at the ILO MP unit?

I have had this rx2600 since they were first offered to developers at a bootcamp itanium training session. For years i have logged into the ILO via the network port a few times per month and checked the logs.

I also purchased a newer rx2600 that has an ILO that has SSH loaded. It has the same issue.

Once i power cycle the units, it seems ok for a few months; so I'm kinda ruling out that it is just hacker bots keeping the port locked up.

Any help or links are appreciated.

[Stephen Hoffman]

On 2018-04-23 15:14:51 +0000, k...@kayceesoftware.com said:

> Is there a way to reset the ILO on a rx2600 without pulling the plug?

Are the control and escape key sequences not working?

Escape Shift-9 to get to the BMC CLI and Escape Shift-Q to get back to
the operating system, use cli> P 0 and cli> P 1 to control the power.

To attempt access back to the iLO, use some combination of the Ctrl/B
and the Ctrl/E chord followed by the characters c and f — CTRL/B or
Ctrl/E cf , with no spaces — on the console. (This assumes a
less-than-fossil iLO 2 version.) Use the iLO commands to control power.

Absent that with iLO 2, AFAIK, you'll end up using a remote power
switch if the iLO 2 is well and truly wedged; if it's not responding to
a request to return to the iLO from the console or the XD command once
you're there.

Related:
http://h41379.www4.hpe.com/doc/83final/ba322-90077/ba322-90077.html
http://h20628.www2.hp.com/km-ext/kmcsdirect/emr_na-c03351063-2.pdf
https://mirrors.pdp-11.ru/_alpha/_VMS/OpenVMS_8.4_I64/HP%20Integrity%20iLO%202%20Operations%20Guide.pdf


Page 230 for the iLO 3
https://h20628.www2.hp.com/km-ext/kmcsdirect/emr_na-c02774507-6.pdf —
or use a remote power switch.

In general, upgrade your firmware to current, maybe swap the iLO board
as a test, and generally firewall all remote ssh access or require VPN
access into the network as there are known security flaws in the older
iLO implementations. Given security reports and fixes associated with
the HPE ProLiant iLO products, it seems possible that there might be
comparable flaws in more current versions of the older Integrity iLO
options, too.



--
Pure Personal Opinion | HoffmanLabs LLC

[k...@kayceesoftware.com]

On Monday, April 23, 2018 at 10:58:28 AM UTC-6, Stephen Hoffman wrote:

Thanks Hoff,
the control and escape sequences do not work.
Most days i get in and when finished I logout.
Then there are times I get NOTHING. telnet and ssh just get a blinking cursor. I try the escape and ctrl sequences thinking perhaps that my last logout did not fully logout. But no luck. I do use remote power switches that use pager technology for some of my remote routers, so I may have to resort to that tech until i can get some newer servers once VSI gets VMS stable on x86.

I was really hoping there was some connection from the VMS side to force a reset as I can still get to VMS when this has happened. I can start a reboot from that side, but I'm concerned that a true emergency where the VMS side is also hosed that i have to drive to office. My servers are NOT mission critical, but i'm concerned about my client sites which are mission critical. So far, none of them have complained, but I'm also not their VMS nor hardware support manager. I just manage my app and related database. It's possible they already have this issue. All clients are on Itanium blades, so maybe they have no issues with OS management.

If things really turn south I have all data now stored as JSON within RMS and all backups go to AWS cloud storage for all clients. And almost all my VMS C apps are also now optional as javascript clients served by VMS apache. About half my clients use the js clients, but the clients where speed and stability is mission critical still use ssh and the VMS C apps. Almost all clients have tested the js clients, but half have gone back. It's interesting that I have had high school and college students give me a LOT of FLAK over the touchscreen/tablet versions and demand to go back to the ssh version. I even have some running ssh apps on ipads.

Strange software market we are in.

[Stephen Hoffman]

On 2018-04-23 17:49:18 +0000, k...@kayceesoftware.com said:

> I was really hoping there was some connection from the VMS side to
> force a reset as I can still get to VMS when this has happened.

Nope. With the exception of the IPMI APIs and the VGA device that can
be presented by the iLO and some logging access, the iLO and BMC and
related baggage are external to and largely hidden from OpenVMS. The
iLO and BMC widgets manage OpenVMS, OpenVMS doesn't manage those
widgets.

Older Integrity firmware had gremlins, and some things (such as
graphics console support) were never made workable on the rx2600 and
zx6000 boxes. Check the system and iLO firmware versions, if you've
not already done so.

I'd also VPN-restrict access into the iLO, but that — or remote
gremlins — may or may not be the trigger here.

> All clients are on Itanium blades, so maybe they have no issues with OS
> management.

Itanium blades and rx2800 boxes are iLO 3. HPE is installing ILO 5 on
current systems. Much newer firmware than what's found in rx2600, too.

> Strange software market we are in.

OpenVMS and RMS are very strange, too. OpenVMS and RMS are usually
more familiar to the folks lurking around the comp.os.vms newsgroup,
though.

As for iLO security...
https://github.com/airbus-seclab/ilo4_toolbox
https://www.synacktiv.com/posts/exploit/rce-vulnerability-in-hp-ilo.html
https://support.hpe.com/hpsc/doc/public/display?docId=c04496345
There are a number of others...
Etc.

[k...@kayceesoftware.com]

Thanks again,
Both my rx2600 units firmware is:
MP FW : E.03.30
BMC FW : 04.03
EFI FW : 05.48
System FW : 04.21

Where do i look these days to find the latest?

[Stephen Hoffman]

Most are under entitlement, AFAIK.

But for this case, see...

https://support.hpe.com/hpsc/doc/public/display?sp4ts.oid=null&docLocale=en_US&docId=emr_na-c01666415


The associated bootable firmware image is apparently available via the
ftp.hp.com site:

> ftp> ls
> 229 Entering Extended Passive Mode (|||40702|).
> 150 Here comes the directory listing.
> -rwxrwxr-x 1 32227 14180 16461191 Feb 03 2009 PF_CPEAKSYS0231C.zip
> 226 Directory send OK.
> ftp> pwd
> Remote directory: /pub/softlib/software10/COL4171/ux-67853-1

Some of the other files mentioned in that support article might also be
available, but this Cougar Peak system CD image should get you to
~2009. Which was the last release, AFAIK.

[k...@kayceesoftware.com]

Thanks Hoff,
I had found that file earlier today after reading varying forums.
I should have time this weekend to load it, but i was planning on more searching to see if there was anything after that.

since i don't plan on keeping these long term, perhaps i'll just modify both so they only have access via my vpn.

[Hans Vlems]

Op maandag 23 april 2018 22:34:54 UTC+2 schreef Stephen Hoffman:

Access to the ftp.hp.com site was possible thru anonymous logon.
my next commands were:
> bin
> cd /pub/softlib/software10/COL4171/ux-67853-1
> ls
Which gave this back:
500 Illegal PORT command.

This was on a Windows 10 pro system, I64/VMS V8.4 (on an rx2600) gave the same result.
What am I missing....

Hans

[Jan-Erik Söderholm]

I just clicked on the ftp.hp.com link, and it opened in my "File Explorer"
(or "Windows Explorer" or whatever its name is). I then just clicked my
way down to the ZIP file, opened it and there is an ISO image. No need to
run any specific FTP tool, Win10 (at least here) does it out of the box.

[erga...@gmail.com]

On Tuesday, 24 April 2018 17:04:20 UTC+1, Hans Vlems wrote:

> This was on a Windows 10 pro system, I64/VMS V8.4 (on an rx2600) gave the same result.
> What am I missing....

Try SET PASSIVE ON, or another client that lets you use passive mode (pretty much anything, except perhaps the Windows command line one).

[Warren Kahle]

Does HP-I64VMS-OSIFM_RX2600-V0231-A-1.ZIPEXE do the same update? I got
it from https://support.hpe.com/hpsc/swd/public/detail?swItemId=ux_68115_1

*Warren Kahle* CISSP
CSA CSE Security+
PointSecure Technologies Inc.
Phone: 713-868-1222
Cell: 713-906-5600
warren...@pointsecure.com

> _______________________________________________
> Info-vax mailing list
> Info...@rbnsn.com
> http://rbnsn.com/mailman/listinfo/info-vax_rbnsn.com
>

[Hans Vlems]

Thanks Erik-Jan , that worked for me.

[Rod Regier]

The last known firmware update for the RX2600 iLO was version E.03.32 released on 04-Feb-2009. This is presumably the "final" release.

An iLO or iLO2 interface that is hung can only be reset by removing AC power from the entire server and then restoring it. (The iLO can perform power control for the main processor and so is unconditionally energized when the server is plugged in).

Possibly applicable -

As of 07-May-2014 the RX2600 and RX2660 Integrity server iLO and iLO2 firmware is vulnerable to being hung (rendered unresponsive) to available Heartbleed vulnerability scanning software. A later release of the then-current firmware could resolve this issue.

[Jan-Erik Söderholm]

Den 2018-04-24 kl. 20:12, skrev Hans Vlems:
> Thanks Erik-Jan , that worked for me.
>

It's Jan-Erik, but still glad that it worked... :-)

[Steven Schweda]

> Both my rx2600 units firmware is:
> MP FW : E.03.30
> BMC FW : 04.03
> EFI FW : 05.48
> System FW : 04.21

If those are really model rx2600, then they're not much
like mine:

[rex-m] MP:CM> SYSREV

SYSREV

Current firmware revisions

MP FW : E.03.32
BMC FW : 01.53
EFI FW : 01.22
System FW : 02.31


Which seems to be what you get from that
HP-I64VMS-OSIFM_RX2600-V0231-A-1.ZIPEXE kit. I also have an
older one, HP-I64VMS-OSIFM_RX2600-V0231.ZIPEXE, which seems
to offer "MP FW : E.03.30". Strangely, the files in the
"-A-1" look older than those in the plain kit:

its $ unzipx -l
[-.hp.rx2600]HP-I64VMS-OSIFM_RX2600-V0231.ZIPEXE;
Archive: ITS$DKA1:[SMS.HP.RX2600]HP-I64VMS-OSIFM_RX2600-V0231.ZIPEXE;1
Length Date Time Name
--------- ---------- ----- ----
8754176 01-16-2009 06:08 hp-i64vms-osifm_rx2600-v0231--1.pcsi$compressed
8891 01-16-2009 06:08 hp-i64vms-osifm_rx2600-v0231--1.pcsi$compressed_esw

its $ unzipx -l
[-.hp.rx2600]HP-I64VMS-OSIFM_RX2600-V0231-A-1.ZIPEXE
Archive: ITS$DKA1:[SMS.HP.RX2600]HP-I64VMS-OSIFM_RX2600-V0231-A-1.ZIPEXE;1
Length Date Time Name
--------- ---------- ----- ----
8836096 01-08-2009 06:12 hp-i64vms-osifm_rx2600-v0231-a-1.pcsi$compressed
8901 01-08-2009 06:12 hp-i64vms-osifm_rx2600-v0231-a-1.pcsi$compressed_esw

> The last known firmware update for the RX2600 iLO was
> version E.03.32 released on 04-Feb-2009. This is presumably
> the "final" release.

That was certainly my impression.

Around here:

[rex-m] MP:CM> df -specific 0000
[...]
PRODUCT INFO:
Manufacturer : hp
Product Name : server rx2600
Part/Model : A6873A
[...]

[Hans Vlems]

Awfully sorry about that!

[Stephen Hoffman]

On 2018-04-24 16:04:17 +0000, Hans Vlems said:

> Access to the ftp.hp.com site was possible thru anonymous logon.
> my next commands were:
>> bin
>> cd /pub/softlib/software10/COL4171/ux-67853-1
>> ls
> Which gave this back:
> 500 Illegal PORT command.
>
> This was on a Windows 10 pro system, I64/VMS V8.4 (on an rx2600) gave
> the same result.
> What am I missing....

I have no Windows systems around to try to replicate that behavior.

Try a different (and preferably working) IP and FTP stack?

HP does have a semi-wonky FTP implementation, and with a very short
timeout. Apparently running an FTP daemon that's also not the current
release as well, based on the vsFTP site.

As for whether or not this works, here's a trace from a local Unix box:

> $ ftp ftp.hp.com
> Trying 15.73.48.57...
> Connected to ftp-hpcom.glb1.hp.com.
> 220 (vsFTPd 3.0.2)
> Name ({expurgated{): anonymous
> 230 Login successful.
> Remote system type is UNIX.
> Using binary mode to transfer files.
> ftp> cd pub/softlib/software10/COL4171/ux-67853-1
> 250 Directory successfully changed.

> ftp> pwd
> Remote directory: /pub/softlib/software10/COL4171/ux-67853-1

> ftp> ls
> 229 Entering Extended Passive Mode (|||40625|).

> 150 Here comes the directory listing.
> -rwxrwxr-x 1 32227 14180 16461191 Feb 03 2009 PF_CPEAKSYS0231C.zip
> 226 Directory send OK.

> ftp> bin
> 200 Switching to Binary mode.
> ftp> get PF_CPEAKSYS0231C.zip
> local: PF_CPEAKSYS0231C.zip remote: PF_CPEAKSYS0231C.zip
> 229 Entering Extended Passive Mode (|||40597|).
> 150 Opening BINARY mode data connection for PF_CPEAKSYS0231C.zip
> (16461191 bytes).
> 100%
> |*********************************************************************************************************************************|
> 16075 KiB {expurgated} 00:00 ETA
> 226 Transfer complete.
> 16461191 bytes received in {expurgated}
> ftp>


As was asked else-thread, I prefer to avoid the PCSI patch kits,
preferring the use of the EFI mechanisms directly when those are
available. That for various reasons.

And I don't expect that something that identifies itself as 231A to be
quite the same as something showing itself as 231C, FWIW.

And there are various security issues with more recent ("newer")
ProLiant iLO implementations, and I'd suspect that some of those same
issues might also or do apply to earlier iLO implementations and to the
Integrity iLO implementations.

[Simon Clubley]

Yoda, Hans is. :-)

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world

[Steven Schweda]

I found a document which mentions numbers like:

BMC 04.03
System 04.21

But it's for an rx2620, not an rx2600:

ftp://ftp.hp.com/pub/servers/software/WindowsIntegrity/cp007679.html

[k...@kayceesoftware.com]

hmmmmm, you are CORRECT to question that firmware listing. I checked my rack and the unit i posted was for an rx2620. I did not even remember having a 2620 model. One of my rx2600's is the same rev's as yours (the latest), the other rx2600 is:
MP FW : E.03.15
BMC FW : 01.52

EFI FW : 01.22
System FW : 02.31

And while i was looking closer I found a rx1620 also in a small stack of DS10's. I guess i picked these up cheap when clients upgraded to I64 blades.
I guess i better search and see what the latest firmware should be for the rx2620. I don't think i have ever fired up the rx1620. If it's quieter than the others, maybe i should test it.

[Hans Vlems]

Is ;-)

[Stephen Hoffman]

On 2018-04-25 04:36:19 +0000, k...@kayceesoftware.com said:


> I don't think i have ever fired up the rx1620. If it's quieter than
> the others, maybe i should test it.

rx1600 and rx1620 are LOUD.

[Hans Vlems]

Compared to, say, an rx6600 which makes more noise?
Hans

[MG]

Op 25-apr-2018 om 22:06 schreef Hans Vlems:

> Compared to, say, an rx6600 which makes more noise?

Hoffman is right and I'd say that the 1U systems are some
of the loudest (Integrity) systems around. On top of being
loud, they are high-pitched, too! Which is arguably worse
than the loudness. (In other words: generating a very loud,
far-reaching and penetrating 'whine'.)

Definitely not an ideal small business, freelance developer,
home/enthusiast/hobbyist, etc. type or class of systems...

I've never heard or used an rx6600, but I assume it's on
par with, say, an rx4640. So it will be 'heaven' compared
to those 1U systems and probably also to some 2U systems
(like the rx2620, especially with "Montecito" CPUs and in
a room without some kind of climate control).

- MG