Any USB to serial adapters that work with emulab tipservs for console access?

[burnettb317]

Adding a bunch of new nodes to our emulab and forgot to make sure they had serial ports.  Have you found any USB to serial adapters that work with emulab tipservs for console access?  I tried one, and the first attempt has not worked (but I still need to try some more options and bios settings), but wanted to ask if anyone has found any that work.

-Ben

[Mike Hibler]

If the nodes support IPMI (e.g., HP iLO or Dell iDRAC management interfaces)
then you can use the serial-over-lan (SOL) feature. That is all we use any
more as serial concentrators are hard to find and not cheap. This does mean
that you do need some cheap unmanaged network switches to connect the
management interface to.

That said, we do have one 8-port serial to USB box. I want to say it is this
guy:

https://www.startech.com/en-us/cards-adapters/icusb23208fd

It seems to work, but it is the console for some switches and the ports don't
get a lot of use.

> --
> You received this message because you are subscribed to the Google Groups
> "emulab-admins" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to emulab-admin...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/
> emulab-admins/7ffe88cc-73cf-45f3-b0d1-dcffe06a684bn%40googlegroups.com.

[Ben Burnett]

Thanks Mike, they are Dell Optiplex 3020 PCs I'll look if they support
iDRAC, they have some management stuff, but I forget what... how are
those configured on the tipservs then? is there a link with instructions
you could point me to?

Do they use another ethernet port on the node in addition to the normal
control interface? can they use the same control interface? I could
setup another LAN for the Serial-over-LAN ports, but then we'd lose
another interface for experimental nodes.

I had tried this Sabrent Serial-to-USB Adapter Cable:
https://www.amazon.com/Sabrent-Serial-Adapter-Chipset-CB-FTDI/dp/B006AA04K0/
so far it isn't working...

That hub looks like it is the wrong "end" I would need something that
doesn't use DB9 ports. The PCs don't have a visible DB9 serial port
(although might have one on the motherboard, looking in to that).

Thanks for the help and info,
-Ben

--
Ben Burnett
Principal Investigator / R&D Software Engineer / Computer Scientist
ATC (Architecture Technology Corporation, www.atcorp.com)
wk: (952) 829-5864 x167
bbur...@atcorp.com

[Mike Hibler]

Okay, I see what you are saying, the nodes have no DB9 connector. I somehow
missed that. :-) We only support a console via a serial UART, if it doesn't
have that, then you are left with IPMI SOL.

However, the Optiplex line are "desktop" machines, so they probably don't
have any management engine. If they have some non-DRAC thing, then it must
support IPMI.

Note that you do not need to have a remote console for the nodes to work
in the testbed, it just make experimentation more pleasant.

> To view this discussion on the web visit https://groups.google.com/d/msgid/emulab-admins/696543a1-8d7e-1b1d-3b51-27a24c8f7277%40gmail.com.

[burnettb317]

thanks.

"you do not need to have a remote console" 

I got the nodes added through the new node interface fine, but they will not free, I've been attributing that to lack of console, is there a console option I need to set to "none" or something?  

[burnettb317]

Mike,

     Still trying to get these nodes to boot without a console, do I need to make a change in the MFS?  

In the DEFS file we have NODECONSOLE="sio"  can we change that to "null" or "vga" then refresh the DEFS file somehow? or is there another place to change that?

I'm trying  the different pxeboot.emu versions   (pxeboot.emu-vga,  pxeboot.emu-null, etc.)  is there another way?

Thanks,

-Ben

On Friday, March 26, 2021 at 9:17:43 AM UTC-5 Mike Hibler wrote:

[Mike Hibler]

Were you able to get a console on the VGA? That is what you should work
toward if not, so you will have a better idea of exactly what the boot
problem is. So hook a monitor up to one of the nodes. If the nodes are
unknown to the testbed, then manually add a dhcpd.conf entry of that
particular machine's MAC address and make sure it has filename set to
pxeboot.emu-vga. Something along the lines of:

host 155.98.36.208 {
filename "/tftpboot/pxeboot.emu-vga";
hardware ethernet 3c:ec:ef:46:20:80;
fixed-address 155.98.36.208;
option host-name "155.98.36.208";
}

in the appropriate subnet section with the appropriate IP/mac addresses.
It will then try to boot your "newnode" MFS which should use the console
setting from pxeboot. Check /usr/testbed/log/bootinfo.log to make sure
it checked in. Then hopefully, you will see some action as it starts to
boot. If you get messages from the BTX loader and then all output stops,
then the console in the kernel may still be wrong.

Normally, all you have to change is the version of the pxeboot program
to change consoles. But really old installs might still have MFSes with
hardwired console settings.

NODECONSOLE in the defs file is only used for the initial install and
for emulab-in-emulab installs.

> emulab-admins/a2b39eb7-42a6-42da-9b92-172ac03401c0n%40googlegroups.com.

[burnettb317]

Mike,

    Thanks for the help, unfortunately I still haven't gotten these nodes booting. Only in the office once in awhile and some of this debugging requires direct eyes/hands-on work.... ugh.

We've tried various serial boards but no luck on those (have "official" dell serial boards on order from china someplace, few weeks out)...

I did get the vga consoles booting and it goes through PXE, and then hangs at: "Emulab looking for control net among: ..."

here are the various log entries from the boot (tries to frisbee, but never loads MFS to frisbee...):

power.log:

Apr 22 17:39:18 boss power[35814]: [root] on: pc13

dhcpd.log:

Apr 22 17:39:37 boss dhcpd: DHCPDISCOVER from f8:bc:12:85:e8:95 via em0

Apr 22 17:39:37 boss dhcpd: DHCPOFFER on 192.168.10.22 to f8:bc:12:85:e8:95 via em0

Apr 22 17:39:37 boss dhcpd: DHCPREQUEST for 192.168.10.22 (192.168.10.3) from f8:bc:12:85:e8:95 via em0

Apr 22 17:39:37 boss dhcpd: DHCPACK on 192.168.10.22 to f8:bc:12:85:e8:95 via em0

stated.log:

Apr 22 17:39:40 [1515]: OBJTYPE='TBNODESTATE', OBJNAME='pc13', EVENTTYPE='PXEBOOTING'

Apr 22 17:39:40 [1513]: pc13: RELOAD/SHUTDOWN => RELOAD/PXEBOOTING

Apr 22 17:39:40 [1513]: pc13: Mode change RELOAD => PXEKERNEL forced

Apr 22 17:39:40 [1513]: pc13: RELOAD/PXEBOOTING => PXEKERNEL/PXEBOOTING

Apr 22 17:39:40 [1515]: OBJTYPE='TBNODESTATE', OBJNAME='pc13', EVENTTYPE='BOOTING'

Apr 22 17:39:40 [1513]: pc13: PXEKERNEL/PXEBOOTING => PXEKERNEL/BOOTING

Apr 22 17:39:40 [1513]: pc13: BootWhat says 10007 (mode RELOAD).

Apr 22 17:39:40 [1513]: pc13: Mode change PXEKERNEL => RELOAD forced

Apr 22 17:39:40 [1513]: pc13: PXEKERNEL/BOOTING => RELOAD/BOOTING

Apr 22 17:39:40 [1513]: HEAD: pc13 in 180, queue=1

bootinfo.log:

Apr 22 17:39:40 boss bootinfo[1486]: tbdb: query failed: MySQL server has gone away

Apr 22 17:39:40 boss bootinfo[1486]: Lost connection to DB; Attempting to reconnect ...

Apr 22 17:39:40 boss bootinfo[1486]: 192.168.10.22: REQUEST (vers 1)

Apr 22 17:39:40 boss bootinfo[1486]: 192.168.10.22: REPLY(1): boot from mfs /tftpboot/frisbee

tftpd.log:

Apr 22 17:39:40 boss tftpd[35820]: 192.168.10.22/2748: RRQ for /tftpboot/frisbee/boot/loader.rc.gz (remapped to /fri                      sbee/boot/loader.rc.gz)

Apr 22 17:39:40 boss tftpd[35820]: 192.168.10.22/2748: RRQ done

Apr 22 17:39:40 boss tftpd[1889]: pid 35820 exits, numchildren=0

Apr 22 17:39:40 boss tftpd[35821]: 192.168.10.22/2749: RRQ for /tftpboot/frisbee/boot/loader.conf.gz (remapped to /f                      risbee/boot/loader.conf.gz)

Apr 22 17:39:40 boss tftpd[35821]: 192.168.10.22/2749: RRQ done

Apr 22 17:39:40 boss tftpd[1889]: pid 35821 exits, numchildren=0

Apr 22 17:39:40 boss tftpd[35822]: 192.168.10.22/2750: RRQ for /tftpboot/frisbee/boot/kernel.gz (remapped to /frisbe                      e/boot/kernel.gz)

Apr 22 17:39:41 boss tftpd[35822]: 192.168.10.22/2750: RRQ done

Apr 22 17:39:41 boss tftpd[1889]: pid 35822 exits, numchildren=0

Apr 22 17:39:41 boss tftpd[35823]: 192.168.10.22/2751: RRQ for /tftpboot/frisbee/boot/mfsroot.gz (remapped to /frisb                      ee/boot/mfsroot.gz)

Apr 22 17:39:42 boss tftpd[35823]: 192.168.10.22/2751: RRQ done

Apr 22 17:39:42 boss tftpd[1889]: pid 35823 exits, numchildren=0

Thought maybe it was something in how I added the nodes and the database got messed up or ports not assigned correct and switchmac was messed up so I deleted a node and have it waiting in newnode, but switchmac fails, because switchmac is getting leakage from someplace and 2 other switches are reporting the macs from the new node (as well as listing 2 other mac addrs, one is ops on vlan 2, and one is a monitor pc we have on vlan 2, is the a way to add mac addrs in an ignore list?  still haven't figured out the bleedover issue, rechecked all switch configs and found some inconsistencies, but nothing fixed switchmac yet.....)

Thoughts? ideas? - how can we get these to boot and play nice without serial consoles?

Thanks,

-Ben

[Mike Hibler]

If it didn't show you any possibilities for the control net, that is
a problem. You should see something like:

Emulab looking for control net among: igb0 igb1 ixl0 bge0 bge1 ...

though in your case you may only have a couple of interfaces. Could be it
doesn't recognize your control net NIC. You should be able to ^C on the
console and have it continue.

> emulab-admins/be574fe7-0223-4db3-a3b6-b65cf5cb1ac1n%40googlegroups.com.

[burnettb317]

Thought I'd update you on the status for anyone that might also be looking at this thread.....

1. Got the official Dell serial add-on boards (for the Optiplex_3020 PCs) and those worked.

2. switchmac was getting extra copies from switches at the "end" of there stacks or paths....  was able to work around this by removing from expt stack temporarily (changed type to "other" in nodes DB and remove wires uplinks, this removed the "noise" from switchmac output, and newnode could then find the correct ports for the new nodes) - also rechecked all our switch configs and made sure they were all consistent

3. our newnode MFS was using freebsd 9 and our frisbee and freebsd MFS were using the 8-64 kernel, 9 saw the NICs but 8-64 did not... got the latest MFSes from utah based on another post from 2017? - these contained the 10-64 kernel (also 11, but I used the 10) - this allowed the nodes to reload correctly.  so replaced all MFSes with the new 10-64 kernel versions

4. this worked, and I could swap them in but the ports were all misconfigured (switchmac was wrong when I added, so I added manually and messed up the port order since quad NIC logical order does not follow physical order)

5. deleted the nodes then re-added through the newnode page, found that for some reason the freebsd.newnode10-64 never completed and added the nodes to the web interface (working remote, so don't know why) went back to the kernel 9 version of freebsd.newnode MFS and they booted and were detected and added

6. able to run switchmac for all expt ports, had to manually enter the control ports, but this was minor, and shouldn't affect nodes once added

7. added nodes and swapped-in and success!!

Also note: not all pcs have the serial add-on boards, and were able to be added (without serial), but the debugging of the entire thing was very nice with the serial capability, so serial capability was not required

still need to clean-up a bunch of stuff....  (including making sure all other nodes boot ok with new MFSes)

Thanks for the help (this time, and the previous posts I used to solve this as well)

-Ben