Harvard NICs

[George Silvis, III]

Orran brought up an idea for our Harvard/MGHPCC cluster networking
setup---instead of replacing the broken 1Gb switch with another 1Gb
switch, put a 10Gb switch there, and use it to provide failover
redundancy to the other one. (I don't understand the details of how to
do that, but perhaps an ethernet bond?) The problem would be if we
don't have two 10Gb NICs on the physical machines.

I tried to investigate this on the MOC openstack controller, and got
pretty confused pretty quickly. First of all, there are /four/ NICs:
em1, em2, p3p1, p3p2. em1 is definitely 10Gb; em2 appears to be unused,
and I can't tell for p3p1 and p3p2. Could someone with a little more
experience investigate this?

Thanks
-George

[Ian Denhardt]

Quoting George Silvis, III (2015-01-20 16:34:35)

> I tried to investigate this on the MOC openstack controller, and got
> pretty confused pretty quickly. First of all, there are /four/ NICs:
> em1, em2, p3p1, p3p2. em1 is definitely 10Gb; em2 appears to be unused,
> and I can't tell for p3p1 and p3p2. Could someone with a little more
> experience investigate this?

em1 is the only one that's set to start on boot. All of them look to
support 10Gb.

-Ian

[Orran Krieger]

Okay, I don't think we should mess with the existing running environment, so we should just replace the messed up switch. However, it would be good to wire additional NICs on a couple of other nodes to the 10Gig network if we have available ports. That way we can set up a new controller on one of the other nodes, and then take over the other compute. I would, in a new setup, isolate the external networking from the tenant networks those controllers.

Nick, is this an option, i.e., can we wire some of the other NICs to additional ports on the 10Gig network for a couple of other compute? We spent over a week until we found a kernel bug that was because we where using a single NIC for both tenant networks and external networks on the controller node. While we fixed (turned off TCP receiver offload) it would be nice to have the extra bandwidth and be deploying things in a better supported configuration.

> --
> You received this message because you are subscribed to the Google Groups "MOC technical" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to moc-technica...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ian Denhardt]

Quoting Orran Krieger (2015-01-20 19:42:41)

> On Jan 20, 2015, at 5:14 PM, Ian Denhardt <i...@zenhack.net> wrote:
> > em1 is the only one that's set to start on boot. All of them look to
> > support 10Gb.

To be clear: I only looked at the controller. The other nics may be in
use on the compute nodes, I'd have to check.

-Ian