How to handle predictable network interface names

[Marc Haber]

Hi,

I would like to discuss how to handle systemd's new feature of
predictable network interface names. This is a rather hot topic in the
team I'm currently working in, and I'd like to solicit your opinions
about that.

On systems with more than one interface, the canonical way to handle
this issue in the past was "assume that eth0 is connected to network
foo, eth1 is connected to network bar, and eth2 is connected to
network baz" and to accept that things fail horribly if the order in
which network interfaces are detected changes.

While upstream's focus is as usual on desktop machines where Ethernet,
USB and WWAN interfaces come and go multiple times a day (see
upstream's reasoning in
https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/),
this seldomly happens in our happy server environment, which reduces
the breakage potential to disruptive kernel updates or vendor firmware
changes peddling with the order in which network interfaces are
enumerated.

This happens rather seldomly in my experience.

I would, however, like to stay with the new scheme since I see its
charms.

But, how would I handle this in a puppetized environment?

Currently, the code that is already, for example for a firewall
assumes that eth0 is the external interface, eth1 the internal one and
eth2 the perimeter networks.

In the new setup, all those interfaces can have different names
depending on different hardware being used. That means that the same
puppet code cannot be used on one firewall instance running on Dell
hardware and a second one running on HP hardware because BIOS indices
and/or PCI paths will vary. If I used the MAC scheme, things are even
worse since interface names will be different even on different pieces
of otherwise identical hardware.

Many of my team members thinkt hat one should simply turn of
predictable network interface names altgether and so that our old code
continues to work. I think that this would be a bad idea, but don't
have any logical arguments other than my gut feeling.

Generating udev rules to fix the network names (and assign names like
ext1, int1, per1) already in postinst of the OS does not work since we
don't know how the machine is going to be wired and even used.

Any ideas? How do _you_ do this?

Greetings
Marc

--
-----------------------------------------------------------------------------
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421

[Rob Nelson]

Marc,

We use VMware's vSphere, which still results in "random" but predictable interface names - eth0 is now eno16780032, eth1 is now eno3359296, etc. We've stuck with that because while it's somewhat painful (eth# is soooooo much easier to comprehend), it's far less painful to memorize that than to maintain some udev rules that may need tweaked across time. However, if we were on bare metal, it might be worth disabling the rules to get the older style back. That's probably still less optimal than customized names, but it's well documented at least. For example, http://carminebufano.com/?p=108 or http://amolkg.blogspot.in/2015/05/centos-7-change-network-interface-name.html - though there are multiple ways to do it even then.

Rob Nelson
rnel...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/20160824135527.GP2471%40torres.zugschlus.de.
For more options, visit https://groups.google.com/d/optout.

[Luke Bigum]

Here we have very strict control over our hardware and what interface goes where. We keep CentOS 6's naming scheme on Dell hardware, so p2p1 is PCI slot 2, Port 1, and don't try rename it. We have a 3rd party patch manager tool (patchmanager.com), LLDP on our switches, and a Nagios check that tells me if an interface is not plugged into the switch port it is supposed to be plugged into (according to patchmanager). This works perfectly on Dell hardware because the PCI name mapping works. On really old HP gear it doesn't work, so we fall back on always assuming eth0 is the first onboard port, etc. If the kernel scanned these devices in a different order we'd get the same breakage you describe, but that's never happened on it's own, it's only happened if an engineer has gone and added re-arranged cards.

We still need some sort of "glue record" that says "this interface should be up and have this IP". In our older designs this was managed entirely in Hiera - so there's a giant multi-level hash that we run create_resources() over to define every single network interface. You can imagine the amount of Hiera data we have. In the newer designs which are a lot more of a role/profile approach I've been trying to conceptualise the networking based on our profiles. So if one of our servers is fulfilling function "database" there will be a Class[profile::database]. This Class might create a bonded interface for the "STORAGE" network and another interface for the "CLIENT" network. Through various levels of Hiera I can define the STORAGE network as VLAN 100, because it might be a different vlan tag at a different location. Then at the Hiera node level (on each individual server) I will have something like:

profile::database::bond_storage_slaves: [ 'p2p1', 'p2p2' ]

That's the glue. At some point I need to tell Puppet that on this specific server, the storage network is a bond of p2p1 and p2p2. If I took that profile to a HP server, I'd be specifying a different set of interface names. In some situations I even just put in one bond interface member, which is useless, but in most situations I find less entropy is worth more than having a slightly more efficient networking stack.

I have bounced around the idea of removing this step and trusting the switch - ie: write a fact to do an LLDP query for the VLAN of the switch port each interface is connected to, that way you wouldn't need the glue, there'd be a fact called vlan_100_interfaces. Two problems with this approach: we end up trusting the switch to be our source of truth (it may not be correct, and, what if the switch port is down?). Secondly the quality and consistency of LLDP information you get out of various manufacturers of networking hardware is very different, so relying on LLDP information to define your OS network config is a bit risky for me.

It's a different story for our VMs. Since they are Puppet defined we specify a MAC address and so we "know" which MAC will be attached to which VM bridge. We drop a MAC based udev rule into the guest to name them similarly, ie: eth100 is on br100. I could technically use the same Puppet code to write udev rules for my hardware, but the PCI based naming scheme is fine so far.

That's what we do, but it's made easy by an almost homogeneous hardware platform and strict physical patch management.

When I read about your problem, it sounds like you are missing a "glue record" that describes your logical interfaces to your physical devices. If you were to follow something along the lines of our approach, you might have something like this:

class profile::some_firewall(

  $external_interface_name = 'eth0',

  $internal_interface_name = 'eth1',

  $perimiter_interface_name = 'eth2'

) {

  firewall { '001_allow_internal':

    chain   => 'INPUT',

    iniface => $internal_interface_name,

    action  => 'accept',

    proto => 'all',

  }

  firewall { '002_some_external_rule':

    chain   => 'INPUT',

    iniface => $external_interface_name,

    action  => 'accept',

    proto => 'tcp',

    dport => '443',

  }

}

That very simple firewall profile probably already works on your HP hardware, and on your Dell hardware you'd need to override the 3 parameters in Hiera:

profile::some_firewall::internal_interface_name: 'em1'

profile::some_firewall::external_interface_name: 'p3p1'

profile::some_firewall::perimiter_interface_name: 'p1p1'

Hope that helps,

-Luke

[Dan White]

Very nice, Luke.

Does the code that lets you custom-name your interfaces live in github or puppet-forge anywhere ?

If not, would you be willing to share ?  I can bring brownies and/or beer to the collaboration :)

Dan White | d_e_...@icloud.com
------------------------------------------------
“Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.”  (Bill Waterson: Calvin & Hobbes)

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/0521ee3e-519d-4c02-9da1-2fe37aad3fc5%40googlegroups.com.

[Luke Bigum]

No, not really :-( It's a very "internal" module that I forked from someone's Google Summer of Code project over 5 years ago (way before voxpupuli/puppet-network). You know all those Hiera keys about vlan tags I mentioned? The defaults are in this module and are the default VLAN interfaces for all of our networks. if I gave out the module the Security team would throttle me for releasing what is part of a map of internal network architecture ;-)

I can however, just post the bit that does the UDEV rules...

*********************

$ cat ../templates/etc/udev/rules.d/70-persistent-net.rules.erb

# Managed by Puppet

<% if @interfaces.is_a?(Hash) -%>

<%   @interfaces.sort.each do |key,val| -%>

<%     if val['hwaddr'] -%>

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="<%= val['hwaddr'] -%>", ATTR{type}=="1", KERNEL=="eth*", NAME="<%= key -%>"

<%     end # if val['hwaddr'] -%>

<%   end # @interfaces.sort.each -%>

<% end -%>

<% if @extra_udev_static_interface_names.is_a?(Hash) -%>

<%   @extra_udev_static_interface_names.sort.each do |interface,hwaddr| -%>

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="<%= hwaddr.downcase -%>", ATTR{type}=="1", KERNEL=="eth*", NAME="<%= interface -%>"

<%   end -%>

<% end -%>

*********************

The template will create udev rules from two sources. The first is @interfaces, which is the giant multi-level hash of network interfaces that our old designs use. A VM might look like this in Hiera:

networking::interfaces:

  eth0:

    ipaddr: 1.1.1.1

    hwaddr: 52:54:00:11:22:33

The second source of udev rules is also a Hash and also from Hiera, but rather than it be embedded in the giant hash of networking information, it is there to compliment the newer role/profile approach where we don't specify MAC addresses. This is purely a cosmetic thing for VMs to make our interface names look sensible. Here is a sanitised Hiera file for a VM with the fictitious "database" profile:

profile::database::subnet_INTERNAL_slaves:

  - 'eth100'

profile::database::subnet_CLIENT_slaves:

  - 'eth214'

networking::extra_udev_static_interface_names:

  eth100: '52:54:00:11:22:33'

  eth214: '52:54:00:44:55:66'

[Dan White]

It is a starting point.

Many thanks for sharing what you can.

Dan White | d_e_...@icloud.com
------------------------------------------------
“Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.”  (Bill Waterson: Calvin & Hobbes)

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/089f4796-5251-4b82-a588-5577b15cfe43%40googlegroups.com.

[Luke Bigum]

Now that I think about it, I might be able to post a sanitised version of the module online with most of the internal stuff stripped out. It might prove useful for educating our own staff in the concepts, as well as other people. It's not a 5 minute job though so if/when it's done, I'll write a new Group post instead of continuing to hijack this one :-)

[Rob Nelson]

Just tell them you wanted to make sure you were satisfying the external pen testing requirements of PCI ;)

On Wednesday, August 24, 2016, Luke Bigum <Luke....@lmax.com> wrote:

 if I gave out the module the Security team would throttle me for releasing what is part of a map of internal network architecture ;-)

--

Rob Nelson
rnel...@gmail.com

[Marc Haber]

On Wed, Aug 24, 2016 at 08:36:49AM -0700, Luke Bigum wrote:
> Here we have very strict control over our hardware and what interface goes
> where. We keep CentOS 6's naming scheme on Dell hardware, so p2p1 is PCI
> slot 2, Port 1, and don't try rename it.

Isn't CentOS 6 still using eth0, 1, 2, 3? How do you handle different
hardware having different slot numbers, or PCI bridges shifting bus
numbers?

> We have a 3rd party patch manager tool (patchmanager.com), LLDP on
> our switches, and a Nagios check that tells me if an interface is not
> plugged into the switch port it is supposed to be plugged into
> (according to patchmanager).

Nice ;-) Is the code for the Nagios stuff public?

> This works perfectly on Dell hardware because the PCI name mapping
> works.

And you don't have many different kinds of servers.

> On really old HP gear it doesn't work,

What does that mean?

> We still need some sort of "glue record" that says "this interface should
> be up and have this IP". In our older designs this was managed entirely in
> Hiera - so there's a giant multi-level hash that we run create_resources()
> over to define every single network interface. You can imagine the amount
> of Hiera data we have.

That's what we're trying to avoid. Can you share example snippets?

> In the newer designs which are a lot more of a role/profile approach
> I've been trying to conceptualise the networking based on our
> profiles. So if one of our servers is fulfilling function "database"
> there will be a Class[profile::database]. This Class might create a
> bonded interface for the "STORAGE" network and another interface for
> the "CLIENT" network.

That is interesting and a nice concept. But nothing one introduces
just to remedy an error report named "help, my interface names do not
fit any more".

So you do create network interfaces in the profile and not in the
role?

> Through various levels of Hiera I can define the STORAGE network as
> VLAN 100, because it might be a different vlan tag at a different
> location. Then at the Hiera node level (on each individual server) I
> will have something like:
>
> profile::database::bond_storage_slaves: [ 'p2p1', 'p2p2' ]
>
> That's the glue. At some point I need to tell Puppet that on this specific
> server, the storage network is a bond of p2p1 and p2p2.

So you need to know when writing this code what kind of hardware the
system is running on, probably down to firmware version and hardware
extras?

> I have bounced around the idea of removing this step and trusting the
> switch - ie: write a fact to do an LLDP query for the VLAN of the switch
> port each interface is connected to, that way you wouldn't need the glue,
> there'd be a fact called vlan_100_interfaces.

So the fact would be coming from the _server_ based on what lldpcli
show neighbors detail returns, which is supposed to include the VLAN
information? Would this work on 801.1q trunks as well?

> Two problems with this approach: we end up trusting the switch to be
> our source of truth (it may not be correct,

The switch uses its own source of truth which also influences which
network traffic gets sent down the link, so trusting the switch will
at least fail to the safe side and avoid accidentally putting full
trust on an Internet link.

> and, what if the switch port is down?).

One would have to fall back to a certain safety net then.

> Secondly the quality and consistency of LLDP information you get out
> of various manufacturers of networking hardware is very different, so
> relying on LLDP information to define your OS network config is a bit
> risky for me.

Is it really this bad? I do have experience with HP and Cisco, and
their LLDP/CDP information is usually fine.

> It's a different story for our VMs. Since they are Puppet defined we
> specify a MAC address and so we "know" which MAC will be attached to which
> VM bridge. We drop a MAC based udev rule into the guest to name them
> similarly, ie: eth100 is on br100.

How do you puppet define your MAC addresses? Which virtualization do
you use? Can i see a code snippet?

> That's what we do, but it's made easy by an almost homogeneous hardware
> platform and strict physical patch management.

Yes. The homogenous hardware platform is probably something that can
only be maintained for really large installations.

> When I read about your problem, it sounds like you are missing a "glue
> record" that describes your logical interfaces to your physical devices.

We're desperately trying to avoid having this in Hiera.

> If you were to follow something along the lines of our approach, you
> might have something like this:
>
> class profile::some_firewall(
> $external_interface_name = 'eth0',
> $internal_interface_name = 'eth1',
> $perimiter_interface_name = 'eth2'
> ) {
> firewall { '001_allow_internal':
> chain => 'INPUT',
> iniface => $internal_interface_name,
> action => 'accept',
> proto => 'all',
> }
>
> firewall { '002_some_external_rule':
> chain => 'INPUT',
> iniface => $external_interface_name,
> action => 'accept',
> proto => 'tcp',
> dport => '443',
> }
> }
>
> That very simple firewall profile probably already works on your HP
> hardware, and on your Dell hardware you'd need to override the 3 parameters
> in Hiera:
>
> profile::some_firewall::internal_interface_name: 'em1'
> profile::some_firewall::external_interface_name: 'p3p1'
> profile::some_firewall::perimiter_interface_name: 'p1p1'

On the Dell R680, yes. A hypothetical "R680s" would need some other
definition, and a VMware Vm comes up with eno<number> interfaces with
eight-digit <number>, basically random.

Thanks for your input, I appreciate it.

[Marc Haber]

On Wed, Aug 24, 2016 at 09:20:56AM -0700, Luke Bigum wrote:
> Now that I think about it, I might be able to post a sanitised version of
> the module online with most of the internal stuff stripped out. It might
> prove useful for educating our own staff in the concepts, as well as other
> people. It's not a 5 minute job though so if/when it's done, I'll write a
> new Group post instead of continuing to hijack this one :-)

Thanks in advance!

[Marc Haber]

Hi Rob,

On Wed, Aug 24, 2016 at 10:30:20AM -0400, Rob Nelson wrote:
> We use VMware's vSphere, which still results in "random" but predictable
> interface names - eth0 is now eno16780032, eth1 is now eno3359296, etc.

In my experience, the numbers are rather random, different on each VM.

> We've stuck with that because while it's somewhat painful (eth# is soooooo
> much easier to comprehend), it's far less painful to memorize that than to
> maintain some udev rules that may need tweaked across time.

having net.ifnames=0 looks easier than working with those eight-digit
numbers.

> However, if we were on bare metal, it might be worth disabling the
> rules to get the older style back. That's probably still less optimal
> than customized names, but it's well documented at least. For
> example, http://carminebufano.com/?p=108 or
> http://amolkg.blogspot.in/2015/05/centos-7-change-network-interface-name.html
> - though there are multiple ways to do it even then.

Yes, that's what I'd see as a last moment manoeuvre. My gut feeling
says "bad idea, don't do that". I don't have any evidence-based
arguments for that though.

[Marc Haber]

On Wed, Aug 24, 2016 at 09:03:16AM -0700, Luke Bigum wrote:
> The template will create udev rules from two sources. The first is
> @interfaces, which is the giant multi-level hash of network interfaces that
> our old designs use. A VM might look like this in Hiera:
>
> networking::interfaces:
> eth0:
> ipaddr: 1.1.1.1
> hwaddr: 52:54:00:11:22:33
>
> The second source of udev rules is also a Hash and also from Hiera, but
> rather than it be embedded in the giant hash of networking information, it
> is there to compliment the newer role/profile approach where we don't
> specify MAC addresses. This is purely a cosmetic thing for VMs to make our
> interface names look sensible. Here is a sanitised Hiera file for a VM with
> the fictitious "database" profile:
>
> profile::database::subnet_INTERNAL_slaves:
> - 'eth100'
> profile::database::subnet_CLIENT_slaves:
> - 'eth214'
> networking::extra_udev_static_interface_names:
> eth100: '52:54:00:11:22:33'
> eth214: '52:54:00:44:55:66'

So the "database" machine wouldn't have an entry in
networking::interfaces at all, or could one define, for example, the
management interface in networking::interfaces and the database
interfaces in the machine-specific hiera tree?

[Rob Nelson]

We are provisioning everything with vmxnet3 drivers, only weird old OVAs from vendors have non-vmxnet3, and we don't manage those with puppet. We are cloning from template and getting the same results on different vSphere/vCenter versions and instances. Not sure why they would come out different since the "slot" ends up being the same on the virtual hardware.

Rob Nelson
rnel...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/20160825123035.GZ2471%40torres.zugschlus.de.

[Luke Bigum]

On Thursday, 25 August 2016 13:21:24 UTC+1, Marc Haber wrote:

On Wed, Aug 24, 2016 at 08:36:49AM -0700, Luke Bigum wrote:
> Here we have very strict control over our hardware and what interface goes
> where. We keep CentOS 6's naming scheme on Dell hardware, so p2p1 is PCI
> slot 2, Port 1, and don't try rename it.

Isn't CentOS 6 still using eth0, 1, 2, 3? How do you handle different
hardware having different slot numbers, or PCI bridges shifting bus
numbers?

I find this depends on the manufacturer. I've never come across a Dell server newer than an R510 that *doesn't* give you PCI based names. I just checked an R510 and it does. All of our ancient HP gear (7 years, older than the R510s which is old) give the ethX names. Also random SuperMicro hardware gives ethX. I don't really know what's missing for the kernel / udev to name them so, but for us it doesn't really matter.

>  We have a 3rd party patch manager tool (patchmanager.com), LLDP on
>  our switches, and a Nagios check that tells me if an interface is not
>  plugged into the switch port it is supposed to be plugged into
>  (according to patchmanager).

Nice ;-) Is the code for the Nagios stuff public?

Unfortunately no :-( Another one of those LMAX modules that's had years of development but too much company specific stuff hard coded in it to release. It's not a huge amount though, and I did just ask my Lead if I could clean up our networking module and release it and he was more than happy, I'm sure I could do the same for our nagios module. Watch this space, but don't hold your breath.

>  This works perfectly on Dell hardware because the PCI name mapping
>  works.

And you don't have many different kinds of servers.

We try keep as few as possible, but it's not that small a list:

*******************

[root@puppet ~]# mco facts productname

Report for fact: productname

        .................................        found 1 times

        KVM                                      found 603 times

        OptiPlex 7010                            found 1 times

        OptiPlex 7020                            found 2 times

        PowerEdge FC430                          found 15 times

        PowerEdge FC630                          found 56 times

        PowerEdge R220                           found 1 times

        PowerEdge R320                           found 92 times

        PowerEdge R330                           found 1 times

        PowerEdge R510                           found 17 times

        PowerEdge R520                           found 66 times

        PowerEdge R720                           found 36 times

        PowerEdge R720xd                         found 30 times

        PowerEdge R730                           found 7 times

        PowerEdge R730xd                         found 37 times

        Precision Tower 5810                     found 10 times

        Precision WorkStation T5500              found 7 times

        ProLiant DL360 G6                        found 2 times

        ProLiant DL380 G5                        found 16 times

        ProLiant DL380 G6                        found 11 times

        To Be Filled By O.E.M.                   found 1 times

        X9SCL/X9SCM                              found 6 times

*************************

 

 

>  On really old HP gear it doesn't work,

What does that mean?

I meant that on our very old HP servers the PCI device name mapping doesn't come up, so you end up with eth0, eth1, etc.

 

> We still need some sort of "glue record" that says "this interface should
> be up and have this IP". In our older designs this was managed entirely in
> Hiera - so there's a giant multi-level hash that we run create_resources()
> over to define every single network interface. You can imagine the amount
> of Hiera data we have.

That's what we're trying to avoid. Can you share example snippets?

Here is a snippet of the older style, in a Node's Hiera. It is what I'm trying to move away from, because if you want to create 20 of these machines you've got to copy this Hiera hash around 20 times over. Oh the number of typos... You can probably interpolate the defined types that this data has create_resources() run over, the key names are pretty Red Hat specific:

*******************************

networking::interfaces:

  bond1:

    bonding_opts: mode=802.3ad xmit_hash_policy=layer3+4 lacp_rate=slow miimon=100

    enable: true

    onboot: 'yes'

    type: Bonding

  bond1.3:

    broadcast: 1.1.3.255

    enable: true

    ipaddr: 1.1.3.7

    netmask: 255.255.255.0

    network: 1.1.3.0

    onboot: 'yes'

    vlan: 'yes'

  p4p1:

    enable: true

    master: bond1

    onboot: 'yes'

    slave: 'yes'

    type: Ethernet

  p4p2:

    enable: true

    master: bond1

    onboot: 'yes'

    slave: 'yes'

    type: Ethernet

networking::routes:

  bond1:

    device: bond1

    routes:

    - 1.1.2.0/24 via 1.1.3.1

*******************************

>  In the newer designs which are a lot more of a role/profile approach
>  I've been trying to conceptualise the networking based on our
>  profiles. So if one of our servers is fulfilling function "database"
>  there will be a Class[profile::database]. This Class might create a
>  bonded interface for the "STORAGE" network and another interface for
>  the "CLIENT" network.

That is interesting and a nice concept. But nothing one introduces
just to remedy an error report named "help, my interface names do not
fit any more".

Probably not, it's a lot of work for burying an error message if that's just your aim. What I get from the abstraction above is being able to take our profiles and re-use them in a completely different site on the other side of the world, or in a staging / testing environment. So I don't have the concept of "VLAN 123 in Production UK", I've just got "The STORAGE network" which in Production UK happens to be vlan 123 (buried low down in Hiera, and only specified once once), but in Dev it's 456, and over there it doesn't exist so we'll give it the same vlan tag as the CLIENT network, etc... The physical-ness of the network is abstracted from the concepts our software relies on.

 

So you do create network interfaces in the profile and not in the
role?

We try to follow the design rule that "Roles only include Profiles". Since our software stack is heavily dependent on the networking architecture, our profiles for our software are not designed exist on the same server. I would never have profile::database and profile::frontend on the same server, as they might both try create a STORAGE network and fail catalog compilation. As such our roles generally only contain one "business level" profile, and look something like:

*******************

class role::database {

  include profile::mandatory         #Everything mandatory on EL6

  include profile::authentication    #Authentication is not mandatory

  include profile::database           #The profile that does most of the work for our software

}

*******************

This is more a function of our software being heavily tied to networking layout though. I do have other roles and profiles for internal (office) systems that can move around a lot easier, as they are just your standard "Run a webapp and DB" thing.

 

>  Through various levels of Hiera I can define the STORAGE network as
>  VLAN 100, because it might be a different vlan tag at a different
>  location. Then at the Hiera node level (on each individual server) I
>  will have something like:
>
> profile::database::bond_storage_slaves: [ 'p2p1', 'p2p2' ]
>
> That's the glue. At some point I need to tell Puppet that on this specific
> server, the storage network is a bond of p2p1 and p2p2.

So you need to know when writing this code what kind of hardware the
system is running on, probably down to firmware version and hardware
extras?

No, the exact opposite ideally.  You need to know *conceptually* what the requirements of our software are. So sticking with the same fictitious "database" example, you must have a STORAGE network and you must have a CLIENT network otherwise the App simply won't run (we're a little bit more complicated than a LAMP-stack-in-AWS company). When we've got this coded correctly it should be hardware independent, but, there is this "mandatory data" that we need to supply to get it to build (what interfaces are for what network). These are the "glue records" I keep talking about (to borrow a term from DNS). Ideally this would be zero. We *could* programatically determine it, but the arguments against are part "effort vs gain" and part "what do you want your source of truth to be". Maybe the best way for us to do it auto-magically would be to query Patch Manager to determine what networking interfaces should be present and what logical networks they attach to. We already Nagios check against it... That's not trivial though, and it also means my Puppet builds rely on an externally hosted SaaS (not going to fly).

> I have bounced around the idea of removing this step and trusting the
> switch - ie: write a fact to do an LLDP query for the VLAN of the switch
> port each interface is connected to, that way you wouldn't need the glue,
> there'd be a fact called vlan_100_interfaces.

So the fact would be coming from the _server_ based on what lldpcli
show neighbors detail returns, which is supposed to include the VLAN
information? Would this work on 801.1q trunks as well?

That was the idea, yes. Don't know about 801.1q, depends on what the switch OS does for such interfaces.

>  Two problems with this approach: we end up trusting the switch to be
>  our source of truth (it may not be correct,

The switch uses its own source of truth which also influences which
network traffic gets sent down the link, so trusting the switch will
at least fail to the safe side and avoid accidentally putting full
trust on an Internet link.

Yeah if that suits your use case, you could do that. For me though, I'd much prefer a Puppet manifest to fail to compile because someone hasn't supplied the correct data. It forces an engineer to think about what they are building, and where it's attached.

>  and, what if the switch port is down?).

One would have to fall back to a certain safety net then.

>  Secondly the quality and consistency of LLDP information you get out
>  of various manufacturers of networking hardware is very different, so
>  relying on LLDP information to define your OS network config is a bit
>  risky for me.

Is it really this bad? I do have experience with HP and Cisco, and
their LLDP/CDP information is usually fine.

In my opinion it is, yes.  One our Network Engineers changed a Dell FX2 chassis internal I/O switch between one mode and the other to get MLAG workign (these are Dell Force 10 internally) and the structure of the LLDP information changed, and this was simple shit too - the switch description just "disappeared" :-(

Here's one part of our client side Nagios monitoring, a script that converts the LLDP information into a parse-able CSV. Our Nagios servers query this data via SNMP and compare it to Patch Manager, there by telling us if something is plugged in to the wrong port. It is sanitised to the "database" example, it looks like this:

[root@server ~]$ sudo .//interfaces.py 

p3p2,yes,clientswitch01.example.com,16,456,Arista DCS-7124SX

em1,yes,storageswitch01.example.com,8/1/20,123,Brocade ICX6450-48

em2,yes,storageswitch01.example.com,8/1/20,123,Brocade ICX6450-48

p4p1,yes,clientswitch02.example.com,16,456,Arista DCS-7124SX

And here's the Python that generates that output. Note the number of if statements in the function parse_switch_type_from_data(), and how I have to fall back on MAC address checks because some models simply don't want to report that they are a "Brocade", etc:

https://gist.github.com/lukebigum/efb5b789bfeaf962ef15128092015d08

I haven't read the LLDP standard, but from personal experience I assume it reads something like "Here is a list of optional fields, put whatever you want in them".

 

> It's a different story for our VMs. Since they are Puppet defined we
> specify a MAC address and so we "know" which MAC will be attached to which
> VM bridge. We drop a MAC based udev rule into the guest to name them
> similarly, ie: eth100 is on br100.

How do you puppet define your MAC addresses? Which virtualization do
you use? Can i see a code snippet?

KVM. MAC addresses statically defined to a deterministic formulae - so if the IP is 1.2.3.4 the MAC address is 52:54:00:02:03:04 - the last 3 IP bytes are the same as the last three MAC Hex numbers. This means no MAC address clash :-) Unfortunately I haven't got everything magically defined once, so we can must  define a VM in Hiera (not the best place for it, but it's what we've got):

*****************

libvirt::vms:

  database:

    cpus: '4'

    ensure: running

    interfaces:

    - bridge:br123,54:52:00:01:02:03

    - bridge:br456,54:52:00:

    memory: '4096'

    on_crash: restart

    on_poweroff: destroy

    on_reboot: restart

    virt_disk: path=/var/lib/libvirt/images/ld4deploy01/ld4deploy01.img,size=16,bus=virtio,sparse=false

    virt_type: kvm

*****************

And then we must duplicate the MAC address in the Hiera of the VM itself when creating the networking inside the VM. This is crap, as it's the same MAC address in multiple places, but it's tricky to fix. I might be able to solve it with exported resources... but I'd probably get VM definitions out of Hiera first before I fixed this.

> That's what we do, but it's made easy by an almost homogeneous hardware
> platform and strict physical patch management.

Yes. The homogenous hardware platform is probably something that can
only be maintained for really large installations.

> When I read about your problem, it sounds like you are missing a "glue
> record" that describes your logical interfaces to your physical devices.

We're desperately trying to avoid having this in Hiera.

I can understand that, and it's good you've got that mindset. I'd like to get to the same place eventually. For me, going from 100s of lines of Hiera for a node to < 20 is good enough so far.

[Luke Bigum]

That's technically possible with our module, yes, although I personally don't want to mix the styles. It has to do with our Hiera hierarchy being mostly based on physical location and entire "instances" of our application, where what we're talking about here is functionality based. If we had a business rule where "every server in this data centre has management interface eth76" then yeah, that would match our Hiera hierarchy perfectly. We don't have those hard and fast rules though, we've got several management networks, with different levels of security applied, appropriate for different layers of our application. So our management networks are a function of defence in depth security design alongside our software, rather than a simple physical location or group of VMs. Since they're a function of design or "business logic", rather than location, our management networks are defined in profiles (on new systems) because it's only at the role/profile level do you know that a "database" server should have a certain type of management network.

In my current profiles though I started with the management interfaces inside the same software profiles. Turns out this was not the best idea as they are not directly related, and what our roles should really look like is this:

*******************

class role::database {

  include profile::mandatory                        #Everything mandatory on EL6

  include profile::authentication                   #Authentication is not mandatory

  include profile::database                          #The profile that does most of the work for our software

  class { 'profile::management':                   #management network definition and dependent services (sshd, etc)

     type => 'database'                               #but for a specific type of machine

  }

}

*******************

So management would be separate. This would allow me to do smarter ordering of Puppet classes for management services like SSH (and remove a little bit more Hiera glue).

[Marc Haber]

On Thu, Aug 25, 2016 at 09:00:46AM -0700, Luke Bigum wrote:
> On Thursday, 25 August 2016 13:31:17 UTC+1, Marc Haber wrote:
> > So the "database" machine wouldn't have an entry in
> > networking::interfaces at all, or could one define, for example, the
> > management interface in networking::interfaces and the database
> > interfaces in the machine-specific hiera tree?
> >
>
> That's technically possible with our module, yes, although I personally
> don't want to mix the styles.

I understand.

> In my current profiles though I started with the management interfaces
> inside the same software profiles. Turns out this was not the best idea as
> they are not directly related, and what our roles should really look like
> is this:
>
> *******************
> class role::database {
> include profile::mandatory #Everything mandatory on EL6
> include profile::authentication #Authentication is not mandatory
> include profile::database #The profile that does most of the work for our software
> class { 'profile::management': #management network definition and dependent services (sshd, etc)
> type => 'database' #but for a specific type of machine
> }
> }
> *******************
>
> So management would be separate. This would allow me to do smarter ordering
> of Puppet classes for management services like SSH (and remove a little bit
> more Hiera glue).

That looks smart, thanks.

[Marc Haber]

On Thu, Aug 25, 2016 at 08:08:13AM -0700, Luke Bigum wrote:
> On Thursday, 25 August 2016 13:21:24 UTC+1, Marc Haber wrote:
> > On Wed, Aug 24, 2016 at 08:36:49AM -0700, Luke Bigum wrote:
> > > Here we have very strict control over our hardware and what interface
> > goes
> > > where. We keep CentOS 6's naming scheme on Dell hardware, so p2p1 is PCI
> > > slot 2, Port 1, and don't try rename it.
> >
> > Isn't CentOS 6 still using eth0, 1, 2, 3? How do you handle different
> > hardware having different slot numbers, or PCI bridges shifting bus
> > numbers?
> >
>
> I find this depends on the manufacturer. I've never come across a Dell
> server newer than an R510 that *doesn't* give you PCI based names. I just
> checked an R510 and it does. All of our ancient HP gear (7 years, older
> than the R510s which is old) give the ethX names. Also random SuperMicro
> hardware gives ethX. I don't really know what's missing for the kernel /
> udev to name them so, but for us it doesn't really matter.

Can you run
$ for iface in /sys/class/net/*; do echo $iface; sudo udevadm info -q all -p $iface | grep ID_NET_NAME; done
on some of your gear? I'd like to learn what different vendors deliver.

My Thinkpad T450:
/sys/class/net/br0
/sys/class/net/enx000011121314
E: ID_NET_NAME=enp0s20u4
E: ID_NET_NAME_MAC=enx000011121314
E: ID_NET_NAME_PATH=enp0s20u4
/sys/class/net/eth0
E: ID_NET_NAME=enp0s25
E: ID_NET_NAME_MAC=enx507b9d681b36
E: ID_NET_NAME_PATH=enp0s25
/sys/class/net/lo

My APU at home (with a lot of bridges and VLANs):
[2/501]mh@aida:~$ for iface in /sys/class/net/*; do echo $iface; sudo udevadm info -q all -p $iface | grep ID_NET_NAME; done
/sys/class/net/br181
/sys/class/net/br182
/sys/class/net/br183
/sys/class/net/br184
/sys/class/net/br185
/sys/class/net/br186
/sys/class/net/br187
/sys/class/net/br188
/sys/class/net/br189
/sys/class/net/br191
/sys/class/net/br192
/sys/class/net/br281
/sys/class/net/br381
/sys/class/net/br382
/sys/class/net/br383
/sys/class/net/brenp1s0
/sys/class/net/brenp2s0
/sys/class/net/brenp3s0
/sys/class/net/enp1s0
E: ID_NET_NAME_MAC=enx000db9342afc
E: ID_NET_NAME_PATH=enp1s0
/sys/class/net/enp2s0
E: ID_NET_NAME_MAC=enx000db9342afd
E: ID_NET_NAME_PATH=enp2s0
/sys/class/net/enp3s0
E: ID_NET_NAME_MAC=enx000db9342afe
E: ID_NET_NAME_PATH=enp3s0
/sys/class/net/int181
/sys/class/net/int182
/sys/class/net/int191
/sys/class/net/int192
/sys/class/net/lo
/sys/class/net/per281
/sys/class/net/unt381
/sys/class/net/unt382
/sys/class/net/unt383
[3/502]mh@aida:~$

A KVM VM with a lot of interfaces:
/sys/class/net/eth0
E: ID_NET_NAME=enx525400422c88
E: ID_NET_NAME_MAC=enx525400422c88
/sys/class/net/eth1
E: ID_NET_NAME=enx525400d22ad3
E: ID_NET_NAME_MAC=enx525400d22ad3
/sys/class/net/eth2
E: ID_NET_NAME=enx52540095dfa6
E: ID_NET_NAME_MAC=enx52540095dfa6
/sys/class/net/int181
/sys/class/net/int182
/sys/class/net/int183
/sys/class/net/int184
/sys/class/net/int185
/sys/class/net/int186
/sys/class/net/int187
/sys/class/net/int188
/sys/class/net/int189
/sys/class/net/int191
/sys/class/net/int192
/sys/class/net/lo
/sys/class/net/per281
/sys/class/net/unt381
/sys/class/net/unt382
/sys/class/net/unt383

> > We have a 3rd party patch manager tool (patchmanager.com), LLDP on
> > > our switches, and a Nagios check that tells me if an interface is not
> > > plugged into the switch port it is supposed to be plugged into
> > > (according to patchmanager).
> >
> > Nice ;-) Is the code for the Nagios stuff public?
> >
>
> Unfortunately no

Too bad ;-)

> :-( Another one of those LMAX modules that's had years of
> development but too much company specific stuff hard coded in it to
> release. It's not a huge amount though, and I did just ask my Lead if
> I could clean up our networking module and release it and he was more
> than happy, I'm sure I could do the same for our nagios module. Watch
> this space, but don't hold your breath.

That would be really really nice.

That's rather nice and straight through, but probably you need to
distinguish what kind of cards is plugged in. There have been cases
where bus numbers changed depending whether cards with PCI bridges
were plugged in. Also, platforms like the raspi do spontaneously
reassign their USB bus numbers, I guess this happens in the PCI world
als well.

> > > We still need some sort of "glue record" that says "this interface
> > should
> > > be up and have this IP". In our older designs this was managed entirely
> > in
> > > Hiera - so there's a giant multi-level hash that we run
> > create_resources()
> > > over to define every single network interface. You can imagine the
> > amount
> > > of Hiera data we have.
> >
> > That's what we're trying to avoid. Can you share example snippets?
> >
>
>
> Here is a snippet of the older style, in a Node's Hiera. It is what I'm
> trying to move away from, because if you want to create 20 of these
> machines you've got to copy this Hiera hash around 20 times over.

I understand. But somewhere you need to maintain this info. This
really belongs into a CMDB, but if one doesn't have one, hiera is
probably the next best place.

> > > In the newer designs which are a lot more of a role/profile approach
> > > I've been trying to conceptualise the networking based on our
> > > profiles. So if one of our servers is fulfilling function "database"
> > > there will be a Class[profile::database]. This Class might create a
> > > bonded interface for the "STORAGE" network and another interface for
> > > the "CLIENT" network.
> >
> > That is interesting and a nice concept. But nothing one introduces
> > just to remedy an error report named "help, my interface names do not
> > fit any more".
>
>
> Probably not, it's a lot of work for burying an error message if that's
> just your aim.

No, that's the subject of the ticket that prompted the discussion
here. People kept hardcoding ethX in their code...

> What I get from the abstraction above is being able to take our
> profiles and re-use them in a completely different site on the other
> side of the world, or in a staging / testing environment. So I don't
> have the concept of "VLAN 123 in Production UK", I've just got "The
> STORAGE network" which in Production UK happens to be vlan 123
> (buried low down in Hiera, and only specified once once), but in Dev
> it's 456, and over there it doesn't exist so we'll give it the same
> vlan tag as the CLIENT network, etc... The physical-ness of the
> network is abstracted from the concepts our software relies on.

Yes, that is a really nice concept with should have been considered
here years ago. Alas, people didn't.

> > So you do create network interfaces in the profile and not in the
> > role?
> >
>
> We try to follow the design rule that "Roles only include Profiles".

... "and don't define their own resources", you mean?

That's one of the aspects of the role-and-profiles approach that I
have never seen spelled out explicitly, but still honored by nearly
anybody, and I have not yet fully grokked the reasons for doing so.

I think have understood now.

> > The switch uses its own source of truth which also influences which
> > network traffic gets sent down the link, so trusting the switch will
> > at least fail to the safe side and avoid accidentally putting full
> > trust on an Internet link.
>
> Yeah if that suits your use case, you could do that. For me though, I'd
> much prefer a Puppet manifest to fail to compile because someone hasn't
> supplied the correct data. It forces an engineer to think about what they
> are building, and where it's attached.

Yes, that's of course a valid assumption. I was more thinking about
the intern who was sent to the datacenter to re-wire the networking
cables of Server 3847 but instead works on 3874. The lldp setup would
catch that.

> > and, what if the switch port is down?).
> >
> > One would have to fall back to a certain safety net then.
> >
> > > Secondly the quality and consistency of LLDP information you get out
> > > of various manufacturers of networking hardware is very different, so
> > > relying on LLDP information to define your OS network config is a bit
> > > risky for me.
> >
> > Is it really this bad? I do have experience with HP and Cisco, and
> > their LLDP/CDP information is usually fine.
>
> In my opinion it is, yes. One our Network Engineers changed a Dell FX2
> chassis internal I/O switch between one mode and the other to get MLAG
> workign (these are Dell Force 10 internally) and the structure of the LLDP
> information changed, and this was simple shit too - the switch description
> just "disappeared" :-(

Ouch. All software sucks.

> Here's one part of our client side Nagios monitoring, a script that
> converts the LLDP information into a parse-able CSV. Our Nagios servers
> query this data via SNMP and compare it to Patch Manager, there by telling
> us if something is plugged in to the wrong port. It is sanitised to the
> "database" example, it looks like this:
>
> [root@server ~]$ sudo .//interfaces.py
> p3p2,yes,clientswitch01.example.com,16,456,Arista DCS-7124SX
> em1,yes,storageswitch01.example.com,8/1/20,123,Brocade ICX6450-48
> em2,yes,storageswitch01.example.com,8/1/20,123,Brocade ICX6450-48
> p4p1,yes,clientswitch02.example.com,16,456,Arista DCS-7124SX
>
> And here's the Python that generates that output. Note the number of if
> statements in the function parse_switch_type_from_data(), and how I have to
> fall back on MAC address checks because some models simply don't want to
> report that they are a "Brocade", etc:
>
> https://gist.github.com/lukebigum/efb5b789bfeaf962ef15128092015d08
>
> I haven't read the LLDP standard, but from personal experience I assume it
> reads something like "Here is a list of optional fields, put whatever you
> want in them".

Nice, thanks. It is always enlightening to be given a show into other
people's machine rooms. People should do that way more often.

> > > It's a different story for our VMs. Since they are Puppet defined we
> > > specify a MAC address and so we "know" which MAC will be attached to
> > which
> > > VM bridge. We drop a MAC based udev rule into the guest to name them
> > > similarly, ie: eth100 is on br100.
> >
> > How do you puppet define your MAC addresses? Which virtualization do
> > you use? Can i see a code snippet?
>
> KVM. MAC addresses statically defined to a deterministic formulae - so if
> the IP is 1.2.3.4 the MAC address is 52:54:00:02:03:04 - the last 3 IP
> bytes are the same as the last three MAC Hex numbers. This means no MAC
> address clash :-)

My first reaction to that was "stupid idea. Don't do this". But after
thinking about it and especially realizing that IPv6 SLAAC basically
does the same thing with the entire MAC addresses, it's actually
pretty smart.

> *****************
> libvirt::vms:
> database:
> cpus: '4'
> ensure: running
> interfaces:
> - bridge:br123,54:52:00:01:02:03
> - bridge:br456,54:52:00:
> memory: '4096'
> on_crash: restart
> on_poweroff: destroy
> on_reboot: restart
> virt_disk:
> path=/var/lib/libvirt/images/ld4deploy01/ld4deploy01.img,size=16,bus=virtio,sparse=false
> virt_type: kvm
>
> *****************
>
> And then we must duplicate the MAC address in the Hiera of the VM itself
> when creating the networking inside the VM. This is crap, as it's the same
> MAC address in multiple places, but it's tricky to fix. I might be able to
> solve it with exported resources... but I'd probably get VM definitions out
> of Hiera first before I fixed this.

I like it.

> > That's what we do, but it's made easy by an almost homogeneous hardware
> > > platform and strict physical patch management.
> >
> > Yes. The homogenous hardware platform is probably something that can
> > only be maintained for really large installations.
> >
> > > When I read about your problem, it sounds like you are missing a "glue
> > > record" that describes your logical interfaces to your physical devices.
> >
> > We're desperately trying to avoid having this in Hiera.
>
> I can understand that, and it's good you've got that mindset. I'd like to
> get to the same place eventually. For me, going from 100s of lines of Hiera
> for a node to < 20 is good enough so far.

It's a matter of style and history. A someone who changes teams often,
I have stopped thinking about stuff like that (those decisions tend to
be taken before I come in) and just adopt what the environment is used
to.

[Luke Bigum]

On Friday, 26 August 2016 10:57:25 UTC+1, Marc Haber wrote:

On Thu, Aug 25, 2016 at 08:08:13AM -0700, Luke Bigum wrote:
> On Thursday, 25 August 2016 13:21:24 UTC+1, Marc Haber wrote:
> > On Wed, Aug 24, 2016 at 08:36:49AM -0700, Luke Bigum wrote:
> > > Here we have very strict control over our hardware and what interface
> > goes
> > > where. We keep CentOS 6's naming scheme on Dell hardware, so p2p1 is PCI
> > > slot 2, Port 1, and don't try rename it.
> >
> > Isn't CentOS 6 still using eth0, 1, 2, 3? How do you handle different
> > hardware having different slot numbers, or PCI bridges shifting bus
> > numbers?
> >
>
> I find this depends on the manufacturer. I've never come across a Dell
> server newer than an R510 that *doesn't* give you PCI based names. I just
> checked an R510 and it does. All of our ancient HP gear (7 years, older
> than the R510s which is old) give the ethX names. Also random SuperMicro
> hardware gives ethX. I don't really know what's missing for the kernel /
> udev to name them so, but for us it doesn't really matter.

Can you run
$ for iface in /sys/class/net/*; do echo $iface; sudo udevadm info -q all -p $iface | grep ID_NET_NAME; done
on some of your gear? I'd like to learn what different vendors deliver.

My Dell XPS 13, 2016 model:

 /sys/class/net/docker0

/sys/class/net/enp0s20u1u3i5

E: ID_NET_NAME_MAC=enx9cebe824ebee

E: ID_NET_NAME_PATH=enp0s20u1u3i5

/sys/class/net/lo

/sys/class/net/virbr0

/sys/class/net/virbr0-nic

/sys/class/net/virbr1

/sys/class/net/virbr1-nic

/sys/class/net/virbr2

/sys/class/net/virbr2-nic

/sys/class/net/wlp2s0

E: ID_NET_NAME=wlp2s0

E: ID_NET_NAME_MAC=wlxacd1b8c05607

E: ID_NET_NAME_PATH=wlp2s0

For both the Dell R720 and R730, there's no NET_NAME stuff:

[root@r720 ~]# udevadm info -q all -p /sys/class/net/p4p2

P: /devices/pci0000:40/0000:40:02.0/0000:42:00.1/net/p4p2

E: UDEV_LOG=3

E: DEVPATH=/devices/pci0000:40/0000:40:02.0/0000:42:00.1/net/p4p2

E: INTERFACE=p4p2

E: IFINDEX=7

E: SUBSYSTEM=net

And this is an FC430, which is a blade-like chassis with internal PCI switches:

[root@FC430 ~]# udevadm info -q all -p /sys/class/net/p5p1/

P: /devices/pci0000:00/0000:00:03.0/0000:02:00.0/0000:03:01.0/0000:04:00.0/0000:05:0c.0/0000:08:00.0/net/p5p1

E: UDEV_LOG=3

E: DEVPATH=/devices/pci0000:00/0000:00:03.0/0000:02:00.0/0000:03:01.0/0000:04:00.0/0000:05:0c.0/0000:08:00.0/net/p5p1

E: INTERFACE=p5p1

E: IFINDEX=6

E: SUBSYSTEM=net

>  What I get from the abstraction above is being able to take our
>  profiles and re-use them in a completely different site on the other
>  side of the world, or in a staging / testing environment. So I don't
>  have the concept of "VLAN 123 in Production UK", I've just got "The
>  STORAGE network" which in Production UK happens to be vlan 123
>  (buried low down in Hiera, and only specified once once), but in Dev
>  it's 456, and over there it doesn't exist so we'll give it the same
>  vlan tag as the CLIENT network, etc... The physical-ness of the
>  network is abstracted from the concepts our software relies on.

Yes, that is a really nice concept with should have been considered
here years ago. Alas, people didn't.

To be fair we didn't design it this way from the start, it's only in the last couple evolutions that abstraction appeared. What we did have from the start though was the concept that the same network segment in different environments would have the same IP address segments, so the DATABASE network over here is 1.15.7.0, and over there it's 1.40.7.0. The third octet for the same network segment at different sites is the same (and hopefully the same VLAN tag on switches, but not mandatory). It's easy to abstract the numbers into names from there. However there's no reason why we couldn't use the same abstraction idea for vastly different or public IP ranges, it would just require more Hiera glue.

 

> > So you do create network interfaces in the profile and not in the
> > role?
> >
>
> We try to follow the design rule that "Roles only include Profiles".

... "and don't define their own resources", you mean?

That's one of the aspects of the role-and-profiles approach that I
have never seen spelled out explicitly, but still honored by nearly
anybody, and I have not yet fully grokked the reasons for doing so.

Yes, we definitely don't define resources, and don't include component / base level classes.  I think we pulled it from an early Gary Larizza post, along with "roles don't have parameters, if you need to configure a role you've got two different roles".  All rules subject to be broken if someone is in a hurry/lazy (which is too often).

I've always understood the logic to be based on the layers of abstraction the role/profile design is trying to achieve. I always envision this tree:

Node (has one) ->

  Role (has one or more) ->

     Profile (has one or more) ->

         Defined Type (has zero or more) ->

             Resources

         or Class (has zero or more) ->

             Resources

         or Resource

[Marc Haber]

On Fri, Aug 26, 2016 at 08:40:49AM -0700, Luke Bigum wrote:
> My Dell XPS 13, 2016 model:
>
> /sys/class/net/docker0
> /sys/class/net/enp0s20u1u3i5
> E: ID_NET_NAME_MAC=enx9cebe824ebee
> E: ID_NET_NAME_PATH=enp0s20u1u3i5

What a name!

> For both the Dell R720 and R730, there's no NET_NAME stuff:
>
> [root@r720 ~]# udevadm info -q all -p /sys/class/net/p4p2
> P: /devices/pci0000:40/0000:40:02.0/0000:42:00.1/net/p4p2
> E: UDEV_LOG=3
> E: DEVPATH=/devices/pci0000:40/0000:40:02.0/0000:42:00.1/net/p4p2
> E: INTERFACE=p4p2
> E: IFINDEX=7
> E: SUBSYSTEM=net

Maybe OS too old? The interface name "p4p2" also looks fishy.

> > What I get from the abstraction above is being able to take our
> > > profiles and re-use them in a completely different site on the other
> > > side of the world, or in a staging / testing environment. So I don't
> > > have the concept of "VLAN 123 in Production UK", I've just got "The
> > > STORAGE network" which in Production UK happens to be vlan 123
> > > (buried low down in Hiera, and only specified once once), but in Dev
> > > it's 456, and over there it doesn't exist so we'll give it the same
> > > vlan tag as the CLIENT network, etc... The physical-ness of the
> > > network is abstracted from the concepts our software relies on.
> >
> > Yes, that is a really nice concept with should have been considered
> > here years ago. Alas, people didn't.
>
> To be fair we didn't design it this way from the start, it's only in the
> last couple evolutions that abstraction appeared.

Yes. I have never seen an installation doing it _this_ right in the
first iteration.

> > > > So you do create network interfaces in the profile and not in the
> > > > role?
> > > >
> > >
> > > We try to follow the design rule that "Roles only include Profiles".
> >
> > ... "and don't define their own resources", you mean?
> >
> > That's one of the aspects of the role-and-profiles approach that I
> > have never seen spelled out explicitly, but still honored by nearly
> > anybody, and I have not yet fully grokked the reasons for doing so.
> >
>
> Yes, we definitely don't define resources, and don't include component /
> base level classes. I think we pulled it from an early Gary Larizza post,
> along with "roles don't have parameters, if you need to configure a role
> you've got two different roles".

Yes, but dropping a supplementary file does not mean that a role has
parameters. And also, it would be duplication if one had two distinct
roles that would only differ in single setting?

> I've always understood the logic to be based on the layers of abstraction
> the role/profile design is trying to achieve. I always envision this tree:
>
> Node (has one) ->
> Role (has one or more) ->
> Profile (has one or more) ->
> Defined Type (has zero or more) ->
> Resources
> or Class (has zero or more) ->
> Resources
> or Resource

That's the way it's in the books. What I'd like to see clarified (and
explained why) is that this tree should have "and nothing else" spelled
out in multiple places:

Node (has one) ->
Role (and nothing else!)

Role (has one or more) ->

Profile (and nothing else!)

Profile (has one or more) ->
Defined Type (has zero or more) ->
Resources
or Class (has zero or more) ->
Resources
or Resource

[John Gelnaw]

I went the other direction-- we deploy all of our servers (and most of our workstations) via cobbler, so all new RHEL 7 and CentOS 7 boxes have "net.ifnames=0" in the default profile, both pre and post install.

I don't agree with upstream's complaint... udev-persistent rules, while a bit of a hack, work just fine, especially when the vast majority of workstations and servers I support have exactly one ethernet device and no wireless.

[Luke Bigum]

On Saturday, 27 August 2016 18:51:09 UTC+1, Marc Haber wrote:

On Fri, Aug 26, 2016 at 08:40:49AM -0700, Luke Bigum wrote:
> My Dell XPS 13, 2016 model:
>
>  /sys/class/net/docker0
> /sys/class/net/enp0s20u1u3i5
> E: ID_NET_NAME_MAC=enx9cebe824ebee
> E: ID_NET_NAME_PATH=enp0s20u1u3i5

What a name!

http://accessories.euro.dell.com/sna/productdetail.aspx?c=uk&l=en&s=bsd&cs=ukbsdt1&sku=452-bboo

 

> For both the Dell R720 and R730, there's no NET_NAME stuff:
>
> [root@r720 ~]# udevadm info -q all -p /sys/class/net/p4p2
> P: /devices/pci0000:40/0000:40:02.0/0000:42:00.1/net/p4p2
> E: UDEV_LOG=3
> E: DEVPATH=/devices/pci0000:40/0000:40:02.0/0000:42:00.1/net/p4p2
> E: INTERFACE=p4p2
> E: IFINDEX=7
> E: SUBSYSTEM=net

Maybe OS too old? The interface name "p4p2" also looks fishy.

Nope, CentOS 6 (which I guess is pretty old now).

 

> Yes, we definitely don't define resources, and don't include component /
> base level classes.  I think we pulled it from an early Gary Larizza post,
> along with "roles don't have parameters, if you need to configure a role
> you've got two different roles".

Yes, but dropping a supplementary file does not mean that a role has
parameters. And also, it would be duplication if one had two distinct
roles that would only differ in single setting?

Mmmm that's one of the two big questions in all our Puppet design discussions here. Do I introduce some sort of boolean switch to drop a new file,

or do I create a new role / profile. (The other big question is "Should this value be hard coded in a profile or as a parameter in Hiera?")

If it is just one file difference, then I would be persuaded to allow a boolean switch in the profile. If the file did functionality differences

inside a given location/environment, or or was necessary in this location but not others, I'd be ok with the boolean being set in Hiera for those locations/environments.

If on the other hand it the role needed to be used in two different ways inside an environment (maybe the file was the difference between Master and Slave) then I would

advocate two roles. In my mind that's quite clear that there are two different "jobs to do" here, rather than something being done slightly differently in a different location:

class  role::master {

   class { 'profile::something': master => true }

}

class  role::slave {

   class { 'profile::something': master => false }

}

The problem I've found with such boolean switches is once someone sees one of them as a potential solution, they tend to explode into a mass of if statements and 

make things really complicated to read. I think it's because it's easy - I can just solve my problem right now with an if statement here, rather than spend a few hours

thinking about and refactoring all the related classes.

Getting a good balance between duplication (and testing) versus purist design is difficult, and I don't think there will ever be a right answer.

-Luke