A question about a custom "datacenter" grain and data vs. logic, now with AWS!

[jll...@gmail.com]

Hi all,

I'm re-posting an update to my original question to /r/saltstack here for wider visibility.

We're moving some stuff to multiple AWS regions, so that muddies the concept of "datacenter" a bit. In this case, it seems reasonable to treat AWS regions (as opposed to Availability Zones, which will serve other purposes) as logical datacenters, hence the two range-based conditions above that I've just recently added.

This grain implementation was an initial pass a couple of years ago to get the ball rolling, but now I'm trying to figure out how to reasonably separate the data from the logic here. I have a Pillar file that defines a bunch of datacenter-specific values and a jinja map that imports the datacenter-specific values via a "datacenter" variable, which I posted last year to this list. However, trying to move the datacenter determination logic into the Pillar file is problematic since the grain needs to be defined before the pillar data is usable, and I use the grain in other places.

Has anyone tried to implement anything along these lines? Any suggestions for how to better approach this datacenter design issue?

import re
import logging
import salt.utils.network

log = logging.getLogger(__name__)

dc_by_octet = {
    '20': 'corp',
    '30': 'foo',
    '40': 'foo',
    '31': 'bar',
    '41': 'bar',
    '50': 'baz',
    '51': 'ugh'
}

aws_by_octet = dict(
    { str(i): 'us-east-1'    for i in range(60, 80)  }.items() +
    { str(i): 'eu-central-1' for i in range(80, 100) }.items()
)

def get_datacenter():
    '''
    Return the server's datacenter based on its IP address. For simplicity's sake,
    treat the local DMZ (192.168.34.0/24) as a datacenter.
    '''

    aws = False

    ipaddrs = salt.utils.network.ip_addrs()

    matches_10 = filter(lambda ip: re.search(r'^10\.', ip), ipaddrs)
    matches_192 = filter(lambda ip: re.search(r'^192\.168\.34\.', ip), ipaddrs)

    if len(matches_192) == 1 and len(matches_10) == 0:
        datacenter = 'dmz'
    elif len(matches_192) == 0 and len(matches_10) > 0:
        octet = matches_10[0].split('.')[1]
        if octet in dc_by_octet.keys():
            datacenter = dc_by_octet[octet]
        elif octet in aws_by_octet.keys():
            datacenter = aws_by_octet[octet]
            aws = True
        else:
            datacenter = 'unknown'
    else:
        datacenter = 'unknown'

    return { 'datacenter': datacenter, 'aws': aws }

if __name__ == '__main__':
    print get_datacenter()

[Ethan Erchinger]

Why not assign dc name and details using the ipcidr matcher?

https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.match.html

I also happen to setup a master or two per logical dc, so you could target minions based on the master grain.

[jll...@gmail.com]

Interesting approach, however I can't seem to get it to work. I may be running into this bug (old and supposedly fixed) or something like it. I'll keep tinkering, thanks for the idea. :)

[jll...@gmail.com]

So I got this working. First I used this template-and-symlink idea for getting the datacenter name and then put the following code in my pillar top.sls:

{% macro dc_by_octets(octets, dc) %}
{%   for octet in octets %}
{%     set cidr = '10.' ~ octet ~ '.0.0/16' %}
{{ cidr | indent(2, true) }}:
    - match: ipcidr
    - datacenter.{{ dc }}
{%   endfor %}
{% endmacro %}

  '192.168.10.0/24':
    - match: ipcidr
    - datacenter.dmz

{{ dc_by_octets([20],     'corp') }}
{{ dc_by_octets([30, 31],  'foo') }}
{{ dc_by_octets([40, 41],  'bar') }}
{{ dc_by_octets([50],      'baz') }}
{{ dc_by_octets([60],      'meh') }}

{{ dc_by_octets(range(60,  80), 'us-east-1') }}
{{ dc_by_octets(range(80, 100), 'eu-central-1') }}

It could be made more generic to deal with a wider variation in CIDR subnet masks and the 192.168 hack is annoying but this is a good first pass. Always open to suggestions, though! :) 

I found that currently you can't reference a previous pillar from within the top file by adding datacenters:<dc>:subnets per dc in datacenters.sls as a list of CIDR blocks:

base:
  '*':
    - datacenters

{% for dc in pillar.keys('datacenters') %}
{%   for cidr in pillar.get('datacenters:' ~ dc ~ ':subnets') %}
{{ cidr | indent(2, true) }}:
    - match: ipcidr
    - datacenter.{{ dc }}
{%   endfor %}
{% endfor %}

So I have a workable solution but I'll keep an eye on the above pull request to hopefully simplify to the double for-loop.

[Justin Lloyd]

Hi all,

So I changed my entire design to use an external pillar but it doesn't appear to work the way I was hoping. The external pillar reads in a yaml file containing all information about all datacenters, including a list of each datacenters' subnets. Here's a distilled gist with the external pillar code and comments illustrating the datacenters.yaml format and contents.

I think the problem I'm having is that Pillar code is executed on the master so it's the master's IP addresses being used by match.ipcidr. As I originally described, the idea here was to eliminate my custom grain and this external pillar design seemed like a really elegant solution to multiple issues. However, I'm still scratching my head over how to get this working for minions to determine their subnets without having to separate the subnet data and, even worse, keep such info in a custom grain.

Thoughts? Am I missing an obvious approach here?

Justin

[Justin Lloyd]

Actually I think this is working. I just had to restart the minions and make sure the syndics all had the proper settings in /etc/salt/master for the external pillar and had salt-master and salt-syndic restarted as well.

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/3eaa4104-1a7f-4eb1-bfb6-78864aa8bd97%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.