Barriers to Network Automation

[Matt O]

I'll kick things off.

Let's start a discussion regarding barriers to entry. I know many have heard about all of the newfangled automation approaches to networking, or even to compute, but just haven't moved on it for various reasons. I know there are a ton of factors that go into this, but I'd like some perspective outside my own.

If you haven't really started down this path, why not? Do you not have enough time in the week to even begin looking at this stuff? Are you unsure as to how such methodologies could be applied in your organization? Or do you think this is all fluff and you're essentially not viewing these things as relevant to your infrastructure or daily job (essentially you view these things as only something the "big companies" can/should take advantage of)?

[Eric P]

Speaking for myself, my biggest barrier is that I'm still learning the requisite systems and coding skills.  I spent the first 10 years of my career hyper focused on "traditional" networking and only within my current role in the last year or so have I been given the latitude to look at emerging trends/tech like automation.  Our existing systems environment heavily leverages automation (via openstack and saltstack) but I'm still figuring out how to plug my mixed-vendor networking environment into that.

[Willard Dennis]

Things preventing me from taking advantage of network automation:

A) legacy infrastructure (up until recently, we've been an all Cisco Catalyst shop; trying to migrate the plant over to Juniper, but this will take time... onePK isn't available for most of our equipment, and haven't heard good things about onePK automation ability anyways...)

B) the current lack of an automation framework like Ansible that's truly cross-vendor... Hoping here that Schprokits fits the bill and is affordable...

- Will

[dwca...@wisc.edu]

 We have quite a bit of automation for legacy devices, but nearly all of it is fundamentally based on Expect.  It hasn't been clear to me that any of the current (newish) automation things are actually geared for the real world enterprise network or linking the old & the new.

[Steven Iveson]

Oh, I could go on and on about this subject.

1) Yes, coding skills are required and thus represent a barrier. As I've mentioned elsewhere, network staff won't be coding functional applications but skills are still needed in this area. Most staff are probably smart enough to pick up what they need to know (Ruby for Puppet for instance) but simply don't have the inclination for one reason or another (valid or otherwise).

2) There are no network specific or focussed tools/toolsets/chains. I wrote more code than Puppet and F5 put together to actually do something useful with the 'original' Puppet F5 module (SOAP). Even now, the REST module is sorely lacking.

3) Devops tools don't make sense to network people. Related to the above but really, these tools are aimed at server people. The websites, the literature, the support sites, everything. I can't imagine trying to setup Jenkins and do something useful without some serious sysadmin backup, which few have.

4) I recently did some work for a large UK bank (20k+ staff) and such is their conservative nature (and overall structure where technical staff and suppliers are concerned) that they are essentially 10+ years behind from a technology standpoint. There's no incentive for staff to push, certainly none for suppliers either. This is reflected elsewhere in the financial services industry in particular. The money is their, not the desire.

5) Money. Who has a reasonably funded test lab, let alone something good and modern enough to do some real testing with? Home lab, sure, production level testing, I don't think so.

6) Fear. Lets say I have a team of six, I automate all the boring, mundane and time consuming things. Great, lets all move up the stack and focus on the good stuff. Err, no, perhaps two of us can.

7) Silos. These won't break down until Merlin the wizard turns up and makes magic. This can't happen bottom up, it has to be the other way around. Thus, in many cases, this won't happen.

8) Idiots. There's plenty of them in management and the executive.

Just as an aside and for context, I've been working as part of a devops team for about 4 months now, working for a global financial 'brand' in the UK. I have nothing to do, despite being the only network person on a floor of 400 or so. Wire once, virtualise all the things and I'm here just in case, and to help a bit with those 'damned' F5 thingies.

[Aaron Paxson]

The largest hurdle, in my mind, is the lack of API's for network equipment.  As mentioned before, OnePK is a good start, but that is not backwards compatible to older equipment.  NETCONF works well, but is not meant for real-time operations.  EXPECT is *not* a good way to get it done.  Using EXPECT scripts for DevOps would be like using windows batch file applications.

More vendor's are starting to build API's into their products, which helps.  But, I truly think that the SDN controller would be the best way to abstract the network, while still handing off API's for our use.  That way, we don't have to worry about the equipment. Let the controller deal with it.

[Jason Edelman]

From a technical perspective, I agree, but there are barriers beyond technology.  With open hardware platforms and new APIs (device and controller based), we are looking good for the future.

--
You received this message because you are subscribed to the Google Groups "network.toCode()" group.
To unsubscribe from this group and stop receiving emails from it, send an email to networktocod...@googlegroups.com.
To post to this group, send email to networ...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/networktocode/15773537-a03a-40f3-afa9-723a465a1f51%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Ed Henry]

I seem to know an SDN company that does just that. ;)

Joking aside. I agree. There are challenges in this respect though, just like everything else.

The nice thing is, we have our inflection point in the market as operators to make an impact. And we should make sure we do just that - which I think we're making a decent movement toward. :)

[David Barroso]

In my opinion, network automation today, is a joke. But is getting better. And this comes from a guy whose network is 95% automated. I don't remember the last time I had to connect to an access switch, a spine, a LB or a firewall. Only border routers are managed manually.

What are the challenges? As someone said already, APIs; as of today they are just CLI wrappers that in the best of the cases return structured data but in most of the cases they just return plain text.

If you plan to automate stuff with tools like ansible or puppet you have to start thinking as sysadmins. Stop adding or removing vlans from ports. Just configure the port as you want it to be without caring how it was before (as if you were configuring apache or squid).

Once you have that ability, you will be able template and deploy any scenario you want. Unfortunately, AFAIK only two vendors have that capability; Juniper and Arista (Arista released that feature a couple of weeks  ago and once that feature was in place it took me less than a week to build a python API to take advantage of it and build an ansible module).

So, IMHO we should stop asking our vendors to implement API calls to do specific stuff like adding vlans, configure IPs, etc. and ask instead for tools that allow us to integrate the network into existing devops tools. Believe me, it's not that hard. You just have to educate your vendors.

--

You received this message because you are subscribed to the Google Groups "network.toCode()" group.
To unsubscribe from this group and stop receiving emails from it, send an email to networktocod...@googlegroups.com.
To post to this group, send email to networ...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/networktocode/7e63a718-3642-45bd-9b69-4d36e4c1ee1c%40googlegroups.com.

[Jason Edelman]

Hi David,

IMHO, on being CLI wrappers, this is actually a good thing (temporarily) for the network industry since it's something network engineers know already.  Going to pure REST or something would make it a little more difficult for those who want to make a leap into automation and some programming who don't already have a dev background (assuming no libs exist b/c in that case it wouldn't matter at all what the API is).  So while not the ideal long term approach for an API, it helps in the learning process FWIW.  Just my 2 cents.

On the other stuff...

Are you saying you would rather give up idempotent modules in Ansible for a single "push" or "deploy" module that just pushes what you've templated?  I've never used that feature on the vendors that offer it, but do they have the capability to show what has changed?  This makes sense for initial deployments, but for use on deploying minor updates, etc., leveraging smaller idempotent could prove to be more valuable because there are safety checks you could/should want to implement, within a module for example.  Can you give more background on the feature you were referring to regarding the template and deploy?  I tried looking for Arista's latest release notes and it seems they may be behind a regwall...

On your last point, can you provide examples?  The vendor offering any kind of an API is what is allowing you to create Ansible modules, so when you say "ask instead for tools" what are you referring to?

To view this discussion on the web visit https://groups.google.com/d/msgid/networktocode/CAObAkcaoufV6TkBUUEC32vUQ52T9ry5sqoEohZ%3DrF0r9zF-%3DvQ%40mail.gmail.com.

[David Barroso]

> Going to pure REST or something would make it a little more difficult for those who want to make a leap into automation and some programming who don't already have a dev background

I disagree. Check how JunOS works. JunOS is built around their XML API. Their CLI is actually a wrapper of that API. I am pretty sure most of the people didn't even notice or know about this but still is by far the best API and CLI out there.

> Are you saying you would rather give up idempotent modules in Ansible for a single "push" or "deploy" module that just pushes what you've templated?

Exactly, that is exactly how we managed to automate our entire network (with very few exceptions). We template our services, then we add the services to our network devices, assign the variables for that service/device and ansible the only thing that has to do is "compile" all the templates and "push" them. Actually, is a bit more complex than that. Ansible will first report the "config diff" which is gathered by gerrit, then someone has to approve the change and then it's finally pushed.

Just think how you configure an apache server or squid. Do you start typing commands (deleting and adding stuff) until you get to your desired state? Or do you instead compile the state you want into a file and then reload the service? So why don't we manage the network in the same way? Why not pushing a new configuration file for BGP with the configuration I want and then reload BGP?

> The vendor offering any kind of an API is what is allowing you to create Ansible modules, so when you say "ask instead for tools" what are you referring to?

What I don't think it's useful is for a vendor to give me an API call to configure a VLAN or assign an IP to an interface, or an ansible/puppet module to do that. My network is extremely more complex than that. What I want is a way (like the config replace I mentioned) to template my services and deploy them. Then I can leverage tools like ansible or puppet.

> On your last point, can you provide examples? 

Sure, I can do probably something better. We opensourced the python API I wrote for EOS.

https://github.com/spotify/pyeos

Here is a quick guide. Check "Managing configuration", there you will find the building blocks that I was talking about before (load_running_config, load_candidate_config, compare_config, commit and rollback):

http://pyeos.readthedocs.org/en/latest/first_steps.html

The ansible module is also there:

https://github.com/spotify/pyeos/blob/master/ansible/eos_install_config

[Jeremy Schulman]

Hi Will - Thank you for the Schprokits mention.  Still plugging away and making progress; hopefully good things coming soon!

Reading through the rest of this thread; props to Steven Iveson for his "Reality Accounting"; this is a great discussion!

 

[Jason Edelman]

See inline below...

On Tue, Jan 27, 2015 at 9:08 AM, David Barroso <dbar...@dravetech.com> wrote:

> Going to pure REST or something would make it a little more difficult for those who want to make a leap into automation and some programming who don't already have a dev background

I disagree. Check how JunOS works. JunOS is built around their XML API. Their CLI is actually a wrapper of that API. I am pretty sure most of the people didn't even notice or know about this but still is by far the best API and CLI out there.

   

<je> I agree with you, just saying wrapping the CLI is a good mid-step...if we had to wait until all vendors re-architected their software, it would be awhile. 

> Are you saying you would rather give up idempotent modules in Ansible for a single "push" or "deploy" module that just pushes what you've templated?

Exactly, that is exactly how we managed to automate our entire network (with very few exceptions). We template our services, then we add the services to our network devices, assign the variables for that service/device and ansible the only thing that has to do is "compile" all the templates and "push" them. Actually, is a bit more complex than that. Ansible will first report the "config diff" which is gathered by gerrit, then someone has to approve the change and then it's finally pushed.

Just think how you configure an apache server or squid. Do you start typing commands (deleting and adding stuff) until you get to your desired state? Or do you instead compile the state you want into a file and then reload the service? So why don't we manage the network in the same way? Why not pushing a new configuration file for BGP with the configuration I want and then reload BGP?

<je> I suppose maybe it's the lack of the ability to load a config snippet into all devices.  My approach has been to build smart ansible mods so far, but with the goal of the user focusing on the desired state, not the commands required to get there (deleting and adding stuff).   Will def check out your examples though.  Thanks.

> The vendor offering any kind of an API is what is allowing you to create Ansible modules, so when you say "ask instead for tools" what are you referring to?

What I don't think it's useful is for a vendor to give me an API call to configure a VLAN or assign an IP to an interface, or an ansible/puppet module to do that. My network is extremely more complex than that. What I want is a way (like the config replace I mentioned) to template my services and deploy them. Then I can leverage tools like ansible or puppet.

> On your last point, can you provide examples? 

Sure, I can do probably something better. We opensourced the python API I wrote for EOS.

https://github.com/spotify/pyeos

Here is a quick guide. Check "Managing configuration", there you will find the building blocks that I was talking about before (load_running_config, load_candidate_config, compare_config, commit and rollback):

http://pyeos.readthedocs.org/en/latest/first_steps.html

The ansible module is also there:

https://github.com/spotify/pyeos/blob/master/ansible/eos_install_config

<je> Ah, I think I saw this the other day, but thought it came from Arista.  I'll definitely check it out.  Many thanks for sharing this!  Really appreciate it.

[David Barroso]

> My approach has been to build smart ansible mods so far, but with the goal of the user focusing on the desired state, not the commands required to get there (deleting and adding stuff).

I followed that path for a while, the problem is that it gets crazy quite soon. You have to parse all the current config, figure out what's not part of the default config and then figure out how to remove what you don't need/want anymore (sometimes is not as simple as prepending 'no'). Then I realized that I shouldn't be fixing that problem, my vendor should : )

[Jason Edelman]

Yes, Those are what my days are like right now!!! :)  Guess maybe I need to live it first..

I've thought about your approach, just not to the extent of what you are saying. 

Quick example.  If you have a BGP service and you are making a slight BGP change to your device (by updating your template), when you are pushing it out, do you do a 'no router bgp <asn>' and then copy over your new bgp config?  Or do you only send the deltas?

[David Barroso]

I always send the entire configuration, from the beginning to the end (including interfaces, users, aaa, logging, etc). It's the OS responsibility to calculate the delta and apply only the changes needed. For me it's totally transparent, I don't care about the current state of the device. That means that if someone implements a manual change it will be overwritten next time ansible is run.

JunOS has had that feature for ages and EOS implemented it since a couple of weeks.

[Dale W. Carder]

Thus spake David Barroso (dbar...@dravetech.com) on Tue, Jan 27, 2015 at 10:31:58AM +0100:

> In my opinion, network automation today, is a joke. But is getting better.
> And this comes from a guy whose network is 95% automated. I don't remember
> the last time I had to connect to an access switch, a spine, a LB or a
> firewall. Only border routers are managed manually.
>
> What are the challenges? As someone said already, APIs; as of today they
> are just CLI wrappers that in the best of the cases return structured data
> but in most of the cases they just return plain text.

I wish that were the case. My experience (particularly with the ASR
9000) was that it became immediately obvious that the XML API's did not
receive any sort of QA. We found missing data compared to the cli, and my
personal favorite was endian issues with x86 vs risc linecards. So, back
to Expect and screen scraping.

JunOS thankfully has a design that does not suffer from these bugs as
blatantly as their CLI is essentially parsing the XML API just like a
customer would directly.

I also now can't remember what platform it was, but we have seen where
the API is much slower than the CLI, when the API requires the data
returned to be ordered. So, if you are trying to do anything at scale,
this is doomed. SNMP in particular suffers from this.

We talk about DevOps, but it is obvious there is no VendorOps. Nobody
would build equipment in such an unusable manner.

Dale

[Willard Dennis]

(forgot to cc: list on initial reply...)

On Tuesday, January 27, 2015, Dale W. Carder <dw...@wisc.edu> wrote:

We talk about DevOps, but it is obvious there is no VendorOps.  Nobody
would build equipment in such an unusable manner.

 

Methinks because we users vote with our dollars... "We get the ______ we deserve" (fill in the blanks, "DevOps", "OS", "vendor-provided features" etc.)  I don't expect an incumbent vendor to turn their aircraft carrier about for a small (if vocal) segment of their customer base... Thankfully we have an incumbent with a decent design available (Juniper Junos), and now many startups who are actually innovating and pushing borders (we'll see who makes it long-term however...) But until we the users demand "better" (careful what you wish for, however...) then we'll get what we've gotten. I for one am glad there's a ONUG out there (although my co. doesn't fall into that particular user base...)

Then there's the forklift upgrade problem that some of us are faced with - not only operationally tricky, but also until all the current equipment is deprecated (++), the CFO ain't gonna go for swapouts. So the changeover to something better will take time... So I for one am interested in a multiple-method way of automating, including such methods as (shudder!) screen-scraping, Expect and SNMP. 

- Will

[Ethan Banks]

On 27 Jan 2015, at 14:22, Willard Dennis wrote:

> (forgot to cc: list on initial reply...)
>
> On Tuesday, January 27, 2015, Dale W. Carder <dw...@wisc.edu> wrote:
>
>>
>> We talk about DevOps, but it is obvious there is no VendorOps.
>> Nobody
>> would build equipment in such an unusable manner.
>>
>
> Methinks because we users vote with our dollars... "We get the ______
> we
> deserve" (fill in the blanks, "DevOps", "OS", "vendor-provided
> features"
> etc.) I don't expect an incumbent vendor to turn their aircraft
> carrier
> about for a small (if vocal) segment of their customer base...
> Thankfully
> we have an incumbent with a decent design available (Juniper Junos),
> and
> now many startups who are actually innovating and pushing borders
> (we'll
> see who makes it long-term however...) But until we the users demand
> "better" (careful what you wish for, however...) then we'll get what
> we've
> gotten. I for one am glad there's a ONUG out there (although my co.
> doesn't
> fall into that particular user base...)

I think part of the problem is certainly vendors having a hard time
changing. But the larger problem is that vendors are not operators. In
conversations I've had with several vendors over the last few months
especially, I've found they don't understand business processes that
drive technology. They certainly don't understand how IT shops operate.
They build features with a vague idea that they are useful, but have no
sense of how that feature is consumed. And they certainly have no sense
of IT in its totality, i.e. how networking and network automation fit
into the larger IT scheme an organization has.

They look to the operator community to help them understand these
things. In general, I don't think vendors are unwilling to build us the
tools that we are asking for. They simply don't know how to give us what
we want. These early attempts are, at least for some, a stab in the
dark.

It doesn't help that the vast majority of networking practitioners have
no idea that the CLI needs, ultimately, to die. CLI-fu is still a
venerated skill.

> Then there's the forklift upgrade problem that some of us are faced
> with -
> not only operationally tricky, but also until all the current
> equipment is
> deprecated (++), the CFO ain't gonna go for swapouts. So the
> changeover to
> something better will take time... So I for one am interested in a
> multiple-method way of automating, including such methods as
> (shudder!)
> screen-scraping, Expect and SNMP.

I agree with you, and hate having to do so. Not being able to move to
gear that's got an OS with automation in mind is a problem. But it
describes a problem that, in a larger sense, I don't know that we're
getting away from with APIs. All the APIs are different so far, as has
come up in this thread.

What I would really like to see is a standard way of modeling and
abstracting network devices, such that standard APIs that transcend
network vendors could be written. I do not want another SNMP, where most
of the interesting OIDs show up under vendor-specific branches of the
MIB tree. 30 different APIs from 30 different vendors to accomplish the
same configuration tasks does not move the ball forward for the
industry, or for us who are consumers. Vendors might claim that such a
standard would limit their ability to innovate, but I disagree. At the
base of it, networking is a commodity meeting straightforward needs.
There is no differentiation to be made at that level of functionality
(802.1q, L2, L3, IPv4/6, OSPF, BGP). Just like in the world of
hypervisors - the hypervisor itself is no longer exciting. It's a
commodity. The differentiation for VMware, et. al. is in the
higher-level functionality.

Point being, what effect can we have as practitioners to drive industry
towards a standardized API model for consuming network resources? Or am
I wrong in thinking such a thing is desirable? ONUG and OCP seem to have
some clout (because money), but I don't know how strongly they are
addressing this specific issue.

/Ethan

[Ed Henry]

TL;DR : Vendors are legitimately trying to provide what you want.

You have no idea how right you are Ethan. It's a bit of a systemic problem with networking vendors.

Disclaimer : I work for Plexxi and I came from about 8 years as an infrastructure operator

Working at Plexxi has taught me that you're right in that vendors WANT to provide the functionality that their users seek, but they lack the operational knowledge to be able to. And it isn't even a lack of talent problem, its a lack of experience problem. The engineers that have designed the last 15 - 20 years of networking equipment has followed the idea that each device is a single system and its never accounted for that its dropped into a system of systems. And this is something that shouldn't be overlooked and we, operators, know that. The old anecdote of hiding the engineer in the closet because they're not good with customers has hurt the industry indescribably.

Working through the process of getting features tooled on top of platforms(used loosely), within a vendor, has been eye opening to me. I've a new found respect for what it takes to get things 'right' in the eyes of the customer. And I've wedged myself into a position where I'll be working directly with the customers and with the engineering team in prototyping these types of features you're looking for. It's going to take a lot to change the way the engineering teams think(the same way it takes a lot to get operators to change the way they think ;)), but I hope to do some of that through these prototyping exercises and explaining what - exactly - operators are wanting and looking for in terms of requested features, from vendors.

My previous post eluded to the fact that this is our time to change the direction of the industry. Connectivity is a commodity, but also a tool, and generally not interesting for the time being, as Ethan said, but what we build on top of that connectivity is what is interesting. And that includes building tooling to build the tool (the network).

As for the standardized API discussion - look at how well each vendor has worked toward staying 'standardized' with something like 802.1q, or even OSPF. It might be best that we just try and get vendors to agree on a data model for the time being and move forward with that. YANG for NETCONF is a good example of that. And the IETF, as much as everyone bashes them, is working on vendor agnostic data models for different protocols that exist out there. (https://datatracker.ietf.org/wg/netmod/documents/)

I took a job at a vendor because I know that, ultimately, someone needs to make the products that are consumed and this is how I felt I could make the largest impact for the time being.

[Zach Brown]

I think one of the major barriers to automation beyond simple things is that the network needs to be treated as a holistic entity and automation tools don't necessarily do that. It's not too hard to automate simple things like interface configs. It's much harder to automate a topology of devices. The barrier is figuring out what's worth while to automate (especially so once the low hanging fruit is taken care of)

It's much easier to automate when you have a small number of vendors involved (especially so with API/code friendly interfaces like JunOS) but when you have many vendors and many versions of code (each requiring a specific syntax to accomplish the same task) the problem gets exponentially more complex. I would say the companies that would benefit the most from automation are also the one with much larger vendor/platform diversity.

I think that's still the main appeal of something like an SDN fabric. You can treat your network as a single holistic entity. That's something easy to work with from a programability standpoint. The problem is that it's not always practical to rip/replace your whole environment with a shiny new SDN fabric... So how do you continue to work with what you have.

[Nick Buraglio]

Oh Ed, I so want to make a "I've got people skills, damnit!" joke based on your "between the customer and the engineers" sentence. So there, I did. 

My last 2 jobs had decent automation out of necessity due to sheer size. The majority of it was SNMPv2/3 based although there were some expect bits as well as some netconf goo. The largest hurdle I see is that there isn't a standard protocol or API for access if you remove SNMP. NetConf tried. Yang is kinda there but neither are supported across the board. OpenFlow has the potential to be a great control protocol but it's green and it has been so run through the marketing machine that tentative network engineers and operators recoil at its mention. Vendor specific solutions look fantastic to enterprises and data centers but it is likely lost to anyone that operates at scale or doesn't sport brand loyalty.   

To me a vendor based, proprietary solution is a non-starter for any kind of scale and applications controlling the network are too far out and frankly, wrought with security issues even at small scale.  There needs to be a standardized and ubiquitous protocol that is lightweight, flexible and modular that aids in the adoption of automation-centric network architecture, and I don't know if such a thing exists yet. I know that sounds a little "debbie downer", but I really think we are still looking for a stanard way to do this. 

I don't necessarily buy the statement that engineers think of the network as a set of systems and not an ecosystem in and of itself, wireless has proven that to be false for nearly 10 years and other management systems exist (think Enterasys) that treat the network as a single managed element. The problem is the lack of a standard. 

[Nick Buraglio]

*Yang models are there if you build them*

[Zach Brown]

The standard API will be the game changer. There is no motivation for most big network vendors to make a non-vendor specific API though.
If there was one and we could just buy any switch, use the same commands and it does the same job, why wouldn't we all just buy the cheapest one out there. (ahem, there will be lots of hardware quality counter arguments to this likely but you get my point)

OpenFlow may be a good choice (eventually) although it's the programming equivalent of assembly language. You'd still need a standard to describe at a high level what you want and have a way to get it translated into OpenFlow for implementation.

[Nick Buraglio]

There will have to be a standard. Think BGP. Although an edge case, if any SDX is going to happen there will have to be a standard set of primitives they support. I assert that there will need to be a standard for internal networks too and that vendors will come around just like with legacy routing protocols. Right now there is a lot of land grab going on, something new and glittery with a gob of marketing hype is ripe for the picking. 

[Steven Iveson]

I'm hardly an 'expert' in this area but, certainly where I'm currently plying my trade (in a DevOps team for an online financial product) each environment is torn down and rebuilt from scratch (sans required mgmt interfaces and firewall rulesets) with each release. This avoids all the diff and merge logic required with other approaches and allows for relatively simple version control. It's all very much in the Matrix Reloaded style, with Neo (mgmt) allowing for reseeded and the machines as the server and network underlay that essentially remain unchanged.

I can't say I've enough experience (or a view on why they do it this way) to discuss the merits of this approach in comparison with others but I quite like it. The actual service is atomic, at least as far as you can take it with automation of everything that can be automated and version control the same. Nothing is forgotten or left behind or remains unnecessarily; the entire service can be rolled forward or backward at will.

Would this be useful in a the broader context of a large WAN or a campus, probably not, I suspect you would hit chicken or egg scenarios pretty quickly, however, taking and planning for a zone based approach would certainly help.

As ever, current tools (and vendors efforts) are pretty poor but also, current designs and approaches are pretty inflexible too. It's not just about process and workflow, some of it is about fundamental design that facilitates these things.

[Steven Iveson]

I sincerely agree with almost all of this; well said. The conclusion I've come to is that standards are produced by vendors, academics and 'administrators' (in the committee and paperwork sense). If you want a standard, don't sit back and wait 3 years for some rubbish to come out of some standards organisations body (SOBs as I like to refer to them) and write one yourself, with your community. It only takes a mailing list and a github repository (and of course, precious time). I'm game.

[Pablo Lucena]

Another barrier I see is getting network engineers within an organization excited about it, who have been doing the same thing for decades and dont want to change. Getting the support from management and other parties is also a barrier in some organizations. Being able to count on the support of fellow co-workers, or even better, from management and above, makes adoption much simpler. In other words, selling the idea to people that don't want change.

[Dale W. Carder]

Thus spake Steven Iveson (st...@iveson.eu) on Thu, Jan 29, 2015 at 05:15:16AM -0800:

> I sincerely agree with almost all of this; well said. The conclusion I've
> come to is that standards are produced by vendors, academics and
> 'administrators' (in the committee and paperwork sense). If you want a
> standard, don't sit back and wait 3 years for some rubbish to come out of
> some standards organisations body (SOBs as I like to refer to them) and
> write one yourself, with your community. It only takes a mailing list and a
> github repository (and of course, precious time). I'm game.

Along these lines, you may be interested in what David Meyer not long ago
wrote[1] in response to Vidya Narayanan's thoughts about the standards
bodies[2].

In one essence, working open-source reference implementations can become
the standard.

Dale

[1] http://packetpushers.net/value-proposition-standards-age-open-source/
[2] https://gigaom.com/2014/04/12/why-i-quit-writing-internet-standards/

> --
> You received this message because you are subscribed to the Google Groups "network.toCode()" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to networktocod...@googlegroups.com.
> To post to this group, send email to networ...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/networktocode/6847eee4-c308-433e-9b9b-a844799b8b76%40googlegroups.com.

[Nick Buraglio]

I have found that "getting the greybeards on board" is really not that
important if you're willing to either play the waiting game or bend
the rules a little. I have had rousing success in environments with
rooms of people that are orders of magnitude smarter than me with more
experience by simply waiting things out, taking responsibility and a
little bit of "prove it will work on my own" on the side. Removing the
scary is key for that kind of opposition because in general human
nature dictates that we fear what we don't understand. Making it a
known instead of an unknown not only removes the fear of such things
but it also serves to illuminate how these things can make life easier
and not steal their jobs. The articles about SDN killing the network
engineering jobs, while making for interesting reading only serves to
make the 80% of those engineers out there hate and fear it even more.
I'm guilty of this myself, I will admit.
That said, I don't agree that "the CLI needs to die", but I would
assert that "the CLI needs to be reduced to a minimal state" since
there will almost always have to be some deep level of troubleshooting
or work that will come up from time to time. There are still prople
that need to know esoteric programming languages and protocols, but
this is likely a debate for another thread, I think.
We do vote with our dollars, but more importantly, we also influence
with our words and actions. The bottom line is that we, over time,
will influence and steer toward what we need if we are unwavering and
vocal in what we ask for. I consider it more like aiming a fire hose a
little bit each day over a long period of time or, more realistically,
like geology (or as Andy from ShawShank put it): pressure and time.
In addition, we have the ability to drive the standards bodies if we
can put the time in. It's boring and takes forever, but it can be
done. In parallel we can also dictate with our words and actions that
open source projects (or whatever we decide) become the standards. I
maintain that we write the book, not the vendors.
There seems to be a rush to make all of this happen right now because
it is The Right Thing (tm), but I don't think that can happen. We need
to wear them down with the tools we have.

My legs are very tired from standing on this soapbox.

nb

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "network.toCode()" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/networktocode/iYM8wbXJ014/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

> networktocod...@googlegroups.com.
> To post to this group, send email to networ...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/networktocode/bc73f8fd-2ccc-49a4-afd1-c8b34b94e957%40googlegroups.com.

[Steven Iveson]

Hmmm, thanks, food for thought although combined I almost came away with 'working code' vs 'open source code'. Of course, working code comes at the end of the IETF process.

[Steven Iveson]

I think it's call Nudge Theory these days ;-)

I agree regarding the CLI; I have little patience for nudging, not that I've been in a position to ever do anything else.

Baby steps, sure thing, but a few leaps every now and then would be nice.

I tend to be quite negative regarding future careers in the industry; how can we remove all the legwork, the mundane and time consuming stuff 'in the weeds' and everyone be OK? They can't all move up the stack and become designers, architects and consultants. I see a huge gap developing between people who cable, rack and look at n/w mgmt systems and the fated few in the right positions doing something interesting. This will of course take time and is well off topic, apologies.

[Nick Buraglio]

Just like everything there is a shift, and it is sink or swim, but I
seriously doubt it will be as dramatic as so many fear and the
technical media implies.
Realistically, though, in IT, if someone is so afraid that they will
have to adapt, why are they working in an industry that is notorious
for rapid change? These are personality traits I always try to coax
out in interview processes because there is a fine line between "I
need that shiny thing" and "change bad!, there is a hole in the boat
we're all going to die!". Those able to understand and navigate that
line are the blue chips.
Change and learning is the thing that excites me about this industry.
For those that fear it I always suggest reading "Who moved my
cheese?".

nb

> https://groups.google.com/d/msgid/networktocode/a6a12a68-5598-42ba-8ab4-54434c753b65%40googlegroups.com.

[Steven Iveson]

Agreed.

I think the stasis in the industry over the last decade and more plus the endless ITIL abuse has taken its toll on many unfortunately.

[Willard Dennis]

 http://www.reactiongifs.us/wp-content/uploads/2013/08/picard_clapping.gif

Well said, sir, well said....

[Mike Biancaniello]

I am currently in the process of putting together an ansible-based orchestration for our network. One thing I've observed is that the server world is highly fragmented with every application written and implemented differently. There are so many times when I've almost pulled my hair out trying to figure out which file I need to edit (and where that file is located and how exactly to make the edit) just to make a single change to a single application, that I had often considered just writing a CLI for my servers that contained a single config file and used set/delete/show/etc commands just like a network device CLI.  For this reason, tools like ansible, chef, puppet, etc are attractive for managing servers. For networks ... we kind of already have that. What we (networking) lack is the orchestration to manage 1000s of devices. We need to keep this in mind when deploying these tools. 

This is why so many of the network-centric automations (e.g. ansible modules) tend to push configs rather than follow the traditional idempotent model. At this point, I'm not yet sure if this is an advantage or will prove to hold us back. Sticking strictly to config templating makes removing (e.g. state: absent) things difficult unless you follow the JunOS model of 'load overwrite', which is not supported with IOS-ish configs (you end up issuing 'no access-list xxxx' followed by what you want), which makes simple changes more impactful than necessary.