[netjsonconfig] Motivation of changes in 0.6.0 alpha

[Federico Capoano]

Hi everyone,

recently I started introducing some important changes in netjsonconfig, the library used to generate configuration of routers.

This change should not affect openwisp2 users negatively, I'm already testing the current master on 3 instances and already fixed a few bugs.

Nevertheless, the change is affecting negatively the 2 GSoC students who have started to work on the AirOS and Raspbian backends, so I will try to explain why this change is needed.

Backward conversion

One of the features that is missing in OpenWISP 2 is the possibility to easily store the full configuration of devices in the controller.

Since OpenWRT/LEDE allows merging a remote configuration with the local one, that's the default mode we are using now in OpenWISP 2 (it was also the only mode available in OpenWISP 1).

This method has its own disadvantages:

• operators have to remember what is in their local configuration, in order to write a remote configuration that works well with the local one
• some packages use anonymous UCI blocks by default, which are impossible for openwisp to override in the merge mode, hence creating duplicate configurations

These disadvantages are holding down OpenWISP 2 from becoming easier to use. OpenWISP 2 is very powerful, but hard to use properly. Which is a shame, because it means many people will give up and use some kind of home baked solution with its own class of bugs, reinventing the wheel: no good framework and ecosystem of tools for low cost open source networks can emerge for such a situation.

That's why we need to add a way to store the full configuration of routers in the controller and this needs to happen automatically: here where backward conversion in netjsonconfig comes into play.

Without the backward conversion, we cannot even use the AirOS or Raspbian backends because those don't have a way to easily merge different configurations like OpenWRT/LEDE (uci import --merge), users would need to manually enter the full configuration of their devices in openwisp2: that will never happen!

The new implementation

I've been designing and planning this feature since at least a year, but never got the time to implement because there were more urgent matters. Now is the right time to introduce it because it's very much needed.

The new implementation adds a few new concepts in the codebase:

• intermediate data structure
  
• converters
• parsers

Everything revolves around the data structures: the NetJSON configuration dictionary and the intermediate data structures.

Remember what Linux Torvalds said?

"Bad programmers worry about the code. Good programmers worry about data structures and their relationships."

This intermediate data structure represents the native configuration of the backend in a python data structure (lists and dictionaries).

This data structure is a convenient way to simplify conversion in both direction, parsing and rendering, and will be different for each backend.

Take a look at this diagram:

Converter classes take care of converting between NetJSON and intermediate data structure (and vice versa):

• to_intermediate converts the NetJSON configuration dictionary to the intermediate data structure
• to_netjson (not pushed to the repository yet) converts the intermediate data structure to NetJSON configuration dictionary

These two main methods can then call helper methods in order to split complex logic in smaller methods that are easier for the human mind to grasp.

Renderers take care of rendering the intermediate data structure to the native format.

Parsers perform the opposite operation of Renderers: they take care of parsing native format and build the intermediate data structure.

In case of OpenWRT and AirOS, we can get away with just 1 renderer (with just 1 jinja2 tempate) and 1 parser. In the case of Raspbian, we may need to have a render and a template for each format that needs to be rendered (eg: /etc/network/interfaces, ini files, dnsmasq, iptables, ecc).

Advantages

• Cleaner, well defined workflow - also easier to document and learn
• Less boilerplate code in those cases where the native configuration of the router is centralized in one format (OpenWRT / AirOS / PfSense / vyos), because most of the logic goes in converters while there can be only 1 renderer and 1 parser
• Last but not least, makes it possible to implement backward conversion without going crazy with messy spaghetti code. Our sanity is very important not only for our work but for our life!

Disadvantages

Developers using lower level features of netjsonconfig in their programs and GSOC students working on the netjsonconfig backends will have to rewrite part of their code. A bit of more work needed, but in my opinion a worthy price to pay for a cleaner and more robust implementation.

How to upgrade your code

Proceed as follows:

• the logic that before was contained in the renderers, should most probably go into the converters
• design an intermediate data structure with the following features:
• closely represents your native configuration format (or formats)
• can be easily rendered with jinja2
• can be easily built when parsing the native format (or formats)
• simplify your renderers so that you have 1 renderer and 1 template for each format supported

Conclusion

That's it. Sorry for the length. I hope the explanation is clear enough.

Federico

[Federico Capoano]

I merged the new backward conversion feature and released it in netjsonconfig 0.6.1: https://github.com/openwisp/netjsonconfig/pull/83

In the near future we will be able to collect the whole device configuration and store it in openwisp2, which will make management easier.

A bit more work and building blocks are needed to get there, but the hardest part is done! FINALLY.

Federico