For discussion: modelling duplex fibre links

[Brian Candler]

I am looking to deploy Netbox cabling model in several places, but it's the duplex fibre issue which is causing me the most grief.   I'd like to make some suggestions for how this could be improved, and would welcome constructive discussion.

First I'd like to review quickly the relevant parts of the data model:

* A Device can have interfaces (active), and front and rear ports (passive).

* Inside a device, a rear port links to one or more front ports, i.e. fan-out.  This models patch panels, including MPO break-out panels.

* Physically, a Device can be located at a site and rack and position; or a site and rack without position; or just a site.

* A Cable has two terminations, A and B.  Each end can be associated with an interface id, a frontport id, or a rearport id.  (There are also consoleport, consoleserverport, powerport, poweroutlet; but those aren't important here).  See CABLE_TERMINATION_TYPES in dcim/constants.py

* There is validation of what can connect to what, so for example you can't have a cable between an interface and a poweroutlet.  See COMPATIBLE_TERMINATION_TYPES in dcim/constants.py

The problem

Fibre plant consists of individual fibre strands. These strands need to be modelled individually.  Even when brought out to paired connectors like LC, you can use individual strands for separate paths - e.g. a single LC cable for BiDi SFP, or GPON.  In fact, if all links used a single bidirectional fibre, there would be no problem.

The problem is when you need a duplex (transmit/receive) fibre pair.  Netbox models this as a single interface, which could be an SFP or SFP+ for example.  In real life, you plug a transceiver into this; apart from the BiDi case, you need to provide two fibres for the path.  But Netbox does not model the SFP, nor does not let you put two cables into the same "interface".

Workaround 1? SFP as a "device"

It is possible to create a "device" representing an SFP.  This has one rear port and two front ports (for duplex).  Then you need a cable from the rear port of the SFP to the main device interface, and separate cables from the front ports to the patch panel.

This is ugly for many reasons, but primarily:

- it doesn't match the real world: there is no actual cable between the SFP and the Interface, it plugs in directly.

- the SFP is treated as a floating "device" which resides in the rack but with no rack position.  In reality, it is located within another Device.  Things plugged into a Device are normally modelled as "inventory items", but these don't have interfaces or ports.

- the device's interface neighbour would show as the SFP: you'd have to trace through it to see what's beyond

- it's really tedious to enter

Or SFPs could be modelled as "Child Devices" which fit into "Device Bays", but this is an abuse of the Device Bays feature (and still requires a fake cable between SFP and Interface)

Workaround 2? Split interfaces

If you have a device with interface Gi1/0/1, you could delete this interface and create two new ones: Gi1/0/1-Tx and Gi1/0/1-Rx.

This is fine for recording fibre paths, but is otherwise very messy:

1. the initial set of interfaces is copied from the Device Type template.  You'd be modifying the device each time you change from simplex to duplex and vice versa.

2. interfaces also represent logical interfaces: for example, an IP address needs to be associated with a single interface.  You could create a LAG called Gi1/0/1, with Gi1/0/1-Tx and Gi1/0/1-Rx as members, but that's even more work.

We need something better.

Proposal 1: Interface Module as a model

What we have in real life is an SFP, so what happens if we make an explicit model for it?  Let's called it "interface_module".

- The "location" of an SFP would be an Interface [on a device, not a VM].  This would be a simple foreign key to dcim_interface.  That is: an interface_module can only exist within an interface.

- The SFP would implicitly connect, on the rear side, with this interface.  This implies changing how ports are traced outbound from interfaces: they would have to look for either cables or interface modules.

- The SFP would have one or more outside connectors: these could be "front ports" or "interfaces".

The existing frontport model has a not-null rear_port_id.  So if we wanted to re-use this, this would have to become nullable, and we add a nullable interface_module_id field.  That is: a frontport could be associated either with a device and rearport, or with a device and interface_module.

The existing interface model already has a polymorphic relation: an interface can be associated with a device_id or a virtual_machine_id.  From this point of view, making an interface optionally associated with an interface_module instead might be easier.  It's also technically more correct, in that an SFP is really an active device, not a passive one: it really does have interfaces front and back, rather than passively-connected path.  On the other hand, the tracing logic would need changing to trace through interface modules as well as rearport/frontport.

You could also have an interface_module_port as completely different kind of entity, to avoid changing frontport or interface.  This works too; it would add some more options for cable terminations, and change the tracing logic a bit.

If an interface has an SFP plugged into it, should be made impossible to plug a cable into it at the same time, and vice versa.

In the device detail view, which shows all the interfaces, you'd still want to see the far-end that the SFP connects to as the "neighbour", rather than just the SFP itself.

To be properly useful, and interface_module instance would need to have a manufacturer and type, and optionally serial number and asset number.  This leads onto the second option:

Proposal 2: Interface Module as a type of Inventory Item

Inventory Items already are attached to Devices, and have type and serial number.

The suggestion here is very much along the lines of Proposal 1, except instead of having Interface Modules we have a special kind of Inventory Item.  Inventory Items already link to device, but as in proposal 1, Inventory Items would have to gain the ability to be associated with a specific Interface (that they are plugged into), and to sprout external Interfaces or Front Ports.  Again, tracing would need to be updated.

It certainly makes sense that an SFP plugged into a device should be recorded in the same way as a RAM module or a hard drive.  Remember though that it's not just plugged into a Device, it's plugged into a specific Interface.

Possibly related: there is a separate proposal (#824) to model line cards as inventory items.  If carried through, this would also mean that Inventory Items would sprout interfaces.  However as it stands, these interfaces would be interfaces on the Device itself, rather than interfaces downstream of an existing Device interface.

It should be possible to insert SFPs in those interfaces too.  Inventory Items already have a parent-child hierarchy - although it's not exposed in the UI (#1692).  But again, the association needs to be SFP to Interface, not SFP to Line card, for this use case.

Proposal 3: Cables with more than two ends

Proposals 1 and 2 introduce an "interface module" which has one rear port and two front ports, acting as a sort of adapter between an SFP interface and one or two fibres.

In real life, you can get fibre cables which are duplex at one end and 2 x simplex at the other.  So if we wish not to have SFP modules, but to maintain the pretence that a single "interface" accepts a single "cable", then one possibility is for a cable to have multiple ends: e.g. A, B1 and B2.  This could be modelled in a similar way to rearports and frontports on a device.

However, "cable terminations" don't exist as a separate database table currently; rather a dcim_cable instance has references to A and B terminations.  It would be possible to extend this to A and B and C, but it seems pretty messy.

The advantage of this approach is that cables with multiple ends do exist - I am thinking of things like QSFP breakout cables - and would benefit from being modelled correctly.  But QSFP is a more complex issue which I haven't covered yet.

Even more messy would be the ability to put two cables into the same interface - I don't think that warrants further consideration, but let me know if you disagree.

Issue: Path bifurcation

Netbox has a cable tracing feature.  There is a built-in assumption that a path is just a single path.  It works for frontport to rearport, and rearport to frontport (by exiting at the same offset as it entered).

However a trace which goes via an SFP with two fibres will have two traces for part of the run.  Hopefully those two traces end up at the same remote device!  But whether they do or not, it would be helpful to visualise it (#2633).

Currently Netbox will just traces the first fibre and ignores the second one.

Issue: Tx/Rx path swap

When you use LC-LC or LC-SC duplex cables, they are straight-through.  I don't know off-hand which way round it is, but let's say that when you plug into the patch panel, port 1 is TX and port 2 is RX.

This means that there's a crossover taking place somewhere in your fibre infrastructure.  Nice article here.

I think that Netbox can be made to handle this, if you are careful when configuring your patch panels when using N:1 breakout.  For example:

- Patch panel A: create rear port 1, map this to front ports 1-12.

- Patch panel B: create rear port 1, map this to front ports 2,1,4,3,6,5,8,7,10,9,12,11

- connect Patch panel A rear port to Patch panel B rear port

The ports on Patch panel A can be created in bulk using a range pattern.  Patch panel B is much more laborious to create.

But this isn't actually accurate: it's not the patch panel which is doing the fibre swap, it's the MPO cable between the panels.  I would therefore like to have a cable model which represents the cable types like those in the article linked above.

Netbox currently considers a cable as single virtual link from termination A to termination B; having different cable types with different strand mappings would solve this problem.  It could also simplify the existing tracing logic, which keeps a virtual offset when going from a front port to a rear port, and assumes the same offset when going from rear port to front port.  Instead, it would just follow the strand mapping from the cable type.

The implication here is that we'd add a "cable type" model.  At the moment, cables are just cables, with attributes like colour or length.  Cable types I think would be helpful anyway, especially to keep track of expensive or exotic cables.  This leads to:

Proposal 3a: Cables with more than two ends (again)

If a Cable Type is a model, then an instance of Cable points to the Cable Type.  The Cable Type can have all sorts of metadata including how many ends and what types they are, and how they internally connect.  The Cable instance can have its own attributes like colour and length, as it does today.

Now, suppose the Cable instances does not link to termination_(a|b)_id and termination_(a|b)_type_id; but instead, we make each termination point to the cable, turning it around completely.

That is: interface, frontport and rearport gain a "cable_id" field.  It's null when the port is not connected to anything, and points to a cable if it's connected to a cable.

For cables which are not symmetrical, we also need a "cable_port" field.  This could just be text or a number: if a cable type has 5 ends, then it would be 1-5.  It would be optional for symmetric, 2-ended cables.

I really like this model.  It intrinsically enforces that you can't put two cables into the same interface, and cables become much richer objects which match real world cables.

It's a fairly major change to the model, but I think it is feasible to implement: a migration could move the A/B termination values from the cables to the ports.

Although it would be possible to use such multi-headed cables as a way of avoiding modelling SFPs entirely (using duplex to simplex as mentioned before), ideally I think SFPs should be modelled as well.

Problem: Bundle interfaces

Netbox models physical interfaces and logical interfaces as the same thing, but this is not always true.  A key example is the QSFP 40G interface.

In the simplest case, you can connect QSFP-MPO-QSFP with a single cable, with only one device at each end.  This you can model today with a single 'cable'. It would be better if it had knowledge of how the strands are crossed over, as described earlier, but for establishing a logical point-to-point link it works.

A QSFP is physically one socket: it can appear to the OS as either a single bundled interface or as four 10G interfaces.  So far, that's not too hard: you could just create four logical interfaces if needed.

However, a QSFP is also really four physical interfaces in one port.  You can get a QSFP that drives 8 fibres (or 4 bidi); or a QSFP to SFP+ copper breakout cable which has one QSFP connector and four SFP+ connectors.  The other ends can connect to four separate devices, and are completely independent from each other, with their own IP address etc; they would have to be traced as four separate connections.

Hence it seems you need to model QSFP as four separate interfaces, which you can bond together in a LAG when needed make a virtual 40G interface.  That also makes sense.  The trouble is that a QSFP is a single module which plugs into one hole - even if it carries four separate circuits.  Having four interfaces breaks the model I outlined before, where an interface module has a single foreign key link to a single dcim_interface.

I think the proper way to fix this is to separate the concerns into two objects "ports" and "interfaces".  Each interface is associated with a port, and this can be N:1, e.g. for QSFP, 4 interfaces link to one port.  Cables plug into ports, rather than interfaces.  This is a major change to Netbox just to support this use case.  (But it does make sense in some other ways: for example, a virtual machine would have interfaces but not ports; and a virtual interface would be an interface with null for its associated port)

Apart from that, the simplest workable solution I can come up with is this:

 

* A QSFP port is a single "interface".  If you want to treat it as 4 x 10G interfaces on the OS side you create four virtual interfaces.

* A QSFP module plugs into a single QSFP interface

* A QSFP module can have 8 outside fibre interfaces, or 4 outside bidi fibre interfaces, or 4 outside copper SFP interfaces

* You would patch these with 8 fibres, or 4 fibres, or four copper cables

The problems are:

- an MPO point-to-point link has to be modelled as 8 separate fibre cables (laborious)

- a QSFP to SFP+ copper breakout cable has to be modelled as 4 separate SFP to SFP cables plus a virtual QSPF adaptor (probably the only option unless we model multi-headed cables)

- if you are using four virtual interfaces on a QSFP, you lose the ability to trace them end-to-end.  (To achieve this, you'd need some way to link a virtual interface to its parent physical interface, with an offset)

I'm not really happy with this.  But if you model a QSFP port as four physical "interfaces", you get into the problem about how you plug a single QSFP module into four interfaces at once.  I can't see how to resolve that properly without splitting "port" from "interface" as I already said.

I suppose it would be possible to turn it around: instead of the SFP module linking to an interface, the interface links to an SFP module.  This would allow for multiple interfaces to link to the same module.  This could become very messy in the user interface though.  You want to allow a QSFP to be plugged into four "interfaces" at once, without in general allowing a regular SFP to be abused in this way.  And really a QSFP should only be pluggable into four *related* interfaces.  I really don't like this option.

Other use cases

There are doubtless other use cases, some of which are covered in the discussion on #20 which was the original cable plant requirement.

There are also some usability optimisations.  Duplex cables are A Thing™, and it would halve the amount of time for entering patches if I could create a single duplex cable from SFP to a pair of ports on a patch panel.  (If we had "port pairs" or "port groups", it might also help with the QSFP case)

But from my point of view, if duplex SFPs into two individual fibre strands can be modelled, that's the 90% use case and I would be be very happy.

[Brian Candler]

On Wednesday, 6 February 2019 15:16:59 UTC, Brian Candler wrote:

Issue: Tx/Rx path swap

When you use LC-LC or LC-SC duplex cables, they are straight-through.  I don't know off-hand which way round it is, but let's say that when you plug into the patch panel, port 1 is TX and port 2 is RX.

This means that there's a crossover taking place somewhere in your fibre infrastructure.  Nice article here.

I forgot to add: you could simply ignore this problem for duplex links.  Assume that when linking SFPs it's perfectly OK to connect TX to TX, and RX to RX; that's being pragmatic :-)

But... if you are passing BiDi or GPON through your fibre infrastructure, you *do* care about these swaps that occur in the infrastructure.  A BiDi plugged into port 1 on patch panel A may come out of port 2 on patch panel B, and you definitely want to model that and trace that.

[Oli S.]

As mentioned on the github issue post https://github.com/digitalocean/netbox/issues/2633#issuecomment-457937138 we use from the past owed in the industrial sector exclusively ST connections in the fiber optic patch field infrastructure.

Any other type of fiber optic connector allows Gbic to belong directly to Patchpanelport. With LC- or SC- Duplex a connection fits to be only one, except there are single connectors.

In principle, this type of connection is difficult to document, especially in the LWL area, because there are various pitfalls.

I would also support the idea : Interface Module as a model.

When selecting an Ethernet (modular) interface I automatically have the possibility to use an appropriate cable.

You could of course also build a cable table in parallel, in which the types of connections are preserved as a matrix. Only this kind of cable connections can be connected accordingly when selecting an Ethernet (modular) interface.

[Brian Candler]

On Wednesday, 6 February 2019 15:16:59 UTC, Brian Candler wrote:

2. interfaces also represent logical interfaces: for example, an IP address needs to be associated with a single interface.  You could create a LAG called Gi1/0/1, with Gi1/0/1-Tx and Gi1/0/1-Rx as members, but that's even more work.

I have been trying out this workaround.  It's a lot of work, and you can't configure a LAG within the DeviceType template so you have to do it individually on every device.

More importantly though: you can't create a LAG out of LAGs, so this prevents you from creating LAGs between your SFP interfaces.