Fiber & Switch update

[Max Goldstein]

Hi all,

Got in to Control and found that we do have 4 new strands, with jumpers that even have the right connector on them. The loss as recorded on the ends of the lines was much smaller than Jesse's feared 50dB - around 10 (unitless, presumably dB). But there was also a range of numbers taped to each strand, which I'm hoping isn't loss as well. I also performed the local saturation tests and found the switches can move 800Mbits/sec over a superb connection. All of this is documented.

Up in the tower, I rackmounted the switch and was able to connect to it by an ethernet cable. However, I couldn't get a connection over either pair of strands. I was concerned to notice the tower jumpers did not have caps on the LC connector; five too-large caps were on the shelf. I went ahead and rackmounted CONRS1. It's not impossible I did something stupid like not push the connector into CONS1 all the way or something.

That's the big concern, minor notes follow.

I couldn't locate the rack screws and had to filtch some off the SH cube power supply. Any idea where they went?

Also, negative on the spare Furman, and there are a lot of plugs in the tower.

They're hanging coats on the ladder to the roof. *Shrug*

Max

[Andy Sayler]

I'd recommend going off the air for an hour with a partner in the studio and testing the new strands with our 10/100 single-strand box. We know that it works, and it will at least let you confirm that the strand numbering is correct, etc.

-Andy

[Max Goldstein]

...That's going to be a pain. I need to get in touch with Butch re the range and loss readings, so I can ask neutrally if he has any ideas.

Max

[Andy Sayler]

Not that much of a pain. Only takes about 5 minutes per line. Just walk the fiboxes on either wire down the list anf confirm connectivity over each strand. It would give you a better idea of whether the issue is the new sfps not handling the distance, or whether Datacom screwed up.

[Jesse Weeks]

If I had to guess I'd say the rack screws are in one of the drawers on top of the work bench labeled as such.

[Max Goldstein]

So unless Datacom gets back to me later this week (entirely possible), we've selected the empty timeslot next Tuesday at 4PM. Nick has volunteered, pending checking his schedule, to be my buddy in control. We'll meet in the station at around 3:40, go off air at 4, and hopefully be done in time for my class at 4:30.

Max

[Andy Sayler]

Bear in mind that it takes a few minutes for the fiboxs to acquire a 10/100 link. Then a quick ping test to confirm. That should allow you to test out the labeled lines and see if they work at all.

[Andy Sayler]

Did you guys get a chance to ping out the lines? Any luck?

-Andy

On Tue, Jan 15, 2013 at 7:41 PM, Max Goldstein <maxgol...@gmail.com> wrote:

[Max Goldstein]

Nothing from Datacom, so we're going to ping the lines Tuesday as planned.

Max

[Max Goldstein]

Nick and I will be meeting at the station at 3:30PM tomorrow to plan and gather equipment to take to the tower, and going off air at 4.

Max

[Andy Sayler]

If you don't get any connectivity with the 10/00 fibox on the lines as labeled, and if you have spare time, it may be worth doing an exhaustive end to end test (i.e. 1 -> 1, 1 -> 2... 2 -> 2, 2->3, etc) in case they just got the numbering wrong.

It may also be wise to power down the 10/100 box and power it back up on the currently in-use, known-good line to get a feel for how much time it takes to acquire a link. You can then use that timing (plus a conservative safety margin) as the baseline for testing the new lines. It may also be good to power cycle the 10/100 boxes between each line test to avoid any state issues that might arise from just plugging into a new line without a reboot (shouldn't matter, but why take risks).

Good luck!

-A

[Max Goldstein]

Good ideas Andy.

Nick, have your phone charged.

My saturation tests on the patch cable ran at 800MBits/sec. Gigbabit is 1000Mbits/sec, no? So are we not getting gigabit even over ideal conditions? Or did iperf crap out?

Max

[Andy Sayler]

There is always some overhead, so you never actually get the full 1000 Mbps. But you should be able to get better than 80%. My guess in iperf either hit a bottleneck elsewhere (processor, RAM, HD speeds, etc) or wasn't tuned quite right.

I wouldn't worry about it. Or at least not until we have at least basic connectivity working to the tower. Then we can look into tuning the details to squeeze out the max possible performance.

-A

[Max Goldstein]

It was closer to 850MBits/sec, but still. Glad to hear you're not concerned.

Max

[Nicholas Andre]

I'll bring my radios. They're way more badass.

[Max Goldstein]

We're not going for badass, we're going for quick and effective communication.

We can do exhaustive strand combinations if we have time, but Datacom tested the setup enough to have loss readings. I think the first thing we should test, after seeing how long the fiboxes take to reboot on the known-good strand, is each strand as marked, no mixing. I'll print out two copies of the fiber wiki page for reference.

Max

[Nicholas Andre]

So as an update to the proceedings, there were two mislabeled fiber strands. The not connected strand in Ballou is number 6, not 5, and the data com strands 3 and 4 had crossed, so to compensate we swapped the strands at Curtis. The wiki has been updated.

All strands connected near instantaneously (dare I say at near the speed of light) on the 10/100 unit once the swaps were sorted out.

So... why is it not connecting? We have tried swapping the strands out of the switch in case of polarity mis-match to no avail. We have also tried using different jumpers between the wall and the switches, also to no avail.

My best bet is some protocol issue (a la USB timeout over 7 meters) although that seems unlikely. Line loss (9.xx dB) may be the problem. Is the 10/100 link more resistant to line loss? 

Looking at the specs, it appears the max range is 550 meters. I assume we're exceeding that?

And I assume we made sure that this fiber is compatible with 850nm?

[Andy Sayler]

Inline.

-A

On Tue, Jan 22, 2013 at 2:36 PM, Nicholas Andre <nichola...@tufts.edu> wrote:

So as an update to the proceedings, there were two mislabeled fiber strands. The not connected strand in Ballou is number 6, not 5, and the data com strands 3 and 4 had crossed, so to compensate we swapped the strands at Curtis. The wiki has been updated.

Cool. 

All strands connected near instantaneously (dare I say at near the speed of light) on the 10/100 unit once the swaps were sorted out.

Noted. 

So... why is it not connecting? We have tried swapping the strands out of the switch in case of polarity mis-match to no avail. We have also tried using different jumpers between the wall and the switches, also to no avail.

My best bet is some protocol issue (a la USB timeout over 7 meters) although that seems unlikely. Line loss (9.xx dB) may be the problem. Is the 10/100 link more resistant to line loss? 

Looking at the specs, it appears the max range is 550 meters. I assume we're exceeding that?

And I assume we made sure that this fiber is compatible with 850nm?

A variety of reasons, not the least of which is that we may be pushing it well past spec. You should confirm with Datacom, but I believe they're using old FDDI 62.5 uM fiber. That means that the rated distance is only 220M (http://www.cisco.com/en/US/prod/collateral/modules/ps5455/ps6577/product_data_sheet0900aecd8033f885.html). I've estimated the tower run at about 1000M to 1500M, maybe more. That's the reason we were originally pushing for single-mode fiber lines. It also looks like the max rated loss for our transceivers on FDDI fiber is 2.4 dB, so if we're at 9 dB, we're several orders of magnitude above that (log scale).

I imagine that the 10/100 boxes are more robust over long distances. They use a propriety protocol and more powerful lasers. Unfortunately, we haven't been able to use them to connect the switches, likely due to a lack of support for switch trunking connections. It may be worth looking up their spec sheet and seeing what distance and line loss they are rated for as a comparison.

You may want to ask Datacom what they use to do Ethernet over their multi-mode fiber lines (that's what they use most of their fiber for, although a lot of their gear is single mode). It may give us a hint as to an alternate approach or hardware that may work better.

If you haven't already, you should also test a switch to switch connection in control using the same patch cables that you are using for the tower run on each end. I think you can connect the BNC ends of each patch cable to two sides of a female-to-female bnc connector in the wall box (you'll need to temporarily disconnect a few of the actual tower lines). That will prove out our patch cables and help to make sure that they aren't at fault here.

We could also explore trying to use LX instead of SX transceivers. They'll work with MMF lines if we use special patch cables. Not sure if they'll perform better or worse, but it is an option.

You may also wish to contact Cisco or dive into the docs to see if there is any options we can config on the switches to tune the transceivers or make them more robust.

Finally, if all else fails, you could contact Datacom and see if they will loan us a tech for a few hours to walk us through our line between Curtis and the Tower. There are a number of junction rooms it runs through (starting in teh Curtis basement). You could take a switch with you and jack it in at each junction to see how far you get before you lose the connection. Then we could look in to putting an amplifier at that point, etc.

If we just can't make it work, we can start looking at alternate options. These may include:

• Forgoing a tower switch and using the new spare fiber lines to run a bunch to 10/100 boxes (need to purchase), dedicated for each piece of tower equipment (and maybe a few for the server).
• Finding a way to make a switch to switch connection work over the 10/100 box link (may have been a config issue before)
• Looking into Ethernet radio and other point-to-point solutions for a separate control to tower connection outside of Datacom's control
• Looking into to propity point-to-point etherent fiber solutiosn that support higher line loss
• Having ITS install a few campus Ethernet drops in the tower with static IPs and setting up an intra-campus VPN
• Going back to the original single-mode fiber plan
  
• Some combination of the above

[Max Goldstein]

Thanks for the ideas Andy. I'll come to them in a moment.

To clarify re: mislabeling, the error with Ballou's old strand in my table on the wiki. The strand itself isn't labelled anywhere, just the wall box connectors it uses. It wouldn't have been a problem if Datacom had put a cap over the unused connector...I'll take spare up there next time. The more serious error is that strands 3 and 4, as labelled by Datacom, were swapped. We swapped them back in Curtis (easier to get to) but this likely means the loss readings on them are reversed. I made a note of this on the wiki; Nick has already fixed the table numbering.

Nick and I discussed the problem and came up with excessive line loss or batch patch cables. Because I didn't have a spare patch cable with me in the tower there wasn't a whole lot we could do if the cable issue was in the tower. I find this slightly more likely because the cables themselves were not capped either (I took a pair of spare caps up today). I suggested we could do this later as it does not require us to go off air. However, if a replacement cable is also bad, the problem would go undetected so I like Andy's idea of testing Datacom's patch cables. However if the cables are the problem we have at least two bum cables, one on each pair ... which makes it unlikely.

All of my correspondence with Butch at Datacom has been terse. I told him we couldn't get a connection a week ago ("but that could easily be a misconfiguration on my part") and asked for ideas, maybe to borrow testing equipment, and got no response. Therefore there's an argument to being stoically self sufficient and buying more 10/100 converters. Air feed, PDU, ethernet telemetry, server x2...that's our five strands. I did get Butch to tell me the MMF is 62.5 as Andy suspects, but open-ended ("any ideas?") requests for what they use will likely go poorly. They've been transitioning to single mode, and they don't demand end-to-end dedicated physical connections. Instead they have routers.

(As a curiosity, I ran a traceroute from our machine to the DNS server 130.64.212.28 and it went from the Anderson router to the Pearson router in one hop. Unsurprisingly a DNS server is near the network core. That's likely a single mode connection, because it's about the same distance as we need. The next hop, to the administrative building halfway to Davis, is certainly single mode.)

Back on topic, I'd rather not dump any more money into this than I have to, and the fiboxes run $280 a pop. I think our best first bet is to look at the switch config and docs and see if there's a way to make it more resilient to line loss. Doubtful, but quick and free. Because the switches will work at 100MBits/sec but not 10MBits/sec, it's possible we're only getting 10 across them. I'm not sure there's a way to test that without going off air again.

Next, because we'll want an external NIC for the tower server anyway, I think we should pursue the ethernet drop + VPN over campus idea. There is some new equipment (not ours)  in the tower since I was there a year and a half ago that looks like a switch, so this may be easier than you might think. Tufts internet is generally pretty fast, and by using the existing backbone we basically piggyback on singlemode fiber for free. (Well, probably SMF, can run a traceroute if someone has a Ballou IP. Not important.) We can probably treat the connections like fiber lines and use the switch, rather than a connection for each device. (Switch+tower server external NIC = access PDU even when our fiber goes dead.) I would recommend we keep the airfeed on the fibox+fiber just because it's more under our control, at least until we have proper silence detection and can fall back to it. Technical question: can we use the same NIC for the VPN and the internet? Depending on the answer we'll need two or three connections.

Paying 15 grand for the single mode trunk or more for an ethernet radio is out of the question. If Datacom doesn't cooperate with the ethernet drops we'll buy more fiboxes, and as a long shot see if they can be made to work with the switches. Buying an amplifier may be an option if we exhaust my first choices, and it's not the fiber patch cables.

TL;DR for ops: we'll be looking at the switch config in the coming weeks.
TL;DR for Jameelah and Gavin: please hold off on the UPS purchase for a little bit longer.

Max

[Max Goldstein]

Addendum: the fiboxes are rated for 2km over MMF, which may well explain the issue. We should get in contact with Cisco support and at least ask about LX SFPs, which can get up to 10km over single mode.

Max

[Andy Sayler]

If you can find the money, I'd go with the 5 fiboxes (4 new) + the Ethernet drop. That will avoid the complexity of a VPN while still providing direct tower Internet access when necessary.

Can you link to the fibox models we currently use?

If we have to buy 4 new fiboxes at around a cost of $1000 total, you might be able to cover that by selling the switch we had planned to put in the tower and the SFPs that go with it (although we could also use that switch in control and pull the old 10/100 HP switch for good). Also, if this year is to be a repeat of most years, you're going to have a decent lump of money after Spring Break (especially if you do the donations drive after break) that will need to be spent (or you will lose it) before the end of the semester. There may be fibox money there as well...

But start by testing the patch cables and confirming that you're properly routing the TX port on one switch to the RX port on the other and vice verca. (i.e. testing the patch strands both crossed and uncrossed on one side while leaving the other side alone; I assume this is something you've played with already, yes?)

-Andy

[Gavin Matthews]

Go to TAB if you get a chance and meet with Butch. You'll get your answers.

[Max Goldstein]

Nick, you said you swapped the strands as a test measure? Easy enough to do again.

The fiboxes are these. They're linked to on the fiber wiki page.

I still like the idea of using the switches. If we can get them to work using either a different config, a VPN, or a more powerful laser, I think that's ideal.

We don't know the dB loss on the old, in-service line, but I'm tempted to go off air for a few minutes and run some iperf tests on both the old and a new line. If we only get 10 megabit it can help eliminate the patch cables.

We probably will have the money for more fiboxes, especially if we only need 2 this year (ethernet telemetry and PDU). But let's explore other options first.

Meet with Butch? Crazy talk....

Max

[Andy Sayler]

Yeah, the current Fibox's still look like the best deal for 10/100 using a single strand of MMF. You can find cheaper ones, but they use 2 strands.

Aren't we using different patch cables for the 10/100 boxes then we use for the the switch SFPs? I'm not sure how your iperf test eliminates that patch cables as an issue...

As far as I know, the switch has no direct support for VPN. To use VPN, you'll need to either buy a VPN gateway and put it in the tower between a wall jack and the switch, or build one (i.e. a Linux server configured with OpenVPN and with multiple NICs acting as a VPN gateway). Most off-the-shelf VPN gateways are limited to 10s of Mbps, so that will never be an option for the Axia or really anything other than some SSH or basic PDU/UPS monitoring and control. You might be able to do a high speed VPN if you build a high performance OpenVPN server from scratch, but getting a gigabit link to the tower switch via VPN is going to be difficult.

By all means, explore other options first. Getting the switches to work would be ideal, but we may be running low on options for that. Testing a switch-to-switch connection with the same set of patch cables that you intend to use for the control to tower connection will be a good way to both prove out the patch cables and to confirm that you are getting the RX/TX matching correct (assuming that the control to tower lines are now labeled correctly and not themselves flipped as you corrected in your testing).

Also, a new idea: Cisco does build a 10/100 SFP for MMF rated at 2KM that should fit on our switch Gigabit SFP ports: GLC-GE-100FX (http://www.cisco.com/en/US/docs/interfaces_modules/transceiver_modules/compatibility/matrix/OL632702.html, need to confirm compatibility with our switch models). That would burn 2 fiber lines, however, as opposed to the fibox's one. Still, that might work well to provide a 10/100 connection between the switches for use with the PDU, telemetry  and server admin (ssh, etc). Then we continue to use the existing fibox for Axia, and consider buying a spare fibox for a direct server-to-server data connection. And we'd still have a strand to spare for either another fibox (more direct server bandwidth) or to go with a 2-strand 10/100 MMF NIC for the direct server-to-server connection.

-Andy

[Jesse Weeks]

On the GigE VPN front: our link to Tufts's network from control only runs at 100Mbps and I doubt they'd be very appreciative of us saturation a 1Gbps link.

Also the "new" hardware in the tower is the property of TEMS by way of public safety and is for their coms, so I doubt you'll every be allowed to tie into it in any way whatsoever.

regards,

[Andy Sayler]

The 10/100 2km SFP's () run about $120 for the OEM and $170 for the real thing.

In the documentation listed above, it looks like our switch (Cisco 2960G) is compatible with either the the gigabit SFP or the megabit SFP version of the 10/100 2km SFP: GLC-GE-100FX or GLC-FE-100FX.

[Andy Sayler]

The limiting VPN factor also is rarely the lien rate. It's the rate at which the VPN gateway or software can process the connection.

-Andy

[Max Goldstein]

If iperf reports we're getting a 10 megabit connection we know the switches aren't working because the bandwidth is too low, now because the patch cables are bad. (Well, they could be bad, but there's no longer a reason to think that.)

Jesse: two nails in the coffin for the VPN idea then.

The SFPs are an interesting idea. They go on Amazon for $165, so a pair won't be much more expensive than a fibox pair. That may be the thing to look into after swapping strands and rooting through the config.  A 10/100 connection between the switches is much more valuable than one or two directly between devices. At this point, I'll take the originally planned setup but at 10/100 if I can get it: two strands for the switches, 2 directly between servers, and 1 as an airfeed backup (somehow).

So, swap strands and dig through config, then investigate the long range SFPs.

Max

[Andy Sayler]

Or Newegg for $150 http://www.newegg.com/Product/Product.aspx?Item=N82E16833120291&nm_mc=OTC-FroogleNEW&cm_mmc=OTC-FroogleNEW-_-Network+-+Switch+Modules-_-Cisco+Systems++Inc.-_-33120291

Yea, let's see where we're at after you prove out the patch cables and RX/TX matching. Maybe config too (although I think the switches may treat the SFPs as black boxes with no low level tuning settings available).

[Max Goldstein]

SFP details: be careful. There's the GLC-GE-100FX and the GLC-FE-100FX. That's gigabit ethernet and fast ethernet (100 megabit). Both, judging by the page title, support only 100 megabit. Instead the distinctions refer to the capacity of the port. We have gigabit ports (right?), so we want the GE model which can be found for not much more at

http://www.newegg.com/Product/Product.aspx?Item=N82E16833120330

Except. Our switches are Catalyst 2960G with 24 ports (Control) and 8 ports (tower). The 24 port one appears to be compatible with the gigabit port SFP model. But the 8 port unit does not. We could go for the 100 megabit one, but anything we can do to get a little more power is worth doing. I think we may need to contact Cisco and ask for a clarification.

Before we make the purchase, I'm inclined to iperf the existing lines and see whether they're 100 or just 10 megabit. If the latter, we're hosed either way. Unless we know Livewire doesn't work over 10 megabit, which ... I don't know if that's true or not.

Max

[Andy Sayler]

As per my previous e-mail, it appears that the F (megabit SFP) version is compatible with both of our switches. I imagine the G (gigabit SFP) version is too, even if it's not explicitly documented as such.

I'm 99% sure that the SFP mode makes no difference in the laser power  The only difference is the SFP interface. Performance and speed wise they should be identical. And both are rated at 2km. I would, however  buy the same version for both sides. Just go with the F version since it's definitely supported by both switches and since they both should cover the distance no problem (similar tech to the fiboxes).

-Andy

[Max Goldstein]

Okay then. We'll investigate swapping strands and the switch config this week at ops, and if that doesn't work (I don't have my hopes up), we can order a pair of the long range F SFPs from Amazon in the same order as the UPSs.

They also make 100 megabit, 2km fiber NICs, which we'll investigate more fully when the time comes.

Max

[Max Goldstein]

Negative on swapping strands or promising switch config. Incidentally, the switch does recognize the SFP. We'll order a pair of the long range 100 megabit SFPs soon.

Max

[Andy Sayler]

You tested using the patch cables directly connected to each other (bypassing the long run)?

--
 
 

[Max Goldstein]

No, but I doubt that's the problem because (1) Datacom tested the lines and presumably the jumpers that they felt obligated to provide, and write the loss on, and (2) TWO jumpers, one on each pair, would have needed to fail for that to be the problem, and (3) the SFPs aren't rated for the distance we're asking them to cover. Not even close.

Max

[Andy Sayler]

A single jumper failure would break the connection. I agree its likely not the issue, but its an easy test to do to prove the patch cables are okay.

--
 
 

[Max Goldstein]

Andy, we have four new strands, so two pairs. There are four jumpers: two pairs times two ends. If strands 1&2 worked but not 3&4, I'd suspect the jumpers. But one jumper in EACH pair needs to be bad to explain what we're seeing.

Max

[Andy Sayler]

I though you were using the same jumpers with each pair and that you meant 2 of the 4 jumper strands would have to fail. With the clarification, I agree. Try 10/100 SFPs.

--
 
 

[Max Goldstein]

Ok. You may have gotten confused because we bought our own jumpers, but then Datacom provided their own for both strands and ends. I may not have said that explicitly.

Max