[Status] [MSO] BT: Blip for large number of BT lines (Open)

[Andrews & Arnold]

Posted at 2024-01-08 23:48:00 GMT by Andrews & Arnold

_Started: 2024-01-08 23:40:00_
The Drop and reconnect of a large number of BT circuits was due to
planned work: https://aastatus.net/42596


URL: https://aastatus.net/apost.cgi?incident=42597

--
AAISP Status Feed
URL: https://aastatus.net/

[Andrews & Arnold]

Posted at 2024-01-08 23:48:00 GMT by Andrews & Arnold

Update #1: 2024-01-08 23:51:30 GMT

_Started: 2024-01-08 23:40:00_
The Drop and reconnect of a large number of BT circuits was due to
planned work: https://aastatus.net/42596
>

> _Update 8 Jan 2024 23:48:17_ Lines are reconnecting, most are back
> online now.
>
> _Update 8 Jan 2024 23:51:15_ We have four links to BT. Two in each of
> our London Datacentres. BT have overnight work that will take one pair
> of these links out of service and would be disruptive. Usually we are
> able to cleanly shut down these links to drain traffic off them.
> Tonight, we shut down one of the links without a problem, but shutting
> down the second did cause a large number of BT circuits to drops and
> reconnect. This was not the intent, and we've been able to move traffic
> like this successfully in the past. We do apologise to those who had
> their service interrupted this evening.

[Andrews & Arnold]

Posted at 2024-01-08 23:48:00 GMT by Andrews & Arnold

Update #2: 2024-01-09 00:31:16 GMT

_Started: 2024-01-08 23:40:00_
The Drop and reconnect of a large number of BT circuits was due to
planned work: https://aastatus.net/42596

_Update 8 Jan 2024 23:48:17_ Lines are reconnecting, most are back
online now.

_Update 8 Jan 2024 23:51:15_ We have four links to BT. Two in each of
our London Datacentres. BT have overnight work that will take one pair
of these links out of service and would be disruptive. Usually we are
able to cleanly shut down these links to drain traffic off them.
Tonight, we shut down one of the links without a problem, but shutting
down the second did cause a large number of BT circuits to drops and
reconnect. This was not the intent, and we've been able to move traffic
like this successfully in the past. We do apologise to those who had
their service interrupted this evening.
>

> _Update 8 Jan 2024 23:59:56_ Lines should be back, many but not all are
> - we are investigating.
>
> _Update 9 Jan 2024 00:05:13_ We restored one of the links we shutdown
> and this restored service for those who were still offline. This does
> mean, that when BT do their work a bit later, that there is a risk of
> another blip or possible outage.
>
> _Resolution_ Affected lines are back online. This post will be closed
> now - but refer back to the parent post for further updates of BTs work
> on 9th January in the early hours of the morning:
> https://aastatus.net/42596

[Andrews & Arnold]

Posted at 2024-01-08 23:48:00 GMT by Andrews & Arnold

Update #3: 2024-01-09 08:18:13 GMT

[Andrews & Arnold]

Posted at 2024-01-08 23:48:00 GMT by Andrews & Arnold

Update #4: 2024-01-10 09:08:28 GMT

[Andrews & Arnold]

Posted at 2024-01-08 23:48:00 GMT by Andrews & Arnold

Update #5: 2024-01-16 16:10:23 GMT

_Started: 2024-01-08 23:40:00_
The Drop and reconnect of a large number of BT circuits was due to
planned work: https://aastatus.net/42596

_Update 8 Jan 2024 23:48:17_ Lines are reconnecting, most are back
online now.

_Update 8 Jan 2024 23:51:15_ We have four links to BT. Two in each of
our London Datacentres. BT have overnight work that will take one pair
of these links out of service and would be disruptive. Usually we are
able to cleanly shut down these links to drain traffic off them.
Tonight, we shut down one of the links without a problem, but shutting
down the second did cause a large number of BT circuits to drops and
reconnect. This was not the intent, and we've been able to move traffic
like this successfully in the past. We do apologise to those who had
their service interrupted this evening.

_Update 8 Jan 2024 23:59:56_ Lines should be back, many but not all are
- we are investigating.

_Update 9 Jan 2024 00:05:13_ We restored one of the links we shutdown
and this restored service for those who were still offline. This does
mean, that when BT do their work a bit later, that there is a risk of
another blip or possible outage.

_Resolution_ Affected lines are back online. This post will be closed
now - but refer back to the parent post for further updates of BTs work
on 9th January in the early hours of the morning:

| https://aastatus.net/42596. The cause of this drop has been identified
> as a mis-configuration within BTs network which prevented our traffic
> from failing over to our links to BT in Telehouse. This has been fixed
> by BT and we will retest.