Anyone else seeing "(invalid or corrupt AS path) 3 bytes E01100" ?

[Adam Hebert]

Multiple providers are seeing this right now. I assume someone is
advertising an extremely long AS_PATH again?

anyone else seeing this?

Adam

[Ballard, Eric]

We are seeing the same thing, has anyone found the offending AS yet?

Thanks
ERIC

[James Bensley]

Throw your coffee at them!

Just my two pence ;)

...James

-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GIT/MU/U dpu s: a--> C++>$ U+> L++> B-> P+> E?> W+++>$ N K W++ O M++>$ V-
PS+++ PE++ Y+ PGP t 5 X+ R- tv+ b+> DI D+++ G+ e(+++++) h--(++) r++ z++
------END GEEK CODE BLOCK------

[randal k]

Yep, we started seeing this right around 12:20pm MST. We saw it from a
customer's rapidly-flapping BGP peer. We told them to configure bgp
maxas-limit, but apparently CRS1s don't have that command.

Anybody have a handy route-map that will deny anything with a as-path
longer than say 15-20? ;-)

Cheers,
Randal

[Joe Provo]

On Mon, Aug 17, 2009 at 03:37:07PM -0600, randal k wrote:
> Yep, we started seeing this right around 12:20pm MST. We saw it from a
> customer's rapidly-flapping BGP peer. We told them to configure bgp
> maxas-limit, but apparently CRS1s don't have that command.
>
> Anybody have a handy route-map that will deny anything with a as-path
> longer than say 15-20? ;-)

Been a while since I had to throw this on cisco, but I since it lacks
sane repeat constraint, you have to either choose to iterate over your
acceptable space or deny on the longer-than-acceptable. For the latter,
^[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_([0-9]+_)+
clobbers 15 ASNs and longer.

--
RSUC / GweepNet / Spunk / FnB / Usenix / SAGE

[Ballard, Eric]

With the help from our transit providers and Cisco TAC the issues looks to be that AS9354 is sending AS0 and causing the corruption when processed in our Cisco CRS routers.

AS9354 shows to be Community Network Center Inc. (CNCI) or TDNC and directly connected to KDDI AS2516.

If anyone from AS9354 is on this list please contact me or stop this advertisement or someone from KDDI please assist.

Thanks
ERIC

[Jared Mauch]

On Aug 17, 2009, at 5:37 PM, randal k wrote:

> Yep, we started seeing this right around 12:20pm MST. We saw it from a
> customer's rapidly-flapping BGP peer. We told them to configure bgp
> maxas-limit, but apparently CRS1s don't have that command.
>
> Anybody have a handy route-map that will deny anything with a as-path
> longer than say 15-20? ;-)

Is there some significant barrier to people getting recent code on the
devices that is not impacted by this and the other fun bgp 'attacks'
that can happen? We usually see customers drop bgp sessions all over,
making me wonder ... if you're not able to upgrade, what is the
issue? Just that most people don't see these as an attack against
their infrastructure? That people are unwilling to upgrade code
unless it has a long-term impact to their operations? An outage once
every few months is OK?

- Jared

[dele...@gmail.com]

I'd have to _assume_ that a lot of those impacted don't have a maint contract with their router vendor of choice and therefore don't have an easy path to upgrade.
-jim

------Original Message------
From: Jared Mauch
To: randal k
Cc: na...@nanog.org
Subject: Re: Anyone else seeing "(invalid or corrupt AS path) 3 bytes E01100" ?

Sent from my BlackBerry device on the Rogers Wireless Network

[Jared Mauch]

On Aug 17, 2009, at 6:45 PM, dele...@gmail.com wrote:

> I'd have to _assume_ that a lot of those impacted don't have a maint
> contract with their router vendor of choice and therefore don't have
> an easy path to upgrade.
> -jim

Cisco gives out free software upgrades for any security(PSIRT) issue.

Take the recent 4-byte asn crash advisory:

-- snip --
http://www.cisco.com/en/US/products/products_security_advisory09186a0080aea4c9.shtml

Customers should have their product serial number available and be
prepared to give the URL of this notice as evidence of entitlement to
a free upgrade. Free upgrades for non-contract customers must be
requested through the TAC.

-- snip --

I would view this as an active attack against the internet
infrastructure and be working with the PSIRT team if it impacted my
network ...

- Jared

[Ricky Beam]

On Mon, 17 Aug 2009 18:40:39 -0400, Jared Mauch <ja...@puck.nether.net>
wrote:

> Is there some significant barrier to people getting recent code on the
> devices that is not impacted by this and the other fun bgp 'attacks'
> that can happen?

In a word: YES.

Any respectable ISP will not load code that has not been extensively
tested. Failure to do so can, and WILL, lead to even greater impact
outages. (we've all made that mistake. Once.) Unless you do millions
with Cisco and can therefore get custom IOS builds, you won't get a newer
version with *just* the intended bug fixed. Their maint "rebuilds" end up
with multiple "fixes" and all too often, previous fixes reverted. (I
stopped counting the number of times I had to bitch at them to refix the
SNMP DLCI interface counters on the 7401... "we don't test frame relay on
the 7401" -- sure, that's eons ago, but nothing has changed over there.)

And then there's the question of support... again, any respectable ISP
maintains maint contracts with their vendors. But, things tend to fall
through the cracks... contracts expire, people forget to list all the
equipment, vendors drop support for various hardware and software, etc.

You've obviously not gone to Cisco for any "non-contract" software
updates. It's faster to bribe someone with an active service contract or
use google.

Also... Never underestimate the power of Lazy!

--Ricky

[Paul Ferguson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, Aug 17, 2009 at 4:26 PM, Ricky Beam <jfb...@gmail.com> wrote:

>
> Any respectable ISP will not load code that has not been extensively

> tested. [...]

Just an observation on how things have changed in ~15 years:

I recall Cisco code bugs that were fixed in semi- real-time, and quotes
from tli: "Code still warm from compiler. Confidence level: Boots in lab."

:-)

- - ferg

-----BEGIN PGP SIGNATURE-----
Version: PGP Desktop 9.5.3 (Build 5003)

wj8DBQFKifMaq1pz9mNUZTMRAkIVAKD/G3MTDXLx9QydkEQq4azQZW4VGwCgrI+t
y30sajrGH98vhOhcyyW0wm0=
=CzXf
-----END PGP SIGNATURE-----

--
"Fergie", a.k.a. Paul Ferguson
Engineering Architecture for the Internet
fergdawgster(at)gmail.com
ferg's tech blog: http://fergdawg.blogspot.com/

[Marko Milivojevic]

> Confidence level: Boots in lab."

One could argue that certain things haven't actually changed that much ;-).

Marko.

--
Marko
CCIE #18427 (SP)
My network blog: http://cisco.markom.info/

[Brett Watson]

On Aug 17, 2009, at 5:17 PM, Paul Ferguson wrote:

> I recall Cisco code bugs that were fixed in semi- real-time, and
> quotes
> from tli: "Code still warm from compiler. Confidence level: Boots in
> lab."

IETF Dallas, 1995 I think. MCI Reston engg and Cisco (Ravi and others)
in the terminal room cutting a new image of IS-IS for us because we'd
tripped on the 24-day timer bug wherein all adjacencies would drop.
Loaded on production routers that evening to fix the problem... good
times :)

-b

[Kevin Oberman]

> From: Brett Watson <br...@the-watsons.org>
> Date: Mon, 17 Aug 2009 19:11:06 -0700

I remember back in 1993 or 1994 getting a note from Tony Li chastising
us for running BGP4 code that was over a WEEK old. How could we possibly
expect things to work with code that far out of date?

The amazing part is that BGP4 not only worked, but was very stable less
then a month later when NSFnet shut down an se had to go to BGP4 to peer
with everyone (like MCI, Sprint, NASA, and UUnet).

Yep, it booted in the lab, all right.
--
R. Kevin Oberman, Network Engineer
Energy Sciences Network (ESnet)
Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab)
E-mail: obe...@es.net Phone: +1 510 486-8634
Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751

[Ivan Pepelnjak]

> Anybody have a handy route-map that will deny anything with a
> as-path longer than say 15-20? ;-)

http://wiki.nil.com/Filter_excessively_prepended_BGP_paths

Ivan

http://www.ioshints.info/about
http://blog.ioshints.info/

[Dylan Ebner]

Ivan-
Thanks for posting this how-to on excessive as prepends. I have a couple of questions that some of the less BGP savvy out their may find helpfull

1. In my enviornment, we are not doing full routes. We have partial routes from AS209 and then fail to AS7263. Is their any advantage for someone like me to do this, as we are not providing any IP transit so we are not passing the route table to anyone else?

2. When I run the "sh ip bgp quote-regexp "_([0-9]+)_\1_\1_\1_\1_ \1_" | begin Network" I am seeing many many ASes that would be filtered by this access-list. What happens to those networks, are they unreachable from my AS, or do I just route those networks to my upstream provider and let them deal with it?

3. This last question is a little OT, but relates to your access list
In the event of some kind if DOS attack coming from one of a few AS numbers (in this case we will use 14793), what is the feesability of using
ip as-path access-list 100 deny _([0-9]+)_\1_\1_\1_\1_
ip as-path access-list 100 deny 14793
ip as-path access-list 100 permit .*

Would this have any affect at all, or would my pipe from my upstream still be congested with garbage traffic ?

Thanks
Dylan Ebner

[Ivan Pepelnjak]

> Ivan-
> Thanks for posting this how-to on excessive as prepends. I
> have a couple of questions that some of the less BGP savvy
> out their may find helpfull
>
> 1. In my enviornment, we are not doing full routes. We have
> partial routes from AS209 and then fail to AS7263. Is their
> any advantage for someone like me to do this, as we are not
> providing any IP transit so we are not passing the route
> table to anyone else?

You could do it to protect your BGP table, but as you're not a transit AS,
it does not make much sense.

> 2. When I run the "sh ip bgp quote-regexp
> "_([0-9]+)_\1_\1_\1_\1_ \1_" | begin Network" I am seeing
> many many ASes that would be filtered by this access-list.

Obviously a lot of people are prepend-happy.

> What happens to those networks, are they unreachable from my
> AS, or do I just route those networks to my upstream provider
> and let them deal with it?

If I understood correctly, you're using a default route toward AS7263, which
means that anything that is not in your BGP table (and consequently your IP
routing table) will be sent out of your AS via the default route. If you're
getting the paths you're filtering from AS209 that means that more traffic
will go to AS7263.

> 3. This last question is a little OT, but relates to your access list
> In the event of some kind if DOS attack coming from one of
> a few AS numbers (in this case we will use 14793), what is
> the feesability of using
> ip as-path access-list 100 deny _([0-9]+)_\1_\1_\1_\1_
> ip as-path access-list 100 deny 14793
> ip as-path access-list 100 permit .*
>
> Would this have any affect at all, or would my pipe from my
> upstream still be congested with garbage traffic ?

No. You cannot influence the inbound traffic apart from not advertising some
of your prefixes to some of your neighbors or giving them hints with BGP
communities or AS-path prepending. Whatever you do with BGP on your routers
influences only the paths the outbound traffic is taking. What you'd
actually need is remote-triggered black hole. Search the Nanog archives for
RTBH, you'll find a number of links in a message from Frank Bulk sent a few
days ago.

Hope this helps

[Dylan Ebner]

Ivan-
This helps vey much.

Thanks
Dylan Ebner

-----Original Message-----
From: Ivan Pepelnjak [mailto:i...@ioshints.info]
Sent: Tuesday, August 18, 2009 1:58 PM
To: Dylan Ebner; 'randal k'; 'Adam Hebert'
Cc: na...@nanog.org
Subject: RE: Anyone else seeing "(invalid or corrupt AS path) 3 bytes E01100" ?

[Joe Provo]

On Tue, Aug 18, 2009 at 09:37:22AM +0200, Ivan Pepelnjak wrote:
> > Anybody have a handy route-map that will deny anything with a
> > as-path longer than say 15-20? ;-)
>
> http://wiki.nil.com/Filter_excessively_prepended_BGP_paths

It will still be a while before we see unbroken 4byte AS behavior
(that whole 'fix the teardown on a anyone sneezing' problem). But
like with stale bogon filters, I expect folks inclined to use this
to drop it in and forget about it. So it would be wise to adjust
the recommended filter to anticipate a 2byteAS view allowing multiple
instances of AS-TRANS; there's likely a more elegant approach, but
the quick step of explicitly allowing _(23465_)+ before you deny

_([0-9]+)_\1_\1_\1_\1_

Cheers,

Joe

[Nathan Ward]

On 19/08/2009, at 6:58 AM, Ivan Pepelnjak wrote:

> No. You cannot influence the inbound traffic apart from not
> advertising some
> of your prefixes to some of your neighbors or giving them hints with
> BGP
> communities or AS-path prepending. Whatever you do with BGP on your
> routers
> influences only the paths the outbound traffic is taking. What you'd
> actually need is remote-triggered black hole. Search the Nanog
> archives for
> RTBH, you'll find a number of links in a message from Frank Bulk
> sent a few
> days ago.

Or, you can prepend your advertisement with the troublesome ASN.

Works for one or two troublesome ASNs as a quick hack at 3am - don't
do it unless you understand why it works and why you shouldn't do it.

--
Nathan Ward

[Nick Boyadjiev]

Dear Colleagues,

Sorry for the late response.

The problem was due to faulty firmware on one of our Alaxala routers.
We resolved the problem the same day (Aug. 18) by downgrading firmware.

For more details, please see Alaxala page here (English):

http://www.alaxala.com/en/information/20090827.html

If you have filters etc. blocking AS9354, please remove them.

Thank you,

Nick Boyadjiev
AS9354 Nagoya JAPAN