Subject: Ghost Route Hunter

From: "Jeroen Massar" <[email-masked]>
Date: Sun, 1 Dec 2002 17:34:52 +0100

I am proud to present a smallish analytical tool
we've developed for tracking down Ghost Routes.

The tool is called: Ghost Route Hunter and can be found at:
https://www.sixxs.net/tools/grh/

It analyzes routing table dumps and filters out all 'good'
ASpaths. The remaining ASpaths are too long and are so called
'ghost routes'. These Ghost Routes cause the blackholing effect
seen many times before when upgrading RIR space from a /35 to a /32.

We've just conducted a small test by announcing the Easynet
prefix 2001:6f8:e000::/35, which wasn't used and announced before.
Easynet only announces 2001:6f8::/35 at this moment due to the
danger of ghostroutes when upgrading it to a /32.

At 14:30* it wasn't visible yet as a ghost route, at the
next table collection at 14:45 it was visible as a ghost route
in both the routers of Tilab and Noris.
At 15:00 it was only visible on the Intouch router but it had
spread quite rapidly already around the world creating a long ASpath.
Then we retracted the route again and at 15:20 it fortunatly vanished.

Would this been a real announcement, eg by upgrading a /35 to a /32
this would have caused a blackhole for the complete /32 unless
the /35 would have been announced forever.

One very important thing we saw with this small test was the fact
that VERAT where originating the prefix at one moment.
Also DFN (JOIN) which appears in about 90%+ of all the ghost routes
should check up their equipment. Another possible important player
in this could be AS10318 (Cablevision S.A.) which isn't even in the
european continent nor peering directly with the ghosted prefixes.

Currently there are still 4 big ghost routes floating around:
- 3ffe:100::/24
netname:      TELEBIT
descr:        pTLA delegation for the 6bone
Which dropped of the internet around tuesday when looking at the latency
graphs*.
ipv6telebit.tbit.dk is unreachable over IPv4. The graphs also show that
it was
only reachable from two out six sites.

- 3ffe:1400::/24
netname:      UNI-C
descr:        pTLA delegation for the 6bone
remarks:      ***************************************
remarks:      * * * *  no longer operational  * * * *
remarks:      ***************************************

But still visible and ghosted and not officially retracted!

Last changed line:
changed:      [email-masked] 20010420

This would mean that the route would have been gone for over a year and
a half!
This route is currently announced by VERAT and Deutsche Telekom though.

- 3ffe:1e00::/24
ipv6-site:    SWISSCOM
origin:       AS3303
descr:        Swisscom Innovations

No netname available apparently. But it is currently announced by:
source: APNIC  
aut-num: AS4697  
as-name: NTTV6NET  
descr: NTT Software Laboratories  

I have already contacted these people seperatly, no response as yet.
Nothing in their 6bone object seems reachable.

- 3ffe:8010::/28
ipv6-site:    ICM-PL
origin:       AS8664
descr:        Interdisciplinary Centre for Mathematical and
Computational Modelling
              Warsaw University, Poland

Origin is currently ICM-PL and ICP-AS.
6bone-gw.6bone.pl is unreachable over IPv4 and IPv6

http://www.6bone.net/6bone_pTLA_list.html doesn't show that NL-BIT6/NL
was
returned. Apparently 3ffe:1400::/24 should say that too.

* direct links:
14:30
https://www.sixxs.net/tools/grh/ghosts/?year=2002&month=12&day=01&time=1
43017
15:00
https://www.sixxs.net/tools/grh/ghosts/?year=2002&month=12&day=01&time=1
50009
15:20
https://www.sixxs.net/tools/grh/ghosts/?year=2002&month=12&day=01&time=1
52013
Latency graphs: https://www.sixxs.net/misc/latency/

Companies mentioned have been CC'd.
This goes to both v6ops and 6bone as it causes many effects in both RIR
and 6bone space.

Greets,
 Jeroen

---

From: Gert Doering <[email-masked]>
Date: Mon, 2 Dec 2002 00:16:25 +0100

Hi,

> Then we retracted the route again and at 15:20 it fortunatly vanished.

I just want to point out that this is mostly normal BGP behaviour in
the face of highly meshed topologies and/or participating routers
with a slow CPU.  

The way BGP works upon withdraw (figure out what's the "second best
path", and happily announce that path to all your neighbors, and
do not announce the next update before minute has passed!) it's pretty
normal for a route to "ghost around" for 30 minutes or even longer
after a full withdraw.  If you look at a BGP table in that time, the
path will be changing every few minutes.

You can observe this in the IPv4 world as well - if you withdraw a prefix
completely, and then do traceroutes on that route, you'll see that it
will be routed for quite a while before it finally is dropped from the
DFZ.  It usually takes 15-20 minutes, but the IPv4 world is much less
tightly meshed due to policitical reasons.

(This behaviour is also known from RIP, and from there it's also called
"count to infinity" for BGP)

The thing that I have called "Ghost Routes" was something different - one
of the routers involved not properly propagating the withdraw to all
his neighbors.  After all other paths have disappeared, this prefix is
still seen by the neighbor (no regular "full sync" in BGP) and believed
until the next session clear.  This can be seen in the routing table as
a static BGP path that isn't changing over many hours or in some cases
even over many days or weeks.

[..]
> the /35 would have been announced forever.

... for those 75 minutes, yes.

> european continent nor peering directly with the ghosted prefixes.

I wouldn't point fingers at ASes and equipment yet (not without
evidence that the prefix really gets stuck for a longer time).

But it is strong evidence that the "6mess" needs to cleaned up 
Real Soon Now, i.e.: don't give everybody *and the whole world behind
him* transit everyhwere.

[..]
> Currently there are still 4 big ghost routes floating around:

Seconded, this looks very much like a ghost.  The paths that I observe
are completely unchanged since over 24 hrs.  From a cursory view, it
seems to be stuck in 1275 or between 1275 and 2603.

> netname:      UNI-C

I don't have that prefix.

> ipv6-site:    SWISSCOM

Seconded.  Overly long AS paths, static since > 24 hrs.

> ipv6-site:    ICM-PL

Seconded.  Overly long AS paths, static since > 24 hrs.

Gert Doering
        -- NetMaster
-- 
Total number of prefixes smaller than registry allocations:  54136  (50279)

SpaceNet AG                 Mail: [email-masked]
Joseph-Dollinger-Bogen 14   Tel : +49-89-32356-0
80807 Muenchen              Fax : +49-89-32356-299