Subject: [GROW] RouteLeaks - problem or not?

From: Christopher Morrow <[email-masked]>
Date: Wed, 14 Nov 2012 17:18:19 -0500

GROW Folks,
The SIDR working group is working on security for origination and path
data related to BGP routes. There has been a note (a few) about SIDR's
effect(s) or not on 'route leaks'. There have even been a few notes on
'what is a route leak'. To date there is a draft which discusses route
leaks:
  <http://tools.ietf.org/html/draft-foo-sidr-simple-leak-attack-bgpsec-no-help-02>

where the authors have attempted to describe one (or many possible)
situations which are called 'route leaks'. They also attempt to
outline security issues which are follow-on effects of the situation
described.

SIDR attempted to look at route-leaks and came up a bit stymied, they
asked IDR for some assistance with the issue, IDR pushed back to GROW
to decide:
  1) What is a 'route leak' (perhaps the above draft identifies one
examplar to be used in that definition)
  2) Are 'route leaks' a problem that Operations folks care about
  3) Should IDR (or the IETF proper) address 'route leaks' with some
form(s) of fix action.

The end result of the above 3 steps is to push back into IDR one of
two action requests:
  1) "Yes, route leaks are a problem, please fix them."
     or
  2) "No, route leaks are not a problem, take no action."

If #1 above is the answer, and IDR decides that changes to the BGP
protocol are warranted (or are a possible solution to the problem)
then SIDR has agreed to do what they can to 'secure' the bits
added/changed/used in that endeavor.

Could we have some discussion on-list about this problem, and some
discussion about whether or not the draft referenced above fits the
definition we would like to use for 'route leak'? I would also like
the authors of the draft to decide where they would like to take their
draft:
   1) SIDR
   2) IDR
   3) GROW
   4) other

Thanks!
-Chris
(co-chair 1:2 of grow, and 1:3 in sidr)

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 14 Nov 2012 17:44:25 -0500

Apologies for the extra spam, there was a topic point I missed in this
mail, see below please.

> outline security issues which are follow-on effects of the situation

Additionally there were several drafts written by Brian Dickson aiming
to provide some definitions about route-leaks and some direction for a
solution, they are:
  <http://tools.ietf.org/html/draft-dickson-sidr-route-leak-def-03>
  <http://tools.ietf.org/html/draft-dickson-sidr-route-leak-solns-01>
  <http://tools.ietf.org/html/draft-dickson-sidr-route-leak-reqts-02>

These are probably best read in reverse numerical order:
   definition
   requirements
   solution

I believe the author aimed to talk about this in a GROW meeting, or on
the GROW list, I have not seen him pipe up as of yet to that end,
however.

-chris

---

From: Jared Mauch <[email-masked]>
Date: Wed, 14 Nov 2012 18:14:06 -0500

>  2) Are 'route leaks' a problem that Operations folks care about

Side comment:

I fixed two problems in the tool in the past day or two.  It appears that 3561<->209 integration is happening and another was related to Comcast which has been corrected.

I certainly support the premise presented int his draft and that this is a significant issue.  The IETF network was subject to a problem like this for Atlanta…

130.129.0.0/16	 3549 2828 10026 4538 4538 4538 4538 3320 7922 7725 56554 

Not sure if you can still view the ticket, but It was here: 

trac.meeting.ietf.org/ticket/536#comment:1

- Jared

---

From: "McPherson, Danny" <[email-masked]>
Date: Wed, 14 Nov 2012 23:34:03 +0000

On Nov 14, 2012, at 6:14 PM, Jared Mauch wrote:
> 
> I certainly support the premise presented int his draft and that this is a significant issue. 

In addition to Jared, Shane, Eric and I, other folks said route leaks are a problem as well on SIDR.  I'll provide full fragment for context below, and they can chime in if they so choose...

===============================================
Begin forwarded message:

> From: Christopher Morrow <[email-masked]>
> Subject: Re: [sidr] The need for SIDR - Google limited outage today due to bogus route announcement
> Date: November 7, 2012 3:56:33 PM EST
> To: Danny McPherson <[email-masked]>
> Cc: "[email-masked] list" <[email-masked]>
> 
> On Wed, Nov 7, 2012 at 2:06 PM, Danny McPherson <[email-masked]> wrote:
>> 
>> On Nov 7, 2012, at 1:48 PM, Christopher Morrow wrote:
>> 
>>> 1) show/agree that this is a problem (route leaks)
>> 
>> Do you believe this is a problem?  When describing events such as this as
>> of late, what did you call it?
> 
> sure, but I'm not the grow-wg ... right? aim that question over ->
> <mailto: [email-masked]> there.

===============================================

Begin forwarded message:

> From: Randy Bush <[email-masked]>
> Subject: Re: [sidr] The need for SIDR - Google limited outage today due to bogus route announcement
> Date: November 7, 2012 10:22:16 PM EST
> To: Christopher Morrow <[email-masked]>
> Cc: sidr wg list <[email-masked]>
> 
> <pedantry>
> 
>> "'Route leaks" are viewed as a routing security problem..."
> 
> route leaks, as we anecdotally know them, are an operational problem.
> imiho, they are not particularly a security problem.  but that does not
> mean i think the ietf routing community should not be working on a
> solution, quite the opposite.
> 
> randy

===============================================

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 14 Nov 2012 23:52:53 +0000

> examplar to be used in that definition)

dunno really.  Some foo which involves prefixes being sent to places where
they oughtn't have been sent to.  This is possibly more of a legal /
wordsmithing problem than anything else.

>   2) Are 'route leaks' a problem that Operations folks care about

speaking with my IXP operator hat on, hell yeah.  INEX operates a route
server cluster which provides service to ~70% of our members.  If we were
to operate an unprotected system and had a prefix leak from a client, it
would destroy confidence in the system and people would depeer.
Connectivity would be directly harmed.  In my experience, it takes very
little for people to lose trust in a system like this.

We use strict prefix filtering based on IRRDB data with a vengeance, but
ended up also having to implement maxprefixes after someone attached the
INEX route server as-set macro to their own as set macro (mistake #1), then
announced all the prefixes they received back to INEX (mistake #2).  As it
happened, their prefixes were depreferred because of the asn path length,
but if they hadn't been depreferred, it could have taken down the entire
system.  Ouch.

Anyway, point being: yes: route leaks matter, yes: at least some operations
people care, and belt + braces is sensible.

>   3) Should IDR (or the IETF proper) address 'route leaks' with some

mmmm, difficult one.

So the current thinking seems to be heading in the direction of RPKI which
has a lot of interesting technical characteristics, probably the most
interesting of which is the max prefix length.  One interesting
characteristic of RPKI is that in order for it to have any real function on
a live network, it will be necessary to completely drop rpki-invalid
prefixes.  Merely depreferring them is a bit useless.

Problem is, the current implementations of rpki only deal with origin
validation, and not as path validation.  Work on this has begun in terms of
a list of specifications (draft-ietf-sidr-bgpsec-reqs), but hasn't
progressed much beyond that as far as I can tell.  I wonder if some of the
sine-qua-nons in this document are mutually exclusive.

RPKI will only be fully functional from the point of view of preventing
leaks (whatever leaks are) when we have full as path validation.

Being honest, RPKI concerns me from two points of view.  From a technical
point of view, BGP is a fully distributed system which forms the basis of
global network reachability, yet RPKI seeks to impose a top-down control /
validation mechanism for it, albeit based on many potential roots.  I
cannot seem to stop being concerned that this is an architecturally bad
idea from a technical point of view, even though there are many safeguards,
including:

- recommendations for using local caches
- lazy validation
- recommendations for validation only at ASN boundaries
- recommendations in draft-ietf-sidr-origin-ops

... even still, there is a double architectural inversion going on here,
where a full-distributed, very low level protocol is being made to depend
on a top-down, much higher level protocol.

>From a non-technical point of view I also have serious concerns about RPKI
in terms of it being used as a hammer that politicians / bureaucrats / LEAs
/ judges / etc are (and that's "are" as in present tense) extremely
interested in as a potential future means of blocking content / services
which they happen not to like.  Although GROW is not the place for this
particular discussion, this aspect of RPKI concerns me to the extent that I
am disturbed about the wisdom of developing the technology in the first
place:  we're building an on/off button for prefixes throughout the
Internet.  We need to be careful what we wish for because one day, we might
get it.

Nick

---

From: Jared Mauch <[email-masked]>
Date: Wed, 14 Nov 2012 18:57:07 -0500

> wordsmithing problem than anything else.

I would say unexpected paths that appear as a result of poorly constructed routing policies or software defaults and lax filtering policies.

My "leak info" tool does fuzz detection and find a lot of these things all day long.

- Jared

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 14 Nov 2012 19:02:17 -0500

On Wed, Nov 14, 2012 at 6:52 PM, Nick Hilliard <[email-masked]> wrote:
> validation, and not as path validation.

one clarifying point, the RPKI doesn't actually say anything at all
about 'path' data. I merely says which origin-as(es) should be able to
announce a prefix.

---

From: Nick Hilliard <[email-masked]>
Date: Thu, 15 Nov 2012 00:12:31 +0000

> about 'path' data. I merely says which origin-as(es) should be able to

i'm being slack about my usage of the term rpki here, and should have
referred to bgpsec instead.

Nick

---

From: Arturo Servin <[email-masked]>
Date: Wed, 14 Nov 2012 22:38:02 -0200

Nick,

	Perhaps this is not the place, but ...

	I am still finding these arguments about RPKI as a bunch of fud (please
do not take it personally). I think that are much easier, cheap and
practical methods (or hammers) that would accomplish the same goal. In
fact, any entity trying to use RPKI as a hammer I would say that it is
very badly advised (to said the least).

Regards,
as

<snip>

---

From: Nick Hilliard <[email-masked]>
Date: Thu, 15 Nov 2012 00:46:54 +0000

> 	Perhaps this is not the place, but ...

It probably isn't.  Let's discuss the technical issues.

Nick

> _______________________________________________
> GROW mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/grow
> 

-- 
Network Ability Ltd. | Chief Technical Officer | Tel: +353 1 6169698
3 Westland Square    | INEX - Internet Neutral | Fax: +353 1 6041981
Dublin 2, Ireland    | Exchange Association    | Email: [email-masked]

---

From: Arturo Servin <[email-masked]>
Date: Wed, 14 Nov 2012 23:08:16 -0200

Chris,

	I think that draft-foo-sidr-simple-leak-attack-bgpsec-no-help-02 shows
very well why rpki-origin-validation and bgpsec are vulerable to route
leaks. Also it has a good and simple example of a route leak. One
drawback that I see is that it does not explain the problem in general,
the document is focused on route leaks as result of
Man-in-the-middle-attack. Recent events relating leaks showed us that an
attack it is not necessary, just some fat fingers in the right place and
you are done.

	draft-dickson-sidr-route-leak-def-03 is more general, it tries to
explain route leaks as an event and not as an attack. However the level
of granularity and detail makes the document and the concept of leak
hard to understand (perhaps it is because English is not my first
language) and I found the simpler example in
draft-foo-sidr-simple-leak-attack-bgpsec-no-help-02 much easier to follow.

	The other two documents (draft-dickson-sidr-route-leak-solns-01 and
draft-dickson-sidr-route-leak-reqts-02) are about solutions and
requirements of a solution. I think for now are beyond us. First I would
prefer to define very well a route leak and then finding a solution.

	In my opinion some merging or re-writting of
draft-foo-sidr-simple-leak-attack-bgpsec-no-help-02 and
draft-dickson-sidr-route-leak-def-03 could result in a good explanation
of leaks and why they are a problem. However, what ever were the next
step, I would suggest to keep
draft-foo-sidr-simple-leak-attack-bgpsec-no-help-02 or some sort of it
because it may be useful in the future (or now) to describe the
limitations of rpki-origin-validation and bgpsec.
	
	And finally, I would prefer (now that sidr and idr have stepped out to)
to work on the definition of the route-leak problem here. After defined,
we could move it either to sidr or idr (although I think both have a
said on the possible solution).

my 20 cents,
as

---

From: Shane Amante <[email-masked]>
Date: Wed, 14 Nov 2012 21:33:33 -0700

Chris,

See below.

On Nov 14, 2012, at 3:18 PM, Christopher Morrow <[email-masked]> wrote:
[--snip--]
> To date there is a draft which discusses route leaks:

See aforementioned draft.  It's the most _concise_ definition of the problem, as observed repeatedly in the Internet.

>  2) Are 'route leaks' a problem that Operations folks care about

Yes!

>  3) Should IDR (or the IETF proper) address 'route leaks' with some

WRT "Should IDR" ... that's like Henry Ford asking what color "Model T" I'd like, right?  Look, no offense to the folks in IDR :-), but as with every other WG their present charter is narrowly defined to 'enhancements to BGP'.  I would hope that _when_the_time_comes_ we remain open-minded about conducting a thorough evaluation of the solution space, before deciding to further refine one, or more, of those solutions to a standard, or set of standards.  So, presently, I would not be in favor of exclusively asking IDR to fix 'route leaks', given the concerns I've raised above.

FWIW, I'm not clear on what you're proposing with "the IETF proper" ... so I can't say "yes" to that option either.

> added/changed/used in that endeavor.

Dare I ask what happens if IDR decides they do not have an answer?

> definition we would like to use for 'route leak'?

Um, yes, but then again I'm a co-author, so clearly you should take this answer with a healthy dose of salt.  :-)

> I would also like

IMO, since you're asking GROW, the answer should hopefully present itself.  :-)  

-shane

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 15 Nov 2012 00:27:37 -0500

Shane,

See below.

> WRT "Should IDR" ... that's like Henry Ford asking what color "Model T" I'd like, right?  Look, no offense to the folks in IDR :-), but as with every other WG their present charter is narrowly defi…

apologies, I was a bit rushed in my message (and I note my first
question was poorly chosen, as well) but I had meant to say that the
placement of a  solution doesn't have to be in bgp (so doesn't have to
involve IDR), if the WG here takes a look at sees other directions to
go.

> FWIW, I'm not clear on what you're proposing with "the IETF proper" ... so I can't say "yes" to that option either.

'should the ietf care about this problem, and spend resources
attempting to provide solutions?'

is that better? I'm really asking whether or not there is a problem
and if we (ietf in general) can get a solution (or even the
requirements for a solution?) defined...

-chris

---

From: Job Snijders <[email-masked]>
Date: Thu, 15 Nov 2012 08:06:27 +0100

Hi,

On Nov 15, 2012, at 12:57 AM, Jared Mauch <[email-masked]> wrote:

> I would say unexpected paths that appear as a result of poorly constructed
> routing policies or software defaults and lax filtering policies.

I'd define it as following: 

	Route Leak: When a misconfigured BGP speaker sends an NLRI Update to another
	Autonomous System and subsequently becomes a potential transit network for
	the given NLRI while this transit service is not aligned with the routing
	policy between the two networks, the NLRI Update is considered a route leak.

Kind regards,

Job

---

From: Warren Kumari <[email-masked]>
Date: Thu, 15 Nov 2012 10:21:41 -0500

> GROW Folks,
Hi!

[SNIP]
>  2) Are 'route leaks' a problem that Operations folks care about

Yes… 

>  3) Should IDR (or the IETF proper) address 'route leaks' with some

Yes… 

> added/changed/used in that endeavor.

Personally I'm not convinced that the resolution to the "route leaks problem" needs to occur in BGP / needs to update BGP, so not sure if this gets 'fixed" in IDR, but I think that is does need to be worked on.

--
Credo quia absurdum est.

---

From: Shane Amante <[email-masked]>
Date: Thu, 15 Nov 2012 21:10:59 -0700

Chris,

> attempting to provide solutions?'

Yes.

> requirements for a solution?) defined...

Yes, there is a (big) operational problem wrt route-leaks.

Yes, we (IETF in general) should work toward a solution.  As I stated previously, I hope we remain open-minded as to the solution space and do not pigeon-hole ourselves into thinking that one must exist inside BGP, either solely or at all.

-shane

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 15 Nov 2012 23:46:31 -0500

> Yes, there is a (big) operational problem wrt route-leaks.

it'd be good to outline that as well in the document... often the
'mitm' problem is brought up, but honestly there are probably more
cases of:
  1) suboptimal routing (why am I routing through Chile to get to my
next-door-asn?)
  2) traffic loss (via too-small-pipes + congestion)
  3) link costs (lookie at the transit pipe being full instead of the
other one!)

that happen and are simpler to show than other issues.

---

From: Warren Kumari <[email-masked]>
Date: Fri, 16 Nov 2012 21:18:45 -0500

> that happen and are simpler to show than other issues.

… and easier to talk about as well :-)

I think that you left out:
2.5) Hey, where the hell did my traffic go?! Bah, someone over there fat fingered and all my traffic follo .. and it's back...

W

--
Do not meddle in the affairs of dragons, for you are crunchy and taste good with ketchup.

---

From: MAWATARI Masataka <[email-masked]>
Date: Sat, 17 Nov 2012 13:47:29 +0900

Greetings,

* On Wed, 14 Nov 2012 23:52:53 +0000

> people care, and belt + braces is sensible.

I agree.
Max-prefix-limits feature is one of the operative tools for mitigating
route leaks.  However in case of operating the route servers in IXP,
threshold of limit also is a important consideration.
Sometimes new members advertise the full routes into route servers,
so IXP operators would like to prevent that.
Combination of route filter and max-prefix-limit is reasonable.

Kind Regards,
Masataka MAWATARI

---

From: Rob Shakir <[email-masked]>
Date: Mon, 19 Nov 2012 15:57:29 +0000

> Yes, we (IETF in general) should work toward a solution.  As I stated previously, I hope we remain open-minded as to the solution space and do not pigeon-hole ourselves into thinking that one must…

Hi GROW, All,

I would support the comments that Shane made here - essentially, this is an operational concern, and we should at least properly document what the problem is. If we can use this problem statement to work towards a solution, then this would be a good aim.

However, I would urge that we consider the operational complexity, manageability and overhead of any suggested solutions. A number of the mechanisms in the sidr space have significant cost of implementation, which I'm not sure is wholly being considered, and may present a challenge for deployment of these solutions.

Specifically regarding route leaks, I have some concerns around the ease of classification of relationships between networks (that I raised in idr when this was discussed) -- whilst it may be convenient to consider there to be clearly defined categories in this space, in the real world, I do not think that it is as clear cut. 

My view is that GROW is the right place to begin this work within the ietf -- however, some consultation with the wider operational community (e.g., NANOG etc.) is likely to be advantageous to tap into the best sources of knowledge on this matter. A solution specified without considering those operators which are not represented in the ietf is likely to be limited in its deployment.

Cheers,
r.

---

From: "Sonalker, Anuja" <[email-masked]>
Date: Mon, 19 Nov 2012 11:50:29 -0500

Hello GROW

Bit slow on the replying over here, sorry for the late chime. 

Thoughts in line below....

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Rob Shakir
> Sent: Monday, November 19, 2012 10:57 AM
> To: Shane Amante; [email-masked] [email-masked]
> Cc: [email-masked]; draft-foo-sidr-simple-leak-attack-bgpsec-
> [email-masked]; [email-masked]
> Subject: Re: [GROW] RouteLeaks - problem or not?
> 
> 

Agree with everyone!! 

> >
> > Yes, we (IETF in general) should work toward a solution.  As I stated
> previously, I hope we remain open-minded as to the solution space and
> do not pigeon-hole ourselves into thinking that one must exist inside
> BGP, either solely or at all.

Completely agree with this comment. imho part of our problem is that even though we know what route leaks are when we start defining them it gets complicated very quickly. There are several aspects that have a qualitative/subjective and to address those we may have to think outside of bgp. The operational perspective is the most important one and therefore GROW is the best place for taking up this problem, imo. 

> 
> Hi GROW, All,
> 
> I would support the comments that Shane made here - essentially, this
> is an operational concern, and we should at least properly document
> what the problem is. If we can use this problem statement to work
> towards a solution, then this would be a good aim.
>

This is a great start. There are probably a few more cases that fall into the Route leaks bucket that we should consider. I would be happy to share them with the authors and everyone on the mailing list. 

> However, I would urge that we consider the operational complexity,
> manageability and overhead of any suggested solutions. A number of the
> mechanisms in the sidr space have significant cost of implementation,
> which I'm not sure is wholly being considered, and may present a
> challenge for deployment of these solutions.
> 
> Specifically regarding route leaks, I have some concerns around the
> ease of classification of relationships between networks (that I raised
> in idr when this was discussed) -- whilst it may be convenient to
> consider there to be clearly defined categories in this space, in the
> real world, I do not think that it is as clear cut.
> 

Similar to yours, but one concern I have is that there are several non-engineering rules that also contribute to route leaks. Should we even open the can on them, or just keep it to engineering and operations while defining the problem?

> My view is that GROW is the right place to begin this work within the
> ietf -- however, some consultation with the wider operational community
> (e.g., NANOG etc.) is likely to be advantageous to tap into the best
> sources of knowledge on this matter. A solution specified without
> considering those operators which are not represented in the ietf is
> likely to be limited in its deployment.

+1

--Anuja

---

From: Michael Hallgren <[email-masked]>
Date: Mon, 19 Nov 2012 22:06:20 +0100

> Specifically regarding route leaks, I have some concerns around the ease of classification of relationships between networks (that I raised in idr when this was discussed) -- whilst it may be conve…

Right, I think we all know by now that modeling commercial or social
desire or need is 'hard'. A good starting or way point would thus likely
be to carefully study the problem space, in order to keep our pieces to
develop as simple (and well defined) as ever possible.

For instance, as you hint, inter domain routing knobs are currently
quite often "used" to patch billing (or collection of sufficient or
cheap enough billing data) shortcomings, which IMHO is a bit
unfortunate. 

> My view is that GROW is the right place to begin this work within the ietf -- however, some consultation with the wider operational community (e.g., NANOG etc.) is likely to be advantageous to tap…

Yep.

Cheers

mh

---

From: Russ White <[email-masked]>
Date: Tue, 20 Nov 2012 09:34:55 -0500

> attack it is not necessary, just some fat fingers in the right place and

In reality, what we are calling a "man in the middle attack," has always
just been one instance of the more general "route leak" problem. The
only way to know if a route has been "leaked," is to know what the table
_should_ look like.

BGPSEC (and the original drafts, S-BGP, on which the BGPSEC
"requirements" are based) is focused on "this is what really happened,"
rather than, "this is what should be happening." The interesting point
of security (if we want to prevent problems rather than just provide
forensics) is, "this is what should be happening."

>>>   1) "Yes, route leaks are a problem, please fix them."

Yes, route leaks are a problem, and there needs to be some way to guard
against them, which means, we're going to have to talk about intent.

>>> added/changed/used in that endeavor.

This, OTOH, I disagree with. The most direct way to solve this problem
isn't to "add bits to BGP." It's to provide a general framework in which
intent can be exposed where necessary. Let's not pretend that this
"route leak" problem is the last and only problem to be solved, nor that
the solution to all possible future problem is to continue to "add bits."

>>>    1) SIDR

SIDR is the correct place to address this problem, though I know it
means reconsidering the entire suite of requirements and currently
proposed solutions. OTOH, that's a good thing, because it might make us
go back and look at real requirements, rather than trying to narrow the
requirements to the set that can be solved with an already proposed
solution.

:-)

Russ

-- 
<><
[email-masked]
[email-masked]

---

From: <[email-masked]>
Date: Wed, 21 Nov 2012 14:59:49 +0000

>cheap enough billing data) shortcomings, which IMHO is a bit

Yes - exactly.  The other consideration here is around how willing certain
networks are to expose such "patches" being used.  The situation is
unfortunate, but at the risk of stating the obvious, if we do not consider
real world behaviour (as Russ commented), then a solution will not solve
real world problems.

r.