Subject: RE: Last Call: <draft-ietf-grow-blackholing-00.txt> (BLACKHOLE BGP Community for Blackholing) to Proposed Standard

From: Michel Py <[email-masked]>
Date: Tue, 21 Jun 2016 03:39:17 +0000

I have to point out that, although it looks like a move in the right direction, the newly created BLACKHOLE community is likely to meet severe max-prefixes resistance. It is way too broad.

I did read the draft, and I do understand it is targeted at IXPs; the skeptical part in me is suggesting that the max-prefixes limit will limit the efficiency of this method. In order for this to be efficient to mitigate a DDOS attack, it would require the prefix limit for the very generic BLACKHOLE community to be in the tens of thousands. I just don't see this happening in the real world. A BGP community with global significance will face significant challenges. I don't see operators trusting this community.

> Extreme caution should be used when purposefully propagating IP prefixes
> tagged with the BLACKHOLE BGP community outside the local routing domain.

This is the part that I find out-of-touch with reality. Extreme caution should be used not to announce RFC1918 prefixes, and not to announce the entire Internet routing table. It happens all the time. 

This draft creates a DDOS vector of its own : an attacker with good BGP feeds to their upstreams could use the well-known community to craft a new DDOS attack by injecting the target prefixe(s). Unlike the NO_ADVERTISE or NO_EXPORT communities, this is a global DDOS bait.

As the operator of a large BGP Blackhole feed (1), the first requests that came out of the beta-testers were asking for more granularity. My BGP blackhole feed is over 100K prefixes; it works for me and my beta-tester buddies, but it won't work for everyone.

I oppose this draft on the grounds that it creates more opportunities for DDOS attacks than it solves.

Michel.

(1) http://arneill-py.sacramento.ca.us/cbbc/

---

From: Christopher Morrow <[email-masked]>
Date: Tue, 21 Jun 2016 12:33:49 -0400

On Mon, Jun 20, 2016 at 11:39 PM, Michel Py <
[email-masked]> wrote:

> I have to point out that, although it looks like a move in the right
> direction, the newly created BLACKHOLE community is likely to meet severe
> max-prefixes resistance. It is way too broad.
>

'​Max prefix pressure'​ care to elaborate a bit more on this? I seems, to
me, that the community is not really 'max prefix' relevant, as much as 'if
the IXP (or whomever) sends more prefixes, I should be aware and take
appropriate actions.

​'way too broad'​ how so?

I did read the draft, and I do understand it is targeted at IXPs; the
> skeptical part in me is suggesting that the max-prefixes limit will limit
> the efficiency of this method. In order for this to be efficient to
> mitigate a DDOS attack, it would require the prefix limit for the very
> generic BLACKHOLE community to be in the tens of thousands. I just don't
> see this happening in the real world. A BGP community with global
> significance will face significant challenges. I don't see operators
> trusting this community.
>

​perhaps not all operators, but perhaps some will, right? not everyone
trusts/abides-by 'no-export' either.​ This is really about 'what value
should we all use?" not about accepting the routes, or really even what to
do if you see this community on a route? You could just QoS the particular
packets in the community tagged prefix not drop traffic to it...

> This is the part that I find out-of-touch with reality. Extreme caution
> should be used not to announce RFC1918 prefixes, and not to announce the
> entire Internet routing table. It happens all the time.
>
> This draft creates a DDOS vector of its own : an attacker with good BGP
> feeds to their upstreams could use the well-known community to craft a new
> DDOS attack by injecting the target prefixe(s). Unlike the NO_ADVERTISE or
> NO_EXPORT communities, this is a global DDOS bait.
>
> As the operator of a large BGP Blackhole feed (1), the first requests that
> came out of the beta-testers were asking for more granularity. My BGP
> blackhole feed is over 100K prefixes; it works for me and my beta-tester
> buddies, but it won't work for everyone.
>
>
​ok, but not everyone has to peer with your server, and not everyone may
agree that they want to listen-to/use such a new community in the wider
network, right?

There are many filter knobs to turn in BGP peering, this particular
document just talks about agreeing on 'what communty value should we all
use here?' instead of: "hey Job uses X, I use Y, Sam uses Z... now how do I
sort out my policy such that I send the right commnuities to the right
peer??"​

I oppose this draft on the grounds that it creates more opportunities for
> DDOS attacks than it solves.
>
>
​you'll have to provide more detail I think... at least for me to
understand how this is worse than what exists today... Essentially an IXP
would have to choose to support ​this community and all peers would have to
agree to accept/use the prefixes so marked.

---

From: Michel Py <[email-masked]>
Date: Wed, 22 Jun 2016 00:24:19 +0000

> Christopher Morrow wrote :
> 'Max prefix pressure' care to elaborate a bit more on this?

Let's look at a real-world example : Team Cymru's UTRS.
https://www.cymru.com/jtk/misc/utrs.html
> A participant is only permitted to have up to 25 active route announcements
> through UTRS at a time. Additional routes will be rejected.

I foresee that IXPs and other organizations who would adopt the BLACKHOLE community would put limits just the same as UTRS does. At 25 routes per participant, the mitigation of a DDOS potentially coming from thousands of IP addresses is limited. I totally understand the reasons to permit only 25 (or n) blackholes routes. 

> 'way too broad' how so?

Because it's "only" one global community.

> There are many filter knobs to turn in BGP peering

That is the point : there are no knobs to turn with only one community; IMHO, it would be better to have ASN:666 (with ASN being the source AS for the blackhole prefix _or_ the AS of the IXP) and even better to standardize more granularity about the reason for being blackholed.
AS:666 : blackhole
AS:6601 : blackhole because of spam
AS:6602 : blackhole because of ssh brute-force attack
.....

> ok, but not everyone has to peer with your server, and not everyone may agree that
> they want to listen-to/use such a new community in the wider network, right?

Correct, but I realized recently that the only way to have some adoption was to provide more knobs. If you don't block enough prefixes, the blackhole mechanism is not efficient. If you block enough prefixes, people are scared of using it.

​
> at least for me to understand how this is worse than what exists today...

Because some people out there are in the DDOSAAS business; with the standardization of the BLACKHOLE community, some script kiddo could write something that probes to see if it actually works; maybe I'm full of BAAS but there are is a lot of creativity being put into DDOS; if this was to become standard, it could become a new attack vector.

See it the same way as logging in as root : as soon as you have the telnet or ssh port open to the outside, you will have attempts to log in as root. The same reasoning applies to a global blackhole community : compromised systems would to inject a blackhole prefix to see if it works and report to their C&C.

Your idea is good, but it's too broad, too generic.

Michel.

http://arneill-py.sacramento.ca.us/cbbc/

---

From: John Kristoff <[email-masked]>
Date: Tue, 21 Jun 2016 20:33:57 -0500

On Wed, 22 Jun 2016 00:24:19 +0000

> https://www.cymru.com/jtk/misc/utrs.html

Hello.  :-)  For those that hadn't heard, I've recently left Team
Cymru just a few weeks ago, but can obviously speak to that project
that is still running as I did briefly at the security track at NANOG
67.

> I foresee that IXPs and other organizations who would adopt the
> BLACKHOLE community would put limits just the same as UTRS does. At
> 25 routes per participant, the mitigation of a DDOS potentially
> coming from thousands of IP addresses is limited. I totally
> understand the reasons to permit only 25 (or n) blackholes routes. 

That was admittedly an arbitrary limit.  One potential peer had said
they would need the max to be at least a few hundred.  This person was
from a DDoS mitigation company unsurprisingly.

> That is the point : there are no knobs to turn with only one
> community; IMHO, it would be better to have ASN:666 (with ASN being
> the source AS for the blackhole prefix _or_ the AS of the IXP) and
> even better to standardize more granularity about the reason for
> being blackholed. AS:666 : blackhole AS:6601 : blackhole because of
> spam AS:6602 : blackhole because of ssh brute-force attack .....

UTRS had started with community tag ending in :666, but then at the
suggestion of discussion on the list, we opted to set the default to
local_asn:0 where local_asn is the asn of the announcing network and
the 0 being the default value, but with potential future use of it to
signify a remote peer with which action is directed.  I don't think it
would it was likely I would have gotten around to implementing anything
like that soon, but it at least prepared us for that potential
flexibility.

John

---

From: Michel Py <[email-masked]>
Date: Wed, 22 Jun 2016 03:29:42 +0000

John,

> John Kristoff wrote :
> For those that hadn't heard, I've recently left Team Cymru just a few weeks ago

I hadn't, you will be missed. The community as a whole owes Team Cymru, thanks for being part of it.

> One potential peer had said they would need the max to be at least a few hundred.  This person was from a DDoS mitigation company unsurprisingly.

At least indeed, unsurprisingly. I surprised myself with the CBBC, when I started a few months ago, blackholing 20K prefixes seemed like it would break something, then it went to 50 K and then 100K and then 135K and it still did not break anything that the beta-testers commented on.

> we opted to set the default to local_asn:0 where local_asn is the asn of the announcing network and the 0 being the default value

Maybe we should transfer ASN 666 to blackhole use :P

> but it at least prepared us for that potential flexibility.

Ack that.

Michel.

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 23 Jun 2016 12:29:48 -0400

On Tue, Jun 21, 2016 at 8:24 PM, Michel Py <
[email-masked]> wrote:

> I foresee that IXPs and other organizations who would adopt the BLACKHOLE
> community would put limits just the same as UTRS does. At 25 routes per
> participant, the mitigation of a DDOS potentially coming from thousands of
> IP addresses is limited. I totally understand the reasons to permit only 25
> (or n) blackholes routes.
>
>
​this implies that src-route blackholing (discard route + uRPF or similar)
is required or targeted for this ​use-case, which I don't think is a
given.  Surely, if you want to do that you'd have to accept very, very
large prefix sets from your peer(s).

this concern doesn't seem to be a blocker for this draft though...

> Because it's "only" one global community.
​ok, that doesn't explain it for me though.​

> That is the point : there are no knobs to turn with only one community;
> IMHO, it would be better to have ASN:666 (with ASN being the source AS for
> the blackhole prefix _or_ the AS of the IXP) and even better to standardize
> more granularity about the reason for being blackholed.
> AS:666 : blackhole
​the state today is that three are 'many different communities' which is
painful for an operator to manage. It means custom policy for each peer,
which is going to (has many times already) bite someone one.​

> Correct, but I realized recently that the only way to have some adoption
> was to provide more knobs. If you don't block enough prefixes, the
> blackhole mechanism is not efficient. If you block enough prefixes, people
> are scared of using it.
>
>
​that seems contradictory... 'if not enough, bad, if enough, bad'​

> Because some people out there are in the DDOSAAS business; with the
> standardization of the BLACKHOLE community, some script kiddo could write
> something that probes to see if it actually works; maybe I'm full of BAAS
> but there are is a lot of creativity being put into DDOS; if this was to
> become standard, it could become a new attack vector.
>
>
​could, could, could... err, nothing definite here that you've pointed out.
I think the field of DoS (attacking) is pretty simple, and has been for a
very long while. there's no real creativity here. I also don't imagine that
this community gets used for the vast majority of dos attacks, but only for
those where the potential harm to the IX or the direct participants is a
large.​

> See it the same way as logging in as root : as soon as you have the telnet
> or ssh port open to the outside, you will have attempts to log in as root.
> The same reasoning applies to a global blackhole community : compromised
> systems would to inject a blackhole prefix to see if it works and report to
> their C&C.
>
>
​'compromised systmes' meaning a router at the IX? if somoene compromises a
router on the IX fabric we've all got much larger problems than 'someone
could blackhole something with a community'.​

---

From: Michel Py <[email-masked]>
Date: Sat, 25 Jun 2016 03:10:31 +0000

>> Michel Py wrote : 
>> I foresee that IXPs and other organizations who would adopt the BLACKHOLE community would put limits just the
>> same as UTRS does. At 25 routes per participant, the mitigation of a DDOS potentially coming from thousands
>> of IP addresses is limited. I totally understand the reasons to permit only 25 (or n) blackholes routes.

>​ this implies that src-route blackholing (discard route + uRPF or similar) is required or targeted for this ​use-case, which
> I don't think is a given.  Surely, if you want to do that you'd have to accept very, very large prefix sets from your peer(s).

The number of prefixes and the use of uRPF are orthogonal. BCP38 would be nice, OTOH I understand why it's not widely deployed.

> this concern doesn't seem to be a blocker for this draft though...

That's the difference between a draft that will become a deployed standard and one that will eventually be deprecated. Acceptance / adoption in the real world.

​
> the state today is that three are 'many different communities' which is painful for an operator to manage.
> It means custom policy for each peer, which is going to (has many times already) bite someone one.​

It means custom policy for each peer-group, which is what operators do and want.

> ​'compromised systmes' meaning a router at the IX? if somoene compromises a router on the IX fabric
> we've all got much larger problems than 'someone could blackhole something with a community'.​

You should see what "IX fabric" means, in many places. I rest my case.

Michel.

---

From: Christopher Morrow <[email-masked]>
Date: Sat, 25 Jun 2016 21:52:52 -0400

On Fri, Jun 24, 2016 at 11:10 PM, Michel Py <
[email-masked]> wrote:

> BLACKHOLE community would put limits just the
> >​ this implies that src-route blackholing (discard route + uRPF or
> similar) is required or targeted for this ​use-case, which
> > I don't think is a given.  Surely, if you want to do that you'd have to
> accept very, very large prefix sets from your peer(s).
>
> The number of prefixes and the use of uRPF are orthogonal. BCP38 would be
> nice, OTOH I understand why it's not widely deployed.
>
>
​sure, it's the particular holistic goal represented...  One usecase for
'send routes which you prefer to see blackholed remotely' is:
  "I am seeing 100gbps of traffic to this /32, please stop filling up my
interconnect"

Another is:
  "All of these things are bad, I would like you to filter them before they
appear on my network"

I don't think the use-case envisioned for the community is the second one.

Further there's no reason that a route-server operator needs to REQUIRE the
use of any particular community. They could, very well, choose to say:
"Yes, I know there's that well known community, but we like 0:2525 so we're
just going to keep on using that, thanks!"

That seems fine, to me. I'm sure the RS participants in question can either
follow along or pressure the RS operator to use the 'well known' option.
(or both!)

> That's the difference between a draft that will become a deployed standard
> and one that will eventually be deprecated. Acceptance / adoption in the
> real world.
>
>
​not everyone has to use it? so... not really a blocker.
not everyone uses (or adheres to the wishes of) NO-EXPORT either.​

​
> It means custom policy for each peer-group, which is what operators do and
​toma-toe, tomA-toe ... (group/peer/etc)​

Here's the thing, speaking as an operator of a not-small network... I don't
really want to manage 1800 different outbound policies and their associated
knobs. I really do want to manage a minimal number, where 'minimal number'
is ~3.

Having to remember / configure each peer which provides 'blackhole
community' features has a different community to send is bonkers. it's also
prone to errors, oversharing and all sorts of other problems...

> You should see what "IX fabric" means, in many places. I rest my case.
​I've seen quite a few, thanks!
-chris​

---

From: Nick Hilliard <[email-masked]>
Date: Sun, 26 Jun 2016 14:38:33 +0100

The IESG wrote:
> The IESG has received a request from the Global Routing Operations WG
> (grow) to consider the following document:
> - 'BLACKHOLE BGP Community for Blackholing'
>   <draft-ietf-grow-blackholing-00.txt> as Proposed Standard
> 
> The IESG plans to make a decision in the next few weeks, and solicits
> final comments on this action. Please send substantive comments to the
> [email-masked] mailing lists by 2016-07-04. Exceptionally, comments may be
> sent to [email-masked] instead. In either case, please retain the
> beginning of the Subject line to allow automated sorting.

There has been no discussion on the GROW mailing list about having this
document published as Standards Track rather than informational and it's
coming as a surprise to see that this was only announced at IESG Last
Call a couple of days ago.  At the least, there ought to be some
discussion about this before pushing it up into the publication queue.

Personally - and I say this as an IXP operator who has had yet another
week-end ruined due to prolonged DDoS problems on an IXP fabric - I
don't think this is an appropriate document for standards track, or even
for publication as an RFC.  The reason for this is that section 3.4
creates the expectation that IXPs could or should be involved in
facilitating blackholing of IP addresses.

The problem is layer 9: if a mechanism of this form is standardised, it
will be viewed by governments, courts and law-enforcement a centralised
big red button which can be pressed at will to block IP access to their
bêtes-noires du jour.  And it turns out that there are lots of things
that governments, courts and LEAs don't like, ranging from file sharing
to witchcraft (one of the default blocking categories in the UK) to
youtube (lots of countries) to google (france), to whatever. It's not
just DDoS that will be targeted here.

The proposal itself has raised an unusual level of disquiet among the
IXP community, which seems to be split down the middle about whether
standardising blackhole communities in an RFC is a good idea or not.
Some IXPs think it's great.  Others think it's a terrible idea.  For
sure, there is no consensus about this in the IXP world.

Nick

---

From: Job Snijders <[email-masked]>
Date: Sun, 26 Jun 2016 16:01:30 +0200

Dear Nick,

On Sun, Jun 26, 2016 at 02:38:33PM +0100, Nick Hilliard wrote:
> <snip>

> I don't think this is an appropriate document for standards track, or
> even for publication as an RFC.
> The reason for this is that section 3.4 creates the expectation that
> IXPs could or should be involved in facilitating blackholing of IP
> addresses.

You are correct, IXPs _could_ be involved in facilitating blackholing of
traffic. This draft does not change today's reality.

Follow-up question: without section 3.4 - would you still object?

> The problem is layer 9: if a mechanism of this form is standardised,
> it will be viewed by governments, courts and law-enforcement a
> centralised big red button which can be pressed at will to block IP
> access to their bêtes-noires du jour.  And it turns out that there are
> lots of things that governments, courts and LEAs don't like, ranging
> from file sharing to witchcraft (one of the default blocking
> categories in the UK) to youtube (lots of countries) to google
> (france), to whatever. It's not just DDoS that will be targeted here.

This is a stifling argument. A blackhole mechanism is already today
standardized as RFC 5635. This draft merely exists to standardize a
codepoint. 

> sure, there is no consensus about this in the IXP world.

There is also the carrier community to consider. Currently a ton of
carriers have implemented their own codepoint to trigger blackholing, a
very incomplete overview follows of communities currently in use by
carriers for the purpose of blackholing: 2914:666, 209:0, 701:9999,
1239:66, 1273:666, 1299:999, 1759:666, 2828:1650, 3212:9999, 3257:2666,
65000:0, 3327:666, 3356:9999, 6762:666, 3561:666, 4323:187, 5580:666,
6461:5990, 7029:4506, 7922:666, 8100:6666, 8218:9999, 8220:63999,
8708:9999, 49544:666, 11537:911, 15756:666, 19401:911, 23265:666 and
286:66.

It is long overdue to standardize all these towards 65535:666 (BLACKHOLE).

Kind regards,

Job

---

From: Nick Hilliard <[email-masked]>
Date: Sun, 26 Jun 2016 15:23:53 +0100

> Follow-up question: without section 3.4 - would you still object?

I don't think that IXPs should be mentioned anywhere in this document.
For the general case of blackholing, an IXP is a clearing house so
should not get involved in the business of dropping its participants'
traffic.  In the case of route servers, blackholing turns the IXP into a
legal target.

Nick

---

From: Job Snijders <[email-masked]>
Date: Sun, 26 Jun 2016 16:43:14 +0200

On Sun, Jun 26, 2016 at 03:23:53PM +0100, Nick Hilliard wrote:
> traffic. In the case of route servers, blackholing turns the IXP into
> a legal target.

I feel that this is not the appropiate forum to define what IXPs can,
can't, should and shouldn't in context of legal enforcement agencies.

Kind regards,

Job

---

From: Nick Hilliard <[email-masked]>
Date: Sun, 26 Jun 2016 17:04:22 +0100

> can't, should and shouldn't in context of legal enforcement agencies.

I wasn't suggesting it was.  What I said was two things:

1. regarding everything except section 3.4: if two organisations decide
to share blackholing information over bgp, that is orthogonal to the
connectivity medium that the two organisations use to connect to each
other.  If there's an IXP in the middle, this is no different to having
a direct point-to-point ethernet connection, or even tin cans and some
string.  From this point of view, it's completely irrelevant to mention
IXPs in the text.

2. relating to section 3.4: there is no consensus in the IXP world that
using route servers to exchange blackhole information is a good idea,
because it offers the opportunity to turn the IXP into a target for
implementing wide-scale blackholing.  This is something that a lot of
IXPs are anxious to avoid, and having an RFC floating around which
promotes this as a viable idea is viewed by many IXPs as something which
would be unhelpful.

Nick

---

From: joel jaeggli <[email-masked]>
Date: Sun, 26 Jun 2016 09:31:17 -0700

> can't, should and shouldn't in context of legal enforcement agencies.

It's not clear to me how that would even work. assuming for the sake of
arguement that the IXP by way of configured policy on the route-server
adds this community to a prefix.

If the route is withdrawn by the participant from the mlpe the
black-hole goes away...

Since the ixp doesn't control the prefix announcement the addition of
transitive attributes in a fashion counter to the wishes of the parties
involved seems self-defeating.

If the ixp mlpe is engaged in prefix hijacking (and that includes more
specifcs for announced blocks), well... that's not an exchange we deal
with anymore.

---

From: John Kristoff <[email-masked]>
Date: Sun, 26 Jun 2016 12:06:01 -0500

On Sun, 26 Jun 2016 16:31:17 +0000
joel jaeggli <[email-masked]> wrote:

> It's not clear to me how that would even work. assuming for the sake
> of arguement that the IXP by way of configured policy on the
> route-server adds this community to a prefix.

Here is some detail on how DE-CIX implements it:

  <https://www.de-cix.net/products-services/de-cix-frankfurt/blackholing/>

John

---

From: Michel Py <[email-masked]>
Date: Sun, 26 Jun 2016 17:19:47 +0000

> Nick Hilliard wrote :
> Personally - and I say this as an IXP operator who has had yet another week-end ruined due to prolonged DDoS problems on an IXP
> fabric - I don't think this is an appropriate document for standards track, or even for publication as an RFC. The reason for this
> is that section 3.4 creates the expectation that IXPs could or should be involved in facilitating blackholing of IP addresses.

I'm with you here. I have never been an IXP operator myself, but I think it's a bad idea that IXPs themselves get involved in the blackholing business. I habe been very specific in the CBBC FAQ (1) that it was not something I thought IXPs should use.

> It's not just DDoS that will be targeted here.

Correct. Although some blackholing by _participants_ in an IXP could be useful, for the layer 9 issues you mentioned earlier it makes total sense that the IXP itself would not be involved. Maybe a route-server operated by a completely independent third party.

> Job Snijders wrote :
> This is a stifling argument. A blackhole mechanism is already today standardized as RFC 5635. This draft merely exists to standardize a codepoint.

I have to disagree. This drafts does more than to standardize a codepoint, it creates a community with global significance. If the draft was about standardizing the :666 part of it, I could support it (with the removal of 3.4).

As of RFC5635, I thought it was a step in the right direction. A draft that reserves 1 digit in the community for BCP38 control (including a "don't care" option) and some other of the remaining digits in the community for some other knobs regarding the reason for blackholing would be the way to go, IMHA.

Michel.

---

From: joel jaeggli <[email-masked]>
Date: Sun, 26 Jun 2016 20:54:21 -0700

>   <https://www.de-cix.net/products-services/de-cix-frankfurt/blackholing/>

At the the possible expense of belaboring my observation still further,
i'm aware of how the community is implemented, I'm on those fabrics.
What I wasn't and am not clear on is how that would lead to:

Nick

>>  In the case of route servers, blackholing turns the IXP into
>>  a legal target.

Job

> can't, should and shouldn't in context of legal enforcement agencies.

Short of the IXP engaging in prefix hijacking, or unilaterally applying
the community to an existing prefix; the ixp is in not position to
black-hole traffic except on request of the sender of the desitnation
prefix. Likewise if you withdraw the prefix from the routeserver, the
blackhole goes away, unless the route-server is engaged in prefix hijacking.

I don't see either of those issues as serious threats. if you live under
a regime that considers prefix hijacking acceptable, the community adds
no capability that the exchange does not already have;they can afterall
change the nexthop today, announce whatever prefix you're willing to
accept and so on; any of those activities in most places would be
immediate grounds for depeering and departure.

---

From: joel jaeggli <[email-masked]>
Date: Sun, 26 Jun 2016 21:47:19 -0700

> discussion about this before pushing it up into the publication queue.

The looking back at the document that was WGLCed (which is the current
version) the state was standards track. the individual sumbission draft
draft-ymbk-grow-blackholing-00 was flagged as such since initial
submission.  It's status therefore I think is well understood. Whether
that's appropriate or not is another question.

Afaik well know assignments (and there are precious few really) have
been generally regarded as standards action thought the ranges is split
like so.

Range 	Registration Procedures
0xFFFF0000-0xFFFF8000	First Come First Served
0xFFFF8001-0xFFFFFFFF	Standards Action

---

From: Christopher Morrow <[email-masked]>
Date: Mon, 27 Jun 2016 09:50:42 -0400

> that's appropriate or not is another question.

https://datatracker.ietf.org/doc/draft-ietf-grow-blackholing/

​has it marked as PS... which I too thought was correct.​

​-chris​

---

From: joel jaeggli <[email-masked]>
Date: Mon, 27 Jun 2016 07:13:41 -0700

On 6/27/16 6:50 AM, Christopher Morrow wrote:
> 
> 
> On Mon, Jun 27, 2016 at 12:47 AM, joel jaeggli <[email-masked]
> <mailto:[email-masked]>> wrote:
> 

ps is the first step in standards track.

---

From: Christopher Morrow <[email-masked]>
Date: Tue, 28 Jun 2016 11:11:30 -0400

On Sun, Jun 26, 2016 at 11:54 PM, joel jaeggli <[email-masked]> wrote:

> >   <
> https://www.de-cix.net/products-services/de-cix-frankfurt/blackholing/>
>
>
​Perhaps Nick is reacting to language like:
"​
 This well-known advisory transitive BGP
   community, namely BLACKHOLE, allows an origin AS to specify that a
   neighboring IP network or IXP should blackhole a specific IP prefix.
​"​

​which could be cleaned up a bit like:
"​This well-known advisory transitive BGP
   community, namely BLACKHOLE, allows an origin AS to specify that a
   neighboring IP network or IXP PARTICIPANT should blackhole a
   specific IP prefix."

This transform doesn't work through out the document though.

---

From: Gert Doering <[email-masked]>
Date: Tue, 28 Jun 2016 17:17:25 +0200

Hi,

On Tue, Jun 28, 2016 at 11:11:30AM -0400, Christopher Morrow wrote:
> ???Perhaps Nick is reacting to language like:
> "???
> ???"???
> ???which could be cleaned up a bit like:
> "???This well-known advisory transitive BGP

Well, the intention *is* that "if the IXP supports black-holing, please do".

In implementations like DECIX', the neighbouring IXP participant does not
have to do anything in particular, except "accept the prefix with the
black-hole nexthop".

Maybe more along the lines of

  This well-known advisory transitive BGP
    community, namely BLACKHOLE, allows an origin AS to specify that a
    neighboring IP network or IXP that has appropriate mechanisms in place
    is requested to blackhole a specific IP prefix.

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444           USt-IdNr.: DE813185279

---

From: Christopher Morrow <[email-masked]>
Date: Tue, 28 Jun 2016 11:27:52 -0400

> >    neighboring IP network or IXP should blackhole a specific IP prefix.
​​<sorry about these wonky characters>​

> Well, the intention *is* that "if the IXP supports black-holing, please
​yup.​

>     is requested to blackhole a specific IP prefix.
​that seems reasonable, though I don't think it necessarily addresses
Nick's issue that: "YIKES! please don't mention that IXP's can/will do this
sort of function"

I think even in the cases where the IXP may add the right community and
reset nexthop, the participants have to actually accept these prefixes and
agree with the IXP about the nexthop, and throw away the traffic destined
to the prefix(es).

You could make an ixp have a next-hop internal to the fabric which was the
discard, but I dont' think that solves the larger problem of: "oh, our ixp
fabric is full of packets :(" since the participants would still be putting
packets  onto the fabric.

maybe that's not the problem which is trying to be solved though (fabric
full, instead of 'participant interface is full')​

---

From: Job Snijders <[email-masked]>
Date: Tue, 28 Jun 2016 17:36:56 +0200

Dear all,

On Tue, Jun 28, 2016 at 11:27:52AM -0400, Christopher Morrow wrote:
> that seems reasonable, though I don't think it necessarily addresses
> Nick's issue that: "YIKES! please don't mention that IXP's can/will do
> this sort of function"

I still question the validity of Nick's concern. Since when do LEA read
RFCs/drafts like draft-ietf-grow-blackholing and then conclude "oh, this
RFC says that blackholing at IXPs can't be done, alrighty then, we'll
show ourselves out!".

LEAs will request what LEAs will request, regardless of technical
feasibility or IETF's acknowledgement/denial a facilitating mechanism
exists.

> I think even in the cases where the IXP may add the right community
> and reset nexthop, the participants have to actually accept these
> prefixes and agree with the IXP about the nexthop, and throw away the
> traffic destined to the prefix(es).
> 
> You could make an ixp have a next-hop internal to the fabric which was
> the discard, but I dont' think that solves the larger problem of: "oh,
> our ixp fabric is full of packets :(" since the participants would
> still be putting packets  onto the fabric.
> 
> maybe that's not the problem which is trying to be solved though
> (fabric full, instead of 'participant interface is full')

Some IXPs can do actual blackholing inside their fabric, a mechanism
which does _not_ require any of the IXP participants to participate or
adjust their local routing policy to honor the BLACKHOLE community. I've
described such a non-cooperative mechanism on the NANOG mailing-list and
I know of one IXP which has implemented this. (This is different from
DE-CIX's current implementation.)

Already today, the reality is that some IXPs can and will blackhole
traffic at the request of a participant, and some IXPs can't (vendor
limitations) or won't (miscellaneous concerns) blackhole traffic. This
draft does not change any of that.

Kind regards,

Job

---

From: Job Snijders <[email-masked]>
Date: Tue, 28 Jun 2016 17:36:56 +0200

Dear all,

Kind regards,

Job

---

From: "Thomas Mangin" <[email-masked]>
Date: Tue, 28 Jun 2016 17:09:20 +0100

> feasibility or IETF's acknowledgement/denial a facilitating mechanism

Job,

Not putting any particular IXP’s hat on, but as an individual I 
_strongly_ agree with Nick. I see no reason to mention IX in the 
document.

The IX community is trying to make sure we do not become the filtering 
sink of the internet (it would not do anyone any good - except perhaps 
transit providers :wink: :wink:)

A long time ago, working for another employer, my team wrote a software 
called ’CleanFeed’ (it was trademarked - a trademark now owned by 
BT) and then demo’ed to BT, which code named a similar project with 
the same name. Today, the whole UK industry has to contend with ‘ISP 
level filtering’.

You do NOT want to open any pandora’s box.

Sincerely,

Thomas

---

From: Job Snijders <[email-masked]>
Date: Tue, 28 Jun 2016 19:07:37 +0200

Dear all,

On Tue, Jun 28, 2016 at 05:09:20PM +0100, Thomas Mangin wrote:
> > > that seems reasonable, though I don't think it necessarily
> > > addresses Nick's issue that: "YIKES! please don't mention that
> > > IXP's can/will do this sort of function"

Alright, I concede.

I'll work with the authors on a version that generalizes the wording to
be non-IXP/ISP specific, e.g. removal of section 3.4 and some
wordsmithing on other sentences.

Kind regards,

Job

---

From: joel jaeggli <[email-masked]>
Date: Tue, 28 Jun 2016 12:48:26 -0700

> DE-CIX's current implementation.)

sure l3 acls can be applied to l2 ports.

most ixps are going to have a set of filters that prevent certain kinda
of activity, e.g. spanning tree PDUs, router-advertisement, proxy-arp
and so  on. these are all within the technical capabilties of most
high-end-ethernet switch platforms.

> draft does not change any of that.

agree

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 28 Jun 2016 22:29:55 +0100

> high-end-ethernet switch platforms.

this is a vast overstatement of ingress port filtering capabilities.
STP BPDUs are ignored because disabling STP on a port will cause the
packet to be silently dropped.  Most hardware which uses tcam for acls
will choke on even ostensibly simple configs, e.g. uniquely keyed L2 +
ipv4/ipv6 + L4 port filters applied on all ports.  Inspecting deep into
packets is rarely easy.  Overall the entire area is fraught with
limitations and corner cases.

Nick

---

From: Job Snijders <[email-masked]>
Date: Wed, 29 Jun 2016 00:54:37 +0200

> Alright, I concede.

We've worked all night and the above work has been completed. :)

https://www.ietf.org/rfcdiff?url2=draft-ietf-grow-blackholing-01

Executive summary of changes between -00 and -01:

    - "IP networks and IXPs" references have been collapsed into "networks"
    - Section 3.4 "IXPs: Peering at Route Servers" removed
    - Author details brought up to date 
    - Clarify paragraph about exhaustion
    - Make the introduction less colloquial

Note that although the wording in the "IANA Considerations" section was
updated, the action requested from IANA has not changed. 

I believe this update addresses the concerns raised in this phase of the
document.

Kind regards,

Job

---

From: Michel Py <[email-masked]>
Date: Wed, 29 Jun 2016 03:03:34 +0000

> https://www.ietf.org/rfcdiff?url2=draft-ietf-grow-blackholing-01

Great job. I no longer oppose this text.

I have a suggestion : make it somehow more clear than the purpose of this draft is to blackhole the _targets_ of DDoS attacks, not the sources. It is not obvious when one uses BGP blackhole feeds to block the sources of attacks.

Thanks,
Michel.

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 29 Jun 2016 13:30:49 +0100

> I believe this update addresses the concerns raised in this phase of the

yes, thanks, it addresses these concerns, and the document is a lot
better as a result.

The second major area of concern I have about this proposal is the
transitive nature of the bgp community.  The issue is that the draft
specifies a mechanism to cause traffic to be dropped on the floor, that
the signaling mechanism is globally transitive in scope, and the
specific intent is that prefixes tagged in this way are exported to
other ASNs. In other words, the draft specifies behaviour that is risky
by default.

Prefix hijacking rates suggest that adding a new compromise vector is
something that should be considered carefully in the context of
standardisation.

The obvious way to work around this would be to specify a non-transitive
community, but that would defeat the purpose of the draft.  Other
options might include a requirement that BLACKHOLE should automatically
be marked as NO-EXPORT when received by a third party ASN. This doesn't
deal with backwards compatibility, though, and given the lifetime of bgp
implementations, there would be a large time window opened where this
may be a problem.

Nick

---

From: Job Snijders <[email-masked]>
Date: Wed, 29 Jun 2016 14:54:49 +0200

On Wed, Jun 29, 2016 at 01:30:49PM +0100, Nick Hilliard wrote:
> transitive nature of the bgp community.

I thought Section 3.2 provides enough detail on scoping routes tagged
with BLACKHOLE, however with your concern and the following in mind:

This draft intends to unify the wild variety of blackholing communities
which exist in the wild by providing a single Well-known codepoint, see
bottom of [1] for an incomplete list.

I do not think it is feasible to mandate new behaviour (like
automatically adding a NO_EXPORT community) in this draft. It is up to
the network operator to decide when and where to honor a blackhole
community and when not to honor it.

Automatically adding NO_EXPORT (or any of the other scope limiting
Well-known communities) is likely to lead to implementation difficulty
for operators when prefixes with a blackhole community attached need to
be redistributed to some sort of sibling network.

So, with your concern and the above in mind:

Should it be somehow clarified that router vendors are not supposed to
implement mechanisms, which are by default enabled, that discard traffic
for BLACKHOLE'ed prefixes?

Just like the deployment of honoring the NOPEER [rfc3765] community is
100% driven by the operator, out of the box the vendor implementations
do nothing with the community except transit it. 

Kind regards,

Job

[1]: https://www.ietf.org/mail-archive/web/ietf/current/msg98740.html

---

From: Gert Doering <[email-masked]>
Date: Tue, 28 Jun 2016 17:32:29 +0200

Hi,

> fabric is full of packets :(" since the participants would still be putting

This is what DECIX is doing: a special next-hop IP which ARPs to a special
MAC address which is MAC-ACL-filtered at all ingress ports to the fabric -
so the nice effect is "fabric not full" (plus "participant interface not
full").

Gert Doering
        -- NetMaster

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 29 Jun 2016 21:46:15 +0100

> for BLACKHOLE'ed prefixes?

I would have said the opposite, i.e. that any traffic tagged with this
prefix is dropped via e.g. null0 or martian mechanisms / etc.  But it
definitely needs to be defined because at the moment it's ambiguous.
Ambiguity is fine when it's your own network, but not fine when you're
defining something with global scope.

Also, as Michael Py mentioned, it's not clear whether this refers to
source based blackholing or destination based blackholing.

Nick

---

From: Job Snijders <[email-masked]>
Date: Wed, 29 Jun 2016 22:54:30 +0200

> defining something with global scope.

Why would you say the opposite? That goes counter to what the vendors
are shipping today. The suggestion "do not do anything" is compatible
with what ships today! :)

We can add a new section "3.4 - Vendor recommendations" and describe
what it is we'd expect a network device vendor to implement or not to
implement. 

> source based blackholing or destination based blackholing.

The word 'source' does not appear in the draft. In my reading of section
3.1 it is obvious destination based blackholing, but I welcome a
suggestion to reword a sentence in the introduction to include the
phrase 'destination based blackholing'.

Do you have any more comments or concerns queued up?

Kind regards,

Job

---

From: joel jaeggli <[email-masked]>
Date: Wed, 29 Jun 2016 13:59:42 -0700

> source based blackholing or destination based blackholing.

It should be an inherent property that what is being blackholed is
traffic bound for the prefix that the community is attached to is it not?

Source based RTBH requires some  explicit coordination between the
parties using it.

> Nick
> 
> _______________________________________________
> GROW mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/grow
>

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 29 Jun 2016 22:10:44 +0100

> Do you have any more comments or concerns queued up?

I don't think the draft is well specified in terms of its intended
semantics.  This is a problem with a standards track document,
particularly one with big scary warnings in the security considerations
section.  It needs to be tightened up substantially before publication
could be considered.

Nick

---

From: Jared Mauch <[email-masked]>
Date: Wed, 29 Jun 2016 17:22:44 -0400

> On Jun 29, 2016, at 5:10 PM, Nick Hilliard <[email-masked]> wrote:
> 
> Job Snijders wrote:

Looking at section 5 of https://www.rfc-editor.org/rfc/rfc5635.txt

I think provides some guidance of helpful text to improve this section. While
some hardware may not support certain capabilities, limiting a specification
to an exact generation of operator hardware would be shortsighted.

I suspect one could re-use much of the rfc5635 text without much if any
additional commentary.

- Jared

---

From: David Farmer <[email-masked]>
Date: Wed, 29 Jun 2016 16:23:23 -0500

> 0xFFFF8001-0xFFFFFFFF   Standards Action

And given that the code point that is being defined is 0xFFFF029A and that
falls in the First Come First Served range, technically standards action
isn't required.  Furthermore, this draft is really dependant on RFC3882 and
RFC5635, both Informational Status, for a full description of the intended
functionality.   If this draft is Standards Track, some may mistakenly
think there is some kind of preference for this mechanism for triggering
Blackholes, vs. the other mechanisms discussed in RFC3882 and RFC5635.  I
fully support this defining this Well-Known BGP Community, however I
believe it should be of equal stature as the other mechanisms discussed in
RFC3882 and RFC5635.

Therefore, for the above reasons, I think the more appropriate status for
this draft is Informational.

Thanks.

-- 
===============================================
David Farmer               Email:[email-masked]
Networking & Telecommunication Services
Office of Information Technology
University of Minnesota
2218 University Ave SE        Phone: 612-626-0815
Minneapolis, MN 55414-3029   Cell: 612-812-9952
===============================================

---

From: Job Snijders <[email-masked]>
Date: Wed, 29 Jun 2016 23:52:56 +0200

Hi David,

Thank you for your feedback.

On Wed, Jun 29, 2016 at 04:23:23PM -0500, David Farmer wrote:
> And given that the code point that is being defined is 0xFFFF029A and
> that falls in the First Come First Served range, technically standards
> action isn't required.  Furthermore, this draft is really dependant on
> RFC3882 and RFC5635, both Informational Status, for a full description
> of the intended functionality.   If this draft is Standards Track,
> some may mistakenly think there is some kind of preference for this
> mechanism for triggering Blackholes, vs. the other mechanisms
> discussed in RFC3882 and RFC5635.  I fully support this defining this
> Well-Known BGP Community, however I believe it should be of equal
> stature as the other mechanisms discussed in
> RFC3882 and RFC5635.
> 
> Therefore, for the above reasons, I think the more appropriate status
> for this draft is Informational.

In light of what you described here, I don't mind changing the intended
status from STD to INFO. Also, I see similarities with NOPEER/RFC3765
which is "Informational" too.

Would anyone object to this change?

Kind regards,

Job

---

From: heasley <[email-masked]>
Date: Wed, 29 Jun 2016 23:33:46 +0000

> what it is we'd expect a network device vendor to implement or not to

It may be useful to be able to forward BH traffic off a router for analysis,
so discarding may not be desired.  I'd leave what is done with traffic by
default for configuration by operator.

---

From: heasley <[email-masked]>
Date: Wed, 29 Jun 2016 23:46:47 +0000

> Looking at section 5 of https://www.rfc-editor.org/rfc/rfc5635.txt

Why wouldn't you want to propogate BH routes?  If you want to BH the
traffic, then let it be dumped closer to the source.  You might
accidentally make things exciting for yourself, but it seems like
desirable behavior to me.

Since BH routes tend to be more specific, most are unlikely to be
propogated very far anyhow.

I think that the security concern lies far more in the lack of origin
validation [with rfc7908 usw.].

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 29 Jun 2016 21:16:15 -0400

> Source based RTBH requires some  explicit coordination between the
​ I like this game... but, "Why would the document specify that this is
either/or src/dest blackholing?"

maybe I'm expecting consenting adults to still be playing at the end of the
day...​

---

From: Randy Bush <[email-masked]>
Date: Thu, 30 Jun 2016 12:27:10 +0900

> specifies a mechanism to cause traffic to be dropped on the floor,
> that the signaling mechanism is globally transitive in scope, and the
> other ASNs. In other words, the draft specifies behaviour that is
> risky by default.

risky?  this is a disasterous vulnerability large enough to handle a
very large truck.

we really do not need a global mecahnism by which an attacker can spoof
a bgp announcement of someone's prefixes and cause traffic to the
specified address space(s) to be discarded on a significant portion of
the internet.

until bgp annoucements can be rigorously authenticated, this is a
disaster waiting to happen.  and it will not wait long.

randy

---

From: Michel Py <[email-masked]>
Date: Thu, 30 Jun 2016 03:36:47 +0000

> Nick Hilliard wrote :
> I would have said the opposite, i.e. that any traffic tagged with this prefix is dropped via e.g. null0
> or martian mechanisms / etc. But it definitely needs to be defined because at the moment it's ambiguous.
> Ambiguity is fine when it's your own network, but not fine when you're defining something with global scope.

Totally agree. Default behavior on vendor "C" router : if the prefix comes with NO-EXPORT, there is nothing to do so that it is not announced out of the local AS. Prefix not announced to eBGP peers because of well-known community. That's what well-known communities are, right ? DEFAULT behavior is coded-in by the vendor.

Same as BLACKHOLE : well-known community, router should drop it by default, and here is the major concern : DEFAULT behavior. If BLACKHOLE as currently proposed is handled the same way NO-EXPORT is, having it enabled by default is a danger.

As said in my initial reaction : too broad.

> Job Snijders wrote :
> The word 'source' does not appear in the draft. In my reading of section 3.1 it is obvious destination based blackholing,
> but I welcome a suggestion to reword a sentence in the introduction to include the phrase 'destination based blackholing'.

The destination-based was not obvious to me on the first read. Besides, my concerns are not what the intent of a draft is, but what people will do with it if implemented. A reword will definitely make it more palatable.

As of what people will do with it, what tells you that your intent of blackholing the destination will not be perverted as blackholing the source ?
There are commercial services out there already that provide source-based BGP blackhole feeds. My personal one happens to be on the wild side, but again what's the difference ?

> joel jaeggli wrote:
> It should be an inherent property that what is being blackholed is traffic bound for the prefix that the community is attached to is it not?

Yes, but how do you make that happen ? There are some people out there that have implemented uRPF. What you suggest is a standardized codepoint to kill the victim in order to protect the infrastructure. I understand that, sometimes necessary. However, if you are the prefix under attack, there is much better : feed a source-based blackhole to your upstreams, which kills the aggressor instead of killing you.

> Source based RTBH requires some explicit coordination between the parties using it.

How so ? if you trust that the party you announce your blackhole to will check that it is indeed your prefix, you are being unrealistic. Especially lately with the thing known as "PA leasing", people do announce prefixes that are not coming from the ASN they are supposed to, and it's not prefix hijacking. I'm not saying it's right, because it's basically multi-homing a PA, but it does happen and it's not going to get better.

> heasley wrote :
> Why wouldn't you want to propogate BH routes?  If you want to BH the traffic, then let it be dumped closer to the
> source.  You might accidentally make things exciting for yourself, but it seems like desirable behavior to me.

Indeed, and this is a lot more true when the blackhole targets the source. The further it is propagated, the better.

> I like this game... but, "Why would the document specify that this is either/or src/dest blackholing?"
> maybe I'm expecting consenting adults to still be playing at the end of the day...

Because it would make it less controversial ?

Michel.