Subject: [Idr] some questions from {RC, LC, EC} analysis presentation in GROW

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Tue, 3 Aug 2021 15:14:39 +0000

Some questions raised at the mike or in the chat window during my GROW presentation last week on the analysis of {Regular, Large, Extended} Communities are worth a revisit. The presentation slides are here:
https://datatracker.ietf.org/meeting/111/materials/slides-111-grow-bgp-regularextendedlarge-community-analysis-01 

Comment (Jeff): Your slides have proven that stuff is being passed around in a transitive fashion. The thing that is not necessarily a clear conclusion is that service providers are willing to pass them around in a way that allows you to safely use them in a transitive fashion for any given application.  

Related comment (Ruediger): There is a need for consideration of what hygiene should be applied to the communities you are propagating. Typically, people are concerned about the hygiene of what they accept. But in a peering relationship, you as the sender are also responsible for what you are sending; you should not propagate without understanding the security [safety] implications. Today this type of hygiene is generally lacking.

Response: We welcome any ideas that may help compile RC/LC/EC application types and their propagation requirements to perform measurements against them. Also, it is good to follow Ruediger’s suggestion and have a GROW document that provides operator guidance on what hygiene to apply on the sender side so that the propagation happens safely. Having said that, our measurements focused on whether or not ASes propagate transitive communities. The results are correct in that respect. One AS in the path may not have removed the community or stopped propagating the route further, but the other ASes that propagated are indeed correctly propagating transitive stuff. When it is not participating in the specific community application, it seems correct behavior for an AS to simply pass on the transitive community to the next AS. In general, LC and RC are transitive (unless NO_EXPORT or NO_ADVERTISE is also present). Minor point: The measurements also show that non-transitive ECs do not propagate; only transitive ECs are seen propagating (slide 15). 

Question (Chris): Why is it assumed (on slide 11) that the Blackhole community is added at the origin AS? 

Response: The RTBH service is requested by the prefix owner. That is why it is assumed that it is added at the AS where the prefix is located, i.e., the origin AS. Are there legitimate circumstances where an AS that is 2 or 3 hops upstream from the prefix can make that request? 

Thanks.
Sriram

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 3 Aug 2021 17:38:13 +0100

Sriram, Kotikalapudi (Fed) wrote on 03/08/2021 16:14:
> Response: We welcome any ideas that may help compile RC/LC/EC
> application types and their propagation requirements to perform
> measurements against them. Also, it is good to follow Ruediger’s
> suggestion and have a GROW document that provides operator guidance
> on what hygiene to apply on the sender side so that the propagation
> happens safely.

good idea.

> Response: The RTBH service is requested by the prefix owner. That is 
> why it is assumed that it is added at the AS where the prefix is 
> located, i.e., the origin AS. Are there legitimate circumstances 
> where an AS that is 2 or 3 hops upstream from the prefix can make 
> that request?
short answer: yes, although this would be unusual.

Nick

---

From: Zhuangshunwan <[email-masked]>
Date: Wed, 4 Aug 2021 02:37:26 +0000

Hi Sriram,

The community attribute example 3356:666 on page 10 may not match the actual function.
"
Example: AS path = 25160 3356 12956 6147 and RC = 3356:666
 This means that the client is at AS 6147 (origin AS) and AS 3356 is the RTBH provider
 AS Distance to RTBH provider = 2
 Propagation (#hops): The Blackhole Community propagated 3 hops in this case (AS 6147 to AS 25160)
"

According to https://onestep.net/communities/as3356/
...
--------------------------------------------------------
prefix type communities
--------------------------------------------------------
3356:123 - Customer route
3356:666 - Peer route
--------------------------------------------------------
...
--------------------------------------------------------
customer traffic engineering communities - Blackhole
--------------------------------------------------------
3356:9999 - blackhole (discard) traffic

Traffic destined for any prefixes tagged with this
community will be discarded at ingress to the Level 3
network. The prefix must be one permitted by the
customer's existing ingress BGP filter.
For some router vendors the peering
must be changed to an eBGP multihop session on the Level
3 side of the connection.
...

Regards,
Shunwan

-----Original Message-----
From: Idr [mailto:[email-masked]] On Behalf Of Sriram, Kotikalapudi (Fed)
Sent: Tuesday, August 3, 2021 11:15 PM
To: GROW WG <[email-masked]>
Cc: IDR <[email-masked]>
Subject: [Idr] some questions from {RC, LC, EC} analysis presentation in GROW

Thanks.
Sriram

_______________________________________________
Idr mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/idr

---

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Wed, 4 Aug 2021 16:58:00 +0000

Hi Shunwan,

Yes, that is a curious thing ... it seems peculiar and specific to AS 3356. 
I have started a discussion on NANOG about 3356:666, 3356:9999, etc. 
Please take a look:
https://mailman.nanog.org/pipermail/nanog/2021-August/thread.html#214447 

Only AS 3356 may be an outlier. Most other AS operators use ASN:666 or WKC 65535:666 for Blackhole Community:
https://www.google.com/search?q=BGP+community+%3A666&rlz=1C1GCEV_enUS847US847&oq=BGP+community+%3A666&aqs=chrome..69i57j69i64.9798j1j15&sourceid=chrome&ie=UTF-8&safe=active&ssui=on 

Also, we'll check -- on slide 12 of my GROW presentation -- out of the roughly 265K count of unique {Prefix, AS Path, RC = Any:666}, how many are with 3356:666. I will let you know.

Sriram 

________________________________________
From: GROW <[email-masked]> on behalf of Zhuangshunwan <[email-masked]>
Sent: Tuesday, August 3, 2021 10:37 PM
To: Sriram, Kotikalapudi (Fed); GROW WG
Cc: IDR
Subject: Re: [GROW] some questions from {RC, LC, EC} analysis presentation in GROW

Hi Sriram,

Regards,
Shunwan

---

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Mon, 9 Aug 2021 17:06:46 +0000

I have heard back from Lumen/Level3 and they have confirmed the following: 

remarks:        prefix type communities
remarks:        --------------------------------------------------------
remarks:        3356:123 - Customer route
remarks:        3356:666 - Peer route

They also stated, “The 123 and 666 communities are announced to our customers intentionally.”

I think the above info is good from the point of view of our measurements. We no longer treat 3356:666 as a Blackhole community. So, we separate them from other ASN:666. We look at the propagation of 3356:666 and 3356:123. Both are meant to start at AS 3356 and are expected to propagate down the customer cone (according to the info from Lumen/Level3 above). We do observe very substantial numbers of 3356:666 and 3356:123:

RIB data (RouteViews3, 2021-07-15.0000):
Total # Unique {Prefix, RC = 3356:666} ; 509900
Total # Unique {Prefix, RC = 3356:123} ; 399567
Total # Unique {Prefix, RC = 3356:9999} ; 28

This is somewhat along the lines of what Jeff was also requesting: measure the propagation against known applications. So, there are about 510K Unique {Prefix, RC = 3356:666} and 400K Unique {Prefix, RC = 3356:123}. They are observed propagating multiple hops starting from AS 3356 (we’ll update the slides with this distribution). Hopefully, much of this propagation is down the customer cone as expected. We don't know if some of them are route leaks, but we can try to check that as part of further investigation.

Any further thoughts/comments?

Sriram   
------------------------------------------

________________________________________
From: Sriram, Kotikalapudi (Fed) <[email-masked]>
Sent: Wednesday, August 4, 2021 12:58 PM
To: Zhuangshunwan; Sriram, Kotikalapudi (Fed); GROW WG
Cc: IDR
Subject: Re: some questions from {RC, LC, EC} analysis presentation in GROW

Hi Shunwan,

Sriram

Hi Sriram,

Regards,
Shunwan

---

From: Zhuangshunwan <[email-masked]>
Date: Tue, 10 Aug 2021 01:56:42 +0000

Hi Sriram,

Thanks for your great job! Your work has given me a very in-depth understanding of the propagation behavior of BGP community attributes on the Internet.
Regarding " Total # Unique {Prefix, RC = 3356:9999} ; 28", why is the number only 28? It may be that the mask of black hole routes is usually greater than 24 (for IPv4 prefixes), preventing such routes from spreading widely on the Internet?
If the answer to the above question is "yes", then if other communities "ASN:666" are widespread in the wild, then such "ASN:666" may not be a black hole community attribute too? As far as I know, the other two examples are 263:666 and 5511:666.

Regards,
Shunwan

-----Original Message-----
From: Sriram, Kotikalapudi (Fed) [mailto:[email-masked]] 
Sent: Tuesday, August 10, 2021 1:07 AM
To: Zhuangshunwan <[email-masked]>
Cc: Jeffrey Haas <[email-masked]>; GROW WG <[email-masked]>; IDR <[email-masked]>
Subject: Re: some questions from {RC, LC, EC} analysis presentation in GROW

RIB data (RouteViews3, 2021-07-15.0000):
Total # Unique {Prefix, RC = 3356:666} ; 509900 Total # Unique {Prefix, RC = 3356:123} ; 399567 Total # Unique {Prefix, RC = 3356:9999} ; 28

Sriram   

Hi Shunwan,

Sriram

Hi Sriram,

The community attribute example 3356:666 on page 10 may not match the actual function.
"
Example: AS path = 25160 3356 12956 6147 and RC = 3356:666  This means that the client is at AS 6147 (origin AS) and AS 3356 is the RTBH provider  AS Distance to RTBH provider = 2  Propagation (#hops): The Blackhole Community propagated 3 hops in this case (AS 6147 to AS 25160) "

Traffic destined for any prefixes tagged with this community will be discarded at ingress to the Level 3 network. The prefix must be one permitted by the customer's existing ingress BGP filter.
For some router vendors the peering
must be changed to an eBGP multihop session on the Level
3 side of the connection.
...

Regards,
Shunwan

---

From: Zhuangshunwan <[email-masked]>
Date: Tue, 10 Aug 2021 02:00:44 +0000

Sorry, there is a typo, 263:666 should be 2603:666.

-----Original Message-----
From: Idr [mailto:[email-masked]] On Behalf Of Zhuangshunwan
Sent: Tuesday, August 10, 2021 9:57 AM
To: Sriram, Kotikalapudi (Fed) <[email-masked]>
Cc: IDR <[email-masked]>; GROW WG <[email-masked]>
Subject: Re: [Idr] some questions from {RC, LC, EC} analysis presentation in GROW

Hi Sriram,

Regards,
Shunwan

-----Original Message-----
From: Sriram, Kotikalapudi (Fed) [mailto:[email-masked]] 
Sent: Tuesday, August 10, 2021 1:07 AM
To: Zhuangshunwan <[email-masked]>
Cc: Jeffrey Haas <[email-masked]>; GROW WG <[email-masked]>; IDR <[email-masked]>
Subject: Re: some questions from {RC, LC, EC} analysis presentation in GROW

Sriram   

Hi Shunwan,

Sriram

Hi Sriram,

Regards,
Shunwan

---

From: Michel Py <[email-masked]>
Date: Tue, 10 Aug 2021 02:27:41 +0000

> Zhuangshunwan wrote :
> then if other communities "ASN:666" are widespread in the wild

They are.

I am the operator of one of the largest ASN:666 BGP blacklist feeds; in the past, I have opposed the standardization of ASN:666 because the text was too vague.
Long story made short : there is not enough separation between source-based BGP backlists and destination-based ones.

As of now, it appears to me that destination-based ASN:666 communities are becoming a de-facto standard; which means that my own source-based ASN:666 BGP feed needs to adopt another community.

I suggest that, if some standardization effort is to take place again, the ASN:666 scheme is used for destination-based BGP blacklist feeds, and that the ASN:888 scheme is used for source-based BGP backlist feeds.
In there, I am happy to follow the lead of Team Cymru in their bogon BGP feed, which is the origin of all BGP blacklist feeds.
https://team-cymru.com/community-services/bogon-reference/bogon-reference-bgp/

In other words : the :666 community shall be used when one wants to backlist one's own prefixes (possibly a /32), a destination-based backlist. While the :888 community shall be used when one wants to blacklist an IP address by the source, which means a high level of trust in the feed, as any contributor to said feed has potentially the ability to blacklist a source IP.

Respectfully submitted.

Michel.

---

From: Zhuangshunwan <[email-masked]>
Date: Tue, 10 Aug 2021 03:25:36 +0000

Hi Michel,

Thank you for your valuable information!
Per the experience of RFC 7999, maybe it is better to use 65535:888 to implement the function of ASN:888？

Regards,
Shunwan

-----Original Message-----
From: Michel Py [mailto:[email-masked]] 
Sent: Tuesday, August 10, 2021 10:28 AM
To: Zhuangshunwan <[email-masked]>; Sriram, Kotikalapudi (Fed) <[email-masked]>
Cc: IDR <[email-masked]>; GROW WG <[email-masked]>
Subject: RE: some questions from {RC, LC, EC} analysis presentation in GROW

They are.

Respectfully submitted.

Michel.

---

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Thu, 12 Aug 2021 17:07:14 +0000

Hi Shunwan,

>Thanks for your great job! Your work has given me a very in-depth understanding of the propagation behavior of BGP community attributes on the Internet.

Glad to hear that. I share the compliments with my colleague Lilia Hannachi.

>Regarding " Total # Unique {Prefix, RC = 3356:9999} ; 28", why is the number only 28? It may be that the mask of black hole routes is usually greater than 24 (for IPv4 prefixes), preventing such rou…

The routes with Blackhole community 3356:9999 or (more generally) ASN:666 (where ASN is not 3356, 5511, or 2603) should be short-lived. The AS providing the corresponding RTBH service should clean up those Blackhole routes from the RIBs after the DDoS mitigation is done. See additional explanations below.
  
>If the answer to the above question is "yes", then if other communities "ASN:666" are widespread in the wild, then such "ASN:666" may not be a black hole community attribute too? As far as I know, t…

Since you mentioned that 5511 and 2603 also do not use ASN:666 for Blackhole, we were able to confirm the same and measured the following:

RIB data (RouteViews3, 2021-07-15.0000):
# Unique {Prefix, RC = 65535:666} = 221
# Unique {Prefix, RC = 3356:666} = 509900
# Unique {Prefix, RC = 5511:666} = 15157
# Unique {Prefix, RC = 2603:666} = 0  (this zero is based on Routeviews3 RIB, 
      but we do see a substantial # 2603:666 in RIPE-RIS BGP Updates 
      since AS 2603 is located in Europe!)  
# Unique {Prefix, RC = ASN:666} where ASN is NOT equal to 3356, 2603, or 5511 = 4638

So, when we eliminate prefixes with 3356:666, 5511:666, or 2603:666, the remaining prefixes with ASN:666 (presumed Blackhole) are much fewer ( = 4638). This is a good thing. Not too many Blackhole ASN:666 should be seen propagating on the Internet because of three reasons: (1) They should propagate typically only one or two hops and then they should be prevented from propagating further by the corresponding AS providing RTBH service; (2) (as you said) they also do not propagate because often their route mask (prefix length) is greater than 24 (IPv4) or 48 (IPv6); and (3) the AS providing the RTBH service should clean up the Blackhole communities from its RIBs after the DDoS attack is mitigated. So, at any given time there should not be too many routes with Blackhole communities in the RIB.         

As the above data shows that after eliminating just the three ASNs that you pointed out the remaining presumed Blackhole ASN:666 are already much fewer. 

I think you’ll find the following measurements about observed prefix lengths interesting as well:

Frequency distribution of IPv4 prefix lengths in the set of Unique {Prefix, RC = ASN:666} where ASN is NOT equal to 3356, 2603, or 5511: 

12 ; 2
14 ; 8
15 ; 5
16 ; 40
17 ; 12
18 ; 9
19 ; 34
20 ; 58
21 ; 80
22 ; 262
23 ; 275
24 ; 2185
30 ; 4
32 ; 1641

Most of the mass is at /24 and /32 (in the above), possibly indicative of genuine use as ASN:666 Blackhole communities.

Frequency distribution of IPv6 prefix lengths in the set of Unique {Prefix, RC = ASN:666} where ASN is NOT equal to 3356, 2603, or 5511 : 

25 ; 1
32 ; 7
36 ; 1
44 ; 1
48 ; 12
128 ; 1

In the above IPv4/IPv6 distribution data, some prefixes with large prefix lengths made it to the collector, but most such prefixes were likely not propagated (correctly so). 

Please let me know if you find other ASNs for which ASN:666 is not Blackhole. Thanks.

Sriram

---

From: Zhuangshunwan <[email-masked]>
Date: Fri, 20 Aug 2021 09:44:46 +0000

Hi Sriram,

Thank you so much for the information, it was very helpful and interesting!
As a fan of the BGP protocol, I particularly like seeing Internet routes carry various community attribute information as they propagate, which gives us the opportunity to see some of the details of the actual operation of the Internet rather than just a big black box. In particular, AS3356 carries various rich community attributes. (Some identify business relationships, some identify geographic information, and so on), It gives us the opportunity to learn about the mysterious Internet..
For security reasons, many ISPs delete received community attributes at their ingress border and then tag their own community attributes, which are deleted at the egress border of that ISP, and these ISPs seem to be a tight black box.
By default, some BGP software does not send community attributes to its neighbors. Instead, it needs to be explicitly enabled a knob before sending community attributes. In addition, many software inherits the community attribute behavior when implementing Large Community.  As a result, contrary to our expectations, large communities may not be widely spread on the Internet.

Looking forward to more interesting output from your research work!

Regards,
Shunwan

-----Original Message-----
From: Sriram, Kotikalapudi (Fed) [mailto:[email-masked]] 
Sent: Friday, August 13, 2021 1:07 AM
To: Zhuangshunwan <[email-masked]>
Cc: GROW WG <[email-masked]>; IDR <[email-masked]>
Subject: Re: some questions from {RC, LC, EC} analysis presentation in GROW

Hi Shunwan,

  

RIB data (RouteViews3, 2021-07-15.0000):
# Unique {Prefix, RC = 65535:666} = 221
# Unique {Prefix, RC = 3356:666} = 509900 # Unique {Prefix, RC = 5511:666} = 15157 # Unique {Prefix, RC = 2603:666} = 0  (this zero is based on Routeviews3 RIB, 
      but we do see a substantial # 2603:666 in RIPE-RIS BGP Updates 
      since AS 2603 is located in Europe!) # Unique {Prefix, RC = ASN:666} where ASN is NOT equal to 3356, 2603, or 5511 = 4638

I think you'll find the following measurements about observed prefix lengths interesting as well:

12 ; 2
14 ; 8
15 ; 5
16 ; 40
17 ; 12
18 ; 9
19 ; 34
20 ; 58
21 ; 80
22 ; 262
23 ; 275
24 ; 2185
30 ; 4
32 ; 1641

25 ; 1
32 ; 7
36 ; 1
44 ; 1
48 ; 12
128 ; 1

Sriram