Subject: Re: [Sidrops] [GROW] Deprecation of AS_SET and AS_CONFED_SET -- feedback requested

From: Jared Mauch <[email-masked]>
Date: Wed, 25 Sep 2019 20:13:35 -0400

(I didn’t see any follow-up) but I suspect the intent was adoption.

I did notice after this a few more BGP announcements with AS_SET occurring on a very regular basis.  I suspect someone is doing some research on it, perhaps someone reading this e-mail :-)

- Jared

> On Aug 7, 2019, at 12:23 PM, Susan Hares <[email-masked]> wrote:
> 
> Sriram: 
> 
> Are you asking for WG Adoption of this draft at this time or just feedback?
> 
> 
> Sue 
> 
> -----Original Message-----
> From: GROW [mailto:[email-masked]] On Behalf Of Sriram, Kotikalapudi
> (Fed)
> Sent: Wednesday, August 7, 2019 11:43 AM
> To: IDR; GROW WG
> Cc: [email-masked]; [email-masked]
> Subject: [GROW] Deprecation of AS_SET and AS_CONFED_SET -- feedback
> requested
> 
> There was significant interest expressed during a SIDROPS presentation in
> Montreal and in the discussion that followed at the mike about advancing the
> following individual draft:
> https://tools.ietf.org/html/draft-kumari-deprecate-as-set-confed-set-14  
> (standards track)   
> 
> The authors have revised and uploaded a new version (-14).
> Your inputs/comments are welcome. In particular, please share any specific
> operational considerations/concerns you may have regarding this topic.
> 
> (Note: A BCP was published on this topics in 2011 --
> https://tools.ietf.org/html/rfc6472 )   
> 
> Thank you.
> 
> Sriram
> _______________________________________________
> GROW mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/grow
> 
> _______________________________________________
> GROW mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/grow

---

From: Jared Mauch <[email-masked]>
Date: Wed, 25 Sep 2019 20:16:45 -0400

(Obviously my search in e-mail today went afoul, I see it’s adopted.. but like I said.. there’s a lot more AS_SETs that appeared in BGP announcements since this was posted).

- Jared

> On Sep 25, 2019, at 8:13 PM, Jared Mauch <[email-masked]> wrote:
>

---

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Thu, 26 Sep 2019 03:32:36 +0000

Jared,

>(Obviously my search in e-mail today went afoul, I see it’s adopted.. 

Yes, the draft was adopted. You can see the discussion here:
https://mailarchive.ietf.org/arch/browse/idr/?gbt=1&q=WG%20adoption%20for%20draft-kumari-deprecate-as-set-confed-set  

One key point of the discussion was about the specification for action at 
a receiver when an update with AS_SET or AS_CONFED_SET is received.
The consensus seemed to be: treat as Withdraw.
We'll soon update the draft to add that.

You had observed in that thread:
>We should also (in parallel) reach out to some of the networks 
>still using as_sets that are visible in the global BGP table.   

How do we do this? On NANOG, RIPE, etc. lists? Please advise.

>I did notice after this a few more BGP announcements with AS_SET 
>occurring on a very regular basis.  I suspect someone is doing some 
>research on it, perhaps someone reading this e-mail :-)

The percentage of updates with AS_SET is observed to be about 0.05%.
This has been in that ballpark for a long time.
Are you computing the percentages in what you observe? 
Is it much different from 0.05%?

Some recent measurements we have at NIST about this are as follow:
(thanks to Lilia Hannachi - my colleague)

******* OREGON3 Collector (20190827.00) **************
August 27, 2019

IPV4 : {normal=16540660, AGGREGATOR=1718902, ATOMIC_AGGREGATE=1086313, AS_CONFED_SET=0, AS_SET=8971, AS_CONFED_SEQUENCE=0}
IPV4 Total 18323517
IPV4 Percentage
normal : 90.27%
AGGREGATOR : 9.38%
ATOMIC_AGGREGATE : 5.93%
AS_CONFED_SET : 0.0%
AS_SET : 0.05%
AS_CONFED_SEQUENCE : 0.0%
Number of ASs in the AS_SET Paths (IPV4) {1=7837, 2=876, 3=144, 4=50, 5=53, 6=2, 7=9}
Position of AS_SET (IPV4) {Beginning=8971, Middle=0, Left_Most=0}
Number of AS_SET with Reserver ASes (IPV4) 248
Number of no-export and no-advertise (IPV4) {no-export=0, no-advertise=751278}
no-export : 0.0%
no-advertise : 4.95%
---------------------------------------------------------------
IPV6 : {normal=67633, AGGREGATOR=8297, ATOMIC_AGGREGATE=3548, AS_CONFED_SET=0, AS_SET=36, AS_CONFED_SEQUENCE=0}
IPV6 Total 76128
IPV6 Percentage
normal : 88.84%
AGGREGATOR : 10.9%
ATOMIC_AGGREGATE : 4.66%
AS_CONFED_SET : 0.0%
AS_SET : 0.05%
AS_CONFED_SEQUENCE : 0.0%
Number of ASs in the AS_SET Paths (IPV6) {1=24, 2=8, 3=1, 4=2, 8=1}
Position of AS_SET (IPV6) {Beginning=36, Middle=0, Left_Most=0}
Number of AS_SET with Reserver ASes (IPV6) 0
Number of no-export and no-advertise (IPV6) {no-export=0, no-advertise=3765}
no-export : 0.0%
no-advertise : 4.95%
---------------------------------------------------------------------------

******* LINX Collector (20190827.00) **************
August 27, 2019

IPV4 : {normal=21697743, AGGREGATOR=2255396, ATOMIC_AGGREGATE=1425305, AS_CONFED_SET=0, AS_SET=11353, AS_CONFED_SEQUENCE=0}
IPV4 Total 24036253
IPV4 Percentage
normal : 90.27%
AGGREGATOR : 9.38%
ATOMIC_AGGREGATE : 5.93%
AS_CONFED_SET : 0.0%
AS_SET : 0.05%
AS_CONFED_SEQUENCE : 0.0%
Number of ASs in the AS_SET Paths (IPV4) {1=9890, 2=1130, 3=188, 4=64, 5=65, 6=2, 7=14}
Position of AS_SET (IPV4) {Beginning=11353, Middle=0, Left_Most=0}
Number of AS_SET with Reserver ASes (IPV4) 348
Number of no-export and no-advertise (IPV4) {no-export=6, no-advertise=0}
no-export : 0.0%
no-advertise : 0.0%
*********************************
IPV6 : {normal=69177, AGGREGATOR=8335, ATOMIC_AGGREGATE=3543, AS_CONFED_SET=0, AS_SET=37, AS_CONFED_SEQUENCE=0}
IPV6 Total 77712
IPV6 Percentage
normal : 89.02%
AGGREGATOR : 10.73%
ATOMIC_AGGREGATE : 4.56%
AS_CONFED_SET : 0.0%
AS_SET : 0.05%
AS_CONFED_SEQUENCE : 0.0%
Number of ASs in the AS_SET Paths (IPV6) {1=24, 2=9, 3=1, 4=2, 8=1}
Position of AS_SET (IPV6) {Beginning=37, Middle=0, Left_Most=0}
Number of AS_SET with Reserver ASes (IPV6) 0
Number of no-export and no-advertise (IPV6) {no-export=0, no-advertise=0}
no-export : 0.0%
no-advertise : 0.0%

Sriram

---

From: Jared Mauch <[email-masked]>
Date: Thu, 26 Sep 2019 11:01:11 -0400

> On Sep 25, 2019, at 11:32 PM, Sriram, Kotikalapudi (Fed) <[email-masked]> wrote:
> 
> Jared,
> 

Looking at RIPE RIS Live data, it seems to be very often.  I rewrote my parser to something lazy and the way the RIS Live message shows up in the JSON is array of array which breaks my laziness.  The majority seem to be AS# 27738

Unparsable AS_PATH: /[196753, 3356, 2914, 12252, 23487, 27738, [27738]]/196753 3356 2914 12252 23487 27738 [27738]/ {'type': 'ris_message', 'data': {'timestamp': 1569509923.51, 'peer': '217.29.67.51', 'peer_asn': '196753', 'id': '217.29.67.51-1569509923.51-9791488', 'host': 'rrc10', 'type': 'UPDATE', 'path': [196753, 3356, 2914, 12252, 23487, 27738, [27738]], 'community': [[2914, 410], [2914, 1009], [2914, 2000], [2914, 3000], [3356, 2], [3356, 22], [3356, 86], [3356, 505], [3356, 666], [3356, 2074], [17152, 1]], 'origin': 'igp', 'aggregator': '23487:200.124.247.2', 'announcements': [{'next_hop': '217.29.67.51', 'prefixes': ['201.183.255.0/24']}]}}
Unparsable AS_PATH: /[57381, 42708, 2914, 12252, 23487, 27738, [27738]]/57381 42708 2914 12252 23487 27738 [27738]/ {'type': 'ris_message', 'data': {'timestamp': 1569509924.6, 'peer': '193.150.22.240', 'peer_asn': '57381', 'id': '193.150.22.240-1569509924.6-9205783', 'host': 'rrc00', 'type': 'UPDATE', 'path': [57381, 42708, 2914, 12252, 23487, 27738, [27738]], 'community': [[2914, 410], [2914, 1009], [2914, 2000], [2914, 3000], [42708, 200], [42708, 208]], 'origin': 'incomplete', 'aggregator': '23487:200.124.247.2', 'announcements': [{'next_hop': '193.150.22.240', 'prefixes': ['201.183.255.0/24']}]}}
Unparsable AS_PATH: /[14061, 3356, 6762, 23487, 27738, 27738, 27738, 27738, 27738, 27738, [27738]]/14061 3356 6762 23487 27738 27738 27738 27738 27738 27738 [27738]/ {'type': 'ris_message', 'data': {'timestamp': 1569509923.5, 'peer': '198.32.176.147', 'peer_asn': '14061', 'id': '198.32.176.147-1569509923.5-38413780', 'host': 'rrc14', 'type': 'UPDATE', 'path': [14061, 3356, 6762, 23487, 27738, 27738, 27738, 27738, 27738, 27738, [27738]], 'community': [[3356, 3], [3356, 22], [3356, 86], [3356, 575], [3356, 666], [3356, 2011], [6762, 1], [6762, 92], [6762, 10110], [14061, 402], [14061, 2000], [14061, 2005], [14061, 4000], [14061, 4004]], 'origin': 'igp', 'aggregator': '23487:200.124.224.2', 'announcements': [{'next_hop': '198.32.176.147', 'prefixes': ['200.124.229.0/24', '201.183.252.0/24', '200.124.225.0/24', '200.124.231.0/24', '200.124.227.0/24', '201.183.248.0/24', '200.124.228.0/24', '200.124.230.0/24', '201.183.243.0/24']}]}}
Unparsable AS_PATH: /[49605, 9002, 6762, 23487, 27738, 27738, 27738, 27738, 27738, 27738, [27738]]/49605 9002 6762 23487 27738 27738 27738 27738 27738 27738 [27738]/ {'type': 'ris_message', 'data': {'timestamp': 1569509924.54, 'peer': '91.206.52.102', 'peer_asn': '49605', 'id': '91.206.52.102-1569509924.54-9415897', 'host': 'rrc20', 'type': 'UPDATE', 'path': [49605, 9002, 6762, 23487, 27738, 27738, 27738, 27738, 27738, 27738, [27738]], 'community': [[9002, 9002], [9002, 64615], [49605, 9002]], 'origin': 'igp', 'aggregator': '23487:200.124.224.2', 'announcements': [{'next_hop': '91.206.52.102', 'prefixes': ['200.124.230.0/24', '201.183.248.0/24', '200.124.227.0/24', '200.124.231.0/24', '200.124.226.0/24', '200.124.229.0/24', '201.183.252.0/24', '201.183.243.0/24', '200.124.225.0/24', '200.124.224.0/24', '200.124.228.0/24']}]}}
Unparsable AS_PATH: /[60501, 196753, 6762, 23487, 27738, 27738, 27738, 27738, 27738, 27738, [27738]]/60501 196753 6762 23487 27738 27738 27738 27738 27738 27738 [27738]/ {'type': 'ris_message', 'data': {'timestamp': 1569509924.52, 'peer': '217.29.66.138', 'peer_asn': '60501', 'id': '217.29.66.138-1569509924.52-24972855', 'host': 'rrc10', 'type': 'UPDATE', 'path': [60501, 196753, 6762, 23487, 27738, 27738, 27738, 27738, 27738, 27738, [27738]], 'community': [[6762, 1], [6762, 31], [6762, 40], [6762, 10110], [17152, 1]], 'origin': 'igp', 'aggregator': '23487:200.124.224.2', 'announcements': [{'next_hop': '217.29.66.138', 'prefixes': ['201.183.252.0/24', '200.124.227.0/24', '200.124.230.0/24', '201.183.243.0/24', '200.124.225.0/24', '200.124.229.0/24', '200.124.228.0/24', '200.124.231.0/24', '201.183.248.0/24']}]}}

---

From: Warren Kumari <[email-masked]>
Date: Thu, 26 Sep 2019 18:43:39 -0400

> (thanks to Lilia Hannachi - my colleague)

This is nice, but what would make it more useful would be if it also
reported if there are *useful* AS_SETS / if the AS_SET means anything.

For example, from Jared's email below:
AS path:  14061 3356 6762 23487 27738 27738 27738 27738 27738 27738
{27738} -- the 27738 AS already shows up as a non-AS_SET in the path.

I've also seen a number of instances where the AS_SET contains many
repeated instances, e.g:
AS path: 6939 3356 42020 39010 {39275 39275 39275 39275 39275 39275
39275 39275 39275 39275} -- this doesn't seem to actually mean
anything...

Looking at the top few AS paths with AS_SETs I see:
  28 12127
  27 27738
  24 11139
  17 23383
  13 12849

AS12127 - Telefonica Moviles El Salvador S.A. de C.V., SV (these are
all behind AS263783 - Telefonica Moviles El Salvador S.A. de C.V., SV
)
AS27738 - Ecuadortelecom S.A., EC
AS11139 - CWC-ROC-11139 - Cable & Wireless Dominica, DM
AS23383 - METRORED S.A. DE C.V., HN   (this also seems to provide
transit to AS267737? (advertising 45.168.172.0/24))
AS12849 HOTNET-IL AMS-IX Admin LAN, IL

W

--
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 26 Sep 2019 22:52:19 -0400

> On Sep 26, 2019, at 6:43 PM, Warren Kumari <[email-masked]> wrote:
> 

This one is on the buggy end of things, but still reasonably valid.  It smells like something that passed through remove-private of some flavor.

> 39275 39275 39275 39275} -- this doesn't seem to actually mean

This one seems very buggy.  The protocol still knows what to do with it.

It'd be interesting to find out what code these folk are running. Hopefully not one of my bugs. :-)

-- Jeff

---

From: Warren Kumari <[email-masked]>
Date: Fri, 27 Sep 2019 08:49:01 -0400

> This one is on the buggy end of things, but still reasonably valid.  It smells like something that passed through remove-private of some flavor.

Yes, it is valid, but it doesn't "mean" anything more than AS path:
14061 3356 6762 23487 27738 27738 27738 27738 27738 27738 27738 - I
cannot see a scenario where the lack of {} causes loop detection or
anything else to fail.
I *guess* it's possible that there was something like ... 27738 65533
{ 27738 65534 }. The { 27738 65534 } is "interesting" to 65533, and
remote-private could have removed both 65533 and 65534.

Ugh, aggregation for others is ugly (yes, I know, but...)

> It'd be interesting to find out what code these folk are running. Hopefully not one of my bugs. :-)

They are all your bugs, Jeff :-)

W

---

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Fri, 27 Sep 2019 13:58:12 +0000

Warren,

Thanks for the additional data/suggestions. Comments below.

.....
>AS12849 HOTNET-IL AMS-IX Admin LAN, IL
....

Interesting! We can extend our analysis at NIST to examine these things as needed.
In the data that I shared earlier, another thing you might have noticed
is that there are ~ 350 instances where special purpose ASNs occur in AS_SETs.
An earlier study we did and presented at IETF is here:
https://www.nist.gov/sites/default/files/documents/itl/antd/AS_SET_Aggregator_Stats-2.pdf  
There we noticed other strange things:
In some cases, AS in the AGGREGATOR doesn't match the right most AS in the AS_PATH
that follows the AS_SET. So AGGREGATOR is different from the AS that created the AS_SET! Hmm..! 
In the slides, we also reported numbers on private ASNs seen in AS_SETs.

Sriram