Subject: [Sidrops] IXP Route Server question

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Tue, 8 Mar 2022 19:36:33 +0000

This question has relevance to the ASPA method for route leak detection.

Is it possible that an ISP AS A peers with a customer AS C via a non-transparent IXP AS B? 
IOW, the AS path in routes propagated by the ISP A for customer C's prefixes looks like this:  A B C.
I.e., can the AS of a non-transparent IXP/RS appear in an AS path in the middle between an ISP and its customer?

Sriram

---

From: Christopher Morrow <[email-masked]>
Date: Tue, 8 Mar 2022 15:02:40 -0500

> middle between an ISP and its customer?
it seems unlikely to me that an ISP would pick up a 'customer' (someone
that pays them to transport packets) at an IXP fabric.
Might it happen? sure? is it messy? yes!

1) that's probably a shared port
2) there are other folk feeding routes and packets into the mix
3) how many came through the 'customer' port (which you can't really know
easily) vs other participants on the ix
4) what capacity planning could the 'customer' do here? (none, basically
with respect to the remote ISP port)

Your question might work also as:
  "ISP A has a customer C on a direct link in location Y.
   ISP A is present at IXP-Z, so is customer C, though they do not
bilaterally peer (not do they interconnect at the IXP).
  ISP A can still see Customer C's routes through the IXP-Z Route Server."

that seems plausible, but not a desired outcome for the ISP :) since they
will be unlikely to collect pesos for the traffic
which MAY pass across that interconnect.

---

From: Christopher Morrow <[email-masked]>
Date: Tue, 8 Mar 2022 15:26:28 -0500

On Tue, Mar 8, 2022 at 3:15 PM Robert Raszuk <[email-masked]> wrote:

> Well I think the answer is - it depends.
>
> First IXP fabric can be used as pure L3 share LAN or can be used (and it
> is often the case) as a p2p emulated VLAN over such L3 shared LAN.
>
> Now if this is L3 shared LAN still customer and ISP may peer directly and
> no third party traffic would be accepted at either end.
>
> If we talk about emulating L2 IXP fabric becomes just an emulated circuit
> and from the perspective of routing it a p2p interface.
>
> Sure the other aspects of the IXP quality, port monitoring,
> oversubscription etc... always will apply but there are ways to mitigate or
> handle those in real IXPs.
>
>
I don't dispute your content here, except that Sriram's question was about
seeing 'customer routes via the RS'... which I think would obviate the
emulation examples you provided.
(well in a bunch of cases it would, you COULD hook up some tomfoolery to
get this to work, but... that sounds complex and prone to disaster)

> Best,
> R.
>
>
>
>
>
>
>
>
> On Tue, Mar 8, 2022 at 9:05 PM Christopher Morrow <
> [email-masked]> wrote:
>
>>
>>
>> _______________________________________________
>> GROW mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/grow
>>
>

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 8 Mar 2022 21:28:09 +0000

Sriram, Kotikalapudi (Fed) wrote on 08/03/2022 19:36:
> I.e., can the AS of a non-transparent IXP/RS appear in an AS path in
> the middle between an ISP and its customer?

Almost all RS's are transparent these days.  Usually IXPs go to lengths 
to ensure that the RS ASN doesn't appear in the AS path.

Some organisations provide transit over IXPs, but it's a minority thing. 
It would be very peculiar if an organisation provided transit over an 
IXP via an RS.

Some organisations provide transit to ASNs over a direct physical 
connection while maintain peering with their customer over an IXP port. 
Usually this happens by accident, but occasionally it can happen by design.

The answer to your question is that it would be technically possible, 
but it would be so peculiar and stupid that it should be considered a 
mistake in the situations where it was intentional. In all other 
situations, it would be a leak.  Generally it would be safe to assume 
that this sort of configuration was in error.

Nick

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 8 Mar 2022 21:16:18 +0100

First IXP fabric can be used as pure L3 share LAN or can be used (and it is
often the case) as a p2p emulated VLAN over such L3 shared LAN.

Best,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 8 Mar 2022 21:34:01 +0100

Right - but IMO route leaking can happen both in the Internet or in
customer <- via IXP -> content provider interconnects.

And in the latter case - especially for those with open peering policy -
often going via RS. After all this is how route servers are mainly used
today :) So both sides will be peering to IXP RS while IXP RS will (in most
cases) not appear in the AS_PATH.

Kind regards,
R.

On Tue, Mar 8, 2022 at 9:26 PM Christopher Morrow <
[email-masked]> wrote:

>>> 3) how many came through the 'customer' port (which you can't really
>>> know easily) vs other participants on the ix
>>> that seems plausible, but not a desired outcome for the ISP :) since
>>> they will be unlikely to collect pesos for the traffic

---

From: Christopher Morrow <[email-masked]>
Date: Tue, 8 Mar 2022 16:30:23 -0500

> cases) not appear in the AS_PATH.
sure! that was my re-wording of sriram's question, effectively. (or I think
that was the re-wording!)

>>> Now if this is L3 shared LAN still customer and ISP may peer directly
>>> and no third party traffic would be accepted at either end.
>>>
>>> If we talk about emulating L2 IXP fabric becomes just an emulated
>>> circuit and from the perspective of routing it a p2p interface.
>>>
>> I don't dispute your content here, except that Sriram's question was
>> about seeing 'customer routes via the RS'... which I think would obviate
>> the emulation examples you provided.
>>>> 4) what capacity planning could the 'customer' do here? (none,
>>>> basically with respect to the remote ISP port)
>>>>

---

From: Ben Maddison <[email-masked]>
Date: Tue, 8 Mar 2022 23:39:07 +0200

> Might it happen? sure? is it messy? yes!
I know of several transit providers that will allow customers to use an
IXP as a kind of virtual access circuit (which itself is a poor idea),
but I would be *very* surprised if any of them allow RS peerings to be
the control plane interconnection (intentionally, at least).

If the underlying question is "should the ASPA path validation algorithm
have a corner case that accommodates this?", that is a very, very firm
"no" from me!

Cheers,

Ben

---

From: Randy Bush <[email-masked]>
Date: Tue, 08 Mar 2022 15:05:48 -0800

> If the underlying question is "should the ASPA path validation
> algorithm have a corner case that accommodates this?", that is a
> very, very firm "no" from me!

aol

---

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Thu, 10 Mar 2022 03:31:25 +0000

Nick and all,

Thank you. What you all shared/discussed is very useful info.

>Almost all RS's are transparent these days.  Usually IXPs go to lengths to ensure that the RS ASN doesn't appear in the AS path.

Good to know that. Well, that means there can be an occasional RS that is non-transparent. When there is a non-transparent RS, could there be big ISPs (Tier-1, Tier-2) present there as RS-clients?

The ASPA verification draft treats the relationship of RS to RS-client as similar to that of Provider to Customer. Seems reasonable? The AS of an RS client includes the RS's AS in its ASPA as a "Provider".

Sriram  

-----Original Message-----
From: Nick Hilliard <[email-masked]> 
Sent: Tuesday, March 8, 2022 4:28 PM
To: Sriram, Kotikalapudi (Fed) <[email-masked]>
Cc: [email-masked]; [email-masked]
Subject: Re: [GROW] IXP Route Server question

> the middle between an ISP and its customer?

Almost all RS's are transparent these days.  Usually IXPs go to lengths to ensure that the RS ASN doesn't appear in the AS path.

Some organisations provide transit over IXPs, but it's a minority thing. 
It would be very peculiar if an organisation provided transit over an IXP via an RS.

Some organisations provide transit to ASNs over a direct physical connection while maintain peering with their customer over an IXP port. 
Usually this happens by accident, but occasionally it can happen by design.

The answer to your question is that it would be technically possible, but it would be so peculiar and stupid that it should be considered a mistake in the situations where it was intentional. In all other situations, it would be a leak.  Generally it would be safe to assume that this sort of configuration was in error.

Nick

---

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Thu, 10 Mar 2022 03:40:51 +0000

Ben,

>I know of several transit providers that will allow customers to use an IXP as a kind of virtual access circuit (which itself is a poor idea), but I would be *very* surprised if any of them allow RS peerings to be the control plane interconnection (intentionally, at least).

Good. Thanks for that insight. Consistent with what Nick and others observed.

>If the underlying question is "should the ASPA path validation algorithm have a corner case that accommodates this?", that is a very, very firm "no" from me!

No. I didn't have that question or idea in mind.

But see a couple of other new questions I asked in my reply to Nick.

Thank you.

Sriram

---

From: Nick Hilliard <[email-masked]>
Date: Thu, 10 Mar 2022 23:11:47 +0000

>> Almost all RS's are transparent these days.  Usually IXPs go to
>> lengths to ensure that the RS ASN doesn't appear in the AS path.

> Good to know that. Well, that means there can be an occasional RS
> that is non-transparent. When there is a non-transparent RS, could
> there be big ISPs (Tier-1, Tier-2) present there as RS-clients?

I would seriously doubt it.  There are some "Tier 1" and "Tier 2" 
providers who peer using route servers at IXPs, but it's a bit unusual 
(and entertaining to hear them strenuously denying that this ever happens).

Non-transparent RS's were used until a couple of years ago but I'm not 
aware of any left at this point. Even when non-transparent RS's were 
more widespread, I would have expected the intersection of these two 
groups to be null, and that if there were an intersection, the 
Tier1/Tier2 providers would not be upset if those paths were dropped / 
marked as invalid.

> The ASPA verification draft treats the relationship of RS to
> RS-client as similar to that of Provider to Customer. Seems
> reasonable? The AS of an RS client includes the RS's AS in its ASPA
> as a "Provider".

yes, this is reasonable.

Nick

---

From: Ben Maddison <[email-masked]>
Date: Fri, 11 Mar 2022 09:23:07 +0200

Hi Sriram,

> Sriram, Kotikalapudi (Fed) wrote on 10/03/2022 03:31:
[...]
> > reasonable? The AS of an RS client includes the RS's AS in its ASPA
Essential, I would think: how could a far end relying party know that an
AS in the middle of a received AS_PATH is a non-transparent IXP RS in
order to apply any other treatment?

A special mark in the ASPA itself perhaps, but yuk ;-)

Cheers,

Ben

---

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Fri, 11 Mar 2022 16:20:38 +0000

Sounds good. Thank you so much for the discussions and info.

Sriram 

-----Original Message-----
From: Ben Maddison <[email-masked]> 
Sent: Friday, March 11, 2022 2:23 AM
To: Nick Hilliard <[email-masked]>
Cc: Sriram, Kotikalapudi (Fed) <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: [GROW] IXP Route Server question

Hi Sriram,

[...]
> > reasonable? The AS of an RS client includes the RS's AS in its ASPA 
Essential, I would think: how could a far end relying party know that an AS in the middle of a received AS_PATH is a non-transparent IXP RS in order to apply any other treatment?

Cheers,

Ben

---

From: Zhuangshunwan <[email-masked]>
Date: Sat, 12 Mar 2022 01:43:19 +0000

Hi Sriram,

> The ASPA verification draft treats the relationship of RS to RS-client as similar
> to that of Provider to Customer. Seems reasonable? The AS of an RS client
> includes the RS's AS in its ASPA as a "Provider".
>

IMO, the ASPA verification draft regards the relationship between RS and RS-client as "peer-to-peer", maybe my understanding is wrong, I will read the ASPA verification draft again.

BR,
Shunwan

> From: GROW [mailto:[email-masked]] On Behalf Of Sriram,
> Kotikalapudi (Fed)
> Sent: Thursday, March 10, 2022 11:31 AM
> >Almost all RS's are transparent these days.  Usually IXPs go to lengths to
> ensure that the RS ASN doesn't appear in the AS path.
> 
> Good to know that. Well, that means there can be an occasional RS that is
> non-transparent. When there is a non-transparent RS, could there be big ISPs
> (Tier-1, Tier-2) present there as RS-clients?
> 
> The ASPA verification draft treats the relationship of RS to RS-client as similar
> to that of Provider to Customer. Seems reasonable? The AS of an RS client
> includes the RS's AS in its ASPA as a "Provider".
> 
> Sriram
> 
> Almost all RS's are transparent these days.  Usually IXPs go to lengths to
> ensure that the RS ASN doesn't appear in the AS path.
> 
> Some organisations provide transit to ASNs over a direct physical connection
> while maintain peering with their customer over an IXP port.
> The answer to your question is that it would be technically possible, but it
> would be so peculiar and stupid that it should be considered a mistake in the
> situations where it was intentional. In all other situations, it would be a leak.
> Generally it would be safe to assume that this sort of configuration was in
> error.
> 
> Nick

---

From: Nick Hilliard <[email-masked]>
Date: Sat, 12 Mar 2022 21:59:28 +0000

> order to apply any other treatment?

given that they're a shrinking rarity, would it not make sense to 
completely exclude non-transparent RSs from the ASPA definition? In the 
short term this would cause problems for ASNs which connect to 
non-transparent RSs, but there are hardly any left, and only one 
sizeable one.

I wonder whether it's a good idea to design a long term security 
mechanism which includes a specific carve-out for a legacy corner case 
like this.

Nick

---

From: Randy Bush <[email-masked]>
Date: Sat, 12 Mar 2022 14:41:28 -0800

> completely exclude non-transparent RSs from the ASPA definition? In
> the short term this would cause problems for ASNs which connect to

i do not wonder about adding complexity to specs and code in a security
application.  i know it's a bad idea.

randy

---

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Sun, 13 Mar 2022 16:20:02 +0000

Nick,

>> Essential, I would think: how could a far end relying party know that 
>> an AS in the middle of a received AS_PATH is a non-transparent IXP RS 
>> in order to apply any other treatment?

>given that they're a shrinking rarity, would it not make sense to completely exclude non-transparent RSs from the ASPA definition? In the short term this would cause problems for ASNs which connect to non-transparent RSs, but there are hardly any left, and only one sizeable one.

>I wonder whether it's a good idea to design a long term security mechanism which includes a specific carve-out for a legacy corner case like this.

Not sure why Ben even raised that question. To me, it doesn't seem relevant. In the route leak detection procedures, the receiving/validating AS does not require information about the nature of ASes (RS or not RS) in the AS Path except for the sending/neighbor AS which it knows to be an RS in case it knows itself to be an RS-client. The procedures rely only on ASPA objects for the origin AS and ASes in the middle. 

Sriram

---

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Sun, 13 Mar 2022 16:44:53 +0000

Hi Shunwan,

>> The ASPA verification draft treats the relationship of RS to RS-client 
>> as similar to that of Provider to Customer. Seems reasonable? The AS 
>> of an RS client includes the RS's AS in its ASPA as a "Provider".

>IMO, the ASPA verification draft regards the relationship between RS and RS-client as "peer-to-peer", 

That is not correct.

>maybe my understanding is wrong, I will read the ASPA verification draft again.

OK. Just FYI... the draft needs a revision. It does not contain yet the enhanced ASPA upstream/downstream verification algorithms/procedures. Not sure if you've followed sidrops discussions on the algorithms since January 2021. Some pointers:
https://datatracker.ietf.org/meeting/110/materials/slides-110-sidrops-sriram-aspa-alg-accuracy-01
https://mailarchive.ietf.org/arch/msg/sidrops/C6Ethp2aC9sJsxDtBrLQDjmCqk4/

Thanks.

Sriram

---

From: Nick Hilliard <[email-masked]>
Date: Sun, 13 Mar 2022 23:56:14 +0000

Sriram, Kotikalapudi (Fed) wrote on 13/03/2022 16:20:
> Not sure why Ben even raised that question. To me, it doesn't seem
> relevant. In the route leak detection procedures, the
> receiving/validating AS does not require information about the nature
> of ASes (RS or not RS) in the AS Path except for the sending/neighbor
> AS which it knows to be an RS in case it knows itself to be an
> RS-client. The procedures rely only on ASPA objects for the origin AS
> and ASes in the middle.
Stepping back a bit, there's no fundamental difference between the 
behaviour of a transparent RS and a non-transparent RS, except that the 
transparent RS happens to elide the RS AS number from the AS path.  The 
remainder of the AS path will be the same in either case.

The algorithm for check_ix_path() implements a check to flip between 
upstream path verification and downstream path verification based on 
whether the RS is transparent or non-transparent.

Even if it gives a workable outcome, I think this is the wrong approach 
from an algorithmic point of view. The algorithm needs to accept that 
the two situations are basically the same except that in one (rare) 
case, the RS ASN is present in the AS path, and in the other, it isn't.

Section 5.3 says that the client has prior knowledge about whether the 
peer is a RS.  I.e. the draft already admits that rs client 
implementations will need a config flag.

So rather than discussing whether non-transparent ASNs are included in 
the specification, the discussion needs to focus on whether RS ASN's 
should be included in the ASPA verification algorithm at all, and if so 
/ if not, how this should be done.

The answer to this question is (surprisingly) not immediately clear. 
RSs do not partake in traffic forwarding, so they are not part of the 
data path between two ASNs; they're only part of the signaling path 
between the two ASNs.  This is a useful hack from a practical point of 
view, but it causes algorithmic breakage in places which assume 
congruency between the data plane and the signaling plane.  One of these 
places is AS path verification.

This changes the question of whether or not a hack is deployed to 
mitigate the effects of RS hackiness, but simply what style of hack 
should be used.  The current proposal is a mitigation approach, but can 
I propose an alternative approach, namely that the draft reintroduces 
congruence between the data plane and the signaling plane from the point 
of view of ASPA verification?

I.e. if the WG opinion is to include RS's as being in scope, then the AS 
path that the aspa algorithm operates on should include the RS ASN, 
regardless of how the RS is transparent or non-transparent. 
Alternatively, if WG opinion is to exclude RS's from the scope of this 
draft, then ASPA should be handled on the basis that the RS ASN is 
deleted from the as path list.

The only information necessary is whether the rs-client is aware that 
it's talking to a RS, but we already know that.

If RS's are kept in scope, then the RS should be treated as a provider 
by the rs client; the rs client should include the RS ASN in its SPAS; 
the rs client should evaluate ASPA as if the RS ASN were included in the 
path; the rs should evaluate clients as customers

If they're out of scope, then ASPA verification should elide the RS ASN 
from the AS PATH if it has not already been elided, and ASPA 
verification should proceed as if the two rs-client ASNs were directly 
connected and each should treat the other as a provider.

Nick

---

From: Zhuangshunwan <[email-masked]>
Date: Mon, 14 Mar 2022 01:45:20 +0000

Hi Sriram,

Thanks for your pointers! I will read them carefully.

Best regards,
Shunwan

> From: Sriram, Kotikalapudi (Fed) [mailto:[email-masked]]
> Sent: Monday, March 14, 2022 12:45 AM
> To: Zhuangshunwan <[email-masked]>
> Subject: RE: [GROW] IXP Route Server question
> 
> Hi Shunwan,
> 
> >> reasonable? The AS of an RS client includes the RS's AS in its ASPA as a
> "Provider".
> 
> OK. Just FYI... the draft needs a revision. It does not contain yet the enhanced
> ASPA upstream/downstream verification algorithms/procedures. Not sure if
> you've followed sidrops discussions on the algorithms since January 2021.
> Some pointers:

---

From: "Mosher, Rob" <[email-masked]>
Date: Fri, 11 Mar 2022 02:37:27 +0000

HKIX inserts their ASN 4635 in the path, and they're quite large.  Having connected to route servers on hundreds of exchanges, this is the only one that I can think of that still does this. 

https://www.hkix.net/hkix/route-server.htm

https://bgp.he.net/AS4635

-rm

-----Original Message-----
From: Sidrops <[email-masked]> On Behalf Of Nick Hilliard
Sent: Thursday, March 10, 2022 6:12 PM
To: Sriram, Kotikalapudi (Fed) <[email-masked]>
Cc: [email-masked]; [email-masked]
Subject: RE: [EXTERNAL] [Sidrops] [GROW] IXP Route Server question

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

> Good to know that. Well, that means there can be an occasional RS that 
> is non-transparent. When there is a non-transparent RS, could there be 
> big ISPs (Tier-1, Tier-2) present there as RS-clients?

I would seriously doubt it.  There are some "Tier 1" and "Tier 2"
providers who peer using route servers at IXPs, but it's a bit unusual (and entertaining to hear them strenuously denying that this ever happens).

Non-transparent RS's were used until a couple of years ago but I'm not aware of any left at this point. Even when non-transparent RS's were more widespread, I would have expected the intersection of these two groups to be null, and that if there were an intersection, the
Tier1/Tier2 providers would not be upset if those paths were dropped / marked as invalid.

yes, this is reasonable.

Nick

_______________________________________________
Sidrops mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/sidrops