Subject: Re: [IPsec] draft-xu-ipsecme-esp-in-udp-lb-07

From: "Bottorff, Paul" <[email-masked]>
Date: Wed, 14 Jul 2021 23:12:41 +0000

Hi Antony, Tero, Valery:

I've put forward draft-bottorff-ipsecme-mtdcuc-ipsec-lb-00 to show the use case for esp in udp load balancing I'm concerned with. This case does not require NAT traversal.

With that said I've spent some time looking into NAT traversal and believe it is possible to combine LB and NAT. To do so the responder must not respond with the source port received in the ESP in UDP packets. Instead, the responder should use the port received by the responder in the IKE exchanges. I believe this means turning off dynamic port updating when combining NAT with LB, which does mean that a change in the NAT mapping for IKE packets (resulting from a NAT timeout or reset) could result in packets being lost, however this event would be rare. Though there may be ways to cover this boundary case I don't believe it is worth the effort to fix this problem to support long term use of IPv4.

Though my application is not concerned with RSS, it does seems using MOBIKE to generate entropy for RSS is very clumsy since this requires assigning new IP addresses just to force entropy. The LB proposal seems the best way forward for a general RSS solution.

Cheers,

Paul

-----Original Message-----
From: IPsec [mailto:[email-masked]] On Behalf Of Bottorff, Paul
Sent: Friday, April 2, 2021 2:59 PM
To: Valery Smyslov <[email-masked]>; 'Tero Kivinen' <[email-masked]>
Cc: 'IPsec' <[email-masked]>; [email-masked]
Subject: Re: [IPsec] draft-xu-ipsecme-esp-in-udp-lb-07

Hi Valery:

Agreed that LB only needs control of the source port to provide entropy. 

Our application is for traversal of highly meshed data centre fabrics. We encapsulate and de-encapsulate at the server using smart NICs and so don't have any impact on RSS or host software (nor improvement over their standard operation). We perform the encapsulation/de-encapsulation after the ESP packet is formed. Since encapsulation occurs after and de-encapsulation occurs before the IPsec stack, IPsec does not see any of the new port assignments so we don't have any issues with the SADB or IKE.

Cheers,

Paul

-----Original Message-----
From: Valery Smyslov [mailto:[email-masked]]
Sent: Thursday, April 1, 2021 11:08 PM
To: 'Tero Kivinen' <[email-masked]>; Bottorff, Paul <[email-masked]>
Cc: 'IPsec' <[email-masked]>; [email-masked]
Subject: RE: [IPsec] draft-xu-ipsecme-esp-in-udp-lb-07

Hi Tero,

> For the load balancing I think it is enough for just one of the ports 
> to be different, thus initiator could simply allocate n random source 
> port numbers, and initiate IKE from each of them to responder, and 
> then create SAs for each of them separately, thus allowing load 
> balancing using UDP encapsulation using existing hardware.

RFC 7791 + MOBIKE can be used to clone IKE SA  and move it to a different local IP+port.

Regards,
Valery.

> --
> [email-masked]
> 
> _______________________________________________
> IPsec mailing list
> [email-masked]
> INVALID URI REMOVED
> man_listinfo_ipsec&d=DwICAg&c=C5b8zRQO1miGmBeVZ2LFWg&r=CCwOcKkISkWxd8Y
> my11M8VW3U6Peq8aJ_DDlgVbQW5E&m=ykseXYzNH5MG1guNwTPMGiGby4o46mBhv92vwoS
> pb0U&s=nCqbPzmEc1xdTkL0jPmKmNgH252j3dURPVnH8bt4OtE&e=

_______________________________________________
IPsec mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ipsec

---

From: Tobias Brunner <[email-masked]>
Date: Thu, 15 Jul 2021 10:35:55 +0200

Hi Paul,

> Instead, the responder should use the port received by the responder in the IKE exchanges.

Note that if these packets have random source ports, this will only work 
if the NAT implementation plays along or there is static port forwarding 
configured.  NATs might filter inbound packets from endpoints that don't 
equal the IP/port to which the host behind the NAT originally sent 
packets when the NAT mapping was created (address and port-dependent 
filtering in terms of RFC 4787).  I guess the same could happen in 
scenarios where there are no NATs but restrictive firewalls that block 
traffic from endpoints to which the host behind the firewall did not 
send traffic.

Regards,
Tobias

---

From: "Bottorff, Paul" <[email-masked]>
Date: Thu, 15 Jul 2021 17:47:14 +0000

Hi Tobias:

The ports used for IKE packets would not be randomized since IKE would not use source port for LB and so should be stable at the NAT.

Cheers,

Paul

-----Original Message-----
From: Tobias Brunner <[email-masked]> 
Sent: Thursday, July 15, 2021 1:36 AM
To: Bottorff, Paul <[email-masked]>; Valery Smyslov <[email-masked]>; 'Tero Kivinen' <[email-masked]>; [email-masked]; 'IPsec' <[email-masked]>
Subject: Re: [IPsec] draft-xu-ipsecme-esp-in-udp-lb-07

Hi Paul,

> Instead, the responder should use the port received by the responder in the IKE exchanges.

Note that if these packets have random source ports, this will only work if the NAT implementation plays along or there is static port forwarding configured.  NATs might filter inbound packets from endpoints that don't equal the IP/port to which the host behind the NAT originally sent packets when the NAT mapping was created (address and port-dependent filtering in terms of RFC 4787).  I guess the same could happen in scenarios where there are no NATs but restrictive firewalls that block traffic from endpoints to which the host behind the firewall did not send traffic.

Regards,
Tobias

---

From: Tobias Brunner <[email-masked]>
Date: Fri, 16 Jul 2021 08:51:33 +0200

Hi Paul,

> The ports used for IKE packets would not be randomized since IKE would not use source port for LB and so should be stable at the NAT.

I was not referring to the IKE but the ESP packets sent by the responder 
to the natted IKE port for LB.  Wasn't that what you were proposing?

Regards,
Tobias

---

From: "Bottorff, Paul" <[email-masked]>
Date: Fri, 16 Jul 2021 18:02:45 +0000

Hi Tobias:

Somehow I think we are mis-understanding. Please excuse the long introduction to answer your question.

Consider an IPSEC initiator sitting behind a NAPT talking with an IPSEC responder on the Internet (within a DC).

The initiator will start with IKE. The IKE packets will not be LB and since a NAPT would be detected, the IKE packets would use source and destination port 4500.

After IKE established the SA, load balancing will be used with ESPinUDP packet exchanges. 

Both the initiator and the responder can exchange ESPinUDP with LB after the SA is established. For ESPinUDP with LB the source port will be used for entropy. This does not necessary mean the source port is random, however does mean the source port must cover a range of values sufficient to provide the required level of entropy. Draft-xu-ipsecme-esp-in-udp-lb does not provide specific guidance on entropy values, however for NAT interoperability it is desirable to limit the number of entropy values.

At an IPSEC initiator (behind NAPT) using draft-xu-ipsecme-esp-in-udp-lb will generate LB ESPinUDP packets with the entropy value in the source port and a new destination port, however the use of a new port to designate LB complicates the situation. Instead, if we consider both ends of the SA being configured for LB, then we can simply use the existing 4500 port to designate the ESPinUDP encapsulation. Using this observation IPSEC LB packets transmitted from the initiator (behind the NAPT) would have entropy in the source port and 4500 in the destination port.

Each IPSEC initiator LB packet with a different entropy value will result in a new mapping at the NAPT.

The normal behaviour of the responder (outside the NAPT) is to transmit to the destination address and port of the last packet source address and port (from the initiator), however this is where we deviate for a LB implementation. Instead the responder (outside the NAPT) transmits all responses to the address and port established for IKE, however with an entropy value in the source port. Since the NAPT has the IKE mapping this will deliver the ESPinUDP packet to the initiator where the destination port will be restored to 4500 and the source port will be the entropy value.

So to your question about NAPT filtering. If the NAPT follows REQ-8 in RFC 4787, then filtering will pass the responder packets since the destination address and port will be mapped and the source address will also be known. In the event a NAPT violates REQ-8 and does Address and Port - Dependent Filtering then the responder packets might be filtered. There may be solutions for this case, however for the present discussion it does not seem worth pursuing the case were a NAPT violates REQ-8.

Cheers,

Paul

-----Original Message-----
From: Tobias Brunner [mailto:[email-masked]] 
Sent: Thursday, July 15, 2021 11:52 PM
To: Bottorff, Paul <[email-masked]>; Valery Smyslov <[email-masked]>; 'Tero Kivinen' <[email-masked]>; [email-masked]; 'IPsec' <[email-masked]>
Subject: Re: [IPsec] draft-xu-ipsecme-esp-in-udp-lb-07

Hi Paul,

> The ports used for IKE packets would not be randomized since IKE would not use source port for LB and so should be stable at the NAT.

I was not referring to the IKE but the ESP packets sent by the responder to the natted IKE port for LB.  Wasn't that what you were proposing?

Regards,
Tobias

---

From: Paul Wouters <[email-masked]>
Date: Fri, 16 Jul 2021 15:46:05 -0400 (EDT)

> Somehow I think we are mis-understanding. Please excuse the long introduction to answer your question.

I am also very confused.

> The initiator will start with IKE. The IKE packets will not be LB and since a NAPT would be detected, the IKE packets would use source and destination port 4500.

Just to clarify, on the initiator those source/dest ports will be 4500,
but on the responder the source port can be anything and the destination
port will be 4500.

> Both the initiator and the responder can exchange ESPinUDP with LB after the SA is established.

Which SA? The initial IPsec SA ?

> For ESPinUDP with LB the source port will be used for entropy. This does not necessary mean the source port is random, however does mean the source port must cover a range of values sufficient to provide the required level of entropy. Draft-xu-ipsecme-esp-in-udp-lb does not provide specific guidance on entropy values, however for NAT interoperability it is desirable to limit the number of entropy values.

I think you are saying "pick a random port and keep using it" and not
"use a random port for each packet", right ?

> At an IPSEC initiator (behind NAPT) using draft-xu-ipsecme-esp-in-udp-lb will generate LB ESPinUDP packets with the entropy value in the source port and a new destination port

Where does this "new destination port" come from? You only have a NAT
mapping for port 500 and a NAT mapping for port 4500. Did a new
destination port get negotiated over IKE? I don't see that specified in
the draft. Or do you expect the non-NAT IPsec endpoint to auto-detect
these packets to random ports based on encap payload and SPI ?

> , however the use of a new port to designate LB complicates the situation. Instead, if we consider both ends of the SA being configured for LB, then we can simply use the existing 4500 port to designate the ESPinUDP encapsulation. Using this observation IPSEC LB packets transmitted from the initiator (behind the NAPT) would have entropy in the source port and 4500 in the destination port.

But when using only destination port 4500, then you run into the
problem of NAT mapping updates. If you send a ESPinUDP packet to
port 4500 from source port X, the peer will update its IPsec SA
endpoint and start sending all packets to that single source port.
So any entropy gain is lost.

> Each IPSEC initiator LB packet with a different entropy value will result in a new mapping at the NAPT.

Yes, a new mapping, but the old mapping will be deleted. Is your draft
saying we should _keep_ those mappings? If so, how would you ever detect
a _rela_ IKE/ESP port remapping from the NAPT router between the
endpoints?

> The normal behaviour of the responder (outside the NAPT) is to transmit to the destination address and port of the last packet source address and port (from the initiator)

Not without MOBIKE. And with MOBIKE this requires explicit signaling via
the INFORMATIONAL IKE message containing an UPDATE_ADDRESS payload.

> however this is where we deviate for a LB implementation. Instead the responder (outside the NAPT) transmits all responses to the address and port established for IKE, however with an entropy value in the source port. Since the NAPT has the IKE mapping this will deliver the ESPinUDP packet to the initiator where the destination port will be restored to 4500 and the source port will be the entropy value.

I don't see how this can properly work. It would be really helpful if
you can explain this with ASCII art in your draft document.

> So to your question about NAPT filtering. If the NAPT follows REQ-8 in RFC 4787, then filtering will pass the responder packets since the destination address and port will be mapped and the source…

I still don't see how you avoid reducing everything to 1 NAT mapping on
the NAPT router, and thus with no extra entropy, as it seems multiple
different source IPs are racing with the regular endpoint NAT mapping
update. And if you change that behaviour to not update but keep the
old NAT mappings, you need to not only change the kernel behaviour,
you also need to signal whether this is supported through IKE, and
whether you want it activated. Eg you would need some _SUPPORTED notify
and then some USE_ notify.

Paul W

---

From: "Bottorff, Paul" <[email-masked]>
Date: Fri, 16 Jul 2021 20:50:23 +0000

Hi Paul:

Below,

Cheers,

Paul

-----Original Message-----
From: Paul Wouters [mailto:[email-masked]] 
Sent: Friday, July 16, 2021 12:46 PM
To: Bottorff, Paul <[email-masked]>
Cc: Tobias Brunner <[email-masked]>; Valery Smyslov <[email-masked]>; 'Tero Kivinen' <[email-masked]>; [email-masked]; 'IPsec' <[email-masked]>; Mahendra Maddur Puttaswamy <[email-masked]>; Shraddha Hegde <[email-masked]>; 徐小虎 <[email-masked]>
Subject: Re: [IPsec] draft-xu-ipsecme-esp-in-udp-lb-07

I am also very confused.

> The initiator will start with IKE. The IKE packets will not be LB and since a NAPT would be detected, the IKE packets would use source and destination port 4500.

Just to clarify, on the initiator those source/dest ports will be 4500, but on the responder the source port can be anything and the destination port will be 4500.

PB>No, on the responder the destination port is not 4500, but is the source port received from the initiator. The reason for this is the NAPT translated the source port of the initiator IKE packet and to reverse the mapping as the responder sends to the initiator it must use the port mapped by the NAPT as the destination. This port will be translated back to 4500 by the NAPT before it is delivered to the initiator behind the NAPT.

> For ESPinUDP with LB the source port will be used for entropy. This does not necessary mean the source port is random, however does mean the source port must cover a range of values sufficient to p…

I think you are saying "pick a random port and keep using it" and not "use a random port for each packet", right ?

PB>More precisely, pick a range of ports for use as entropy code points and use them by mapping the flows of the packets within the ESP to them. The source port picked is not really random since for LB to work properly we must retain flow integrity. Typically the port would be determined by a hash of the packet now encrypted with as an ESP. Also we can't choose a single port for entropy since the objective is to use the source port for spreading traffic over multiple paths. What I am saying is that for most applications we can choose (preferably contiguous) a port range (which might be as small as 2 ports or as large as 256 ports) which would be used for LB. The choise of these ports would then depend on the hash function applied to inside packet before ESP.

> At an IPSEC initiator (behind NAPT) using 
> draft-xu-ipsecme-esp-in-udp-lb will generate LB ESPinUDP packets with 
> the entropy value in the source port and a new destination port

Where does this "new destination port" come from? You only have a NAT mapping for port 500 and a NAT mapping for port 4500. Did a new destination port get negotiated over IKE? I don't see that specified in the draft. Or do you expect the non-NAT IPsec endpoint to auto-detect these packets to random ports based on encap payload and SPI ?

PB>The new source port is proposed by draft-xu-ipsecme-esp-in-udp-lb and is to be obtained from IANA. This is a point of disagreement between myself and the authors of draft-xu-ipsecme-esp-in-udp-lb which I describe in a little more detail as follows.

> , however the use of a new port to designate LB complicates the situation. Instead, if we consider both ends of the SA being configured for LB, then we can simply use the existing 4500 port to desi…

But when using only destination port 4500, then you run into the problem of NAT mapping updates. If you send a ESPinUDP packet to port 4500 from source port X, the peer will update its IPsec SA endpoint and start sending all packets to that single source port.
So any entropy gain is lost.

PB>We are in agreement here. What I am proposing is that this IPSEC behaviour (dynamic updating) needs to change to support LB through NAT. To support IPSEC LB through NAPT. For LB the IPsec SA endpoint needs to NOT update its end point, but instead use the end point established for IKE.

> Each IPSEC initiator LB packet with a different entropy value will result in a new mapping at the NAPT.

Yes, a new mapping, but the old mapping will be deleted. Is your draft saying we should _keep_ those mappings? If so, how would you ever detect a _rela_ IKE/ESP port remapping from the NAPT router between the endpoints?

PB>The mappings generated by ESPinUDP with LB don't need to be preserved, only the IKE mapping needs to be preserved. Since IKE used a different source port (4500) from the ESPinUDP LB packets these should appear at different non-overlapping translations.

> The normal behaviour of the responder (outside the NAPT) is to 
> transmit to the destination address and port of the last packet source 
> address and port (from the initiator)

Not without MOBIKE. And with MOBIKE this requires explicit signaling via the INFORMATIONAL IKE message containing an UPDATE_ADDRESS payload.

PB>I not certain we disagree since this is a matter of frame of reference. The initiator will always see the same destination port and addresses.

> however this is where we deviate for a LB implementation. Instead the responder (outside the NAPT) transmits all responses to the address and port established for IKE, however with an entropy value…

I don't see how this can properly work. It would be really helpful if you can explain this with ASCII art in your draft document.

PB>I have not put forward any draft on this topic, because my use case does not have any need for NAT. See draft-bottorff-ipsecme-mtdcuc-psec-lb. The only reason I bring it up is that NAT was raised as an issue in the previous email thread on draft-xu-ipsecme-esp-in-udp-lb. My only intent is to suggest that there probably is a way to solve NAPT traversal combined with LB.

> So to your question about NAPT filtering. If the NAPT follows REQ-8 in RFC 4787, then filtering will pass the responder packets since the destination address and port will be mapped and the source…

I still don't see how you avoid reducing everything to 1 NAT mapping on the NAPT router, and thus with no extra entropy, as it seems multiple different source IPs are racing with the regular endpoint NAT mapping update. And if you change that behaviour to not update but keep the old NAT mappings, you need to not only change the kernel behaviour, you also need to signal whether this is supported through IKE, and whether you want it activated. Eg you would need some _SUPPORTED notify and then some USE_ notify.

Paul W

---

From: Paul Wouters <[email-masked]>
Date: Thu, 22 Jul 2021 14:42:57 -0400 (EDT)

> Cheers,

The below did not clear things up for me. I think it would be useful to
describe things in the draft much more carefully and details with some
ascii art.

Paul