Subject: UDP source ports for HTTP/3 and QUIC

From: Mark Nottingham <[email-masked]>
Date: Thu, 15 Jul 2021 10:20:39 +1000

[ bringing this up on both lists because it's not yet clear what the right scope is ]

It's not uncommon for servers to block certain UDP source ports to avoid being overwhelmed by certain reflection attacks. In particular:

* 53 - DNS
* 123 - NTP
* 1900 - SSDP
* 5353 - mDNS
* 11211 - memcached

... among other candidates.

See, eg., <https://blog.cloudflare.com/reflections-on-reflections/>. This isn't done to avoid protocol vulnerabilities as such -- it's to avoid volumetric attacks (usually DDoS). 

If a client chooses source ports from the ephemeral port range, this shouldn't be an issue. However, some implementations (or deployments) extend the source port range "downwards" to avoid exhaustion:

* On Linux, it's pretty common to tune to extend the ephemeral port range, e.g., <https://ma.ttias.be/linux-increase-ip_local_port_range-tcp-port-range/>

* On some versions of Windows, ports as low as 1025 are used: <https://docs.microsoft.com/en-us/troubleshoot/windows-server/networking/service-overview-and-network-port-requirements>

* FreeBSD has used ports as low as 1024 in the past; their documentation is... confused right now. <https://github.com/freebsd/freebsd-src/blob/373ffc62c158e52cde86a5b934ab4a51307f9f2e/sys/netinet/in.h#L272> (I don't run a FreeBSD box to check this)

* NAT and CGNAT are a thing. <https://www.rfc-editor.org/rfc/rfc4787.html#section-4.2.1> recommends that source ports in the range 1024-65535 be reassigned within that range, which can cause collisions. 

Overall, the chance of collision (i.e., a client or a NAT choosing a blocked source port) is quite low, but at Internet scale it'll happen, leading to the client needing to open a new connection, thereby adding perceived latency. 

At a minimum, it seems like a good thing for client and NAT developers to be aware that some ports are likely to be blocked, so that they can avoid them if they care about perceived performance. This e-mail might be enough for that immediate purpose.

My question is whether there's a need for more coordination -- e.g., a very short RFC outlining such ports, to help bring this to folks' attention (especially in the NAT crowd)?

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 15 Jul 2021 11:29:37 +0200

On Thu, Jul 15, 2021 at 10:56:28AM +0200, Mikkel Fahnøe Jørgensen wrote:
> It is perhaps worth noting that due to QUIC (optionally) having unique
> connection identifiers, it is feasible to have many connections on the same
> source port. Therefore that could be a recommendation in cases where some
> source ports might be blocked.

I think that this is an excellent idea! The simple fact that this is
being discussed precisely is because the source port serves no purpose
here other than being compatible with UDP. So basically we could have
a recommendation that each application preferably uses a single socket
and source port for outgoing communication. This will also lower the
stress on source port allocation (and recycling) as well as the need
for file descriptors.

Willy

---

From: Stefan Eissing <[email-masked]>
Date: Thu, 15 Jul 2021 11:53:46 +0200

> Am 15.07.2021 um 11:29 schrieb Willy Tarreau <[email-masked]>:
> 

Yikes, all QUIC packets will potentially arrive a the same child listener. Let's go all the way and defined a fixed source port. ^^

- Stefan

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 15 Jul 2021 12:01:49 +0200

On Thu, Jul 15, 2021 at 11:53:46AM +0200, Stefan Eissing wrote:
> Yikes, all QUIC packets will potentially arrive a the same child listener.

That's a good point, indeed. But we could suggest that for low network
resource usage (i.e. browsers), reusing sockets that proved to work fine
is recommended.

> Let's go all the way and defined a fixed source port. ^^

:-)

Willy

---

From: David Schinazi <[email-masked]>
Date: Thu, 15 Jul 2021 10:10:20 -0700

Chrome doesn't plan on having multiple connections share a UDP source port.
In part because sharing would require the overhead of client connection
IDs, but mainly the fact that switching away from the
one-port-per-connection model we have today is work towards solving a
problem that we don't have.

Back to Mark's original email, I think there's value in writing an RFC to
document this behavior - that way clients can have code that requests a
different source port from the OS if it initially assigns one that is
likely to be blocked by servers. And while modifying NATs is a long game,
RFCs do tend to move the needle over time.

David

On Thu, Jul 15, 2021 at 6:40 AM Nick Banks <nibanks=
[email-masked]> wrote:

> You should definitely not make any assumptions around having unique source
> ports for QUIC connections. MsQuic already supports sharing the local port
> and is looking to make it a default behavior (for at least some scenarios)
> to avoid the fairly common "port exhaustion" problems we see with TCP. This
> doesn't necessarily mean all connections would be on the same source port,
> but only a few ports might be used for all connections.
>
> Thanks,
> - Nick
>
> Sent from Outlook <http://aka.ms/weboutlook>
> ------------------------------
> *From:* QUIC <[email-masked]> on behalf of Töma Gavrichenkov <
> [email-masked]>
> *Sent:* Thursday, July 15, 2021 6:36 AM
> *To:* Mikkel Fahnøe Jørgensen <[email-masked]>
> *Cc:* Mark Nottingham <[email-masked]>; IETF QUIC WG <[email-masked]>; HTTP
> Working Group <[email-masked]>
> *Subject:* Re: UDP source ports for HTTP/3 and QUIC
>
> Peace,
>
> On Thu, Jul 15, 2021, 11:57 AM Mikkel Fahnøe Jørgensen <[email-masked]>
> wrote:
>
> As others have pointed out, I would suspect an RFC with a port list would
> quickly become outdated.
>
>
> Speaking generally of lists of a content too dynamic and too host-specific
> to hardcode in RFCs, there once was a habit of putting them into DNS
> records.
>
> --
> Töma
>
>

---

From: Erik Nygren <[email-masked]>
Date: Thu, 15 Jul 2021 19:25:00 -0400

On Wed, Jul 14, 2021 at 8:21 PM Mark Nottingham <[email-masked]> wrote:

> It's not uncommon for servers to block certain UDP source ports to avoid
> being overwhelmed by certain reflection attacks. In particular:
>
> * 53 - DNS
> * 123 - NTP
> * 1900 - SSDP
> * 5353 - mDNS
> * 11211 - memcached
>
> See, eg., <https://blog.cloudflare.com/reflections-on-reflections/>. This
> isn't done to avoid protocol vulnerabilities as such -- it's to avoid
> volumetric attacks (usually DDoS).
>
>

Closely related, the Fetch spec's "bad port" list is fairly TCP-specific
and could likely
use additions for some of these.  I opened
https://github.com/whatwg/fetch/issues/1268
to track that.

    Erik

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 16 Jul 2021 10:27:20 +1000

Adding source ports to Fetch to complement their list of "bad" destination ports is interesting, but probably not sufficient -- NAT and CGNAT vendors tend not to read that document.

If we believe both that this should be written down and that more reflection-capable applications are going to emerge, we should probably put this information in a registry.

It strikes me that it could be done by adding a column to the port registry <https://www.iana.org/assignments/service-names-port-numbers/>. In other words, TSVWG. Thoughts?

Cheers,

> On 16 Jul 2021, at 9:25 am, Erik Nygren <[email-masked]> wrote:
> 
> 
> Closely related, the Fetch spec's "bad port" list is fairly TCP-specific and could likely
> use additions for some of these.  I opened https://github.com/whatwg/fetch/issues/1268
> to track that.
> 
>     Erik
> 
>

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 16 Jul 2021 03:40:10 +0200

Inline

On Thu, Jul 15, 2021 at 11:29:37AM +0200, Willy Tarreau wrote:
> here other than being compatible with UDP.

Huh ?? The UDP source port serves as the UDP destination port for
return packets. Otherwise how do you redirect the packet to he
right socket ? 

> So basically we could have
> and source port for outgoing communication. 

Sure, but different UDP source port (socket) for different applications.

> This will also lower the

Yes, especially for systems with many simultaneous connections,
likely servers/responders.

Unless scale is a problem its somewhat hard to say whats better or worse.
Using separate sockets/port-numbers even within a sinle app allows
to distinguish the connections easier with the usual suspsects such
as per-socket kernel diagnostics, network tools such a IPfix or the
like. If i where to implement a stack for clients i'd certainly
wanted to offer the option of whether to allocate separate sockets
(source ports) per connection or not - and see what users do with it.

Toerless

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 16 Jul 2021 08:06:44 +0200

> return packets. Otherwise how do you redirect the packet to he

That's exactly what I meant by "compatible with UDP", i.e. a datagram
oriented, bidirectional transport protocol. The source port is solely
used to know what socket to deliver the return traffic to.

> Sure, but different UDP source port (socket) for different applications.

Sure :-)

> Unless scale is a problem its somewhat hard to say whats better or worse.

Stefan made a good point about the problem that might result, with
inbound load balancing between multiple listeners (typically what's
achieved by L3 switches doing L3+L4 hash between multiple servers,
and operating systems hashing the source+destination port to pick a
different listening socket). Thus a suggestion might be to possibly
save resources by using a small amount of sockets, with "small" left
to the appreciation of the implementation.

> like.

Sure, but at the application level it does require to check the
connection ID anyway.

> If i where to implement a stack for clients i'd certainly

Yep, at least as a debug option to help with captures or to trace
file descriptors activities.

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 16 Jul 2021 06:34:32 +0000

--------

> to the appreciation of the implementation.

We should run the question "few or many UDP ports?" past some some
friendly 100G and 400G device driver maintainers.

The NICs I have been working with for the ESO ELT project all
included UDP ports in the hash they used to decide which CPU core
to deliver packets/interrupts to and we had to spread across UDP
ports to or all the traffic would hit one single core.

I dont know if QUIC has registered with the NIC designers and device
drivers writers yet, but given the opacity of QUIC packets, it is
very hard to see what else than the UDP port they can feed into
their hash, so I expect the answer to be "many".

But it would best to ask, and find out how important they think it
will be, and what "many" means for them.

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 16 Jul 2021 09:16:40 +0200

> their hash, so I expect the answer to be "many".

That's true but queues are not the only parameter, as using many
sockets also results in a massive cache miss ratio and many more
syscalls. Even with TCP it's quite visible that using a few hundred
connections tends to be incredibly more efficient at 100Gbps than
using tens of thousands.

Ideally we want all queues to be used with a good balance while
limiting the amount of operations to transport these data.

Probably that some wording evocating that possibility offered by
the protocol and a suggestion to take various factors into account
(L4 LB, NIC queues, listeners, syscalls etc) is sufficient to let
implementors ask the relevant people for advice in their context.

Willy

---

From: Stefan Eissing <[email-masked]>
Date: Fri, 16 Jul 2021 09:51:01 +0200

> Am 16.07.2021 um 09:16 schrieb Willy Tarreau <[email-masked]>:
> 

It seems a good idea to actively pull in people knowledgable in 
these implementation and hear their recommendations/expectations.

As far as my limited knowledge goes, having a QUIC application
share a single source port for many QUIC connections does not
pose any problems for servers.

The concern is more about NAT (and esp. CGNAT) to coalesce
many Source:address+port into a few NAT-address+ports,
using QUIC connection IDs. On the extreme, once these NATs
use connection IDs, they could theoretically live with a single
NAT port. That would not work well with any UDP load balancing.

The advice for QUIC aware NAT implementations could then be
something like (IANAUBE):
- distribute QUIC connections across a minimum of N ports
- The N ports would ideally be random or a complete interval
  (they should not all give the same number module a prime)
- distribute new NATed connections in some random fashion
  or round-robin across your port pool.

- Stefan

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 16 Jul 2021 23:53:11 +0200

> to the appreciation of the implementation.

That would be an example getting more resources on a load-balanced
server farm when you use multiple 5-tuple flows to the same
(cluster) server by reson of client UDP port. 

The same (more/different) resource allocation could also occur for
network bandwidth along the path when there is any type of per-flow
traffic operation, even if heuristically, such as attempts for
bandwidth fairness. Or even alternative paths choosen when there
is 5-tuple load-spliting along the path.

We have been around this block pre-HTTP2.0, using multiple
sockets to increase performance. But any HTTP2.0 fixes did
not change the network or server-farm aspects.

> Sure, but at the application level it does require to check the

This again would be on the network side.

Cheers
    Toerless

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 17 Jul 2021 10:28:09 +1000

I appreciate the enthusiasm for the tangential here :) but to return the original topic -- I realised that this is obliquely discussed in the applicability draft:
  https://quicwg.org/ops-drafts/draft-ietf-quic-applicability.html#name-port-selection-and-applicat

So at a minimum, it seems like we should expand upon that text to make this issue more clear. I'll try to start a PR.

If I read the thread correctly, people seem to think (somewhat pessimistically, but realistically) that protocols susceptible to reflection will continue to be created, so hardcoding a list into the RFC isn't workable.

Instead, I'm currently thinking the best approach will be to:

1. Expand the applicability document as per above, using examples from the list I gave, since they're already pervasive
2. Start a discussion in the TSVWG about the possibility of adding a new column to the port registry to capture this information 

Make sense?