Subject: 5-tuple uniquess

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sun, 6 Aug 2017 06:18:21 -0400

Have argued before as comments to various issues, but I’ll try once more on
the list:

Please consider making QUIC independent of the underlying transport,
including assumptions on the uniqueness of the UDP protocol.

1. If a http server is able to receive 100K connections, it will not be
able to forward them to the same backend service because there are not
enough ephemeral port numbers to create a reverse proxy connection per
incoming connection.

2. It would be fragile to port to QUIC to alternative transports because of
implied assumptions on the connectivity. For example on an I2C bus.

3. If a client receives its own initial connection id as a response from a
server, even if the server has decided on a new connection id, the client
can trivially identify the connection with a simple lookup, and trivially
reject other connections. This requires that the connection id is visible
(recent discussions on encryption of cleartext might challenge this, but
the initial connection id could be part of authenticated data).

4. On connection migration there would need to be special considerations,
but again, handling this up front would simplify 2.

The obvious drawback of this is the connection id must remain in clear text
which takes space and might cause some privacy concerns. Regarding privacy,
I don’t think it is more or less secret than a unique 5-tuple for an
on-path interceptor. As to size, this is unfortunate. If designed
carefully, it may be possible to omit the connection id as it is today for
scenarios where the 5-tuple is sufficient. For example, public network
might use 5-tuple without explicit connection id, reverse proxy might not.
But overall, I would prefer the simplicity of always having the connection
id.

Kind Regards,
Mikkel Fahnøe Jørgensen

---

From: Ryan Hamilton <[email-masked]>
Date: Sun, 6 Aug 2017 07:16:59 -0700

Is there particular language in the transport doc which you would like to
change? For example, I believe the current document specifies that the
connection id is required unless omit_connection_id is sent by the receiver
during the handshake. Does this satisfy your requirement about the
connection id being visible? The docs also talk about the underlying 5
tuple changing during the connection, which sounds like what you're asking
for. So it would be good to have a bit more detail on what you would like
to see change in the spec.

Cheers,

Ryan

On Sun, Aug 6, 2017 at 3:18 AM, Mikkel Fahnøe Jørgensen <[email-masked]>
wrote:

> Have argued before as comments to various issues, but I’ll try once more
> on the list:
>
> 2. It would be fragile to port to QUIC to alternative transports because
> of implied assumptions on the connectivity. For example on an I2C bus.
>
> The obvious drawback of this is the connection id must remain in clear
> text which takes space and might cause some privacy concerns. Regarding
> privacy, I don’t think it is more or less secret than a unique 5-tuple for
> an on-path interceptor. As to size, this is unfortunate. If designed

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sun, 6 Aug 2017 07:22:13 -0700

Hi Ryan,

I can dig down in more detail if there is interest. Meanwhile, some
examples where the transport document is not compatible with my request is:
https://quicwg.github.io/base-drafts/draft-ietf-quic-transport.html#packet-version

Kind Regards,
Mikkel Fahnøe Jørgensen

On 6 August 2017 at 16.17.01, Ryan Hamilton ([email-masked]) wrote:

Cheers,

Ryan

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sun, 6 Aug 2017 07:22:53 -0700

Mail sent prematurely, will follow up

Kind Regards,
Mikkel Fahnøe Jørgensen

On 6 August 2017 at 16.22.13, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

Hi Ryan,

Kind Regards,
Mikkel Fahnøe Jørgensen

Cheers,

Ryan

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sun, 6 Aug 2017 07:43:34 -0700

Hi Ryan,

The version negotiation packet is ok - it reflects the client version
https://quicwg.github.io/base-drafts/draft-ietf-quic-transport.html#packet-version

But
https://quicwg.github.io/base-drafts/draft-ietf-quic-transport.html#packet-server-cleartext
"The connection ID field in a Server Cleartext packet contains a connection
ID that is chosen by the server (see Section 5.6
<https://quicwg.github.io/base-drafts/draft-ietf-quic-transport.html#connection-id>
)."
This breaks linkage to the original client connection id so the client
cannot tell which connection the response belongs to unless looking into
5-tuple, or testing all recent connections against AEAD signature
(depending on where this lands).

I have not covered all other possible server responses, but similar
concerns apply.
It is not a major change, but currently it is not possible.

Another issue is hostile response to initial connection setup, and plain
errors - how to you reply with an error. Having the connection id from the
client makes it reasonably easy to deal with - similar to version
negotiation. Alternatively you have to rely on 5-tuples. This requires
careful analysis of both the transport doc and possibly TLS and thins are
not fully settled. Stateless reset is related to this. I will not argue
strongly on this because since I last looked at it, new AEAD encryption is
being discussed, and I have not analysed this in detail.

Then there is connection migration. This also requires careful analysis.
I’m not sure if it currently covers my request.

I’m sure there are several other places with implicit assumptions that I am
not able to list off memory.

The (my) basic rule is the any change in connection ID should refer to the
previous ID without relying on external state such as 5-tuples or TLS
extension state (which requires crypto state to access and complicates
things). Also, due concerns of privacy concerns this linkage between
connection ID’s should only be possible by peers, excepts for the initial
client to server connection transition that is also visible to any party
on-path.

In addition to the above encoding concerns, it may also be helpful to
explicitly state that the same UDP port may be used to initiate multiple
independent QUIC connection to the same server UDP port when connection ID
has been negotiated to be present - or further detail this possibility as
something that can be negotiated.

Some other points I forget to mention regarding benefits of non-unique
tuples:

5) no interference with OS is necessary in order to establish a new
connection to an existing endpoint because no new ephemeral port is needed.
This may also simplify clean up and state management, especially server to
server.

6) Netmap and similar high speed interfaces would be simpler without having
to deal with ephemeral port allocation, especially server to server where
both end points can reference each other, and where it might be random
which endpoint actually initiates to the connection (as in Cord / Kademlia
style overlay networks)

Kind Regards,
Mikkel Fahnøe Jørgensen

---

From: Ian Swett <[email-masked]>
Date: Tue, 8 Aug 2017 19:00:47 -0400

You're right, ephemeral port exhaustion is a very real potential issue, and
I think we should ensure multiple QUIC connections can run over the same
5-tuple and we don't specify QUIC in a way that precludes that.

I believe you're correct that the current Server Cleartext makes this
impossible if the server changes the connection ID.  Possibly it should
specify the client's connection ID in the packet header and then the new
server connection ID in the transport parameters?
https://github.com/quicwg/base-drafts/issues/714

The text is also missing some details on how changing the connection ID
should work for Stateless Retry, see https://github.com/quicwg/
base-drafts/issues/713, so I'd expect some improvements in the near future.

On Sun, Aug 6, 2017 at 10:43 AM, Mikkel Fahnøe Jørgensen <[email-masked]

> "The connection ID field in a Server Cleartext packet contains a
> connection ID that is chosen by the server (see Section 5.6
> I’m sure there are several other places with implicit assumptions that I
> am not able to list off memory.
>
> 6) Netmap and similar high speed interfaces would be simpler without
> having to deal with ephemeral port allocation, especially server to server
> where both end points can reference each other, and where it might be
> random which endpoint actually initiates to the connection (as in Cord /
> Kademlia style overlay networks)
>
> Kind Regards,
> Mikkel Fahnøe Jørgensen
>
>
>

---

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 9 Aug 2017 17:03:29 +0000

Can someone elaborate on “Possibly it should specify the client's connection ID in the packet header and then the new server connection ID in the transport parameters?”

Is that only for “Server Cleartext”? What about connections that start out with 0RTT packets – how would the NAT know which flow Server’s 1RTT reply belongs to (if Server changes the ConenctionID)?

Maybe relying on a 5-tuple is not such a bad idea after all?  A NAT does not need a different ephemeral port for every outgoing connection.  It only needs a different ephemeral port for outgoing connections to the same IP.

Even then, if the NAT is willing to do something special for QUIC, it can be a much simpler thing than examine QUIC transport params.  The NAT can simply reserve a single ephemeral port for all 1RTT QUIC packets.  That is, all outgoing cleartext and 0rtt QUIC packets would require a distinct 5-tuple, but once the connection migrates to 1RTT, the ephemeral port it used can be freed, and all 1RTT packets sourced from the reserved QIUC port.  That would work, since all 1RTT packets (both from client and server) would be using Server-selection ConnectionID.

  *   Igor

From: Ian Swett [mailto:[email-masked]]
Sent: Tuesday, August 08, 2017 7:01 PM
To: Mikkel Fahnøe Jørgensen <[email-masked]>
Cc: IETF QUIC WG <[email-masked]>; Ryan Hamilton <[email-masked]>
Subject: Re: 5-tuple uniquess

You're right, ephemeral port exhaustion is a very real potential issue, and I think we should ensure multiple QUIC connections can run over the same 5-tuple and we don't specify QUIC in a way that precludes that.

I believe you're correct that the current Server Cleartext makes this impossible if the server changes the connection ID.  Possibly it should specify the client's connection ID in the packet header and then the new server connection ID in the transport parameters?  https://github.com/quicwg/base-drafts/issues/714<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_quicwg_base-2Ddrafts_issues_714&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=mXkiKKh_FkRO-MANo0WqyEn2M1LkcNJFMOkgYdp4_q8&s=-DtvtkmD_lZJgEtjvugx4OxskRAUmqpy5NgCgNzahGw&e=>

The text is also missing some details on how changing the connection ID should work for Stateless Retry, see https://github.com/quicwg/base-drafts/issues/713<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_quicwg_base-2Ddrafts_issues_713&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=mXkiKKh_FkRO-MANo0WqyEn2M1LkcNJFMOkgYdp4_q8&s=cV_5AUJPoW46SQHJT1Niy_zuRCx0E7x288-L46-zhdY&e=>, so I'd expect some improvements in the near future.

On Sun, Aug 6, 2017 at 10:43 AM, Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>> wrote:

Hi Ryan,

I can dig down in more detail if there is interest. Meanwhile, some examples where the transport document is not compatible with my request is:

The version negotiation packet is ok - it reflects the client version
https://quicwg.github.io/base-drafts/draft-ietf-quic-transport.html#packet-version<https://urldefense.proofpoint.com/v2/url?u=https-3A__quicwg.github.io_base-2Ddrafts_draft-2Dietf-2Dquic-2Dtransport.html-23packet-2Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=mXkiKKh_FkRO-MANo0WqyEn2M1LkcNJFMOkgYdp4_q8&s=gnz8PJ4RQMq6w-_WFa1RHYMTgpgtgImakR1qZwm_uTk&e=>

But
https://quicwg.github.io/base-drafts/draft-ietf-quic-transport.html#packet-server-cleartext<https://urldefense.proofpoint.com/v2/url?u=https-3A__quicwg.github.io_base-2Ddrafts_draft-2Dietf-2Dquic-2Dtransport.html-23packet-2Dserver-2Dcleartext&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=mXkiKKh_FkRO-MANo0WqyEn2M1LkcNJFMOkgYdp4_q8&s=DjWdo8Yhu8t7h1zqd7BIxk9_TLU36AxmdZ0DmhZn6vE&e=>
"The connection ID field in a Server Cleartext packet contains a connection ID that is chosen by the server (see Section 5.6<https://urldefense.proofpoint.com/v2/url?u=https-3A__quicwg.github.io_base-2Ddrafts_draft-2Dietf-2Dquic-2Dtransport.html-23connection-2Did&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=mXkiKKh_FkRO-MANo0WqyEn2M1LkcNJFMOkgYdp4_q8&s=5SH6u7AaH_arxGY3yEeqwyhio8dTX9SMQWFc2TIlhY0&e=>)."
This breaks linkage to the original client connection id so the client cannot tell which connection the response belongs to unless looking into 5-tuple, or testing all recent connections against AEAD signature (depending on where this lands).

I have not covered all other possible server responses, but similar concerns apply.
It is not a major change, but currently it is not possible.

Another issue is hostile response to initial connection setup, and plain errors - how to you reply with an error. Having the connection id from the client makes it reasonably easy to deal with - similar to version negotiation. Alternatively you have to rely on 5-tuples. This requires careful analysis of both the transport doc and possibly TLS and thins are not fully settled. Stateless reset is related to this. I will not argue strongly on this because since I last looked at it, new AEAD encryption is being discussed, and I have not analysed this in detail.

Then there is connection migration. This also requires careful analysis. I’m not sure if it currently covers my request.

I’m sure there are several other places with implicit assumptions that I am not able to list off memory.

The (my) basic rule is the any change in connection ID should refer to the previous ID without relying on external state such as 5-tuples or TLS extension state (which requires crypto state to access and complicates things). Also, due concerns of privacy concerns this linkage between connection ID’s should only be possible by peers, excepts for the initial client to server connection transition that is also visible to any party on-path.

In addition to the above encoding concerns, it may also be helpful to explicitly state that the same UDP port may be used to initiate multiple independent QUIC connection to the same server UDP port when connection ID has been negotiated to be present - or further detail this possibility as something that can be negotiated.

Some other points I forget to mention regarding benefits of non-unique tuples:

5) no interference with OS is necessary in order to establish a new connection to an existing endpoint because no new ephemeral port is needed. This may also simplify clean up and state management, especially server to server.

6) Netmap and similar high speed interfaces would be simpler without having to deal with ephemeral port allocation, especially server to server where both end points can reference each other, and where it might be random which endpoint actually initiates to the connection (as in Cord / Kademlia style overlay networks)

Kind Regards,
Mikkel Fahnøe Jørgensen

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 9 Aug 2017 10:33:21 -0700

This may not address all your concerns, but I think perhaps you are looking
at the problem from the wrong end with respect to ephemeral port exhaustion:

In a typical NAT scenario you have many clients reaching for one server,
each client likely passing a different NAT.
Hence each connection will be a unique 5-tuple, even if QUIC itself is not
making any effort to achieve this.

Now, if one of these clients creates two connections to the same server and
chooses to reuse the outbound UDP port number, say reaching for two
different images, then the NAT might think that the two connections are the
same connection. And in some sense it is - not the same QUIC connection,
but the same operational context. Thus, the two connections helps reduce
NAT overhead and also cooperates in keeping the NAT hole open.

On the other end, at the server, there is a separate problem: The server
(S) does not have the requested images. It reaches to one of a few possible
backend servers (B1, B2). Since the server S is not attempting to be
clever, it creates a new QUIC connection from S to B1 or B2 for each
inbound client connection. There may or may not be any NAT involved here,
but the server S is only assigned a single IP address for outbound requests
and cannot create more concurrent connections that the number of ephemeral
ports + safety margin permits, that is, unless it chooses to reuse outbound
port numbers which solves the problem of port exhaustion.

If, on the other hand, a QUIC client were co-located with other services,
it might not be able to receive the correct packets when these different
services compete for port numbers. But here it suffice that they either
agree to use different ports (like PING vs. DNS), or each service allocate
a single ephemeral port for its service via UDP socket creation. In either
case the NAT would route packets to the correct service by relying on
5-tuples even if QUIC itself does not.

So, going further - what if several hosts use the same IP address, as is
the case with load balancers? Again, 5-tuples are used to bind connections
to each unique host, only with fewer UDP ports there is much less state to
maintain. The middleware only need to route traffic to the correct
endpoint, not distinguish between logical QUIC connections that happen
between the same two entities.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 9 August 2017 at 19.03.35, Lubashev, Igor ([email-masked]) wrote:

Can someone elaborate on “Possibly it should specify the client's
connection ID in the packet header and then the new server connection ID in
the transport parameters?”

Is that only for “Server Cleartext”? What about connections that start out
with 0RTT packets – how would the NAT know which flow Server’s 1RTT reply
belongs to (if Server changes the ConenctionID)?

Maybe relying on a 5-tuple is not such a bad idea after all?  A NAT does
not need a different ephemeral port for every outgoing connection.  It only
needs a different ephemeral port for outgoing connections to the same IP.

Even then, if the NAT is willing to do something special for QUIC, it can
be a much simpler thing than examine QUIC transport params.  The NAT can
simply reserve a single ephemeral port for all 1RTT QUIC packets.  That is,
all outgoing cleartext and 0rtt QUIC packets would require a distinct
5-tuple, but once the connection migrates to 1RTT, the ephemeral port it
used can be freed, and all 1RTT packets sourced from the reserved QIUC
port.  That would work, since all 1RTT packets (both from client and
server) would be using Server-selection ConnectionID.

   - Igor

*From:* Ian Swett [mailto:[email-masked]]
*Sent:* Tuesday, August 08, 2017 7:01 PM
*To:* Mikkel Fahnøe Jørgensen <[email-masked]>
*Cc:* IETF QUIC WG <[email-masked]>; Ryan Hamilton <[email-masked]>
*Subject:* Re: 5-tuple uniquess

<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_quicwg_base-2Ddrafts_issues_714&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=mXkiKKh_FkRO-MANo0WqyEn2M1LkcNJFMOkgYdp4_q8&s=-DtvtkmD_lZJgEtjvugx4OxskRAUmqpy5NgCgNzahGw&e=>

The text is also missing some details on how changing the connection ID
should work for Stateless Retry, see
https://github.com/quicwg/base-drafts/issues/713
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_quicwg_base-2Ddrafts_issues_713&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=mXkiKKh_FkRO-MANo0WqyEn2M1LkcNJFMOkgYdp4_q8&s=cV_5AUJPoW46SQHJT1Niy_zuRCx0E7x288-L46-zhdY&e=>,
so I'd expect some improvements in the near future.

On Sun, Aug 6, 2017 at 10:43 AM, Mikkel Fahnøe Jørgensen <[email-masked]>
wrote:

Hi Ryan,

https://quicwg.github.io/base-drafts/draft-ietf-quic-transport.html#packet-version
<https://urldefense.proofpoint.com/v2/url?u=https-3A__quicwg.github.io_base-2Ddrafts_draft-2Dietf-2Dquic-2Dtransport.html-23packet-2Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=mXkiKKh_FkRO-MANo0WqyEn2M1LkcNJFMOkgYdp4_q8&s=gnz8PJ4RQMq6w-_WFa1RHYMTgpgtgImakR1qZwm_uTk&e=>

But

https://quicwg.github.io/base-drafts/draft-ietf-quic-transport.html#packet-server-cleartext
<https://urldefense.proofpoint.com/v2/url?u=https-3A__quicwg.github.io_base-2Ddrafts_draft-2Dietf-2Dquic-2Dtransport.html-23packet-2Dserver-2Dcleartext&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=mXkiKKh_FkRO-MANo0WqyEn2M1LkcNJFMOkgYdp4_q8&s=DjWdo8Yhu8t7h1zqd7BIxk9_TLU36AxmdZ0DmhZn6vE&e=>

"The connection ID field in a Server Cleartext packet contains a connection
ID that is chosen by the server (see Section 5.6
<https://urldefense.proofpoint.com/v2/url?u=https-3A__quicwg.github.io_base-2Ddrafts_draft-2Dietf-2Dquic-2Dtransport.html-23connection-2Did&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=mXkiKKh_FkRO-MANo0WqyEn2M1LkcNJFMOkgYdp4_q8&s=5SH6u7AaH_arxGY3yEeqwyhio8dTX9SMQWFc2TIlhY0&e=>
)."

Kind Regards,

Mikkel Fahnøe Jørgensen

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 10 Aug 2017 03:09:32 +0000

I see “port reuse” as an instance of a generic “NAT rebinding” that QUIC is built for.  We’ve maintained previously, however, that QUIC tolerance to “connection migration / NAT rebinding” during connection establishment is out of scope.  This proposal would seem to revisit that scope.

You may indeed want to open multiple concurrent connections from S to B, hoping to spread your connections among multiple servers behind B.  But it may actually be counter-productive to use a single 5-tuple, if B is behind an ECMP router acting as a load balancer.  Moreover, this will also defeat RSS/RPS/RFS<https://www.kernel.org/doc/Documentation/networking/scaling.txt> load balancing mechanisms within servers.  Be aware.

  *   Igor

From: Mikkel Fahnøe Jørgensen [mailto:[email-masked]]
Sent: Wednesday, August 09, 2017 1:33 PM
To: Lubashev, Igor <[email-masked]>; Ian Swett <[email-masked]>
Cc: Ryan Hamilton <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: RE: 5-tuple uniquess

This may not address all your concerns, but I think perhaps you are looking at the problem from the wrong end with respect to ephemeral port exhaustion:

In a typical NAT scenario you have many clients reaching for one server, each client likely passing a different NAT.
Hence each connection will be a unique 5-tuple, even if QUIC itself is not making any effort to achieve this.

Now, if one of these clients creates two connections to the same server and chooses to reuse the outbound UDP port number, say reaching for two different images, then the NAT might think that the two connections are the same connection. And in some sense it is - not the same QUIC connection, but the same operational context. Thus, the two connections helps reduce NAT overhead and also cooperates in keeping the NAT hole open.

On the other end, at the server, there is a separate problem: The server (S) does not have the requested images. It reaches to one of a few possible backend servers (B1, B2). Since the server S is not attempting to be clever, it creates a new QUIC connection from S to B1 or B2 for each inbound client connection. There may or may not be any NAT involved here, but the server S is only assigned a single IP address for outbound requests and cannot create more concurrent connections that the number of ephemeral ports + safety margin permits, that is, unless it chooses to reuse outbound port numbers which solves the problem of port exhaustion.

If, on the other hand, a QUIC client were co-located with other services, it might not be able to receive the correct packets when these different services compete for port numbers. But here it suffice that they either agree to use different ports (like PING vs. DNS), or each service allocate a single ephemeral port for its service via UDP socket creation. In either case the NAT would route packets to the correct service by relying on 5-tuples even if QUIC itself does not.

So, going further - what if several hosts use the same IP address, as is the case with load balancers? Again, 5-tuples are used to bind connections to each unique host, only with fewer UDP ports there is much less state to maintain. The middleware only need to route traffic to the correct endpoint, not distinguish between logical QUIC connections that happen between the same two entities.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 9 August 2017 at 19.03.35, Lubashev, Igor ([email-masked]<mailto:[email-masked]>) wrote:
Can someone elaborate on “Possibly it should specify the client's connection ID in the packet header and then the new server connection ID in the transport parameters?”

  *   Igor

From: Ian Swett [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Tuesday, August 08, 2017 7:01 PM
To: Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>>
Cc: IETF QUIC WG <[email-masked]<mailto:[email-masked]>>; Ryan Hamilton <[email-masked]<mailto:[email-masked]>>
Subject: Re: 5-tuple uniquess

Hi Ryan,

But

Kind Regards,
Mikkel Fahnøe Jørgensen

---

From: Ian Swett <[email-masked]>
Date: Wed, 9 Aug 2017 23:33:55 -0400

In Prague, we decided that the 5-tuple couldn't change during the
handshake, but we didn't say that a 5-tuple indicated a single QUIC
connection.

For IPv6, a flow label can be used to spread load.  From my perspective,
the goal here is to allow multiple QUIC connections to share a 5-tuple, not
to recommend it for all use cases.  For example, it may make sense to
preclude it for HTTP over QUIC clients?

On Wed, Aug 9, 2017 at 11:09 PM, Lubashev, Igor <[email-masked]> wrote:

> I see “port reuse” as an instance of a generic “NAT rebinding” that QUIC
> is built for.  We’ve maintained previously, however, that QUIC tolerance to
> “connection migration / NAT rebinding” during connection establishment is
> out of scope.  This proposal would seem to revisit that scope.
>
>
>
> You may indeed want to open multiple concurrent connections from S to B,
> hoping to spread your connections among multiple servers behind B.  But it
> may actually be counter-productive to use a single 5-tuple, if B is behind
> an ECMP router acting as a load balancer.  Moreover, this will also defeat
> RSS/RPS/RFS
> <https://www.kernel.org/doc/Documentation/networking/scaling.txt> load
> balancing mechanisms within servers.  Be aware.
>
>
>
>    - Igor
>
>
>
>
>
>
>
> *From:* Mikkel Fahnøe Jørgensen [mailto:[email-masked]]
> *Sent:* Wednesday, August 09, 2017 1:33 PM
> *To:* Lubashev, Igor <[email-masked]>; Ian Swett <[email-masked]
> >
> *Cc:* Ryan Hamilton <[email-masked]>; IETF QUIC WG <[email-masked]>
> *Subject:* RE: 5-tuple uniquess
>
>
>
> This may not address all your concerns, but I think perhaps you are
> looking at the problem from the wrong end with respect to ephemeral port
> exhaustion:
>
>
>
> Now, if one of these clients creates two connections to the same server
> and chooses to reuse the outbound UDP port number, say reaching for two
> You're right, ephemeral port exhaustion is a very real potential issue,
> and I think we should ensure multiple QUIC connections can run over the
> same 5-tuple and we don't specify QUIC in a way that precludes that.
>
>
>
> base-drafts/issues/713
> On Sun, Aug 6, 2017 at 10:43 AM, Mikkel Fahnøe Jørgensen <
> [email-masked]> wrote:
>
>
>
> Hi Ryan,
>
>
>

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 10 Aug 2017 04:03:50 +0000

I am not particularly against 5-tuple-sharing, if it can be made to work.  But allowing such scenarios is effectively making the 5-tuples unusable for connection identification at any point.  Hence, I do not see what value “5-tuple cannot change” assertion has anymore.  As long as a packet somehow get to an endpoint, only connection ids are used to identify the flows.

Yes, flow label would work for ipv6.

From: Ian Swett [mailto:[email-masked]]
Sent: Wednesday, August 09, 2017 11:34 PM
To: Lubashev, Igor <[email-masked]>
Cc: Mikkel Fahnøe Jørgensen <[email-masked]>; Ryan Hamilton <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: 5-tuple uniquess

In Prague, we decided that the 5-tuple couldn't change during the handshake, but we didn't say that a 5-tuple indicated a single QUIC connection.

For IPv6, a flow label can be used to spread load.  From my perspective, the goal here is to allow multiple QUIC connections to share a 5-tuple, not to recommend it for all use cases.  For example, it may make sense to preclude it for HTTP over QUIC clients?

On Wed, Aug 9, 2017 at 11:09 PM, Lubashev, Igor <[email-masked]<mailto:[email-masked]>> wrote:
I see “port reuse” as an instance of a generic “NAT rebinding” that QUIC is built for.  We’ve maintained previously, however, that QUIC tolerance to “connection migration / NAT rebinding” during connection establishment is out of scope.  This proposal would seem to revisit that scope.

You may indeed want to open multiple concurrent connections from S to B, hoping to spread your connections among multiple servers behind B.  But it may actually be counter-productive to use a single 5-tuple, if B is behind an ECMP router acting as a load balancer.  Moreover, this will also defeat RSS/RPS/RFS<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.kernel.org_doc_Documentation_networking_scaling.txt&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=KUNIt2ar4Cclkpvbx3rSn-aNPr6BlUcFKurjrSGlWAs&s=XAy9X4IKcMEReLT7qu7BocLWBA9IZ9BJtIODwj5qtM8&e=> load balancing mechanisms within servers.  Be aware.

  *   Igor

From: Mikkel Fahnøe Jørgensen [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Wednesday, August 09, 2017 1:33 PM
To: Lubashev, Igor <[email-masked]<mailto:[email-masked]>>; Ian Swett <[email-masked]<mailto:[email-masked]>>
Cc: Ryan Hamilton <[email-masked]<mailto:[email-masked]>>; IETF QUIC WG <[email-masked]<mailto:[email-masked]>>
Subject: RE: 5-tuple uniquess

Kind Regards,
Mikkel Fahnøe Jørgensen

  *   Igor

Hi Ryan,

But

Kind Regards,
Mikkel Fahnøe Jørgensen

---

From: Ian Swett <[email-masked]>
Date: Thu, 10 Aug 2017 00:07:50 -0400

Ensuring the 5-tuple cannot change during the handshake is useful to
prevent a variety of off-path attacks, as well as a handshake
simplification.

On Thu, Aug 10, 2017 at 12:03 AM, Lubashev, Igor <[email-masked]>
wrote:

> I am not particularly against 5-tuple-sharing, if it can be made to work.
> But allowing such scenarios is effectively making the 5-tuples unusable for
> connection identification at any point.  Hence, I do not see what value
> “5-tuple cannot change” assertion has anymore.  As long as a packet somehow
> get to an endpoint, only connection ids are used to identify the flows.
>
>
>
> *Sent:* Wednesday, August 09, 2017 11:34 PM
> *To:* Lubashev, Igor <[email-masked]>
> *Cc:* Mikkel Fahnøe Jørgensen <[email-masked]>; Ryan Hamilton <
> [email-masked]>; IETF QUIC WG <[email-masked]>
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.kernel.org_doc_Documentation_networking_scaling.txt&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=KUNIt2ar4Cclkpvbx3rSn-aNPr6BlUcFKurjrSGlWAs&s=XAy9X4IKcMEReLT7qu7BocLWBA9IZ9BJtIODwj5qtM8&e=>
> load balancing mechanisms within servers.  Be aware.
>
>
>
>    - Igor
>
>
>
>
>
>
>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 9 Aug 2017 22:33:29 -0700

On HTTP clients being excluding from port-reuse - it might make sense for
browser clients but not necessarily for reverse proxy clients, and thus
most programming language standard libraries.

On RFS (flow steering) - CloudFlare made a contribution to Netmap so they
could steer some traffic to Netmap while the remaining traffic landed on
the Linux TCP stack.
https://blog.cloudflare.com/single-rx-queue-kernel-bypass-with-netmap/
In this case you could pro-actively use this to partially take over the
network traffic - or - an application could internally manage multiple core
loads by actively managing port allocations. So yes, there is a concern if
not used with care, but it also opens up opportunities.

On backend routing - it may be a good idea to allocate port numbers from a
limited sized pool avoiding both extremes.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 10 August 2017 at 06.08.11, Ian Swett ([email-masked]) wrote:

---

From: joel jaeggli <[email-masked]>
Date: Sat, 12 Aug 2017 22:14:53 -0700

On 8/9/17 20:33, Ian Swett wrote:
> the goal here is to allow multiple QUIC connections to share a 5-tuple,
> not to recommend it for all use cases.  For example, it may make sense
> to preclude it for HTTP over QUIC clients?

Flow labels are known to change during handshakes...

Source / dest and flow label is not at lot of entropy , esp if the
latter happens to be zero.

>     I see “port reuse” as an instance of a generic “NAT rebinding” that
>     QUIC is built for.  We’ve maintained previously, however, that QUIC
>     tolerance to “connection migration / NAT rebinding” during
>     connection establishment is out of scope.  This proposal would seem
>     to revisit that scope.____
> 
>     __ __
> 
>     You may indeed want to open multiple concurrent connections from S
>     to B, hoping to spread your connections among multiple servers
>     behind B.  But it may actually be counter-productive to use a single
>     5-tuple, if B is behind an ECMP router acting as a load balancer. 
>     Moreover, this will also defeat RSS/RPS/RFS
>     <https://www.kernel.org/doc/Documentation/networking/scaling.txt>
>     load balancing mechanisms within servers.  Be aware.____
> 
>     __ __
> 
>       * Igor____
> 
>     __ __
> 
>     __ __
> 
>     __ __
> 
>     *From:*Mikkel Fahnøe Jørgensen [mailto:[email-masked]
>     <mailto:[email-masked]>]
>     *To:* Lubashev, Igor <[email-masked]
>     <mailto:[email-masked]>>; Ian Swett <[email-masked]
>     <mailto:[email-masked]>>
>     *Cc:* Ryan Hamilton <[email-masked] <mailto:[email-masked]>>; IETF
>     QUIC WG <[email-masked] <mailto:[email-masked]>>
>     *Subject:* RE: 5-tuple uniquess____
> 
>     __ __
> 
>     looking at the problem from the wrong end with respect to ephemeral
>     port exhaustion:____
> 
>     __ __
> 
>     In a typical NAT scenario you have many clients reaching for one
>     server, each client likely passing a different NAT.____
> 
>     Hence each connection will be a unique 5-tuple, even if QUIC itself
>     is not making any effort to achieve this.____
> 
>     __ __
> 
>     Now, if one of these clients creates two connections to the same
>     server and chooses to reuse the outbound UDP port number, say
>     reaching for two different images, then the NAT might think that the
>     two connections are the same connection. And in some sense it is -
>     not the same QUIC connection, but the same operational context.
>     Thus, the two connections helps reduce NAT overhead and also
>     cooperates in keeping the NAT hole open.____
> 
>     __ __
> 
>     On the other end, at the server, there is a separate problem: The
>     server (S) does not have the requested images. It reaches to one of
>     a few possible backend servers (B1, B2). Since the server S is not
>     attempting to be clever, it creates a new QUIC connection from S to
>     B1 or B2 for each inbound client connection. There may or may not be
>     any NAT involved here, but the server S is only assigned a single IP
>     address for outbound requests and cannot create more concurrent
>     connections that the number of ephemeral ports + safety margin
>     permits, that is, unless it chooses to reuse outbound port numbers
>     which solves the problem of port exhaustion.____
> 
>     __ __
> 
>     If, on the other hand, a QUIC client were co-located with other
>     services, it might not be able to receive the correct packets when
>     these different services compete for port numbers. But here it
>     suffice that they either agree to use different ports (like PING vs.
>     DNS), or each service allocate a single ephemeral port for its
>     service via UDP socket creation. In either case the NAT would route
>     packets to the correct service by relying on 5-tuples even if QUIC
>     itself does not.____
> 
>     __ __
> 
>     So, going further - what if several hosts use the same IP address,
>     as is the case with load balancers? Again, 5-tuples are used to bind
>     connections to each unique host, only with fewer UDP ports there is
>     much less state to maintain. The middleware only need to route
>     traffic to the correct endpoint, not distinguish between logical
>     QUIC connections that happen between the same two entities.____
> 
>     __ __
> 
>     Kind Regards,____
> 
>     Mikkel Fahnøe Jørgensen____
> 
>     __ __
> 
>     On 9 August 2017 at 19.03.35, Lubashev, Igor ([email-masked]
>     <mailto:[email-masked]>) wrote:____
> 
>         Can someone elaborate on “Possibly it should specify the
>         client's connection ID in the packet header and then the new
>         server connection ID in the transport parameters?”____
> 
>          ____
> 
>         Is that only for “Server Cleartext”? What about connections that
>         start out with 0RTT packets – how would the NAT know which flow
>         Server’s 1RTT reply belongs to (if Server changes the
>         ConenctionID)?____
> 
>          ____
> 
>         Maybe relying on a 5-tuple is not such a bad idea after all?  A
>         NAT does not need a different ephemeral port for every outgoing
>         connection.  It only needs a different ephemeral port for
>         outgoing connections to the same IP.____
> 
>          ____
> 
>         Even then, if the NAT is willing to do something special for
>         QUIC, it can be a much simpler thing than examine QUIC transport
>         params.  The NAT can simply reserve a single ephemeral port for
>         all 1RTT QUIC packets.  That is, all outgoing cleartext and 0rtt
>         QUIC packets would require a distinct 5-tuple, but once the
>         connection migrates to 1RTT, the ephemeral port it used can be
>         freed, and all 1RTT packets sourced from the reserved QIUC
>         port.  That would work, since all 1RTT packets (both from client
>         and server) would be using Server-selection ConnectionID.____
> 
>          ____
> 
>           * Igor____
> 
>          ____
> 
>          ____
> 
>          ____
> 
>         *From:*Ian Swett [mailto:[email-masked]
>         <mailto:[email-masked]>]
>         *To:* Mikkel Fahnøe Jørgensen <[email-masked]
>         <mailto:[email-masked]>>
>         *Cc:* IETF QUIC WG <[email-masked] <mailto:[email-masked]>>; Ryan
>         Hamilton <[email-masked] <mailto:[email-masked]>>
>         *Subject:* Re: 5-tuple uniquess____
> 
>          ____
> 
>         You're right, ephemeral port exhaustion is a very real potential
>         issue, and I think we should ensure multiple QUIC connections
>         can run over the same 5-tuple and we don't specify QUIC in a way
>         that precludes that.____
> 
>          ____
> 
>         I believe you're correct that the current Server Cleartext makes
>         this impossible if the server changes the connection ID. 
>         Possibly it should specify the client's connection ID in the
>         packet header and then the new server connection ID in the
>         transport parameters? 
>         <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_quicwg_base-2Ddrafts_issues_714&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=mXkiKKh_FkRO-MANo0WqyEn2M1LkcNJFMOkgYdp4_q8&s=-DtvtkmD_lZJgEtjvugx4OxskRAUmqpy5NgCgNzahGw&e=>____
> 
>          ____
> 
>         The text is also missing some details on how changing the
>         connection ID should work for Stateless Retry,
>         see https://github.com/quicwg/base-drafts/issues/713
>         so I'd expect some improvements in the near future.____
> 
>          ____
> 
>         On Sun, Aug 6, 2017 at 10:43 AM, Mikkel Fahnøe Jørgensen
>         <[email-masked] <mailto:[email-masked]>> wrote:____
> 
>              ____
> 
>             Hi Ryan,____
> 
>              ____
> 
>             I can dig down in more detail if there is interest.
>             Meanwhile, some examples where the transport document is not
>             compatible with my request is:____
> 
>              ____
> 
>             The version negotiation packet is ok - it reflects the
>             client version____
> 
>             <https://urldefense.proofpoint.com/v2/url?u=https-3A__quicwg.github.io_base-2Ddrafts_draft-2Dietf-2Dquic-2Dtransport.html-23packet-2Dversion&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=mXkiKKh_FkRO-MANo0WqyEn2M1LkcNJFMOkgYdp4_q8&s=gnz8PJ4RQMq6w-_WFa1RHYMTgpgtgImakR1qZwm_uTk&e=>____
> 
>              ____
> 
>             But ____
> 
>             <https://urldefense.proofpoint.com/v2/url?u=https-3A__quicwg.github.io_base-2Ddrafts_draft-2Dietf-2Dquic-2Dtransport.html-23packet-2Dserver-2Dcleartext&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=mXkiKKh_FkRO-MANo0WqyEn2M1LkcNJFMOkgYdp4_q8&s=DjWdo8Yhu8t7h1zqd7BIxk9_TLU36AxmdZ0DmhZn6vE&e=>____
> 
>             "The connection ID field in a Server Cleartext packet
>             contains a connection ID that is chosen by the server
>             (see Section 5.6
>             <https://urldefense.proofpoint.com/v2/url?u=https-3A__quicwg.github.io_base-2Ddrafts_draft-2Dietf-2Dquic-2Dtransport.html-23connection-2Did&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=mXkiKKh_FkRO-MANo0WqyEn2M1LkcNJFMOkgYdp4_q8&s=5SH6u7AaH_arxGY3yEeqwyhio8dTX9SMQWFc2TIlhY0&e=>)."____
> 
>             This breaks linkage to the original client connection id so
>             the client cannot tell which connection the response belongs
>             to unless looking into 5-tuple, or testing all recent
>             connections against AEAD signature (depending on where this
>             lands).____
> 
>              ____
> 
>             I have not covered all other possible server responses, but
>             similar concerns apply.____
> 
>             It is not a major change, but currently it is not possible.____
> 
>              ____
> 
>             Another issue is hostile response to initial connection
>             setup, and plain errors - how to you reply with an error.
>             Having the connection id from the client makes it reasonably
>             easy to deal with - similar to version negotiation.
>             Alternatively you have to rely on 5-tuples. This requires
>             careful analysis of both the transport doc and possibly TLS
>             and thins are not fully settled. Stateless reset is related
>             to this. I will not argue strongly on this because since I
>             last looked at it, new AEAD encryption is being discussed,
>             and I have not analysed this in detail.____
> 
>              ____
> 
>             Then there is connection migration. This also requires
>             careful analysis. I’m not sure if it currently covers my
>             request.____
> 
>              ____
> 
>             I’m sure there are several other places with implicit
>             assumptions that I am not able to list off memory.____
> 
>              ____
> 
>             The (my) basic rule is the any change in connection ID
>             should refer to the previous ID without relying on external
>             state such as 5-tuples or TLS extension state (which
>             requires crypto state to access and complicates things).
>             Also, due concerns of privacy concerns this linkage between
>             connection ID’s should only be possible by peers, excepts
>             for the initial client to server connection transition that
>             is also visible to any party on-path.____
> 
>              ____
> 
>             In addition to the above encoding concerns, it may also be
>             helpful to explicitly state that the same UDP port may be
>             used to initiate multiple independent QUIC connection to the
>             same server UDP port when connection ID has been negotiated
>             to be present - or further detail this possibility as
>             something that can be negotiated.____
> 
>              ____
> 
>              ____
> 
>             Some other points I forget to mention regarding benefits of
>             non-unique tuples:____
> 
>              ____
> 
>             5) no interference with OS is necessary in order to
>             establish a new connection to an existing endpoint because
>             no new ephemeral port is needed. This may also simplify
>             clean up and state management, especially server to server.____
> 
>              ____
> 
>             6) Netmap and similar high speed interfaces would be simpler
>             without having to deal with ephemeral port allocation,
>             especially server to server where both end points can
>             reference each other, and where it might be random which
>             endpoint actually initiates to the connection (as in Cord /
>             Kademlia style overlay networks)____
> 
>              ____
> 
>             Kind Regards,____
> 
>             Mikkel Fahnøe Jørgensen____
> 
>                  ____
> 
>          ____
> 
>

---

From: Christian Huitema <[email-masked]>
Date: Sat, 12 Aug 2017 22:49:35 -0700

> latter happens to be zero.
Not to mention that there is hardly any cross-platform API to set or
view flow labels on IPv6/UDP packets, and that we may also want a
solution for IPv6.

I would personally prefer a symmetric solution, in which the Server
Clear Text packets are labelled with the initial Connection ID chosen by
the client. If the server wants to change that, it could include a
proposed connection ID as a parameter -- maybe as a transport parameter
in the TLS extension.

One of the really neat features of Quic is that a server can listen for
multiple connections on a single UDP socket. This reduce the overhead of
select or poll calls, because there is just one system interface to
manage. I would like to be able to do exactly the same for clients.
Maybe not a huge deal in HTTP/Web scenarios, but rather useful in proxy
scenarios, such as for example DNS resolvers.

Also, I would like to be able to support server mobility in the same
manner as client mobility, and with adequate server privacy.

-- 
Christian Huitema