Subject: [Masque] Updated version of QUIC-aware proxying using HTTP capsules

From: Tommy Pauly <[email-masked]>
Date: Tue, 12 Oct 2021 11:42:25 -0700

Hi all,

David and I recently revised the QUIC-aware proxying draft (which adds the ability to share sockets and perform forwarding on top of CONNECT-UDP).

This new revision aligns with the latest CONNECT-UDP and HTTP datagram drafts by adopting capsules and using CONNECT-UDP’s extended CONNECT method.

We ended up not needing to use any context IDs (as defined in HTTP datagrams), but instead are just using some new capsule messages to add control signaling between the client and the proxy. Hopefully, this informs some of our broader discussion about how the extensibility provided by HTTP datagrams can be used, and which parts are most useful.

Best,
Tommy

> Begin forwarded message:
> 
> From: [email-masked]
> Subject: New Version Notification for draft-pauly-masque-quic-proxy-02.txt
> Date: October 11, 2021 at 11:04:58 AM PDT
> To: David Schinazi <[email-masked]>, Tommy Pauly <[email-masked]>
> 
> 
> A new version of I-D, draft-pauly-masque-quic-proxy-02.txt
> has been successfully submitted by Tommy Pauly and posted to the
> IETF repository.
> 
> Name:		draft-pauly-masque-quic-proxy
> Revision:	02
> Title:		QUIC-Aware Proxying Using HTTP
> Document date:	2021-10-11
> Group:		Individual Submission
> Pages:		19
> URL:            https://www.ietf.org/archive/id/draft-pauly-masque-quic-proxy-02.txt
> Status:         https://datatracker.ietf.org/doc/draft-pauly-masque-quic-proxy/
> Html:           https://www.ietf.org/archive/id/draft-pauly-masque-quic-proxy-02.html
> Htmlized:       https://datatracker.ietf.org/doc/html/draft-pauly-masque-quic-proxy
> Diff:           https://www.ietf.org/rfcdiff?url2=draft-pauly-masque-quic-proxy-02
> 
> Abstract:
>   This document defines an extension to UDP Proxying over HTTP that
>   adds specific optimizations for proxied QUIC connections.  This
>   extension allows a proxy to reuse UDP 4-tuples for multiple
>   connections.  It also defines a mode of proxying in which QUIC short
>   header packets can be forwarded using an HTTP/3 proxy rather than
>   being re-encapsulated and re-encrypted.
> 
> 
> 
> 
> The IETF Secretariat
> 
>

---

From: Ben Schwartz <[email-masked]>
Date: Wed, 13 Oct 2021 09:13:15 -0400

I think there's an idea here that is worth pursuing, but I'm not sure about
this specific approach.

To start with a concrete issue, RFC 9000 says "an endpoint MUST NOT reuse a
connection ID when sending to more than one destination address".  To
preserve user privacy, the client connection ID must change when the
client's IP changes.  However, this draft is based on the idea that the
proxy uses essentially a single target-facing port, with the result that
the target doesn't know when the client's address has changed.

There's also a problem in the handling of Stateless Reset responses, which
generate a random Connection ID.  The proxy will have to drop any Reset
responses, because it does not know which client to forward them to.

The requirement that the QUIC packets to forward land on the same
client-facing port as the HTTP/3 connection also strikes me as unnatural
and possibly inconvenient.  The forwarding service really has nothing to do
with the HTTP/3 transport.  There's no reason to even require HTTP/3; the
forwarding service could just as well be controlled using a session that is
running in HTTP/2.

I believe these problems are consequences of trying to conserve port
numbers on the proxy.  This strikes me as a premature optimization.

I think we should benchmark this design against something closer to a
classic NAT:
- Registration ACKs indicate an explicit client-facing IP and port, which
is distinct from the HTTP endpoint
- Each client-facing port is associated with one tunnel
- Each (target-facing port, target) pair is used for at most one tunnel
- Connections should move to a different target-facing port (and possibly
IP) whenever the client's address (as viewed from the proxy) changes

This would remove the need for juggling of Connection IDs, and allow
application to any UDP payload, not just QUIC.  It would allow use with any
HTTP version, and be unaffected by arbitrary HTTP intermediaries.  It can
even be extended to forward TCP.

The port number would impose some scaling limits, but only on the
assumption that IP addresses are scarce.  In IPv6 this is definitely not
true, and scaling seems likely to be manageable even in IPv4, as evidenced
by the successful operation of large-scale TURN servers for WebRTC.

On Tue, Oct 12, 2021 at 2:42 PM Tommy Pauly <tpauly=
[email-masked]> wrote:

> Hi all,
>
> David and I recently revised the QUIC-aware proxying draft (which adds the
> ability to share sockets and perform forwarding on top of CONNECT-UDP).
>
> This new revision aligns with the latest CONNECT-UDP and HTTP datagram
> drafts by adopting capsules and using CONNECT-UDP’s extended CONNECT method.
>
> We ended up not needing to use any context IDs (as defined in HTTP
> datagrams), but instead are just using some new capsule messages to add
> control signaling between the client and the proxy. Hopefully, this informs
> some of our broader discussion about how the extensibility provided by HTTP
> datagrams can be used, and which parts are most useful.
>
> Best,
> Tommy
>
> Begin forwarded message:
>
> *From: [email-masked]
> *Subject: **New Version Notification for
> draft-pauly-masque-quic-proxy-02.txt*
> *Date: *October 11, 2021 at 11:04:58 AM PDT
> *To: *David Schinazi <[email-masked]>, Tommy Pauly <
> [email-masked]>
>
>
> URL:
> https://www.ietf.org/archive/id/draft-pauly-masque-quic-proxy-02.txt
> Status:
> https://datatracker.ietf.org/doc/draft-pauly-masque-quic-proxy/
> Html:
> https://www.ietf.org/archive/id/draft-pauly-masque-quic-proxy-02.html
> Htmlized:
> https://datatracker.ietf.org/doc/html/draft-pauly-masque-quic-proxy
> Diff:
> https://www.ietf.org/rfcdiff?url2=draft-pauly-masque-quic-proxy-02
>
> Abstract:
> --
> Masque mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/masque
>

---

From: Tommy Pauly <[email-masked]>
Date: Wed, 13 Oct 2021 08:56:03 -0700

Hi Ben,

> On Oct 13, 2021, at 6:13 AM, Ben Schwartz <bemasc=[email-masked]> wrote:
> 
> I think there's an idea here that is worth pursuing, but I'm not sure about this specific approach.
> 
> To start with a concrete issue, RFC 9000 says "an endpoint MUST NOT reuse a connection ID when sending to more than one destination address".  To preserve user privacy, the client connection ID must change when the client's IP changes.  However, this draft is based on the idea that the proxy uses essentially a single target-facing port, with the result that the target doesn't know when the client's address has changed.

I don’t think there’s any violation of that requirement here. The fact that a proxy hides a client’s original IP address, and any changes that might have (due to passive NAT rebinding or active migration) isn’t any different than with all of CONNECT-UDP proxying.
> 
> There's also a problem in the handling of Stateless Reset responses, which generate a random Connection ID.  The proxy will have to drop any Reset responses, because it does not know which client to forward them to.
> 

Yes, the stateless reset case is an interesting corner case to explore.

> The requirement that the QUIC packets to forward land on the same client-facing port as the HTTP/3 connection also strikes me as unnatural and possibly inconvenient.  The forwarding service really has nothing to do with the HTTP/3 transport.  There's no reason to even require HTTP/3; the forwarding service could just as well be controlled using a session that is running in HTTP/2.

I don’t think you’d want these to be different. From a network observer’s point of view, they wouldn’t be able to tell apart traffic bound to the proxy and traffic forwarded on—they’ll just see a sea of different QUIC SH packets with different connection IDs. It’s also a clear benefit to share address validation state for the return path, and handle NAT rebindings, etc.

> 
> I believe these problems are consequences of trying to conserve port numbers on the proxy.  This strikes me as a premature optimization.

Both the port number usage and the connection forwarding are things we doing today at large scale—this extension is the basis for how we do multi-hop proxies with iCloud Private Relay, and both aspects are hugely valuable optimizations in practice. The setup here is where you have many thousands to millions of concurrent connections that just need lightweight IP blinding to go to a next hop, with an authenticated proxy.

> 
> I think we should benchmark this design against something closer to a classic NAT:
> - Registration ACKs indicate an explicit client-facing IP and port, which is distinct from the HTTP endpoint
> - Connections should move to a different target-facing port (and possibly IP) whenever the client's address (as viewed from the proxy) changes
> 
> This would remove the need for juggling of Connection IDs, and allow application to any UDP payload, not just QUIC.  It would allow use with any HTTP version, and be unaffected by arbitrary HTTP intermediaries.  It can even be extended to forward TCP.
> 
> The port number would impose some scaling limits, but only on the assumption that IP addresses are scarce.  In IPv6 this is definitely not true, and scaling seems likely to be manageable even in IPv4, as evidenced by the successful operation of large-scale TURN servers for WebRTC.

As given in the reasons above, I think this kind of change would be a regression for both privacy and scalability.

Best,
Tommy

> 
> On Tue, Oct 12, 2021 at 2:42 PM Tommy Pauly <tpauly=[email-masked] <mailto:[email-masked]>> wrote:
> Hi all,
> 
>> From: [email-masked] <mailto:[email-masked]>
>> To: David Schinazi <[email-masked] <mailto:[email-masked]>>, Tommy Pauly <[email-masked] <mailto:[email-masked]>>
>> 
>> 
>> URL:            https://www.ietf.org/archive/id/draft-pauly-masque-quic-proxy-02.txt <https://www.ietf.org/archive/id/draft-pauly-masque-quic-proxy-02.txt>
>> Status:         https://datatracker.ietf.org/doc/draft-pauly-masque-quic-proxy/ <https://datatracker.ietf.org/doc/draft-pauly-masque-quic-proxy/>
>> Html:           https://www.ietf.org/archive/id/draft-pauly-masque-quic-proxy-02.html <https://www.ietf.org/archive/id/draft-pauly-masque-quic-proxy-02.html>
>> Htmlized:       https://datatracker.ietf.org/doc/html/draft-pauly-masque-quic-proxy <https://datatracker.ietf.org/doc/html/draft-pauly-masque-quic-proxy>
>> Diff:           https://www.ietf.org/rfcdiff?url2=draft-pauly-masque-quic-proxy-02 <https://www.ietf.org/rfcdiff?url2=draft-pauly-masque-quic-proxy-02>
>> 
>> Abstract:
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/masque <https://www.ietf.org/mailman/listinfo/masque>

---

From: Ben Schwartz <[email-masked]>
Date: Wed, 13 Oct 2021 13:00:31 -0400

On Wed, Oct 13, 2021 at 11:56 AM Tommy Pauly <tpauly=
[email-masked]> wrote:

> Hi Ben,
>
> On Oct 13, 2021, at 6:13 AM, Ben Schwartz <
> bemasc=[email-masked]> wrote:
>
> To start with a concrete issue, RFC 9000 says "an endpoint MUST NOT reuse
> a connection ID when sending to more than one destination address".  To
> I don’t think there’s any violation of that requirement here. The fact
> that a proxy hides a client’s original IP address, and any changes that
> might have (due to passive NAT rebinding or active migration) isn’t any
> different than with all of CONNECT-UDP proxying.
>

It's different from CONNECT-UDP because the client-proxy traffic is not
encrypted.  An observer between the client and proxy can see a connection
ID move from one client IP to another, and conclude that those two IPs are
the same client.  That would nullify the linkability defense from RFC 9000.

...

> I don’t think you’d want these to be different. From a network observer’s
> point of view, they wouldn’t be able to tell apart traffic bound to the
> proxy and traffic forwarded on—they’ll just see a sea of different QUIC SH
> packets with different connection IDs.
>

>From a privacy perspective, I don't see a clear difference between traffic
with different connection IDs on the same port and traffic with no
connection IDs on different ports.  Regardless, you're welcome to use the
same server port if you prefer and can manage it.

It’s also a clear benefit to share address validation state for the return
> path, and handle NAT rebindings, etc.
>

What happens if forwardable packets arrive from an unexpected client
address?  Does this trigger address revalidation on the client-proxy
connection?   This seems very tricky, given that the proxy can't validate
the authenticity of these packets at all, but they might represent nearly
all packets on the connection.

Overall, the amount of trickery required to get this right seems out of
proportion to the benefit.  For example, the need to apply conditions on
target connection IDs, despite there being no coordination with the target,
results in stack-crossing complexity and failure modes that must be handled
by the client.  This could be avoided by putting a very small header on
upstream packets to be forwarded, like TURN Channels [1].

Similarly, the deep integration with QUIC and extensive layer-crossing
makes this impossible to implement on top of a generic HTTP stack, or even
a generic QUIC implementation, due to the need to share a socket (both
client and server), learn and control which connection IDs are in use,
etc.  As a standards-track RFC, this draft would complicate, or maybe
erase, the interface between QUIC and HTTP/3.

I would be happier with this if the QUIC elements were decoupled from HTTP
and analyzed alongside QUIC-LB, perhaps in quicwg, exposing a well-defined
control interface that could be consumed by a MASQUE draft.

[1] https://datatracker.ietf.org/doc/html/rfc5766#section-2.5

---

From: Tommy Pauly <[email-masked]>
Date: Wed, 13 Oct 2021 11:18:02 -0700

> On Oct 13, 2021, at 10:00 AM, Ben Schwartz <bemasc=[email-masked]> wrote:
> 
> 
> 
> On Wed, Oct 13, 2021 at 11:56 AM Tommy Pauly <tpauly=[email-masked] <mailto:[email-masked]>> wrote:
> Hi Ben,
> 
>> On Oct 13, 2021, at 6:13 AM, Ben Schwartz <bemasc=[email-masked] <mailto:bemasc=[email-masked]>> wrote:
>> 
> It's different from CONNECT-UDP because the client-proxy traffic is not encrypted.  An observer between the client and proxy can see a connection ID move from one client IP to another, and conclude that those two IPs are the same client.  That would nullify the linkability defense from RFC 9000.

I think the linkability defense was more about recognizing a connection as it changes between two different network paths (and sent different traffic at different times on these paths), not about recognizing a connection along a single network path. Yes, forwarding mode does mean that an observer looking at both segments of the network path can recognize traffic (as would be possible anyway unless a lot of work was done to hide traffic patterns based on timing and size), but I don’t believe that is a violation of the spec. The client’s IP address didn’t change, and thus the connection ID doesn’t need to change.

> I don’t think you’d want these to be different. From a network observer’s point of view, they wouldn’t be able to tell apart traffic bound to the proxy and traffic forwarded on—they’ll just see a sea of different QUIC SH packets with different connection IDs.
> 
> From a privacy perspective, I don't see a clear difference between traffic with different connection IDs on the same port and traffic with no connection IDs on different ports.  Regardless, you're welcome to use the same server port if you prefer and can manage it.

It’s quite different, since there isn’t a guaranteed number of connection IDs that a single QUIC connection can use on a specific UDP socket. It thus becomes harder to observe the exact number of proxied connections going on through a tunnel, whereas opening a new connection for every request would provide a much stronger signal to a network observer.

Beyond that, trying to split up traffic would add a lot of overhead for validating these other client addresses/ports, if that even would be possible. I don’t think that’s a viable direction, and it doesn’t seem to add value.

> 
> It’s also a clear benefit to share address validation state for the return path, and handle NAT rebindings, etc.
> 
> What happens if forwardable packets arrive from an unexpected client address?  Does this trigger address revalidation on the client-proxy connection?   This seems very tricky, given that the proxy can't validate the authenticity of these packets at all, but they might represent nearly all packets on the connection.

The proxy can initiate path validation using its own QUIC connection to the client, since it recognizes which client has a mapping for the target CID.

> 
> Overall, the amount of trickery required to get this right seems out of proportion to the benefit.  For example, the need to apply conditions on target connection IDs, despite there being no coordination with the target, results in stack-crossing complexity and failure modes that must be handled by the client.  This could be avoided by putting a very small header on upstream packets to be forwarded, like TURN Channels [1].

The conditions to map target connection IDs doesn’t need coordination with the target. They essentially turn into simple BPF rules on the proxy side.

> 
> Similarly, the deep integration with QUIC and extensive layer-crossing makes this impossible to implement on top of a generic HTTP stack, or even a generic QUIC implementation, due to the need to share a socket (both client and server), learn and control which connection IDs are in use, etc.  As a standards-track RFC, this draft would complicate, or maybe erase, the interface between QUIC and HTTP/3.

Please note that this document is defined as experimental. However, I would also note that this has been implemented in multiple stacks already. Please don’t assume that QUIC implementations are so tied to having sole ownership over unique UDP sockets. It’s already possible for multiple QUIC connections to share a single socket.

I also don’t understand why this would change any interface between QUIC and HTTP/3. It certainly didn’t require any such changes in our implementations.

> 
> I would be happier with this if the QUIC elements were decoupled from HTTP and analyzed alongside QUIC-LB, perhaps in quicwg, exposing a well-defined control interface that could be consumed by a MASQUE draft.

I believe that proxying of unreliable packets using an HTTP proxy falls squarely within MASQUE’s domain. Outside of the scope of proxying, this doesn’t make sense.

Thanks,
Tommy
> 
> [1] https://datatracker.ietf.org/doc/html/rfc5766#section-2.5 <https://datatracker.ietf.org/doc/html/rfc5766#section-2.5>-- 
> Masque mailing list
> [email-masked]

---

From: Ben Schwartz <[email-masked]>
Date: Wed, 13 Oct 2021 14:34:33 -0400

On Wed, Oct 13, 2021 at 2:18 PM Tommy Pauly <tpauly=
[email-masked]> wrote:

> I think the linkability defense was more about recognizing a connection as
> it changes between two different network paths (and sent different traffic
> at different times on these paths)
>

Yes, I'm discussing the case where the client has moved, and is now
speaking to the proxy from a different client address.  The proxy changes
its Client Connection ID in response, but the target does not, because it
does not know that the client has moved.

..

> The proxy can initiate path validation using its own QUIC connection to
> the client, since it recognizes which client has a mapping for the target
> CID.
>

Yes, but this means that packet forwarding can't simply be a rule in a
kernel or smart-NIC.  It needs to be handled by the QUIC stack, which needs
to have a first-class notion of which connection IDs are being controlled
via an application running inside which other connection that it actually
owns.

> The conditions to map target connection IDs doesn’t need coordination with
> the target.
>

Yes.

> They essentially turn into simple BPF rules on the proxy side.
>

Unless they are rejected due to a collision, in which case the client sees
a registration failure, which adds a codepath.

Please note that this document is defined as experimental.

OK, in that case I'm happy to put aside architectural considerations.

> I believe that proxying of unreliable packets using an HTTP proxy falls
> squarely within MASQUE’s domain. Outside of the scope of proxying, this
> doesn’t make sense.
>

It seems to me that the transport logic here (demultiplexing by QUIC ID,
port sharing rules, etc.) has nothing to do with HTTP, and could be defined
entirely separately.  MASQUE is scoped exclusively to HTTP.

---

From: Tommy Pauly <[email-masked]>
Date: Wed, 13 Oct 2021 12:27:56 -0700

> On Oct 13, 2021, at 11:34 AM, Ben Schwartz <[email-masked]> wrote:
> 
> 
> 
> On Wed, Oct 13, 2021 at 2:18 PM Tommy Pauly <tpauly=[email-masked] <mailto:[email-masked]>> wrote:
> I think the linkability defense was more about recognizing a connection as it changes between two different network paths (and sent different traffic at different times on these paths)
> 
> Yes, I'm discussing the case where the client has moved, and is now speaking to the proxy from a different client address.  The proxy changes its Client Connection ID in response, but the target does not, because it does not know that the client has moved.

Ah, in this case the client just needs to change connection IDs used to the target, which should trigger the target to also start changing connections IDs. We ought to describe that more in detail for the migration considerations section, but I think it’s fairly straightforward.

If the target didn’t provide other CIDs or doesn’t support migration, then the client shouldn’t try to maintain a single forwarded connection across two different paths.

> Yes, but this means that packet forwarding can't simply be a rule in a kernel or smart-NIC.  It needs to be handled by the QUIC stack, which needs to have a first-class notion of which connection IDs are being controlled via an application running inside which other connection that it actually owns.
> The conditions to map target connection IDs doesn’t need coordination with the target.
> 
> Yes.
>  
> Unless they are rejected due to a collision, in which case the client sees a registration failure, which adds a codepath.

Yes, but this falls back to simply tunneling within DATAGRAM frames, which always needs to be implemented.

> OK, in that case I'm happy to put aside architectural considerations.

Sounds good. I’m not advocating that everyone do this—but it’s a very useful thing for specific deployment models, and we are being transparent about what the protocol is doing.

> It seems to me that the transport logic here (demultiplexing by QUIC ID, port sharing rules, etc.) has nothing to do with HTTP, and could be defined entirely separately.  MASQUE is scoped exclusively to HTTP.

MASQUE is scoped to using HTTP as the protocol to communicate with the proxy. However, MASQUE certainly not scoped to only allow proxying of HTTP traffic. It specifically is for proxying UDP, IP, etc. 

The ability for two QUIC connections to share a UDP socket is something that’s already allowed and defined in RFC 9000. This document about QUIC-aware proxying is specifically defining how the HTTP proxy layer adds control signaling to negotiate this sharing.

Thanks,
Tommy

---

From: Ben Schwartz <[email-masked]>
Date: Wed, 13 Oct 2021 15:44:35 -0400

> Ah, in this case the client just needs to change connection IDs used to
> the target, which should trigger the target to also start changing
> connections IDs.
>

Should it?  I'm not aware of a rule like that in QUIC.

---

From: Tommy Pauly <[email-masked]>
Date: Wed, 13 Oct 2021 13:35:40 -0700

> On Oct 13, 2021, at 12:44 PM, Ben Schwartz <[email-masked]> wrote:
> 
> On Wed, Oct 13, 2021 at 3:27 PM Tommy Pauly <[email-masked] <mailto:[email-masked]>> wrote:
>> On Oct 13, 2021, at 11:34 AM, Ben Schwartz <[email-masked] <mailto:[email-masked]>> wrote:
> 
> ... 
> Ah, in this case the client just needs to change connection IDs used to the target, which should trigger the target to also start changing connections IDs.
> 

Sorry, this is something that we need to explain more clearly in the document. Specifically, when the proxy detects a migration event, it would likely need to change the local port or address it uses on its target-facing socket to make the address change event be visible to the target. Then, (and the part I was referring to), as long as the client is changing the connection ID to the target, it would ensure that it wouldn’t hit this exception clause that allows the target server to not change connection IDs if the received connection ID didn’t change:

Similarly, an endpoint MUST NOT reuse a connection ID when sending to more than one destination address. Due to network changes outside the control of its peer, an endpoint might receive packets from a new source address with the same Destination Connection ID field value, in which case it MAY continue to use the current connection ID with the new remote address while still sending from the same local address.

---

From: Ben Schwartz <[email-masked]>
Date: Wed, 13 Oct 2021 17:11:10 -0400

OK, sounds good.

On Wed, Oct 13, 2021 at 4:35 PM Tommy Pauly <[email-masked]> wrote:

> Sorry, this is something that we need to explain more clearly in the
> document. Specifically, when the proxy detects a migration event, it would
> likely need to change the local port or address it uses on its
> target-facing socket to make the address change event be visible to the
> target. Then, (and the part I was referring to), as long as the client is
> changing the connection ID to the target, it would ensure that it wouldn’t
> hit this exception clause that allows the target server to not change
> connection IDs if the received connection ID didn’t change:
>
> Similarly, an endpoint MUST NOT reuse a connection ID when sending to more
> than one destination address. Due to network changes outside the control of
> its peer, an endpoint might receive packets from a new source address with
> the same Destination Connection ID field value, in which case
> it MAY continue to use the current connection ID with the new remote
> address while still sending from the same local address.
>
>

---

From: "Martin Thomson" <[email-masked]>
Date: Thu, 14 Oct 2021 15:30:04 +1100

Let's leave aside for the moment the questions about relative utility and the expected threat model - those remain unanswered for me.  Those questions are more important than questions of how connection IDs might be set when migration occurs, because that determines whether this is worth pursuing at all.  The draft assumes a *very* weak adversary.  That might be appropriate if we are only concerned about obscuring source IP addresses, but we might then ask what benefit this has relative to a (far simpler) NAT.

In any case, to the discussion at hand...

This draft proposes a design for use of connection IDs that is the antithesis of the design in QUIC.  In QUIC, the endpoint that receives a packet decides what connection IDs are used.  Here, we have clients informing proxies of what connection IDs might appear on packets that the proxy might receive.  The proxy has no say, which could make things very difficult for it.

That difficulty is manifest in the non-determinism of the design and Section 2.4 where you need a strategy to manage collisions.  This says that a proxy might reject a mapping, but that might not work out well for connection IDs that come from  a server that is ignorant of the proxy's constraints.  As the protocol is described, you can probably build a proxy that isn't forced to reject ALL connection IDs from a server, but there are designs that this proposal completely forecloses on.

A different design involves the proxy telling the client what connection IDs can be used.  Or, better yet, having the proxy generate a mapping from Server -> Proxy connection IDs into Proxy -> Client connection IDs (and the same in the other direction).  This can be achieved with a tiny amount of packet rewriting at the proxy and client and even less proxy state than the draft currently uses.  Having connection IDs constrained to a single hop makes all the routing easier for all involved and avoids any collision issues.

That still requires a bit of thinking about what the proxy might need to do for unvalidated client addresses, which will be hidden from servers and so might be used for DoS.  My initial thought is that dropping packets to meet QUIC's rules is a solid strategy that will produce an acceptable outcome.

On Thu, Oct 14, 2021, at 08:11, Ben Schwartz wrote:
> Attachments:
> * smime.p7s

---

From: Christian Huitema <[email-masked]>
Date: Wed, 13 Oct 2021 22:28:08 -0700

> This draft proposes a design for use of connection IDs that is the antithesis of the design in QUIC.  In QUIC, the endpoint that receives a packet decides what connection IDs are used.  Here, we ha…

Isn't that very much the problem addressed by QUIC-LB?

-- Christian Huitema

---

From: "Martin Thomson" <[email-masked]>
Date: Thu, 14 Oct 2021 17:00:54 +1100

> Isn't that very much the problem addressed by QUIC-LB?

You could frame things that way, certainly.  However, I think that there are certain assumptions there that might not hold in this case.  For instance, a server behind a load balancer can be allocated a small number to identify it.  Clients of a proxy might not.  A server behind a load balancer might expect its outbound packets to be sent more or less directly.  Here, the proxy might need to perform some routing based on information that is not in the IP/UDP header.

We might learn a lot from QUIC-LB here, but it might not be a perfect fit.

---

From: Tommy Pauly <[email-masked]>
Date: Thu, 14 Oct 2021 09:00:02 -0700

One discussion we’ve been having with Lucas about how to make the QUIC-aware forwarding play better with QUIC-LB might be a useful solution here.

We could change things to have the proxy’s ACK_CID capsules define a proxy-chosen CID to use instead of a target-chosen CID, which could be swapped out at the proxy. The process could work like:

- Client registers Target CID 0x1234 with the proxy
- Proxy acks, and says to replace Target CID 0x1234 with Proxy CID 0x5678 when forwarding to the proxy
- For a packet that the client is sending in forwarding mode, it would create the packet to the target with DCID 0x1234, and then overwrite that with DCID 0x5678 before sending (but not changing anything else in the packet)

For cases where the Proxy CID length is equal to or shorter than the Target CID length, this works no problem. For cases where the Proxy CID is longer than the Target CID, the client would need to reserve some extra room when creating the packets (essentially shifting over), and the proxy would need to shrink the packet on its transformation. This should be doable, still.

We can certainly make this the standard approach, which could make this simpler to reason about.

Thanks,
Tommy

> On Oct 13, 2021, at 11:00 PM, Martin Thomson <[email-masked]> wrote:
>

---

From: Christian Huitema <[email-masked]>
Date: Thu, 14 Oct 2021 11:23:12 -0700

> - For a packet that the client is sending in forwarding mode, it would create the packet to the target with DCID 0x1234, and then overwrite that with DCID 0x5678 before sending (but not changing an…

Swapping bytes in packets and hoping that it won't break encryption? 
That seems brittle. It might be simpler for clients to just ask the 
proxy to provide them a set of suitable CID, and then pick CID from that 
pool for individual connections.

Maybe we need to start by stating core assumptions, such as whether only 
clients or both clients and servers seat behind a proxy, and whether a 
client using a proxy always sends all of its traffic through that proxy, 
instead of, for example, using connection migration and shortcut the 
proxy for some connections. Or whether we want a client to be able to 
split traffic between two uncoordinated proxies.

> For cases where the Proxy CID length is equal to or shorter than the Target CID length, this works no problem. For cases where the Proxy CID is longer than the Target CID, the client would need to…

The key assumption is QUIC-LB is that the servers behind the LB agree to 
abide by the LB's rules. The rules include a minimum length.

It might make sens to have similar agreement on rules as part of the 
explicit decision to use a proxy.

-- Christian Huitema

---

From: Tommy Pauly <[email-masked]>
Date: Thu, 14 Oct 2021 12:45:41 -0700

> On Oct 14, 2021, at 11:23 AM, Christian Huitema <[email-masked]> wrote:
> 
> 
> Swapping bytes in packets and hoping that it won't break encryption? That seems brittle.

To be clear, the encryption and integrity checks for the packet would only work with the end-to-end Target CID. Swapping in the Proxy CID would only be for the link forwarding, and processing by a load balancer, and would be switched back to a valid CID packet before it hits the actual QUIC stack on the other end.

> It might be simpler for clients to just ask the proxy to provide them a set of suitable CID, and then pick CID from that pool for individual connections.
> 
> Maybe we need to start by stating core assumptions, such as whether only clients or both clients and servers seat behind a proxy, and whether a client using a proxy always sends all of its traffic through that proxy, instead of, for example, using connection migration and shortcut the proxy for some connections. Or whether we want a client to be able to split traffic between two uncoordinated proxies.

The setup here is:

Client —> Proxy —> non-coordinating QUIC target (doesn’t know about the proxy)

With the load balancing variant:

Client —> LB -> Proxy —> non-coordinating QUIC target (doesn’t know about the proxy)

I’m not aware of any case where we’d want the client to talk to the target directly, but it could theoretically switch to send the traffic through another proxy via migration.
> The key assumption is QUIC-LB is that the servers behind the LB agree to abide by the LB's rules. The rules include a minimum length.
> 
> It might make sens to have similar agreement on rules as part of the explicit decision to use a proxy.

Agreed, any case with a QUIC LB needs to have coordination about CID lengths here, and if an LB knows it has a proxy behind it that wants to do this, it may need to have additional coordination.

Thanks,
Tommy

---

From: Ben Schwartz <[email-masked]>
Date: Thu, 14 Oct 2021 15:56:38 -0400

...

I think it's already there in Section 4.1:

> In order to avoid conflicts, clients SHOULD select Client Connection IDs
of at least 8 bytes in length with unpredictable values.

Agreed, any case with a QUIC LB needs to have coordination about CID
> lengths here, and if an LB knows it has a proxy behind it that wants to do
> this, it may need to have additional coordination.
>

I don't think the topic is about putting a QUIC-LB in front of the proxy.
I think the point is that the proxy _is_ a QUIC-LB, just upside-down, with
the role of client and target/backend swapped.

There are some important differences (migration, mutual trust among
participating endpoints) but it's definitely a strong parallel.