Subject: [Masque] Updates to HTTP Datagrams and CONNECT-UDP

From: David Schinazi <[email-masked]>
Date: Wed, 6 Oct 2021 19:58:14 -0700

Hi MASQUE enthusiasts,

Lucas and I have been hard at work and we're excited to share that we've
made changes discussed by the WG at the last meeting and submitted updates
to both draft-ietf-masque-h3-datagram and draft-ietf-masque-connect-udp.

In particular, the main changes are:
* capsules are no longer HTTP/3 frames and are instead sent on the data
stream
* extensibility of HTTP datagrams has been simplified via Datagram Format
Types
* CONNECT-UDP is no longer its own method and now uses extended CONNECT
* CONNECT-UDP is now configured using a URI template
* all versions of HTTP are now supported (as opposed to it being a TODO)

Here are links to the latest versions:
https://datatracker.ietf.org/doc/html/draft-ietf-masque-h3-datagram-04
https://datatracker.ietf.org/doc/html/draft-ietf-masque-connect-udp-05

And here are diffs:
https://www.ietf.org/rfcdiff?url2=draft-ietf-masque-h3-datagram-04
https://www.ietf.org/rfcdiff?url2=draft-ietf-masque-connect-udp-05

As usual, we'd love to hear your thoughts and comments on GitHub issues for
these drafts.

Google QUICHE implements the latest versions, please contact me if you'd
like to test interop.

Thanks,
David

---

From: Ben Schwartz <[email-masked]>
Date: Wed, 13 Oct 2021 13:21:01 -0400

One thing that seems to have been lost in these revisions is the ability to
transfer UDP packets reliably.  Endpoints can use DATAGRAM capsules, but
intermediaries might "decapsulate" them, making them unreliable.  This is
especially relevant in the "first flight" context: in typical UDP
scenarios, sending the first packet reliably and in-order after the request
is good for performance and avoids the need for servers to buffer incoming
datagrams.

On Wed, Oct 6, 2021 at 10:58 PM David Schinazi <[email-masked]>
wrote:

> As usual, we'd love to hear your thoughts and comments on GitHub issues
> for these drafts.
>
> --
> Masque mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/masque
>

---

From: David Schinazi <[email-masked]>
Date: Wed, 13 Oct 2021 10:35:19 -0700

Hi Ben,

The client can still send the first UDP packet reliably in a capsule.
Intermediaries might convert it to an unreliable datagram, but at the end
of the day the proxy is going to convert it to an unreliable UDP packet. So
you're not getting client-to-target datagram reliability regardless. Also I
don't think this changed since previous drafts, it just wasn't explicit in
previous drafts but that would happen anyway if you had client ---- h3 ----
intermediary 1 ---- h2 ---- intermediary 2 ---- h3 ---- proxy.

David

On Wed, Oct 13, 2021 at 10:21 AM Ben Schwartz <[email-masked]> wrote:

> One thing that seems to have been lost in these revisions is the ability
> to transfer UDP packets reliably.  Endpoints can use DATAGRAM capsules, but

---

From: Ben Schwartz <[email-masked]>
Date: Wed, 13 Oct 2021 13:57:13 -0400

I think the ability to transfer UDP packets reliably between client and
proxy is valuable and worth retaining, for a few reasons.

1. Slow loss recovery - The first flight on a connection has no RTT or
bandwidth estimate, so loss detection and retry is generally very slow and
conservative.  The client-proxy connection is already warm, has smaller
actual RTT than the end-to-end path, and likely has higher loss rate than
the proxy-target path.  Sending the first packet "reliably" allows us to
recover from packet loss on that link much faster.

2. MTU - If an active H3 intermediary hop has a small MTU, the endpoints
may need some way to force the intermediary to send their packets over the
reliable channel in order to emulate acceptable client-proxy MTU.  (This is
perhaps clearer in the CONNECT-IP context.)

3. Reordering/buffering - If we assume that reasonable tolerance to packet
reordering is expected, then in the absence of a reliable delivery
mechanism, a datagram that arrives before its stream or context
registration has to be buffered by the recipient, or it will be lost (see
1).  This complicates both endpoints and increases the potential for abuse.

My preferred solution is to add a RELIABLE_DATAGRAM capsule type, which is
semantically identical to DATAGRAM but opaque.

On Wed, Oct 13, 2021 at 1:35 PM David Schinazi <[email-masked]>
wrote:

>>> * extensibility of HTTP datagrams has been simplified via Datagram
>>> Format Types

---

From: Tommy Pauly <[email-masked]>
Date: Wed, 13 Oct 2021 11:22:37 -0700

It sounds like a RELIABLE _DATAGRAM would be a possible extension for CONNECT-UDP, but doesn’t seem to be something the core spec should support. The nuances of when this would be appropriate are subtle.

The one case I see here that is compelling is the MTU case, where the datagram could be sent as a capsule not in a QUIC DATAGRAM frame. However, it seems simple to just have guidance in CONNECT-UDP that an intermediary SHOULD forward any datagram that it can’t fit in a DATAGRAM frame using a capsule rather than dropping the packet.

To that end, you could even suggest to intermediaries that they SHOULD always use a capsule instead of a DATAGRAM frame if the datagram arrived over a capsule using H3 when it could have used a DATAGRAM frame, but the client chose to use a capsule instead.

Thanks,
Tommy

> On Oct 13, 2021, at 10:57 AM, Ben Schwartz <bemasc=[email-masked]> wrote:
> 
> I think the ability to transfer UDP packets reliably between client and proxy is valuable and worth retaining, for a few reasons.
> 
> 1. Slow loss recovery - The first flight on a connection has no RTT or bandwidth estimate, so loss detection and retry is generally very slow and conservative.  The client-proxy connection is already warm, has smaller actual RTT than the end-to-end path, and likely has higher loss rate than the proxy-target path.  Sending the first packet "reliably" allows us to recover from packet loss on that link much faster.
> 
> 2. MTU - If an active H3 intermediary hop has a small MTU, the endpoints may need some way to force the intermediary to send their packets over the reliable channel in order to emulate acceptable client-proxy MTU.  (This is perhaps clearer in the CONNECT-IP context.)
> 
> 3. Reordering/buffering - If we assume that reasonable tolerance to packet reordering is expected, then in the absence of a reliable delivery mechanism, a datagram that arrives before its stream or context registration has to be buffered by the recipient, or it will be lost (see 1).  This complicates both endpoints and increases the potential for abuse.
> 
> My preferred solution is to add a RELIABLE_DATAGRAM capsule type, which is semantically identical to DATAGRAM but opaque.
> 
> 
> On Wed, Oct 13, 2021 at 1:35 PM David Schinazi <[email-masked] <mailto:[email-masked]>> wrote:
> Hi Ben,
> 
> The client can still send the first UDP packet reliably in a capsule. Intermediaries might convert it to an unreliable datagram, but at the end of the day the proxy is going to convert it to an unreliable UDP packet. So you're not getting client-to-target datagram reliability regardless. Also I don't think this changed since previous drafts, it just wasn't explicit in previous drafts but that would happen anyway if you had client ---- h3 ---- intermediary 1 ---- h2 ---- intermediary 2 ---- h3 ---- proxy.
> 
> David
> 
> On Wed, Oct 13, 2021 at 10:21 AM Ben Schwartz <[email-masked] <mailto:[email-masked]>> wrote:
> One thing that seems to have been lost in these revisions is the ability to transfer UDP packets reliably.  Endpoints can use DATAGRAM capsules, but intermediaries might "decapsulate" them, making them unreliable.  This is especially relevant in the "first flight" context: in typical UDP scenarios, sending the first packet reliably and in-order after the request is good for performance and avoids the need for servers to buffer incoming datagrams.
> 
> On Wed, Oct 6, 2021 at 10:58 PM David Schinazi <[email-masked] <mailto:[email-masked]>> wrote:
> Hi MASQUE enthusiasts,
> 
> Lucas and I have been hard at work and we're excited to share that we've made changes discussed by the WG at the last meeting and submitted updates to both draft-ietf-masque-h3-datagram and draft-ietf-masque-connect-udp.
> 
> * capsules are no longer HTTP/3 frames and are instead sent on the data stream
> * extensibility of HTTP datagrams has been simplified via Datagram Format Types
> https://datatracker.ietf.org/doc/html/draft-ietf-masque-h3-datagram-04 <https://datatracker.ietf.org/doc/html/draft-ietf-masque-h3-datagram-04>
> https://datatracker.ietf.org/doc/html/draft-ietf-masque-connect-udp-05 <https://datatracker.ietf.org/doc/html/draft-ietf-masque-connect-udp-05>
> 
> And here are diffs:
> https://www.ietf.org/rfcdiff?url2=draft-ietf-masque-h3-datagram-04 <https://www.ietf.org/rfcdiff?url2=draft-ietf-masque-h3-datagram-04>
> https://www.ietf.org/rfcdiff?url2=draft-ietf-masque-connect-udp-05 <https://www.ietf.org/rfcdiff?url2=draft-ietf-masque-connect-udp-05>
> 
> As usual, we'd love to hear your thoughts and comments on GitHub issues for these drafts.
> 
> Google QUICHE implements the latest versions, please contact me if you'd like to test interop.
> 
> Thanks,
> David
> -- 
> Masque mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/masque <https://www.ietf.org/mailman/listinfo/masque>
> -- 
> Masque mailing list
> [email-masked]

---

From: David Schinazi <[email-masked]>
Date: Wed, 13 Oct 2021 11:30:19 -0700

> proxy is valuable and worth retaining, for a few reasons.

Note that this isn't retaining, none of the CONNECT-UDP drafts have
supported this feature so far. So adding this new feature as an extension
sounds like a reasonable path forward.

> recover from packet loss on that link much faster.

Connections are hop-by-hop, so there's no such thing as a client-to-proxy
connection. There's a client-to-first-intermediary connection, and having
the client send the first packet in a capsule is currently supported and
that'll solve the issue you're describing. Because in practice the great
majority of loss happens on the client-to-first-intermediary hop.

> perhaps clearer in the CONNECT-IP context.)

Can you elaborate on this use-case? Can you perhaps share a diagram with
MTUs on the various hops?

> 1).  This complicates both endpoints and increases the potential for abuse.

Buffering a small number of datagrams is no more of an attack vector than
buffering capsule/stream data when it arrives out of order.

> semantically identical to DATAGRAM but opaque.

This sounds like a reasonable extension, but I don't think it should be
introduced in the base spec because it isn't clear it provides benefit in
real world scenarios.

David

>>>> Lucas and I have been hard at work and we're excited to share that
>>>> we've made changes discussed by the WG at the last meeting and submitted
>>>> updates to both draft-ietf-masque-h3-datagram and
>>>> draft-ietf-masque-connect-udp.
>>>>

---

From: Ian Swett <[email-masked]>
Date: Wed, 13 Oct 2021 14:46:14 -0400

IMO, RELIABLE_DATAGRAM is not necessary, because reliability is something
implemented by the sender in QUIC.

Put another way, this is an implementation/API choice, not a protocol
design point.

On Wed, Oct 13, 2021 at 2:30 PM David Schinazi <[email-masked]>
wrote:

>>>> One thing that seems to have been lost in these revisions is the
>>>> ability to transfer UDP packets reliably.  Endpoints can use DATAGRAM
>>>> capsules, but intermediaries might "decapsulate" them, making them
>>>> unreliable.  This is especially relevant in the "first flight" context: in
>>>> typical UDP scenarios, sending the first packet reliably and in-order after
>>>> the request is good for performance and avoids the need for servers to
>>>> buffer incoming datagrams.
>>>>
>>>> On Wed, Oct 6, 2021 at 10:58 PM David Schinazi <
>>>> [email-masked]> wrote:
>>>>
>>>>> Hi MASQUE enthusiasts,
>>>>>

---

From: Ben Schwartz <[email-masked]>
Date: Wed, 13 Oct 2021 15:05:27 -0400

...

> in practice the great majority of loss happens on the
> client-to-first-intermediary hop.
>

Sure.

> MTUs on the various hops?

OK.  Imagine CONNECT-IP with:
             H3                                        H2
Client ---------> Active Intermediary ---------> Proxy --------> Target

The MTU on all these links is 1280 (worst case), so the H3 Datagram MTU is
<1280.  Target sends Proxy a 1280-byte packet, which goes in a DATAGRAM
capsule.  Intermediary decapsulates it, and wraps it into a >1280-byte QUIC
packet, which is lost or dropped, as it exceeds MTU.

If we accept that the client-target DPLPMTUD is permitted to assume MTU >=
1280, then communication fails.

If Proxy uses PING to measure the Datagram MTU M and finds M < 1280, it can
solve this by using RELIABLE_DATAGRAM for all packets M < size(packet) <=
1280.

> buffering capsule/stream data when it arrives out of order.

It's always better to give other parties the least ability to tie up your
resources.  Mainly, I think the simplicity of "datagram is recognized ?
send() : drop()" is valuable.

> introduced in the base spec because it isn't clear it provides benefit in

I think it should be in the base spec.  It's absolutely trivial to specify
and implement, it's needed for correct MTU handling (under certain
assumptions), and the performance and removal-of-buffering benefits are
largely lost if the endpoints can't assume that it's supported.

---

From: David Schinazi <[email-masked]>
Date: Wed, 13 Oct 2021 17:45:47 -0700

Let me attempt to summarize an offline discussion I had with Ben on this:
in the CONNECT-IP world, if we want to tunnel IPv6 we'll need to support
1280-byte datagrams. This can become an issue if there's an h3 hop along
the way that has a low MTU such that this h3 hop can only send QUIC
DATAGRAM frames of length < 1280 (let's say 1000 for simplicity). In the
main deployment model for CONNECT-IP, the h3 leg will be on the
client-to-first-intermediary hop, so the client can choose to send QUIC
initials of length 1330 to ensure that if the MTU is too small the client
will use h2 instead of h3 on the first hop. The scenario where things
aren't solvable by the client nor proxy is when we have a setup like
this: client ---- h2 ---- intermediary 1 ---- h3 ---- intermediary 2 ----
h2 ---- proxy where the h3 leg has a datagram MTU of 1000. In this example,
when the intermediaries receive a DATAGRAM capsule over h2, they need to
decide whether to send it as a DATAGRAM capsule or a DATAGRAM frame over
h3. If the incoming datagram is of length 1280, that means the intermediary
needs to either drop the datagram, or send it as a capsule. But different
use cases might prefer different outcomes: for example if the endpoints are
performing DPLPMTUD then having the intermediary make their probes reliable
will break DPLPMTUD. But if the endpoints have a minimum required MTU (such
as 1280 in IPv6) then not making it reliable will break tunneled IPv6 MTU
requirements. So because the intermediary isn't aware of which is which, we
need a way for the endpoints to tell the intermediary whether to
drop/DATAGRAM frame or to capsule. Having an opaque RELIABLE_DATAGRAM
capsule fits the bill for that signal.

TLDR (of that long ramble, my apologies) is that while most use cases don't
need RELIABLE_DATAGRAM, you can construct some uncommon deployments that
won't work well without it. Because this adds virtually no complexity, I
think it makes sense to add RELIABLE_DATAGRAM to the core HTTP Datagrams
draft.

Thoughts?
David

On Wed, Oct 13, 2021 at 12:05 PM Ben Schwartz <[email-masked]> wrote:

> If Proxy uses PING to measure the Datagram MTU M and finds M < 1280, it
> can solve this by using RELIABLE_DATAGRAM for all packets M < size(packet)
> <= 1280.
>

---

From: Lucas Pardue <[email-masked]>
Date: Thu, 14 Oct 2021 02:02:34 +0100

On Thu, Oct 14, 2021 at 1:46 AM David Schinazi <[email-masked]>
wrote:

> TLDR (of that long ramble, my apologies) is that while most use cases
> don't need RELIABLE_DATAGRAM, you can construct some uncommon deployments
> that won't work well without it. Because this adds virtually no complexity,
> I think it makes sense to add RELIABLE_DATAGRAM to the core HTTP Datagrams
> draft.
>
> Thoughts?
>

I might be convinced to support the addition of a capsule that does such a
thing but I don't like the proposed name much. Something like
SEALED_DATAGRAM seems a bit more accurate in so far as intermediaries
cannot meddle with the capsule.

Lucas

---

From: Tommy Pauly <[email-masked]>
Date: Wed, 13 Oct 2021 20:04:21 -0700

I agree with Lucas that the name shouldn’t be “reliable datagram”, since the datagram certainly won’t be reliable end-to-end if it is egressing a UDP proxy on the server side to talk to a traditional UDP server. 

David, I’m still not convinced by the scenario you present here. My initial reaction is that an intermediary (which will know it’s an intermediary here) will easily know that it must send datagrams that are too large as capsules, not quic datagrams.

The example of MTU discovery here seems to be artificially wanting to discover a particular MTU in the middle of the intermediaries, which include H2 hops. Why would a client need to care about that? My assumption is that MTU discovery is about knowing what size packet I can successfully send end to end. If the datagram is too large after it leaves the proxy, that’s what I want to detect. The fact that there’s some intermediary that can has a crazy small link between it and the next hop doesn’t matter, because it can successfully send my packets to the next hop in a capsule.

Even if what you’re trying to detect is “what’s my max size I can send without having to use a reliable capsule” in order to make sure you don’t need to add unnecessary reliability, this new capsule doesn’t meaningfully help you. The only time it wouldn’t be clear and you’d have a loss of information is when at least one intermediary is using a version that doesn’t support datagrams, like H1 or H2. In that case, your packets are necessarily going over an extra reliable link anyhow, so the deployment clearly doesn’t care that much about adding extra reliability to datagrams. In the example you give, the client itself is using H2 to talk to the proxy, so it really shouldn’t care.

If there’s some deployment that actually *needs* to have this signal in a mixed H2/h3 environment that couldn’t simply put the datagrams in capsules rather than dropping, I’d be quite curious to hear it. But so far, I’m not seeing justification. 

Tommy

> On Oct 13, 2021, at 6:03 PM, Lucas Pardue <[email-masked]> wrote:
> 
> 
> 
> 
>> On Thu, Oct 14, 2021 at 1:46 AM David Schinazi <[email-masked]> wrote:
>> Let me attempt to summarize an offline discussion I had with Ben on this: in the CONNECT-IP world, if we want to tunnel IPv6 we'll need to support 1280-byte datagrams. This can become an issue if there's an h3 hop along the way that has a low MTU such that this h3 hop can only send QUIC DATAGRAM frames of length < 1280 (let's say 1000 for simplicity). In the main deployment model for CONNECT-IP, the h3 leg will be on the client-to-first-intermediary hop, so the client can choose to send QUIC initials of length 1330 to ensure that if the MTU is too small the client will use h2 instead of h3 on the first hop. The scenario where things aren't solvable by the client nor proxy is when we have a setup like this: client ---- h2 ---- intermediary 1 ---- h3 ---- intermediary 2 ---- h2 ---- proxy where the h3 leg has a datagram MTU of 1000. In this example, when the intermediaries receive a DATAGRAM capsule over h2, they need to decide whether to send it as a DATAGRAM capsule or a DATAGRAM frame over h3. If the incoming datagram is of length 1280, that means the intermediary needs to either drop the datagram, or send it as a capsule. But different use cases might prefer different outcomes: for example if the endpoints are performing DPLPMTUD then having the intermediary make their probes reliable will break DPLPMTUD. But if the endpoints have a minimum required MTU (such as 1280 in IPv6) then not making it reliable will break tunneled IPv6 MTU requirements. So because the intermediary isn't aware of which is which, we need a way for the endpoints to tell the intermediary whether to drop/DATAGRAM frame or to capsule. Having an opaque RELIABLE_DATAGRAM capsule fits the bill for that signal.
>> 
>> TLDR (of that long ramble, my apologies) is that while most use cases don't need RELIABLE_DATAGRAM, you can construct some uncommon deployments that won't work well without it. Because this adds virtually no complexity, I think it makes sense to add RELIABLE_DATAGRAM to the core HTTP Datagrams draft.
>> 
>> Thoughts?
> 
> I might be convinced to support the addition of a capsule that does such a thing but I don't like the proposed name much. Something like SEALED_DATAGRAM seems a bit more accurate in so far as intermediaries cannot meddle with the capsule.
> 
> Lucas
> -- 
> Masque mailing list
> [email-masked]

---

From: Ben Schwartz <[email-masked]>
Date: Thu, 14 Oct 2021 11:29:54 -0400

On Wed, Oct 13, 2021 at 11:04 PM Tommy Pauly <[email-masked]> wrote:

> The fact that there’s some intermediary that can has a crazy small link
> between it and the next hop doesn’t matter, because it can successfully
> send my packets to the next hop in a capsule.
>

It sounds like you're proposing that the intermediary encapsulate datagrams
that are too large for the next-hop MTU.  This not currently allowed:

> If the HTTP Datagram is too large to fit in a DATAGRAM frame (for example
because the path MTU of that QUIC connection is too low or if the maximum
UDP payload size advertised on that connection is too low), the
intermediary SHOULD drop the HTTP Datagram instead of converting it to a
DATAGRAM capsule. This preserves the end-to-end unreliability
characteristic that methods such as Datagram Packetization Layer Path MTU
Discovery (DPLPMTUD) depend on [RFC8899]. An intermediary that converts
QUIC DATAGRAM frames to DATAGRAM capsules allows HTTP Datagrams to be
arbitrarily large without suffering any loss; this can misrepresent the
true path properties, defeating methods such a DPLPMTUD.

---

From: Tommy Pauly <[email-masked]>
Date: Thu, 14 Oct 2021 08:52:35 -0700

> On Oct 14, 2021, at 8:29 AM, Ben Schwartz <[email-masked]> wrote:
> 
> 
> 
> On Wed, Oct 13, 2021 at 11:04 PM Tommy Pauly <[email-masked] <mailto:[email-masked]>> wrote:
> The fact that there’s some intermediary that can has a crazy small link between it and the next hop doesn’t matter, because it can successfully send my packets to the next hop in a capsule.
> 
> It sounds like you're proposing that the intermediary encapsulate datagrams that are too large for the next-hop MTU.  This not currently allowed:
> 
> > If the HTTP Datagram is too large to fit in a DATAGRAM frame (for example because the path MTU of that QUIC connection is too low or if the maximum UDP payload size advertised on that connection is too low), the intermediary SHOULD drop the HTTP Datagram instead of converting it to a DATAGRAM capsule. This preserves the end-to-end unreliability characteristic that methods such as Datagram Packetization Layer Path MTU Discovery (DPLPMTUD) depend on [RFC8899]. An intermediary that converts QUIC DATAGRAM frames to DATAGRAM capsules allows HTTP Datagrams to be arbitrarily large without suffering any loss; this can misrepresent the true path properties, defeating methods such a DPLPMTUD.

Right, I’m suggesting that a deployment that knows it has mixed protocol intermediaries should be allowed to covert DATAGRAMs into datagram capsules, up to a configured max limit. It may not want to allow packets that are bigger than it knows would ever reasonably work, but if you have a deployment where you know you have a small MTU on the H3 intermediary link, you could decide to not drop the packets.

Tommy

---

From: Ben Schwartz <[email-masked]>
Date: Thu, 14 Oct 2021 12:27:47 -0400

On Thu, Oct 14, 2021 at 11:52 AM Tommy Pauly <[email-masked]> wrote:

> It sounds like you're proposing that the intermediary encapsulate
> datagrams that are too large for the next-hop MTU.  This not currently
> allowed:
>
> > If the HTTP Datagram is too large to fit in a DATAGRAM frame (for
> example because the path MTU of that QUIC connection is too low or if the
> maximum UDP payload size advertised on that connection is too low), the
> Right, I’m suggesting that a deployment that knows it has mixed protocol
> intermediaries should be allowed to covert DATAGRAMs into datagram
> capsules, up to a configured max limit. It may not want to allow packets
> that are bigger than it knows would ever reasonably work, but if you have a
> deployment where you know you have a small MTU on the H3 intermediary link,
> you could decide to not drop the packets.
>

It sounds like you're proposing that origins would have to manually
configure an "MTU floor" on all H3 intermediaries that might be used to
reach them.  This cutoff would presumably be different for different
":protocol" values (e.g. webtransport vs. CONNECT-UDP), and possibly even
for different requests (e.g. CONNECT-IP address family).

I would highly prefer an arrangement where intermediaries are
":protocol"-agnostic, which seems both simpler and more flexible.