Subject: [Masque] MASQUE extensibility

From: "Christopher Wood" <[email-masked]>
Date: Fri, 04 Dec 2020 12:18:43 -0800

One point raised during our IETF 109 meeting was extensibility. A number of different extensions have already been discussed, including: IP header compression, ECN signals, and QUIC-aware proxy support. It seems clear that the MASQUE use-cases require some amount of extensibility in the core mechanism. What we’d like to determine is how much extensibility is needed.

Given that our charter is somewhat vague on what extensions are in scope [1], there seem to be (at least) three questions we should answer if we are to embark on this extension work:

1) Do we want CONNECT-UDP and a future mechanism for IP proxying to be extensible?
2) If yes, which parts of each protocol need extensibility, and to what extent?
3) If yes, do we want to adopt these extensions as distinct documents? (An ECN signal, for example, might be included as an extension in the CONNECT-UDP document, or it might be part of a separate document. Conversely, extensions for QUIC-aware proxying seem likely to be a separate document.)

We'd like to hear answers to these three questions from the WG. Converging here will help us better determine what is in scope going forward. To that end, please share your thoughts on these questions before December 18. We'll assess and see where we are at that time.

Thanks,
Chris and Eric

[1] Relevant text in the charter (https://datatracker.ietf.org/doc/charter-ietf-masque/) reads as follows:

The primary goal of this working group is to develop mechanism(s) that allow configuring and concurrently running multiple proxied stream- and datagram-based flows inside an HTTPS connection. These mechanism(s) are collectively called MASQUE. ***The group will specify HTTP and/or HTTP/3 extensions to enable this functionality.***

(emphasis ours)

---

From: David Schinazi <[email-masked]>
Date: Fri, 4 Dec 2020 17:54:22 -0800

My personal thoughts:
1) Absolutely! Both CONNECT-UDP and IP proxying need to be extensible,
discussion has already surfaced multiple desired extensions.
2) Since we're building over HTTP, we get headers on request and response
so that's already a great extension point. Additionally, from the ideas
we've seen so far, we may need a way to communicate extension information
during the lifetime of the request stream (as opposed to only at the start)
so the wire format of the bidirectional stream might need an extension
point. We may need the ability of associating multiple datagram flow IDs
with one request, for example to indicate which UDP packets had ECN-CE set.
3) I think that separate drafts makes things easier and prevents extensions
from delaying the main deliverables

Cheers,
David

On Fri, Dec 4, 2020 at 12:19 PM Christopher Wood <[email-masked]>
wrote:

> One point raised during our IETF 109 meeting was extensibility. A number
> of different extensions have already been discussed, including: IP header
> compression, ECN signals, and QUIC-aware proxy support. It seems clear that
> the MASQUE use-cases require some amount of extensibility in the core
> mechanism. What we’d like to determine is how much extensibility is needed.
>
> Given that our charter is somewhat vague on what extensions are in scope
> [1], there seem to be (at least) three questions we should answer if we are
> to embark on this extension work:
>
> 3) If yes, do we want to adopt these extensions as distinct documents? (An
> ECN signal, for example, might be included as an extension in the
> CONNECT-UDP document, or it might be part of a separate document.
> Conversely, extensions for QUIC-aware proxying seem likely to be a separate
> document.)
>
> We'd like to hear answers to these three questions from the WG. Converging
> here will help us better determine what is in scope going forward. To that
> end, please share your thoughts on these questions before December 18.
> We'll assess and see where we are at that time.
>
> Thanks,
> Chris and Eric
>
> [1] Relevant text in the charter (
> https://datatracker.ietf.org/doc/charter-ietf-masque/) reads as follows:
>
> The primary goal of this working group is to develop mechanism(s) that
> allow configuring and concurrently running multiple proxied stream- and
> datagram-based flows inside an HTTPS connection. These mechanism(s) are
> collectively called MASQUE. ***The group will specify HTTP and/or HTTP/3
> extensions to enable this functionality.***
>
> (emphasis ours)
>
> --
> Masque mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/masque
>

---

From: Lucas Pardue <[email-masked]>
Date: Sat, 5 Dec 2020 04:59:26 +0000

Hey,

Please see my points in line.

On Fri, 4 Dec 2020, 20:19 Christopher Wood, <[email-masked]> wrote:

> [1],

I don't find the point you emphasise as too vague. My comprehension, which
may well differ from others, is that in the MASQUE WG we are developing a
tunneling design that is an extension to another protocol. We picked HTTP.
This is in contrast to a design which builds a new application protocol on
top of TCP or QUIC.

Whether we make the new thing itself extensible is very interesting but
tangential to the point you highlight IMO. It is a bit vague if we are
chartered to build with extensibility in mind (using HTTP makes preventing
extensibility a difficult cat to keep in the bag), and/or if we should
actively develop extensions ourselves. And so I thank the chairs for
reading the WG and kicking off this thread!

there seem to be (at least) three questions we should answer if we are to
> embark on this extension work:
>

That horse has already bolted.

2) If yes, which parts of each protocol need extensibility, and to what
> extent?

All the bits that are already extensible.

I wonder if this question is actually, "do we want to develop extensions on
top of the extensibility.

> Conversely, extensions for QUIC-aware proxying seem likely to be a separate

Separate docs to start. There's nothing stopping things getting merged
later if we discover the friction is low.

Cheers
Lucas

---

From: Eric Rescorla <[email-masked]>
Date: Sat, 5 Dec 2020 12:08:03 -0800

I basically concur with David here.

> so that's already a great extension point.
>

Exactly.

Additionally, from the ideas we've seen so far, we may need a way to
> communicate extension information during the lifetime of the request stream
> (as opposed to only at the start) so the wire format of the bidirectional
> stream might need an extension point. We may need the ability of
> associating multiple datagram flow IDs with one request, for example to
> indicate which UDP packets had ECN-CE set.
>

Agreed. The question, I think, is if we want to bear that cost with every
datagram, even if it's just a single marker byte. If not -- and if this is
a common enough case -- maybe we want to have two datagram frame types in
QUIC.

> from delaying the main deliverables

Agreed.

-Ekr

>> 3) If yes, do we want to adopt these extensions as distinct documents?
>> (An ECN signal, for example, might be included as an extension in the
>> We'd like to hear answers to these three questions from the WG.
>> Converging here will help us better determine what is in scope going
>> forward. To that end, please share your thoughts on these questions before
>> December 18. We'll assess and see where we are at that time.
>>
>> Thanks,
>> Chris and Eric
>>

---

From: "Martin Thomson" <[email-masked]>
Date: Mon, 07 Dec 2020 20:52:17 +1100

On Sun, Dec 6, 2020, at 07:08, Eric Rescorla wrote:
> > Additionally, from the ideas we've seen so far, we may need a way to communicate extension information during the lifetime of the request stream (as opposed to only at the start) so the wire format of the bidirectional stream might need an extension point. We may need the ability of associating multiple datagram flow IDs with one request, for example to indicate which UDP packets had ECN-CE set.
> 
> Agreed. The question, I think, is if we want to bear that cost with 
> every datagram, even if it's just a single marker byte. If not -- and 
> if this is a common enough case -- maybe we want to have two datagram 
> frame types in QUIC.

So I agree here with David, but I'm not sure that you (both) have correctly identified the extension points we need.

There is the "framing" on the CONNECT-UDP stream, which might naively contain data plus something else.  It seems to me like this needs the ability to include stuff other than data.  Stealing a byte (or some bits) there might be a good thing.

Once you have that signaling channel available, I don't think that you necessarily need multiple ways to signal inside DATAGRAM frames.  As David suggests, you can use the flow ID to carry some information.  Or, you could define a way to negotiate the use of a flow ID that includes a header.  Or, if you think that is too expensive, then you could look to modify QUIC to add a new frame type.  That seems like an option that should be reserved as a nuclear option.

---

From: Tommy Pauly <[email-masked]>
Date: Mon, 07 Dec 2020 08:45:37 -0800

1) Yes, extensibility for both is important.
2) I think the key metric here is that we ought to design things such that any reasonable use of tunneling/proxying UDP or IP over HTTP/3 can use the primary methods we define (CONNECT-UDP… and CONNECT-IP?) without needing to define yet other methods. I think the current design of CONNECT-UDP satisfies this, but trying to restrict the meaning and mapping of flow IDs would likely harm this extensibility.
3) It sounds like it will be best to have extensions as separate documents. I imagine this working out as some core CONNECT-UDP and “CONNECT-IP” documents that define the most basic elements that any endpoint that understands these methods must support, with optional behavior for different use cases being defined in extension documents.

Regarding charter scope, it seems that if we take the choice to have a core CONNECT-UDP document with separate documents for how to use CONNECT-UDP in different scenarios, that doesn’t violate the charter. The choice to split things up is about improving accessibility and clarity. The charter does not mandate that the WG must only produce the most minimal versions of CONNECT-UDP and CONNECT-IP in single documents, but rather gives room for these to have complete protocol solutions, modulo areas like multicast and server-initiated flows.

Thanks,
Tommy

> On Dec 4, 2020, at 12:18 PM, Christopher Wood <[email-masked]> wrote:
>

---

From: Mirja Kuehlewind <[email-masked]>
Date: Wed, 9 Dec 2020 18:01:29 +0000

1) Yes, every new protocol should be extensible. Lucas and others mentioned that H3 provides already some extensibility, however, as soon as you have send the CONNECT there is not much H3 "left" and therefore we should make sure that we consider options for additional extension mechanisms later in the life time of a CONNECT request.

2) People did mention in-stream (or I would say in-forwarding-association - I  think we need to agree on terminology here to make sure we talk about the same thing) control data. So I think there is information that you want to signal that is associated to a forwarding stream or even a certain packet, where it probably make sense to send this data within the respective forwarding stream, however, there might be other cases where you want to signal something based on some event in the proxy, including negotiation when or before the CONNECT is sent, or general information about proxy state or capabilities which maybe be independent of any active forwarding association. We also need to consider appropriate ways to signal such information and I think all (new) signaling we define should be extensible.

3) I think the extension points itself need to be described in the base spec. I also still think that ECN handling should be part of the base spec as this is an important functionality that is in my view not optional if we ever want to see it used. However we can of course start with separate documents and merge in when ready. Still, we first must agree about the kind of extension points we want to use.

Mirja

On 04.12.20, 21:19, "Masque on behalf of Christopher Wood" <[email-masked] on behalf of [email-masked]> wrote:

    Thanks,
    Chris and Eric

    (emphasis ours)

---

From: Martin Duke <[email-masked]>
Date: Wed, 9 Dec 2020 12:03:44 -0800

Thanks for starting this discussion, Chris.

With my AD hat on, I would personally interpret the charter to allow
CONNECT-UDP to provide all the functionality one might have in a UDP
socket, except those (like multicast) explicitly forbidden. This includes
ECN, DSCP, etc, although the WG is welcome to decide that some or all of
these functions aren't important. As AD, I don't have a strong opinion
today as to whether these end up as one draft or multiple drafts.
Certainly, it makes sense to spell out the design in a separate draft first
and later decide whether or not to integrate it into the base design, as
quicwg did with the spin bit and ECN support.

If the CONNECT-UDP method is designed so as to be literally not extensible
to support these use cases, I would consider that a significant flaw. I
don't think that's the case.

Speaking as an individual, the hardest extensibility problem would seem to
be per-datagram information. There are four different resources we could
use for this:
- an additional QUIC frame type codepoint, with the second type having
additional per-datagram information
- an additional H3 frame type codepoint, with the second type having
additional per-datagram information
- flow ID codepoints, as David suggests. If we want to support DSCP, this
means we need 8 bits per flow just for this (but then, we have 2^62 of
them!)
- Make every H3 datagram frame encode this information -- if we were to
choose this option, we will probably end up using more H3 frame types as
Webtransport, etc. will probably not want the overhead.

To me, the first two are the least profligate, and the second option
(another H3 frame) is the right layer to do it. But none of these resources
are scarce and this is mainly an aesthetic preference.

Broadening from the per-datagram issue, we are likely filling up some sort
of HTTP or H3 registry with whatever extension points we specify, so that
community's opinion is valuable here.

Martin

On Wed, Dec 9, 2020 at 10:01 AM Mirja Kuehlewind <mirja.kuehlewind=
[email-masked]> wrote:

> 1) Yes, every new protocol should be extensible. Lucas and others
> mentioned that H3 provides already some extensibility, however, as soon as
> you have send the CONNECT there is not much H3 "left" and therefore we
> should make sure that we consider options for additional extension
> mechanisms later in the life time of a CONNECT request.
>
> 2) People did mention in-stream (or I would say in-forwarding-association
> - I  think we need to agree on terminology here to make sure we talk about
> the same thing) control data. So I think there is information that you want
> to signal that is associated to a forwarding stream or even a certain
> packet, where it probably make sense to send this data within the
> respective forwarding stream, however, there might be other cases where you
> want to signal something based on some event in the proxy, including
> negotiation when or before the CONNECT is sent, or general information
> about proxy state or capabilities which maybe be independent of any active
> forwarding association. We also need to consider appropriate ways to signal
> such information and I think all (new) signaling we define should be
> extensible.
>
> 3) I think the extension points itself need to be described in the base
> spec. I also still think that ECN handling should be part of the base spec
> as this is an important functionality that is in my view not optional if we
> ever want to see it used. However we can of course start with separate
> documents and merge in when ready. Still, we first must agree about the
> kind of extension points we want to use.
>
> Mirja
>
>
> On 04.12.20, 21:19, "Masque on behalf of Christopher Wood" <
> [email-masked] on behalf of [email-masked]> wrote:
>
>     One point raised during our IETF 109 meeting was extensibility. A
> number of different extensions have already been discussed, including: IP
> header compression, ECN signals, and QUIC-aware proxy support. It seems
> clear that the MASQUE use-cases require some amount of extensibility in the
> core mechanism. What we’d like to determine is how much extensibility is
> needed.
>
>     Given that our charter is somewhat vague on what extensions are in
> scope [1], there seem to be (at least) three questions we should answer if
> we are to embark on this extension work:
>