Subject: [Masque] Capsules

From: "Martin Thomson" <[email-masked]>
Date: Tue, 02 Nov 2021 17:24:40 +1100

I've just reacquainted myself with the ever-shifting content of draft-ietf-masque-h3-datagram (-05 specifically).

My conclusion: this document is about 10x longer and more complex than it needs to be and maybe three or four times more complex than it could be if you accept some of the underlying assumptions.

---
The source of this added complexity is a desire to construct an abstract framework that allows intermediaries to participate in the protocols that this document attempts to facilitate.

The key functional role for an intermediary here is for it to take datagrams and forward them in a way that best suits the HTTP version in use, allowing the same datagram to traverse different versions.  Forwarding in HTTP/3 wants to use native DATAGRAM frames, whereas fowarding elsewhere needs to use the stream established by extended CONNECT.  The goal in creating a generic framework was to enable some common handling between different protocols (UDP proxying, IP proxying, WebTransport).

This is a bit of narrow use case.   An intermediary that understands that a capsule-based protocol can take datagrams from a stream of capsules and send them in DATAGRAM frames.  In the opposite direction, they can fold DATAGRAM frames into streams.

This forwarding behaviour depends on the intermediary understanding that the protocol in use is a capsule-based protocol.  This is done by looking at the :protocol pseudo header field on the extended CONNECT and looking that up in a list of known capsule-based protocols.  This is necessary because not all protocols that use extended CONNECT use capsules.  After all, the only use of extended CONNECT that currently exists, websockets, doesn't.

That's not so far removed from a system where the intermediary understands the contents of different CONNECT-based protocols that do not share a framework.  It is possible to have different protocols that use extended CONNECT with shared design elements that maximize code reuse at intermediaries WITHOUT a framework like this.  Those protocols can all use the same format, name, and registry if that helps maximize code reuse.  At that point that is functionally indistinguishable from the system that uses a framework.

>From this perspective, a framework like this becomes something of an academic exercise.  A lot of work is done to document the framework and describe its properties, but as the effect on deployed code is non-existent, it has no real effect on what people do.  And it comes with risks, like leading people who build intermediaries to start making bad assumptions about how extended CONNECT works generally.

The editors of the WebTransport over HTTP/2 draft are currently debating whether to propose the use of capsules there.  There, we have some interesting design constraints that might be cause to use a different format.  One option that is being considered is reusing design elements from QUIC to make the protocol easier to implement.  I don't think that we should feel constrained to use capsules, except to the extent that it might be nice to make implementation in intermediaries easier.

---
I'd like to propose a different approach:

1. When datagram support is negotiated at the QUIC transport layer, HTTP uses datagrams by associating them with request streams.  This uses the system described in the draft (prepending the stream id divided by four).

2. Different uses of datagrams need to be individually negotiated using settings.  (WebTransport negotiates use of both datagrams and extended CONNECT with a single setting.)

3. Each of the protocols we have describe the use of capsules.  Separately.

That's it.  It's perhaps a shade minimal, but it is sufficient.

I can see the advantage of a shared registry for capsule types.  That has some real advantages in terms of code reuse.  This document might be a good home for doing a little unification of the clerical stuff.  I'm OK with doing that (this is the 10x saving -> 3x saving I alluded to), but it does not need the supporting machinery in the current draft.  It just needs to make a registry, describe the generic capsule format and define the DATAGRAM capsule and forwarding rules for that capsule.  The rules that talk about opaque capsules and whatnot is unnecessary.

---
This would mean losing all the text about datagram contexts with their format types.  I can't see any justification for a generic framework for those functions.  I can see protocols in which these might be useful constructs.  However, use of those constructs would benefits from closer tuning of the mechanisms to the problem domain, something that a framework cannot do.

draft-schwartz-masque-h3-datagram-ping proposes using the format type to create a separation between that and "other" uses of datagrams.  It's important to note that identical treatment at intermediaries is a necessary feature for that draft, but I don't think that is necessary.  It also depends on datagram contexts being used, which is a big assumption as they are effectively pointless for many protocols.  WebTransport doesn't seem inclined to want them.  Should WebTransport want this, it has to pay a pretty high price (>=1 byte on every datagram sent), but that should be something that protocol decides.  Maybe that protocol can come up with a better means of distinguishing datagrams that results in less overhead.

Earlier, it was suggested that datagram contexts might be used for ECN signaling.  At the time, it seemed clever.  But now I realize that it is more like one-step forward, four-steps backwards compression.

---
Sorry for the long missive.  I hope that I was able to capture the rationale so we can discuss this next week in more detail.

Regards,
Martin

---

From: Ben Schwartz <[email-masked]>
Date: Tue, 2 Nov 2021 11:20:30 -0400

It seems like a lot of these concerns are about document refactoring.  For
example, the special behaviors for HTTP 2<->3 intermediaries could be
extracted into a separate document and described as an optimization, but
that's not functionally different from the current text.

However, you also identified some technical issues that we should explore.

On Tue, Nov 2, 2021 at 2:25 AM Martin Thomson <[email-masked]> wrote:
...

> This forwarding behaviour depends on the intermediary understanding that
> the protocol in use is a capsule-based protocol.  This is done by looking
> at the :protocol pseudo header field on the extended CONNECT and looking
> that up in a list of known capsule-based protocols.  This is necessary
> because not all protocols that use extended CONNECT use capsules.  After
> all, the only use of extended CONNECT that currently exists, websockets,
> doesn't.
>

This is a good observation, and we can correct it.  We can define an
additional request header to indicate that the datagram+capsule system is
in use (like the old Transfer-Encoding header), to decouple this from the
:protocol.

The editors of the WebTransport over HTTP/2 draft are currently debating
> whether to propose the use of capsules there.  There, we have some
> interesting design constraints that might be cause to use a different
> format.  One option that is being considered is reusing design elements
> from QUIC to make the protocol easier to implement.  I don't think that we
> should feel constrained to use capsules, except to the extent that it might
> be nice to make implementation in intermediaries easier.
>

I think the purpose of a standards body is to avoid producing multiple
similar but incompatible systems if it can be avoided.  If WebTransport has
additional requirements, MASQUE might be able to adopt a subset of its
framing.

---

From: David Schinazi <[email-masked]>
Date: Tue, 2 Nov 2021 10:27:42 -0700

Hi Martin, thanks for reading the latest draft. Responses inline.

David

On Mon, Nov 1, 2021 at 11:25 PM Martin Thomson <[email-masked]> wrote:

> I've just reacquainted myself with the ever-shifting content of
> draft-ietf-masque-h3-datagram (-05 specifically).
>
> My conclusion: this document is about 10x longer and more complex than it
> needs to be and maybe three or four times more complex than it could be if
> you accept some of the underlying assumptions.
>
> ---
> The source of this added complexity is a desire to construct an abstract
> framework that allows intermediaries to participate in the protocols that
> this document attempts to facilitate.
>

Speaking for myself as co-author of the draft, constructing an abstract
framework was never my desire. I care about the properties of the system
we're building, not about abstract concepts.

The key functional role for an intermediary here is for it to take
> datagrams and forward them in a way that best suits the HTTP version in
> use, allowing the same datagram to traverse different versions.  Forwarding
> in HTTP/3 wants to use native DATAGRAM frames, whereas fowarding elsewhere
> needs to use the stream established by extended CONNECT.  The goal in
> creating a generic framework was to enable some common handling between
> different protocols (UDP proxying, IP proxying, WebTransport).
>

Agreed. My implementation supports both WebTransport and CONNECT-UDP, and
there's definitely value in reusing the code that sends and parses
datagrams and capsules.

This is a bit of narrow use case.   An intermediary that understands that a
> capsule-based protocol can take datagrams from a stream of capsules and
> send them in DATAGRAM frames.  In the opposite direction, they can fold
> DATAGRAM frames into streams.
>
> That's not so far removed from a system where the intermediary understands
> the contents of different CONNECT-based protocols that do not share a
> framework.  It is possible to have different protocols that use extended
> CONNECT with shared design elements that maximize code reuse at
> intermediaries WITHOUT a framework like this.  Those protocols can all use
> the same format, name, and registry if that helps maximize code reuse.  At
> that point that is functionally indistinguishable from the system that uses
> a framework.
>

I'm confused here. In order to reuse code, it makes sense for multiple
protocols to share the capsule TLV wire format and the IANA capsule type
registry. Since protocols will share the IANA registry, you need to define
it in this doc. At that point it makes sense to also define the capsule TLV
format here as well since we see value in reusing it. Copy-pasting the TLV
format in every protocol document doesn't help us. If you don't like the
word "framework", then don't use that word?
draft-ietf-masque-h3-datagram-05 doesn't contain the word framework,
because it doesn't attempt to create a framework. Only to define pieces
that can be reused.

>From this perspective, a framework like this becomes something of an
> academic exercise.  A lot of work is done to document the framework and
> describe its properties, but as the effect on deployed code is
> non-existent, it has no real effect on what people do.  And it comes with
> risks, like leading people who build intermediaries to start making bad
> assumptions about how extended CONNECT works generally.
>

I fully agree with you that we would get the exact same software if we took
some parts out of this document and copied them verbatim into every
protocol that uses datagrams. I don't see that as helpful though.

> be nice to make implementation in intermediaries easier.

You shouldn't feel constrained to use capsules, you should only use them if
they provide value. Out of curiosity: what are the design constraints that
would steer you to a different protocol?

---
> 1. When datagram support is negotiated at the QUIC transport layer, HTTP
> uses datagrams by associating them with request streams.  This uses the
> system described in the draft (prepending the stream id divided by four).
>
> 2. Different uses of datagrams need to be individually negotiated using
> settings.  (WebTransport negotiates use of both datagrams and extended
> CONNECT with a single setting.)
>

What does "different uses of datagrams" mean? If one of my backends
supports CONNECT-UDP, why do I need a SETTING on the client-frontend leg?

This is just moving rocks, and increases the total number of lines of spec
written. What's the benefit of copy-pasting something in multiple documents
if they all depend on a common draft?

>
> I can see the advantage of a shared registry for capsule types.  That has
> some real advantages in terms of code reuse.  This document might be a good
> home for doing a little unification of the clerical stuff.  I'm OK with
> doing that (this is the 10x saving -> 3x saving I alluded to), but it does
> not need the supporting machinery in the current draft.  It just needs to
> make a registry, describe the generic capsule format and define the
> DATAGRAM capsule and forwarding rules for that capsule.  The rules that
> talk about opaque capsules and whatnot is unnecessary.
>

The introduction of opaque and transparent capsules was done as an
editorial clarification to help readers understand the concept of capsules.
We can remove those terms, but we still need the concept of injecting a
datagram into the stream when going from h3 to h2.

---
> This would mean losing all the text about datagram contexts with their
> format types.  I can't see any justification for a generic framework for
> those functions.  I can see protocols in which these might be useful
> constructs.  However, use of those constructs would benefits from closer
> tuning of the mechanisms to the problem domain, something that a framework
> cannot do.
>
> draft-schwartz-masque-h3-datagram-ping proposes using the format type to
> create a separation between that and "other" uses of datagrams.  It's
> important to note that identical treatment at intermediaries is a necessary
> feature for that draft, but I don't think that is necessary.  It also
> depends on datagram contexts being used, which is a big assumption as they
> are effectively pointless for many protocols.  WebTransport doesn't seem
> inclined to want them.  Should WebTransport want this, it has to pay a
> pretty high price (>=1 byte on every datagram sent), but that should be
> something that protocol decides.  Maybe that protocol can come up with a
> better means of distinguishing datagrams that results in less overhead.
>

I think the value here is the definition of extensions that work across
multiple protocols. draft-schwartz-masque-h3-datagram-ping is a perfect
example of that: there's been discussion in WebTransport of adding a
JavaScript API at some later date that would tell the JavaScript what the
datagram MTU is. Having this MTU information would also be useful for
CONNECT-UDP and CONNECT-IP. What you're suggesting is that every single
protocol redefines this. That's not reducing any complexity, just kicking
the can down the road.

Earlier, it was suggested that datagram contexts might be used for ECN
> signaling.  At the time, it seemed clever.  But now I realize that it is
> more like one-step forward, four-steps backwards compression.
>
> ---
> Sorry for the long missive.  I hope that I was able to capture the
> rationale so we can discuss this next week in more detail.
>

Can you clarify what your goal is? It sounds like you would prefer for this
document to be shorter. I agree with that goal, but it shouldn't be done in
a vacuum - we should look at the total length of the MASQUE/WebTransport
documents.

Regards,
> Martin
>
> --
> Masque mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/masque
>

---

From: Alan Frindell <[email-masked]>
Date: Tue, 2 Nov 2021 22:04:43 +0000

> Out of curiosity: what are the design constraints that would steer you to a different protocol?

For context on the WebTransport over H2 potential use of Capsules, see https://github.com/ietf-wg-webtrans/draft-webtransport-http2/issues/29

WebTransport over H2 in its current form defines “WebTransport Frames”, which are identical to Capsules (TLV, using QUIC VarInts, https://ietf-wg-webtrans.github.io/draft-webtransport-http2/draft-ietf-webtrans-http2.html#name-webtransport-frames).

Since WebTransport over H2 can mostly be abstracted as “Some QUIC frames over a byte-stream”, we were discussing on the issue whether we could re-use existing QUIC frame parsers.  Since QUIC frames do not contain a Length field, we discussed putting the Length first in the WebTransport Frame, so it could be stripped before using an existing frame parser.  Alternately, we could remove the generic Frame type altogether (as QUIC has), and just kill the stream when an unknown Frame type is detected.  Either of those two alternatives are incompatible with Capsules as currently specified.

-Alan

---

From: "Martin Thomson" <[email-masked]>
Date: Wed, 03 Nov 2021 10:24:03 +1100

> additional request header to indicate that the datagram+capsule system 
> is in use (like the old Transfer-Encoding header), to decouple this 
> from the :protocol.

It's good to know that we disagree.  You seem to think that the addition of another protocol layer is desirable.  I do not.  That shouldn't be a surprise: I asked if we could simplify things; your response was to propose making them more complex.

What I'm saying is that it would be better to NOT have a generic framework that exists to serve the intermediation use case.  Instead, intermediaries can understand all the protocols if they want to interact with the protocol more extensively.  That is the purpose of using settings to negotiate support (a decision already made).  Of course, that negotiation is only necessary if the protocol wants to escape the confines of a single stream.

Document factoring is secondary to that, of course.

> similar but incompatible systems if it can be avoided.  If WebTransport 
> has additional requirements, MASQUE might be able to adopt a subset of 
> its framing.

I don't agree that that is the purpose of a standards body at all.  We're here so that different parties can agree in their (and hopefully everybody's) shared interest.  What you might be looking for here is good engineering practice, which says that you don't want too much diversity in how the same problem is solved in different contexts.

For this example, we have overlapping contexts.  HTTP/2 puts length fields ahead of type fields (fixed LTVs), Capsules have varint TLVs, QUIC has TVs.  These are not naturally compatible and so we have to decide how to align the format with these existing things.  Of course, this is an easy thing to answer if you are predisposed toward the addition of another protocol layer.

---

From: "Martin Thomson" <[email-masked]>
Date: Wed, 03 Nov 2021 10:47:35 +1100

On Wed, Nov 3, 2021, at 04:27, David Schinazi wrote:
> protocols to share the capsule TLV wire format and the IANA capsule 
> type registry. Since protocols will share the IANA registry, you need 
> to define it in this doc. At that point it makes sense to also define 
> the capsule TLV format here as well since we see value in reusing it. 
> Copy-pasting the TLV format in every protocol document doesn't help us. 
> If you don't like the word "framework", then don't use that word? 

I'm curious, how much of a hit to reuse would it be if:

1. The protocols had their own registries?

2. The protocols used different framing layouts (TLV vs. LTV or varint vs. fixed)?

My guess is that the incremental cost is tiny.  I know that I never reuse framing code from an old project, because it's generally just as easy to copy-paste or even just retype that tiny bit of code.  And you always have different rules regarding which frame/capsule/packet types are permitted in different contexts, making reuse more complex.

> You shouldn't feel constrained to use capsules, you should only use 
> them if they provide value. Out of curiosity: what are the design 
> constraints that would steer you to a different protocol?

I hinted at these in my response to Ben, but we're essentially putting QUIC frames into the data stream, which has some interesting implications.  There are a few options being tossed around:

* Encapsulate QUIC frames by adding a leading length field (which would make them look like LTV format, mostly)
* Use capsules and do some sort of mapping of QUIC frame types into that space (mapping TBD)
* Use HTTP/2 framing (I don't think that this is serious)

> supports CONNECT-UDP, why do I need a SETTING on the client-frontend 

That's a good question.  I started a thread in HTTPbis about that exact question.  The answer being "I don't know" or maybe "it depends".

> editorial clarification to help readers understand the concept of 
> capsules. We can remove those terms, but we still need the concept of 
> injecting a datagram into the stream when going from h3 to h2.

I think that the only thing you need is the special treatment for datagram capsules.

> JavaScript API at some later date that would tell the JavaScript what 
> the datagram MTU is. Having this MTU information would also be useful 
> for CONNECT-UDP and CONNECT-IP. What you're suggesting is that every 
> single protocol redefines this. That's not reducing any complexity, 
> just kicking the can down the road.

I don't think that I was suggesting kicking the can down the road. I was suggesting that building a framework is unnecessary, especially when you have just one use case for it.

If MTU probing is useful - and I have some reservations about that - probing can be added to this draft.  Or it could be added in a separate document as a capsule type with identical rules for forwarding as datagram capsules, though that creates some uncertainty that it isn't fully supported on the path.  The whole datagram contexts and datagram formats framework is extraordinarily heavyweight for those cases that aren't using it (i.e., most of them), even if it makes this one thing a tiny bit easier.

> Can you clarify what your goal is? It sounds like you would prefer for 
> this document to be shorter. I agree with that goal, but it shouldn't 
> be done in a vacuum - we should look at the total length of the 
> MASQUE/WebTransport documents.

My goal is to make it easier to implement the whole suite.  It starts here, but I expect the simplifications to flow on to the other documents also.  It's conceptually easier to reason about this when each protocol is self-contained.  I think that you can get that - and shared design elements - without working nearly so hard.

---

From: Victor Vasiliev <[email-masked]>
Date: Tue, 2 Nov 2021 19:56:37 -0400

On Tue, Nov 2, 2021 at 7:24 PM Martin Thomson <[email-masked]> wrote:

> What I'm saying is that it would be better to NOT have a generic framework
> that exists to serve the intermediation use case.  Instead, intermediaries
> can understand all the protocols if they want to interact with the protocol
> more extensively.  That is the purpose of using settings to negotiate
> support (a decision already made).  Of course, that negotiation is only
> necessary if the protocol wants to escape the confines of a single stream.
>

>From what I understand, the only reason you need to negotiate a SETTINGS
parameter is if you need to use native HTTP/3 datagrams.  There's nothing
fundamentally preventing anyone from using any capsule protocols in the
current iterations of H2/H3, provided that you support extended CONNECT.

---

From: David Schinazi <[email-masked]>
Date: Tue, 2 Nov 2021 17:17:32 -0700

> 2. The protocols used different framing layouts (TLV vs. LTV or varint vs.

Right now I have one class called CapsuleParser that is used for both
WebTransport and CONNECT-UDP. If either (1) or (2), then I'd need separate
classes (or some non-trivial inheritance) which leads to the usual set of
bugs where you modify one class and forget to update the other.

My guess is that the incremental cost is tiny.  I know that I never reuse
> framing code from an old project, because it's generally just as easy to
> copy-paste or even just retype that tiny bit of code.  And you always have
> different rules regarding which frame/capsule/packet types are permitted in
> different contexts, making reuse more complex.
>

I'm not too worried about the incremental cost - copy-paste is easy - but I
am concerned about the maintenance costs that come from code duplication.

> I hinted at these in my response to Ben, but we're essentially putting
> QUIC frames into the data stream, which has some interesting implications.
> There are a few options being tossed around:
>

Defining one new capsule type per QUIC frame type you care about sounds
fairly straightforward to me, am I missing something?

> That's a good question.  I started a thread in HTTPbis about that exact
> question.  The answer being "I don't know" or maybe "it depends".
>

Sounds like we agree. These are just two different ways to frame the
discussion:
1) all capsules must be forwarded without messing with them, except datagram
2) define concepts of opaque vs transparent, with datagram being the only
transparent one
This is just an editorial matter.

> I don't think that I was suggesting kicking the can down the road. I was
> suggesting that building a framework is unnecessary, especially when you
> have just one use case for it.
>

I still don't think there's any framework happening here, just reuse of
spec text and code between WebTransport, CONNECT-UDP and CONNECT-IP.

If MTU probing is useful - and I have some reservations about that -
> probing can be added to this draft.  Or it could be added in a separate
> document as a capsule type with identical rules for forwarding as datagram
> capsules, though that creates some uncertainty that it isn't fully
> supported on the path.

I think this is the key property we're after here: being able to deploy
extensions by modifying the endpoints but without having to modify
intermediaries.

> The whole datagram contexts and datagram formats framework is
> extraordinarily heavyweight for those cases that aren't using it (i.e.,
> most of them), even if it makes this one thing a tiny bit easier.
>
> My goal is to make it easier to implement the whole suite.  It starts
> here, but I expect the simplifications to flow on to the other documents
> also.  It's conceptually easier to reason about this when each protocol is
> self-contained.  I think that you can get that - and shared design elements
> - without working nearly so hard.
>

What are your thoughts on extensibility? Most of the perceived complexities
in the design stem from ensuring future-proof extensibility, so perhaps we
need to agree on how much extensibility we want before deciding how to
implement it.

---

From: Eric Rescorla <[email-masked]>
Date: Sun, 7 Nov 2021 14:08:47 -0800

On Mon, Nov 1, 2021 at 11:25 PM Martin Thomson <[email-masked]> wrote:

> This is a bit of narrow use case.   An intermediary that understands that
> a capsule-based protocol can take datagrams from a stream of capsules and
I'd actually like to push on this use case a bit.

To a first order, there are two ways one can have an HTTP
intermediary:

1. It's associated with the client (i.e., a forward proxy)
2. It's associated with the server (i.e., a reverse proxy)

To a first order, we no longer need to worry about the first type of
proxy: because we are using encrypted transport, if that kind of proxy
is in play, we just tunnel through that (effectively using RFC 2817
CONNECT) so it's irrelevant.

This leaves us with the situation where the proxy is associated with
the server. However, unlike the forward proxy, this is invisible to
the client; it's just an internal implementation detail whether the
server is implemented as one device, multiple devices with different
versions of HTTP in between them, or indeed multiple devices with some
totally non-HTTP protocol in between them.

For this reason, protocol machinery which is intended to deal with the
case where there are multiple chained HTTP elements seems like added
complexity that we're imposing on the client for a case which is
fundamentally a local matter for the server. I think if we were
instead to just assume that the client was talking directly to the
server and the server would sort out this kind of internal detail,
life would be much simpler and we should consider that.

-Ekr