Subject: HTTP extensions, semantics and HTTP datagrams / MASQUE / WEBTRANS

From: Mark Nottingham <[email-masked]>
Date: Tue, 23 Jun 2020 09:40:40 +1000

Hi all,

The new MASQUE[1] and WEBTRANS[2] working groups both wish to define new protocols using QUIC DATAGRAM frames[3] that are not conveying (existing) HTTP semantics, but are intended to be usable over (at least some versions of) a HTTP connection.

MASQUE's purpose is to proxy UDP and potentially IP (like a VPN) over a HTTP/3 connection. For WEBTRANS, the purpose is to create a browser-exposed, WebSocket-like protocol that allows unreliable delivery (which may also use CONNECT-UDP). These summaries are just my understanding; for a more complete picture, see the respective charters and related WG discussion.

It's not yet clear how or even if these protocols will map into HTTP semantics. Some have proposed something like a CONNECT-UDP method; others seem to favour negotiating a new protocol using SETTINGS or a similar version-specific mechanism directly, avoiding the definition of any generic HTTP semantics.

So far, the most concrete proposal we have is draft-schinazi-quic-h3-datagram,[4] which defines how QUIC DATAGRAM frames are surfaced in HTTP/3 without any mapping them into generic HTTP semantics (like a method, status code, header, etc.). Thus, a specification using them has the choice of defining a generic HTTP extension (like CONNECT-UDP, mapping them into HTTP semantics) or a version-specific extension(like SETTINGS, bypassing generic semantics).

Before these groups get too far down the road, we have an opportunity to give them some guidance about how to extend both HTTP's semantics overall, and specific versions of HTTP -- keeping in mind recent core discussions.[5] 

The most relevant question is whether new, non-HTTP protocols on a HTTP connection should be bootstrapped by generic HTTP semantics (e.g., a method), version-specific semantics (e.g., SETTINGS), or something else. 

There's also a question as to whether something like draft-schinazi-h3-datagram is the right approach, or whether it should also include generic HTTP semantics "bundled in."

Do folks have any thoughts about this? 

Cheers,

1. <https://datatracker.ietf.org/wg/masque/about/>.
2. <https://datatracker.ietf.org/wg/webtrans/about/>.
3. 'An Unreliable Datagram Extension to QUIC', <https://tools.ietf.org/html/draft-ietf-quic-datagram>.
4. <https://tools.ietf.org/html/draft-schinazi-quic-h3-datagram>.
5. <https://github.com/httpwg/http-core/issues/44>.

--
Mark Nottingham   https://www.mnot.net/

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 23 Jun 2020 18:45:41 +1000

My personal .02 on this (and I'd be very curious to hear others' reactions) --

We don't have a good track record of creating extensions or new capabilities that don't map into HTTP semantics. Server Push is probably the poster child here. As such, creating a version-specific extension for DATAGRAMs without mapping them into a method (or whatever) seems like it's leaving an attractive nuisance around; how will it show up in APIs? How will it interact with intermediaries? Etc.

We also don't have a lot of precedent for a HTTP version-specific extension that does something completely separate to the HTTP interaction that's happening on the connection. I'm not sure that it would be something we want to encourage.

All that said, we're not the protocol police; I don't think we should _stop_ this, but I do think that if we want this to succeed, it might work better as something packaged together with CONNECT-UDP (or however it gets expressed in HTTP).

AIUI the one big argument for factoring it out into a separate draft is that MASQUE might want to use it *without* a method (etc.) for IP proxying. So that might be a conversation to have (although that use case has been delayed, I think).

Another potential consumer could be a header on CONNECT (or even other methods?) that allows unreliable delivery but degrades gracefully to reliable delivery on hops that don't provide it. Even then, it seems like such a spec could reference the wire formats, etc., in CONNECT-UDP once that's defined.

Overall - I have a preference for this to be rolled into a CONNECT-UDP (or similar) spec, but can live with it as a separate doc, provided that some thought is put into warning potential users about some risks.

What do other folks think?

> On 23 Jun 2020, at 9:40 am, Mark Nottingham <[email-masked]> wrote:
> 
> Hi all,
>

---

From: Ben Schwartz <[email-masked]>
Date: Wed, 24 Jun 2020 14:30:11 -0400

One direction I think could be interesting is to embrace the Extended
CONNECT as used by WebSocket-H2 (RFC 8441).  If we define ":protocol = tcp"
as a synonym for the old-fashioned CONNECT, then ":protocol = udp" could
represent the basic MASQUE case nicely, and ":protocol = webtransport"
might go naturally as well (if we can't reuse ":protocol = websocket").

The combination of :protocol and transport (H3 vs. H2) might be sufficient
to indicate whether datagrams are allowed.

On Tue, Jun 23, 2020 at 4:49 AM Mark Nottingham <[email-masked]> wrote:

> We don't have a good track record of creating extensions or new
> capabilities that don't map into HTTP semantics. Server Push is probably
> the poster child here. As such, creating a version-specific extension for
> DATAGRAMs without mapping them into a method (or whatever) seems like it's
> leaving an attractive nuisance around; how will it show up in APIs? How
> will it interact with intermediaries? Etc.
>
> We also don't have a lot of precedent for a HTTP version-specific
> extension that does something completely separate to the HTTP interaction
> that's happening on the connection. I'm not sure that it would be something
> we want to encourage.
>
> All that said, we're not the protocol police; I don't think we should
> _stop_ this, but I do think that if we want this to succeed, it might work
> better as something packaged together with CONNECT-UDP (or however it gets
> expressed in HTTP).
>
> AIUI the one big argument for factoring it out into a separate draft is
> that MASQUE might want to use it *without* a method (etc.) for IP proxying.
> So that might be a conversation to have (although that use case has been
> delayed, I think).
>
> Another potential consumer could be a header on CONNECT (or even other
> methods?) that allows unreliable delivery but degrades gracefully to
> reliable delivery on hops that don't provide it. Even then, it seems like
> such a spec could reference the wire formats, etc., in CONNECT-UDP once
> that's defined.
>
> Overall - I have a preference for this to be rolled into a CONNECT-UDP (or
> similar) spec, but can live with it as a separate doc, provided that some
> thought is put into warning potential users about some risks.
>
> > The new MASQUE[1] and WEBTRANS[2] working groups both wish to define new
> protocols using QUIC DATAGRAM frames[3] that are not conveying (existing)
> HTTP semantics, but are intended to be usable over (at least some versions
> of) a HTTP connection.
> >
> > MASQUE's purpose is to proxy UDP and potentially IP (like a VPN) over a
> HTTP/3 connection. For WEBTRANS, the purpose is to create a
> browser-exposed, WebSocket-like protocol that allows unreliable delivery
> (which may also use CONNECT-UDP). These summaries are just my
> understanding; for a more complete picture, see the respective charters and
> related WG discussion.
> >
> > It's not yet clear how or even if these protocols will map into HTTP
> semantics. Some have proposed something like a CONNECT-UDP method; others
> seem to favour negotiating a new protocol using SETTINGS or a similar
> version-specific mechanism directly, avoiding the definition of any generic
> HTTP semantics.
> >
> > So far, the most concrete proposal we have is
> draft-schinazi-quic-h3-datagram,[4] which defines how QUIC DATAGRAM frames
> are surfaced in HTTP/3 without any mapping them into generic HTTP semantics
> (like a method, status code, header, etc.). Thus, a specification using
> them has the choice of defining a generic HTTP extension (like CONNECT-UDP,
> mapping them into HTTP semantics) or a version-specific extension(like
> SETTINGS, bypassing generic semantics).
> >
> > Before these groups get too far down the road, we have an opportunity to
> give them some guidance about how to extend both HTTP's semantics overall,
> and specific versions of HTTP -- keeping in mind recent core
> discussions.[5]
> >
> > The most relevant question is whether new, non-HTTP protocols on a HTTP
> connection should be bootstrapped by generic HTTP semantics (e.g., a
> method), version-specific semantics (e.g., SETTINGS), or something else.
> >
> > There's also a question as to whether something like
> draft-schinazi-h3-datagram is the right approach, or whether it should also
> include generic HTTP semantics "bundled in."
> >
> > 3. 'An Unreliable Datagram Extension to QUIC', <
> https://tools.ietf.org/html/draft-ietf-quic-datagram>.

---

From: David Schinazi <[email-masked]>
Date: Wed, 24 Jun 2020 14:33:19 -0700

Thanks for bringing this up, Mark!

The split between draft-schinazi-quic-h3-datagram
and draft-schinazi-masque-connect-udp mainly came from my intuition at the
time, and is definitely something we could change. The rationale at the
time was that there might be some applications that are interested in using
datagrams over an h3 connection, but not associating those with HTTP
semantics. After conversations with Mark, I'm no longer sure that these are
necessarily useful.

CONNECT-UDP has the nice property of fitting in with HTTP semantics and
being able to work across multiple HTTP hops. One possible solution could
be to fold draft-schinazi-quic-h3-datagram
into draft-schinazi-masque-connect-udp, and then have the combined document
tie QUIC DATAGRAM frames over an HTTP/3 connection with a given CONNECT-UDP
stream. My main concern is that if we come up with a new HTTP mechanism
that would like to use datagrams but couldn't be solved by extending
CONNECT-UDP, then we'd be in a bind because it might be impossible to use
both CONNECT-UDP and this new mechanism over the same connection. Another
solution could be to keep the drafts separate, and
have draft-schinazi-quic-h3-datagram mention that any application that uses
it MUST define associated HTTP semantics and how datagrams work across
multiple HTTP hops.

@Ben I don't think reusing extended CONNECT is an option for CONNECT-UDP,
because of backwards compatibility: if a proxy were to receive a CONNECT
request with ":protocol" set to "udp", it might ignore the ":protocol"
header and create a TCP connection to the host and port instead of UDP. I
think it's important that CONNECT-UDP to a non-supporting proxy fail
instead of fallback to TCP.

David

On Wed, Jun 24, 2020 at 11:30 AM Ben Schwartz <[email-masked]> wrote:

>> Overall - I have a preference for this to be rolled into a CONNECT-UDP
>> (or similar) spec, but can live with it as a separate doc, provided that
>> some thought is put into warning potential users about some risks.
>>
>> > The new MASQUE[1] and WEBTRANS[2] working groups both wish to define
>> new protocols using QUIC DATAGRAM frames[3] that are not conveying
>> (existing) HTTP semantics, but are intended to be usable over (at least
>> some versions of) a HTTP connection.
>> >
>> > Before these groups get too far down the road, we have an opportunity
>> to give them some guidance about how to extend both HTTP's semantics
>> overall, and specific versions of HTTP -- keeping in mind recent core
>> discussions.[5]
>> >

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 25 Jun 2020 09:53:01 +1000

Hey Ben,

I don't want to get too much into the mechanism yet - this is more about general guidance.

That said, you remind me that we should probably have a discussion about how pseudo-headers are (ab)used; we've already seen cases where people are using them for private purposes, and overloading :protocol has its own tricky bits...

Cheers,

> On 25 Jun 2020, at 4:30 am, Ben Schwartz <[email-masked]> wrote:
> 
> One direction I think could be interesting is to embrace the Extended CONNECT as used by WebSocket-H2 (RFC 8441).  If we define ":protocol = tcp" as a synonym for the old-fashioned CONNECT, then ":protocol = udp" could represent the basic MASQUE case nicely, and ":protocol = webtransport" might go naturally as well (if we can't reuse ":protocol = websocket").
> 
> The combination of :protocol and transport (H3 vs. H2) might be sufficient to indicate whether datagrams are allowed.
> 
> 
>

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 25 Jun 2020 09:56:05 +1000

> On 25 Jun 2020, at 7:33 am, David Schinazi <[email-masked]> wrote:
> 
> My main concern is that if we come up with a new HTTP mechanism that would like to use datagrams but couldn't be solved by extending CONNECT-UDP, then we'd be in a bind because it might be impossible to use both CONNECT-UDP and this new mechanism over the same connection.

Could you not embed it in CONNECT-UDP in such a way that it could be reused by other extensions?

> Another solution could be to keep the drafts separate, and have draft-schinazi-quic-h3-datagram mention that any application that uses it MUST define associated HTTP semantics and how datagrams work across multiple HTTP hops.

I'm not sure that MUST would have much force, but perhaps we could just position the document as 'this is a generic layer for attaching HTTP semantics to unreliable DATAGRAM delivery in HTTP/3' or similar.

---

From: Ben Schwartz <[email-masked]>
Date: Wed, 24 Jun 2020 21:28:39 -0400

On Wed, Jun 24, 2020 at 5:33 PM David Schinazi <[email-masked]>
wrote:

> header and create a TCP connection to the host and port instead of UDP.
>

I don't think that's true.  Support for :protocol has to be signaled at
startup with SETTINGS_ENABLE_CONNECT_PROTOCOL, so it cannot be sent to a
server that does not understand it.  RFC 8441 also says "On requests
bearing the :protocol pseudo-header field ... the server MUST NOT create a
tunnel to the host indicated by the :authority as it would with a
CONNECT method
request that was not modified by this extension.", so it's quite explicit
that extended-CONNECT does not silently fall back to old-fashioned CONNECT.

I think it's important that CONNECT-UDP to a non-supporting proxy fail
> instead of fallback to TCP.

I believe Extended-CONNECT gives you this behavior.

David
>> The combination of :protocol and transport (H3 vs. H2) might be
>> sufficient to indicate whether datagrams are allowed.
>>
>>
>>
>>> > MASQUE's purpose is to proxy UDP and potentially IP (like a VPN) over
>>> a HTTP/3 connection. For WEBTRANS, the purpose is to create a
>>> > The most relevant question is whether new, non-HTTP protocols on a
>>> HTTP connection should be bootstrapped by generic HTTP semantics (e.g., a

---

From: Eric Kinnear <[email-masked]>
Date: Wed, 24 Jun 2020 18:48:36 -0700

> On Jun 24, 2020, at 6:28 PM, Ben Schwartz <[email-masked]> wrote:
> 
> 
> 
> On Wed, Jun 24, 2020 at 5:33 PM David Schinazi <[email-masked] <mailto:[email-masked]>> wrote:
> @Ben I don't think reusing extended CONNECT is an option for CONNECT-UDP, because of backwards compatibility: if a proxy were to receive a CONNECT request with ":protocol" set to "udp", it might ignore the ":protocol" header and create a TCP connection to the host and port instead of UDP.
> 
> I don't think that's true.  Support for :protocol has to be signaled at startup with SETTINGS_ENABLE_CONNECT_PROTOCOL, so it cannot be sent to a server that does not understand it.  RFC 8441 also says "On requests bearing the :protocol pseudo-header field ... the server MUST NOT create a tunnel to the host indicated by the :authority as it would with a CONNECT method request that was not modified by this extension.", so it's quite explicit that extended-CONNECT does not silently fall back to old-fashioned CONNECT.
> 
> I think it's important that CONNECT-UDP to a non-supporting proxy fail instead of fallback to TCP.
> 

We used Extended-CONNECT in one of the iterations of the WebTransport drafts in almost exactly this way, and I do think it was not un-reasonable if only looking at the text as written. 

However, as Mark mentioned general direction over implementation mechanism at the moment, I think one of the main questions around H3 datagrams is how they differ from QUIC datagrams and whether that’s something that should have semantic meaning in HTTP. 

I know that one of the reasons to split out into H3 datagrams was to have a flow ID present to allow multiple different flows of data to coexist, and there was quite lively discussion at the time which was fairly split as to the utility vs. dangers of such flow IDs. 

The biggest questions in my mind in that area are: How much do we care about sharing an HTTP connection that is using HTTP semantics on some streams with these other frames? And does that, in turn, mean that we need to define HTTP semantics for the datagrams (H3 or H2 or otherwise)? 

My personal gut reaction would be (a) yes, we do care and (b) probably not necessary as long as we can correctly demux what information is intended for what extension/protocol living at the other end.

>> My main concern is that if we come up with a new HTTP mechanism that would like to use datagrams but couldn't be solved by extending CONNECT-UDP, then we'd be in a bind because it might be impossi…

Thanks,
Eric

> 
> David
> 
> On Wed, Jun 24, 2020 at 11:30 AM Ben Schwartz <[email-masked] <mailto:[email-masked]>> wrote:
> On Tue, Jun 23, 2020 at 4:49 AM Mark Nottingham <[email-masked] <mailto:[email-masked]>> wrote:
> > On 23 Jun 2020, at 9:40 am, Mark Nottingham <[email-masked] <mailto:[email-masked]>> wrote:
> > 
> > Hi all,
> > 
> > 1. <https://datatracker.ietf.org/wg/masque/about/ <https://datatracker.ietf.org/wg/masque/about/>>.
> > 2. <https://datatracker.ietf.org/wg/webtrans/about/ <https://datatracker.ietf.org/wg/webtrans/about/>>.
> > 3. 'An Unreliable Datagram Extension to QUIC', <https://tools.ietf.org/html/draft-ietf-quic-datagram <https://tools.ietf.org/html/draft-ietf-quic-datagram>>.
> > 4. <https://tools.ietf.org/html/draft-schinazi-quic-h3-datagram <https://tools.ietf.org/html/draft-schinazi-quic-h3-datagram>>.
> > 5. <https://github.com/httpwg/http-core/issues/44 <https://github.com/httpwg/http-core/issues/44>>.
> > 
> > --
> > Mark Nottingham   https://www.mnot.net/ <https://www.mnot.net/>
> > 
> > 
> 
> --
> Mark Nottingham   https://www.mnot.net/ <https://www.mnot.net/>
> 
>

---

From: Lucas Pardue <[email-masked]>
Date: Thu, 25 Jun 2020 20:04:00 +0100

Lots a food for thought in this thread, I'll pick on one aspect for now:

On Thu, Jun 25, 2020 at 2:52 AM Eric Kinnear <[email-masked]> wrote:

> I know that one of the reasons to split out into H3 datagrams was to have
> a flow ID present to allow multiple different flows of data to coexist, and
> there was quite lively discussion at the time which was fairly split as to
> the utility vs. dangers of such flow IDs.
>
> The biggest questions in my mind in that area are: How much do we care
> about sharing an HTTP connection that is using HTTP semantics on some
> streams with these other frames? And does that, in turn, mean that we need
> to define HTTP semantics for the datagrams (H3 or H2 or otherwise)?
>
> My personal gut reaction would be (a) yes, we do care and (b) probably not
> necessary as long as we can correctly demux what information is intended
> for what extension/protocol living at the other end.
>

Over on the datagram spec repo [1] we are still discussing the topic of
flow IDs. From that I've been more deeply considering my implementation of
QUIC DATAGRAM and looking ahead to HTTP/3 DATAGRAM, especially how that
layering would work and the API presented by the layers*. So, devil's
advocate, I'm wondering why we need the abstract notion of a flow ID in
HTTP/3 - draft-schinazi-quic-h3-datagram-03 [2] defines the requirement
"Implementations of HTTP/3 that support the DATAGRAM extension MUST provide
a flow identifier allocation service.", which as a library implementer I
struggle to satisfy. It also offers little guidance on things like security
considerations or DoS analysis, which is really where a common definition
would help because it would force the issue of determining if we need to
communicate things such as limiting the number of total concurrent flows,
how to retire a flow etc, how to report limit violations/error, how to
gracefully close an HTTP/3 connection with active flows etc. The closest
parallel for me is HTTP/3's management of server push via push ID, it fixes
an accounting loophole that exists in HTTP/2 and provides explicit guidance
for use with GOAWAY. If HTTP/3 DATAGRAM were to stay generic then IMO it
needs to think about some factors like that. However, if the use of
DATAGRAMs is always required to be defined alongside by an HTTP semantic
then it might be able to play by the rules of that semantic; for example,
consider that we define a tight binding between DATAGRAM and CONNECT-UDP -
an implementation could apply the security considerations and resource
management that it already does for HTTP requests. Thus the number of
concurrent flows is bound by the number of concurrent requests and flow ID
fate fate is tied to request ID fate, which is tied to connection fate.
Furthermore it would support the carriage of per-flow limits expressed as
headers.

So to address the OP, I think we need to fully consider semantics up front:
either true version-independent HTTP semantics or, as the QUIC charter
likes to call them version-specific semantics. The danger is not doing so
is that people start deploying things that we have to retroactively attempt
to describe e.g. "A payload within a CONNECT request message has no defined
semantics;   sending a payload body on a CONNECT request might cause some
existing implementations to reject the request"

Cheers,
Lucas

* in quiche, we present both QUIC and HTTP/3 API layers and document how
applications are expected to use them when using one or both of the layers.

[1] - https://github.com/quicwg/datagram/issues/6
[2] https://tools.ietf.org/html/draft-schinazi-quic-h3-datagram-03