Subject: Final DATA frames

From: Mike Bishop <[email-masked]>
Date: Thu, 6 Dec 2018 18:57:13 +0000

One other change to draw the working groups' attention to, on the opposite end of the stream from initial PRIORITY frames:

Issue #1885<https://github.com/quicwg/base-drafts/issues/1885> points out that, as in HTTP/2, it is inefficient to have to length-prefix each chunk of a response which is being generated incrementally.  In H2, this is unavoidable, because the frames are also the unit of multiplexing.  In H3, the QUIC STREAM frames are always length-prefixed, but we can have very large HTTP-layer frames because they're within a QUIC stream.  This was briefly mentioned in Bangkok.

There are multiple possible approaches to making this better:

  *   Move DATA off-stream.  Considered too extreme a change for this stage of the process and not helpful in all cases, but this can be implemented as an extension.  (See this unsubmitted draft<https://github.com/MikeBishop/quic-external-data>.)
  *   Frames of length 0 extend to the end of the stream.  This seemed like the architecturally cleanest way to solve it, but the prospect of arbitrary frames whose length isn't known from the beginning was potentially challenging for some implementations.
  *   DATA frames of length 0 extend to the end of the stream.  This is a more scoped change, but special-cases the DATA frame.

After discussion on the issue and on the submitted PR (#2098)<https://github.com/quicwg/base-drafts/pull/2098>, the consensus seemed roughly split between the latter two options, but most participants in the discussion seemed amenable to either.  I've merged the PR, which changed the behavior only of DATA frames.

However, Martin pointed out<https://github.com/quicwg/base-drafts/pull/2098#issuecomment-444763104> that this discussion has been entirely on GitHub, and that perhaps an opportunity for list discussion was in order.  If there's not WG consensus to proceed with this change at all, I'll revert it.  If there's consensus to use a different solution instead, I'll create a follow-up PR.

Thanks,
Mike Bishop

---

From: Dmitri Tikhonov <[email-masked]>
Date: Thu, 6 Dec 2018 14:23:56 -0500

On Thu, Dec 06, 2018 at 06:57:13PM +0000, Mike Bishop wrote:
> However, Martin pointed out [1] that this discussion has been
> entirely on GitHub

This is not quite correct, we did discuss it on the mailing list
first, albeit not in a dedicated thread:

  https://www.ietf.org/mail-archive/web/quic/current/msg05003.html

  - Dmitri.

1. https://github.com/quicwg/base-drafts/pull/2098#issuecomment-444763104

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 6 Dec 2018 18:04:05 -0800

We had also briefly discussed this in Bangkok, hadn't we?
I'm ok with this change, FWIW.  I'd like us to not be making large and
gratuitous changes at this point, but this is neither.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 7 Dec 2018 18:15:42 +0000

AFAICT, the github conversation didn’t touch on the interrelation of framing-on-stream and push-promise, or trailers.

Given the current inline (interleaved) expression of push-promise within a stream, you don’t want to prevent yourself from being able to insert the push-promise description into the stream when you need to do it.
It’d also make trailers “interesting”, i.e. likely prevent them from working properly.

If having a zero-length frame is considered a problem, then another way to solve that would be to assert that the length of a frame is the stated size + 1, i.e. if that field is zero, then the length of the payload is one.
-=R

From: QUIC <[email-masked]> on behalf of Jana Iyengar <[email-masked]>
Date: Thursday, December 6, 2018 at 6:04 PM
To: Mike Bishop <[email-masked]>, "[email-masked]" <[email-masked]>, QUIC WG <[email-masked]>
Subject: Re: Final DATA frames

We had also briefly discussed this in Bangkok, hadn't we?
I'm ok with this change, FWIW.  I'd like us to not be making large and gratuitous changes at this point, but this is neither.

On Thu, Dec 6, 2018 at 11:24 AM Dmitri Tikhonov <[email-masked]<mailto:[email-masked]>> wrote:

  https://www.ietf.org/mail-archive/web/quic/current/msg05003.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mail-2Darchive_web_quic_current_msg05003.html&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=4E1OreHTngWjXbIHz7eVjX5C2sQ0tW0A61_fSORHwIs&s=f4AndezxB6bkwfBj1QPQ5zpyVdGWIpAlSDajCjEpIpU&e=>

  - Dmitri.

---

From: Dmitri Tikhonov <[email-masked]>
Date: Fri, 7 Dec 2018 13:45:41 -0500

On Fri, Dec 07, 2018 at 06:15:42PM +0000, Roberto Peon wrote:
> Given the current inline (interleaved) expression of push-promise within
> a stream, you don’t want to prevent yourself from being able to insert
> the push-promise description into the stream when you need to do it.

I agree.  I made similar argument here:

  https://github.com/quicwg/base-drafts/issues/1885#issuecomment-431478990

  - Dmitri.

---

From: Ryan Hamilton <[email-masked]>
Date: Fri, 7 Dec 2018 12:11:23 -0800

On Fri, Dec 7, 2018 at 10:16 AM Roberto Peon <[email-masked]> wrote:

> AFAICT, the github conversation didn’t touch on the interrelation of
> framing-on-stream and push-promise, or trailers.
>
>
> Given the current inline (interleaved) expression of push-promise within a
> stream, you don’t want to prevent yourself from being able to insert the
> push-promise description into the stream when you need to do it.
> If having a zero-length frame is considered a problem, then another way to
> solve that would be to assert that the length of a frame is the stated size
> + 1, i.e. if that field is zero, then the length of the payload is one.
>

Well, the motivation for the change is to avoid extra overhead in the very
common case of an HTTP message with no trailers and no associated pushes
(and dynamically generated content). If a particular HTTP message does not
have those properties then it can easily use a non-zero length to terminate
the DATA frame and follow it up with pushes or trailers.

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 7 Dec 2018 20:22:11 +0000

I was swayed by the discussion on this topic to support the proposal we
ended up with. As Ryan puts it so well, we don't discount such other use
cases, we just need the server to have enough information to decide how to
respond on the initial DATA frame it responds with.

And to that end, I think it has enough enpowerment. If risk averse, it
would not use this optimisation. If sensitive to push or trailers, it can
take the care to allow them.

Lucas

On Fri, 7 Dec 2018, 20:12 Ryan Hamilton <rch=[email-masked]
wrote:

>> a stream, you don’t want to prevent yourself from being able to insert the
>> If having a zero-length frame is considered a problem, then another way
>> to solve that would be to assert that the length of a frame is the stated
>> size + 1, i.e. if that field is zero, then the length of the payload is one.
>>
>

---

From: Roberto Peon <[email-masked]>
Date: Fri, 7 Dec 2018 21:27:19 +0000

It will become very easy for an implementation which otherwise seems conforming to mess up and have no way of using push, or communicating trailers.
This really seems like premature optimization, especially in the case where we’re taking options off the table already (like having the payload be separate).
A little inefficiency seems much less scary than features no longer interoperating (e.g. because a proxy is very unlikely to know whether the response will have trailers, etc. without some very interesting coordination). That would feel very HTTP/1.1 pipelining (which was a good idea, but couldn’t effectively be implemented interoperably) to me. That was not at all a fun situation.

This makes me feel very uneasy.
-=R

From: Lucas Pardue <[email-masked]>
Date: Friday, December 7, 2018 at 12:23 PM
To: Ryan Hamilton <rch=[email-masked]>
Cc: Roberto Peon <[email-masked]>, Jana Iyengar <[email-masked]>, IETF QUIC WG <[email-masked]>, Mike Bishop <[email-masked]>, HTTP Working Group <[email-masked]>
Subject: Re: Final DATA frames

I was swayed by the discussion on this topic to support the proposal we ended up with. As Ryan puts it so well, we don't discount such other use cases, we just need the server to have enough information to decide how to respond on the initial DATA frame it responds with.

And to that end, I think it has enough enpowerment. If risk averse, it would not use this optimisation. If sensitive to push or trailers, it can take the care to allow them.

Lucas

On Fri, 7 Dec 2018, 20:12 Ryan Hamilton <rch=[email-masked]<mailto:[email-masked]> wrote:
On Fri, Dec 7, 2018 at 10:16 AM Roberto Peon <[email-masked]<mailto:[email-masked]>> wrote:
AFAICT, the github conversation didn’t touch on the interrelation of framing-on-stream and push-promise, or trailers.

Well, the motivation for the change is to avoid extra overhead in the very common case of an HTTP message with no trailers and no associated pushes (and dynamically generated content). If a particular HTTP message does not have those properties then it can easily use a non-zero length to terminate the DATA frame and follow it up with pushes or trailers.

---

From: Mike Bishop <[email-masked]>
Date: Fri, 7 Dec 2018 22:50:25 +0000

Trailers SHOULD be declared in the headers, but it’s true that a proxy whose origin violates that SHOULD could find itself in a state where it has to drop the trailers.

From: Roberto Peon <[email-masked]>
Sent: Friday, December 7, 2018 1:27 PM
To: Lucas Pardue <[email-masked]>; Ryan Hamilton <rch=[email-masked]>
Cc: Jana Iyengar <[email-masked]>; IETF QUIC WG <[email-masked]>; Mike Bishop <[email-masked]>; HTTP Working Group <[email-masked]>
Subject: Re: Final DATA frames

From: Lucas Pardue <[email-masked]<mailto:[email-masked]>>
Date: Friday, December 7, 2018 at 12:23 PM
To: Ryan Hamilton <rch=[email-masked]<mailto:rch=[email-masked]>>
Cc: Roberto Peon <[email-masked]<mailto:[email-masked]>>, Jana Iyengar <[email-masked]<mailto:[email-masked]>>, IETF QUIC WG <[email-masked]<mailto:[email-masked]>>, Mike Bishop <[email-masked]<mailto:[email-masked]>>, HTTP Working Group <[email-masked]<mailto:[email-masked]>>
Subject: Re: Final DATA frames

Lucas

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 7 Dec 2018 23:38:46 +0000

Perhaps I'm overlooking a few things but the way I'd see it is that such a
proxy could be conservative and chose never to use len=0 on responses that
it is in control of.

If it wanted to be more flexible at cost of risk, a proxy might also use
the content-length header to make an informed decision. The header is not
required, which sort of aligns with an origin generating a dynamic response
of unknown total length.

Wrt to push. A proxy could use reception of a link header, with preload, to
decide to push before generating the parent request-response payload. If
the upstream wants to support "pass through" of push promise properly,
being conservative with length=0 would also be sensible.

Trailers, well they are are hard I agree. Mike's provided some options but
I'd ask exactly what trailers are applicable to H3. Is this design really a
blocker to  use?

On Fri, 7 Dec 2018, 22:50 Mike Bishop <[email-masked] wrote:

> Trailers SHOULD be declared in the headers, but it’s true that a proxy
> whose origin violates that SHOULD could find itself in a state where it has
> to drop the trailers.
>
>
>
> *From:* Roberto Peon <[email-masked]>
> *Sent:* Friday, December 7, 2018 1:27 PM
> *To:* Lucas Pardue <[email-masked]>; Ryan Hamilton <rch=
> [email-masked]>
> *Cc:* Jana Iyengar <[email-masked]>; IETF QUIC WG <[email-masked]>;
> Mike Bishop <[email-masked]>; HTTP Working Group <
> [email-masked]>
> *Subject:* Re: Final DATA frames
>
>
>
> It will become very easy for an implementation which otherwise seems
> conforming to mess up and have no way of using push, or communicating
> trailers.
> This really seems like premature optimization, especially in the case
> where we’re taking options off the table already (like having the payload
> be separate).
>
> A little inefficiency seems much less scary than features no longer
> interoperating (e.g. because a proxy is very unlikely to know whether the
> response will have trailers, etc. without some very interesting
> coordination). That would feel very HTTP/1.1 pipelining (which was a good
> idea, but couldn’t effectively be implemented interoperably) to me. That
> was not at all a fun situation.
>
>
> *From: *Lucas Pardue <[email-masked]>
> *Date: *Friday, December 7, 2018 at 12:23 PM
> *To: *Ryan Hamilton <rch=[email-masked]>
> *Cc: *Roberto Peon <[email-masked]>, Jana Iyengar <[email-masked]>,
> IETF QUIC WG <[email-masked]>, Mike Bishop <[email-masked]>, HTTP
> Working Group <[email-masked]>
> *Subject: *Re: Final DATA frames
>
>
>

---

From: Mike Bishop <[email-masked]>
Date: Sat, 8 Dec 2018 00:01:03 +0000

Not a blocker in my view, but it’s a fair critique that this design is mutually exclusive with those features.  A server (whether proxy or application library) that wants to retain the ability to do mid-content push or trailers can’t use this feature.  A server that is reasonably certain that it won’t need them could use this feature as an optimization.  It’s worth noting that a “final DATA frame” is only ever that, and need not be used.

I wasn’t planning to get into this yet, but since it’s relevant, I have an alternate design as an extension (not yet published) here<https://mikebishop.github.io/quic-external-data/draft-bishop-quic-external-data.html>.  One option is to revert this change and interested parties can work on the extension instead, if there’s interest.

From: Lucas Pardue <[email-masked]>
Sent: Friday, December 7, 2018 3:39 PM
To: Mike Bishop <[email-masked]>
Cc: Roberto Peon <[email-masked]>; Ryan Hamilton <rch=[email-masked]>; Jana Iyengar <[email-masked]>; IETF QUIC WG <[email-masked]>; HTTP Working Group <[email-masked]>
Subject: Re: Final DATA frames

Perhaps I'm overlooking a few things but the way I'd see it is that such a proxy could be conservative and chose never to use len=0 on responses that it is in control of.

If it wanted to be more flexible at cost of risk, a proxy might also use the content-length header to make an informed decision. The header is not required, which sort of aligns with an origin generating a dynamic response of unknown total length.

Wrt to push. A proxy could use reception of a link header, with preload, to decide to push before generating the parent request-response payload. If the upstream wants to support "pass through" of push promise properly, being conservative with length=0 would also be sensible.

Trailers, well they are are hard I agree. Mike's provided some options but I'd ask exactly what trailers are applicable to H3. Is this design really a blocker to  use?

On Fri, 7 Dec 2018, 22:50 Mike Bishop <[email-masked]<mailto:[email-masked]> wrote:
Trailers SHOULD be declared in the headers, but it’s true that a proxy whose origin violates that SHOULD could find itself in a state where it has to drop the trailers.

From: Roberto Peon <[email-masked]<mailto:[email-masked]>>
Sent: Friday, December 7, 2018 1:27 PM
To: Lucas Pardue <[email-masked]<mailto:[email-masked]>>; Ryan Hamilton <rch=[email-masked]<mailto:[email-masked]>>
Cc: Jana Iyengar <[email-masked]<mailto:[email-masked]>>; IETF QUIC WG <[email-masked]<mailto:[email-masked]>>; Mike Bishop <[email-masked]<mailto:[email-masked]>>; HTTP Working Group <[email-masked]<mailto:[email-masked]>>
Subject: Re: Final DATA frames

Lucas

---

From: Lucas Pardue <[email-masked]>
Date: Sat, 8 Dec 2018 00:06:09 +0000

Hmm, an out of order delivery issue there. I was defending this design
rather than poking holes is. Regardless, the reason I like it is that it
has flexibility down to the per-request level. I understand concerns but
think a server actor can mitigate them by simply not supporting len=0. I
can't see the interop issue, but perhaps I'm just missing it.

Licas

On Sat, 8 Dec 2018, 00:01 Mike Bishop <[email-masked] wrote:

> Not a blocker in my view, but it’s a fair critique that this design is
> mutually exclusive with those features.  A server (whether proxy or
> application library) that wants to retain the ability to do mid-content
> push or trailers can’t use this feature.  A server that is reasonably
> certain that it won’t need them could use this feature as an optimization.
> It’s worth noting that a “final DATA frame” is only ever that, and need not
> be used.
>
>
>
> I wasn’t planning to get into this yet, but since it’s relevant, I have an
> alternate design as an extension (not yet published) here
> <https://mikebishop.github.io/quic-external-data/draft-bishop-quic-external-data.html>.
> One option is to revert this change and interested parties can work on the
> extension instead, if there’s interest.
>
>
>
> *From:* Lucas Pardue <[email-masked]>
> *Sent:* Friday, December 7, 2018 3:39 PM
> *To:* Mike Bishop <[email-masked]>
> *Cc:* Roberto Peon <[email-masked]>; Ryan Hamilton <rch=
> [email-masked]>; Jana Iyengar <[email-masked]>; IETF
> QUIC WG <[email-masked]>; HTTP Working Group <[email-masked]>
> Wrt to push. A proxy could use reception of a link header, with preload,
> to decide to push before generating the parent request-response payload. If

---

From: Mike Bishop <[email-masked]>
Date: Sat, 8 Dec 2018 00:10:57 +0000

The interop issue is that a proxy might inspect the headers, see no indication of trailers, use a len=0 DATA frame, then arrive at the end of the origin’s response and discover trailers there anyway that it now can’t send.  As you say, the proxy can be cautious, and proxies are also allowed to discard trailers, so it’s not a hard interop failure.  But it’s an unfortunate corner-case in an otherwise beneficial change.

Sent: Friday, December 7, 2018 4:06 PM

Hmm, an out of order delivery issue there. I was defending this design rather than poking holes is. Regardless, the reason I like it is that it has flexibility down to the per-request level. I understand concerns but think a server actor can mitigate them by simply not supporting len=0. I can't see the interop issue, but perhaps I'm just missing it.

Licas

On Sat, 8 Dec 2018, 00:01 Mike Bishop <[email-masked]<mailto:[email-masked]> wrote:
Not a blocker in my view, but it’s a fair critique that this design is mutually exclusive with those features.  A server (whether proxy or application library) that wants to retain the ability to do mid-content push or trailers can’t use this feature.  A server that is reasonably certain that it won’t need them could use this feature as an optimization.  It’s worth noting that a “final DATA frame” is only ever that, and need not be used.

From: Lucas Pardue <[email-masked]<mailto:[email-masked]>>
Sent: Friday, December 7, 2018 3:39 PM
To: Mike Bishop <[email-masked]<mailto:[email-masked]>>
Cc: Roberto Peon <[email-masked]<mailto:[email-masked]>>; Ryan Hamilton <rch=[email-masked]<mailto:[email-masked]>>; Jana Iyengar <[email-masked]<mailto:[email-masked]>>; IETF QUIC WG <[email-masked]<mailto:[email-masked]>>; HTTP Working Group <[email-masked]<mailto:[email-masked]>>
Subject: Re: Final DATA frames

Lucas

---

From: David Schinazi <[email-masked]>
Date: Fri, 7 Dec 2018 16:49:34 -0800

I don't see the proxy case as an issue. Maybe this can be resolved by
adding some implementation guidance along the lines of "do not send length
0 unless you are absolutely sure there will be no trailers or other bits
after your payload". This is a performance optimization for a widespread
use-case of the web today, and no one is required to take advantage of it -
if someone is writing a proxy they can simply not use the optimization.

David

On Fri, Dec 7, 2018 at 4:11 PM Mike Bishop <[email-masked]> wrote:

> The interop issue is that a proxy might inspect the headers, see no
> indication of trailers, use a len=0 DATA frame, then arrive at the end of
> the origin’s response and discover trailers there anyway that it now can’t
> send.  As you say, the proxy can be cautious, and proxies are also allowed
> to discard trailers, so it’s not a hard interop failure.  But it’s an
> unfortunate corner-case in an otherwise beneficial change.
>
>
>
> *Sent:* Friday, December 7, 2018 4:06 PM

---

From: Kazuho Oku <[email-masked]>
Date: Sat, 8 Dec 2018 10:18:04 +0900

> The interop issue is that a proxy might inspect the headers, see no indication of trailers, use a len=0 DATA frame, then arrive at the end of the origin’s response and discover trailers there anywa…

I tend to agree with what Mike and Roberto says. Use of len=0 is a
hack that does not always work.

At least, there are two better ways of addressing the issue. One way
is to use a different stream for carrying trailers.

The other way (which is my preference) is to let the transport layer
carry the boundaries of streams. For example, we could add a
START_OF_CHUNK flag to STREAMs frame, and also a NEXT_CHUNK frame
(with no payload) that indicates the start of a next chunk of the
preceding STREAM frame. Then, we can get rid of length field of every
HTTP frame.

Admittedly, the alternatives have less chance in making it to v1. But
the question is, is always using length field so bad that we cannot
wait for a more ideal fix?

> I wasn’t planning to get into this yet, but since it’s relevant, I have an alternate design as an extension (not yet published) here.  One option is to revert this change and interested parties can work on the extension instead, if there’s interest.
>
>
>
> On Fri, 7 Dec 2018, 20:12 Ryan Hamilton <rch=[email-masked] wrote:
>

-- 
Kazuho Oku

---

From: Dmitri Tikhonov <[email-masked]>
Date: Sun, 9 Dec 2018 16:26:33 -0500

> wait for a more ideal fix?

I don't think it's so bad.  The overhead incurred by the DATA frames
is small: 0.2% to 0.02% [1].  I think we can live with it for v1.

  - Dmitri.

1. https://github.com/quicwg/base-drafts/issues/1885#issuecomment-431348155

---

From: Mike Bishop <[email-masked]>
Date: Mon, 10 Dec 2018 23:35:49 +0000

I'm personally hoping that the HTTP spec will change less frequently than the QUIC spec will, and that changes will be primarily via extensions.  But the choice I'd like to make on list right now is:

  *   Keep the change - it's an optimization for a common case, and proxies can disable it if they expect the uncommon case to apply
  *   Revert the change - the optimization isn't worth the possibility that proxies shoot themselves in the foot

Whether we can do better in an extension or by adding new transport features in v2 doesn't block either option - we can keep this change and still publish something better later.  My count of people on this thread so far is split about 50/50 between keeping the change in versus various reasons to reject it.

-----Original Message-----
From: Dmitri Tikhonov <[email-masked]>
Sent: Sunday, December 9, 2018 1:27 PM
To: Kazuho Oku <[email-masked]>
Cc: Mike Bishop <[email-masked]>; Jana Iyengar <[email-masked]>; Roberto Peon <[email-masked]>; rch=[email-masked]; HTTP Working Group <[email-masked]>; [email-masked]; IETF QUIC WG <[email-masked]>
Subject: Re: Final DATA frames

> wait for a more ideal fix?

I don't think it's so bad.  The overhead incurred by the DATA frames is small: 0.2% to 0.02% [1].  I think we can live with it for v1.

  - Dmitri.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 11 Dec 2018 11:18:55 +1100

FWIW, there is a valid use case here, but I'm not happy with the
specific design.

For instance, an extension that used a unidirectional stream for the
body of a request might be a better option.

On that basis, I would revert the change.
On Tue, Dec 11, 2018 at 10:36 AM Mike Bishop <[email-masked]> wrote:
> Keep the change – it’s an optimization for a common case, and proxies can disable it if they expect the uncommon case to apply
> Revert the change – the optimization isn’t worth the possibility that proxies shoot themselves in the foot
>
>
>
> Whether we can do better in an extension or by adding new transport features in v2 doesn’t block either option – we can keep this change and still publish something better later.  My count of people on this thread so far is split about 50/50 between keeping the change in versus various reasons to reject it.
>
>
>

---

From: Ryan Hamilton <[email-masked]>
Date: Mon, 10 Dec 2018 17:34:34 -0800

As you say, there's a valid use case here. There's nothing about this
particular design which would prevent any sort of "send the body on a
unidirectional stream" extension from being worked on or implemented. That
some future extension might (or might not) be a better solution to this use
case does not seem to me to be a terribly compelling argument for reverting
this, particularly given the simplicity of this design.

(I'll also point out that delivering the body on a unidirectional stream
doesn't necessarily work will with server push, as the PUSH_PROMISE is
required to arrive before the reference to the promised resource. So if the
PUSH_PROMISE happens on the main stream and the body goes on a different
stream, that ordering is not guaranteed without some addition properties.)

---

From: Roberto Peon <[email-masked]>
Date: Tue, 11 Dec 2018 01:46:56 +0000

This simplicity of the design here is also a trap. Sure, the extension is simple, but it also makes it simple to fail to interoperate!
To be clear, only those proxies which have established some prior knowledge that there will be no PUSH_PROMISE can safely use this optimization with the protocol as designed today.
-=R

From: Ryan Hamilton <[email-masked]>
Date: Monday, December 10, 2018 at 5:35 PM
To: Martin Thomson <[email-masked]>
Cc: Mike Bishop <[email-masked]>, Dmitri Tikhonov <[email-masked]>, Kazuho Oku <[email-masked]>, Jana Iyengar <[email-masked]>, Roberto Peon <[email-masked]>, HTTP Working Group <[email-masked]>, Lucas Pardue <[email-masked]>, IETF QUIC WG <[email-masked]>
Subject: Re: Final DATA frames

As you say, there's a valid use case here. There's nothing about this particular design which would prevent any sort of "send the body on a unidirectional stream" extension from being worked on or implemented. That some future extension might (or might not) be a better solution to this use case does not seem to me to be a terribly compelling argument for reverting this, particularly given the simplicity of this design.

(I'll also point out that delivering the body on a unidirectional stream doesn't necessarily work will with server push, as the PUSH_PROMISE is required to arrive before the reference to the promised resource. So if the PUSH_PROMISE happens on the main stream and the body goes on a different stream, that ordering is not guaranteed without some addition properties.)

On Mon, Dec 10, 2018 at 4:19 PM Martin Thomson <[email-masked]<mailto:[email-masked]>> wrote:
FWIW, there is a valid use case here, but I'm not happy with the
specific design.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 11 Dec 2018 12:55:11 +1100

> As you say, there's a valid use case here. There's nothing about this particular design which would prevent any sort of "send the body on a unidirectional stream" extension from being worked on or…

It creates uncertainty about whether the design is appropriate.

> (I'll also point out that delivering the body on a unidirectional stream doesn't necessarily work will with server push, as the PUSH_PROMISE is required to arrive before the reference to the promis…

I apologize.  I didn't mean to imply that I thought this had better
properties WRT PUSH_PROMISE.

---

From: Ryan Hamilton <[email-masked]>
Date: Mon, 10 Dec 2018 17:55:58 -0800

Not all endpoints are proxies :) And some of those endpoint which are
proxies are able to communicate with their backends in order to establish
such knowledge. The endpoints I work with are both able to establish such
knowledge, fwiw.

On Mon, Dec 10, 2018 at 5:47 PM Roberto Peon <[email-masked]> wrote:

> This simplicity of the design here is also a trap. Sure, the extension is
> simple, but it also makes it simple to fail to interoperate!
> To be clear, only those proxies which have established some prior
> knowledge that there will be no PUSH_PROMISE can safely use this
> optimization with the protocol as designed today.
> -=R
>
>
>
> *From: *Ryan Hamilton <[email-masked]>
> *Date: *Monday, December 10, 2018 at 5:35 PM
> *To: *Martin Thomson <[email-masked]>
> *Cc: *Mike Bishop <[email-masked]>, Dmitri Tikhonov <
> [email-masked]>, Kazuho Oku <[email-masked]>, Jana
> Iyengar <[email-masked]>, Roberto Peon <[email-masked]>, HTTP Working
> Group <[email-masked]>, Lucas Pardue <[email-masked]>,
> IETF QUIC WG <[email-masked]>

---

From: Dmitri Tikhonov <[email-masked]>
Date: Mon, 10 Dec 2018 21:41:13 -0500

This change is intended to fix a purported shortcoming in the
HTTP/3 framing mechanism.  The claim is that

  " DATA frame encoding is inefficient for long dynamically
  " generated bodies. [1]

As shown in this thread and elsewhere, the framing overhead is
small.  What is the definition of "inefficient" that is used to
make this claim?

  - Dmitri.

1. https://github.com/quicwg/base-drafts/issues/1885

---

From: Ryan Hamilton <[email-masked]>
Date: Tue, 11 Dec 2018 08:03:56 -0800

That's a fair question. One of the features of QUIC which is so exciting is
the performance improvements it offers. This comes from things like 0-RTT,
no HoL blocking, better loss recovery, etc. The former affects initial
latency, and the others come into play in the face of packet loss. But on
low loss links with long lived flows, different dynamics come into play.
Comparing TCP + TLS to QUIC, for example, we see that QUIC's per-packet
encryption overhead compares unfavorably to TLS's per-record encryption
overhead (assuming the senders is using large records). This is true with
Google QUIC's 12 byte AEAD hashes, and is even worse with IETF QUIC's 16
byte overhead (but of course, the 16 byte overhead provides desirable
security properties). All of this is to say, that for applications with low
loss links and long lived flow, increasing QUIC overhead eats into the
performance and threatens to tip the balance in favor of TCP. In my
experience with 12 byte auth hashes and no DATA frame overhead, QUIC
performance is not 10x better than TCP or 2x better than TCP. It's small
percentages better than TCP. So a .2% increase in overhead because of DATA
frame headers is like 10% of the gap between QUIC and TCP. This is
significant. And when there is an extremely simple feature to avoid this
overhead, I have a hard time thinking that overhead is justified.

On Mon, Dec 10, 2018 at 6:41 PM Dmitri Tikhonov <[email-masked]>
wrote:

> > > Group <[email-masked]>, Lucas Pardue <[email-masked]
> >,

---

From: Dmitri Tikhonov <[email-masked]>
Date: Tue, 11 Dec 2018 12:13:43 -0500

Let's look at the efficiency cost in a bit more detail.  The 0.2%
figure is the 3 bytes per packet.  This is in the worst case scenario
when a DATA frame is included in each packet.  Certainly since we
have a *long* dynamically generated response, one could choose to
make the frames longer.  If anything, the longer the response, the
more the framing penalty is amortized.  (A 1-byte response, in
contrast, would incur 200% framing penalty.)  Of course, at some
point the packets must actually be sent out -- and this is what
dictates just how long the DATA frame would have to be.  Hence, unless
one sends one packet at a time, the 3-byte cost is spread over
several packets and 0.2% becomes that many times smaller.

The framing penalty is already small.  I think one would have
trouble even measuring the impact -- throughput improvement -- of
the proposed change.

The benefit should be weighed against the drawbacks of the proposal.
They have been presented in this thread already so I won't repeat
them.

We are to choose between

    1. reducing the risk of interoperability problems; and

    2. reducing data framing penalty.

If the magnitude of (1) and (2) is about the same (this is, of
course, subjective), then I think we should not change anything.

  - Dmitri.

---

From: Roberto Peon <[email-masked]>
Date: Tue, 11 Dec 2018 17:43:16 +0000

Yeah, I’m aware, having helped to write at least some of them with you and others here as partners!

It isn’t like I hate optimization-- I’m just really scared of having to deal with this gone wrong. History has proven it is very likely that someone won’t get this right, and then protocol features will stop working, and pain will ensue, and the protocol will be less useful.

If we could put the header/payload into separate streams, then that’d be a step forward along a path to enabling such optimization (and compatible with the additional changes you’d need to handle PP)
… but I don’t want to think about that for V1.

I’d rather get what we have now done, and look at further optimizations in a V2 that we get out the door after a very short time window.
-=R

From: Ryan Hamilton <[email-masked]>
Date: Monday, December 10, 2018 at 5:56 PM
To: Roberto Peon <[email-masked]>
Cc: Martin Thomson <[email-masked]>, Mike Bishop <[email-masked]>, Dmitri Tikhonov <[email-masked]>, Kazuho Oku <[email-masked]>, Jana Iyengar <[email-masked]>, HTTP Working Group <[email-masked]>, Lucas Pardue <[email-masked]>, IETF QUIC WG <[email-masked]>
Subject: Re: Final DATA frames

Not all endpoints are proxies :) And some of those endpoint which are proxies are able to communicate with their backends in order to establish such knowledge. The endpoints I work with are both able to establish such knowledge, fwiw.

On Mon, Dec 10, 2018 at 5:47 PM Roberto Peon <[email-masked]<mailto:[email-masked]>> wrote:
This simplicity of the design here is also a trap. Sure, the extension is simple, but it also makes it simple to fail to interoperate!
To be clear, only those proxies which have established some prior knowledge that there will be no PUSH_PROMISE can safely use this optimization with the protocol as designed today.
-=R

From: Ryan Hamilton <[email-masked]<mailto:[email-masked]>>
Date: Monday, December 10, 2018 at 5:35 PM
To: Martin Thomson <[email-masked]<mailto:[email-masked]>>
Cc: Mike Bishop <[email-masked]<mailto:[email-masked]>>, Dmitri Tikhonov <[email-masked]<mailto:[email-masked]>>, Kazuho Oku <[email-masked]<mailto:[email-masked]>>, Jana Iyengar <[email-masked]<mailto:[email-masked]>>, Roberto Peon <[email-masked]<mailto:[email-masked]>>, HTTP Working Group <[email-masked]<mailto:[email-masked]>>, Lucas Pardue <[email-masked]<mailto:[email-masked]>>, IETF QUIC WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: Final DATA frames

---

From: Mike Bishop <[email-masked]>
Date: Wed, 12 Dec 2018 00:40:05 +0000

Based on the discussion, it sounds like we at least don’t have consensus to include this particular solution in the HTTP/3 draft.  I’ve reverted the change for the time being; if we get consensus on this or a different solution, I’m happy to re-add whatever we agree on.

I’d also encourage people on the list to monitor the GitHub issue e-mail, etc. and drop in to comment on issues that they don’t want addressed.  The fact that there was an apparent gap in consensus between the mailing list and the GitHub discussion in a working group that uses GitHub so heavily is… troubling.

From: Roberto Peon <[email-masked]>
Sent: Tuesday, December 11, 2018 9:43 AM
To: Ryan Hamilton <[email-masked]>
Cc: Martin Thomson <[email-masked]>; Mike Bishop <[email-masked]>; Dmitri Tikhonov <[email-masked]>; Kazuho Oku <[email-masked]>; Jana Iyengar <[email-masked]>; HTTP Working Group <[email-masked]>; Lucas Pardue <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: Final DATA frames

From: Ryan Hamilton <[email-masked]<mailto:[email-masked]>>
Date: Monday, December 10, 2018 at 5:56 PM
To: Roberto Peon <[email-masked]<mailto:[email-masked]>>
Cc: Martin Thomson <[email-masked]<mailto:[email-masked]>>, Mike Bishop <[email-masked]<mailto:[email-masked]>>, Dmitri Tikhonov <[email-masked]<mailto:[email-masked]>>, Kazuho Oku <[email-masked]<mailto:[email-masked]>>, Jana Iyengar <[email-masked]<mailto:[email-masked]>>, HTTP Working Group <[email-masked]<mailto:[email-masked]>>, Lucas Pardue <[email-masked]<mailto:[email-masked]>>, IETF QUIC WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: Final DATA frames

---

From: Ian Swett <[email-masked]>
Date: Tue, 11 Dec 2018 21:02:01 -0500

TLDR: If the focus is on shipping v1 and making sure we don't introduce
premature optimizations into HTTP/3, I thInk we should seriously consider
moving PUSH and the existing priorities to an extension.

It's too late on this issue, but I was going to add that this is very
useful for the CONNECT case.  Other comments below.

On Tue, Dec 11, 2018 at 12:46 PM Roberto Peon <[email-masked]> wrote:

> It isn’t like I hate optimization-- I’m just really scared of having to
> deal with this gone wrong. History has proven it is very likely that
> someone won’t get this right, and then protocol features will stop working,
> and pain will ensue, and the protocol will be less useful.
>

Why won't they get THIS right?  Your example below of PUSH_PROMISE is not
convincing, given the one proxy I am familiar with puts all PUSH_PROMISEs
before the response.

> If we could put the header/payload into separate streams, then that’d be a
> step forward along a path to enabling such optimization (and compatible
> with the additional changes you’d need to handle PP)
>

I'll note this is strictly more complex to implement.

>
> I’d rather get what we have now done, and look at further optimizations in
> a V2 that we get out the door after a very short time window.
> -=R
>

This issue is/was not blocking us getting QUIC V1 done, particularly the
HTTP draft.  There are things we could simplify that would help the HTTP
draft along, such as moving priorities and PUSH to extensions.

> *Date: *Monday, December 10, 2018 at 5:56 PM
> *To: *Roberto Peon <[email-masked]>
> *Cc: *Martin Thomson <[email-masked]>, Mike Bishop <
> [email-masked]>, Dmitri Tikhonov <[email-masked]>,
> Kazuho Oku <[email-masked]>, Jana Iyengar <[email-masked]>, HTTP
> Working Group <[email-masked]>, Lucas Pardue <
> [email-masked]>, IETF QUIC WG <[email-masked]>

---

From: Dmitri Tikhonov <[email-masked]>
Date: Wed, 12 Dec 2018 09:00:49 -0500

> moving PUSH and the existing priorities to an extension.

The EOS DATA frame and HTTP prioritization and push promises are not
equivalent.  One is simply fiddling with the way data is framed and
is not visible to the application.  The others are HTTP/2's primary
features.

  - Dmitri.

---

From: Ian Swett <[email-masked]>
Date: Wed, 12 Dec 2018 11:38:09 -0500

I don't want to derail this too much(probably too late for that), and no
one has a PR anywhere close to landing, but I do think we should start
thinking of these as potential extensions.  I disagree that they're
HTTP/2's primary features, both due to lack of use and lack of measurable
benefits years after standardization.  Push might actually benefit from
being an extension, because then potential improvements(ie: cache-digest)
could be integrated into the extension more quickly.

On the other hand, QPACK is relatively complex but we have clearer
demonstrations of it's benefit.

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 12 Dec 2018 16:48:43 +0000

> demonstrations of it's benefit.

The corollary of Ian's point is: extracting something like push allows for
iteration of HTTP features without affecting the core HTTP/3 syntax.
Innovation without having to write quic-transport-bis, ter etc. which may
require more coordination effort that iterating using extensions.

That of course would assume that H3 is flexible enough or has has enough
extension points.

Lucas

---

From: Dmitri Tikhonov <[email-masked]>
Date: Wed, 12 Dec 2018 12:02:04 -0500

But you would agree that these are completely different discussions?

One is whether to modify HTTP/3 framing mechanism.  The other is
changing HTTP/3 feature set.

My point is that it is unjustified to to say that since we have
decided not to change the framing mechanism we should consider
dropping prioritization and push promises.

---

From: Ian Swett <[email-masked]>
Date: Wed, 12 Dec 2018 12:26:51 -0500

>From a technical perspective, yes.  But from a goals/process perspective, I
think they related.  We've spent a lot of time on very complex work in the
QUIC WG, and I realize everyone has a different perspective, but I'm
confused by the amount of pushback this got relative to all the work we
have done to make PUSH and something approximating(since they're actually
different) H2 priorities work on top of QUIC.

If I remember correctly, your main argument against this framing change was
that the existing DATA frame with a length becomes largely useless and
might be under-exercised.  That makes me feel like we've over-optimized for
the complex special cases and made a design error or two at least one step
prior to this proposal.

To make a specific and relevant suggestion, maybe DATA should never have a
length and if we want features that most requests don't
use(trailers/PUSH_PROMISE/etc) we should put them on a separate
unidirectional stream?

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 12 Dec 2018 10:18:42 -0800

I think the current 0 length DATA makes sense in the context of solving a
real world problem.

But thinking forward to to v2, or possible late changes in v1, it makes
more sense to have DATA in a separate uni stream. If that is added later,
it will be messy to have a DATA frame not to mention having a 0-length DATA
frame.
The complexity is then supporting all kinds of old “hacks” when viewed from
a future version. In this light it makes sense to do as little as possible
now, iff a uni DATA stream makes sense in the future.

I kind of like Ians proposal:

because it is very workable and simple without preventing complex use
cases. But it also appears a bit like a hack but it has some ordering
properties that uni DATA streams don’t have, but my PUSH_PROMISE foo is
weak.

A separate uni DATA stream has the advantage that you can hand off data
flow to a separate processor while QPACK chews on head compression.

What are the real downsides to using using a uni DATA stream? timeline for
QUIC v1, coordination receiver side? …

Even if DATA frame len 0 is dropped while waiting for something better in
v2 (uni DATA streams), that will still leave an odd DATA frame that needs
to be back supported or removed. So I don’t think len 0 changes things
fundamentally here.

---

From: Roberto Peon <[email-masked]>
Date: Wed, 12 Dec 2018 18:41:07 +0000

The biggest downside is that it is very different, and will require changes to interop.
The next biggest is that we’ve not done it yet, and thus don’t understand any of the potential 2nd order effects, especially on performance.
For instance, how does one think about MAX_STREAM_ID when every HTTP request/response takes two uni streams?

-=R

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wednesday, December 12, 2018 at 10:18 AM
To: Jana Iyengar <[email-masked]>, Mike Bishop <[email-masked]>, IETF QUIC WG <[email-masked]>, Kazuho Oku <[email-masked]>, HTTP Working Group <[email-masked]>, Ryan Hamilton <[email-masked]>, Martin Thomson <[email-masked]>, Lucas Pardue <[email-masked]>, Ian Swett <ianswett=[email-masked]>, Roberto Peon <[email-masked]>
Subject: Re: Final DATA frames

I think the current 0 length DATA makes sense in the context of solving a real world problem.

But thinking forward to to v2, or possible late changes in v1, it makes more sense to have DATA in a separate uni stream. If that is added later, it will be messy to have a DATA frame not to mention having a 0-length DATA frame.
The complexity is then supporting all kinds of old “hacks” when viewed from a future version. In this light it makes sense to do as little as possible now, iff a uni DATA stream makes sense in the future.

To make a specific and relevant suggestion, maybe DATA should never have a length and if we want features that most requests don't use(trailers/PUSH_PROMISE/etc) we should put them on a separate unidirectional stream?

because it is very workable and simple without preventing complex use cases. But it also appears a bit like a hack but it has some ordering properties that uni DATA streams don’t have, but my PUSH_PROMISE foo is weak.

A separate uni DATA stream has the advantage that you can hand off data flow to a separate processor while QPACK chews on head compression.

What are the real downsides to using using a uni DATA stream? timeline for QUIC v1, coordination receiver side? …

Even if DATA frame len 0 is dropped while waiting for something better in v2 (uni DATA streams), that will still leave an odd DATA frame that needs to be back supported or removed. So I don’t think len 0 changes things fundamentally here.

---

From: Dmitri Tikhonov <[email-masked]>
Date: Wed, 12 Dec 2018 14:53:47 -0500

On Wed, Dec 12, 2018 at 12:26:51PM -0500, Ian Swett wrote:
> I'm confused by the amount of pushback this got relative to all
> the work we have done to make PUSH and something approximating
> (since they're actually different) H2 priorities work on top of
> QUIC.

The main knock on the EOS DATA proposal is that its benefits are
marginal relative to the perceived issues its adoption may engender.
The same cannot (or could not) be said about priorities and push
promises.

> that the existing DATA frame with a length becomes largely useless and
                                                             ^^^^^^^
> might be under-exercised.

s/useless/unused/

> That makes me feel like we've over-optimized for the complex
> special cases and made a design error or two at least one step
> prior to this proposal.

I disagree.  The current HTTP/3 framing mechanism is not over-
optimized; it is designed to handle the general case.  It is the
EOS DATA proposal that is an actual optimization.

  - Dmitri.

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 12 Dec 2018 18:13:34 -0800

Let's pull this train back on its track. I don't think it makes sense to
litigate for removing PUSH or anything else into an extension in this
thread. I have my thoughts on that, but I don't think this is the place to
discuss them.

There are two common deployment scenarios: (i) a server does not
use/support PUSH or trailers, or (ii) a server uses/supports PUSH or
trailers. In both cases, the server knows exactly where it falls. There may
be a third uncommon case where (iii) a server supports PUSH or trailers and
does not know if the backend server wants to use it or not.

This PR optimizes for (i), which, based on your deployment scenario, may be
not just common but exclusively true. I think we all agree that neither the
benefits nor the downsides are earth-shattering, but if you never have a
reason to send the length frame, it is quite a helpful optimization to your
code path.  You never have to know the size of the response.

FWIW, I don't see this as a particularly complex optimization, and I do see
the benefits as outweighing potential downsides. Dmitri brought up the
downside of a DATA frame with length not being used as much, but I'll note
that that only happens in a world where most deployed servers don't use
PUSH/trailers. And in that world, this optimization seems useful.

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 13 Dec 2018 11:44:01 +0900

2018年12月13日(木) 11:13 Jana Iyengar <[email-masked]>:
>
> Let's pull this train back on its track. I don't think it makes sense to litigate for removing PUSH or anything else into an extension in this thread. I have my thoughts on that, but I don't think this is the place to discuss them.
>
> There are two common deployment scenarios: (i) a server does not use/support PUSH or trailers, or (ii) a server uses/supports PUSH or trailers. In both cases, the server knows exactly where it falls. There may be a third uncommon case where (iii) a server supports PUSH or trailers and does not know if the backend server wants to use it or not.

I am not sure if calling (iii) being uncommon is correct. Unless you
have some out-of-band coordination, a reverse proxy cannot tell if a
H2 backend is going to send trailers (in case of H1, use of
content-length header implies that the backend is not going to send
trailers). The reverse proxy needs to assume that the backend is going
to send trailers.

>
> This PR optimizes for (i), which, based on your deployment scenario, may be not just common but exclusively true. I think we all agree that neither the benefits nor the downsides are earth-shattering, but if you never have a reason to send the length frame, it is quite a helpful optimization to your code path.  You never have to know the size of the response.
>
> FWIW, I don't see this as a particularly complex optimization, and I do see the benefits as outweighing potential downsides.

I agree that the downsides are small. However, the benefits of
omitting the length field are small too.

That is because typically, you know the length of the response (i.e.
content-length field). In such case, what you can do is to set the
length field of the DATA frame to that value. That's it.

The proposed optimization only makes sense if both of the following
conditions are true:
* the server knows that it is not going to send trailers or push promises
* the server does not know the length of the content beforehand

I am not sure if the second point is something we often observe. All
the static contents would have their length field predetermined. Web
applications have traditionally been built to emit content-length
headers, because without them, it was impossible to use a persistent
connection in HTTP/1.0.

> Dmitri brought up the downside of a DATA frame with length not being used as much, but I'll note that that only happens in a world where most deployed servers don't use PUSH/trailers. And in that world, this optimization seems useful.
>
>
> On Wed, Dec 12, 2018 at 11:53 AM Dmitri Tikhonov <[email-masked]> wrote:
>>

-- 
Kazuho Oku