Subject: [QUIC] Drafts structure and split

From: Jana Iyengar <[email-masked]>
Date: Fri, 21 Oct 2016 12:13:15 -0700

Hello folks,

There's been some really useful feedback on some of the input drafts from
folks on the list, and I'd like to update them before the drafts deadline,
so the changes are part of the wg discussions in Seoul. I also wondered if
folks had any specific comments on draft structure and split at this stage,
so the authors of the input drafts could get those in before the
conversation about adoption.

Thanks for all the feedback so far,
- jana

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 21 Oct 2016 22:25:29 +0200

Hi,

I been wondering about this. Sometimes, i feel that the loss detection 
and reliability belongs to the protocol document (at least the basic 
loss detection and retransmision) and leave the congestion control in a 
separated document. I mean, it seems that loss detection and 
retransmission is too closely related to the basic protocol.

But i dont really have a strong opinion.
Regards, marcelo

El 21/10/16 a las 21:13, Jana Iyengar escribió:
> Hello folks,
>
> There's been some really useful feedback on some of the input drafts 
> from folks on the list, and I'd like to update them before the drafts 
> deadline, so the changes are part of the wg discussions in Seoul. I 
> also wondered if folks had any specific comments on draft structure 
> and split at this stage, so the authors of the input drafts could get 
> those in before the conversation about adoption.
>
> _______________________________________________
> QUIC mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/quic

---

From: Christian Huitema <[email-masked]>
Date: Fri, 21 Oct 2016 13:35:42 -0700

There is a natural layering of stream controls on top of a packet transmission layer, which itself depends on security negotiations performed on stream 1.

The packet transmission layer would specify packet transmission, ack and error detection, and packet level flow control and connection control. In my mind, it should not specify retransmission, since that is stream specific, but it should notify the stream level when a stream frame has been lost, or acknowledged.

Congestion control should be pluggable.

-- Christian Huitema 

> On Oct 21, 2016, at 1:25 PM, marcelo bagnulo braun <[email-masked]> wrote:
> 
> Hi,
> 
> I been wondering about this. Sometimes, i feel that the loss detection and reliability belongs to the protocol document (at least the basic loss detection and retransmision) and leave the congestion control in a separated document. I mean, it seems that loss detection and retransmission is too closely related to the basic protocol.
> 
>> There's been some really useful feedback on some of the input drafts from folks on the list, and I'd like to update them before the drafts deadline, so the changes are part of the wg discussions in Seoul. I also wondered if folks had any specific comments on draft structure and split at this stage, so the authors of the input drafts could get those in before the conversation about adoption.
>>

---

From: Jana Iyengar <[email-masked]>
Date: Fri, 21 Oct 2016 14:53:32 -0700

I think perhaps what the document should really be called is not "loss
recovery", but "loss detection". Pretty much everything in the doc is
expected to be about how to detect losses, and not so much about
retransmitting. What gets retransmitted and/or rebundled in actually
specified currently in the main protocol spec. Since loss detection is
intertwined with the currently specified congestion controllers for TCP, it
seems natural to combine them in a single doc.

There were two reasons for moving the loss detection and CC out into a
separate doc: so the main spec could be smaller, and so the loss detection
mechanisms could be independently evolved (later) as updates to the loss
recovery spec. Much like 793 and 2581 (though admittedly separate evolution
was not the concern then.) It might also help with reviews later on, since,
for example, folks in tcpm ought to be able to read, review, and understand
the similarities with and differences from TCP in terms of loss detection
without having to closely follow the rest of the QUIC spec.

On Fri, Oct 21, 2016 at 1:25 PM, marcelo bagnulo braun <[email-masked]>
wrote:

> Hi,
>
> I been wondering about this. Sometimes, i feel that the loss detection and
> reliability belongs to the protocol document (at least the basic loss
> detection and retransmision) and leave the congestion control in a
> separated document. I mean, it seems that loss detection and retransmission
> is too closely related to the basic protocol.
>

---

From: Jana Iyengar <[email-masked]>
Date: Fri, 21 Oct 2016 15:15:26 -0700

Christian,

I definitely like your layering in terms of an intellectual separation.

Tell me if you think differently, but I think the documents roughly reflect
some of the separation you suggest, but we've tried to keep the number of
docs from becoming too large (we already have 4). The stream abstraction
and the general protocol mechanisms for packet transmission are described
in the core protocol spec. Since streams and connection signaling may
require reliability, retransmissions and rebundling logic is described in
this doc.

The loss detection and congestion control spec describes how to use various
signals to detect losses, and (will describe) how to modulate the
congestion controller based on losses and received acks. This makes both
the congestion controller and the loss detection pluggable. To be clear,
even if these were to become a part of the core protocol document, I would
have a strong recommendation in there for implementers to keep their
congestion controller and loss detection machinery separate and pluggable.
I think keeping the docs separate helps for reviews and later updates,
i.e., "document pluggability."

- jana

On Fri, Oct 21, 2016 at 1:35 PM, Christian Huitema <[email-masked]>
wrote:

> There is a natural layering of stream controls on top of a packet
> transmission layer, which itself depends on security negotiations performed
> on stream 1.
>
> The packet transmission layer would specify packet transmission, ack and
> error detection, and packet level flow control and connection control. In
> my mind, it should not specify retransmission, since that is stream
> specific, but it should notify the stream level when a stream frame has
> been lost, or acknowledged.
>
> and reliability belongs to the protocol document (at least the basic loss
> deadline, so the changes are part of the wg discussions in Seoul. I also
> wondered if folks had any specific comments on draft structure and split at
> this stage, so the authors of the input drafts could get those in before
> the conversation about adoption.
> >>

---

From: Jana Iyengar <[email-masked]>
Date: Fri, 21 Oct 2016 15:23:56 -0700

One thing I meant to but forgot to mention:

> require reliability, retransmissions and rebundling logic is described in

(Since this doc has streams, it also has Flow Control.) I can see this doc
being separated into two parts, split at the interface between the stream
abstractions and the packet transport logic. I wonder how useful it would
be to split this into two docs. We could point out that there are these two
distinct parts to the transport (which we can do within the core protocol
spec.)

---

From: "Eggert, Lars" <[email-masked]>
Date: Sat, 22 Oct 2016 07:28:39 +0000

On 2016-10-21, at 21:13, Jana Iyengar <[email-masked]> wrote:
> I also wondered if folks had any specific comments on draft structure and split at this stage, so the authors of the input drafts could get those in before the conversation about adoption.

Without my chair hat on, I had been wondering if it makes sense to break out the overview parts from core-protocols and stick into a new "start reading here" document.

Lars

---

From: Brian Trammell <[email-masked]>
Date: Sat, 22 Oct 2016 18:24:27 +0200

> On 22 Oct 2016, at 09:28, Eggert, Lars <[email-masked]> wrote:
> 

FWIW, I had the same thought.

Cheers,

Brian

---

From: Victor Vasiliev <[email-masked]>
Date: Sat, 22 Oct 2016 17:55:27 -0400

My intuitive split would be:
* Overview document
* Packet layer document -- describes packet wire format, record-level
protection, ACK/STOP_WAITING, loss detection and congestion control.
* Stream layer document -- describes streams, flow control, and other
higher-level issues.
* Session establishment using TLS 1.3
* HTTP/2 mapping

I believe this split would be useful for the implementers given that this
layering seems to have worked out well in Google's current QUIC
implementation.

On Fri, Oct 21, 2016 at 6:23 PM, Jana Iyengar <[email-masked]> wrote:

> Tell me if you think differently, but I think the documents roughly
>> reflect some of the separation you suggest, but we've tried to keep the
>> number of docs from becoming too large (we already have 4). The stream
>> abstraction and the general protocol mechanisms for packet transmission are
>> described in the core protocol spec. Since streams and connection signaling
>> may require reliability, retransmissions and rebundling logic is described
>> in this doc.
>>
>

---

From: Kyle Rose <[email-masked]>
Date: Sun, 23 Oct 2016 14:21:28 -0400

> may require reliability, retransmissions and rebundling logic is described

Even though FEC implementations are out of scope for the current charter,
it may make sense to structure the documents and the interfaces between
layers such that loss detection and ECN are fully independent of correction
and compensation mechanisms: these could be retransmits or FEC, but they
could also be AQM or notifications to the application to, for example,
reduce media quality.

Kyle

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Sun, 23 Oct 2016 21:50:57 -0500

Speaking as an AD who has been wrong before (references available on
request),

> Kyle

Whether or not the QUIC protocol ever does FEC, I'm thinking that the
division between loss detection and loss recovery would be helpful.

The observations that TCP ended up with the same split by accident that you
folks are considering doing intentionally was helpful for me.

Thanks,

Spencer

---

From: Jim Roskind <[email-masked]>
Date: Sun, 23 Oct 2016 20:29:10 -0700

First off, it is super exciting to me personally to see QUIC progressing so
well in the IETF!!!

The Design and Rationale of QUIC stressed several things:

a) The goal of QUIC was to reduce latency
b) There were numerous gains to be made by integrating elements of highly
evolved portions of traditional layered protocols.

As a result, I'm hopefully to see fewer, rather than more, documents. I'm
hoping to see tight and clear integration, rather than a layered protocol,
with layered documentation, falling victim to the many latency losses of
historical layers such HTTP over SPDY over TLS over TCP over IP.

That cross layer integration has already shown remarkable gains in QUIC.
For example, the integration of the crypto with awareness about
packetization boundaries was a great boon for both connection establishment
(low-latency handshakes; key change; etc.), as well as precluding of head
of line blocking (induced at the level of IP packet loss, but historically
surfacing in block encryption, if not TCP sequential streaming).

A second example is the clarity of packet loss information provided by QUIC
stream-content-rebundling (rather than packet re-transmission) has
apparently enhanced even traditional TCP (RENO?) congestion control results.

I'm very excited about the (finally published in ACM Queue) details of
Google's Congestion Control algorithm entitled BBR (Buffer Bloat
Reduction), which provides tremendous latency savings to TCP by minimizing
buffer size, and hence buffer latency.  I'm am filled with anticipation
about the integrated benefits with QUIC (and its primary goal, to reduce
latency), which should provide more precise transit-time information atop
BBRs incredible published results, and hence still better monitoring of
bottleneck buffer variations. With latency as a goal for QUIC, it is very
desirable to tightly integrate such latency-enhancing advances.

As a result, I'm hoping to see fewer, rather than more, independent
documents.  Each separate document will seemingly make it more difficult to
discuss variations, and potential tight integration in the QUIC protocol.

I do think it makes sense to separate discussion of packet loss detection
(which serves as one of the signals to any congestion control algorithm)
 from stream-content restransmission (which is surely dependent on the
application which QUIC serves), I don't see that such separation requires a
separate document.  I would however like some plugable API (for instance)
to support unreliable transport (probably outside of standardization of
existing practice).  That said, I would like to see direct (status quo)
discussion of how QUIC responds to retranmitted stream content, independent
of cause/motivation (loss recovery?  external IP packet replication due to
transmission media events?).

Bottom line: Hopefully fewer, well integrated documents.

I'm certainly willing to help edit and work on any of these documents.

Thanks,

Jim Roskind (who is only an IETF noob, and could be getting confused by
this discussion)

---

From: Kyle Rose <[email-masked]>
Date: Mon, 24 Oct 2016 10:24:10 -0400

> historical layers such HTTP over SPDY over TLS over TCP over IP.

Layering definitely requires more effort to get right (for some definition
of "right": even the very wise cannot see all ends). Intentionally building
a monolith creates a different set of problems.

> surfacing in block encryption, if not TCP sequential streaming).

To get this right, one had to notice the tight interaction between the
crypto and packetization in advance. No matter how the documents are
structured, there are still two layers there and therefore still an
interface between them: the question is whether to make the interface
explicit and pluggable, or implicit and difficult to reuse.

I don't think the number of documents actually matters all that much, but
defining clear, bounded layers, along with appropriate information flow
between them to enable current design goals (e.g., low latency) and
foreseeable expanded functionality, does: putting effort in now will
hopefully land the inevitable ossification in places that are easier for
future protocol designers to deal with.

Kyle

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Mon, 24 Oct 2016 18:06:58 -0500

If I might offer an opinion, speaking as an individual ...

> future protocol designers to deal with.

If people are expecting a loss recovery mechanism beyond the current
version of BBR to ever appear in QUIC, what I would love is a plugable
*document structure*, where the loss recovery text that is being updated,
obsoleted, or will just be competing with existing mechanism(s) is clear,
without opening parts of the spec that aren't changing.

Thanks,

Spencer, speaking as an individual

---

From: Ian Swett <[email-masked]>
Date: Mon, 24 Oct 2016 20:31:07 -0400

Agreed, I think that structure makes sense, and as Kyle noted, makes
experimentation with FEC more straightforward.  One can easily imagine a
loss recovery approach where a given stream frame is never retransmitted,
but coded information is.

It's worth noting that BBR is only a change to the congestion controller,
and RACK is designed as a change the loss recovery.

On Mon, Oct 24, 2016 at 7:06 PM, Spencer Dawkins at IETF <
[email-masked]> wrote:

>>> As a result, I'm hopefully to see fewer, rather than more, documents.
>>> I'm hoping to see tight and clear integration, rather than a layered
>>> protocol, with layered documentation, falling victim to the many latency
>>> losses of historical layers such HTTP over SPDY over TLS over TCP over IP.
>>>
>>
>> Layering definitely requires more effort to get right (for some
>> definition of "right": even the very wise cannot see all ends).
>> Intentionally building a monolith creates a different set of problems.
>>
>>

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 24 Oct 2016 20:33:44 -0700

> could also be AQM or notifications to the application to, for example,

Agreed, to detection vs. recovery. I think we could articulate this more
explicitly in the core spec, so I'll give it a shot. I'll also ask you to
remember this (and maybe suggest text if necessary) when reviewing the next
rev of the document :-)

On AQM/ECN, I expect that the signal will be passed to the congestion
controller, and should therefore form part of the interface between the
core protocol and the congestion controller. Any notificaitons to the
application can be interactions with the congestion controller. Does that
make sense?

---

From: Kyle Rose <[email-masked]>
Date: Tue, 25 Oct 2016 15:07:50 -0400

> remember this (and maybe suggest text if necessary) when reviewing the next

Will do.

> application can be interactions with the congestion controller. Does that

Some applications might care only about loss or preventing future loss
("does/will the client have enough information to reconstruct the stream,
or do I need to retransmit/send more ECC?"), while others might care about
congestion separately from loss (the previous question vs. "do I need to
reduce the bitrate?"), so having separate signaling for the two cases makes
a lot of sense.

Kyle

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 25 Oct 2016 12:37:09 -0700

> reduce the bitrate?"), so having separate signaling for the two cases makes

ECN signals indicate only congestion, not loss. My point was that
congestion signals - explicit ones via ECN or implicit ones via loss/delay
-- are all fed to the congestion controller. I agree that loss recovery is
an entirely different matter, and that's a signal that comes out of the
Loss Detection component. Make sense?

If you're pointing out that congestion control and loss detection are
separable, I agree. Perhaps this separation could be made explicit in the
Loss Detection and Congestion Control doc? Or would you want to separate
the two out? My earlier inclination was to keep fewer documents until they
become large enough to need separation, but I can see arguments for
document pluggability too.

- jana

---

From: Kyle Rose <[email-masked]>
Date: Tue, 25 Oct 2016 17:34:30 -0400

> Loss Detection component. Make sense?

Yep, I agree completely. I think maybe what I meant was more clearly stated
as "Applications may (will?) want to react differently to loss or
prospective loss from congestion, versus loss for other reasons (like poor
wifi signal)." We may currently have difficulty distinguishing the former
from the latter when ECN isn't a universal congestion signal, but having
the proper interface in place allowing applications to tailor behavior
based on improved future telemetry could be useful.

> document pluggability too.

I agree with Spencer's point about document pluggability: it's less
important how many documents the WG produces than that they use
encapsulation to make it easy to swap out or replace components later
without having to individually update sentences from 17 different sections.

Kyle

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 25 Oct 2016 18:18:40 -0700

On Tue, Oct 25, 2016 at 2:34 PM, Kyle Rose <[email-masked]> wrote:

> Yep, I agree completely. I think maybe what I meant was more clearly
> stated as "Applications may (will?) want to react differently to loss or

Sounds good to me.

- jana

---

From: Jim Roskind <[email-masked]>
Date: Tue, 25 Oct 2016 19:13:11 -0700

On Mon, Oct 24, 2016 at 5:31 PM, Ian Swett <[email-masked]> wrote:

> ...
> ....
BBR relies on getting estimates of transit time, so as to estimate
variations in buffer sizes.  As a relatively rough estimate, I believe BBR
in TCP relies on traditional-TCP RTT calculations, which includes the round
trip sum of transit times in both directions.  I expect that better one-way
transit time estimates can be harvested in QUIC, by changing the data
included in ACK frames to better illuminate transit times in each
directions (or more precisely, illuminate changes, from one packet to the
next, in one-way transit times).

Such modifications appear at first blush to be notably larger than just a
change to the congestion controller.

Similarly, I'd expect modifications to support ECN in QUIC to run much
deeper than a congestion controller.

Jim

---

From: Ryan Hamilton <[email-masked]>
Date: Tue, 25 Oct 2016 21:09:28 -0700

> next, in one-way transit times).

​Correct me if I'm wrong, but QUIC ACK frames already have this
information. The contain the arrival time of each packet at the peer, which
provides the one-way transfer time.
​

---

From: Jim Roskind <[email-masked]>
Date: Wed, 26 Oct 2016 08:39:32 -0700

On Tue, Oct 25, 2016 at 9:09 PM, Ryan Hamilton <[email-masked]> wrote:

>>> It's worth noting that BBR is only a change to the congestion
>>> controller, ....
>>>
>>>
>>>

Absolutely (and since you implemented and are probably maintaining this in
the Google implementation, you'd know if there was any change in that
status quo). The cross-stack requirements in the QUIC Rationale and Design
Doc motivated the placement and use of timestamps in ACK frames for
evaluating bottleneck bandwidth back in 2012. All that "frame level" design
was part of congestion control design, even though at the time we were only
using traditional TCP-style congestion control (which did not call for the
time extra timestamps).

All of this is indeed a reflection of the tight integration between the
elements, and across the stack. That is why I'd argue that it is difficult
to divide a QUIC standard (into a multitude of separate docs) across
layers, when delicate details in one layer are heavily depended upon, and
perchance varied greatly, to better suit needs of would-be "other layers."

For instance, consideration may be given to adjust the resolution, range,
or coding of those ACK-frame timestamps specifically for congestion
algorithms like BBR, but such a change would sadly not be confined to
otherwise isolated docs about congestion avoidance. :-(  As other folks
look at, and experiment with, other variants of congestion control, such as
ECN, it becomes plausible that different metrics will be needed at
different levels of the protocol.

It would be a shame if standardization in one "layer" made it more
difficult to rapidly evolve broader advances for QUIC.

I guess on the other side, it sure would be nice if we were all brilliant
enough to standardize ALL required hooks...but that seems more than hard.

If we can find nice partitioning points, that would be great, but I was
hard pressed to expect much more than the crypto separation, and still
allow for rapid ongoing evolution. Even with crypto, there were interesting
cross-stack requirements, but getting crypto right is soooo hard, that
separation seemed very prudent (along with somewhat freezing supporting
issues elsewhere in the protocol).

Jim

[Opinions expressed are my own, and not that of my employer]