Subject: HTTP/2 extensibility <draft-ietf-httpbis-http2-17>

From: Bob Briscoe <[email-masked]>
Date: Thu, 5 Mar 2015 12:55:07 +0000

HTTP/2 folks,

I've reviewed the whole draft. I know the draft has just successfully 
passed IESG review, but I hope this posting is still useful.

My credentials for this: first role in the IETF in 1995 was to ensure 
HTTP/1.1 generalised from Web pages to objects, but for the last 15 
years my focus has shifted down the layers into transport.
Non-credentials: I've been paying insufficient attention to HTTP/2 
until now, but I have tried to research back over the ML for the 
rationale behind design decisions.

So consider this as a late review from a clueful but fresh pair of eyes.

My main concerns are
* extensibility
* flow control
* numerous open issues left dangling
I'll cover extensibility here, and my other concerns (as well as 
nits) in subsequent posting.

Achieving this milestone on time has been impressive. I understand 
the reasons for having to get something standardised. However, I see 
potential problems. And that would be fine, but only if there were a 
more granular mechanism to extend the protocol to fix it in future.

For instance, a number of potential issues around DoS are left open. 
If the protocol has to be hardened against new attacks, I believe the 
recommended extensibility path is to design and implement a 
completely new protocol release, then upgraded endpoints negotiate it 
during the initial handshake. The timescale for such a process is 
measured in years, during which the vulnerability has to be lived 
with. Surely we need more granular extensibility; to introduce new 
frame types and structures, and/or to deprecate outdated/vulnerable ones.

Rather than a blanket statement saying that an endpoint discards and 
ignores a frame type that it does not recognise, we should include a 
field in the generic frame header to specify this behaviour. In this 
way, the currently defined extensibility behaviour could be required 
or relaxed depending on what the future brings - instead of having to 
decide the extensibility model now.

This would be safe, because the worst an attacker can do is inject a 
new unrecognised header and get it forwarded, but not acted on. 
Unless it is a type known to at least one implementation, it will 
never be acted on.

The same point applies to extending known frame types. There is no 
point having a length field on frames if any length other than that 
specified produces a FRAME_SIZE_ERROR. For extensibility, a frame 
with an unexpected length should at least be ignored, rather than 
leading to a stream error. Ideally, similar to above, there should be 
a field that specifies what action to take (forward or not) if the 
structure of the frame type is unrecognised.

One of the greatest strengths of HTTP/1.x was the general rule about 
unrecognised headers, which enabled 'a thousand flowers to bloom'. 
The few most beautiful ones survived. If evolution can only proceed 
in giant steps, the natural selection process will be glacially slow.

HTTP is important to us all. It has now become the only way to extend 
transport capabilities. The last thing we should do is build over the 
only pathway we have left with an evolutionary cul-de-sac.

Bob

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: Martin Thomson <[email-masked]>
Date: Thu, 5 Mar 2015 09:57:10 -0800

Hi Bob,

Thanks for your input.  Just two notes, to hopefully provide more
context for this decisions on this point.

On 5 March 2015 at 04:55, Bob Briscoe <[email-masked]> wrote:
> Achieving this milestone on time has been impressive. I understand the
> reasons for having to get something standardised. However, I see potential
> problems. And that would be fine, but only if there were a more granular
> mechanism to extend the protocol to fix it in future.

This was the subject of lengthy discussion, a good part of which you
won't find on the mailing list unfortunately.  However, there was
pretty strong consensus for the model you see before you.

Basically, there is a tension between the desire to be arbitrarily
extensible and the competing desires for both protocol robustness and
actually finishing on time.

Robustness (or robust interoperability) was considered top priority
for HTTP/2, trumping even the concerns you describe, which were
debated at some length.  However, the general principle driving the
design here was to make it very easy to detect when a peer has
diverged from the well-defined core protocol.  Detecting those errors
immediately and failing requests and connections ensures that
implementation errors do not persist long term.  Instead, they get
fixed.

Part of the reason HTTP/1.1 is such a pain to work with is the
numerous places where flexibility is permitted. Variations in HTTP/1.1
implementations have become entrenched, causing new implementations to
be forced into including all the same sort of ugly hacks or risk
interoperability failure.  An interoperable HTTP/1.1 implementation is
MUCH harder to build than it should be.

Many of the implementers of HTTP/2 had experience with the pain that
causes and so we were very careful to identify places where
implementations could deviate (to the extent possible) and
aggressively foreclose on those.

You might like to think of this from another perspective: Postel's
famous statement[1] is not a principle that guides protocol design, it
is in fact a fatalistic observation about the effect of entropic decay
of a protocol.

Finally, designing good extensibility is really, really hard.  It
takes time.  And as the saying goes, shipping is a feature.

> For instance, a number of potential issues around DoS are left open. If the
> protocol has to be hardened against new attacks, I believe the recommended
> extensibility path is to design and implement a completely new protocol
> release, then upgraded endpoints negotiate it during the initial handshake.
> The timescale for such a process is measured in years, during which the
> vulnerability has to be lived with. Surely we need more granular
> extensibility; to introduce new frame types and structures, and/or to
> deprecate outdated/vulnerable ones.

I don't know what others think, but I would expect that a small,
necessary change to the protocol could be deployed by those affected
in much the same timescale as an extension might be.  I do appreciate
the fact that calling something HTTP/3 to fix a small DoS bug might
*seem* drastic, but the only risk is in avoiding people with an axe to
grind trying to get in other changes when you do that.

(I'll also note that your concerns are largely only relevant in the
presence of intermediaries, for what that is worth.)

---

From: Mike Bishop <[email-masked]>
Date: Thu, 5 Mar 2015 18:22:09 +0000

Hi, Bob -

Let me try to address a few of your points.  The logic behind "discard unknown" is precisely so that new frame types can be added.  If you receive a frame type that you know and can validate the length, then you're able to reject it as an improper length -- but if it's a frame type you don't recognize, the length allows you to skip past it.  So, just as you suggest, the worst case is to inject something that gets ignored.  There were concerns raised with forwarding frame types you don't understand, particularly when you consider a connection to an intermediary where different streams may be going to different servers on the back-end.  If the frames on different streams were related in some way, but you could only untangle the relation if you understood the frames, the eventual recipients would get fragmentary conversations.  (Think of a shared compression context across streams, for example.)

For this reason, it was decided that extension frames should be hop-by-hop, but could be defined such that a recipient who understands the frame would relay it onto subsequent connections.  New extensions can define additional frames, and can define the processing of them.  That processing may include whether/how to forward them to the next hop, if you're an intermediary.  I would consider it good practice for any extension which defines frames to also define intermediary behavior, but the WG opted not to make that good practice a normative requirement.

It was also an explicit choice not to allow an endpoint to *unilaterally* change the definition of existing frames, because that leads to unknown behavior when you receive a known frame type of an unexpected length.  You don't know where the changes are, and whether fields still mean what you think they do.  However, the flexibility to modify rules by mutual agreement is the obvious way around that.  Define, as part of your extension, a setting that indicates your willingness to accept, say, an "opinion" string embedded in each SETTINGS ACK.  Upon receipt of that setting, someone who doesn't speak your extension ignores it and sends normal ACKs.  Someone who does speak your extension knows they can send you non-standard ACKs and you'll handle them appropriately without generating a FRAME_SIZE_ERROR.

-----Original Message-----
From: Bob Briscoe [mailto:[email-masked]] 
Sent: Thursday, March 5, 2015 4:55 AM
To: [email-masked]; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: HTTP/2 extensibility <draft-ietf-httpbis-http2-17>

HTTP/2 folks,

I've reviewed the whole draft. I know the draft has just successfully passed IESG review, but I hope this posting is still useful.

My credentials for this: first role in the IETF in 1995 was to ensure
HTTP/1.1 generalised from Web pages to objects, but for the last 15 years my focus has shifted down the layers into transport.
Non-credentials: I've been paying insufficient attention to HTTP/2 until now, but I have tried to research back over the ML for the rationale behind design decisions.

My main concerns are
* extensibility
* flow control

Bob

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Fri, 6 Mar 2015 00:47:29 +0000

Hi Bob,

> On 5 Mar 2015, at 12:55, Bob Briscoe <[email-masked]> wrote:

Can you expand a bit more on what these DoS vectors might be or is it more of a general concern?

> If the protocol has to be hardened against new attacks, I believe the recommended extensibility path is to design and implement a completely new protocol release, then upgraded endpoints negotiate it during the initial handshake. The timescale for such a process is measured in years, during which the vulnerability has to be lived with. Surely we need more granular extensibility; to introduce new frame types and structures, and/or to deprecate outdated/vulnerable ones.

As Martin points out, I don't see how deployment of a new extension/frame type/etc is likely to be any faster than a new protocol release.

Anyway, if necessary there is nothing to stop a future RFC modifying the base behaviour without a new protocol release (along with all the complexities of backwards compatibility that go along with that) if that is what the WG decides to do at the time a vulnerability is discovered IMO.

Also as Martin points out dealing with the flexibility allowed by HTTP/1.1 is a royal PITA and leads to lots of extra rarely executed code paths to deal with the numerous corner cases.

> HTTP is important to us all. It has now become the only way to extend transport capabilities. The last thing we should do is build over the only pathway we have left with an evolutionary cul-de-sac.

I don't think HTTP/2.0 is perfect but for the use cases it addresses it is an improvement over HTTP/1.1. It'll be a long time (if ever) IMO before HTTP/2.0 completely supplants HTTP/1.1. In that time there is plenty of scope to evolve along a different path if that is what is required, so I don't see it as a cul-de-sac but merely one possible evolutionary path. Time will tell whether HTTP/2.0 becomes the dominant species or another evolution branches from HTTP/1.1 or HTTP/2.0 itself evolves.

Ben

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 6 Mar 2015 13:35:10 +0000

Mike,

At 18:22 05/03/2015, Mike Bishop wrote:
>Hi, Bob -
>
>Let me try to address a few of your points.  The logic behind 
>"discard unknown" is precisely so that new frame types can be 
>added.  If you receive a frame type that you know and can validate 
>the length, then you're able to reject it as an improper length -- 
>but if it's a frame type you don't recognize, the length allows you 
>to skip past it.

That's only an argument for having a length field. It doesn't justify 
'discard unknown' any more than it justifies 'ignore unknown'. And it 
doesn't justify hard-coding the predetermined valid length into the 
implementation.

>So, just as you suggest, the worst case is to inject something that 
>gets ignored.  There were concerns raised with forwarding frame 
>types you don't understand, particularly when you consider a 
>connection to an intermediary where different streams may be going 
>to different servers on the back-end.  If the frames on different 
>streams were related in some way, but you could only untangle the 
>relation if you understood the frames, the eventual recipients would 
>get fragmentary conversations.  (Think of a shared compression 
>context across streams, for example.)

OK, that /is/ a valid argument for 'discard unknown' rather than 
'forward unknown'.

But I suggested a solution to that; if it's unknown, there could be a 
field in the frame that says whether to forward or discard it (aka. 
late binding, like in IPv6 extension headers). Then:
* if a new frame type would be susceptible to inconsistency over 
different paths, it can be flagged as 'discard unknown'.
* Otherwise it would be flagged as 'forward unknown'.
* And if we're not sure, we can play safe and flag it as 'discard 
unknown' anyway.

Then you could keep the tightly consistent design you want today (by 
setting discard unknown on all frame types), but you haven't 
foreclosed on a 'forward unknown' frame type if you really need it in future.

During the design discussions, did anyone suggest this 'late binding' idea?

>For this reason, it was decided that extension frames should be 
>hop-by-hop, but could be defined such that a recipient who 
>understands the frame would relay it onto subsequent 
>connections.  New extensions can define additional frames, and can 
>define the processing of them.  That processing may include 
>whether/how to forward them to the next hop, if you're an 
>intermediary.  I would consider it good practice for any extension 
>which defines frames to also define intermediary behavior, but the 
>WG opted not to make that good practice a normative requirement.

There seems to be an assumption that implementers will add new 
features even though they won't work e2e until everyone else has 
implemented and deployed the same feature. That seems to overlook the 
last two depressing decades of experience (at the transport layer) of 
chicken-and-egg deployment incentives.

>It was also an explicit choice not to allow an endpoint to 
>*unilaterally* change the definition of existing frames, because 
>that leads to unknown behavior when you receive a known frame type 
>of an unexpected length.  You don't know where the changes are, and 
>whether fields still mean what you think they do.  However, the 
>flexibility to modify rules by mutual agreement is the obvious way 
>around that.  Define, as part of your extension, a setting that 
>indicates your willingness to accept, say, an "opinion" string 
>embedded in each SETTINGS ACK.  Upon receipt of that setting, 
>someone who doesn't speak your extension ignores it and sends normal 
>ACKs.  Someone who does speak your extension knows they can send you 
>non-standard ACKs and you'll handle them appropriately without 
>generating a FRAME_SIZE_ERROR.

Yes. So again, my concern is that the spec requires frames to be 
discarded if they are of known type but unknown length; rather than 
using the 'late-binding' idea, to be able to 'ignore but forward unknown'.

Bob

>I've reviewed the whole draft. I know the draft has just 
>successfully passed IESG review, but I hope this posting is still useful.
>

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 6 Mar 2015 14:33:24 +0000

Martin, (and Ben who reinforced Martin's points),

So extensibility experience from the app-layer is being applied to a 
transport layer protocol (an unfortunate consequence of giving a name 
to a new transport protocol that makes it look like a major revision 
of an app-layer protocol).

In the IETF TCP maintenance WG, the bar to changes is very high, 
given insufficiently analysed and tested changes could be devastating 
to the Internet. When doing 'mundane' lower layer stuff like framing 
and multiplexing, it is much less likely that you will get an 
explosion of different ways of doing it. In TCPM, explosion of 
feature interactions is only a minor concern, because the number of 
revisions has remained small.

In the transport layer 'culture' changes take longer to introduce. 
But that's because the cost of failure is higher, and it's much 
harder to fix errors once they have been deployed.

inline...

>is in fact a fatalistic observation about the effect of entropic decay

I assume you're referring to
" In general, an implementation must be conservative
   in its sending behavior, and liberal in its receiving behavior." [RFC791]

This /is/ a principle that guides protocol design, but perhaps 
HTTP/1.x experience shows that it is less appropriate at higher 
layers. At lower layers, it has stood the test of time, and you don't 
hear anyone questioning it.

>takes time.  And as the saying goes, shipping is a feature.

Oh dear. I think we are witnessing a layering violation of cultures.

> > For instance, a number of potential issues around DoS are left open.

Ben, I'm referring to the large number of DoS concerns listed in the 
HTTP/2 spec in open-ended sentences that just state the concern. Each 
one made me think "So, why haven't you redesigned the protocol then?" 
If I had designed a protocol with those concerns, I wouldn't have 
even brought it to standardisation until I had at least some idea of 
a direction to address them. Again, this seems to be a culture 
difference between app-layer and lower layers.

>If the

Has the WG really thought through whether this extensibility approach 
is going to work? For instance, has a revision naming scheme been 
designed that can describe forks?

Also, it's not just about how a revision is /named/. It's about a 
completely different /process/...

HTTP/3 (or even HTTP/2.0.0.1) would need to be agreed by a whole 
standards committee process, no matter how minor the change. Whereas 
a new unilateral frame type could be experimented with locally, or 
used locally to patch a vulnerability.

>presence of intermediaries, for what that is worth.)

Correct, revisions (whether unliateral or standardised) are v hard to 
deploy if intermediate nodes must 'discard if unknown' (see the 
thread with Mike Bishop, that I have forked).

Bob

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 06 Mar 2015 15:02:24 +0000

--------
In message <[email-masked]>, Bob Briscoe 

>hear anyone questioning it.

The amount of things TCP can be liberal about can be counted on the
fingers of a single person, whereas the things HTTP can (have to)
be liberal about are mathematically uncountable.

This is entirely a situation of the HTTP communitys own making,
caused by having ill-conceived ideas battle it out with badly thought
out hacks, only in terms of market-share in an imperfectly operating
market, subject to monopoly size players with much to loose.

Some of us hoped that H2 would react sensibly to this nightmarish
situation, by tightening architecture and cleaning up semantics.

As you can see, that didn't happen, and all HTTP's daemons will
be able to make the transition to H2 effortlessly and new deamons
have already started to materialize.

>Oh dear. I think we are witnessing a layering violation of cultures.

Undoubtedly:  H2 is primarily a political protocol.

>difference between app-layer and lower layers.

I fully agree with you, H2 is at best a valuable prototype, but not
anywhere near the maturity you would expect from a protocol revision
of one of the worlds most used and popular protocols.

>is going to work?

No, it has not.

There's never time for that kind of thought in this WG.

...as you have no doubt already gathered from the answers to your
pertinent observations.

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 6 Mar 2015 17:09:46 +0000

Mike,

I omitted to comment on the rather unconventional structure of 
SETTINGS frames too, which makes SETTINGS both hard to extend and inefficient.

"A SETTINGS frame with a length other than a multiple of 6 octets 
MUST be treated as a connection error"
The SETTINGS frame structure seems not to use the extensible 
type-length-value (TLV) design, that has been tried and tested for 
binary protocols over decades.

The 16b SETTINGS type is currently used for 6 values (1-6). 
Extensibility to allow the other 65,530 values will be fairly 
pointless if each of their values has to fit within 32b.

A [T:L:V] of [8:8:variable] octets would probably have been 
sufficient, more efficient and far more flexible.

Bob

[Side-stepping Poul-Henning's comments that are probably true, but 
perhaps not phrased in a way that will persuade you to reconsider.]

At 13:35 06/03/2015, Bob Briscoe wrote:
>That's only an argument for having a length field. It doesn't 
>justify 'discard unknown' any more than it justifies 'ignore 
>unknown'. And it doesn't justify hard-coding the predetermined valid 
>length into the implementation.
>
>>relation if you understood the frames, the eventual recipients 
>>would get fragmentary conversations.  (Think of a shared 
>>compression context across streams, for example.)
>
>But I suggested a solution to that; if it's unknown, there could be 
>a field in the frame that says whether to forward or discard it 
>(aka. late binding, like in IPv6 extension headers). Then:
>implemented and deployed the same feature. That seems to overlook 
>the last two depressing decades of experience (at the transport 
>layer) of chicken-and-egg deployment incentives.
>
>>someone who doesn't speak your extension ignores it and sends 
>>normal ACKs.  Someone who does speak your extension knows they can 
>>send you non-standard ACKs and you'll handle them appropriately 
>>without generating a FRAME_SIZE_ERROR.
>