Subject: updated http 1.1 rfcs and hop-by-hop

From: Martijn Faassen <[email-masked]>
Date: Mon, 11 Aug 2014 16:42:46 +0200

Hi there,

[I hope this question reaches the right mailing list]

I'm trying to find the equivalent of RFC 2616, section 13.5.1 in the
new specs. This section defines those header fields considered
hop-by-hop, i.e.:

      - Connection
      - Keep-Alive
      - Proxy-Authenticate
      - Proxy-Authorization
      - TE
      - Trailers
      - Transfer-Encoding
      - Upgrade

I need such a list in order to implement a Python WSGI-based proxy.
See PEP 3333:

http://legacy.python.org/dev/peps/pep-3333/#other-http-features

  However, because WSGI servers and applications do not communicate
via HTTP, what
  RFC 2616 calls "hop-by-hop" headers do not apply to WSGI internal
communications.
  WSGI applications must not generate any "hop-by-hop" headers [4],
attempt to use
  HTTP features that would require them to generate such headers, or
rely on the content
  of any incoming "hop-by-hop" headers in the environ dictionary. WSGI
servers must
  handle any supported inbound "hop-by-hop" headers on their own, such
as by decoding
  any inbound Transfer-Encoding, including chunked encoding if applicable.

So, a WSGI proxy has to remove any hop-by-hop headers from the
response of the host it is proxying, and not pass them along. WSGI has
its own iterator based mechanism to support chunking, and a WSGI
server can then add the appropriate headers.

In section 13.5.1 of RFC 2616 these hop-by-hop headers are specified,
but I can't find the equivalent in the newer RFCs.

I do find section 6.1 in rfc7230, so I think this means that a
conforming server would list related headers in the Connection. But
what about Transfer-Encoding, for instance? A server that sends
"Connection: close" may not send Transfer-Encoding?

It's certainly possible that I'm missing something. I haven't read the
complete specs yet. But any guidance would be appreciated.

Regards,

Martijn

---

From: Martijn Faassen <[email-masked]>
Date: Mon, 18 Aug 2014 10:39:01 +0200

Hi there,

No answer to my question at all so far? Perhaps it's the wrong list?
Any feedback would be appreciated!

Thank you!

Martijn

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 19 Aug 2014 17:37:33 +1200

> Any feedback would be appreciated!

There is no longer an implicit list of hop-by-hop headers.

If you are sending any header your proxy needs to comply with the sender
criteria for that header. The headers pointed out by RFC2616 are all
defined with a MUST on being added to Connection: header explicitly, or
were actually having hopX-to-hopY behaviour where X and Y may have any
number of intermediary hops between them.

You are free to code a default list from the header definitions in your
application. But its just a convenience to fix incorrect input (missing
Connection: entries).

Amos

---

From: Martijn Faassen <[email-masked]>
Date: Tue, 19 Aug 2014 10:35:49 +0200

Hi there,

I'm not sure I'm getting it yet. Does this mean that servers should be
sending out a Connection header that looks
like this?

Connection: close, Transfer-Encoding

Instead of just:

Connection: close

Or is 'Transfer-Encoding' a hopX-by-hopY behavior (I don't understand
what this is)?

I'm having trouble finding where the RFC currently specifies that a
particular header, such as Transfer-Encoding, should be in a
'Connection' entry.

Regards,

Martijn

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 19 Aug 2014 21:17:58 +1200

> Or is 'Transfer-Encoding' a hopX-by-hopY behavior (I don't understand

Transfer-Encoding has the hopX-to-hopY behaviour.

The server just emits Transfer-Encoding. A proxy receiving it is allowed
to strip away the header and payload encoding, but it is also allowed to
just relay both through - provided the receiving client is also HTTP/1.1
and can handle the decoding.

The Proxy-Auth* headers are the same. Get sent by whichever hop needs to
auth and stripped by whichever receiving hop requested authentication.

hopX-by-hopY (of hopX-to-hopY) is how I describe a setup like this:
 client-> proxy -> X -> proxy -> proxy -> Y -> proxy -> server

If X and Y are both negotiating the use of features between themselves
and over the proxies between. hop-by-hop and end-to-end being the well
known extreme/special cases of the general X-to-Y situation.

> particular header, such as Transfer-Encoding, should be in a

The TE header is one with mandatory Connection:TE. You can see that
"MUST" in the final paragraph of section 4.3:
"  sender of TE MUST also send a "TE" connection option within the
   Connection header field
"

The Upgrade header is another. You can see that "MUST" in the
4th-to-last paragraph of section 6.7:
"  sender MUST also send a Connection header
   field (Section 6.1) that contains an "upgrade" connection option
"

The Connection header itself is hop-by-hop, its "MUST" is in paragraph 2
of section 6.1:
"  A proxy or gateway MUST parse a received Connection header field
...
   then remove the Connection header field itself
   (or replace it with the intermediary's own connection options for the
   forwarded message).
"

HTH
Amos

---

From: Martijn Faassen <[email-masked]>
Date: Tue, 19 Aug 2014 12:35:26 +0200

Hey,

Thanks, this is helping my comprehension. I've written up a blog post
describing the relation between the new HTTP 1.1 RFCs and Python WSGI
spec here:

http://blog.startifact.com/posts/new-http-rfcs-versus-wsgi.html

Hopefully I didn't get it too wrong. I'll bring up this post to the
Python web special interest group too.

Thanks again!

Regards,

Martijn

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 20 Aug 2014 07:06:37 -0700

> Python web special interest group too.

Transfer-Encoding is not listed in Connection because removing it
without decoding the message body would be a severe error.

The notion that certain header fields are "hop-by-hop" in RFC2616
was poorly considered. Fields are always forwarded unless there exists
a specific requirement to delete them in transit, where each such
requirement effectively depends on the conformance and version of the
recipient.  Hence, the Connection header field has a specific requirement
to delete the listed fields and itself.

Transfer-Encoding does not have a specific requirement regarding
deletion of the field, but does require that its value correspond
to the transfer codings that are applied to the message body.
Hence, when one or more of those codings is decoded, the sender must
adjust the Transfer-Encoding field accordingly.

....Roy