Subject: [Technical Errata Reported] RFC7230 (5163)

From: RFC Errata System <[email-masked]>
Date: Fri, 20 Oct 2017 15:28:45 -0700 (PDT)

The following errata report has been submitted for RFC7230,
"Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing".

--------------------------------------
You may review the report below and at:
http://www.rfc-editor.org/errata/eid5163

--------------------------------------
Type: Technical
Reported by: David Matson <[email-masked]>

Section: 3.2.4

Original Text
-------------
A field value might be preceded and/or followed by optional
whitespace (OWS); a single SP preceding the field-value is preferred
for consistent readability by humans.  The field value does not
include any leading or trailing whitespace: OWS occurring before the
first non-whitespace octet of the field value or after the last
non-whitespace octet of the field value ought to be excluded by
parsers when extracting the field value from a header field.

Corrected Text
--------------
A field value might be preceded and/or followed by optional
whitespace (OWS); a single SP preceding the field-value is preferred
for consistent readability by humans.  The field value does not
include any leading or trailing whitespace: OWS occurring before the
first non-whitespace octet of the field value or after the last
non-whitespace octet of the field value ought to be excluded by
parsers when extracting the field value from a header field.

All optional whitespace between tokens in field-content has the same
semantics as SP. Any sequence of SP / HTAB that occurs between tokens
in field-content MAY be replaced with a single SP before interpreting
the field value or forwarding the message downstream.

Notes
-----
RFC 2616, section 2.2, contained the following text:

All linear white space, including folding, has the same semantics as SP. A recipient MAY replace any linear white space with a single SP before interpreting the field value or forwarding the message downstream.

Similarly, RFC 2616 section 4.2 contained the following text:
Any LWS that occurs between field-content MAY be replaced with a single SP before interpreting the field value or forwarding the message downstream.

In section A.2. Changes from RFC 2616, the document does not list any intended change for how space and tab are handled, but the current text does appear to constitute a change. I suspect the change is accidental due to rewording the document when line folding was made deprecated.

Note that in RFC 2616, LWS is defined as follows:
LWS            = [CRLF] 1*( SP | HT )

In particular, the leading CRLF was optional.

Thus, the wording in RFC 2616 covered two cases:
1. LWS that includes line folding.
2. LWS that does not include line folding.

The current text does cover how to handle case #1 - former LWS that began with a CRLF; later in section 3.2.4 it requires rejecting or replacing with SP. (The old "MAY" language has effectively become a "MUST" for the leading CRLF case.)

However, the current text does not appear to address case #2 - former LWS that does not begin with a CRLF - in other words, SP and HTAB occurring between field-content. I suspect the intention is still that a recipient should treat such whitespace as insignificant, and may replace any sequence of SP and HTAB with a single SP before interpreting the field content, but I believe the text of the current RFC no longer provides this behavior.

(I have not read all of the specifications in full, so please accept my apologies if I have misread or missed a relevant portion elsewhere.)

Instructions:
-------------
This erratum is currently posted as "Reported". If necessary, please
use "Reply All" to discuss whether it should be verified or
rejected. When a decision is reached, the verifying party  
can log in to change the status and edit the report, if necessary. 

--------------------------------------
RFC7230 (draft-ietf-httpbis-p1-messaging-26)
--------------------------------------
Title               : Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing
Publication Date    : June 2014
Author(s)           : R. Fielding, Ed., J. Reschke, Ed.
Category            : PROPOSED STANDARD
Source              : Hypertext Transfer Protocol Bis APP
Area                : Applications
Stream              : IETF
Verifying Party     : IESG

---

From: Julian Reschke <[email-masked]>
Date: Wed, 25 Oct 2017 10:12:41 +0200

On 2017-10-21 00:28, RFC Errata System wrote:
> ...

I believe that RFC 7230 really says what it's supposed to say, so this 
is no erratum:

- we got rid of LWS altogether
- OWS, as the name says, is "OPTIONAL"
- generic code can't handle OWS in field values without knowing the ABNF 
of the header field

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 25 Oct 2017 19:47:31 +0200

On 2017-10-25 19:31, David Matson wrote:
> When generic code does know the ABNF of the header field, is it still allowed to remove spaces in field values before processing/passing downstream? If so, where should this behavior be documented as permissible? (Is this something that each field now needs to document separately?)

By my definition, it wouldn't be generic anymore :-)-

> (FWIW, the spec doesn't technically use OWS inside field-content; the ABNF between field-vchar is equivalent to RWS, I believe.)

OWS appears as part of list productions, see 
<https://www.greenbytes.de/tech/webdav/rfc7230.html#abnf.extension>. It 
also appears in specific header fields, such as 
<https://www.greenbytes.de/tech/webdav/rfc7231.html#media.type>...

> For example, if a proxy understands the ABNF of all the following headers and receives a response with Allow, Server, Vary, etc header fields set where there's more than one space separating items in the list, can it still normalize down to one space before passing the response back to the client? The Allow case is perhaps the most interesting - RFC 7231 specifically prohibits modifying the Allow header field, but I suspect the intent there is more around the list of methods rather than the whitespace characters between them.
> ...

Aha: 
<https://www.greenbytes.de/tech/webdav/rfc7231.html#rfc.section.7.4.1.p.4>:

"A proxy MUST NOT modify the Allow header field — it does not need to 
understand all of the indicated methods in order to handle them 
according to the generic message handling rules."

I agree with you about the intent, and that modifying OWS here would be 
harmless. Do we need to state that, and yes, what header fields are 
affected?

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 30 Oct 2017 14:49:13 +0100

On 2017-10-25 20:04, David Matson wrote:
> Fair point :)
>>> I agree with you about the intent, and that modifying OWS here would be harmless. Do we need to state that, and yes, what header fields are affected?
> 
> I think that would be useful. If the we did make such a statement, should it be in RFC7230 or elsewhere? Perhaps RFC7230 could say something like this (phrased perhaps too technically):
> "If the ABNF for a header field's content includes RWS or OWS, this whitespace MAY be replaced with a single SP before interpreting the field value or forwarding the message downstream."
> 
> For context, the reason this came up is that in Microsoft Azure Storage, certain header fields are canonicalized to be used in a cryptographic hash, and the canonocalized form normalizes this whitespace to a single SP to ensure the signature doesn't break when a proxy makes the kind of transformation formerly allowed under RFC 2616. Had this transformation not been permitted by the RFC, I suspect we would have left the inside of the string as-is when canonicalizing.
> https://docs.microsoft.com/en-us/rest/api/storageservices/authentication-for-the-azure-storage-services
> ...

We probably should clarify

   https://www.greenbytes.de/tech/webdav/rfc7230.html#rule.OWS

with statements about what kind of rewriting is allowed. And, for

   https://www.greenbytes.de/tech/webdav/rfc7231.html#header.allow

we need to clarify that "MUST NOT modify" doesn't apply to the *WS 
rewriting we allow in general.

Chairs: should we handle that as erratum or just add it to 
<https://github.com/httpwg/http11bis/issues>?

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 31 Oct 2017 10:09:08 +1100

I'd lean towards filing an issue.

> On 31 Oct 2017, at 12:49 am, Julian Reschke <[email-masked]> wrote:
> 
> we need to clarify that "MUST NOT modify" doesn't apply to the *WS rewriting we allow in general.
> 
> Chairs: should we handle that as erratum or just add it to <https://github.com/httpwg/http11bis/issues>?
> 
> Best regards, Julian
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: David Matson <[email-masked]>
Date: Wed, 25 Oct 2017 18:05:02 +0000

Fair point :)

Thanks,

David

-----Original Message-----
From: Julian Reschke [mailto:[email-masked]] 
Sent: Wednesday, 25 October 2017 10:48
To: David Matson <[email-masked]>; RFC Errata System <[email-masked]>; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: Re: [Technical Errata Reported] RFC7230 (5163)

On 2017-10-25 19:31, David Matson wrote:
> When generic code does know the ABNF of the header field, is it still 
> allowed to remove spaces in field values before processing/passing 
> downstream? If so, where should this behavior be documented as 
> permissible? (Is this something that each field now needs to document 
> separately?)

> (FWIW, the spec doesn't technically use OWS inside field-content; the 
> ABNF between field-vchar is equivalent to RWS, I believe.)

OWS appears as part of list productions, see <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.greenbytes.de%2Ftech%2Fwebdav%2Frfc7230.html%23abnf.extension&data=02%7C01%7Cdmatson%40microsoft.com%7C649df737b5a6487f6ebb08d51bd085d5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636445504782504514&sdata=ACu9NO8NGmE5KuMNcEG5fwJKbD3NyzUQzZf13T7bwzU%3D&reserved=0>. It also appears in specific header fields, such as <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.greenbytes.de%2Ftech%2Fwebdav%2Frfc7231.html%23media.type&data=02%7C01%7Cdmatson%40microsoft.com%7C649df737b5a6487f6ebb08d51bd085d5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636445504782504514&sdata=PpGHTLxqOhS1CHI3zy%2BVgBYzC6mb8pV8lY4TF7F%2FAWI%3D&reserved=0>...

> For example, if a proxy understands the ABNF of all the following headers and receives a response with Allow, Server, Vary, etc header fields set where there's more than one space separating items…

Aha: 
<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.greenbytes.de%2Ftech%2Fwebdav%2Frfc7231.html%23rfc.section.7.4.1.p.4&data=02%7C01%7Cdmatson%40microsoft.com%7C649df737b5a6487f6ebb08d51bd085d5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636445504782504514&sdata=bTGIQzF%2Fi5SKMKTO5iLtBZetQe%2B%2FflpskHvU6zMtQkc%3D&reserved=0>:

"A proxy MUST NOT modify the Allow header field — it does not need to understand all of the indicated methods in order to handle them according to the generic message handling rules."

Best regards, Julian