Subject: Proposal for #486: Requiring proxies to process warn-date

From: Mark Nottingham <[email-masked]>
Date: Mon, 2 Sep 2013 10:47:58 +1000

<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/486>

We discussed this issue in Berlin briefly, and I had an action to make a concrete proposal.

In <https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p6-cache.html#header.warning>:

OLD:
---8<---
   If an implementation sends a message with one or more Warning header fields
   to a receiver whose version is HTTP/1.0 or lower, then the sender MUST
   include in each warning-value a warn-date that matches the
   Date header field in the message.

   If a system receives a message with a warning-value that includes a
   warn-date, and that warn-date is different from the Date
   value in the response, then that warning-value MUST be deleted from the
   message before storing, forwarding, or using it. (preventing the
   consequences of naive caching of Warning header fields.) If all of the
   warning-values are deleted for this reason, the Warning header field MUST
   be deleted as well.
--->8---

NEW:
---8<---
   RFC2616 made the Warning header field's warn-date component optional; it 
   was only required to be sent when the recipient's version was HTTP/1.0 or lower.

   However, deployment experience has shown that many (if not most) intermediaries
   do not process the Warning header as required by RFC2616. This results in 
   situations where the field can appear in messages where it is not applicable, because
   a warning-value has not been removed by an intermediary.

   As a result, this specification shifts responsibility for processing of Warning from 
   intermediaries to the recipient that is actually consuming them.

   Generators of Warning header fields MUST include in every warning-value
   a warn-date that matches the Date header field in the message. Recipients that
   process a Warning header field MUST ignore (and MAY remove before forwarding)
   a warning-value whose warn-date is different from the Date value in the response. 
--->8---

We'd also remove the brackets in the ABNF that denote that warn-date is optional.

Comments?

--
Mark Nottingham   http://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 2 Sep 2013 06:23:05 +0200

Hi Mark,

> We'd also remove the brackets in the ABNF that denote that warn-date is optional.

Seems good, it follows the same principle we have in the whole draft which
is that the first who is really able to clean up the mess does it, so this
will increase overall compliance of deployed solutions.

Regards,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Mon, 02 Sep 2013 15:04:26 +0200

On 2013-09-02 02:47, Mark Nottingham wrote:
> ...

Indeed; I have to apologize for not paying attention before.

I have now tried to understand what's going on, and I'm confused.

1) "Warning" is not defined in HTTP/1.0 (right?). At least I don't see 
it in <http://tools.ietf.org/html/rfc1945>.

2) RFC 2068 does define Warning, but doesn't mention this issue at all: 
<http://tools.ietf.org/html/rfc2068#section-14.45>.

3) RFC 2616 has in 
<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.14.46.p.19>:

"If an implementation sends a message with one or more Warning headers 
whose version is HTTP/1.0 or lower, then the sender MUST include in each 
warning-value a warn-date that matches the date in the response."

HTTPbis changed this to:

"If an implementation sends a message with one or more Warning header 
fields to a receiver whose version is HTTP/1.0 or lower, then the sender 
MUST include in each warning-value a warn-date that matches the Date 
header field in the message."

Was that an intentional change at all?

And what are we trying to achieve here?

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 3 Sep 2013 10:37:52 +1000

On 02/09/2013, at 11:04 PM, Julian Reschke <[email-masked]> wrote:

> 3) RFC 2616 has in <http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.14.46.p.19>:
> 
> "If an implementation sends a message with one or more Warning headers whose version is HTTP/1.0 or lower, then the sender MUST include in each warning-value a warn-date that matches the date in the response."
> 
> HTTPbis changed this to:
> 
> "If an implementation sends a message with one or more Warning header fields to a receiver whose version is HTTP/1.0 or lower, then the sender MUST include in each warning-value a warn-date that matches the Date header field in the message."
> 

Yes; Warning headers don't have a version, so clearly the intent was for the condition to be upon the recipient.

> And what are we trying to achieve here?

Aligning the spec with reality. Currently, the spec places an requirement upon intermediaries to parse and sometimes remove Warning vales based upon their date (also parsing Date), whether or not they implement a cache. This places an unreasonable burden upon them, and doesn't align well with their motivation -- which has led to this requirement not being implemented widely (or at all?). 

It's much more reasonable to place the burden upon the party that's actually consuming the Warning. Given the state of implementation, that's what they'd have to do anyway.

Cheers,

---

From: Julian Reschke <[email-masked]>
Date: Tue, 03 Sep 2013 08:12:06 +0200

> Yes; Warning headers don't have a version, so clearly the intent was for the condition to be upon the recipient.

But the message they appear in has. I'm not a caching expert, but while 
reviewing the history I got the expression that we are trying to "fix" 
something that we broken ourselves. It would be good to understand what 
the intent of the text in 2616 was.

> It's much more reasonable to place the burden upon the party that's actually consuming the Warning. Given the state of implementation, that's what they'd have to do anyway.

Yes, but that's a problem of the text in httpbis, not in 2616, right?

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 3 Sep 2013 16:14:55 +1000

On 03/09/2013, at 4:12 PM, Julian Reschke <[email-masked]> wrote:

> But the message they appear in has. I'm not a caching expert, but while reviewing the history I got the expression that we are trying to "fix" something that we broken ourselves. It would be good to understand what the intent of the text in 2616 was.

2616 defines what it means to send HTTP/1.1 messages, not 1.0. Besides which, what the sending implementation / message version is has nothing to do with the intent of including the date, which is to make sure that *intervening* 1.0 caches don't cause the Warning to be persisted longer than it's valid for.

Cheers,

---

From: Julian Reschke <[email-masked]>
Date: Tue, 03 Sep 2013 08:46:03 +0200

> 2616 defines what it means to send HTTP/1.1 messages, not 1.0. Besides which, what the sending implementation / message version is has nothing to do with the intent of including the date, which is…

Yes. But I'm still trying to understand the history.

RFC 2616 was the first spec to define "Warning" -- what is the backwards 
compat story here? After all, an 1.0 intermediary will not even know 
what "Warning" is.

What am I missing here?

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 3 Sep 2013 17:23:44 +1000

On 03/09/2013, at 4:46 PM, Julian Reschke <[email-masked]> wrote:

> RFC 2616 was the first spec to define "Warning" -- what is the backwards compat story here? After all, an 1.0 intermediary will not even know what "Warning" is.
> 
> What am I missing here?

The actions of a HTTP/1.0 cache can affect the validity of a warning. For example, if a response has the warning "This response is stale" and a HTTP/1.0 cache revalidates it, the response is no longer stale, but the 1.0 cache will not have removed the warning. 

Warn-date is a mechanism to detect this situation; if warn-date is different to Date, we know that the warning is no longer valid.

However, 2616 places the requirement for detecting it upon *every* sender, including intermediaries, whether or not they cache, whether or not they do anything with Warning. This is pretty clearly Bad Protocol Design, because of the aforementioned misalignment of incentives.

Cheers,

---

From: Julian Reschke <[email-masked]>
Date: Tue, 03 Sep 2013 09:40:59 +0200

> Warn-date is a mechanism to detect this situation; if warn-date is different to Date, we know that the warning is no longer valid.

But it seems we lost the description of that mechanism; in 
<http://greenbytes.de/tech/webdav/rfc2616.html#warnings> I see:

"HTTP/1.0 caches will cache all Warnings in responses, without deleting 
the ones in the first category. Warnings in responses that are passed to 
HTTP/1.0 caches carry an extra warning-date field, which prevents a 
future HTTP/1.1 recipient from believing an erroneously cached Warning."

Maybe it would be good to re-add something like that because it explains 
what the date actually is for.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 3 Sep 2013 18:20:13 +1000

On 03/09/2013, at 5:40 PM, Julian Reschke <[email-masked]> wrote:

> But it seems we lost the description of that mechanism; in <http://greenbytes.de/tech/webdav/rfc2616.html#warnings> I see:
> 
> "HTTP/1.0 caches will cache all Warnings in responses, without deleting the ones in the first category. Warnings in responses that are passed to HTTP/1.0 caches carry an extra warning-date field, which prevents a future HTTP/1.1 recipient from believing an erroneously cached Warning."
> 
> Maybe it would be good to re-add something like that because it explains what the date actually is for.

Perhaps. Do you still have an issue with the proposed change?

---

From: Julian Reschke <[email-masked]>
Date: Tue, 03 Sep 2013 10:49:43 +0200

> Perhaps. Do you still have an issue with the proposed change?

I agree with the intent; I'd restore the RFC 2616 text that explains the 
purpose of the date, and move the description of the change into the 
"Changes from RFC 2616" appendix.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Tue, 03 Sep 2013 12:08:51 +0200

...proposed change 
(<trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/486/486.diff>):

      Warning       = 1#warning-value

      warning-value = warn-code SP warn-agent SP warn-text
                                              SP warn-date

      warn-code  = 3DIGIT
      warn-agent = ( uri-host [ ":" port ] ) / pseudonym
                      ; the name or pseudonym of the server adding
                      ; the Warning header field, for use in debugging
      warn-text  = quoted-string
      warn-date  = DQUOTE HTTP-date DQUOTE

...

    o  1xx Warnings describe the freshness or validation status of the
       response, and so MUST be deleted by a cache after validation.
       They can only be generated by a cache when validating a cached
       entry, and MUST NOT be generated in any other situation.

    o  2xx Warnings describe some aspect of the representation that is
       not rectified by a validation (for example, a lossy compression of
       the representation) and MUST NOT be deleted by a cache after
       validation, unless a full response is sent, in which case they
       MUST be.

    HTTP/1.0 caches will cache all Warnings in responses, without
    deleting the ones in the first category.  Thus, Warnings in responses
    carry a warning-date field, which can help in detecting an
    erroneously cached Warning.

    RFC 2616 made the Warning header field's warn-date component
    optional; it was only required to be sent when the recipient's
    version was HTTP/1.0 or lower.  However, deployment experience has
    shown that many intermediaries do not process the Warning header
    field as required by RFC 2616.  This results in situations where the
    field can appear in messages where it is not applicable, because a
    warning-value has not been removed by an intermediary.

    As a result, this specification shifts responsibility for processing
    of Warning from intermediaries to the recipient that is actually
    consuming them.

    Generators of Warning header fields MUST include in every warning-
    value a warn-date that matches the Date header field in the message.
    Recipients that process a Warning header field MUST ignore (and MAY
    remove before forwarding) a warning-value whose warn-date is
    different from the Date value in the response.

    The following warn-codes are defined by this specification, each with
    a recommended warn-text in English, and a description of its meaning.

... and in the appendix:

    Some requirements regarding production of the Warning header fields
    have been relaxed, as it is not widely implemented.  Furthermore,
    presence of the warn-date component has been made required (dropping
    requirements specific to HTTP/1.0).  Finally, the Warning header
    field no longer uses RFC 2047 encoding, nor allows multiple
    languages, as these aspects were not implemented.  (Section 7.5)

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 4 Sep 2013 14:07:53 +1000

Looks good to me.

On 03/09/2013, at 8:08 PM, Julian Reschke <[email-masked]> wrote:

> ...proposed change (<trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/486/486.diff>):
>

---

From: Julian Reschke <[email-masked]>
Date: Wed, 04 Sep 2013 08:24:32 +0200

> Looks good to me.

Thanks.

-> <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/2368>

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 6 Sep 2013 17:40:38 -0700

> -> <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/2368>

Sorry for missing the boat here, but this changes the warn-date
to be a required field.  As mentioned previously, no implementation
that I am aware of actually sends a warn-date, let alone processes
one, so now we have changed a never-implemented requirement into
an everyone-who-implements-is-noncompliant requirement.

How does that help?

I thought what I heard from the last meeting was that we would
remove all requirements on warn-date except for those recipients
who actually process warning header fields.

....Roy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 6 Sep 2013 17:58:05 -0700

> who actually process warning header fields.

and I should add...

The 2616 requirements (and motivation) for warn-date are
inherently bogus -- they assume that a Date header field
will be originated by a non-conformant cache, such that a
cached warning value will have a different date than a
non-cached warning value.  I have no idea why that assumption
was made, given that Date is defined as the origin's date of
message creation, is supposed to be cached when a response
is cached, and required to not be originated at each hop.
Even non-conformant caches tend to implement that correctly,
aside from the ones that don't save any header fields.

To reiterate again, NOBODY implements the requirements on
warn-date.  Even if it is implemented as specified, the end
result is the same -- warnings are always ignored by recipients.
This feature should be purged.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 9 Sep 2013 11:50:07 +1000

On 07/09/2013, at 10:40 AM, Roy T. Fielding <[email-masked]> wrote:

> to be a required field.

If you're generating a Warning header field, yes.

> As mentioned previously, no implementation

Actually, Apache httpd's mod_proxy_http appears to; see clean_warning_headers() and ap_proxy_clean_warnings(). 

> who actually process warning header fields.

I think we said that we'd remove requirements on those parties that don't do anything specific to Warnings.

> aside from the ones that don't save any header fields.

Revalidation updates Date, because it's required to be sent in a 304. So, you can detect whether revalidation has occurred if the warn-date and Date are different.

The point of warn-date is to detect cache revalidation, not just caching.

(I note that in RFC2616 the definition of 304 Not Modified explicitly requires Date, whereas p4 omits it, relying on the requirements for Date in p2. I'm a bit concerned that'll confuse implementers, since the list in p4 appears to be definitive.)

> warn-date.  
> Even if it is implemented as specified, the end

Right now, Apache *is* generating a lot of Warnings that will be incorrect if they pass through a cache, because most caches don't strip based upon warn-date (ironically, the one that I found is Apache, and it isn't including a warn-date, so that feature can't function). 

It seems to me that warn-date processing by the consuming implementation is necessary for the proper operation of the protocol, both by the spec (since 1.0 didn't include processing for Warning) and in reality (since most 1.1 intermediary / cache implementations are not stripping Warning). 

If we follow your proposal and remove requirements around warn-date, someone can generate a Warning that persists in cache beyond a revalidation, and that condition won't be able to be detected. Apache is doing that right now by generating Warning fields without warn-date.

I think we need to require warn-date in those Warnings that are affected by cache revalidation; specifically, the 1xx series. 

What if we constrained the requirement to include a warn-date to just those warnings?

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 9 Sep 2013 12:28:17 +1000

BTW, the background / motivation is pretty clearly spelled out here:
  http://tools.ietf.org/html/draft-ietf-http-warning-00

---

From: "Roy T. Fielding" <[email-masked]>
Date: Mon, 9 Sep 2013 03:22:30 -0700

> Actually, Apache httpd's mod_proxy_http appears to; see clean_warning_headers() and ap_proxy_clean_warnings(). 

Yes, and I would like to delete that code (or at least the bits
that are not needed).

> I think we said that we'd remove requirements on those parties that don't do anything specific to Warnings.

Is there a distinction that I am missing?

> Revalidation updates Date, because it's required to be sent in a 304. So, you can detect whether revalidation has occurred if the warn-date and Date are different.

Why would revalidation occur (with updated Date) without the
1xx Warnings being deleted?

> The point of warn-date is to detect cache revalidation, not just caching.

Okay.

> (I note that in RFC2616 the definition of 304 Not Modified explicitly requires Date, whereas p4 omits it, relying on the requirements for Date in p2. I'm a bit concerned that'll confuse implementer…

We can add that back in if you think it's needed.

> Right now, Apache *is* generating a lot of Warnings that will be incorrect if they pass through a cache, because most caches don't strip based upon warn-date (ironically, the one that I found is Ap…

I am pretty sure that Apache doesn't care, other than the desire
to be conformant with the standard.

> It seems to me that warn-date processing by the consuming implementation is necessary for the proper operation of the protocol, both by the spec (since 1.0 didn't include processing for Warning) an…

I don't see how.  Warning isn't necessary for the proper operation
of the protocol, which is why clients ignore them.

> If we follow your proposal and remove requirements around warn-date, someone can generate a Warning that persists in cache beyond a revalidation, and that condition won't be able to be detected. Ap…

Apache also generates some Warning header fields with bad syntax,
probably because there are no examples in the spec.  Fixing that
is way down there on my to-do list.

We can keep the requirement that, when a warn-date exists AND
a recipient is going to make some use of the Warning, THEN it
has to be processed in some way.

We cannot add a requirement that senders of Warning must send
the warn-date, which is what the ABNF change does.  Aside from
being a fantasy, it is incompatible with the grammar in 2068,
and that is just as forbidden for us as it was for 2616.

> I think we need to require warn-date in those Warnings that are affected by cache revalidation; specifically, the 1xx series. 

Why?  I don't care how many ways a Warning might become stale;
nobody sees the Warnings, and even if they did we can't trust a
refreshed cache entry to actually be "fresh" in any reasonable
sense if the same cache didn't even bother to implement 1xx
Warning deletion; it is "safer", in that case, to assume the old
Warning is true until that cache kicks the bucket, since these
are all Warnings about how the content *might* be stale.

However, that notion of "safe" is hardly relevant given the
client isn't going to say anything to a user about the Warning.
Do you know of any client code (UA or cache) that changes
behavior based on a received Warning?

> What if we constrained the requirement to include a warn-date to just those warnings?

Still incompatible.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 9 Sep 2013 20:34:27 +1000

> Yes, and I would like to delete that code (or at least the bits

I thought you might...

> Is there a distinction that I am missing?

You said "recipients", I didn't.

> 1xx Warnings being deleted?

Because revalidation doesn't delete headers; it only replaces them.

Consider a HTTP/1.0 cache that has stored:

HTTP/1.1 200 OK
Date: [one hour ago]
Content-Type: foo/bar
Warning: 123 Whatever
X-Thing: 1

When the 304 comes in, it gets:

HTTP/1.1 304 Not Modified
Date: [now]
Content-Type: foo/bar
Other-Thing: 2

which makes the updated cache entry:

HTTP/1.1 200 OK
Date: [now]
Content-Type: foo/var
Warning: 123 Whatever
X-Thing: 1
Other-Thing: 2

<https://svn.tools.ietf.org/svn/wg/httpbis/draft-ietf-httpbis/latest/p6-cache.html#freshening.responses> says that certain Warning headers are to be deleted by caches when this happens, but 1.0 caches don't know about it (and as we've seen, very few 1.1 caches implement it).

> is way down there on my to-do list.

Nod

> has to be processed in some way.

OK

>> What if we constrained the requirement to include a warn-date to just those warnings?

With...?

If we're going to keep the Warning feature, it should be interoperable. That includes making current implementations non-conformant, if it will get us to a place where interop is improved. That's why I want to make this requirement.

Failing that, we can warn generators that omitting warn-date from 1xx warnings can result in them being erroneously cached, because of both the poor state of implementation and 1.0 caches -- in prose. 

I think that's pretty weak, but if you insist...

---

From: Julian Reschke <[email-masked]>
Date: Mon, 09 Sep 2013 12:37:57 +0200

> and that is just as forbidden for us as it was for 2616.

Well, RFC 2068 has:

           Warning    = "Warning" ":" 1#warning-value

           warning-value = warn-code SP warn-agent SP warn-text
           warn-code  = 2DIGIT
           warn-agent = ( host [ ":" port ] ) | pseudonym
                           ; the name or pseudonym of the server adding
                           ; the Warning header, for use in debugging
           warn-text  = quoted-string

(<http://tools.ietf.org/html/rfc2068#section-14.45>) - so adding an 
optional date already was an incompatible change.

Best regards, Julian