Subject: RE: NEW ISSUE: 13.1.2's Definition of 1xx Warn-Codes

From: "Joris Dobbelsteen" <[email-masked]>
Date: Wed, 27 Dec 2006 11:11:16 +0100

>-----Original Message-----
>From: [email-masked] 
>[mailto:[email-masked]] On Behalf Of Travis Snoozy (Volt)
>Sent: vrijdag 22 december 2006 22:55
>To: [email-masked]
>Subject: NEW ISSUE: 13.1.2's Definition of 1xx Warn-Codes
>

Assumption: RFC2616...

>(Supporting references at the end of the message)
>
>The offending part of section 13.1.2 (Warnings, page 77) reads:
>
>   [...]
>
>   1xx  Warnings that describe the freshness or revalidation status of
>     the response, and so MUST be deleted after a successful
>     revalidation. 1XX [sic] warn-codes MAY be generated by a 
>cache only
>     when validating a cached entry. It MUST NOT be generated 
>by clients.
>
>   [...]
>
>
>The problems, in order from simplest to the most complex:
>
>1. 1XX should be 1xx (or vice-versa) everywhere.
>
>Proposed fix: pick one, and use it everywhere. I'm partial to 
>1xx myself :).
>

Valid. Slight typo, should not impair understanding of the spec.

>>2. The use of MAY in the offending part of 13.1.2 conflicts 
>with the MUST
>   requirements in section 14.46 and the definition of MAY in BCP14.
>

Invalid.
Your interpretation is clearly wrong, read it in another way. It is
consistent. I will comment on the fix.

>Proposed fix: "1xx warn-codes MUST NOT be added to any 
>messages except cache entries

Invalid:
Warnings, including 1xx warn-codes, have multiple purposes and are not
limited to HTTP caches only (see section 14.46).
Furthermore its perfectly legal for a server to include a 1xx warn-code
on, perhaps my database refuses and I used the cached information
instead. A warning could be appropiate in some cases.

>, and MUST NOT be added to cache 
>entries except in response to a validation attempt." (As a 
>side note, a definition of a cache entry would be
>nice.)

Valid, but the spec already says this (in the positive sence).

But your proposed fix forgets that 1xx warn-codes must be deleted on
successful validation.
(Side note: Tricky is that section 14.46 does mention that warnings from
the validated response must be used instead. So remove the old 1xx
warnings, as these do not apply any more and add the new warnings
instead. Of course these new warnings may include 1xx warn-codes again,
but for this case.)

I do not suggest any corrections to this paragraph for this reason.

>3. One would think that proxies could include caches (though I 
>have yet to
>   find where this is permitted with a true BCP14 MAY). 

Read the defitions:
Client is a program.
Server is a program.
Proxy is a program.
Cache is a program's local store and ....
Thus a cache may be a proxies local store and ....

Funny you say it correctly below.

>However, the wording
>   in the offending part of 13.1.2 makes it impossible to satisfy the
>   requirements of a cache and the requirements of a client (and, by
>   extension, proxy) simultaneously. A cache is not an 
>independent program;
>   it is part of a program (as per the 1.3 definition). A client is a
>   program, and it can contain a cache (again, from 1.3), but 
>this limits
>   the cache's behavior to the set intersection of allowed 
>behaviors for
>   caches and clients (due to how "client" is defined). This leads to a
>   conflict where the cache MUST generate a 1xx code, but a 

...cache "MAY" generate..., not MUST.
You mentioned that in the above items.

>client MUST NOT
>   generate a 1xx code. Thus, we're left having to conclude 
>that caches can
>   exist only as part of independent servers (which have their content
>   pushed to them, or delivered through some out-of-band method).

Thus, invalid.
Your claim fails on wrong copying of the MUST and MAY clauses in the
spec.

Only a slight typo correction might be in place.

- Joris Dobbelsteen

---

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Wed, 27 Dec 2006 13:46:26 -0800

I apologize in advance for the long-windedness of this letter; you might
want to get a cup of tea before you start reading.

> Assumption: RFC2616...

Correct. Sorry for not being explicit.

> >     cache only when validating a cached entry. It MUST NOT be generated
> >     by clients.
> >
> >   [...]
> >
> >

<snip>

> >with the MUST requirements in section 14.46 and the definition of MAY in
> >BCP14.
> >
>
> Invalid.

Then my point is proven: if I can interpret it wrong, it needs to be
rewritten ;) (see also Murphy's Law).

But seriously, according to BCP14: MAY "mean[s] that an item is TRULY
OPTIONAL" (emphasis mine). Clearly, it is not truly optional for 1xx warn-
codes to "be generated by a cache only when validating a cached entry."
Certain 1xx warn-codes MUST be generated by any conformant/conditionally
conformant cache (section 14.46, page 149). May is being used correctly in
the English sense, but not in the BCP14 MAY sense.

Now, as to whether my interpretation is right or wrong, *that* is arguable.
I have some examples (later on) that show the completely literal
interpretations of the spec. I did _not_ use the literal interpretation of
the original text, since it seemed very unlikely that that was the actual
intent. However, the literal interpretation *does not* have any conflict
with section 14.46. In any case, I ask that you tell me how -you- interpret
this section; telling me to "read it another way" is totally unhelpful.

> instead. A warning could be appropiate in some cases.

1xx codes are almost exclusively cache-related, so I'd say it probably won't
matter in practice (only 110 and 199 make sense for anything but a cache to
generate, and even 110 is iffy, and the warn-code could easily be generated
by the origin server's "caching" subsystem). My new proposal (later) should
sidestep this issue, so it shouldn't really matter either way -- but it's
good that you thought about it (I don't think I did).

> >, and MUST NOT be added to cache entries except in response to a
> >validation attempt." (As a side note, a definition of a cache entry would
> > be nice.)
>

13.1.2 does not actually say this in a positive sense (though -other-
portions of the spec might). What it says is something completely different,
and (I'm guessing) unintentional.

The technical reason that MUST NOT (or a well-crafted MUST) needs to be used
has to do with "may" in English, versus MAY in BCP14. BCP14 effectively
requires authors to emulate the English meaning of "may" in the form of a
MUST or MUST NOT. Following are several examples, starting from the original
wording and moving to my proposed wording.

BAD (original)

"1xx warn-codes MAY be generated by a cache only when validating a cached
entry."

Stated positively, but English and BCP14 conspire to contort the meaning.
This literally means: "A cache can choose to perform 1xx warning generation
for any message [though the cache entry being validated is implied] when
validating a cache entry, and at no other time. A cache can also choose to
generate 1xx warnings for any entry at any other time -- it's totally up to
the implementer how they want to do it." I think that it is *highly
unlikely* that the spec actually means this (but please, speak up if
you think the spec means what it says, here).

BAD

"1xx warn-codes may be generated by a cache only when validating a cached
entry."

Stated positively, but with no requirements (which may, in fact, be all
right). This literally means: "If a cache is validating a cache entry, then
the cache can generate 1xx warnings for any message [but the cache entry
being validated is implied]. This is informative, and any normative sections
that conflict with this statement override."

BAD

"1xx warn-codes MUST be generated by a cache only when validating a cached
entry."

Stated positively, but English and BCP14 are still messing things up.
Literally means: "When validating a cached entry, the cache has to generate
1xx warn-codes for some message or another [though the entry being validated
is implied] -- if it doesn't, it's in violation of the spec."

BAD

"1xx warn-codes MUST only be generated by a cache when validating a cached
entry."

Stated positively, but still screwy. Literally: "Nothing but caches are
allowed to generate 1xx warn-codes, and the caches are allowed to generate
1xx warn-codes for any message if and only if the cache is validating a
cached entry [but the cache entry being validated is the implied target for
the generated warn-code]. Breaking these rules means you're in violation of the spec."

BETTER

"Caches MUST ensure that they generate 1xx warn-codes for cached entries
only, and that 1xx warn-codes are generated for a cached entry during
validation of that entry only."

Stated positively, with the implicit target of the warn-code made explicit.
However, I prefer to use "MUST NOT ... except" instead of "MUST ... only",
because even (native) English-speakers have difficulty using and
understanding the word "only" in a correct, unambiguous way. Since being
correct and unambiguous is especially important in a spec, avoiding "only"
is probably a Good Thing.

BETTER (new proposed)

"A cache MUST NOT generate 1xx warn-codes for any messages except cache
entries, and MUST NOT add 1xx warn-codes to cache entries except in response
to a validation attempt."

Stated negatively, with the warn-code target made explicit. Correct,
unambiguous, and easy to understand. This is my suggestion, as revised to
allow servers to generate warn-codes (by virtue of specifying only cache
behavior).

> But your proposed fix forgets that 1xx warn-codes must be deleted on

Sorry, I wasn't totally clear about my intent: I'm not replacing the
*entire* paragraph, just the last two sentences. There are no issues with
the first sentence. My mistake.

> instead. Of course these new warnings may include 1xx warn-codes again,

I don't think I understand what your point is, here.

> I do not suggest any corrections to this paragraph for this reason.

Well, I still do; hopefully I've convinced you that what this section is
actually saying is probably not what is really meant.

> >have yet to find where this is permitted with a true BCP14 MAY).
>
> Read the defitions:

Yes, lets!

> Client is a program.
Informative.

> Server is a program.
Informative.

> Proxy is a program.
Informative.

> Cache is a program's local store and ....
Informative.

> Thus a cache may be a proxies local store and ....
Informative.

You left out the *only* normative bit in the definitions (which,
consequently, is the one that makes it so that proxies can't have caches):

* Proxies MUST conform to both server and client requirements
Normative (!)

> Funny you say it correctly below.

It's not funny, it's intentional ;). Proxies MUST conform to both server and
client requirements. Clients MUST NOT generate 1xx warn-codes. Caches MUST
generate 110, 111, and 113 warn-codes in certain situations. Thus, a client
MUST NOT have a cache (because if it did, the client would generate warn-
codes in the cache, violating the spec), and by extension a proxy MUST NOT
have a cache (because it would be violating a client rule if it did). This
is clearly not what the spec writers intended, but it's what they wrote
nonetheless.

> >However, the wording in the offending part of 13.1.2 makes it impossible
> >to satisfy the requirements of a cache and the requirements of a client
> >(and, by extension, proxy) simultaneously. A cache is not an independent
> >program; it is part of a program (as per the 1.3 definition). A client is
> >a program, and it can contain a cache (again, from 1.3), but this limits
> >the cache's behavior to the set intersection of allowed behaviors for

Please re-read the references I included in my original message. Section
14.46 page 149, definitions of warn-codes 110, 111, and 113 clearly state
that a cache MUST generate these warn codes under certain conditions. That's
the whole point, here; the MAY in section 13.1.2 can not be correct, and
needs to be removed. The cache MUST generate specific 1xx warn codes, but
only under certain conditions; English "may" means this, but BCP14 MAY does
NOT.

> >client MUST NOT generate a 1xx code. Thus, we're left having to conclude
> >that caches can exist only as part of independent servers (which have
> >their content pushed to them, or delivered through some out-of-band
> >method).
>
> Thus, invalid.

That, too is the point. The conclusion drawn from the conformance points in the spec leads to utter absurdity. Thus, one of the following must be true:

1. I botched my logic somewhere.
2. The spec really intends this silly conclusion.
3. There's an error in the spec.

I argue that (3) is the case. I think that we can both agree that (2) is
very, very unlikely. I also think that (1) will be very hard to show,
because the spec is quite clearly contradicting itself -- but if you can
point me to a clause that breaks my logic (or show that I've made an error
in my logic, or used a false premise), I'll be happy to re-evaluate my
position.

> Your claim fails on wrong copying of the MUST and MAY clauses in the

Are you asking me to copy, verbatim, the entire sections I'm referring to? I
have not taken anything out of context that I am aware of, and you are free
to double-check with the official RFC if you wish (I have included sections
and page numbers for just that purpose). If you think I have taken something
out of context ("wrong copying"), please be more explicit as to the
sections, pages, and clauses that you feel I have wrongfully omitted.
However, I am confident that even if I were to use the complete text of the
sections I referred to, my argument would still hold (since I'm working on a
heavily-annotated hard-copy of the spec, it's hard to take things out of
context).

> Only a slight typo correction might be in place.

You mean MAY -> may, right? ;)

Thanks for your analysis,

-- Travis

---

From: "Larry Masinter" <[email-masked]>
Date: Wed, 27 Dec 2006 23:12:42 -0800

>     cache only when validating a cached entry. It MUST NOT be generated

What does the last sentence mean, anyway? A '1xx warn-code'
isn't a request header. So how would a Client 'generate' one
anyway?

Larry
-- 
http://larry.masinter.net

---

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Thu, 28 Dec 2006 09:15:20 -0800

> isn't a request header. So how would a Client 'generate' one

I interpret "generating" a 1xx warn-code to mean either adding a new Warning header with that warn code, or appending the warn-code to the last Warning header in a given message. I don't know if that's the intent or not (there's no explicit definition of "generate"), but I think it's a reasonable guess.

-- Travis

---

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Thu, 28 Dec 2006 11:39:56 -0800

Travis Snoozy said:
> > >     cache only when validating a cached entry. It MUST NOT be
> > >     generated by clients.
> >
> I interpret "generating" a 1xx warn-code to mean either adding a new
> Warning header with that warn code, or appending the warn-code to the last
> Warning header in a given message. I don't know if that's the intent or
> not (there's no explicit definition of "generate"), but I think it's a
> reasonable guess.

*light bulb goes off*

Ah. Pardon my temporary blindness; I reread your E-mail, and caught your
meaning. While Warning is a general header, 1xx warn-codes only make sense
in *Responses* (since Requests aren't cached and can't be [re]validated).
THAT is a very sensible interpretation; the wording in the spec was just so
non-specific (and in poor context) that I didn't even think about that.

So, another revision...

Proposed fix:

"A cache MUST NOT generate 1xx warn-codes for any messages except cache
entries, and MUST NOT generate 1xx warn-codes for a cache entry except in
response to a validation attempt for that entry. 1xx warn-codes MUST NOT be
generated in Request messages."

This leaves everyone else to generate 1xx warn-codes at their leisure, so long as those warn-codes are added only to Responses and NOT Requests.

Thanks!

-- Travis

---

From: Henrik Nordstrom <[email-masked]>
Date: Fri, 29 Dec 2006 02:43:37 +0100

> instead. Of course these new warnings may include 1xx warn-codes again,

It's already covered elsewhere by a general blanket rule in 10.3.5 304
Not modified.

Regards
Henrik

---

From: Larry Masinter <[email-masked]>
Date: Thu, 28 Dec 2006 20:15:53 -0800

We're talking about

 1xx  Warnings that describe the freshness or revalidation status of
   the response, and so MUST be deleted after a successful
   revalidation. 1XX warn-codes MAY be generated by a cache only when
   validating a cached entry. It MUST NOT be generated by clients.

------------------------------------
First, a few guidelines, as I remember them:

* The roles of 'client', 'proxy' and 'server' were not intended to
be exhaustive; HTTP is used in many contexts with hybrid roles
and complex relationships. In some cases, it's necessary to
specify requirements specifically for clients vs. proxies,
or proxies vs. servers, but in general, it's better to couch
things in terms of request messages and their responses.

* In general, it's better to avoid MUST and MUST NOT rules except
for those cases where it is necessary to ensure interoperability
or operational stability. (There are a few additional cases,
but generally not.)  It's especially troublesome to add
"MUST" or "MUST NOT" requirements to HTTP at this time.
We do better when we describe how things work
and what protocol elements _mean_ and let implementors
use them as appropriate.

* More specifically, it's better to avoid specifying
implementation advice rather than observable network
behavior. (Again, some exceptions.)

The introduction of MUST/MAY/SHOULD keywords happened by
taking English text without them and introducing them
in an editing pass, but (in my opinion) there are too many
of them in the HTTP spec. It's better to avoid introducing
any more. 

-----------------
So, I can't see what the interoperability problems would
be, for allowing responses to contain warnings when
responses are stale generally. 

My proposed rewrite:

 1xx  Warnings that describe the freshness or revalidation status of
    a response. For example, a cache might include a warning
    if it is unable to revalidate the freshness of a cached value,
    but would not include a warning after a successful revalidation.
    These warnings only apply to responses, and SHOULD NOT
    appear in request messages.

I.e., avoid a too narrow (and unnecessary) MUST NOT,
or really specifying anything other than what the
warning means, and letting implementors decide
when to use them.

> "A cache MUST NOT generate 1xx warn-codes for any messages 
> except cache entries

Let's suppose I have a web site where I generate  pages
from a database, and I cache the database values but
not the HTML  pages. If the database cache is stale,
I might still want to include a 1xx warning, even though
the response didn't correspond to a 'cache entry'.
Why would you disallow this? 

> and MUST NOT generate 1xx warn-codes for a cache 
> entry except in
> response to a validation attempt for that entry. 

I don't understand what a 'validation attempt'
might be that isn't implicit in a 'request'.
Is there a reason why some stale results shouldn't
include warnings?                

Larry

---

From: "Travis Snoozy (Volt)" <[email-masked]>
Date: Fri, 29 Dec 2006 09:43:24 -0800

>    validating a cached entry. It MUST NOT be generated by clients.

Then the spec shouldn't require that proxies MUST implement both client and
server behaviors -- because this makes those words very normative-sounding.
In fact, I think it would be pretty hard to specify HTTP without these
delineations for a good chunk of the behaviors that are present.

> and what protocol elements _mean_ and let implementors

That's a can of worms for another day ;).

> behavior. (Again, some exceptions.)

No disagreement here.

> of them in the HTTP spec. It's better to avoid introducing

I don't think there are too many of them, so much as they are too often
misused and/or in the wrong places. Case in point...

[Encompassed by my response to your final paragraph]

> warning means, and letting implementors decide

[Encompassed by my response to your final paragraph]

> Why would you disallow this?

It's not disallowed; the keyword is "cache". What you're talking about is
not a protocol-level cache, and the server is free to add whatever warn-
codes it wants. This requirement applies only to the behavior of caches as
defined by the spec, and the spec clearly states that the caches it refers
to deal with storing and retrieving "response messages". Granted, we could
probably make things more explicit (and I'm all for that), but this is not
as prohibitive as you make it out to be.

> > and MUST NOT generate 1xx warn-codes for a cache entry except in

I sure don't know, either, but the original author thought that that wording
was important for one reason or another. Unfortunately, there is no
explanatory part for this set of requirements, so I can't tell you what the
author was thinking, or why this particular phrasing is (or at least was)
important. It could also be that I misinterpreted the original sentence that
my proposed phrasing is based on ("1XX warn-codes MAY be generated by a
cache only when validating a cached entry"). Since my goal was to make
little to no semantic change (insofar as the _actual intent_ of the spec,
not the literal interpretation), I just took that section and attempted to
make its meaning as explicit as possible; I haven't bothered to question the
semantics themselves, because that goes beyond a simple editorial change.

Thanks,

-- Travis

---

From: "Larry Masinter" <[email-masked]>
Date: Fri, 29 Dec 2006 16:02:37 -0800

I confess: I didn't look hard enough at the context of
the offending paragraph, and missed the fact that the
text  is in the section on "Caching in HTTP" which
provides ample context for narrowing the scope...

As for the original authors, check out 
http://www.hpl.hp.com/personal/ange/archives/http-wg-archive/0763.html 

and

http://www.hpl.hp.com/personal/ange/archives/http-wg-archive/1088.html

Larry