Subject: Parsing the Injection-Info: header field

From: Julien ÉLIE <[email-masked]>
Date: Sat, 9 Jan 2010 00:21:42 +0100

Hi,

I follow a discussion that was initiated in news.software.nntp:

    http://groups.google.fr/group/news.software.nntp/browse_frm/thread/c704d0859dd84d38
    <news:[email-masked]>

According to Section 3.2.8 of RFC 5536:

   injection-info  =  "Injection-Info:" SP [CFWS] path-identity
                      [CFWS] *( ";" [CFWS] parameter ) [CFWS] CRLF

      NOTE: The syntax of <parameter> (Section 5.1 of [RFC2045], as
      amended by [RFC2231]), taken in conjunction with the folding rules
      of [RFC0822] (note: not [RFC2822] or [RFC5322]), effectively
      allows [CFWS] to occur on either side of the "=" inside a
      <parameter>.

I do not understand well that point.
Does it mean that CFWS is allowed both before and after the "="?
Why should RFC 822 be followed here?  And why doesn't RFC 5322 mention
that?

I thought that RFC 5536 was based upon RFC 5322 and therefore would
not allow a space before "=", and would allow a space after "=" only
if <value> is a quoted-string.

Just afterwards:

   <attribute>              format of <value>
   --------------------     -----------------
   "posting-host"           a <host-value>
   "posting-account"        any <value>
   "logging-data"           any <value>
   "mail-complaints-to"     an <address-list>

      NOTE: Since any such <host-value> or <address-list> also has to be
      a syntactically correct <value>, it will usually be necessary to
      encapsulate it as a <quoted-string>, for example:

       posting-host = "posting.example.com:192.0.2.1"

So it shows that a space is allowed before "=".
Isn't it a difference with RFC 5322 that ought to be mentioned in appendix C?

When it is said that <host-value> is encapsulated as a <quoted-string>, does
it mean that:

posting-host = "pos\ting.example.com"
posting-host =   (Comment)   "posting.example.com"  (Comment)

are two valid values for this parameter?  (Both being "posting.example.com".)

Incidentally, I do not know how to parse

posting-host="20bd:20bd::1"

It could be the IPv6 address "20bd:20bd::1" or the host name "20bd" associated
with the IPv6 address "20bd::1"!

-- 
Julien ÉLIE

« En vérité, le chemin importe peu, la volonté d'arriver suffit à tout. »

---

From: Antti-Juhani Kaijanaho <[email-masked]>
Date: Sat, 9 Jan 2010 09:31:19 +0200

> Does it mean that CFWS is allowed both before and after the "="?

I believe so, yes.

> Why should RFC 822 be followed here?

This is how I understand it:

Because RFC 2231 is what defines <parameter>, and RFC 2231 is written in the
context of RFC 822 ABNF, not RFC 5322 (or RFC 2822) ABNF.  Thus, the implicit
CFWS semantics of RFC 822 must be applied to the definition of <parameter>,
even though the rest of RFC 5536 follows RFC 5322 in not having implicit CFWS.

I suspect the same note applies to all other nonterminals imported from the
MIME specs (token, value, attribute).

This confusion is caused by that the MIME specs have not (yet?) been updated to
match RFC 5322.  The yam (Yet Another Mail) working group is, AFAICT, the place
where any decision to do that would be initially made.

> And why doesn't RFC 5322 mention that?

Why would it?

> Isn't it a difference with RFC 5322 that ought to be mentioned in appendix C?

What difference is that?  RFC 5322 does not define <parameter>.

-- 
Antti-Juhani Kaijanaho, Jyväskylä, Finland
http://antti-juhani.kaijanaho.fi/newblog/
http://www.flickr.com/photos/antti-juhani/

---

From: Julien ÉLIE <[email-masked]>
Date: Sat, 9 Jan 2010 11:22:45 +0100

Hi Antti-Juhani,

> even though the rest of RFC 5536 follows RFC 5322 in not having implicit CFWS.

Yet, RFC 5322 mentions in its introduction:

   This document specifies a syntax only for text messages.  In
   particular, it makes no provision for the transmission of images,
   audio, or other sorts of structured data in electronic mail messages.
   There are several extensions published, such as the MIME document
   series ([RFC2045], [RFC2046], [RFC2049]), which describe mechanisms
   for the transmission of such data through electronic mail, either by
   extending the syntax provided here or by structuring such messages to
   conform to this syntax.  Those mechanisms are outside of the scope of
   this specification.

I understand that RFC 2045 now extends the syntax of RFC 5322 (instead
of RFC 822).

RFC 2231 extends the syntax of RFC 2045 so it is normally currently
based on RFC 5322.

> What difference is that?  RFC 5322 does not define <parameter>.

Sure, but USEFOR is said to comply with RFC 5322 which specifies that RFC 2045
syntax is now based upon RFC 5322 (or that RFC 2045 is to structure messages
to conform to RFC 5322 -- which is basically the same for now unauthorized
folding rules).

> match RFC 5322.

It depends on whether a software is compliant to RFC 822 or not.
When a software complies with RFC 2822 or 5322, the new syntax is to be used
since MIME specs are said to extend these new RFCs instead of the obsoleted
RFC 822.

Is RFC 5322 to be read in another way when it speaks about RFC 2045?

-- 
Julien ÉLIE

« -- Essayons d'interroger ce garde habilement sans éveiller ses soupçons...
  -- Hep ! Où est enfermé Assurancetourix ? » (Astérix)

---

From: Antti-Juhani Kaijanaho <[email-masked]>
Date: Sat, 9 Jan 2010 14:22:10 +0200

> I understand that RFC 2045 now extends the syntax of RFC 5322 (instead

That interpretation would mean that suddenly CFWS beciame forbidden in MIME
constructs when RFC 2822 was published.  I don't think that was the intent,
especially when "those mechanisms are outside of the scope".

That paragraph is a bit unfortunately worded, though.

---

From: Julien ÉLIE <[email-masked]>
Date: Sun, 10 Jan 2010 16:10:03 +0100

Hi Antti-Juhani,

> especially when "those mechanisms are outside of the scope".

CFWS is not forbidden by RFC 2822/5322.  It just became obsolete syntax:

4. Obsolete Syntax

   Earlier versions of this specification allowed for different (usually
   more liberal) syntax than is allowed in this version.

   [...]

   Though these syntactic
   forms MUST NOT be generated according to the grammar in section 3,
   they MUST be accepted and parsed by a conformant receiver.

   [...]

   One important difference between the obsolete (interpreting) and the
   current (generating) syntax is that in structured header field bodies
   (i.e., between the colon and the CRLF of any structured header
   field), white space characters, including folding white space, and
   comments could be freely inserted between any syntactic tokens.  This
   allowed many complex forms that have proven difficult for some
   implementations to parse.

That's why I find weird that RFC 5536 still encourages an obsolete syntax
(with the posting-host example) that MUST NOT be generated.

I believe the interpretation of the fact that RFC 2045 now extends the
syntax of RFC 5322 is that:

 * for RFC-5322-compliant implementation, RFC 2045 MUST comply with the
   general syntax of RFC 5322;

 * for RFC-822-compliant implementation, RFC 2045 uses the general syntax
   described in RFC 822.

> That paragraph is a bit unfortunately worded, though.

I don't think so.
It is normal that an RFC-5322-compliant article does not use obsolete
syntax.  That's what RFC 5322 mentions:  if MIME is to be used in compliance
with RFC 5322, then the allowed syntax is restricted to the one of RFC 5322.

I'm trying to write to the IETF YAM group to ask, but I don't manage to
(the same way I still do not manage to write to the W3C URI group!).

-- 
Julien ÉLIE

« Rubor, tumor, dolor, calor et functio laesa. »

---

From: Antti-Juhani Kaijanaho <[email-masked]>
Date: Mon, 11 Jan 2010 08:54:07 +0200

> CFWS is not forbidden by RFC 2822/5322.  It just became obsolete syntax:

On RFC 822, and in specifications based on it, CFWS is *implicitly allowed* in
structured field bodies.  That means that in such specifications, CFWS was not
included in the grammar, but it still was allowed.  What 2822 did was, among
other things, reweite the syntax to make CFWS explicit.

Thus, if we are to believe, as you argue, that all 822-based specifications
(and specifically the MIME specifications) became 2822-based when 2822 was
accepted, then CFWS that was implicitly allowed under 822 rules became suddenly
forbidden (since no CFWS constructs or obs- productions magically appear in old
specs), not just obsolete.

For example, in RFC 2045:

     content := "Content-Type" ":" type "/" subtype
                *(";" parameter)

Under 822 rules, CFWS is implicitly allowed between all the symbols after ":".
Under 2822 rules, no CFWS is allowed here (even as obsolete syntax), since no
CFWS symbol, or an obs-content symbol, is included in this production (and,
equivalently, in the productions of "type", "subtype" and "parameter").

> 4. Obsolete Syntax

I know; it's also irrelevant here.

> syntax.

I don't have a problem with not using obsolete syntax.  My point is that
pre-2822 specs effectively used a different grammar formalism than post-2822
specs, and trying to say "just use 2822 rules" is doomed to fail.  The specs,
and their grammar, need a rewrite.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 11 Jan 2010 13:02:37 GMT

>I understand that RFC 2045 now extends the syntax of RFC 5322 (instead

How can that possibly be so, since it was written umpteen years earlier?
Anything that "extended RFC 5322 would have to be written _after_ that.

RFC 2045, and its relations, provides syntax for many new header fields.
>From reading RFC 2045 and RFC 822, you can deduce *exactly* where CFWS is
allowed. If RFC were to be re-written (a highly desirable thing BTW), then
if would redefine its syntax with explicite CFWS, presumably in exactly
the same places as now, unless it chose to declare some of them to be
obsolete as 5322 has done.

It is clear that, according to RFC 2045, CFWS is allowed on both sides of
the '=' in a <parameter> (and why shouldn't it?). And, just to make sure, I
checked my interpretation with Keith Moore before writing the original
version of the paragraph which you quoted.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131            Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Julien ÉLIE <[email-masked]>
Date: Mon, 11 Jan 2010 23:18:10 +0100

Hi Charles,

> Anything that "extended RFC 5322 would have to be written _after_ that.

A misunderstanding of mine.
I thought that if an article was to comply with RFC 5322, then the syntax
of extended fields (defined by MIME), was to be restricted to what RFC 5322
allows.  But that does not seem to be the case.

Thanks, Antti-Juhani and Charles, for having explained what should be
done for MIME.

> version of the paragraph which you quoted.

Yeah, no problem.  Sorry for the disturbance of my thread.

For what it is worth, I also had an answer from the YAM group meanwhile:

  http://www.ietf.org/mail-archive/web/yam/current/msg00279.html

    [Ned Freed]
    "RFCs 2045 and 2046 were written before DRUMS and RFCs
    2821/2822/5321/5322 and therefore used the syntactic convensions of RFC 822,
    where LWSP is implicitly allowed between tokens. The fact that DRUMS opted to
    make the places where LWSP is allowed explicit rather than implicit doesn't
    change the syntax rules for 2045 and 2046 in any way. shape or form.

    I also note that had RFC 5322 had any effect on MIME syntax, the header of the
    document would show that it updates RFCs 2045 and 2046. The document header
    says no such thing.

    P.S. When revised MIME specifications come out they will probably switch to the
    explicit LWSP approach since it seems that's the preferred way to do it now."

-- 
Julien ÉLIE

« Cela n'a rien de remarquable. Il suffit d'appuyer
  sur la bonne touche au bon moment et l'instrument
  joue tout seul. » (J.-S. Bach)