Subject: Question: Folding and FWS in Path

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 16 Sep 2005 08:24:45 +0200

Question for the WG:

current syntax of PATH has FWS (folding white space) all over the place.

I recall discussion of other fields where folding was strongly discouraged 
- Newsgroups: was the important one.

1) Is there a group consensus to allow folding of Path?
2) Is there a group consensus to allow whitespace in Path?
3) If yes: Are these features in status "use them now", or are they in 
status "MUST accept, SHOULD NOT generate"?

                  Harald

---

From: Richard Clayton <[email-masked]>
Date: Fri, 16 Sep 2005 09:42:40 +0100

In message <153526FD4DE53138862E87A0@B50854F0A9192E8EC6CDA126>, Harald
Tveit Alvestrand <[email-masked]> writes
>1) Is there a group consensus to allow folding of Path?

it looks very dangerous to me -- it will stop Path working at all for
any system that does not unfold before consulting it

that said, all that goes wrong is that peers are offered extra articles
which they then refuse based on the message-ID, so it's not the end of
the world, just a small amount of extra traffic -- recall that paths
across Usenet are often of very similar (very fast speed) so multiple
peers often offer you the same article. Path: is there for a slower
gentler Usenet where if you have an article, the chances are that none
of your peers does, but if they do have it, then it's in a long queue
heading your way and you'd do them a favour by not overloading them
further by offering it to them

of course if systems have stupidly short history files then Path: helps
considerably!  I noted the recent discussion of a week or so, and recall
that a decade ago, UUNet used to run with 90 days plus...

>2) Is there a group consensus to allow whitespace in Path?

it isn't there at the moment...  but if there's this huge encouragement
for adding diagnostics (which I suspect will be mainly ignored, the
place for a MISMATCH is in the receiving machine's syslog, not on the
propagated article) then the Path header lines are going to be rather
longer than at present...

why did they get added ?

>status "MUST accept, SHOULD NOT generate"?

If they must be there, then I think that would be best

- -- 
richard                                              Richard Clayton

They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety.         Benjamin Franklin

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 16 Sep 2005 11:47:38 +0200

> why did they get added ?

I don't know .... they were in draft-ietf-usefor-article-00, in the 
"new-path" format ("old-path" syntax also permitted spaces, but only 
spaces).
(draft fetched from 
<http://bgp.potaroo.net/ietf/all-ids/draft-ietf-usefor-article-00.txt> - 
the draft was not dated, but had an expiry date of November 1998, so we can 
assume that it was from May 1998).

Son-of-1036 (1994) said:

     NOTE: This syntax has the disadvantage of contain-
     ing  no  white space, making it impossible to con-
     tinue a Path header across several lines.   Imple-
     mentors  of relayers and reading agents are warned
     that it is intended that  the  successor  to  this
     Draft will change the definition of path delimiter
     to:

          path-delimiter = "!" [ space ]

     and are urged to  fix  their  software  to  handle
     (i.e.,  ignore) white space following the exclama-
     tion points.  They are urged to hurry;  some  ill-
     behaved  systems  reportedly  already feel free to
     add such white space.

By February 2000 (article-03), this had settled down already to the present 
quite folding-permissive format:

5.6.1.  Format

      Path-content        = *( path-identity [FWS] delimiter [FWS] )
                     tail-entry *FWS
      path-identity       = 1*( ALPHA / DIGIT / "-" / "." / ":" / "_" )
      delimiter           = "/" / "?" / "%" / "," / "!"
      tail-entry          = 1*( ALPHA / DIGIT / "-" / "." / ":" / "_" )

This is from Forrest's very useful site:
<http://mibsoftware.com/userkt/usefor/s-o-1036/Path.htm>

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 16 Sep 2005 13:13:52 +0200

> current syntax of PATH has FWS (folding white space)

Charles' versions limits it to "end of path-entry"
(= one or more identities plus optional diagnostic)

You had it before or after all "!" or "!!"

My last proposal had it before all "!" or "!!"
(because Russ doesn't like "after")

We all have the alibi-pseudo-2882-[FWS] after the
"colon plus magic SP", and another before CRLF.
The prose forbids both, should be *WSP.  <shrug>

> 1) Is there a group consensus to allow folding of Path?

Dunno.

> 2) Is there a group consensus to allow whitespace in Path?

1036 had "blank" as separator, s-o-1036 announced
that the grandson will allow FWS.  If we'd _forbid_
WSP it has to be listed as "difference from 1036".

> 3) If yes: Are these features in status "use them now",
>    or are they in status "MUST accept, SHOULD NOT generate"?

Server territory, same idea as for Control / Message-ID /
Newsgroups / Xref / Expires / Distribution / Lines.

In other words, as long as the WG has conxistent concepts
I'm confident to find ABNF reflecting whatever that is.

                           Bye, Frank

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 16 Sep 2005 08:26:12 -0400

Richard Clayton wrote:

> the world, just a small amount of extra traffic

Some (many?) peering servers keep only a limited message-ID cache, representing
several hours of message IDs, not days.  Am I mistaken?

In this case, path is not some diagnostic nor advisory like 2822 "Received."
It is the only way that traffic is manageable.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 16 Sep 2005 16:05:06 +0200

> several hours of message IDs, not days.  Am I mistaken?

I think the last discussion of the message-id cache ("history file") gave a 
conclusion that "if you do less than a week, you're either an idiot or you 
have special circumstances and know exactly what you're doing"......

---

From: Seth Breidbart <[email-masked]>
Date: Fri, 16 Sep 2005 12:24:32 -0400 (EDT)

> 1) Is there a group consensus to allow folding of Path?

I'm mildly opposed.

> 2) Is there a group consensus to allow whitespace in Path?

Likewise

> 3) If yes: Are these features in status "use them now", or are they in=20

The latter; I suspect there's too much existing software that will be
very unhappy with it (at least, won't check properly the whitespaced
entry, similar to what happened with Newsgroups).

Seth

---

From: Russ Allbery <[email-masked]>
Date: Fri, 16 Sep 2005 11:04:38 -0700

>> any system that does not unfold before consulting it that said, all
>> that goes wrong is that peers are offered extra articles which they
>> then refuse based on the message-ID, so it's not the end of the world,
>> just a small amount of extra traffic

> Some (many?) peering servers keep only a limited message-ID cache,
> representing several hours of message IDs, not days.  Am I mistaken?

They certainly shouldn't!  There's absolutely no guarantee that one's
duplicates will follow a common path, and such a server would accept
duplicate articles in practice unless it also imposed tight restrictions
on Date.

(Unless perhaps you're talking only about the fast in-memory cache as
opposed to the full history file?)

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 19 Sep 2005 17:00:30 GMT

>>1) Is there a group consensus to allow folding of Path?

I had understood that the last time we discussed this (not that long ago)
we agreed to leave it as it is.

>further by offering it to them

Exactly so. If a site really is receiving excessive traffic because some
peer is ignoring the Path after the first fold (N.B. peering sites, if
present, will tend to be near the start of the Path), then he simply has
to lean on the offending peer(s) to upgrade to this new-fangled standard
(and it won't be all that new-fangled by the time folded Paths are
appearing in significant quantity).

The effect is noway as harmful as would be the case of allowing folding in
the Newsgroups header before the bulk of servers had upgraded.

>>2) Is there a group consensus to allow whitespace in Path?

Whitespace was clearly allowed (as a delimiter) by RFC 1036; hence folding
was also implicitly allowed - the problem is that implementations largely
ignored this.

>longer than at present...

MISMATCHes will be relatively uncommon, with little effect on the length.

The problem with receiving machine's syslogs is that those sites most
prone to being involved with spams/scams are those most unlikely to be
willing to divulge what is in their syslogs.

>If they must be there, then I think that would be best

That would be possible. But my preference would be for something like "MAY
generate, though it might result in a small increase of traffic to sites
whose peers are not yet upgraded to this standard".

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Russ Allbery <[email-masked]>
Date: Mon, 19 Sep 2005 12:59:03 -0700

Charles Lindsey <[email-masked]> writes:
> Richard Clayton <[email-masked]> writes:

> MISMATCHes will be relatively uncommon, with little effect on the

I bet that MISMATCHes will be extremely common.  They are often seen
already even though only one server adds them (and then in somewhat
limited situations, I believe).

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 20 Sep 2005 11:24:21 GMT

>limited situations, I believe).

The few that I looked at in detail appeared not to be genuine MISMATCHes -
most likely inadequate checking by the match detection software. The same
sequence of Paths involving the same MISMATCH turned up again and again.

If a newsadmin is regularly inserting a MISMATCH for some site, then he
ought to be discussing with the peer in question what is going wrong, or
else he should fix his software (or more likely his 'sys' file). If there
are too many such false positives, then the whole value of the MISMATCH
notation (which is to catch bogously preloaded Paths) will be lost.

---

From: Richard Clayton <[email-masked]>
Date: Tue, 20 Sep 2005 14:57:57 +0100

In message <[email-masked]>, Charles Lindsey
<[email-masked]> writes

>> but if there's this huge encouragement

so quite what makes you assume that such wicked sites will bother to
record a "MISMATCH" ?

Looking at the Diablo release notes (since it is their invention) I find
the following:

        Added Path: name checking.  If the first element of the Path:
        received by an article does not match any 'alias' statements
        for the incoming connection, the IP address is prepended
        to the path: with .MISMATCH appended.

        >>> NOTE <<< you should grep through newly created spool
        directories every so often looking for .MISMATCH in the spool
        files to locate incoming feeds with improperly configured
        'alias's (in dnewsfeeds).  When I turned this feature on,
        I found that four of my 80+ feeds were misconfigured.

so in fact what it is actually attempting to record is sites who do not
include themselves in the Path: header field, but what it will actually
record is sites who peer with incorrectly configured servers :)

However, it is out there and although I note that Diablo now has a
"nomismatch" command to turn the feature off, it must be useful to
someone and so I suppose we have to allow it somehow.

- -- 
richard @ highwayman . com                       "Nothing seems the same
                          Still you never see the change from day to day
                                And no-one notices the customs slip away"

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 20 Sep 2005 16:20:32 +0200

> willing to divulge what is in their syslogs.

.... and those involved with spams/scams will be willing to put in 
MISMATCH....?

those who wish to lie will lie, no matter what the standard says.
If you presume a manager acting in good faith, the difference is in the 
amount of resources needed to get him to look at the logs.

Note: That MAY be a huge difference....

---

From: Russ Allbery <[email-masked]>
Date: Tue, 20 Sep 2005 09:47:51 -0700

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

> The few that I looked at in detail appeared not to be genuine MISMATCHes
> - most likely inadequate checking by the match detection software. The
> same sequence of Paths involving the same MISMATCH turned up again and
> again.

Right.  This will probably be common.

> else he should fix his software (or more likely his 'sys' file).

I think it's unlikely that anyone is going to do this on a regular basis.

> If there are too many such false positives, then the whole value of the
> MISMATCH notation (which is to catch bogously preloaded Paths) will be
> lost.

Yup, hence my skepticism, although I'm willing to support having the
feature in the standard given how many people have asked for it over the
years.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 21 Sep 2005 10:32:28 GMT

>so quite what makes you assume that such wicked sites will bother to

It's not the wicked sites that insert MISMATCHes. They are inserted by the
good sites downstream from the wicked sites so that their wickedness gets
exposed:

    good.site!wicked.site.MISMATCH!pretend-to-be-good.site!...

The point I was trying to make was that wicked sites usually try to attach
themselves to peers whose abuse-desks are known to be uncooperative to
abuse investigators.

>        to the path: with .MISMATCH appended.

Yes, that is more or less what we are trying to say.

>        I found that four of my 80+ feeds were misconfigured.

And that looks like excellent advice. I have made a note to consider
something like that at the relevant point in USEPRO.

>record is sites who peer with incorrectly configured servers :)

Plus the occasional genuinely wicked site; hence the reason to try to
minimize such false positives.

---

From: Richard Clayton <[email-masked]>
Date: Wed, 21 Sep 2005 12:36:06 +0100

In message <[email-masked]>, Charles Lindsey
<[email-masked]> writes
>    good.site!wicked.site.MISMATCH!pretend-to-be-good.site!...

no, what is currently created is

        ...good.site!192.168.2.1.MISMATCH!other.site!...

where the situation is that "good.site" does not believe (for whatever
reason) that 192.168.2.1 is called "other.site"

they could of course discard the article (believing it to be forged) or
they could write the information into their syslog (my original point)

Harald's observation is pertinent, in that "good.site" may not wish to
check their syslogs just because some random netcop thinks that the
article looks dodgy. However, since the IP who owns "192.168.2.1" is
unlikely to tell you anything about its user without a court order I
don't see that this gets you any further.  Also, in practice, since an
IP address is often entirely useless without a timestamp you are going
to have to get "good.site" to look at their logs anyway.

If you really wanted traceability you'd put

        ...!192.168.2.1.04:17:23.GMT.MISMATCH!...

>themselves to peers whose abuse-desks are known to be uncooperative to

this is entirely soggy thinking. IMNSHO (I regularly advise a major ISP
on exotic abuse issues) no abuse desk on the planet is -- without the
active co-operation of "good.site" -- going to act on an IP address
(without a timestamp) that someone else placed into a Usenet article
Path: header field. There's just too many ways of forging that
information :(

so this is all wishful thinking ... and the preponderance of system
misconfigurations that Diablo has exposed doesn't really sway me from
that.  I note that Russ doesn't think it does any harm, and I suppose I
agree with that -- but please don't consider it does much good either

- -- 
richard                                              Richard Clayton