Subject: Re: [Asrg] Body signature subgroup and the forwarding project

From: Dave Crocker <[email-masked]>
Date: Fri, 5 Aug 2005 10:00:14 +0200

>  That is, the signature
>  involves a hash of the message, but before making the hash, the
>  message is cleaned up to make it more likely that the signature will

John,

Having watched some of the distractions with DKIM vs. MASS, I am a bit 
sensitized.

We have seen some evidence that people believe the names that are used to label 
an activity. 

I believe this potentially useful activity is NOT about signatures.

It is about "survivable" canonicalization.  Since that is such an awkward 
phrase, it is probably reasonable to say it is about  hashes.

calling the activity bodysig is going to draw folk that actually want to talk 
about signatures.

  d/
  ---
  Dave Crocker
  Brandenburg InternetWorking
  +1.408.246.8253
  dcrocker  a t ...
  WE'VE MOVED to:  www.bbiw.net

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Hadmut Danisch <[email-masked]>
Date: Tue, 9 Aug 2005 18:03:56 +0200

On Wed, Jul 27, 2005 at 02:38:26AM -0000, John Levine wrote:
> At this point I haven't figured out what the mechanism for running
> messages through the forwarding gauntlet should be.  Suggestions
> welcome.  Paste them into a web form?  Mail them in as an attachment?

My guess is that it does not make much sense to just try this and that
message, and maybe a harder one.

We need to figure out what could cause transmission and then setup a
program, which automatically generates testmessage trying to provocate
the suspected conversion. 

Things I would like to see:

- White space at top, bottom, line ends of body. 

- Zero bytes. 

- Line ends in  0D, 0A, 0D0A

- Tabs convertet do spaces?

- Character sets: Try all bytes from 0..255 without charset, 
  try all charsets like iso-8859-1..15, arabic, chinese ones, etc. 
  and UTF-8. Try every possible character value (0..255 in most
  charsets, some more in UTF-8). 

- HTML: Tags up- oder downcased? canonicalized? script tags removed?

- Virus- and Spam-Mails (false positives!): Are they tagged or
  modified?

- Forwarding and Bouncing by .forward and MUAs. 

- Offending language: I found that especially US-made spam filter
  programs are prepared to educate users and to block lists of 
  sexual offending or vulgar language. Are such mails modified?

regards
Hadmut

---

From: der Mouse <[email-masked]>
Date: Tue, 9 Aug 2005 12:31:24 -0400 (EDT)

>   programs are prepared to educate users and to block lists of sexual
>   offending or vulgar language.

Which is particularly amusing when applied to mail in other languages,
in which such sequences of characters do not carry the same meaning.
(The closest example I know of offhand is Norwegian, in which "fart" is
a noun meaning, approximately, "speed" - probably related to German
"fahrt".)

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       [email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: Hadmut Danisch <[email-masked]>
Date: Tue, 9 Aug 2005 18:55:14 +0200

> a noun meaning, approximately, "speed" - probably related to German

Recently MSN Search startet to block such words like "Sex". 

Students who were looking for pages about the "Staatsexamen" (= german
for state examination) where notified that their query produces sexual
related content, and they were asked to modify their query.

:-)

Hadmut

---

From: David Wilson <[email-masked]>
Date: Wed, 10 Aug 2005 10:11:13 +0100

> a noun meaning, approximately, "speed" - probably related to German

And then there are programs which take substrings and object to them.
There was the case publicised some years ago about the program which
objected to the name of the East coast town in England: 'Scunthorpe'.

(and if you are too innocent to spot the problem, well done you).

cheers

David Wilson

---

From: David Wilson <[email-masked]>
Date: Wed, 10 Aug 2005 10:28:39 +0100

>   sexual offending or vulgar language. Are such mails modified?

This sounds good.

A few other comments.

Probably this should be called 'contentsig' rather than 'bodysig' for
the reasons outlined in other messages.

Perhaps before a test mechanism is devised, some idea of

- what entity will be generating the signature
- what entity will be checking the signature

is needed. If these are UAs, then rather more types of MTA need to be
checked than if the generating and checking entities are "boundary"
MTAs, communicating over the (actual) Internet. (Bad luck if you have
Exchange in the way anywhere).

If the message is to pass through a boundary MTA, then these can make
various changes to the message. The function of adding some disclaimer
to the body is often done at the boundary. (You know the sort of thing,
"This message is intended for the person I meant to send it to. If you
are not that person and you have read the message down to this point,
please, er, forget that you read it.") You can get similar messages
added when a message gets checked for viruses. (I have seen one example
which added it to a text/html but outside the main <html>...</html>
tags. Outlook displayed it, sensible UAs did not!)

There are a number of transformations which a boundary MTA (i.e. the MTA
which gives Internet access to an organization with a significant
internal Email network) might make to the heading of the message. E.g.

- Remove the Received trace from the internal MTAs
- Change the folding of any field
- 'Normalize' addresses in known fields (e.g. turning internal addresses
into addresses as they are to be visible to the outside world, or
vice-versa)
- Add Message-ID if missing

This kind of thing suggest to me that content based signatures are best
generated and checked at the boundary.

cheers

-- 
David Wilson <[email-masked]>

---

From: Hadmut Danisch <[email-masked]>
Date: Wed, 10 Aug 2005 19:37:24 +0200

...just another effect:

I was just converting a mailsystem from procmail to maildrop and
testing the new mailfilter configuration, when I found a difference
in a mail body:

A line in a mail from the ietf mailing list was modified from 
>From all I hear, the walls of Fortress Europe are still quite a bit lower
to
>>From all I hear, the walls of Fortress Europe are still quite a bit lower

I still have to figure out whether this is a bug or what, but
nevertheless it is a modification of the email body.

regards
Hadmut

---

From: Craig Cockburn <[email-masked]>
Date: Wed, 10 Aug 2005 21:43:33 +0100

In message <[email-masked]>, David Wilson 
<[email-masked]> writes
>(and if you are too innocent to spot the problem, well done you).

And the first four letters of my surname have caused no end of problems
See here
http://tinyurl.com/8f7pr

-- 
Craig Cockburn ("coburn"). SiliconGlen.com Ltd. http://SiliconGlen.com
Home to the first online guide to Scotland, founded 1994.
Scottish FAQ, wedding info, website design, stop spam and more!

---

From: John Levine <[email-masked]>
Date: 11 Aug 2005 03:53:17 -0000

>>>From all I hear, the walls of Fortress Europe are still quite a bit lower

That's pretty common whenever a message is stored in a Unix mbox mailbox.

It's one of the reasons I use maildirs which put each message in a
separate file.

R's,
John

---

From: der Mouse <[email-masked]>
Date: Thu, 11 Aug 2005 00:23:59 -0400 (EDT)

> That's pretty common whenever a message is stored in a Unix mbox

Is it?  In my experience it is not.  It's common to turn "From" at the
beginning of a line into ">From", but not to turn ">From" into
">>From".

> It's one of the reasons I use maildirs which put each message in a

Agreed - mbox format's in-band message boundary is..well, "broken"
might be putting it too strongly, but not by much.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign

---

From: [email-masked] (Hadmut Danisch)
Date: Thu, 11 Aug 2005 08:38:41 +0200

> beginning of a line into ">From", but not to turn ">From" into

But isn't the second a consequence from the first?

If you turn From into >From, then you must turn >From into >>From. 
Because otherwise you can't tell whether a >From in the message
actually was a From that was converted or a >From that was not
modified.

So the algorithm appears to be 

^(>*)From      ->  ${1}>From   when into mailbox and

^(>*)>From     ->  ${1}From    when out of mailbox

regards
Hadmut

---

From: David Wilson <[email-masked]>
Date: Thu, 11 Aug 2005 10:12:24 +0100

> ^(>*)>From     ->  ${1}From    when out of mailbox

Maybe the reported problem was that the system was not symmetrical in
this way, so the delivery agent performed the first conversion, but the
agent reading the message from the mbox file only converts >From to
From.

The RISK (to borrow a convention from elsewhere): the use of ad-hoc and
vague specifications for encoding and decoding.

---

From: "Mark E. Mallett" <[email-masked]>
Date: Thu, 11 Aug 2005 10:21:51 -0400

> beginning of a line into ">From", but not to turn ">From" into

"common" is a fuzzy concept when it comes to mboxes, and even more so
with regard to "From " line quoting.

The non-iterable non-nestable single-">" quoting is an old yet popular
style.

The nestable "From " quoting is known as "mboxrd" (google it, I guess),
and is newer (but not really recent) and also popular.

Some mailbox handling code attempts to be smart and only quote "From "
lines that have a tail part that looks like it might conform to some
known syntax (and conversely, not to quote ones that don't look real).
This kind of guessing just adds to the mess.

Recommendations vary.

> might be putting it too strongly, but not by much.

aye.

mm

---

From: John Levine <[email-masked]>
Date: 12 Aug 2005 03:50:46 -0000

>> ^(>*)>From     ->  ${1}From    when out of mailbox

Right.

>this way, ...

That's often a problem.  The "correct" way to store messages in
mboxes, if anything related to such a bogus format can be described as
correct, is to add a > when you store a message and remove one when
you fetch it, but code that handles mboxes is utterly inconsistent.
Some does that, some just puts a > in front of an unquoted From,
some stacks up the >>> but forgets to unstack it.

R's,
John

---

From: Kenny Thomas <[email-masked]>
Date: Fri, 12 Aug 2005 16:43:12 -0400

Then, you can well imagine the grief I have had getting mail from a man i work 
with who's last name is Porn :-)