Subject: [apps-discuss] multipart/report

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 22 Sep 2010 14:25:02 -0700

Re-sending to a wider audience...

From: Murray S. Kucherawy 
Sent: Tuesday, September 21, 2010 10:53 PM
To: [email-masked]
Subject: multipart/report

I'm doing some work with the OMA in the area of spam reporting.  They have developed an XML message format for reporting spam from a mobile handset.  Because SMS and MMS (and even IM) have different structure than email, they went with something more extensible than what we currently do with DSNs.  I'm helping that group try to converge with what we do in the email world, especially with what the MARF working group is producing (e.g., ARF, specified by recently-published RFC5965).  The first thing is to register the MIME type they've created, and so I've crafted a template for that.

The next thing we'd like to achieve is to be able to specify multiple spam reports in a single message, regardless of transport (they currently use HTTP POST, but there's no reason SMTP couldn't be used).  The most obvious thing to do would be to use a multipart/mixed MIME message, and then include their MIME type in each of "n" parts.  However, this is different than what was done with ARF, which used multipart/report.  The advantage to using multipart/report, which stipulates that it must be the outermost MIME type, is that you know right away that you're processing a report rather than having to parse part way into the MIME tree to figure that out.  (According to Tony Hansen, that's why this rule was put in place.)  But although multipart/report is the most obvious construct for doing this, it doesn't lend itself to multi-report messages.  So to get these to converge, we have to be a little creative or be very creative (in the "create a lot of documents" sense).

We could do a message/digest, each sub-part of which is message/rfc822 and the MIME type within that is multipart/report.  One could argue that this approach conforms to the multipart/report rules by having that be the outermost MIME part of each constituent message in the aggregate message, but that might be hard to defend given the spirit of that rule.

There's also a stipulation that the MIME subtype of the second part in a multipart/report message must be equal to the report-type of the outermost part, with no other constraints.  Thus to keep multipart/report at the outermost MIME but allow multi-message reports, we could comply by using "multipart/report; report-type=mixed" as the outermost MIME type and then use "multipart/mixed" inside.  But this also sure feels hack-ish.

So what would be more palatable to this community?  Could multipart/report be updated to accommodate this use case?  Should we register multipart/multi-report that relaxes some of the existing restrictions without changing the existing media type?

Feedback/guidance is welcome.

-MSK

---

From: Barry Leiba <[email-masked]>
Date: Thu, 23 Sep 2010 11:40:29 -0400

> We could do a message/digest, each sub-part of which is message/rfc822 and
> the MIME type within that is multipart/report.  One could argue that this approach
> conforms to the multipart/report rules by having that be the outermost MIME part
> of each constituent message in the aggregate message, but that might be hard
> to defend given the spirit of that rule.

What happens if you forward a multipart/report as an imbedded message?
 My guess is that you wind up with something like this:

0 multipart/mixed
   1.0   text/plain
   1.1   message/rfc822
      2.1      multipart/report

I don't think anyone would call the forwarded message a violation.
By the same token, I think what you describe above makes sense.

Barry

---

From: Keith Moore <[email-masked]>
Date: Thu, 23 Sep 2010 20:43:20 -0400

[NB: top posting is evil.]

> The next thing we'd like to achieve is to be able to specify multiple spam reports in a single message, regardless of transport (they currently use HTTP POST, but there's no reason SMTP couldn't be…

My vague recollection is that multipart/report was constrained to be the top-level MIME type mostly to ease the burden on mail gateways that would have wanted to automatically map between DSN and another kind of (non)delivery report.   

The requirement that it be the outermost MIME type might be a bit misstated.   I don't think anyone intended that you shouldn't be able to forward a DSN as an attachment to another message.  But there's a subtle but important semantic difference between sending a multipart/report at the top level of a message and sending a multipart/report as a body part within some other multipart or message type.  In the first case, the mail system is sending a report in response to a specific delivery attempt; in the second, a sender is just sending an attachment to a recipient and there's no reason that the mail system should care what that attachment is.   It might be reasonable, say, for a user agent or message submission agent of some kind (could be traditional, fax, or voice) to use received multipart/report messages to keep track of whether a sent message were delivered or whether delivery failed; but it would not be reasonable for that same user agent or message submission agent to look past the top level of received messages to see if they happen to contain a multipart/report.

(In a very ideal world, reporting should have been purely an SMTP function... we're mixing layers by sending reports in ordinary email messages.  But SMTP has worked that way for ~30 years; RFC 1891 et seq just tried to standardize the format.)

So my top-level response would be that multipart/report as a top-level type is appropriate when (and only when) the mail system is issuing a report in response to a delivery attempt.   I don't see anything inherently wrong with shipping multipart/reports around as children of a multipart/mixed, say, but when you do that, it no longer means that the mail system is sending a report in response to a delivery attempt.

I'm not sure I understand in what context you propose to use multipart/report for spam reporting.  It seems obvious that if the mail system is issuing a spam report to an envelope sender ("we refuse to deliver this; we think it's spam"), that a top-level multipart/report would be appropriate, but my guess is that this is not the intended use.   If you're using multipart/report for other purposes, e.g. to transmit a batch of spam reports from one MSP to another, I don't see any reason why it should be the top-level type, and I'd actually suggest that making it the top-level type would be inappropriate.  A multipart/mixed at the top level with multipart/reports body parts seems much more appropriate.  

Note also that multipart/report was designed to report potentially multiple delivery attempts - but for a single message.  It seems inappropriate to try to use a single multipart/report to report information about multiple messages. 

Bottom line: I doubt that using multipart/report as a top-level type is appropriate for what you seem to want to do.  And while I wouldn't have a problem with your using multipart/report underneath a multipart/mixed for this purpose, I have to wonder whether multipart/report is what you need at all.  If you want to use it as a way to send a report about a single message, consisting of human-readable explanatory text, an XML spam report, and optionally the original message - it's probably a good fit and maybe better than defining another type for that purpose.  But if that's not what you want to do, you're probably better off defining a new type for it.

Keith

---

From: Claudio Allocchio <[email-masked]>
Date: Fri, 24 Sep 2010 09:33:18 +0200 (CEST)

Hello all,

> I'm not sure I understand in what context you propose to use 
> multipart/report for spam reporting.  It seems obvious that if the mail 
> system is issuing a spam report to an envelope sender ("we refuse to 
> deliver this; we think it's spam"), that a top-level multipart/report 
> would be appropriate, but my guess is that this is not the intended use. 
> If you're using multipart/report for other purposes, e.g. to transmit a 
> batch of spam reports from one MSP to another, I don't see any reason 
> why it should be the top-level type, and I'd actually suggest that 
> making it the top-level type would be inappropriate.  A multipart/mixed 
> at the top level with multipart/reports body parts seems much more 
> appropriate.

I have an additional concern about the whole concept of sending a report 
stating "we refuse to deliver this; we think it's spam", in whichever 
format (machine or human readable) is could be:

  - it will likely happen in VERY FEW cases (compared to the total number 
of e-mail messages including real spam) that a real, non-spam message is 
mistaken as a spam one, and thus the relivery report is sent to a real 
user which needs to know his e-mail was mistaken by the recipient antispam 
system and hence the sender needs to act somehow.

  - in the great majority of cases the report will be sent becasue it was 
generated by REAM spam messages. But real spam messages have an 
originating (from/sender...) which either:

   - is a real e-mail address of an innocent user, who has nothing to do
     with the spam message itself, and also his/her client and MTA are
     innocent. In this case they will be annoyed by receiving tons of
     reports for a problem where they cannot do any action about. I
     believe this is unacceptable.

   - is a non valid e-mail address. In this other case, the report will
     just generate a useless overload on the MTAs, springing out non
     delivery reports, etc... also this is unacceptable IMHO.

Are we sure we are not just worsening the problems?

For the rest, I fully agree with Keith position.

------------------------------------------------------------------------------ 
Claudio Allocchio G A R R [email-masked]
                         Senior Technical Officer tel: +39 040 3758523 
Italian Academic and G=Claudio; S=Allocchio; fax: +39 040 3758565 Research 
Network P=garr; A=garr; C=it;

            PGP Key: http://www.cert.garr.it/PGP/keys.php3#ca

---

From: Barry Leiba <[email-masked]>
Date: Fri, 24 Sep 2010 11:41:14 -0400

Claudio says...
> generated by REAM spam messages. But real spam messages have an originating
> (from/sender...) which either:
>

I think you misunderstand the point here; perhaps if you read RFC
5965, you'll get the whole picture.  The short version is that these
are not SMTP-level bounces that say "unable to deliver [because I
think it's spam]".  Those, you'd expect would get sent back to the
sending user.

These are reports, triggered either by spam filters or by users, that
will be processed automatically by the sending domain.  The sending
domain MIGHT choose to send some of them back to the sending user, but
that's according to its handling policy.  In practice (this is already
fairly widely deployed), end users are not seeing these for "spam"
they didn't really send.

But Keith brings up an important point that I hadn't thought of, and
that might make us think again about the mechanism here.  As he says,
the intent of multipart/report as the top-level message was exactly to
allow automated processing of these things.  ARF is actually using
that feature as intended, but I now wonder if there might be
unintended consequences, exactly because these reports ARE different
to the usual non-delivery reports.

Do we have good data on how these are being handled by domains that
*don't* play the ARF game?  If my domain helpfully pushes an ARF
report back to your domain, but your domain doesn't know from ARF,
what will it do?  Won't it just treat it as a normal non-delivery
report and try to give it to the (purported) sending user?

Maybe the right thing to do in general is to wrap the multipart/report
in a multipart/mixed, so only ARF-knowing systems will do anything
with it.

Barry