Subject: [imapext] SUBMIT proposals and "will it be used"

From: Bron Gondwana <[email-masked]>
Date: Sun, 05 Aug 2012 20:47:12 +0200

Look what happens when I go away for a few days!  Lots of excitement.

Just summarising the points I've seen so far:

* Any proposed protocol needs to be extensible.

I agree - we can't predict the future of messaging.  Closing off
extensibility is dumb.

* It's going to be horribly complicated to support everything.

I kind of agree, but see below.

* We need to support every SMTP extension ever proposed and be everything
  to everybody.

I strongly disagree.  We have evidence that by far the most common IMAP clients
are a small handful.  Outlook/Outlook Express are going to be a problem of
course, but they already are for any new proposal.

iOS mail is quite responsive, and I suspect they will be willing to add things
which make life easier for their users.  Apple is into that.

Thunderbird is Open Source[tm].  I'm willing to do the patching myself if
necessary.  I already did for COMPRESS=DEFLATE because we needed it for FastMail.

And that's 99% of users covered.  I really feel that we keep banging our heads
on the complex use cases that <1% of users need.

* Firewalling - is it or isn't it a problem

The problem isn't so much firewalling at the mail server end - it's intermediate
blackholes and random firewalling.  The whole point of one protocol vs two here
is that it is either totally blocked, in which case it's very clearly an
intermediate ISP problem and the user yells at their ISP, or totally open.

The hard part about SMTP (or even Submission) and IMAP is that you get stupidly
configured intermediate firewalls who allow IMAP, but not SMTP or Submission.
Then the users get extra confused, and bounced between two different support
desks - nobody wins.

Experience shows that misconfigured firewalls are going to keep happening.  Two
different protocols on different ports with (as also mentioned) different
authentication mechanisms just means this is going to keep causing issues.

* You have to deal with complicated trust relationships between IMAP and
  Submission server anyway.

In 1% of cases.  99% of cases they're the same party, and they already have
an internal trust relationship set up.  If they're also running a webmail
system (very common these days) then they already have a method for injecting
authenticated emails into their outbound queue without an SMTP frontend - and
can SUBMIT straight into that same path.

* Nobody will use it because nobody is using it.

This is the same uphill battle that every extension faces.  As a counterpoint,
I present XLIST by Gmail.  It's widely supported now - so much so that Apple
managed to screw up and they'll send the LIST command but expect the XLIST
flags in the response still.  We added an option to Cyrus to send those
flags even if not asked to.  There's now a standard for it, in theory, but
there's a lot more support for the non-standard thing, because that was
there NOW, and it was useful enough that everyone did it.  We patched it into
FastMail long before pushing it back to Cyrus master.

I argue that a good submit is something that users want.  It's something that
will be jumped on as soon as it's available, in the way that BURL wasn't.
I argue quite strongly that the number of times it has been proposed is a
very good indication that there's a real problem which it would solve.  Maybe
the suggested draft isn't the right way to do it - but it needs to be be done.
The triangle of IMAP server, IMAP/SMTP client and SMTP server has one more
party and two more trust links than the average user wants to deal with.

Maybe this won't be the perfect protocol for mail forever, but neither is what
we have now.  The fact that almost every program just uploads the mail twice
still is a testament to something...

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Michael Fair <[email-masked]>
Date: Sun, 5 Aug 2012 15:50:39 -0700

I've been watching the discussions for some time and was wondering...

Has anyone considered/proposed making SUBMIT a special folder name (I like
OUTBOX) like INBOX is rather than a separate command.

Then using the APPEND or MOVE command to add a message to this special
folder to deliver messages.

This folder would be like the existing OUTBOX folder on most clients.

Only minor client modifications would be needed to handle ordinary messages.

Additional parameters that can't be put into the message itself can be set
by new parameters added to the MOVE or APPEND commands themselves.

Extensibility is handled by allowing a series of key/value flags to be sent
as part of the command.  Perhaps along the lines of JSON or URI encoding.

It's then up to the server to actually handle the rest.  Status flags on
the message could be used to communicate the delivery statuses.  Deleting
the message from the OUTBOX would signal completion of injection into the
SMTP queues.

I imagine servers would typically use a smarthost to handle the submission.

Thoughts, ideas, feedback?

Thanks for considering the idea,
Mike
On Aug 5, 2012 11:47 AM, "Bron Gondwana" <[email-masked]> wrote:

> _______________________________________________
> imapext mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/imapext
>

---

From: "Adrien de Croy" <[email-masked]>
Date: Sun, 05 Aug 2012 23:09:12 +0000

Hi

------ Original Message ------
From: "Bron Gondwana" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 6/08/2012 6:47:12 a.m.
Subject: [imapext] SUBMIT proposals and "will it be used"
>I agree - we can't predict the future of messaging.  Closing off

Yes, and I don't think anyone has been arguing that any potential 
SUBMIT extension should not in itself be extensible.

Even the current draft has room in it for a certain amount of 
extensibility.

We may just need to consider more aspects of the SMTP envelope to be 
extensible, such as forward-path (may need to become an object rather 
than a single parameter).  Etc.

>I kind of agree, but see below.

As long as the skeleton is extensible, I think any future SMTP 
extension will need to prove its worth anyway.

Everyone thinks their proposed extension is invaluable, but the 
community (developers) tend to disagree more often than not.

There have basically been 2 main options proposed.

1. client provides envelope, IMAP server does submission on behalf
2. client does protocol tunneled through to submit server

Personally, option 1 for me is a simple write-envelope-to-routing-slip 
+ file-copy-message-to-delivery-queue.  Probably about 50 lines of 
code.  I'm pretty certain I could get it working in about 1/2 hr, so 
it's a no brainer to adopt.

If one doesn't have an SMTP client implementation with their IMAP 
server, then sure it's going to be more difficult.  Option 2 for me 
would probably be very hard to justify writing.

Option 1 for me has another main benefit.  It allows the IMAP server to 
choose how submission will be done.  This has several potential 
advantages I see.

i. only 1 submission implementation to debug (in terms of why did a 
submit fail etc - of course this only applies when an organisation has 
migrated to IMAP SUBMIT)
ii. centralises management of submission to IMAP server operator.

and probably some more

the downsides are potentially there is more work to do when extensions 
come out.  However, I don't know if tunneling is really a starter 
either, since to avoid double-transmission, you'd need to proxy rather 
than tunnel anyway, which means the IMAP server now has to be able to 
deal with all SMTP protocol, and in fact SMTP extensions would be more 
likely to break submission, e.g. if a client chooses to use an 
extension which breaks message parsing / delineation in the proxy.

Especially since we can't mandate any limits on SMTP to cope with 
tunneled SMTP via IMAP.

>on the complex use cases that <1% of users need.
I agree.  This being an optional extension only needs to be supported 
by those who see value in it.

As long as we don't stuff it up for the future, and for everyone else.

>authentication mechanisms just means this is going to keep causing issues.

Most firewall problems relate to firewalls that are local to the 
client.  You wouldn't believe the number of customers we have who can't 
get a port opened on their local firewall (their own firewall!!!) due 
to various bureacratic / administrative setups.

They need to submit proposals, research etc etc.  Yes, some 
organisations (more than you'd think) are that tight.

>still is a testament to something...
I agree.

Adrien