Subject: [imapext] Accepting an "ESORT" response instead of an ESEARCH

From: Jan Kundrát <[email-masked]>
Date: Mon, 25 Feb 2013 15:59:58 +0100

Hi,
it turned out that a certain IMAP server sends out non-standard "ESORT" responses in response to the ESORT command (the correct way is to respond with an ESEARCH). I've filed a bugreport with an attached patch which hopefully fixes the issue, but even if they accept it soon, there's the usual problem with releases taking time and sysadmins delaying updates etc, and in the end, it's always the client which gets the blame "because other MUAs work just fine". The server the reporter is using was released roughly 10 months ago; I have unfortunately no idea about when they started advertising the ESORT capability, the publicly available git mirror doesn't go that far back in time.

Do you guys see any problem with a client simply accepting "* ESORT ..." in the same way and using the same syntax as an ESEARCH response? Would you leave such a workaround in your codebase indefinitely, or remove it after some time period?

With kind regards,
Jan

-- 
Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

---

From: Barry Leiba <[email-masked]>
Date: Mon, 25 Feb 2013 11:50:46 -0500

> Do you guys see any problem with a client simply accepting "* ESORT ..." in
> the same way and using the same syntax as an ESEARCH response?

Sure, why not.  You might also consider accepting "BURGFLUJ", "XYZZY",
and "SUPERCALIFRAGILISTICEXPIALIDOCIOUS" as well.

Barry

---

From: Dave Cridland <[email-masked]>
Date: Mon, 25 Feb 2013 16:58:42 +0000

I don't see why we need standardized responses anyway; we may as well use a
machine learning technique to extract the semantics of each response at
runtime.

---

From: Jan Kundrát <[email-masked]>
Date: Mon, 25 Feb 2013 18:31:31 +0100

> machine learning technique to extract the semantics of each response at

Well, I have some experience with consulting my crystal ball when deciphering responses from various servers. Most of that is documented on this ML or on the imap-protocol. The bugs are in proprietry and OSS servers alike. I report them when I hit them, and most of them get fixed, eventually. The server instances which manifest the problem are also upgraded most of the time. Eventually. Some of them never are, though.

Trojita is a small project, there's no company backing up its development with a pile of cash to hand out on a regular basis. Work is done in the free time of the contributors (with some occasional consulting in my case). In order to sustain the development, I try to attract new people to the project, and I guess the correct way to do so is by showing some flexibility and accomodating their needs to a certain extent. In this case, it's likely that they want to access their mail even when their company's server of choice sends garbage in violation of the relevant RFCs. Someone made a mistake in the code; that's not a big deal. Mistakes happen, I've done my fair share of them.

Yes, we have to drive the line somewhere. On the other hand, Trojita is probably the only client which has ever sent ESTORE to any IMAP server out there. Yes, it sucks to be the first one. However, this won't be the first or the last interoperability quirk I add to the code base.

I wonder what other client authors do here? Would you accept the ESORT here?

Cheers,
Jan

---

From: Bron Gondwana <[email-masked]>
Date: Tue, 26 Feb 2013 12:05:33 +1100

> I wonder what other client authors do here? Would you accept the ESORT here?

I would.  But I'm a hard-core pragmatist instead of the sticklers for rules
that seem to populate this newsgroup.  What would your users prefer, something
which copes, or something which tells them to make a snarky phonecall to their
email software vendor?

The other possibility is to detect that the response is broken and to fall back
to using other techniques to obtain the information (standard SORT - you need to
support is anyway for server which don't have all the fancy), even if they are
less efficient.

As for how long I'd support workarounds for - until the servers which need them
are no longer common is probably long enough.  The same way that websites are
starting to no longer actively support IE6 quirks.

Then again I work for Opera which has recently stopped building their own browser
engine because too many mobile websites only work with Webkit these days.  The
reality is that unless you have a standard test-suite which implementations have
to pass to claim to be "compliant" and you write that into the standard, you will
get buggy implementations - and you'll either need quirks to work around them or
you will find your users go to someone who gets the job done rather than quotes
a rulebook at them.

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Ned Freed <[email-masked]>
Date: Wed, 27 Feb 2013 19:02:09 -0800 (PST)

> that seem to populate this newsgroup.
                                        
In my case at least, you're waaaaay off base.

> What would your users prefer, something

If only the tradeoff were always this simple. Sure, there are cases, like the
one I encountered the other day in my managesieve server were there were a
couple of clients that wouldn't accept literal responses in one place, only
quoted strings. So I change our server to output quoted strings. Problem
solved.

Oh, wait. There's another client, a much more popular one, as a matter of fact,
that has a bug in its quoted string processing. Tickle that bug and only
one did the managesieve part of the client wedge, when the connection timed
out the entire client basically fell over.

Since the more popular client is the more important case to accomodate I filed
a bug. It turned out the problem had been partly, but not completely, solved in
the development branch. A few iterations with the author and now the problem is
completely fixed.

So much far "pragmatic" accomodation of broken behavior. In this case the truly
pragmatic approach was to actively work with the client author and get the
actual problem resolved.

Now, maybe with this ESORT business it's an incorrect string, nothing more. But
how sure are you of that? You filed a bug, have you followed up on that?

> support is anyway for server which don't have all the fancy), even if they are

Given the complexity of IMAP this sounds safer to me, and less likely to result
in snarky phone calls, this time to you.

				Ned

---

From: Bron Gondwana <[email-masked]>
Date: Fri, 01 Mar 2013 10:17:19 +1100

> quoted strings. So I change our server to output quoted strings. Problem

And we have something in Cyrus interop that's the other way - some old versions
of XFER require everything to be literals.  Thankfully it's all internal, so
we just send literals.
 
> pragmatic approach was to actively work with the client author and get the

That's great if you can also make sure that all the clients upgrade.  Servers
tend to be on a slower cycle - so you have to do both - work with the authors to
get the bug fixed, and try to work around it.

> how sure are you of that? You filed a bug, have you followed up on that?

Is there significant correlation between incorrect strings and other bugs?  There's
(in general) no way to tell how buggy the implementation you're talking to actually
is short of sanity checking its response.  Obviously, the sort could be totally
bogus.

> in snarky phone calls, this time to you.

Yep - good old RFC3501 - fallback for everything!

Bron.
-- 
  Bron Gondwana
  [email-masked]

---

From: Ned Freed <[email-masked]>
Date: Thu, 28 Feb 2013 17:48:40 -0800 (PST)

> Is there significant correlation between incorrect strings and other bugs?

Decades of experience says implementations that are sloppy about one thing are
far more likely to be sloppy about other things.

> There's

Indeed. And of course some imlementations are so buggy it's a waste of time to
even try. In such cases setting customer expectations early and correctly tends
to have a much better outcome then iterating many times before finding yourself
up against something you can't possibly work around.

				Ned

---

From: Bron Gondwana <[email-masked]>
Date: Fri, 01 Mar 2013 13:14:37 +1100

> far more likely to be sloppy about other things.

Yep.  So you fall back to just downloading all the message bodies locally and
doing your own sort.  That tends to work pretty well (unless it's Lotus Notes,
in which case all bets are off, because it mangles the raw message too)

> up against something you can't possibly work around.

Without an agreed-upon test suite, you get into he said/she said.  Java nearly
got this right, except for the whole keeping the test compatibility kit under
obnoxious licensing.

Bron.
-- 
  Bron Gondwana
  [email-masked]