Subject: [lemonade] extended error reporting

From: Peter Coates <[email-masked]>
Date: Mon, 26 Nov 2007 05:12:08 -0800

Attached is a suggestion that I believe addresses a problem in error
reporting in particular in URLFETCH, but which also applies to FETCH.

Peter Coates

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Mark Crispin <[email-masked]>
Date: Mon, 26 Nov 2007 09:41:40 -0800

> reporting in particular in URLFETCH, but which also applies to FETCH.

My reaction to this proposal is one of horror, even as I recognize that it 
is quite well-intentioned.

I prefer to abolish the error cases rather than adding additional 
complexity to IMAP to report errors.

(1) There is no excuse for a FETCH error.  Ever.

A client should NEVER attempt to FETCH something that does not exist.  It 
knows, from the untagged EXISTS response, exactly what sequence numbers 
are valid; and it knows, from the BODYSTRUCTURE, exactly what message 
parts exist.  An invalid sequence number is a BAD, and I should have made 
an invalid body part specifier a BAD as well.

Similarly, a server should NEVER return an error to a FETCH of a valid 
sequence number and message part; notwithstanding RFC 2180 4.1.2 and 4.1.3 
which are cases of poorly-designed (and IMHO non-compliant) server 
implementations.

(2) That leaves UID FETCH and URLFETCH.  The ONLY error condition that 
should exist for these commands is "this item no longer exists"; and thus 
there is one, and only one, meaning for a NIL return.

Complexity is not our friend, boys and girls.  IMAP is already far more 
complex than it needs to be, and the result is that it is nearly 
unimplementable.  There are major servers out there, from major players in 
the industry, which spit in the face of the current specification due to 
its excessive complexity.

It would be better to publish an informational RFC saying "RFC 2180 4.1.2 
and 4.1.3 are bad ideas and should never be done", thus simplifying(!) 
IMAP, than to add more complexity to IMAP that nobody will implement.

There are certainly aspects of IMAP in which better error reporting would 
be highly desirable.  FETCH is not one of them.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Peter Coates <[email-masked]>
Date: Mon, 26 Nov 2007 10:06:08 -0800

I beg to disagree with your assertion that the only possible interpretation
of a NIL in a response to a URLFETCH is "this item no longer exists".  The
URL specified could have become invalid between the time it was returned
from a GENURLAUTH and the time it was presented in a URLFETCH.  It could
have expired.  The UIDVALIDITY could have changed.  Access could have been
denied (or never granted).  The server has information that it could choose
to return to the client, information that could help diagnose the problem.
That URLFETCH does not have a way to return the information currently seem
to me to be a problem.

Peter

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Mark Crispin
Sent: November 26, 2007 09:42
To: Peter Coates
Cc: [email-masked]
Subject: Re: [lemonade] extended error reporting

-- Mark --

---

From: Ned Freed <[email-masked]>
Date: Mon, 26 Nov 2007 10:07:43 -0800 (PST)

> denied (or never granted).

Or the URL could have gotten corrupted in some way. Or the server could be
misconfigured, e.g., no submit user defined.

I ran into several of these working on our BURL implementation. Figuring out
what was going on in some cases was a major, major pain.

> The server has information that it could choose

A big one IMO.

Although this hasn't been an issue for me, I'm also not convinced by the
assertion that FETCH should never return an  error. Just as one example, what
about systems that move inactive message content to some type of secondary
storage?

				Ned

---

From: Mark Crispin <[email-masked]>
Date: Mon, 26 Nov 2007 11:16:57 -0800 (PST)

> misconfigured, e.g., no submit user defined.

These sound like things that a server should NOT disclose to a client due 
to security considerations (as opposed to something in its server log). 
An attacker may find that level of detail to be quite useful.

Remember the lesson of Tenex password validation!

> what was going on in some cases was a major, major pain.

I'm sure; and I don't dispute that these conditions should be in a 
server's error log.  I don't think that they should be disclosed to a 
client.

>> That URLFETCH does not have a way to return the information currently seem

A mechanism exists via status codes and untagged OK/NO/BAD.  It is not 
necessary to add a new syntax element.

> about systems that move inactive message content to some type of secondary

Back in 1987 we thought that such a thing may be necessary while waiting 
for the operator to mount the CD-ROM with the archived file.

It didn't happen then.  I doubt very much that it ever will in this day 
and age of petabyte primary stores.

IMAP has a perfectly mechanism -- multiple mailboxes -- to separate active 
from inactive content.  A mixture of active and inactive content within a 
mailbox did not happen, and IMHO never will.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 26 Nov 2007 20:36:04 +0100

Mark Crispin writes:
> (2) That leaves UID FETCH and URLFETCH.

And URLFETCH.

Arnt

---

From: Peter Coates <[email-masked]>
Date: Mon, 26 Nov 2007 11:53:41 -0800

It reveals nothing that they can't find out to tell a client that a URL has
expired or that they do not have access of a mailbox.  I do not see that as
a valid argument.  What revealing such information does is reduce service
calls.

As for other mechanisms, I grant that an untagged NO could be sent after
each URLFETCH response carrying the reason for the NIL.  But if that is to
be done, then that too needs specifying do different servers do not use
different conventions to correlate the error with the NIL.  And that reeks
of a contrapted solution, rather than the rather modest suggestion I put
forward in the first place.

The alternative solution for URLFETCH would be to limit it to a single URL
per command so the error reason can be carried in a NO response.  But
URLFETCH is already an RFC.

Contrary to your assertion, we do see NIL responses sent in FETCH responses
for transient operational reasons, and I see the demand for two tier
storage, and the like increasing rather than decreasing.  I grant that
multiple mailboxes offer a better solution for separating active from
inactive content, but that does not help the folks running the server.

Peter

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark
Crispin
Sent: November 26, 2007 11:17
To: Ned Freed
Cc: Peter Coates; [email-masked]
Subject: RE: [lemonade] extended error reporting

-- Mark --

http://panda.com/mrc

---

From: Mark Crispin <[email-masked]>
Date: Mon, 26 Nov 2007 14:33:49 -0800 (Pacific Standard Time)

On Mon, 26 Nov 2007, Peter Coates wrote:
> expired

If you are talking about URLAUTH URLs, the expiration is in plaintext in 
the URL and the client is perfectly capable of figuring that out for 
itself without server messages.

I'm not convinced that it is safe to say that the URLAUTH has been revoked 
or other details of how it is invalid.  I won't be without a thorough 
review by the security experts.

> or that they do not have access of a mailbox.

What are we talking about, [UID] FETCH or URLFETCH?

In the case of [UID] FETCH, the argument is silly.  Access is by mailbox, 
not by message; and the proper time for any such error message is at 
SELECT time.

In the case of URLFETCH, what does it mean to have a URLAUTH URL but not 
access?  If you are talking about non-URLAUTH URLs, that should be a NO.

> I do not see that as

Saying "it will reduce service calls" is like saying "it's for the 
children".  Such statements are nothing more than strawmen to stifle 
debate ("you want there to be more service calls, you evil person you").

The only situation in which error messages to the end user are useful are 
when the end user can do something about the condition.  Neither "URL has 
expired" nor "access not permitted" have any obvious remedial action for 
J. Random User other than placing a service call (or giving up).  Once 
again, what is put in the server log for support personnel is another 
matter entirely.

I don't consider it to be a valid tradeoff to reduce service calls but 
opening a potential security hole.  Security holes are not necessarily 
obvious either.  I call your attention, once again, to the Tenex password 
validation security bug.

It is necessary to think these things through carefully and cautiously, 
and not say "it isn't going to be a problem", or "it will reduce service 
calls."

Instead of strawmen arguments, give some concrete examples in which an 
error message would lead to remedial user action without the need for a 
service call.  Then!! we can talk about a means by which this is best 
conveyed.

I sympathize with your need to debug your code, but that's a special case 
that is best off done in a closed environment.  That is not a typical user 
use case.

> forward in the first place.

Bah.

"Yes, we could use an existing framework for this purpose, but it won't 
create as elegant a solution as creating an entirely new framework.  Our 
new framework will be so obviously wonderful that everybody will implement 
it."

Excess complexity is a disease, and the worst form of that disease is 
excessive complexity to create elegant solutions.

> URLFETCH is already an RFC.

Please offer a use case in which a multiple-URL URLFETCH does something 
useful with discrete error messages.  I suspect that in most real use 
circumstances, the interesting cases are "it worked (everything was 
fetched)" or "it didn't work".

> for transient operational reasons

Only broken IMAP servers do that.  If a broken implementation isn't going 
to be fixed, what makes anyone think that it's going to adopt this new 
error framework that you are proposaing?

> and I see the demand for two tier
> storage, and the like increasing rather than decreasing.

Who?  Where?

> I grant that

If this is a bug in a local server implementation, it would be better to 
fix that server rather than expect everything else to change to accomodate 
that server.

IMAP is about making things work, not explaining why it can only do a 
half-assed job of working.

-- Mark --

---

From: Ned Freed <[email-masked]>
Date: Mon, 26 Nov 2007 15:01:32 -0800 (PST)

> itself without server messages.

Not necessarily. A client may opt to slam a URL together and pass it on to
URLAUTH without having any real understanding of the format.

> review by the security experts.

That's just not how these things work. There are no magic security oracles we
can ask such questions of. Completed proposals in the IETF get some measure of
review as part of our standards process but that's about it. We're expected to
work out our security issues for ourselves, performing such analysis ourselves.

In fact these days the security area seems to even have given up on the notion
of assigning advisors to working groups. (Not that this would benefit us in
this case - advisors advise, they don't do our work for us.)

Now, if you have a specific security exposure in mind or better still an
analysis of why certain information would be a bad thing to reveal, that's one
thing, but absent vague concerns about security are not justification for
hiding useful information.

> ...

> when the end user can do something about the condition.

First, this isn't true. The user may not be in a position to fix some problems
but a site administrator may - as long as they have the necessary information.
And the most straightfoward way for them to get that information is from the
user. The alternative if the information isn't presented to the user is for
them to dig through server logs - which in the case of large scale systems
handling millions of users can be a very challenging task indeed, assuming the
error was even logged.

Second, it is not the case that none of the errors can be dealt with by the
user. The obvious one is that the message no longer exists. If the user knows
that's what went wrong they can address it (by referring to a different
message, giving up or whatever). The problem is right now when an error occurs
the user has no way of knowing what the underlying problem is and whether or
not they are in a position to address it.

In fact the main point of having a useful error is to tell the user whether or
not it is something they can address themselves.

> Neither "URL has

We deal with this sort of thing all the time at quite a few very large sites.
We never hear that improved error messages significantly reduce frontline calls
(most of which are probably of the "what's this cord with the two rectangular
things and one round thing for" sort anyhow), but we hear all the time that
they are hugely helpful in getting those calls resolved without having to push
issues "up the chain". Our customers therefore hammer on us constantly to
provide better, more consistent, and more detailed errors _without_ having to
fall back to logs. (They would also very much like having those errors in the
user's native language and not technical English, but that's a different
discussion for a different day.)

> obvious either.  I call your attention, once again, to the Tenex password

I'm afraid I no longer recall what this one was and google turns up nothing
useful.

> and not say "it isn't going to be a problem", or "it will reduce service

Now this, OTOH, is a true strawman. Nobody here is saying, "let's go off half
cocked and define something that spews out user passwords". Rather, several of
us have identified a problem during development that we believe will persist in
actual operation and we're discussing ways to address it without causing
additional trouble.

> service call.  Then!! we can talk about a means by which this is best

I have given several already.

In conclusion, I will add that I have absolutely no religion as to how this
gets addressed. I'm therefore staying out of the mechanism selection part of
this.

				Ned

---

From: Dave Cridland <[email-masked]>
Date: Tue, 27 Nov 2007 00:29:05 +0000

On Mon Nov 26 22:33:49 2007, Mark Crispin wrote:
> If you are talking about URLAUTH URLs, the expiration is in  
> plaintext in the URL and the client is perfectly capable of  
> figuring that out for itself without server messages.
> 
> I'm not convinced that it is safe to say that the URLAUTH has been  
> revoked or other details of how it is invalid.  I won't be without  
> a thorough review by the security experts.

In my experience, the BURL/URLAUTH situation is very weak in error  
reporting, and the client doesn't always know what the failure reason  
is.

Taking this specific example, mobile devices aren't always aware of  
the correct time, and a client trying to generate, then use (via  
BURL) an IMAP URL which has a tight expiration is perfectly capable  
of finding a largely opaque error coming back from the submission  
server, purely because the clock is wrong. (And sometimes, it's wrong  
because the GSM network is helpfully broadcasting the wrong time).

FWIW, I use a ten minute expiry in my client precisely to lessen the  
chances of this happening, but that's not always going to be enough -  
it's one of those shrugging-off implementation choices.

It's also possible for another client to expunge the message prior to  
the URLAUTH ticket being used, and quite possibly other error cases  
which are not clear-cut cases of developer error, but cases of user  
misconfiguration or misuse. I suspect that extending URLAUTH to  
handle conversion would elicit several more use cases.

I'm perfectly in agreement that if an apparently bogus URLAUTH URL  
gets passed in, that's an error I don't mind not reporting, but in  
some cases, the error requires specific user action to resolve - it's  
those errors I'd welcome better error reporting for.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Mark Crispin <[email-masked]>
Date: Mon, 26 Nov 2007 17:37:45 -0800 (Pacific Standard Time)

> URLAUTH without having any real understanding of the format.

A URLAUTH signed URL can't be "slammed together".  Once signed with a 
URLAUTH, it is immutable.  Whatever assemble the original URL has to know 
the semantics of expiration.

> That's just not how these things work. There are no magic security 
> oracles we can ask such questions of. Completed proposals in the IETF 
> get some measure of review as part of our standards process but that's 
> about it. We're expected to work out our security issues for ourselves, 
> performing such analysis ourselves.

In that case, then this group is remiss if it does not perform such 
analysis.

> analysis of why certain information would be a bad thing to reveal, 
> that's one thing, but absent vague concerns about security are not 
> justification for hiding useful information.

For each of the possible error messages, what information does this give 
the attacker that he would not have been able to discover easily 
otherwise?

Does it inform the attacker whether a URLAUTH signing key exists?

Does it inform the attacker that the data contents exist (or do not exist) 
even if he does not know the correct URLAUTH?

Does it, with a specially-tailored URL, give the attacker information that 
reduces the problem scope to determine the URLAUTH signing key?

Vague claims that "this isn't a problem" are not a way to address security 
considerations!

> First, this isn't true. The user may not be in a position to fix some 
> problems but a site administrator may - as long as they have the 
> necessary information. And the most straightfoward way for them to get 
> that information is from the user. The alternative if the information 
> isn't presented to the user is for them to dig through server logs - 
> which in the case of large scale systems handling millions of users can 
> be a very challenging task indeed, assuming the error was even logged.

So, because the server has a poor log implementation, we have to give this 
information to the user trusting that he will convey it accurately to the 
site administrator.  We also have to trust that this user isn't a hacker 
who has been collecting all this information to break in, using some 
vector that he knows about but we don't (yet).

I remain skeptical.

> Second, it is not the case that none of the errors can be dealt with by 
> the user. The obvious one is that the message no longer exists. If the 
> user knows that's what went wrong they can address it (by referring to a 
> different message, giving up or whatever). The problem is right now when 
> an error occurs the user has no way of knowing what the underlying 
> problem is and whether or not they are in a position to address it.

This scenario makes sense only with URLAUTH; and in that case a separate 
entity from the user generated the URLAUTH.

> In fact the main point of having a useful error is to tell the user 
> whether or not it is something they can address themselves.

And I contend that the use cases discussed so far are primarily for 
software developers debugging their implementations and NOT for end users.

Most usage of URLAUTH is via automated mechanisms, and the interesting 
thing to the end user is if it worked or not.  If it doesn't work, the end 
user isn't going to debug the URLAUTH.

> We deal with this sort of thing all the time at quite a few very large 
> sites. We never hear that improved error messages significantly reduce 
> frontline calls (most of which are probably of the "what's this cord 
> with the two rectangular things and one round thing for" sort anyhow), 
> but we hear all the time that they are hugely helpful in getting those 
> calls resolved without having to push issues "up the chain". Our 
> customers therefore hammer on us constantly to provide better, more 
> consistent, and more detailed errors _without_ having to fall back to 
> logs. (They would also very much like having those errors in the user's 
> native language and not technical English, but that's a different 

Who are your customers on this point?  The end users?  Or the site 
administrators?

If the latter, shouldn't you be addressing site administrator needs, 
rather than dump technical jargon on an end user?  You need a way for the 
site administrator to know better what is going on, without sharing that 
information with the Russian Mafia.

> I'm afraid I no longer recall what this one was and google turns up nothing

Tenex very helpfully told you if the userid was invalid, if the password 
was invalid, if there was some error in an argument, etc.  Passwords were 
validated, character-at-a-time, between the stored password and the user's 
argument string.

Once determining a valid userid (admittedly not generally difficult in 
Tenex), you would then pass it a one-character password, where the next 
character was on a no-access page.  "Invalid password" meant that 
character was not the first character of the password.  "Page fault" meant 
that it was the first character of the password.  Repeat with 
progressively longer strings until you got the entire password.

The first attempt at a fix was to fetch the entire password argument from 
the user before validating, so page fault would always happen.  But that 
wasn't good enough.  The more difficult, but equally effective, attack is 
left as an exercise for the reader; but the fix was a rewrite of the 
password validation logic so that the validity test was a single machine 
instruction.

> Now this, OTOH, is a true strawman. Nobody here is saying, "let's go off 
> half cocked and define something that spews out user passwords". Rather, 
> several of us have identified a problem during development that we 
> believe will persist in actual operation and we're discussing ways to 
> address it without causing additional trouble.

Better error reporting is a good thing; but let's have a listing of the 
errors that we want to report instead of adding an open-ended framework 
with just a handwave that "there are no problems."

> I have given several already.

You listed:
  . "slamming together a URLAUTH".  The end user can't do this at all.
  . "it no longer exists".  The end user has to get a new one from whomever
     gave him the URLAUTH.

In fact, for URLAUTH, there really are only the following small set of 
conditions for the end user:
  . You won.  Yay!
  . Foo, you are a hacker.
  . This URLAUTH is no longer valid.  Contact whomever gave it to you to
     get a new one.

For FETCH, there are only the following conditions:
  . You won.  Yay!
  . Your buggy client gave an invalid message sequence number.
  . Your buggy client gave an invalid part part specifier.  I tell you NIL.

For URL FETCH, there are the following conditions:
  . You won.  Yay!
  . This UID is no longer valid.  I'm saying nothing about that message.
  . Your buggy client gave an invalid part part specifier.  I tell you NIL.

Since you claim that there are more conditions than these, are they really 
unique conditions?  Are they brought about by the implementation, and 
would not exist if the implementation was any better?

If it is "foo, you are a hacker", do you really want to tell the hacker 
how he goofed?  Even if it is "no longer valid", do you necessarily want 
to tell the user (as opposed to the issuer) why?

If you have to tell the end user anything, I would be more comfortable 
with a mechanism that issued an incident token, which would then allow 
authorized personnel to get what really happpened.

> gets addressed. I'm therefore staying out of the mechanism selection part of

Interestingly, I have no religion on that point either.

IMAP is so incredibly complex and baroque that its doom is sealed whether 
or not one more gargoyle is added to the tower.

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Mon, 26 Nov 2007 18:22:50 -0800 (Pacific Standard Time)

On Tue, 27 Nov 2007, Dave Cridland wrote:
> In my experience, the BURL/URLAUTH situation is very weak in error reporting, 
> and the client doesn't always know what the failure reason is.

I agree.  But what is the target audience for the error reports?

End user?

Site administrator?

Software developer/debugger?

I submit that this is not a "one shoe fits all" type of situation.

> Taking this specific example, mobile devices aren't always aware of the 
> correct time, and a client trying to generate, then use (via BURL) an IMAP 
> URL which has a tight expiration is perfectly capable of finding a largely 
> opaque error coming back from the submission server, purely because the clock 
> is wrong. (And sometimes, it's wrong because the GSM network is helpfully 
> broadcasting the wrong time).

That scenario suggests that the correct protocol fix is to have a relative 
expiration date/time for the client doing the GENURLAUTH, and have the 
returned URLAUTH replace that relative expiration date/time with an 
absolute value.  For example,

authimapurlrump = authimapurl [expire-relative] iurlauth-rump

expire-relative = ";EXPIRESIN=" 1*DIGIT
                   ; number of seconds

That way, we don't have relative expirations in authimapurlfull; the 
returned authimapurlfull would have to include an absolute expiration.

An alternative is to encode a relative expiration in the mechanism field.

> It's also possible for another client to expunge the message prior to the 
> URLAUTH ticket being used, and quite possibly other error cases which are not 
> clear-cut cases of developer error, but cases of user misconfiguration or 
> misuse.

Yes, but those are cases of "this URLAUTH is no longer valid, contact the 
entity that gave you the URLAUTH for a new one."

> I'm perfectly in agreement that if an apparently bogus URLAUTH URL gets 
> passed in, that's an error I don't mind not reporting, but in some cases, the 
> error requires specific user action to resolve - it's those errors I'd 
> welcome better error reporting for.

Yes -- but once again, for whom?!?

-- Mark --

---

From: Dave Cridland <[email-masked]>
Date: Tue, 27 Nov 2007 10:23:10 +0000

On Tue Nov 27 02:22:50 2007, Mark Crispin wrote:
>> reporting, and the client doesn't always know what the failure  
>> reason is.
> 
And I absolutely agree, but I also submit that some errors do require  
user intervention to correct, and therefore should be presented  
clearly to the user for resolution.

>> Taking this specific example, mobile devices aren't always aware  
>> of the correct time, and a client trying to generate, then use  
>> (via BURL) an IMAP URL which has a tight expiration is perfectly  
>> capable of finding a largely opaque error coming back from the  
>> submission server, purely because the clock is wrong. (And  
>> sometimes, it's wrong because the GSM network is helpfully  
> That scenario suggests that the correct protocol fix is to have a  
> relative expiration date/time for the client doing the GENURLAUTH,  
> and have the returned URLAUTH replace that relative expiration  
> date/time with an absolute value.  For example,

Well, I'm not sure that helps, and more importantly I suspect that  
this would be a more radical change than Peter's proposal anyway.  
BURL/URLAUTH are deployed technologies [ie, not just Polymer], and  
the simplest way of handling this deficiency without breaking  
existing implementations would be to have the BURL command produce  
more useful diagnostics.

Besides which, there are other cases, including any case which  
involves server-side manipulation of a body part during the URLFETCH.  
My proposal of BINARY-mode is one such (consider bogus base64), and  
any similar proposal involving CONVERT-in-URLFETCH.

>> It's also possible for another client to expunge the message prior  
>> to the URLAUTH ticket being used, and quite possibly other error  
>> cases which are not clear-cut cases of developer error, but cases  
>> of user misconfiguration or misuse.
> 
> Yes, but those are cases of "this URLAUTH is no longer valid,  
> contact the entity that gave you the URLAUTH for a new one."
> 
> 
Yes, but often they're not cases where there is a security leak in  
explaining what the problem actually is. Bear in mind that the most  
common scenario for BURL/URLAUTH is the case where a client gets a  
URLAUTH for a message and then uses it quite rapidly in a BURL  
command within an authenticated SUBMIT session. The authorizing  
authority for the URL is the user, here, and moreover, if a hacker  
has got to the stage of being able to use a submit+ authorized URL,  
then he can almost certainly gain access to the IMAP store itself.

This is not always the case, of course, and I'm certainly not  
advocating producing details error responses for all cases. Just  
those that make sense.

>> some cases, the error requires specific user action to resolve -  
>> it's those errors I'd welcome better error reporting for.
> 

Sorry, I thought this was clear - "[...] the error requires specific  
user action to resolve" - hence the user.

Dave.

---

From: Mark Crispin <[email-masked]>
Date: Tue, 27 Nov 2007 07:50:28 -0800 (PST)

On Tue, 27 Nov 2007, Dave Cridland wrote:
> Well, I'm not sure that helps, and more importantly I suspect that this would 
> be a more radical change than Peter's proposal anyway. BURL/URLAUTH are 
> deployed technologies [ie, not just Polymer], and the simplest way of 
> handling this deficiency without breaking existing implementations would be 
> to have the BURL command produce more useful diagnostics.

Given your scenario of the GSM network broadcasting the wrong time, and 
hence the URLAUTH generating entity is making meaningless URLAUTHs, I 
don't see how an error message is going to help the end user at all.

It's the fool software that set up an expiration 10 minutes into the 
future based upon an incorrection notion of absolute time.  The user 
probably knows quite well that the time shown by his phone is hours off 
from the real time, and can't do a damn thing about it.  Even if the phone 
permits a manual reset, the network time update will just set it back.

And it's not just GSM.  I've seen this with CDMA networks.  So much for 
the myth that CDMA network time is guaranteed to be in synchronization 
with GPS time.  [Details on request.  I had someone argue with me that 
this was "impossible" at length on another forum before he finally 
understood.]

It seems to me that if you really want to generate expirations based upon 
a relative interval, we should extend the URLAUTH mechanism to allow this. 
Obvious we can't break the existing implementations, hence the word 
"extend".

> Besides which, there are other cases, including any case which involves 
> server-side manipulation of a body part during the URLFETCH. My proposal of 
> BINARY-mode is one such (consider bogus base64), and any similar proposal 
> involving CONVERT-in-URLFETCH.

In that case, the error report should be part of the conversion mechanism 
framework and not the fetching framework.

>>> URLAUTH ticket being used, and quite possibly other error cases which are 
>>> not clear-cut cases of developer error, but cases of user misconfiguration 
>>> or misuse.
> Yes, but often they're not cases where there is a security leak in explaining 
> what the problem actually is.

But the problem is "this URLAUTH is no longer valid, contact the entity 
that gave you the URLAUTH for a new one."

> Bear in mind that the most common scenario for 
> BURL/URLAUTH is the case where a client gets a URLAUTH for a message and then 
> uses it quite rapidly in a BURL command within an authenticated SUBMIT 
> session. The authorizing authority for the URL is the user, here, and 
> moreover, if a hacker has got to the stage of being able to use a submit+ 
> authorized URL, then he can almost certainly gain access to the IMAP store 
> itself.

Let's assume that the hacker, one way or another, has got to the state of 
knowing a submit+ authorized URL.  That doesn't help him much, since he 
can't use it, but he has an example of a URLAUTH generated by the correct 
key.  URLAUTH resists known-plaintext attacks, but perhaps he can try 
something else.

He now launches a brute force attack to see if he can create other 
URLAUTHs.  Perhaps he uses carefully-crafted URLs that are more likely to 
exploit a weakness in the encryption algorithm or key.  What do more 
detailed error messages tell him?  For example:

Do they make his task of ultimately finding the key any easier?

Short of finding the key, do they disclose details about the mailbox (such 
as the number of messages, or the structure of certain messages)?

> This is not always the case, of course, and I'm certainly not advocating 
> producing details error responses for all cases. Just those that make sense.

The next thing is: a new framework for error responses is proposed, 
instead of using the existing frameworks.  What value does this new 
framework gain that justifies the creation of Yet Another Framework to an 
already extremely overburdened protocol?

The chief benefit to the new framework is in associating the message 
precisely with the URL that triggered it in a multi-URL request; thus the 
desire not to use tagged response.

How is this useful to the end user?  Gee, the 23rd URL in my URLFETCH 
message got a URL expired error message.  I don't think so, especially 
given (by your own agreement) the most common case is "the case where a 
client gets a URLAUTH for a message and then uses it quite rapidly in a 
BURL command within an authenticated SUBMIT session."

How is this useful to an application?  The user's request failed.  In the 
most common case, the application is the submit server; and it's just 
going to bounce the entire thing back.  It's not going to do anything 
intelligent knowing that the 23rd URL was bad.  The unresolved security 
question here is how much goes in the bounce, and how much goes in the log 
with perhaps an incident tag in the bounce.  Even if you give the end user 
all the information, you can give it just as well in the existing 
framework using tagged response or a series of untagged NOs.  The proposed 
new framework doesn't gain anything for the bounce message.

Similarly, in the common case, the proposed new framework doesn't help 
site administrators either.  The important thing for a site administrator 
is how the submit server puts together its bounce for the end user.  The 
site administrator doesn't care if its where it is in some URLFETCH 
request.  Once again, the tagged response or a series of untagged NOs is 
completely adequate.

I submit that the main value for the new framework is software developers 
debugging their URLAUTH/BURL implementations, and NOT end users, 
applications, or site administrators.  Software developer convenience is 
NOT a good reason for adding something that is otherwise of limited or no 
use.

What's more, you are NOT going to get this new framework generally 
accepted with such a paltry value.  We have major IMAP servers that don't 
even implement all of the base specification (and whose vendors question 
the need to implement all of the base specification), much less any 
extensions.

-- Mark --

http://panda.com/mrc

---

From: Dave Cridland <[email-masked]>
Date: Tue, 27 Nov 2007 16:33:15 +0000

On Tue Nov 27 15:50:28 2007, Mark Crispin wrote:
>> involves server-side manipulation of a body part during the  
>> URLFETCH. My proposal of BINARY-mode is one such (consider bogus  
>> base64), and any similar proposal involving CONVERT-in-URLFETCH.
> 
> In that case, the error report should be part of the conversion  
> mechanism framework and not the fetching framework.
> 
> 
Well, indeed, but at a protocol level there is no such distinction -  
both may, and probably will, use URLFETCH.

>> Bear in mind that the most common scenario for BURL/URLAUTH is the  
>> case where a client gets a URLAUTH for a message and then uses it  
>> quite rapidly in a BURL command within an authenticated SUBMIT  
>> session. The authorizing authority for the URL is the user, here,  
>> and moreover, if a hacker has got to the stage of being able to  
>> use a submit+ authorized URL, then he can almost certainly gain  
>> access to the IMAP store itself.
> 
> Let's assume that the hacker, one way or another, has got to the  
> state of knowing a submit+ authorized URL.  That doesn't help him  
> much, since he can't use it, but he has an example of a URLAUTH  
> generated by the correct key.  URLAUTH resists known-plaintext  
> attacks, but perhaps he can try something else.
> 
> URLAUTHs.  Perhaps he uses carefully-crafted URLs that are more  
> likely to exploit a weakness in the encryption algorithm or key.   
> What do more detailed error messages tell him?  For example:
> 
> Short of finding the key, do they disclose details about the  
> mailbox (such as the number of messages, or the structure of  
> certain messages)?
> 
> 
Well, let's assume that the attacker has an expired URL. He can  
detirmine it is expired by checking the timestamp in the URL. Getting  
an error message telling him so is hardly a breach of security -  
we're allowed to tell people the time. RESETKEY or similar issues are  
different - here, the URL has been forcefully invalidated, and  
there's no need to distinguish between this and mailbox deletion.  
Doing so most certainly leaks information.

We can, in addition, reject anything that should have been caught at  
GENURLAUTH time, which includes non-existent body parts.

But there's a wealth of cases - including the message being expunged  
- for which the URL can be validated fully. In these case, you have a  
demonstrably valid URL that remains authorized, but which points to  
missing or unusable data. It can be argued that this grants your  
brute force attacker a greater chance of being able to identify an  
existent message and valid section to use in their URL, but equally,  
they have such an example already, so I'd dispute the strength of  
that argument.

FWIW, it'd be perfectly reasonable to encode within the INTERNAL gunk  
the modseq the message was generated at, so that if you had  
sufficient state (and Cyrus IMAP, for example, does) you could  
distinguish between "The message did not exist at the time of the  
GENURLAUTH" - reject out of hand - and "The message no longer exists"  
- provide an error.

> accepted with such a paltry value.  We have major IMAP servers that  
> don't even implement all of the base specification (and whose  
> vendors question the need to implement all of the base  
> specification), much less any extensions.

Given that this is being proposed by implementors of BURL capable  
submission servers, and you're arguing that it has most value to  
them, I feel that you're arguing that the value is higher to the  
implementor than to the end user, so I think you may be  
self-defeating here - it seems somewhat likely that, amongst URLAUTH  
implementors, and especially those also working on BURL, it would  
have a reasonably high uptake.

Dave.

---

From: Dave Cridland <[email-masked]>
Date: Tue, 27 Nov 2007 16:39:13 +0000

On Tue Nov 27 15:50:28 2007, Mark Crispin wrote:
> The chief benefit to the new framework is in associating the  
> message precisely with the URL that triggered it in a multi-URL  
> request; thus the desire not to use tagged response.
> 
> How is this useful to the end user?

Just to pick up on this point in particular, this allows the  
Submission server to associate the URLAUTH error with the BURL  
command which produced it, in case it happens to have batched up a  
series of BURL requests from different sessions while it's  
authenticating to the IMAP server, for example.

Whilst it may, or may not, be useful to the end user to know which  
particular BURL command caused the failure, it's certainly handy for  
the submission server to know.

Dave.

---

From: Mark Crispin <[email-masked]>
Date: Tue, 27 Nov 2007 12:10:47 -0800 (Pacific Standard Time)

On Tue, 27 Nov 2007, Dave Cridland wrote:
> Just to pick up on this point in particular, this allows the Submission 
> server to associate the URLAUTH error with the BURL command which produced 
> it, in case it happens to have batched up a series of BURL requests from 
> different sessions while it's authenticating to the IMAP server, for example.

OK, this is a scenario worth exploring.

It postulates that a submission server is going to batch multiple BURL 
requests into a single URLFETCH (presuamably to save RTTs); and that this 
sort of thing is going to be common enough that it is worth adding an 
additional framework for errors.

How likely is this scenario?

Is it likely to encompass all the BURL requests for a single message? 
For multiple messages?

What about the theory that claims that it is better NOT to batch multiple 
requests, but instead to have multiple sessions working on parts of the 
problem?  For example, the SMTP strategy advocated by the qmail church?

The point here is a sanity check.  It isn't enough that it can be done; 
the point is will it be done, or are we creating an elaborate kludge tower 
that in the long term won't be used because everybody (at least, everybody 
who matter...and we neither define nor control that) does something else.

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Tue, 27 Nov 2007 12:23:26 -0800 (Pacific Standard Time)

On Tue, 27 Nov 2007, Dave Cridland wrote:
> Well, indeed, but at a protocol level there is no such distinction - both 
> may, and probably will, use URLFETCH.

That presumes that the conversion is encoded into the URL, thus denying 
the client that choice.  Doesn't seem like a good idea to me.

> Well, let's assume that the attacker has an expired URL. He can detirmine it 
> is expired by checking the timestamp in the URL. Getting an error message 
> telling him so is hardly a breach of security - we're allowed to tell people 
> the time.

Correct.  On the other hand, since he can determine that fact himself even 
in the absence of an error message, that is not an interesting case for 
the error message.

> RESETKEY or similar issues are different - here, the URL has been 
> forcefully invalidated, and there's no need to distinguish between this and 
> mailbox deletion. Doing so most certainly leaks information.

Correct again, and if this proposal is to move forward I believe that 
these sorts of issues must be ferreted out and discussed in security 
considerations.  I don't think that it is my responsibility to do so; the 
proponents should do this.

> But there's a wealth of cases - including the message being expunged - for 
> which the URL can be validated fully. In these case, you have a demonstrably 
> valid URL that remains authorized, but which points to missing or unusable 
> data.

Here we diverge in opinion.  I believe that the case of a message being 
expunged MAY be an equivalent case to RESETKEY.  For example, I want to 
revoke just one particular URLAUTH that I issued, and not any others. 
Hence, I want a finer tool than RESETKEY.  I copy the affected message 
back to the same mailbox (people do this all the time!) then expunge the 
original copy.  I may not want

> Given that this is being proposed by implementors of BURL capable submission 
> servers, and you're arguing that it has most value to them, I feel that 
> you're arguing that the value is higher to the implementor than to the end 
> user, so I think you may be self-defeating here - it seems somewhat likely 
> that, amongst URLAUTH implementors, and especially those also working on 
> BURL, it would have a reasonably high uptake.

Maybe.  But I don't foresee large multi-URLAUTH URLFETCHes being widely 
used.  It strikes me as a good sounding idea that isn't so good in 
practice.

-- Mark --

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 27 Nov 2007 21:27:54 +0100

I haven't said anything. So I will: I like the basic idea (+1 on it) and 
will contribute once the discussion moves on to productive detail.

Arnt