Subject: re: Delete-Expunge (Was: Move?)
From: Pete Resnick <[email-masked]>
To: Mark Crispin <[email-masked]>
Cc: [email-masked]
Date: Sun, 21 Jan 2001 16:25:38 -0600
List-Id: <ietf-imapext.imc.org>

On 1/19/01 at 3:29 PM -0800, Mark Crispin wrote:

>Let me ask you again; would SHOULD type wording such as that above be
>satisfactory for your purposes?  You're invited to suggest any wordsmithing.

I think a SHOULD is asking for someone to violate it. However, I 
don't think we need a MUST to solve the problem. Let me suggest the 
following:

2.3.2: Change the description of the deleted flag to:

         \Deleted
            Message is marked for deletion, removed by later EXPUNGE or CLOSE

That makes it explicit that the message is removed by all and only 
the EXPUNGE and CLOSE commands.

7.4.1: Change the description of the EXPUNGE response to:

       The EXPUNGE response reports that the specified message sequence
       number has been permanently removed from the mailbox.  This response
       This response occurs as the result of EXPUNGE or CLOSE command.

       Note: Even though a message with the \Deleted flag set is not removed
       until an EXPUNGE or CLOSE is issued, untagged EXPUNGE responses
       may be the result of EXPUNGE or CLOSE commands issued in another
       session. In particular, a client MUST NOT assume that messages with the
       \Deleted flag will not be expunged simply because that client 
did not issue
       an EXPUNGE or CLOSE.

       Upon receipt of an EXPUNGE response, the message sequence number...

Now, we haven't issued any unenforceable directives. We've just very 
specifically defined the semantics of \Deleted and the EXPUNGE 
response. No MUSTs or SHOULDs; just explicit definition of the 
semantics.

pr

-- 
Pete Resnick <mailto:[email-masked]>
Eudora Engineering - QUALCOMM Incorporated
Ph: (217)337-6377 or (858)651-4478, Fax: (858)651-1102

Date: Fri, 19 Jan 2001 15:29:21 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: re: Delete-Expunge (Was: Move?)
To: Pete Resnick <[email-masked]>
cc: [email-masked]
Sender: [email-masked]

On Fri, 19 Jan 2001 14:43:12 -0600, Pete Resnick wrote:
> >Unfortunately, I do not believe that it is as easy as telling client
> >writers "servers are only allowed to expunge when they are explicitly told
> >to do so."
> I can tell you for us, if we had the statement in the RFC, we would
> do it. It would actually turn out to be less work for us to do it
> only the "IMAP way".

Please believe me when I say that I would like very much to make such a
statement, especially if it induces Qualcomm to make Eudora only use the
delete-expunge mechanism.

My problem is that I don't think that I can make such a statement, or at least
I don't think that I can make it a MUST.

Would you settle for a SHOULD?

For example:
	A server SHOULD NOT unilaterally expunge deleted messages from
	a mailbox unless it receives an EXPUNGE or CLOSE command from
	a client session that has that mailbox selected.  The server
	SHOULD retain deleted messages, with the \Deleted flag set,
	across multiple sessions until an explicit EXPUNGE or CLOSE is
	issued.

	Note that an EXPUNGE or CLOSE in one session MAY cause the
	messages to be removed in all sessions which have that mailbox
	selected.  In particular, a client MUST NOT assume that
	messages will not be expunged unless it issues an EXPUNGE or
	CLOSE.

> The
> only sense in which this is enforceable is that servers which fail to
> follow the directive will behave badly and will get beat up on.

Exactly, but I believe that in this case we are talking BCP and SHOULD, not
MUST.

> Similarly, we can say (if we want) that a successful STORE of a
> \Deleted flag for a message doesn't mean "Mark this for deletion and
> expunge it whenever you feel like it", but rather "Mark this for
> deletion and expunge it when you get an EXPUNGE or CLOSE command."

Historically, the former was the original intent; but the latter is what
everybody seems to have implemented.  There hasn't been much call for a long
time for systemwide auto-EXPUNGE when the disk is full.  Although who knows,
maybe someone will want to do that (ref. certain other messages in this
thread).

> 1. They performed the action in their client which called the EXPUNGE
> or CLOSE command.
> 2. Someone else who shared the mailbox did an EXPUNGE or CLOSE command.
> 3. Their server is not following the IMAP standard.
> We all have to code (and document to the user) dealing with things of
> type 1 and 2. Things of type 3 we can dismiss.

Except that, all the time, we have people who defy the standards with such
statements as "the standards are no good if they get in the way of doing
work."  Just the other day, I had someone wanting me to add "AW:" as an
alternative to "re:" because some broken software in Germany writes it.

> Indeed, by the same people that I initially asked about this
> behavior, I was told that any server that did would be soundly
> beat-up on by the community.

In my observation, practices which are "soundly beat-up on by the community"
become "widely accepted" just a few years later; and such practices that are
enshrined in standards result in the standards being defied or criticized.

If I am so unfortunate as to have to still be defending stipulations in IMAP
10 years from now, I need stronger consensus and support than this.  I've been
burnt a lot by this in the past, and I don't want

> My argument
> is that if we already believe auto-EXPUNGE is prohibited behavior, we
> should document that.

I don't consider it prohibited.  I consider it bad, unfortunate, and totally
unsuitable for any software I would use.  But compliant client software
handles it just fine, and thus I see the issue as one of implementor morality
rather than harm.

Let me ask you again; would SHOULD type wording such as that above be
satisfactory for your purposes?  You're invited to suggest any wordsmithing.
Also, does anyone object to text such as this?

Date: Fri, 19 Jan 2001 14:43:12 -0600
To: Mark Crispin <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: re: Delete-Expunge (Was: Move?)
Cc: [email-masked]
Sender: [email-masked]

On 1/19/01 at 9:44 AM -0800, Mark Crispin wrote:

>Unfortunately, I do not believe that it is as easy as telling client writers
>"servers are only allowed to expunge when they are explicitly told to do so."

I can tell you for us, if we had the statement in the RFC, we would 
do it. It would actually turn out to be less work for us to do it 
only the "IMAP way".

>Yes, "it is compliant behavior according to the IMAP specification 
>for a server to expunge messages which are marked \Deleted even if 
>it receives no EXPUNGE or CLOSE command and even if the mailbox is 
>not shared."  It is compliant because it is not explicitly 
>forbidden.  It is not forbidden because such a prohibition is 
>unenforceable.

Let's start from the end: unenforceability. Take the case of SMTP. 
RFC 1123 says:

       5.3.3  Reliable Mail Receipt

          When the receiver-SMTP accepts a piece of mail (by sending a
          "250 OK" message in response to DATA), it is accepting
          responsibility for delivering or relaying the message.  It must
          take this responsibility seriously, i.e., it MUST NOT lose the
          message for frivolous reasons, e.g., because the host later
          crashes or because of a predictable resource shortage.

Now you could say, "This requirement is unenforceable; a client can't 
tell the difference between a server that loses the message for 
frivolous reasons rather than for good reasons. Therefore, this 
shouldn't have ever been explicitly forbidden." That's nonsense. The 
only sense in which this is enforceable is that servers which fail to 
follow the directive will behave badly and will get beat up on. We're 
not defining syntactic requirements here which are visible from the 
wire; we're defining semantic requirements. A successful DATA command 
doesn't mean "this message has been accepted if I feel like it". It 
is defined as meaning "this message has been accepted and I won't 
lose it for frivolous reasons."

Similarly, we can say (if we want) that a successful STORE of a 
\Deleted flag for a message doesn't mean "Mark this for deletion and 
expunge it whenever you feel like it", but rather "Mark this for 
deletion and expunge it when you get an EXPUNGE or CLOSE command."

>How can a client know that an unexpected EXPUNGE of messages 
>occurred because of the server's capricious behavior, as opposed to 
>some other client having issued an EXPUNGE?  It can't.

Absolutely irrelevant. A client can't tell the difference and must be 
coded to accept untagged EXPUNGE responses. But if the requirement 
were in the document, the user could be guaranteed that if they mark 
a message for deletion (i.e., put it in the virtual "Trash"), it 
won't be expunged unless:

1. They performed the action in their client which called the EXPUNGE 
or CLOSE command.
2. Someone else who shared the mailbox did an EXPUNGE or CLOSE command.
3. Their server is not following the IMAP standard.

We all have to code (and document to the user) dealing with things of 
type 1 and 2. Things of type 3 we can dismiss. We can't crash or do 
something stupid, but we can sure tell our users, "Sorry, your server 
is broken. We're not going to write some special Trash mailbox code 
to fix it."

>I don't know of any server that auto-EXPUNGEs...

Indeed, by the same people that I initially asked about this 
behavior, I was told that any server that did would be soundly 
beat-up on by the community. That fact ought to be documented.

>but there's probably someone out there who would consider it 
>capricious to prohibit server auto-EXPUNGE. Observing that all 
>clients are required to handle the event anyway, he'll just ignore 
>the prohibition.

Implementors ignore countless prohibitions in standards now. That's 
utterly irrelevant too. That a prohibition exists defines what the 
community thinks are the standards of protocol behavior. My argument 
is that if we already believe auto-EXPUNGE is prohibited behavior, we 
should document that. We have to code around stupidity of some broken 
implementations, but having the ability to call something broken goes 
a long way.

If we all believe that auto-EXPUNGE is wrong, please document that.

pr

-- 
Pete Resnick <mailto:[email-masked]>
Eudora Engineering - QUALCOMM Incorporated
Ph: (217)337-6377 or (858)651-4478, Fax: (858)651-1102

Date: Fri, 19 Jan 2001 10:28:37 -0600
To: Mark Crispin <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Delete-Expunge (Was: Move?)
Cc: [email-masked]
Sender: [email-masked]

On 1/18/01 at 1:33 PM -0800, Mark Crispin wrote:

>You are confusing a user interface (and server storage) issue with a 
>protocol issue.  The client can present a "Trash" view to the user, 
>while still using the IMAP delete-expunge model.  It should; and 
>most clients do.

While you're trying to get the IMAP rev to last call, this might be a 
good time to ask:

There is a reason client writers provide a "Trash" mailbox mode 
instead of the IMAP delete-expunge mode. (Eudora, BTW, provides both 
modes.) I have heard the claim that it is compliant behavior 
according to the IMAP specification for a server to expunge messages 
which are marked \Deleted even if it receives no EXPUNGE or CLOSE 
command and even if the mailbox is not shared. Now, I carefully went 
through imapv-14 yesterday looking for support for this claim. There 
are several notes about untagged EXPUNGE responses that I might 
interpret to mean this, but there was no direct note one way or the 
other.

If it is true that servers can expunge at will, then there is no way 
to achieve the semantics of a "Trash" mailbox with the IMAP 
delete-expunge model: There is no way to say, "Mark this message for 
deletion and permanently delete it only when I tell you to." If the 
widespread belief is not true, and servers (barring nasty error 
conditions) are only allowed to expunge when they are explicitly told 
to, that should be clarified in the spec. I firmly believe that with 
such assurance, clients will stop playing this "Trash" mailbox game.

pr

-- 
Pete Resnick <mailto:[email-masked]>
Eudora Engineering - QUALCOMM Incorporated
Ph: (217)337-6377 or (858)651-4478, Fax: (858)651-1102

Date: Fri, 19 Jan 2001 09:44:40 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: re: Delete-Expunge (Was: Move?)
To: Pete Resnick <[email-masked]>
cc: [email-masked]
Sender: [email-masked]

Unfortunately, I do not believe that it is as easy as telling client writers
"servers are only allowed to expunge when they are explicitly told to do so."
I have had client writers tell me in no uncertain terms that they use a
"Trash" mailbox because it's less work for them to program it that way.

I don't see that situation changing, because "less work for me" is a powerful
motivator.  That is what is behind the demands for message MOVE and DELETE
operators.

Yes, "it is compliant behavior according to the IMAP specification for a
server to expunge messages which are marked \Deleted even if it receives no
EXPUNGE or CLOSE command and even if the mailbox is not shared."  It is
compliant because it is not explicitly forbidden.  It is not forbidden because
such a prohibition is unenforceable.

How can a client know that an unexpected EXPUNGE of messages occurred because
of the server's capricious behavior, as opposed to some other client having
issued an EXPUNGE?  It can't.

It's clearly possible for another client to issue an EXPUNGE.  Quite a bit of
protocol work was expended on stipulating when an unsolicited EXPUNGE can
happen.  None of that would have been necessary if shared EXPUNGE was
prohibited (as shared access is in POP3); but it was clear that people wanted
shared EXPUNGE.

I don't know of any server that auto-EXPUNGEs, but there's probably someone
out there who would consider it capricious to prohibit server auto-EXPUNGE.
Observing that all clients are required to handle the event anyway, he'll just
ignore the prohibition.

Date: Fri, 19 Jan 2001 12:20:14 +0000
From: John Haxby <[email-masked]>
Organization: OpenMail R&D
Cc: IMAP Extensions WG <[email-masked]>
Subject: Re: Move?
Sender: [email-masked]

This is a multi-part message in MIME format.
--------------86BF67392CCA2C8ABE2F11C2
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Mark Crispin wrote:

> On Thu, 18 Jan 2001 15:12:59 -0800 (PST), Mark Crispin wrote:
> >       1) maximum message size
> >       2) maximum overall long-term storage use
> >       3) maximum transient (short-term) storage use
> >
> > The point being that the second and third values should be quite different.
>
> Another point needs to be made about the first value.  The need for it is to
> prevent a denial of service problem.  It should be impossible for a single
> message to push a user near his second limit to the third limit.

While modifying the server quota model would permit this, you then run into a
selection of interesting extra problems.   For example, how long does the
transient quota last?  Until the next logout?  For five minutes? Until all
current sessions have logged out?   Does (or can) the server know about this
last state?  What if the client crashes half way through the copy/mark/expunge
(as they often do :-) you're then left badly over quota and, depending on the
implementation, this may mean that messages to the user might be rejected.  You
might wind up with two copies of the message because the mark didn't work and
you don't know, then whether the copy is corrupt ...

Suppose, further, you have a Very Large Message, slightly over half the
available free space in the message store.   You cannot copy that, regardless of
any quota systems (fortunately, the better message stores copy references which
neatly gets around this).

Quite apart from the difficulties of designing, yet alone implementing, a quota
scheme that allows copy to work, there are the various temporal difficulties of
not having an atomic move operation in the protocol.  I've mentioned the
crashing one.   There's also the problem of multiple client access (which, as
I've said, is partially addressed by UID+, partially because it requires all
clients to cooperate and use UID+).

The quota problem is really not going be solved without MOVE and/or
DELETE (message) in the protocol, but then there's the slightly harder problem
of getting users to deploy a client that actually uses the extension ... and
this should never be underestimated.   (Various people, Mark mostly, have often
suggested that the client should be fixed to avoid a particular problem, but
have failed to realise that getting, say, 110,000 users to do *anything* is well
nigh impossible.)

jch

--------------86BF67392CCA2C8ABE2F11C2
Content-Type: text/x-vcard; charset=us-ascii;
 name="jch.vcf"
Content-Transfer-Encoding: 7bit
Content-Description: Card for John Haxby
Content-Disposition: attachment;
 filename="jch.vcf"

begin:vcard 
n:Haxby;John
tel;fax:+44 1344 763686
tel;work:+44 1344 763711
x-mozilla-html:FALSE
url:https://ecardfile.com/id/jch
org:OpenMail R&D
adr:;;Hewlett Packard<br>Nine Mile Ride;Wokingham;Berks;RG40 3LL;United Kingdom
version:2.1
email;internet:[email-masked]
note;quoted-printable:<em>OpenMail for All!</em>&nbsp=3B<img src=3D"http://www.openmail.com/cyc/om/00/graphics/omlinux.jpg" width=3D53 height=3D62 align=3Dbottom>
x-mozilla-cpt:;25408
fn:John Haxby
end:vcard

--------------86BF67392CCA2C8ABE2F11C2--

Date: Fri, 19 Jan 2001 12:04:51 +0000
From: John Haxby <[email-masked]>
Organization: OpenMail R&D
To: Edward Hibbert <[email-masked]>
Cc: [email-masked]
Subject: Re: Move?
Sender: [email-masked]

This is a multi-part message in MIME format.
--------------6DA3E24986CBE1C5325A9D95
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Edward Hibbert wrote:

> Seems to me that deprecating clients which behave like this isn't going to
> be effective as a method of resolving the problem long term - as a user I
> find the Deleted Items folder much more useful than having messages in the
> same folder marked as deleted cluttering up the view.  Opinions clearly vary
> but I think there is a body of support for this approach and for the
> forseeable future there are going to be clients which have this model.
>

I agree.  The trouble with forcing any one model is that different users prefer
different things -- some like mark/delete, some like move to wastebasket, and
some like to just delete.   This leads users to choose specific clients, or
configure clients like Netscape in a particular way.   It seems rather odd that
the IMAP protocol should dictate user preferences ...

>
> Perhaps we should look at an extension for MOVE, though I'm not sure I'm
> competent to draft one.  I know Mark objects to this - what about other
> folk?
>

I'm not competant either.   To draft an extension, I mean.   However, I'll
happily review any extension that would provide MOVE (and preferably its
companion, DELETE).

jch

--------------6DA3E24986CBE1C5325A9D95
Content-Type: text/x-vcard; charset=us-ascii;
 name="jch.vcf"
Content-Transfer-Encoding: 7bit
Content-Description: Card for John Haxby
Content-Disposition: attachment;
 filename="jch.vcf"

begin:vcard 
n:Haxby;John
tel;fax:+44 1344 763686
tel;work:+44 1344 763711
x-mozilla-html:FALSE
url:https://ecardfile.com/id/jch
org:OpenMail R&D
adr:;;Hewlett Packard<br>Nine Mile Ride;Wokingham;Berks;RG40 3LL;United Kingdom
version:2.1
email;internet:[email-masked]
note;quoted-printable:<em>OpenMail for All!</em>&nbsp=3B<img src=3D"http://www.openmail.com/cyc/om/00/graphics/omlinux.jpg" width=3D53 height=3D62 align=3Dbottom>
x-mozilla-cpt:;25408
fn:John Haxby
end:vcard

--------------6DA3E24986CBE1C5325A9D95--

Date: Thu, 18 Jan 2001 15:39:43 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: RE: Move?
To: IMAP Extensions WG <[email-masked]>
Sender: [email-masked]

On Thu, 18 Jan 2001 15:12:59 -0800 (PST), Mark Crispin wrote:
> 	1) maximum message size
> 	2) maximum overall long-term storage use
> 	3) maximum transient (short-term) storage use
>
> The point being that the second and third values should be quite different.

Another point needs to be made about the first value.  The need for it is to
prevent a denial of service problem.  It should be impossible for a single
message to push a user near his second limit to the third limit.

Consequently, Q3 > Q2, Q3-Q2 > Q1

Date: Thu, 18 Jan 2001 15:12:59 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: RE: Move?
To: Edward Hibbert <[email-masked]>
cc: "'Larry Osterman'" <[email-masked]>, Caleb Clausen <[email-masked]>, [email-masked]
Sender: [email-masked]

On Thu, 18 Jan 2001 22:40:12 -0000, Edward Hibbert wrote:
> So, how is one to allow a client MOVE operation - which feels to me as
> though it should work as long as you are below your quota - and
> simultaneously not violate server quota limits?  Violating quota limits is
> anathema to some of our ISP customers.

Today, a client has to do COPY-STORE \DELETED.  I contend that this is
correct, since, unlike the proposed MOVE, it is reversible.  This is
important, not just from the perspective of allowing the user to "undo".

Specifically, if the MOVE operation fails for any reason -- such as a quota
exceeded condition at the target -- the state prior to the MOVE has to be
restored.  You can't do this unless the mail store implements MOVE of multiple
messages as an atomic operation; furthermore that it must lock the source
mailbox against any other operations until the MOVE is complete (especially
append of new messages or EXPUNGE of the moved messages); and on top of that
MOVE must guarantee that it can reverse without quota problems (suppose the
source mailbox is currently over quota).

A further complication exists with any other client which has the mailbox
selected at the time a MOVE is going on; that client can not get the untagged
EXPUNGE until it does a "safe" command.  The question then becomes about how
you maintain the necessary zombie state (which probably requires leaving
ghosts of the message behind...hence not solving the quota problem); since
MOVE forces an EXPUNGE the server can't take the easy out of no-op'ing the
EXPUNGE until such time as there is exclusive access.

These entail a lot of difficult requirements for MOVE, and ones that are
likely to be ignored by careless implementors who would just do a bunch of
rename() calls.  The result would be worse than the original problem.

A better solution is to reconsider your assumptions behind quotas; and in
particular the idea that a user's permanent space allotment should equal his
transient space allotment.  I spent many years in a past life (1973-1984)
dealing with file quotas in which users were constantly at the limit of their
quota.  Far more money was spent in human salaries than would have been
expended by buying more disk -- and that was at a time when disks were
expensive!

More to the point: a user with a 5MB quota should not be receiving 4.5MB MIME
godzillagrams.  Either his quota is much too low, or his expectations of what
size messages he can receive are too high.  A reasonable quota facility needs
at least three values:
	1) maximum message size
	2) maximum overall long-term storage use
	3) maximum transient (short-term) storage use

The point being that the second and third values should be quite different.

From: Edward Hibbert <[email-masked]>
To: "'Mark Crispin'" <[email-masked]>
Cc: "'Larry Osterman'" <[email-masked]>, Caleb Clausen <[email-masked]>, [email-masked]
Subject: RE: Move?
Date: Thu, 18 Jan 2001 22:40:12 -0000
Sender: [email-masked]

Mark Crispin wrote:

> Leaving aside the quota problem, 

But this is precisely the issue I want to address.  We can argue all day
about whether Trash folders are a good idea and how clients can best
represent them.  Let's not do that because it's emotive and outside the
scope of the server.  And I implement servers so I'm not too interested in
it.

Instead, let's concentrate on the quota issue, which applies just as much in
moving mail from one folder to another.  I'm assuming this is an operation
that most clients do.

So, how is one to allow a client MOVE operation - which feels to me as
though it should work as long as you are below your quota - and
simultaneously not violate server quota limits?  Violating quota limits is
anathema to some of our ISP customers.

One way round the problem would be to say that the quota should apply to
individual folders rather than mailboxes (apologies for the violation of
IMAP terminology here).  But that seems artificial to me, and more
irritating to users.

So I don't see a way of solving this without a MOVE extension.  I'm not
arguing that this is the most serious problem in the world, and I'm prepared
to be convinced that it's not worth doing anything about except perhaps
adding something to the implementation notes.  But I'd like to see whether
I'm missing some way of solving this, or whether there's support for a MOVE
extension.

Regards,

Edward.

Regards,

Edward.

From: Lyndon Nerenberg <[email-masked]>
To: Edward Hibbert <[email-masked]>
cc: [email-masked]
Subject: Re: Move? 
Organization: ACI Worldwide
Date: Thu, 18 Jan 2001 14:56:40 -0700
Sender: [email-masked]

> Seems to me that deprecating clients which behave like this isn't going to
> be effective as a method of resolving the problem long term - as a user I
> find the Deleted Items folder much more useful than having messages in the
> same folder marked as deleted cluttering up the view.

So fix the GUI interface. There is nothing on the planet that
says a message marked \Deleted cannot be displayed by the client as if
it was in a different folder (called Trash or whatever else you want
to call it). This is *not* rocket science. (Simeon had functionality
in it similar to this five years ago.)

--lyndon

Date: Thu, 18 Jan 2001 13:33:46 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: RE: Move?
To: Edward Hibbert <[email-masked]>
cc: "'Larry Osterman'" <[email-masked]>, Caleb Clausen <[email-masked]>, [email-masked]
Sender: [email-masked]

> as a user I
> find the Deleted Items folder much more useful than having messages in the
> same folder marked as deleted cluttering up the view.  Opinions clearly vary
> but I think there is a body of support for this approach and for the
> forseeable future there are going to be clients which have this model.

You are confusing a user interface (and server storage) issue with a protocol
issue.  The client can present a "Trash" view to the user, while still using
the IMAP delete-expunge model.  It should; and most clients do.  Doing so also
neatly avoids the quota problem.

Leaving aside the quota problem, as a protocol operation, the Trash model is
fatally flawed -- unlike deleted status, it is impossible to reverse a "Trash"
operation.  The only thing you can do is add a new message (with a new UID!)
to the original mailbox containing the message in question.

Yes, it is highly unfortunate that one commonly-used client made the wrong
choice.  That program is disproportionately represented among my incoming
support question, due to this and certain other unfortunate design decisions.
We all have to make the best of it, but I don't think that breaking IMAP's
fundamentals is a good idea.

> Perhaps we should look at an extension for MOVE, though I'm not sure I'm
> competent to draft one.  I know Mark objects to this - what about other
> folk?

Let me emphasize:

I oppose the standardization of any extension which creates an independent
means from EXPUNGE to remove a message from a mailbox.  This is a very serious
violation of the IMAP architecture.

Even if this extension is promulgated over my objections, there is no reason
to believe that it will solve your problem.  It presupposes widespread
implementation, including in the particular client that you're purposing to
convenience -- assuming there is still any development in that program.

From: Edward Hibbert <[email-masked]>
To: "'Larry Osterman'" <[email-masked]>, Caleb Clausen <[email-masked]>
Cc: [email-masked]
Subject: RE: Move?
Date: Thu, 18 Jan 2001 20:33:08 -0000
Sender: [email-masked]

That's precisely right.  We do already make a reference when a COPY is done
(and for multirecipient messages), but if you don't charge that copy against
the quota then you have the problem of which of the messages actually holds
the size and therefore which one the user has to delete to free up the
space.  I guess you could split the charged size across the number of copies
and recalculate it each time you copy, but that gets silly, and as Larry
says, you end up with users deleting messages and wondering why their
mailbox is still full.

Another point to note is that references themselves consume some space - for
us the minimal message size, because they do have to have their own flags -
and so ought to be charged against the quota.

Seems to me that deprecating clients which behave like this isn't going to
be effective as a method of resolving the problem long term - as a user I
find the Deleted Items folder much more useful than having messages in the
same folder marked as deleted cluttering up the view.  Opinions clearly vary
but I think there is a body of support for this approach and for the
forseeable future there are going to be clients which have this model.

Perhaps we should look at an extension for MOVE, though I'm not sure I'm
competent to draft one.  I know Mark objects to this - what about other
folk?

Regards,

Edward.

-----Original Message-----
From: Larry Osterman [mailto:[email-masked]]
Sent: Thursday, January 18, 2001 8:11 PM
To: Caleb Clausen; Edward Hibbert
Cc: [email-masked]
Subject: RE: Move?

It's possible that Edwards' data store is like the Exchange store, which
natively implements this (we call it "Single Instance Storage").  The
problem is that in order to make the quota management work correctly,
the store needs to charge each instance against the quota - this means
that the amount of quota charged against a particular user is greater
than the sum of the storage they are using, however if we don't do it
this way, there is no way for a client to determine how their quota is
being used - users get REALLY annoyed if they delete an item to get
below their quota only to discover that it doesn't help them.

-----Original Message-----
From: Caleb Clausen [mailto:[email-masked]]
Sent: Thursday, January 18, 2001 11:08 AM
To: Edward Hibbert
Cc: [email-masked]'
Subject: RE: Move?

From:           	Edward Hibbert <[email-masked]>
> My motivation for a MOVE is not based on an infatuation with OS
commands
> :-).  I have a nice transactional database with rollback and all that
good
> stuff.
> 
> Instead it's about quota.  At the moment we enforce the quota limit on
the
> COPY command.  We have come up against a client which maintains a
Deleted
> Items folder (mailbox in IMAP terminology), so it moves items into it
when
> the user deletes them.  To do the move it does a COPY/DELETE/EXPUNGE.

> 
> But this process requires that you have as much free quota as the size
of
> the message you're trying to move - which is obviously not possible if
the
> message is itself more than half your quota limit.
> 
> Changing the client is not really an option, so I'm left with couple
of
> ideas, neither of which I like very much:
> - don't check quota on COPY
> - hardcode the Deleted Items folder name and don't check quota if
we're
> copying to there.
> 
> Obviously with a MOVE command it would be clear that you weren't
trying to
> violate your quota.  It wouldn't help me just now because I can't
change the
> client to use it, but it seems to me that this is an argument in
favour of
> it.
> 
> Do any of you folk have any other ideas as to how to solve this?

well, here's an idea. use reference counts internally on your 
messages. that way, you wouldn't have to have room for an extra 
message when the user does a COPY, just an extra reference.

i haven't given this a lot of thought; i don't know how hard it would 
be to implement. one thing that springs to mind is the flags; each 
reference would have to have it's own set of them.

"For, I say fortunately, I always carry a spare set 
of feathers."
  --Foghorn Leghorn

content-class: urn:content-classes:message
Subject: RE: Move?
Date: Thu, 18 Jan 2001 12:10:56 -0800
From: "Larry Osterman" <[email-masked]>
To: "Caleb Clausen" <[email-masked]>, "Edward Hibbert" <[email-masked]>
Cc: <[email-masked]>
Sender: [email-masked]

It's possible that Edwards' data store is like the Exchange store, which
natively implements this (we call it "Single Instance Storage").  The
problem is that in order to make the quota management work correctly,
the store needs to charge each instance against the quota - this means
that the amount of quota charged against a particular user is greater
than the sum of the storage they are using, however if we don't do it
this way, there is no way for a client to determine how their quota is
being used - users get REALLY annoyed if they delete an item to get
below their quota only to discover that it doesn't help them.

-----Original Message-----
From: Caleb Clausen [mailto:[email-masked]]
Sent: Thursday, January 18, 2001 11:08 AM
To: Edward Hibbert
Cc: [email-masked]'
Subject: RE: Move?

From:           	Edward Hibbert <[email-masked]>
> My motivation for a MOVE is not based on an infatuation with OS
commands
> :-).  I have a nice transactional database with rollback and all that
good
> stuff.
> 
> Instead it's about quota.  At the moment we enforce the quota limit on
the
> COPY command.  We have come up against a client which maintains a
Deleted
> Items folder (mailbox in IMAP terminology), so it moves items into it
when
> the user deletes them.  To do the move it does a COPY/DELETE/EXPUNGE.

> 
> But this process requires that you have as much free quota as the size
of
> the message you're trying to move - which is obviously not possible if
the
> message is itself more than half your quota limit.
> 
> Changing the client is not really an option, so I'm left with couple
of
> ideas, neither of which I like very much:
> - don't check quota on COPY
> - hardcode the Deleted Items folder name and don't check quota if
we're
> copying to there.
> 
> Obviously with a MOVE command it would be clear that you weren't
trying to
> violate your quota.  It wouldn't help me just now because I can't
change the
> client to use it, but it seems to me that this is an argument in
favour of
> it.
> 
> Do any of you folk have any other ideas as to how to solve this?

well, here's an idea. use reference counts internally on your 
messages. that way, you wouldn't have to have room for an extra 
message when the user does a COPY, just an extra reference.

i haven't given this a lot of thought; i don't know how hard it would 
be to implement. one thing that springs to mind is the flags; each 
reference would have to have it's own set of them.

"For, I say fortunately, I always carry a spare set 
of feathers."
  --Foghorn Leghorn

Date: Thu, 18 Jan 2001 11:23:08 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: RE: Move?
To: Caleb Clausen <[email-masked]>
cc: Edward Hibbert <[email-masked]>, "[email-masked]'" <[email-masked]>
Sender: [email-masked]

On Thu, 18 Jan 2001 11:07:32 -0800, Caleb Clausen wrote:
> well, here's an idea. use reference counts internally on your
> messages. that way, you wouldn't have to have room for an extra
> message when the user does a COPY, just an extra reference.

That's an interesting point.  link() is a perfectly valid way of implementing
COPY for the message data since it is static.  Parts of the metadata --
INTERNALDATE, RFC822.SIZE, ENVELOPE, BODYSTRUCTURE -- can also be lunk.
However, each mailbox needs its own copy of other metadata, most notably FLAGS
and UID.

If the underlying problem is that users can't clean their mail due to quotas
and Netscape's use of a "Trash" mailbox, I suggest that the problem is in
Netscape and not in any IMAP server.  The Trash mailbox is a Netscape-ism and
is not sanctioned by IMAP in any way.  Clients which use IMAP the way IMAP was
intended to use don't have this problem.

You may be able to work around the problem by applying separate quotas to the
Trash folder; but IMHO it's also a good answer to say "use another client."

From: "Caleb Clausen" <[email-masked]>
To: Edward Hibbert <[email-masked]>
Date: Thu, 18 Jan 2001 11:07:32 -0800
Subject: RE: Move?
CC: "[email-masked]'" <[email-masked]>
Sender: [email-masked]

From:           	Edward Hibbert <[email-masked]>
> My motivation for a MOVE is not based on an infatuation with OS commands
> :-).  I have a nice transactional database with rollback and all that good
> stuff.
> 
> Instead it's about quota.  At the moment we enforce the quota limit on the
> COPY command.  We have come up against a client which maintains a Deleted
> Items folder (mailbox in IMAP terminology), so it moves items into it when
> the user deletes them.  To do the move it does a COPY/DELETE/EXPUNGE.  
> 
> But this process requires that you have as much free quota as the size of
> the message you're trying to move - which is obviously not possible if the
> message is itself more than half your quota limit.
> 
> Changing the client is not really an option, so I'm left with couple of
> ideas, neither of which I like very much:
> - don't check quota on COPY
> - hardcode the Deleted Items folder name and don't check quota if we're
> copying to there.
> 
> Obviously with a MOVE command it would be clear that you weren't trying to
> violate your quota.  It wouldn't help me just now because I can't change the
> client to use it, but it seems to me that this is an argument in favour of
> it.
> 
> Do any of you folk have any other ideas as to how to solve this?

well, here's an idea. use reference counts internally on your 
messages. that way, you wouldn't have to have room for an extra 
message when the user does a COPY, just an extra reference.

i haven't given this a lot of thought; i don't know how hard it would 
be to implement. one thing that springs to mind is the flags; each 
reference would have to have it's own set of them.

"For, I say fortunately, I always carry a spare set 
of feathers."
  --Foghorn Leghorn

Date: Thu, 18 Jan 2001 12:46:42 -0500
From: Ken Murchison <[email-masked]>
Organization: Oceana Matrix Ltd.
To: Edward Hibbert <[email-masked]>
CC: "[email-masked]'" <[email-masked]>
Subject: Re: Move?
Sender: [email-masked]

Edward Hibbert wrote:
> 
> My motivation for a MOVE is not based on an infatuation with OS commands
> :-).  I have a nice transactional database with rollback and all that good
> stuff.
> 
> Instead it's about quota.  At the moment we enforce the quota limit on the
> COPY command.  We have come up against a client which maintains a Deleted
> Items folder (mailbox in IMAP terminology), so it moves items into it when
> the user deletes them.  To do the move it does a COPY/DELETE/EXPUNGE.
> 
> But this process requires that you have as much free quota as the size of
> the message you're trying to move - which is obviously not possible if the
> message is itself more than half your quota limit.
> 
> Changing the client is not really an option, so I'm left with couple of
> ideas, neither of which I like very much:
> - don't check quota on COPY
> - hardcode the Deleted Items folder name and don't check quota if we're
> copying to there.
> 
> Obviously with a MOVE command it would be clear that you weren't trying to
> violate your quota.  It wouldn't help me just now because I can't change the
> client to use it, but it seems to me that this is an argument in favour of
> it.
> 
> Do any of you folk have any other ideas as to how to solve this?

We had a similar problem here with people using Communicator with a
Trash folder.  If they ran up against their quota limit, they couldn't
delete messages for the reason you describe above.

I solved it by placing a separate QUOTAROOT (via Cyrus imapd) on the
Trash folder so "deleted" items dont count against their "main" quota.

Ken
-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

Date: Thu, 18 Jan 2001 09:50:52 +0000
From: John Haxby <[email-masked]>
Organization: OpenMail R&D
To: Edward Hibbert <[email-masked]>
Cc: "[email-masked]'" <[email-masked]>
Subject: Re: Move?
Sender: [email-masked]

This is a multi-part message in MIME format.
--------------AD66766772BFAF0AAFAAFD54
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Edward Hibbert wrote:

> My motivation for a MOVE is not based on an infatuation with OS commands
> :-).  I have a nice transactional database with rollback and all that good
> stuff.
>
> Instead it's about quota.  At the moment we enforce the quota limit on the
> COPY command.  We have come up against a client which maintains a Deleted
> Items folder (mailbox in IMAP terminology), so it moves items into it when
> the user deletes them.  To do the move it does a COPY/DELETE/EXPUNGE.
>
> But this process requires that you have as much free quota as the size of
> the message you're trying to move - which is obviously not possible if the
> message is itself more than half your quota limit.

We have exactly this problem as well.   We just ignore the limitation.  At some
stage, someone will complain and then we might have to do something about it.
I have also given up on the idea of having a DELETE command as well.   (Quite
apart from the various race conditions in a mark/delete scheme, we get problems
with users who delete a message but since the client doesn't actually delete it
just yet, you don't drop below quota.   One of these days, someone is going to
decide that merely modifying messages when you are over quota is a bad move and
you won't be able to get below it again...)

jch, dispirited

--------------AD66766772BFAF0AAFAAFD54
Content-Type: text/x-vcard; charset=us-ascii;
 name="jch.vcf"
Content-Transfer-Encoding: 7bit
Content-Description: Card for John Haxby
Content-Disposition: attachment;
 filename="jch.vcf"

begin:vcard 
n:Haxby;John
tel;fax:+44 1344 763686
tel;work:+44 1344 763711
x-mozilla-html:FALSE
url:https://ecardfile.com/id/jch
org:OpenMail R&D
adr:;;Hewlett Packard<br>Nine Mile Ride;Wokingham;Berks;RG40 3LL;United Kingdom
version:2.1
email;internet:[email-masked]
note;quoted-printable:<em>OpenMail for All!</em>&nbsp=3B<img src=3D"http://www.openmail.com/cyc/om/00/graphics/omlinux.jpg" width=3D53 height=3D62 align=3Dbottom>
x-mozilla-cpt:;25408
fn:John Haxby
end:vcard

--------------AD66766772BFAF0AAFAAFD54--

From: Edward Hibbert <[email-masked]>
To: "'Mark Crispin'" <[email-masked]>
Cc: "[email-masked]'" <[email-masked]>
Subject: RE: Move?
Date: Thu, 18 Jan 2001 09:06:04 -0000
Sender: [email-masked]

My motivation for a MOVE is not based on an infatuation with OS commands
:-).  I have a nice transactional database with rollback and all that good
stuff.

Instead it's about quota.  At the moment we enforce the quota limit on the
COPY command.  We have come up against a client which maintains a Deleted
Items folder (mailbox in IMAP terminology), so it moves items into it when
the user deletes them.  To do the move it does a COPY/DELETE/EXPUNGE.  

But this process requires that you have as much free quota as the size of
the message you're trying to move - which is obviously not possible if the
message is itself more than half your quota limit.

Changing the client is not really an option, so I'm left with couple of
ideas, neither of which I like very much:
- don't check quota on COPY
- hardcode the Deleted Items folder name and don't check quota if we're
copying to there.

Obviously with a MOVE command it would be clear that you weren't trying to
violate your quota.  It wouldn't help me just now because I can't change the
client to use it, but it seems to me that this is an argument in favour of
it.

Do any of you folk have any other ideas as to how to solve this?

Regards,

Edward Hibbert
Internet Applications Group
Data Connection Ltd
Tel:	+44 131 662 1212		Fax:	+44 131 662 1345
Email:	[email-masked]	Web:	http://www.dataconnection.com

-----Original Message-----
From: Mark Crispin [mailto:[email-masked]]
Sent: Wednesday, January 17, 2001 9:52 PM
To: Edward Hibbert
Cc: [email-masked]'
Subject: re: Move?

On Wed, 17 Jan 2001 21:47:16 -0000, Edward Hibbert wrote:
> There's a COPY command to copy an item from one mailbox to another.  But
so
> far as I'm aware there isn't a MOVE command, so a client has to do a COPY
> and then a DELETE (and possibly an EXPUNGE if it wants the item to
disappear
> from its view of the current folder, depending on how it presents this).

That is correct.

> Is there any particular reason why there isn't a MOVE command?

You just gave the reason: the functionality of a MOVE command can be done
with
the existing commands.  It also preserves the IMAP delete-expunge model in
which it is impossible to remove a message from a mailbox in a single step.

The obvious temptation for implementations which have a one-file/one-message
form of mail store would be to implement MOVE via a rename() system call;
it's
also the primary motivation for asking for it (the secondary reason is to
eliminate the additional RTT).  That violates the IMAP delete-expunge model.

An additional problem which is often overlooked with rename() is that COPY
requires that the operation succeed or fail atomicly.  There is no atomic
rename(); which means that if a rename() fails (e.g. due to disk quota in
the
target) the previous messages have to be rename()d back -- which could also
fail!

The bottom line here is that the rename() idea is an "attractive nuisance";
one that looks desirable on the surface but has serious problems.  Since
these
problems are not necessarily obvious, affirmative measures need to be in
place
to prevent it.

For MOVE, such affirmative measures would require the preservation of a
deleted copy in the source mailbox; hence resulting in the exact equivalent
of
COPY+STORE \DELETED.  This could be done, but such a command would just be
protocol bloat.

Date: Wed, 17 Jan 2001 13:52:09 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: re: Move?
To: Edward Hibbert <[email-masked]>
cc: "[email-masked]'" <[email-masked]>
Sender: [email-masked]

On Wed, 17 Jan 2001 21:47:16 -0000, Edward Hibbert wrote:
> There's a COPY command to copy an item from one mailbox to another.  But so
> far as I'm aware there isn't a MOVE command, so a client has to do a COPY
> and then a DELETE (and possibly an EXPUNGE if it wants the item to disappear
> from its view of the current folder, depending on how it presents this).

That is correct.

> Is there any particular reason why there isn't a MOVE command?

You just gave the reason: the functionality of a MOVE command can be done with
the existing commands.  It also preserves the IMAP delete-expunge model in
which it is impossible to remove a message from a mailbox in a single step.

The obvious temptation for implementations which have a one-file/one-message
form of mail store would be to implement MOVE via a rename() system call; it's
also the primary motivation for asking for it (the secondary reason is to
eliminate the additional RTT).  That violates the IMAP delete-expunge model.

An additional problem which is often overlooked with rename() is that COPY
requires that the operation succeed or fail atomicly.  There is no atomic
rename(); which means that if a rename() fails (e.g. due to disk quota in the
target) the previous messages have to be rename()d back -- which could also
fail!

The bottom line here is that the rename() idea is an "attractive nuisance";
one that looks desirable on the surface but has serious problems.  Since these
problems are not necessarily obvious, affirmative measures need to be in place
to prevent it.

For MOVE, such affirmative measures would require the preservation of a
deleted copy in the source mailbox; hence resulting in the exact equivalent of
COPY+STORE \DELETED.  This could be done, but such a command would just be
protocol bloat.

From: Edward Hibbert <[email-masked]>
To: "[email-masked]'" <[email-masked]>
Subject: Move?
Date: Wed, 17 Jan 2001 21:47:16 -0000
Sender: [email-masked]

I'm sure this is going to turn out to be a stupid question.

There's a COPY command to copy an item from one mailbox to another.  But so
far as I'm aware there isn't a MOVE command, so a client has to do a COPY
and then a DELETE (and possibly an EXPUNGE if it wants the item to disappear
from its view of the current folder, depending on how it presents this).

Is there any particular reason why there isn't a MOVE command?

Edward Hibbert
Internet Applications Group
Data Connection Ltd
Tel:	+44 131 662 1212		Fax:	+44 131 662 1345
Email:	[email-masked]	Web:	http://www.dataconnection.com

Date: Fri, 5 Jan 2001 11:24:19 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: Re: LAST CALL: IMAP base and MULTIAPPEND drafts
To: Simon Josefsson <[email-masked]>
cc: IMAP Interest List <[email-masked]>, IMAP Extensions WG <[email-masked]>
Sender: [email-masked]

On 05 Jan 2001 13:06:45 +0100, Simon Josefsson wrote:
> Suggestion, replace:
>       8-bit characters
>       are permitted in the message.
> with
>       8-bit characters
>       are permitted in the message body.
> since 8-bit characters are not permitted at all in the header section
> of a message, I think.

Nominally, this is a good idea.  However, I have not made this change, and
unless there is strong concensus for it I don't intend to do so, for the
following reasons:

 1) it would require that both the IMAP base (which has similar wording) and
IMAP MULTIAPPEND documents be revised if 8-bit characters are ever allowed in
the header.  I'm thinking of the time when all the world is Unicode and we can
dispense with all other charsets, encoded words, etc.

 2) There is also the question of allowing APPEND for messages that don't
necessarily conform to RFC 822; this is particularly an issue with drafts.

There seems to be a concensus (which I share) that IMAP servers should not
attempt to do message format validation on APPEND because of the issue with
drafts.  However, if such validation is done, I feel that the specification of
the validation should be in the message format document (RFC 822 or successor)
and not in an IMAP document.  As far as the IMAP document is concerned, the
APPEND text is simply a literal.

> Also, regarding UIDPLUS interactions, I'd suggest replacing
>    That is, the APPENDUID response code returns as many UIDs as there
>    were messages appended in the multiple append.
> with
>    That is, the APPENDUID response code returns as many UIDs as there
>    were messages appended in the multiple append.  The UIDs returned
>    should be in the order the articles where appended.  The message
>    set may not contain extraneous UIDs or the symbol '*'.
> to mimic some of the text from RFC 2359 regarding `set's and COPYUID.

This is a good idea and I've made the change.

To: Mark Crispin <[email-masked]>
Cc: IMAP Interest List <[email-masked]>, IMAP Extensions WG <[email-masked]>
Subject: Re: LAST CALL: IMAP base and MULTIAPPEND drafts
From: Simon Josefsson <[email-masked]>
Date: 05 Jan 2001 13:06:45 +0100
Lines: 56
Sender: [email-masked]

Mark Crispin <[email-masked]> writes:

> This is also a last call for
>  http://www.ietf.org/internet-drafts/draft-crispin-imap-multiappend-02.txt
> which is a minor (but exceedingly useful) extension.  It is unchanged from
> the -01 draft except for an updated expiration date.

I haven't had time to read this draft until now, so this is a late
minor clearification of two points...  Sorry for late reaction.

Suggestion, replace:

      The APPEND command appends the literal arguments as new messages
      to the end of the specified destination mailbox.  This argument
      SHOULD be in the format of an [RFC-822] message.  8-bit characters
      are permitted in the message.  A server implementation that is
      unable to preserve 8-bit data properly MUST be able to reversibly
      convert 8-bit APPEND data to 7-bit using a [MIME-IMB] content
      transfer encoding.

with

      The APPEND command appends the literal arguments as new messages
      to the end of the specified destination mailbox.  This argument
      SHOULD be in the format of an [RFC-822] message.  8-bit characters
      are permitted in the message body.  A server implementation that is
      unable to preserve 8-bit data properly MUST be able to reversibly
      convert 8-bit APPEND data to 7-bit using a [MIME-IMB] content
      transfer encoding.

since 8-bit characters are not permitted at all in the header section
of a message, I think.

Also, regarding UIDPLUS interactions, I'd suggest replacing

   Servers which support both MULTIAPPEND and [UIDPLUS] will have the
   "resp-code-apnd" rule modified as follows:

   resp-code-apnd  = "APPENDUID" SP nz-number SP set

   That is, the APPENDUID response code returns as many UIDs as there
   were messages appended in the multiple append.

with

   Servers which support both MULTIAPPEND and [UIDPLUS] will have the
   "resp-code-apnd" rule modified as follows:

   resp-code-apnd  = "APPENDUID" SP nz-number SP set

   That is, the APPENDUID response code returns as many UIDs as there
   were messages appended in the multiple append.  The UIDs returned
   should be in the order the articles where appended.  The message
   set may not contain extraneous UIDs or the symbol '*'.

to mimic some of the text from RFC 2359 regarding `set's and COPYUID.

Date: Thu, 4 Jan 2001 19:44:53 -0800 (PST)
From: Mark Crispin <[email-masked]>
To: IMAP Interest List <[email-masked]>
cc: IMAP Extensions WG <[email-masked]>
Subject: LAST CALL: IMAP base and MULTIAPPEND drafts
Organization: Networks & Distributed Computing
Sender: [email-masked]

As discussed in the IMAPEXT meeting in San Diego,...

This is a last call for
 http://www.ietf.org/internet-drafts/draft-crispin-imapv-13.txt
which is the latest revision of RFC 2060, correcting all known errors and
omissions.  It has been 4 years since RFC 2060 was released, and there are
a lot of changes and clarifications.

This is also a last call for
 http://www.ietf.org/internet-drafts/draft-crispin-imap-multiappend-02.txt
which is a minor (but exceedingly useful) extension.  It is unchanged from
the -01 draft except for an updated expiration date.

Does IMAPEXT want to take on MULTIAPPEND, or just let it advance to
Proposed Standard status by itself?  I don't forsee any further work being
done to MULTIAPPEND, and people have been implementing it...

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

To: IETF-Announce: ;
Cc: [email-masked]
From: [email-masked]
Reply-to: [email-masked]
Subject: I-D ACTION:draft-ietf-imapext-thread-06.txt
Date: Thu, 04 Jan 2001 07:27:27 -0500
Sender: [email-masked]

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Internet Message Access Protocol Extension Working Group of the IETF.

	Title		: INTERNET MESSAGE ACCESS PROTOCOL - THREAD EXTENSION
	Author(s)	: M. Crispin, K. Murchison
	Filename	: draft-ietf-imapext-thread-06.txt
	Pages		: 13
	Date		: 03-Jan-01
	
This document describes the server-based threading extension to the
IMAP4rev1 protocol.  This extension provides substantial performance
improvements for IMAP clients which offer threaded views.
A server which supports this extension indicates this with more or
more capability names consisting of 'THREAD-' followed by a supported
threading algorithm name as described in this document.  This
provides for future upwards-compatible extensions.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-imapext-thread-06.txt

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-ietf-imapext-thread-06.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-ietf-imapext-thread-06.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="[email-masked]"

Content-Type: text/plain
Content-ID:	<[email-masked]>

ENCODING mime
FILE /internet-drafts/draft-ietf-imapext-thread-06.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-ietf-imapext-thread-06.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<[email-masked]>

--OtherAccess--

--NextPart--

To: IETF-Announce: ;
Cc: [email-masked]
From: [email-masked]
Reply-to: [email-masked]
Subject: I-D ACTION:draft-ietf-imapext-sort-06.txt
Date: Thu, 04 Jan 2001 07:27:22 -0500
Sender: [email-masked]

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Internet Message Access Protocol Extension Working Group of the IETF.

	Title		: INTERNET MESSAGE ACCESS PROTOCOL - SORT EXTENSION
	Author(s)	: M. Crispin, K. Murchison
	Filename	: draft-ietf-imapext-sort-06.txt
	Pages		: 10
	Date		: 03-Jan-01
	
This document describes an experimental server-based sorting
extension to the IMAP4rev1 protocol, as implemented by the University
of Washington's IMAP toolkit.  This extension provides substantial
performance improvements for IMAP clients which offer sorted views.
A server which supports this extension indicates this with a
capability name of 'SORT'.  Client implementations SHOULD accept any
capability name which begins with 'SORT' as indicating support for
the extension described in this document.  This provides for future
upwards-compatible extensions.
At the time of this document was written, the IMAP Extensions Working
Group (IETF-IMAPEXT) was considering upwards-compatible additions to
the SORT extension described in this document, tenatively called the
SORT2 extension.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-imapext-sort-06.txt

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-ietf-imapext-sort-06.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-ietf-imapext-sort-06.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="[email-masked]"

Content-Type: text/plain
Content-ID:	<[email-masked]>

ENCODING mime
FILE /internet-drafts/draft-ietf-imapext-sort-06.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-ietf-imapext-sort-06.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<[email-masked]>

--OtherAccess--

--NextPart--

Date: Sun, 21 Jan 2001 16:25:38 -0600
To: Mark Crispin <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: re: Delete-Expunge (Was: Move?)
Cc: [email-masked]
Sender: [email-masked]

On 1/19/01 at 3:29 PM -0800, Mark Crispin wrote:

>Let me ask you again; would SHOULD type wording such as that above be
>satisfactory for your purposes?  You're invited to suggest any wordsmithing.

I think a SHOULD is asking for someone to violate it. However, I 
don't think we need a MUST to solve the problem. Let me suggest the 
following:

2.3.2: Change the description of the deleted flag to:

         \Deleted
            Message is marked for deletion, removed by later EXPUNGE or CLOSE

That makes it explicit that the message is removed by all and only 
the EXPUNGE and CLOSE commands.

7.4.1: Change the description of the EXPUNGE response to:

       The EXPUNGE response reports that the specified message sequence
       number has been permanently removed from the mailbox.  This response
       This response occurs as the result of EXPUNGE or CLOSE command.

       Note: Even though a message with the \Deleted flag set is not removed
       until an EXPUNGE or CLOSE is issued, untagged EXPUNGE responses
       may be the result of EXPUNGE or CLOSE commands issued in another
       session. In particular, a client MUST NOT assume that messages with the
       \Deleted flag will not be expunged simply because that client 
did not issue
       an EXPUNGE or CLOSE.

       Upon receipt of an EXPUNGE response, the message sequence number...

Now, we haven't issued any unenforceable directives. We've just very 
specifically defined the semantics of \Deleted and the EXPUNGE 
response. No MUSTs or SHOULDs; just explicit definition of the 
semantics.

pr

-- 
Pete Resnick <mailto:[email-masked]>
Eudora Engineering - QUALCOMM Incorporated
Ph: (217)337-6377 or (858)651-4478, Fax: (858)651-1102

Date: Fri, 19 Jan 2001 15:29:21 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: re: Delete-Expunge (Was: Move?)
To: Pete Resnick <[email-masked]>
cc: [email-masked]
Sender: [email-masked]

On Fri, 19 Jan 2001 14:43:12 -0600, Pete Resnick wrote:
> >Unfortunately, I do not believe that it is as easy as telling client
> >writers "servers are only allowed to expunge when they are explicitly told
> >to do so."
> I can tell you for us, if we had the statement in the RFC, we would
> do it. It would actually turn out to be less work for us to do it
> only the "IMAP way".

Please believe me when I say that I would like very much to make such a
statement, especially if it induces Qualcomm to make Eudora only use the
delete-expunge mechanism.

My problem is that I don't think that I can make such a statement, or at least
I don't think that I can make it a MUST.

Would you settle for a SHOULD?

For example:
	A server SHOULD NOT unilaterally expunge deleted messages from
	a mailbox unless it receives an EXPUNGE or CLOSE command from
	a client session that has that mailbox selected.  The server
	SHOULD retain deleted messages, with the \Deleted flag set,
	across multiple sessions until an explicit EXPUNGE or CLOSE is
	issued.

	Note that an EXPUNGE or CLOSE in one session MAY cause the
	messages to be removed in all sessions which have that mailbox
	selected.  In particular, a client MUST NOT assume that
	messages will not be expunged unless it issues an EXPUNGE or
	CLOSE.

> The
> only sense in which this is enforceable is that servers which fail to
> follow the directive will behave badly and will get beat up on.

Exactly, but I believe that in this case we are talking BCP and SHOULD, not
MUST.

> Similarly, we can say (if we want) that a successful STORE of a
> \Deleted flag for a message doesn't mean "Mark this for deletion and
> expunge it whenever you feel like it", but rather "Mark this for
> deletion and expunge it when you get an EXPUNGE or CLOSE command."

Historically, the former was the original intent; but the latter is what
everybody seems to have implemented.  There hasn't been much call for a long
time for systemwide auto-EXPUNGE when the disk is full.  Although who knows,
maybe someone will want to do that (ref. certain other messages in this
thread).

> 1. They performed the action in their client which called the EXPUNGE
> or CLOSE command.
> 2. Someone else who shared the mailbox did an EXPUNGE or CLOSE command.
> 3. Their server is not following the IMAP standard.
> We all have to code (and document to the user) dealing with things of
> type 1 and 2. Things of type 3 we can dismiss.

Except that, all the time, we have people who defy the standards with such
statements as "the standards are no good if they get in the way of doing
work."  Just the other day, I had someone wanting me to add "AW:" as an
alternative to "re:" because some broken software in Germany writes it.

> Indeed, by the same people that I initially asked about this
> behavior, I was told that any server that did would be soundly
> beat-up on by the community.

In my observation, practices which are "soundly beat-up on by the community"
become "widely accepted" just a few years later; and such practices that are
enshrined in standards result in the standards being defied or criticized.

If I am so unfortunate as to have to still be defending stipulations in IMAP
10 years from now, I need stronger consensus and support than this.  I've been
burnt a lot by this in the past, and I don't want

> My argument
> is that if we already believe auto-EXPUNGE is prohibited behavior, we
> should document that.

I don't consider it prohibited.  I consider it bad, unfortunate, and totally
unsuitable for any software I would use.  But compliant client software
handles it just fine, and thus I see the issue as one of implementor morality
rather than harm.

Let me ask you again; would SHOULD type wording such as that above be
satisfactory for your purposes?  You're invited to suggest any wordsmithing.
Also, does anyone object to text such as this?

Date: Fri, 19 Jan 2001 14:43:12 -0600
To: Mark Crispin <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: re: Delete-Expunge (Was: Move?)
Cc: [email-masked]
Sender: [email-masked]

On 1/19/01 at 9:44 AM -0800, Mark Crispin wrote:

>Unfortunately, I do not believe that it is as easy as telling client writers
>"servers are only allowed to expunge when they are explicitly told to do so."

I can tell you for us, if we had the statement in the RFC, we would 
do it. It would actually turn out to be less work for us to do it 
only the "IMAP way".

>Yes, "it is compliant behavior according to the IMAP specification 
>for a server to expunge messages which are marked \Deleted even if 
>it receives no EXPUNGE or CLOSE command and even if the mailbox is 
>not shared."  It is compliant because it is not explicitly 
>forbidden.  It is not forbidden because such a prohibition is 
>unenforceable.

Let's start from the end: unenforceability. Take the case of SMTP. 
RFC 1123 says:

       5.3.3  Reliable Mail Receipt

          When the receiver-SMTP accepts a piece of mail (by sending a
          "250 OK" message in response to DATA), it is accepting
          responsibility for delivering or relaying the message.  It must
          take this responsibility seriously, i.e., it MUST NOT lose the
          message for frivolous reasons, e.g., because the host later
          crashes or because of a predictable resource shortage.

Now you could say, "This requirement is unenforceable; a client can't 
tell the difference between a server that loses the message for 
frivolous reasons rather than for good reasons. Therefore, this 
shouldn't have ever been explicitly forbidden." That's nonsense. The 
only sense in which this is enforceable is that servers which fail to 
follow the directive will behave badly and will get beat up on. We're 
not defining syntactic requirements here which are visible from the 
wire; we're defining semantic requirements. A successful DATA command 
doesn't mean "this message has been accepted if I feel like it". It 
is defined as meaning "this message has been accepted and I won't 
lose it for frivolous reasons."

Similarly, we can say (if we want) that a successful STORE of a 
\Deleted flag for a message doesn't mean "Mark this for deletion and 
expunge it whenever you feel like it", but rather "Mark this for 
deletion and expunge it when you get an EXPUNGE or CLOSE command."

>How can a client know that an unexpected EXPUNGE of messages 
>occurred because of the server's capricious behavior, as opposed to 
>some other client having issued an EXPUNGE?  It can't.

Absolutely irrelevant. A client can't tell the difference and must be 
coded to accept untagged EXPUNGE responses. But if the requirement 
were in the document, the user could be guaranteed that if they mark 
a message for deletion (i.e., put it in the virtual "Trash"), it 
won't be expunged unless:

1. They performed the action in their client which called the EXPUNGE 
or CLOSE command.
2. Someone else who shared the mailbox did an EXPUNGE or CLOSE command.
3. Their server is not following the IMAP standard.

We all have to code (and document to the user) dealing with things of 
type 1 and 2. Things of type 3 we can dismiss. We can't crash or do 
something stupid, but we can sure tell our users, "Sorry, your server 
is broken. We're not going to write some special Trash mailbox code 
to fix it."

>I don't know of any server that auto-EXPUNGEs...

Indeed, by the same people that I initially asked about this 
behavior, I was told that any server that did would be soundly 
beat-up on by the community. That fact ought to be documented.

>but there's probably someone out there who would consider it 
>capricious to prohibit server auto-EXPUNGE. Observing that all 
>clients are required to handle the event anyway, he'll just ignore 
>the prohibition.

Implementors ignore countless prohibitions in standards now. That's 
utterly irrelevant too. That a prohibition exists defines what the 
community thinks are the standards of protocol behavior. My argument 
is that if we already believe auto-EXPUNGE is prohibited behavior, we 
should document that. We have to code around stupidity of some broken 
implementations, but having the ability to call something broken goes 
a long way.

If we all believe that auto-EXPUNGE is wrong, please document that.

pr

-- 
Pete Resnick <mailto:[email-masked]>
Eudora Engineering - QUALCOMM Incorporated
Ph: (217)337-6377 or (858)651-4478, Fax: (858)651-1102

Date: Fri, 19 Jan 2001 10:28:37 -0600
To: Mark Crispin <[email-masked]>
From: Pete Resnick <[email-masked]>
Subject: Delete-Expunge (Was: Move?)
Cc: [email-masked]
Sender: [email-masked]

On 1/18/01 at 1:33 PM -0800, Mark Crispin wrote:

>You are confusing a user interface (and server storage) issue with a 
>protocol issue.  The client can present a "Trash" view to the user, 
>while still using the IMAP delete-expunge model.  It should; and 
>most clients do.

While you're trying to get the IMAP rev to last call, this might be a 
good time to ask:

There is a reason client writers provide a "Trash" mailbox mode 
instead of the IMAP delete-expunge mode. (Eudora, BTW, provides both 
modes.) I have heard the claim that it is compliant behavior 
according to the IMAP specification for a server to expunge messages 
which are marked \Deleted even if it receives no EXPUNGE or CLOSE 
command and even if the mailbox is not shared. Now, I carefully went 
through imapv-14 yesterday looking for support for this claim. There 
are several notes about untagged EXPUNGE responses that I might 
interpret to mean this, but there was no direct note one way or the 
other.

If it is true that servers can expunge at will, then there is no way 
to achieve the semantics of a "Trash" mailbox with the IMAP 
delete-expunge model: There is no way to say, "Mark this message for 
deletion and permanently delete it only when I tell you to." If the 
widespread belief is not true, and servers (barring nasty error 
conditions) are only allowed to expunge when they are explicitly told 
to, that should be clarified in the spec. I firmly believe that with 
such assurance, clients will stop playing this "Trash" mailbox game.

pr

-- 
Pete Resnick <mailto:[email-masked]>
Eudora Engineering - QUALCOMM Incorporated
Ph: (217)337-6377 or (858)651-4478, Fax: (858)651-1102

Date: Fri, 19 Jan 2001 09:44:40 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: re: Delete-Expunge (Was: Move?)
To: Pete Resnick <[email-masked]>
cc: [email-masked]
Sender: [email-masked]

Unfortunately, I do not believe that it is as easy as telling client writers
"servers are only allowed to expunge when they are explicitly told to do so."
I have had client writers tell me in no uncertain terms that they use a
"Trash" mailbox because it's less work for them to program it that way.

I don't see that situation changing, because "less work for me" is a powerful
motivator.  That is what is behind the demands for message MOVE and DELETE
operators.

Yes, "it is compliant behavior according to the IMAP specification for a
server to expunge messages which are marked \Deleted even if it receives no
EXPUNGE or CLOSE command and even if the mailbox is not shared."  It is
compliant because it is not explicitly forbidden.  It is not forbidden because
such a prohibition is unenforceable.

How can a client know that an unexpected EXPUNGE of messages occurred because
of the server's capricious behavior, as opposed to some other client having
issued an EXPUNGE?  It can't.

It's clearly possible for another client to issue an EXPUNGE.  Quite a bit of
protocol work was expended on stipulating when an unsolicited EXPUNGE can
happen.  None of that would have been necessary if shared EXPUNGE was
prohibited (as shared access is in POP3); but it was clear that people wanted
shared EXPUNGE.

I don't know of any server that auto-EXPUNGEs, but there's probably someone
out there who would consider it capricious to prohibit server auto-EXPUNGE.
Observing that all clients are required to handle the event anyway, he'll just
ignore the prohibition.

Date: Fri, 19 Jan 2001 12:20:14 +0000
From: John Haxby <[email-masked]>
Organization: OpenMail R&D
Cc: IMAP Extensions WG <[email-masked]>
Subject: Re: Move?
Sender: [email-masked]

This is a multi-part message in MIME format.
--------------86BF67392CCA2C8ABE2F11C2
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Mark Crispin wrote:

> On Thu, 18 Jan 2001 15:12:59 -0800 (PST), Mark Crispin wrote:
> >       1) maximum message size
> >       2) maximum overall long-term storage use
> >       3) maximum transient (short-term) storage use
> >
> > The point being that the second and third values should be quite different.
>
> Another point needs to be made about the first value.  The need for it is to
> prevent a denial of service problem.  It should be impossible for a single
> message to push a user near his second limit to the third limit.

While modifying the server quota model would permit this, you then run into a
selection of interesting extra problems.   For example, how long does the
transient quota last?  Until the next logout?  For five minutes? Until all
current sessions have logged out?   Does (or can) the server know about this
last state?  What if the client crashes half way through the copy/mark/expunge
(as they often do :-) you're then left badly over quota and, depending on the
implementation, this may mean that messages to the user might be rejected.  You
might wind up with two copies of the message because the mark didn't work and
you don't know, then whether the copy is corrupt ...

Suppose, further, you have a Very Large Message, slightly over half the
available free space in the message store.   You cannot copy that, regardless of
any quota systems (fortunately, the better message stores copy references which
neatly gets around this).

Quite apart from the difficulties of designing, yet alone implementing, a quota
scheme that allows copy to work, there are the various temporal difficulties of
not having an atomic move operation in the protocol.  I've mentioned the
crashing one.   There's also the problem of multiple client access (which, as
I've said, is partially addressed by UID+, partially because it requires all
clients to cooperate and use UID+).

The quota problem is really not going be solved without MOVE and/or
DELETE (message) in the protocol, but then there's the slightly harder problem
of getting users to deploy a client that actually uses the extension ... and
this should never be underestimated.   (Various people, Mark mostly, have often
suggested that the client should be fixed to avoid a particular problem, but
have failed to realise that getting, say, 110,000 users to do *anything* is well
nigh impossible.)

jch

--------------86BF67392CCA2C8ABE2F11C2
Content-Type: text/x-vcard; charset=us-ascii;
 name="jch.vcf"
Content-Transfer-Encoding: 7bit
Content-Description: Card for John Haxby
Content-Disposition: attachment;
 filename="jch.vcf"

begin:vcard 
n:Haxby;John
tel;fax:+44 1344 763686
tel;work:+44 1344 763711
x-mozilla-html:FALSE
url:https://ecardfile.com/id/jch
org:OpenMail R&D
adr:;;Hewlett Packard<br>Nine Mile Ride;Wokingham;Berks;RG40 3LL;United Kingdom
version:2.1
email;internet:[email-masked]
note;quoted-printable:<em>OpenMail for All!</em>&nbsp=3B<img src=3D"http://www.openmail.com/cyc/om/00/graphics/omlinux.jpg" width=3D53 height=3D62 align=3Dbottom>
x-mozilla-cpt:;25408
fn:John Haxby
end:vcard

--------------86BF67392CCA2C8ABE2F11C2--

Date: Fri, 19 Jan 2001 12:04:51 +0000
From: John Haxby <[email-masked]>
Organization: OpenMail R&D
To: Edward Hibbert <[email-masked]>
Cc: [email-masked]
Subject: Re: Move?
Sender: [email-masked]

This is a multi-part message in MIME format.
--------------6DA3E24986CBE1C5325A9D95
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Edward Hibbert wrote:

> Seems to me that deprecating clients which behave like this isn't going to
> be effective as a method of resolving the problem long term - as a user I
> find the Deleted Items folder much more useful than having messages in the
> same folder marked as deleted cluttering up the view.  Opinions clearly vary
> but I think there is a body of support for this approach and for the
> forseeable future there are going to be clients which have this model.
>

I agree.  The trouble with forcing any one model is that different users prefer
different things -- some like mark/delete, some like move to wastebasket, and
some like to just delete.   This leads users to choose specific clients, or
configure clients like Netscape in a particular way.   It seems rather odd that
the IMAP protocol should dictate user preferences ...

>
> Perhaps we should look at an extension for MOVE, though I'm not sure I'm
> competent to draft one.  I know Mark objects to this - what about other
> folk?
>

I'm not competant either.   To draft an extension, I mean.   However, I'll
happily review any extension that would provide MOVE (and preferably its
companion, DELETE).

jch

--------------6DA3E24986CBE1C5325A9D95
Content-Type: text/x-vcard; charset=us-ascii;
 name="jch.vcf"
Content-Transfer-Encoding: 7bit
Content-Description: Card for John Haxby
Content-Disposition: attachment;
 filename="jch.vcf"

begin:vcard 
n:Haxby;John
tel;fax:+44 1344 763686
tel;work:+44 1344 763711
x-mozilla-html:FALSE
url:https://ecardfile.com/id/jch
org:OpenMail R&D
adr:;;Hewlett Packard<br>Nine Mile Ride;Wokingham;Berks;RG40 3LL;United Kingdom
version:2.1
email;internet:[email-masked]
note;quoted-printable:<em>OpenMail for All!</em>&nbsp=3B<img src=3D"http://www.openmail.com/cyc/om/00/graphics/omlinux.jpg" width=3D53 height=3D62 align=3Dbottom>
x-mozilla-cpt:;25408
fn:John Haxby
end:vcard

--------------6DA3E24986CBE1C5325A9D95--

Date: Thu, 18 Jan 2001 15:39:43 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: RE: Move?
To: IMAP Extensions WG <[email-masked]>
Sender: [email-masked]

On Thu, 18 Jan 2001 15:12:59 -0800 (PST), Mark Crispin wrote:
> 	1) maximum message size
> 	2) maximum overall long-term storage use
> 	3) maximum transient (short-term) storage use
>
> The point being that the second and third values should be quite different.

Another point needs to be made about the first value.  The need for it is to
prevent a denial of service problem.  It should be impossible for a single
message to push a user near his second limit to the third limit.

Consequently, Q3 > Q2, Q3-Q2 > Q1

Date: Thu, 18 Jan 2001 15:12:59 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: RE: Move?
To: Edward Hibbert <[email-masked]>
cc: "'Larry Osterman'" <[email-masked]>, Caleb Clausen <[email-masked]>, [email-masked]
Sender: [email-masked]

On Thu, 18 Jan 2001 22:40:12 -0000, Edward Hibbert wrote:
> So, how is one to allow a client MOVE operation - which feels to me as
> though it should work as long as you are below your quota - and
> simultaneously not violate server quota limits?  Violating quota limits is
> anathema to some of our ISP customers.

Today, a client has to do COPY-STORE \DELETED.  I contend that this is
correct, since, unlike the proposed MOVE, it is reversible.  This is
important, not just from the perspective of allowing the user to "undo".

Specifically, if the MOVE operation fails for any reason -- such as a quota
exceeded condition at the target -- the state prior to the MOVE has to be
restored.  You can't do this unless the mail store implements MOVE of multiple
messages as an atomic operation; furthermore that it must lock the source
mailbox against any other operations until the MOVE is complete (especially
append of new messages or EXPUNGE of the moved messages); and on top of that
MOVE must guarantee that it can reverse without quota problems (suppose the
source mailbox is currently over quota).

A further complication exists with any other client which has the mailbox
selected at the time a MOVE is going on; that client can not get the untagged
EXPUNGE until it does a "safe" command.  The question then becomes about how
you maintain the necessary zombie state (which probably requires leaving
ghosts of the message behind...hence not solving the quota problem); since
MOVE forces an EXPUNGE the server can't take the easy out of no-op'ing the
EXPUNGE until such time as there is exclusive access.

These entail a lot of difficult requirements for MOVE, and ones that are
likely to be ignored by careless implementors who would just do a bunch of
rename() calls.  The result would be worse than the original problem.

A better solution is to reconsider your assumptions behind quotas; and in
particular the idea that a user's permanent space allotment should equal his
transient space allotment.  I spent many years in a past life (1973-1984)
dealing with file quotas in which users were constantly at the limit of their
quota.  Far more money was spent in human salaries than would have been
expended by buying more disk -- and that was at a time when disks were
expensive!

More to the point: a user with a 5MB quota should not be receiving 4.5MB MIME
godzillagrams.  Either his quota is much too low, or his expectations of what
size messages he can receive are too high.  A reasonable quota facility needs
at least three values:
	1) maximum message size
	2) maximum overall long-term storage use
	3) maximum transient (short-term) storage use

The point being that the second and third values should be quite different.

From: Edward Hibbert <[email-masked]>
To: "'Mark Crispin'" <[email-masked]>
Cc: "'Larry Osterman'" <[email-masked]>, Caleb Clausen <[email-masked]>, [email-masked]
Subject: RE: Move?
Date: Thu, 18 Jan 2001 22:40:12 -0000
Sender: [email-masked]

Mark Crispin wrote:

> Leaving aside the quota problem, 

But this is precisely the issue I want to address.  We can argue all day
about whether Trash folders are a good idea and how clients can best
represent them.  Let's not do that because it's emotive and outside the
scope of the server.  And I implement servers so I'm not too interested in
it.

Instead, let's concentrate on the quota issue, which applies just as much in
moving mail from one folder to another.  I'm assuming this is an operation
that most clients do.

So, how is one to allow a client MOVE operation - which feels to me as
though it should work as long as you are below your quota - and
simultaneously not violate server quota limits?  Violating quota limits is
anathema to some of our ISP customers.

One way round the problem would be to say that the quota should apply to
individual folders rather than mailboxes (apologies for the violation of
IMAP terminology here).  But that seems artificial to me, and more
irritating to users.

So I don't see a way of solving this without a MOVE extension.  I'm not
arguing that this is the most serious problem in the world, and I'm prepared
to be convinced that it's not worth doing anything about except perhaps
adding something to the implementation notes.  But I'd like to see whether
I'm missing some way of solving this, or whether there's support for a MOVE
extension.

Regards,

Edward.

Regards,

Edward.

From: Lyndon Nerenberg <[email-masked]>
To: Edward Hibbert <[email-masked]>
cc: [email-masked]
Subject: Re: Move? 
Organization: ACI Worldwide
Date: Thu, 18 Jan 2001 14:56:40 -0700
Sender: [email-masked]

> Seems to me that deprecating clients which behave like this isn't going to
> be effective as a method of resolving the problem long term - as a user I
> find the Deleted Items folder much more useful than having messages in the
> same folder marked as deleted cluttering up the view.

So fix the GUI interface. There is nothing on the planet that
says a message marked \Deleted cannot be displayed by the client as if
it was in a different folder (called Trash or whatever else you want
to call it). This is *not* rocket science. (Simeon had functionality
in it similar to this five years ago.)

--lyndon

Date: Thu, 18 Jan 2001 13:33:46 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: RE: Move?
To: Edward Hibbert <[email-masked]>
cc: "'Larry Osterman'" <[email-masked]>, Caleb Clausen <[email-masked]>, [email-masked]
Sender: [email-masked]

> as a user I
> find the Deleted Items folder much more useful than having messages in the
> same folder marked as deleted cluttering up the view.  Opinions clearly vary
> but I think there is a body of support for this approach and for the
> forseeable future there are going to be clients which have this model.

You are confusing a user interface (and server storage) issue with a protocol
issue.  The client can present a "Trash" view to the user, while still using
the IMAP delete-expunge model.  It should; and most clients do.  Doing so also
neatly avoids the quota problem.

Leaving aside the quota problem, as a protocol operation, the Trash model is
fatally flawed -- unlike deleted status, it is impossible to reverse a "Trash"
operation.  The only thing you can do is add a new message (with a new UID!)
to the original mailbox containing the message in question.

Yes, it is highly unfortunate that one commonly-used client made the wrong
choice.  That program is disproportionately represented among my incoming
support question, due to this and certain other unfortunate design decisions.
We all have to make the best of it, but I don't think that breaking IMAP's
fundamentals is a good idea.

> Perhaps we should look at an extension for MOVE, though I'm not sure I'm
> competent to draft one.  I know Mark objects to this - what about other
> folk?

Let me emphasize:

I oppose the standardization of any extension which creates an independent
means from EXPUNGE to remove a message from a mailbox.  This is a very serious
violation of the IMAP architecture.

Even if this extension is promulgated over my objections, there is no reason
to believe that it will solve your problem.  It presupposes widespread
implementation, including in the particular client that you're purposing to
convenience -- assuming there is still any development in that program.

From: Edward Hibbert <[email-masked]>
To: "'Larry Osterman'" <[email-masked]>, Caleb Clausen <[email-masked]>
Cc: [email-masked]
Subject: RE: Move?
Date: Thu, 18 Jan 2001 20:33:08 -0000
Sender: [email-masked]

That's precisely right.  We do already make a reference when a COPY is done
(and for multirecipient messages), but if you don't charge that copy against
the quota then you have the problem of which of the messages actually holds
the size and therefore which one the user has to delete to free up the
space.  I guess you could split the charged size across the number of copies
and recalculate it each time you copy, but that gets silly, and as Larry
says, you end up with users deleting messages and wondering why their
mailbox is still full.

Another point to note is that references themselves consume some space - for
us the minimal message size, because they do have to have their own flags -
and so ought to be charged against the quota.

Seems to me that deprecating clients which behave like this isn't going to
be effective as a method of resolving the problem long term - as a user I
find the Deleted Items folder much more useful than having messages in the
same folder marked as deleted cluttering up the view.  Opinions clearly vary
but I think there is a body of support for this approach and for the
forseeable future there are going to be clients which have this model.

Perhaps we should look at an extension for MOVE, though I'm not sure I'm
competent to draft one.  I know Mark objects to this - what about other
folk?

Regards,

Edward.

-----Original Message-----
From: Larry Osterman [mailto:[email-masked]]
Sent: Thursday, January 18, 2001 8:11 PM
To: Caleb Clausen; Edward Hibbert
Cc: [email-masked]
Subject: RE: Move?

It's possible that Edwards' data store is like the Exchange store, which
natively implements this (we call it "Single Instance Storage").  The
problem is that in order to make the quota management work correctly,
the store needs to charge each instance against the quota - this means
that the amount of quota charged against a particular user is greater
than the sum of the storage they are using, however if we don't do it
this way, there is no way for a client to determine how their quota is
being used - users get REALLY annoyed if they delete an item to get
below their quota only to discover that it doesn't help them.

-----Original Message-----
From: Caleb Clausen [mailto:[email-masked]]
Sent: Thursday, January 18, 2001 11:08 AM
To: Edward Hibbert
Cc: [email-masked]'
Subject: RE: Move?

From:           	Edward Hibbert <[email-masked]>
> My motivation for a MOVE is not based on an infatuation with OS
commands
> :-).  I have a nice transactional database with rollback and all that
good
> stuff.
> 
> Instead it's about quota.  At the moment we enforce the quota limit on
the
> COPY command.  We have come up against a client which maintains a
Deleted
> Items folder (mailbox in IMAP terminology), so it moves items into it
when
> the user deletes them.  To do the move it does a COPY/DELETE/EXPUNGE.

> 
> But this process requires that you have as much free quota as the size
of
> the message you're trying to move - which is obviously not possible if
the
> message is itself more than half your quota limit.
> 
> Changing the client is not really an option, so I'm left with couple
of
> ideas, neither of which I like very much:
> - don't check quota on COPY
> - hardcode the Deleted Items folder name and don't check quota if
we're
> copying to there.
> 
> Obviously with a MOVE command it would be clear that you weren't
trying to
> violate your quota.  It wouldn't help me just now because I can't
change the
> client to use it, but it seems to me that this is an argument in
favour of
> it.
> 
> Do any of you folk have any other ideas as to how to solve this?

well, here's an idea. use reference counts internally on your 
messages. that way, you wouldn't have to have room for an extra 
message when the user does a COPY, just an extra reference.

i haven't given this a lot of thought; i don't know how hard it would 
be to implement. one thing that springs to mind is the flags; each 
reference would have to have it's own set of them.

"For, I say fortunately, I always carry a spare set 
of feathers."
  --Foghorn Leghorn

content-class: urn:content-classes:message
Subject: RE: Move?
Date: Thu, 18 Jan 2001 12:10:56 -0800
From: "Larry Osterman" <[email-masked]>
To: "Caleb Clausen" <[email-masked]>, "Edward Hibbert" <[email-masked]>
Cc: <[email-masked]>
Sender: [email-masked]

It's possible that Edwards' data store is like the Exchange store, which
natively implements this (we call it "Single Instance Storage").  The
problem is that in order to make the quota management work correctly,
the store needs to charge each instance against the quota - this means
that the amount of quota charged against a particular user is greater
than the sum of the storage they are using, however if we don't do it
this way, there is no way for a client to determine how their quota is
being used - users get REALLY annoyed if they delete an item to get
below their quota only to discover that it doesn't help them.

-----Original Message-----
From: Caleb Clausen [mailto:[email-masked]]
Sent: Thursday, January 18, 2001 11:08 AM
To: Edward Hibbert
Cc: [email-masked]'
Subject: RE: Move?

From:           	Edward Hibbert <[email-masked]>
> My motivation for a MOVE is not based on an infatuation with OS
commands
> :-).  I have a nice transactional database with rollback and all that
good
> stuff.
> 
> Instead it's about quota.  At the moment we enforce the quota limit on
the
> COPY command.  We have come up against a client which maintains a
Deleted
> Items folder (mailbox in IMAP terminology), so it moves items into it
when
> the user deletes them.  To do the move it does a COPY/DELETE/EXPUNGE.

> 
> But this process requires that you have as much free quota as the size
of
> the message you're trying to move - which is obviously not possible if
the
> message is itself more than half your quota limit.
> 
> Changing the client is not really an option, so I'm left with couple
of
> ideas, neither of which I like very much:
> - don't check quota on COPY
> - hardcode the Deleted Items folder name and don't check quota if
we're
> copying to there.
> 
> Obviously with a MOVE command it would be clear that you weren't
trying to
> violate your quota.  It wouldn't help me just now because I can't
change the
> client to use it, but it seems to me that this is an argument in
favour of
> it.
> 
> Do any of you folk have any other ideas as to how to solve this?

well, here's an idea. use reference counts internally on your 
messages. that way, you wouldn't have to have room for an extra 
message when the user does a COPY, just an extra reference.

i haven't given this a lot of thought; i don't know how hard it would 
be to implement. one thing that springs to mind is the flags; each 
reference would have to have it's own set of them.

"For, I say fortunately, I always carry a spare set 
of feathers."
  --Foghorn Leghorn

Date: Thu, 18 Jan 2001 11:23:08 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: RE: Move?
To: Caleb Clausen <[email-masked]>
cc: Edward Hibbert <[email-masked]>, "[email-masked]'" <[email-masked]>
Sender: [email-masked]

On Thu, 18 Jan 2001 11:07:32 -0800, Caleb Clausen wrote:
> well, here's an idea. use reference counts internally on your
> messages. that way, you wouldn't have to have room for an extra
> message when the user does a COPY, just an extra reference.

That's an interesting point.  link() is a perfectly valid way of implementing
COPY for the message data since it is static.  Parts of the metadata --
INTERNALDATE, RFC822.SIZE, ENVELOPE, BODYSTRUCTURE -- can also be lunk.
However, each mailbox needs its own copy of other metadata, most notably FLAGS
and UID.

If the underlying problem is that users can't clean their mail due to quotas
and Netscape's use of a "Trash" mailbox, I suggest that the problem is in
Netscape and not in any IMAP server.  The Trash mailbox is a Netscape-ism and
is not sanctioned by IMAP in any way.  Clients which use IMAP the way IMAP was
intended to use don't have this problem.

You may be able to work around the problem by applying separate quotas to the
Trash folder; but IMHO it's also a good answer to say "use another client."

From: "Caleb Clausen" <[email-masked]>
To: Edward Hibbert <[email-masked]>
Date: Thu, 18 Jan 2001 11:07:32 -0800
Subject: RE: Move?
CC: "[email-masked]'" <[email-masked]>
Sender: [email-masked]

From:           	Edward Hibbert <[email-masked]>
> My motivation for a MOVE is not based on an infatuation with OS commands
> :-).  I have a nice transactional database with rollback and all that good
> stuff.
> 
> Instead it's about quota.  At the moment we enforce the quota limit on the
> COPY command.  We have come up against a client which maintains a Deleted
> Items folder (mailbox in IMAP terminology), so it moves items into it when
> the user deletes them.  To do the move it does a COPY/DELETE/EXPUNGE.  
> 
> But this process requires that you have as much free quota as the size of
> the message you're trying to move - which is obviously not possible if the
> message is itself more than half your quota limit.
> 
> Changing the client is not really an option, so I'm left with couple of
> ideas, neither of which I like very much:
> - don't check quota on COPY
> - hardcode the Deleted Items folder name and don't check quota if we're
> copying to there.
> 
> Obviously with a MOVE command it would be clear that you weren't trying to
> violate your quota.  It wouldn't help me just now because I can't change the
> client to use it, but it seems to me that this is an argument in favour of
> it.
> 
> Do any of you folk have any other ideas as to how to solve this?

well, here's an idea. use reference counts internally on your 
messages. that way, you wouldn't have to have room for an extra 
message when the user does a COPY, just an extra reference.

i haven't given this a lot of thought; i don't know how hard it would 
be to implement. one thing that springs to mind is the flags; each 
reference would have to have it's own set of them.

"For, I say fortunately, I always carry a spare set 
of feathers."
  --Foghorn Leghorn

Date: Thu, 18 Jan 2001 12:46:42 -0500
From: Ken Murchison <[email-masked]>
Organization: Oceana Matrix Ltd.
To: Edward Hibbert <[email-masked]>
CC: "[email-masked]'" <[email-masked]>
Subject: Re: Move?
Sender: [email-masked]

Edward Hibbert wrote:
> 
> My motivation for a MOVE is not based on an infatuation with OS commands
> :-).  I have a nice transactional database with rollback and all that good
> stuff.
> 
> Instead it's about quota.  At the moment we enforce the quota limit on the
> COPY command.  We have come up against a client which maintains a Deleted
> Items folder (mailbox in IMAP terminology), so it moves items into it when
> the user deletes them.  To do the move it does a COPY/DELETE/EXPUNGE.
> 
> But this process requires that you have as much free quota as the size of
> the message you're trying to move - which is obviously not possible if the
> message is itself more than half your quota limit.
> 
> Changing the client is not really an option, so I'm left with couple of
> ideas, neither of which I like very much:
> - don't check quota on COPY
> - hardcode the Deleted Items folder name and don't check quota if we're
> copying to there.
> 
> Obviously with a MOVE command it would be clear that you weren't trying to
> violate your quota.  It wouldn't help me just now because I can't change the
> client to use it, but it seems to me that this is an argument in favour of
> it.
> 
> Do any of you folk have any other ideas as to how to solve this?

We had a similar problem here with people using Communicator with a
Trash folder.  If they ran up against their quota limit, they couldn't
delete messages for the reason you describe above.

I solved it by placing a separate QUOTAROOT (via Cyrus imapd) on the
Trash folder so "deleted" items dont count against their "main" quota.

Ken
-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

Date: Thu, 18 Jan 2001 09:50:52 +0000
From: John Haxby <[email-masked]>
Organization: OpenMail R&D
To: Edward Hibbert <[email-masked]>
Cc: "[email-masked]'" <[email-masked]>
Subject: Re: Move?
Sender: [email-masked]

This is a multi-part message in MIME format.
--------------AD66766772BFAF0AAFAAFD54
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Edward Hibbert wrote:

> My motivation for a MOVE is not based on an infatuation with OS commands
> :-).  I have a nice transactional database with rollback and all that good
> stuff.
>
> Instead it's about quota.  At the moment we enforce the quota limit on the
> COPY command.  We have come up against a client which maintains a Deleted
> Items folder (mailbox in IMAP terminology), so it moves items into it when
> the user deletes them.  To do the move it does a COPY/DELETE/EXPUNGE.
>
> But this process requires that you have as much free quota as the size of
> the message you're trying to move - which is obviously not possible if the
> message is itself more than half your quota limit.

We have exactly this problem as well.   We just ignore the limitation.  At some
stage, someone will complain and then we might have to do something about it.
I have also given up on the idea of having a DELETE command as well.   (Quite
apart from the various race conditions in a mark/delete scheme, we get problems
with users who delete a message but since the client doesn't actually delete it
just yet, you don't drop below quota.   One of these days, someone is going to
decide that merely modifying messages when you are over quota is a bad move and
you won't be able to get below it again...)

jch, dispirited

--------------AD66766772BFAF0AAFAAFD54
Content-Type: text/x-vcard; charset=us-ascii;
 name="jch.vcf"
Content-Transfer-Encoding: 7bit
Content-Description: Card for John Haxby
Content-Disposition: attachment;
 filename="jch.vcf"

begin:vcard 
n:Haxby;John
tel;fax:+44 1344 763686
tel;work:+44 1344 763711
x-mozilla-html:FALSE
url:https://ecardfile.com/id/jch
org:OpenMail R&D
adr:;;Hewlett Packard<br>Nine Mile Ride;Wokingham;Berks;RG40 3LL;United Kingdom
version:2.1
email;internet:[email-masked]
note;quoted-printable:<em>OpenMail for All!</em>&nbsp=3B<img src=3D"http://www.openmail.com/cyc/om/00/graphics/omlinux.jpg" width=3D53 height=3D62 align=3Dbottom>
x-mozilla-cpt:;25408
fn:John Haxby
end:vcard

--------------AD66766772BFAF0AAFAAFD54--

From: Edward Hibbert <[email-masked]>
To: "'Mark Crispin'" <[email-masked]>
Cc: "[email-masked]'" <[email-masked]>
Subject: RE: Move?
Date: Thu, 18 Jan 2001 09:06:04 -0000
Sender: [email-masked]

My motivation for a MOVE is not based on an infatuation with OS commands
:-).  I have a nice transactional database with rollback and all that good
stuff.

Instead it's about quota.  At the moment we enforce the quota limit on the
COPY command.  We have come up against a client which maintains a Deleted
Items folder (mailbox in IMAP terminology), so it moves items into it when
the user deletes them.  To do the move it does a COPY/DELETE/EXPUNGE.  

But this process requires that you have as much free quota as the size of
the message you're trying to move - which is obviously not possible if the
message is itself more than half your quota limit.

Changing the client is not really an option, so I'm left with couple of
ideas, neither of which I like very much:
- don't check quota on COPY
- hardcode the Deleted Items folder name and don't check quota if we're
copying to there.

Obviously with a MOVE command it would be clear that you weren't trying to
violate your quota.  It wouldn't help me just now because I can't change the
client to use it, but it seems to me that this is an argument in favour of
it.

Do any of you folk have any other ideas as to how to solve this?

Regards,

Edward Hibbert
Internet Applications Group
Data Connection Ltd
Tel:	+44 131 662 1212		Fax:	+44 131 662 1345
Email:	[email-masked]	Web:	http://www.dataconnection.com

-----Original Message-----
From: Mark Crispin [mailto:[email-masked]]
Sent: Wednesday, January 17, 2001 9:52 PM
To: Edward Hibbert
Cc: [email-masked]'
Subject: re: Move?

On Wed, 17 Jan 2001 21:47:16 -0000, Edward Hibbert wrote:
> There's a COPY command to copy an item from one mailbox to another.  But
so
> far as I'm aware there isn't a MOVE command, so a client has to do a COPY
> and then a DELETE (and possibly an EXPUNGE if it wants the item to
disappear
> from its view of the current folder, depending on how it presents this).

That is correct.

> Is there any particular reason why there isn't a MOVE command?

You just gave the reason: the functionality of a MOVE command can be done
with
the existing commands.  It also preserves the IMAP delete-expunge model in
which it is impossible to remove a message from a mailbox in a single step.

The obvious temptation for implementations which have a one-file/one-message
form of mail store would be to implement MOVE via a rename() system call;
it's
also the primary motivation for asking for it (the secondary reason is to
eliminate the additional RTT).  That violates the IMAP delete-expunge model.

An additional problem which is often overlooked with rename() is that COPY
requires that the operation succeed or fail atomicly.  There is no atomic
rename(); which means that if a rename() fails (e.g. due to disk quota in
the
target) the previous messages have to be rename()d back -- which could also
fail!

The bottom line here is that the rename() idea is an "attractive nuisance";
one that looks desirable on the surface but has serious problems.  Since
these
problems are not necessarily obvious, affirmative measures need to be in
place
to prevent it.

For MOVE, such affirmative measures would require the preservation of a
deleted copy in the source mailbox; hence resulting in the exact equivalent
of
COPY+STORE \DELETED.  This could be done, but such a command would just be
protocol bloat.

Date: Wed, 17 Jan 2001 13:52:09 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: re: Move?
To: Edward Hibbert <[email-masked]>
cc: "[email-masked]'" <[email-masked]>
Sender: [email-masked]

On Wed, 17 Jan 2001 21:47:16 -0000, Edward Hibbert wrote:
> There's a COPY command to copy an item from one mailbox to another.  But so
> far as I'm aware there isn't a MOVE command, so a client has to do a COPY
> and then a DELETE (and possibly an EXPUNGE if it wants the item to disappear
> from its view of the current folder, depending on how it presents this).

That is correct.

> Is there any particular reason why there isn't a MOVE command?

You just gave the reason: the functionality of a MOVE command can be done with
the existing commands.  It also preserves the IMAP delete-expunge model in
which it is impossible to remove a message from a mailbox in a single step.

The obvious temptation for implementations which have a one-file/one-message
form of mail store would be to implement MOVE via a rename() system call; it's
also the primary motivation for asking for it (the secondary reason is to
eliminate the additional RTT).  That violates the IMAP delete-expunge model.

An additional problem which is often overlooked with rename() is that COPY
requires that the operation succeed or fail atomicly.  There is no atomic
rename(); which means that if a rename() fails (e.g. due to disk quota in the
target) the previous messages have to be rename()d back -- which could also
fail!

The bottom line here is that the rename() idea is an "attractive nuisance";
one that looks desirable on the surface but has serious problems.  Since these
problems are not necessarily obvious, affirmative measures need to be in place
to prevent it.

For MOVE, such affirmative measures would require the preservation of a
deleted copy in the source mailbox; hence resulting in the exact equivalent of
COPY+STORE \DELETED.  This could be done, but such a command would just be
protocol bloat.

From: Edward Hibbert <[email-masked]>
To: "[email-masked]'" <[email-masked]>
Subject: Move?
Date: Wed, 17 Jan 2001 21:47:16 -0000
Sender: [email-masked]

I'm sure this is going to turn out to be a stupid question.

There's a COPY command to copy an item from one mailbox to another.  But so
far as I'm aware there isn't a MOVE command, so a client has to do a COPY
and then a DELETE (and possibly an EXPUNGE if it wants the item to disappear
from its view of the current folder, depending on how it presents this).

Is there any particular reason why there isn't a MOVE command?

Edward Hibbert
Internet Applications Group
Data Connection Ltd
Tel:	+44 131 662 1212		Fax:	+44 131 662 1345
Email:	[email-masked]	Web:	http://www.dataconnection.com

Date: Fri, 5 Jan 2001 11:24:19 -0800 (PST)
From: Mark Crispin <[email-masked]>
Subject: Re: LAST CALL: IMAP base and MULTIAPPEND drafts
To: Simon Josefsson <[email-masked]>
cc: IMAP Interest List <[email-masked]>, IMAP Extensions WG <[email-masked]>
Sender: [email-masked]

On 05 Jan 2001 13:06:45 +0100, Simon Josefsson wrote:
> Suggestion, replace:
>       8-bit characters
>       are permitted in the message.
> with
>       8-bit characters
>       are permitted in the message body.
> since 8-bit characters are not permitted at all in the header section
> of a message, I think.

Nominally, this is a good idea.  However, I have not made this change, and
unless there is strong concensus for it I don't intend to do so, for the
following reasons:

 1) it would require that both the IMAP base (which has similar wording) and
IMAP MULTIAPPEND documents be revised if 8-bit characters are ever allowed in
the header.  I'm thinking of the time when all the world is Unicode and we can
dispense with all other charsets, encoded words, etc.

 2) There is also the question of allowing APPEND for messages that don't
necessarily conform to RFC 822; this is particularly an issue with drafts.

There seems to be a concensus (which I share) that IMAP servers should not
attempt to do message format validation on APPEND because of the issue with
drafts.  However, if such validation is done, I feel that the specification of
the validation should be in the message format document (RFC 822 or successor)
and not in an IMAP document.  As far as the IMAP document is concerned, the
APPEND text is simply a literal.

> Also, regarding UIDPLUS interactions, I'd suggest replacing
>    That is, the APPENDUID response code returns as many UIDs as there
>    were messages appended in the multiple append.
> with
>    That is, the APPENDUID response code returns as many UIDs as there
>    were messages appended in the multiple append.  The UIDs returned
>    should be in the order the articles where appended.  The message
>    set may not contain extraneous UIDs or the symbol '*'.
> to mimic some of the text from RFC 2359 regarding `set's and COPYUID.

This is a good idea and I've made the change.

To: Mark Crispin <[email-masked]>
Cc: IMAP Interest List <[email-masked]>, IMAP Extensions WG <[email-masked]>
Subject: Re: LAST CALL: IMAP base and MULTIAPPEND drafts
From: Simon Josefsson <[email-masked]>
Date: 05 Jan 2001 13:06:45 +0100
Lines: 56
Sender: [email-masked]

Mark Crispin <[email-masked]> writes:

> This is also a last call for
>  http://www.ietf.org/internet-drafts/draft-crispin-imap-multiappend-02.txt
> which is a minor (but exceedingly useful) extension.  It is unchanged from
> the -01 draft except for an updated expiration date.

I haven't had time to read this draft until now, so this is a late
minor clearification of two points...  Sorry for late reaction.

Suggestion, replace:

      The APPEND command appends the literal arguments as new messages
      to the end of the specified destination mailbox.  This argument
      SHOULD be in the format of an [RFC-822] message.  8-bit characters
      are permitted in the message.  A server implementation that is
      unable to preserve 8-bit data properly MUST be able to reversibly
      convert 8-bit APPEND data to 7-bit using a [MIME-IMB] content
      transfer encoding.

with

      The APPEND command appends the literal arguments as new messages
      to the end of the specified destination mailbox.  This argument
      SHOULD be in the format of an [RFC-822] message.  8-bit characters
      are permitted in the message body.  A server implementation that is
      unable to preserve 8-bit data properly MUST be able to reversibly
      convert 8-bit APPEND data to 7-bit using a [MIME-IMB] content
      transfer encoding.

since 8-bit characters are not permitted at all in the header section
of a message, I think.

Also, regarding UIDPLUS interactions, I'd suggest replacing

   Servers which support both MULTIAPPEND and [UIDPLUS] will have the
   "resp-code-apnd" rule modified as follows:

   resp-code-apnd  = "APPENDUID" SP nz-number SP set

   That is, the APPENDUID response code returns as many UIDs as there
   were messages appended in the multiple append.

with

   Servers which support both MULTIAPPEND and [UIDPLUS] will have the
   "resp-code-apnd" rule modified as follows:

   resp-code-apnd  = "APPENDUID" SP nz-number SP set

   That is, the APPENDUID response code returns as many UIDs as there
   were messages appended in the multiple append.  The UIDs returned
   should be in the order the articles where appended.  The message
   set may not contain extraneous UIDs or the symbol '*'.

to mimic some of the text from RFC 2359 regarding `set's and COPYUID.

Date: Thu, 4 Jan 2001 19:44:53 -0800 (PST)
From: Mark Crispin <[email-masked]>
To: IMAP Interest List <[email-masked]>
cc: IMAP Extensions WG <[email-masked]>
Subject: LAST CALL: IMAP base and MULTIAPPEND drafts
Organization: Networks & Distributed Computing
Sender: [email-masked]

As discussed in the IMAPEXT meeting in San Diego,...

This is a last call for
 http://www.ietf.org/internet-drafts/draft-crispin-imapv-13.txt
which is the latest revision of RFC 2060, correcting all known errors and
omissions.  It has been 4 years since RFC 2060 was released, and there are
a lot of changes and clarifications.

This is also a last call for
 http://www.ietf.org/internet-drafts/draft-crispin-imap-multiappend-02.txt
which is a minor (but exceedingly useful) extension.  It is unchanged from
the -01 draft except for an updated expiration date.

Does IMAPEXT want to take on MULTIAPPEND, or just let it advance to
Proposed Standard status by itself?  I don't forsee any further work being
done to MULTIAPPEND, and people have been implementing it...

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

To: IETF-Announce: ;
Cc: [email-masked]
From: [email-masked]
Reply-to: [email-masked]
Subject: I-D ACTION:draft-ietf-imapext-thread-06.txt
Date: Thu, 04 Jan 2001 07:27:27 -0500
Sender: [email-masked]

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Internet Message Access Protocol Extension Working Group of the IETF.

	Title		: INTERNET MESSAGE ACCESS PROTOCOL - THREAD EXTENSION
	Author(s)	: M. Crispin, K. Murchison
	Filename	: draft-ietf-imapext-thread-06.txt
	Pages		: 13
	Date		: 03-Jan-01
	
This document describes the server-based threading extension to the
IMAP4rev1 protocol.  This extension provides substantial performance
improvements for IMAP clients which offer threaded views.
A server which supports this extension indicates this with more or
more capability names consisting of 'THREAD-' followed by a supported
threading algorithm name as described in this document.  This
provides for future upwards-compatible extensions.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-imapext-thread-06.txt

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-ietf-imapext-thread-06.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-ietf-imapext-thread-06.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="[email-masked]"

Content-Type: text/plain
Content-ID:	<[email-masked]>

ENCODING mime
FILE /internet-drafts/draft-ietf-imapext-thread-06.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-ietf-imapext-thread-06.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<[email-masked]>

--OtherAccess--

--NextPart--

To: IETF-Announce: ;
Cc: [email-masked]
From: [email-masked]
Reply-to: [email-masked]
Subject: I-D ACTION:draft-ietf-imapext-sort-06.txt
Date: Thu, 04 Jan 2001 07:27:22 -0500
Sender: [email-masked]

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Internet Message Access Protocol Extension Working Group of the IETF.

	Title		: INTERNET MESSAGE ACCESS PROTOCOL - SORT EXTENSION
	Author(s)	: M. Crispin, K. Murchison
	Filename	: draft-ietf-imapext-sort-06.txt
	Pages		: 10
	Date		: 03-Jan-01
	
This document describes an experimental server-based sorting
extension to the IMAP4rev1 protocol, as implemented by the University
of Washington's IMAP toolkit.  This extension provides substantial
performance improvements for IMAP clients which offer sorted views.
A server which supports this extension indicates this with a
capability name of 'SORT'.  Client implementations SHOULD accept any
capability name which begins with 'SORT' as indicating support for
the extension described in this document.  This provides for future
upwards-compatible extensions.
At the time of this document was written, the IMAP Extensions Working
Group (IETF-IMAPEXT) was considering upwards-compatible additions to
the SORT extension described in this document, tenatively called the
SORT2 extension.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-imapext-sort-06.txt

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-ietf-imapext-sort-06.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-ietf-imapext-sort-06.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

--OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="[email-masked]"

Content-Type: text/plain
Content-ID:	<[email-masked]>

ENCODING mime
FILE /internet-drafts/draft-ietf-imapext-sort-06.txt

--OtherAccess
Content-Type: Message/External-body;
	name="draft-ietf-imapext-sort-06.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<[email-masked]>

--OtherAccess--

--NextPart--