IMAP UIDBATCHES Extension
Internet-Draft draft-eggert-uidbatches-00
Daniel Eggert, Ed.
Apple Inc
One Apple Park Way
Cupertino
CA
95014
USA
deggert@apple.com
2024-11-04
Art
MailMaint
IMAP
The UIDBATCHES extension of the Internet Message Access Protocol (IMAP) allows clients to retrieve UIDs from the server such that these UIDs split the messages of a mailbox into equally sized batches. This lets the client perform operations such as FETCH/SEARCH/STORE on these specific batches. This limits the number of messages that each command operates on and may limit the size of the response.
Introduction
This document defines an extension to the Internet Message Access Protocol [RFC3501] for retrieving UIDs that split a mailbox's messages into euqally sized batches. This extension is compatible with both IMAP4rev1 [RFC3501] and IMAP4rev2 [RFC9051].
Document Conventions
In protocol examples, this document uses a prefix of "C: " to denote lines sent by the client to the server, and "S: " for lines sent by the server to the client. Lines prefixed with "// " are comments explaining the previous protocol line. These prefixes and comments are not part of the protocol. Lines without any of these prefixes are continuations of the previous line, and no line break is present in the protocol unless specifically mentioned.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.
Other capitalised words are IMAP keywords [RFC3501] or keywords from this document.
The UIDBATCHES extension
An IMAP server advertises support for the UIDBATCHES extension by including the UIDBATCHES capability in the CAPABILITY response / response code.
UIDBATCHES Command
Arguments:
Message count per batch.
Optional batch range.
Responses:
UIDBATCHES response
Result:
OK
BAD command unknown or arguments invalid
When the client sends a UIDBATCHES command to the server, the server will return those UIDs that split the messages in the currently selected mailbox into equal sizes.
For a mailbox with <N> messages, requesting batches of size <M> will return the UIDs of the messages with the sequence numbers

<N-M>, <N-2*M>, <N-3*M>, ...

If e.g. the client sends

C: A302 UIDBATCHES 2000

and the currently selected mailbox has 6823 messages, the server might return

S: * UIDBATCHES (TAG "A302") UID ALL 99696 20351 7830
S: A302 OK UIDBATCHES Completed

where the message with sequence number 4823 has UID 99696, the message with sequence number 2823 has UID 20351, and the message with sequence number 823 has UID 7830.
The server MUST reply with an UIDBATCHES response. The UIDBATCHES response has the same format as an ESEARCH untagged response to a UID SEARCH command with RETURN (). It notably MUST include the tag of the command it relates to, and it MUST include the UID indicator.
Note that this will not return the highest UID since the client is expected to know this value from UIDNEXT or any newly received messages. Similarly, this will not return the UID of the message with sequence number 1.
A client can optionally provide a batch range. The server will then limits its response to the given UIDs at the given indices. E.g. if the client sends

C: A302 UIDBATCHES 2000 100:200

for a mailbox with more than 400,000 messages, the server would return the 100th to 200th batch, corresponding to the 200,000th and 400,000th message respectively. These UIDs would still split the mailboxes messages into batches of size 2000. The first returned UID would thus correspond to the sequence number 200,000, the second returned UID would correspond to the sequence number 202,000.
As such, the client can request the first 4 batches with

C: A302 UIDBATCHES 2000 1:4

If the selected mailbox has more than 8,000 messages, the server would then return an UIDBATCHES response with 4 UIDs.
When the client issues any valid UIDBATCHES command and the mailbox is empty, the server MUST reply with an UIDBATCHES response. This UIDBATCHES response will not have an ALL part, similar to a UID SEARCH that doesn't match any messages, e.g.

S: * UIDBATCHES (TAG "A302") UID
S: A302 OK UIDBATCHES Completed

The server MAY return fewer UIDs than requested by the client even if the mailbox contains more messages. Since the client knows what the count of messages in the mailbox is, it can determine if the server returned all UIDs or not. Servers MUST at least return the first 40 results unless the client requested fewer. Servers SHOULD at least return the first 100 results unless the client requested fewer.
Servers MUST respond with BAD and a response code TOO SMALL if the client uses a batch size that is smaller than the minimum allowed by the server, e.g.

S: A302 BAD [TOO SMALL] Minimum batch size is 500

Servers MUST allow for batch sizes 500 or larger.
Intended Use
When the client selects a mailbox, it can use the UIDBATCHES command to find the UIDs that split the mailboxes messages into batches. E.g.

C: A142 SELECT INBOX
S: * 6823 EXISTS
S: * 1 RECENT
S: * OK [UNSEEN 12] Message 12 is first unseen
S: * OK [UIDVALIDITY 3857529045] UIDs valid
S: * OK [UIDNEXT 215296] Predicted next UID
S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
S: * OK [PERMANENTFLAGS (\Deleted \Seen \*)] Limited
S: A142 OK [READ-WRITE] SELECT completed
C: A143 UIDBATCHES 2000
S: * UIDBATCHES (TAG "A143") UID ALL 99696 20351 7830
S: A143 OK UIDBATCHES Completed

The client can then use these 4 UID ranges:
1. 215295:99695
2. 99696:20350
3. 20351:7829
4. 7830:1
where each range has 2,000 messages in it, except for the last range, which only holds the remaining 823 messages.
Since new messages can not appear within these UID ranges, the number of messages in each range can not grow. It may decrease, though, as messages get deleted.
The client may choose to keep track of the number of EXPUNGE or VANISH messages and re-run UIDBATCHES when many messages have been deleted. The client MUST NOT excessively re-run UIDBATCHES and specifically MUST NOT re-run UIDBATCHES unless at least N/2 messages have been deleted from the mailbox, where N is the batch size the client has requested.
Similarly, once new messages arrive into the mailbox, the client can start a new message batch 215296:*. Once N or more new messages have arrived, the client can then create a second new batch based on the UID of the N'th message. Alternatively, the client may choose to re-run UIDBATCHES. The client MUST NOT re-run UIDBATCHES if fewer than N/2 new messages have been received.
To clarify, the client MUST NOT re-run UIDBATCHES unless at least one of these conditions are met:
1. a new mailbox has been selected
2. more than N/2 messages have been expunged from the mailbox
3. more than N/2 new messages have been received into the mailbox
Similarity to UID SEARCH Command
The UIDBATCHES is in effect nothing more than shorthand for a UID SEARCH command of the form

C: A145 UID SEARCH RETURN () <N-M>,<N-2*M>,<N-3*M>,...

where N is the number of messages in the mailbox and M is the requested batch count.
The special purpose UIDBATCHES command, though, tries to address two problems:
a. for many servers, UID SEARCH commands specifying sequence numbers are costly, especially for mailboxes with many messages.
b. the UIDONLY extension disallows the use of sequence numbers and thus makes it difficult for the client to split its commands into batches of a size that works well for the client and server.
By providing a special purpose command, servers can implement a different, optimized code path for determining message batches. And servers using the UIDONLY extension can provide a facility to let the client determine message batches without using sequence numbers in a UID SEARCH command.
Similarity to PARTIAL Extension
The PARTIAL extension provides a different way for the client to split its commands into batches by using pages SEARCH and FETCH.
The intention of the UIDBATCHES command is the let the client pre-determine message batches of a desired size.
This makes it easier for the client to share implementation between servers regardless of their support of PARTIAL. And additionally, because the client can issue a corresponding UID SEARCH command to servers that do not implement UIDBATCHES, the client can use similar batching implementations for servers that support UIDBATCHES and those that do not.
Interaction with MESSAGELIMIT Extension
When the server supports both the MESSAGELIMIT and UIDBATCHES extension, the client SHOULD request batches no larger than the specified maximum number of messages that can be processed in a single command. The client MAY choose to use a smaller batch size.
Additionally, since servers MAY limit the number of UIDs returned in response to UIDBATCHES, it is reasonable to assume that they would at most return N UIDs where N is the limit the server announced as its MESSAGELIMIT.
Interaction with UIDONLY Extension
As noted above, the UIDBATCHES extension allows for clients to create UID ranges for message batches even when the connection operates in UIDONLY mode which otherwise doesn't allow for using message sequence numbers.
Interaction with SEARCHRES Extension
Servers that support SEARCHRES [RFC5182] MUST NOT store the result of UIDBATCHES in the $ variable.
Formal syntax
The following syntax specification uses the Augmented Backus-Naur Form (ABNF) notation as specified in [RFC5234].
Non-terminals referenced but not defined below are as defined by IMAP4 [RFC3501].
Except as noted otherwise, all alphabetic characters are case-insensitive. The use of upper or lower case characters to define token strings is for editorial clarity only. Implementations MUST accept these strings in a case-insensitive fashion.

capability          =/ "UIDBATCHES"
                       ;; <capability> from [RFC3501]

command-select      =/ message-batches

message-batches     = "UIDBATCHES" SP nz-number
                      [SP nz-number ":" nz-number]

uidbatches-response = "UIDBATCHES" search-correlator SP "UID"
                      [ALL tagged-ext-simple]

mailbox-data        =/ uidbatches-response

Security Considerations
This document defines an additional IMAP4 capability. As such, it does not change the underlying security considerations of [RFC3501] and IMAP4rev2 [RFC9051].
This document defines an optimization that can both reduce the amount of work performed by the server, as well at the amount of data returned to the client. Use of this extension is likely to cause the server and the client to use less memory than when the extension is not used. However, as this is going to be new code in both the client and the server, rigorous testing of such code is required in order to avoid introducing of new implementation bugs.
IANA Considerations
Changes/additions to the IMAP4 capabilities registry
IMAP4 capabilities are registered by publishing a standards track or IESG approved Informational or Experimental RFC. The registry is currently located at:
https://www.iana.org/assignments/imap4-capabilities
IANA is requested to add registrations of the "UIDBATCHES" capability to this registry, pointing to this document.
References
Normative References
[RFC3501]
[RFC9051]
[RFC2119]
[RFC8174]
[RFC5182]
[RFC5234]