Subject: [Extra] IMAP4rev2 body search

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 17 Jan 2020 15:56:06 +0100

Hi,

a discussion the other day^Wweek jars in my mind.

I think we should underspecify the BODY search key more clearly. All it now 
says is "matches if contains", which is IMO correct (it matches the running 
code) but underspecified and vague. I think we ought to eliminate the 
vagueness, and propose the following explicit underspecification:

---

Messages that contain the specified string in the body of the message.

The server SHOULD decdode the content-transfer-encoding, so that a message 
matches independent of its content-transfer-encoding. Apart from that rule, 
this specification explicitly allows much server behaviour that has been 
common in IMAP4rev1 servers, including:

Most servers interpret "contains" on a character level, ie. "BODY range" 
matches a message that contains the word "orange", but some servers 
interpret it on a token or word level, ie. "BODY range" does not match a 
message that contains "orange", because "orange" is one token. It may 
however match a message that contains "ranges", if the server uses stemming 
(and perhaps language detection).

Some servers search only in text/plain. Others also search in other types, 
for example Microsoft Word and PDF attachments.

Some servers search HTML on a source level ("BODY range" does not match 
ra&shy;nge), others search HTML as normally displayed ("BODY range" matches 
ra&shy;nge).

Most servers interpret the messages and search terms independent of 
encoding, such that a message that uses charset=iso-8859-8 may match a 
search term that uses UTF-8. However, not even this is required.

Rationale: IMAP4rev1 servers vary in the kind and quality of their search 
implementation. This document chooses to avoid adding requirements on the 
business logic in such servers.

---

(I'm open to changing any of this. IMO any variant is good, and good is 
better than best. I'd prefer that quarrelsome people can't beat each other 
over the head with the RFC, that's all.)

Arnt

---

From: Brandon Long <[email-masked]>
Date: Wed, 22 Jan 2020 11:01:14 -0800

> over the head with the RFC, that's all.)

This seems overly wordy, almost more like a bcp or something instead of the
spec itself.

I'm not opposed to it, but I wouldn't call myself an expert on the fine
craft of spec writing either.

Can we give intent itself?  Ie, that there's several ways to implement
search (substring, word based)
and to interpret body (raw text, extracted from formatting, or even ocr)

Brandon

Brandon

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 23 Jan 2020 18:41:48 +0100

I think the key question is: Underspecifed and vague about it, or 
underspecified and defiant about it.

God knows that what I suggested isn't well-written. It's a first draft, 
written in one go while I was waiting impatiently for a slow, slow program 
to finish. First drafts are too wordy 99% of the time and the circumstances 
weren't favourable either. But that doesn't matter. Any good text should 
IMO have the traits a) explicit about what it doesn't standardise and b) 
explicit that we want existing servers to remain compliant, warts and all.

The key question is whether you all agree with me about a+b. If you do 
agree with me I can refine the prose. Or Alexey does.

Arnt

---

From: Brandon Long <[email-masked]>
Date: Thu, 23 Jan 2020 14:36:56 -0800

> agree with me I can refine the prose. Or Alexey does.

as an author of one of the servers which definitely took liberties, I do
agree with a+b

Brandon