Subject: Re: draft-ietf-ldapbis-filter-03.txt

From: [email-masked] (Mark C Smith)
Date: Fri, 15 Nov 2002 09:38:56 -0500

Hallvard B Furuseth wrote:
> Why was section 6 "String Search Filter Definition" changed to
> say that "all octets greater than 0x7f" must be escaped?
> 
> Shouldn't that at least be "MUST" esacpe them when generating filters,
> but SHOULD (or MAY) recognize them unescaped, for backwards
> compatibility?

That change was the result of working group last call comments made by 
Steven Legg (but I am the one who wrote the text). But I think I 
unintentionally made a protocol change. Is there a good reason to 
require that all octects > 0x7f be escaped? I do not remember all of the 
details surrounding those older conversations.

 > Also, the change should be noted in Appendix A.

Agreed. Also, if we leave the escaping requirement in, I agree with you 
we should add text that suggests that recognizing octets > 0x7f when 
they are not escaped is good practice.

> Also,
> 
> 
>>7.  Examples
>>       (1.3.6.1.4.1.1466.0;binary=\04\02\48\69)
> 
> 
> ;binary has been removed from [Protocol].

Good catch.

-Mark

---

From: Hallvard B Furuseth <[email-masked]>
Date: Fri, 15 Nov 2002 14:33:30 +0100

Shouldn't that at least be "MUST" esacpe them when generating filters,
but SHOULD (or MAY) recognize them unescaped, for backwards
compatibility?  Also, the change should be noted in Appendix A.

Also,

-- 
Hallvard

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Fri, 15 Nov 2002 12:16:14 -0800

At 06:38 AM 2002-11-15, Mark C Smith wrote:
>That change was the result of working group last call comments made by Steven Legg (but I am the one who wrote the text).

So that everyone change refresh their memory...

Steven's comment:
  http://www.openldap.org/lists/ietf-ldapbis/200105/msg00027.html

and Mark's subsequent reply:
  http://www.openldap.org/lists/ietf-ldapbis/200106/msg00002.html

and last call summary:
  http://www.openldap.org/lists/ietf-ldapbis/200106/msg00003.html

Kurt

---

From: Hallvard B Furuseth <[email-masked]>
Date: Fri, 15 Nov 2002 22:28:19 +0100

>   http://www.openldap.org/lists/ietf-ldapbis/200106/msg00003.html

Thanks.

The change from talking about "characters" in the -01 draft to "octets"
fixed the complaint in that comment.  If that was the reason for the
change, I see no reason to require characters above 0x7f to be quoted.
They are simply octets that have no special meaning in filters, like
all other octets except * ( ) \ NUL.

-- 
Hallvard

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Fri, 15 Nov 2002 17:20:10 -0800

Should <valueencoding> be something like:

      valueencoding  = 0*(normal / escaped)
      normal         = UTFN / UTFMB
      UTFN           = %x01-27 / %x2b-5b / %x5d-7f
      escaped        = "\" hex hex     
      hex            = %x30-39 / %x41-46 / %x61-66

where UTFM is as defined in [Models]:

        UTFMB   = UTF2 / UTF3 / UTF4 / UTF5 / UTF6
        UTF0    = %x80-BF
        UTF1    = %x00-7F
        UTF2    = %xC0-DF 1(UTF0)
        UTF3    = %xE0-EF 2(UTF0)
        UTF4    = %xF0-F7 3(UTF0)
        UTF5    = %xF8-FB 4(UTF0)
        UTF6    = %xFC-FD 5(UTF0)

(I'd like to see the grammar rewrite to the common productions
form [Models]).

Kurt

At 06:38 AM 2002-11-15, Mark C Smith wrote:
>That change was the result of working group last call comments made by Steven Legg (but I am the one who wrote the text). But I think I unintentionally made a protocol change. Is there a good reason to require that all octects > 0x7f be escaped? I do not remember all of the details surrounding those older conversations.
>
>
>Agreed. Also, if we leave the escaping requirement in, I agree with you we should add text that suggests that recognizing octets > 0x7f when they are not escaped is good practice.
>
>
>>Also,
>>
>>>7.  Examples

---

From: Hallvard B Furuseth <[email-masked]>
Date: Thu, 21 Nov 2002 12:54:48 +0100

> where UTFM is as defined in [Models]:

Hm?  Wasn't the recenlty discussed change done in order to allow
non-UTF8 data (octet strings) as value encodings?  This change would
disallow them again.

BTW, I've just noticed that the requirement to quote %x80-%xff dates
to the -01 version of the draft, not to the octetstring change.
That draft restricts unquoted characters to
      normal  = %x01-27 / %x2b-5b / %x5d-7f
rfc2254 does not do that, it just says that
      value   = AttributeValue from Section 4.1.6 of [rfc2251].

-- 
Hallvard

---

From: Hallvard B Furuseth <[email-masked]>
Date: Thu, 21 Nov 2002 13:14:54 +0100

Sorry, truncated message.

I wrote:

> That draft [filter-01] restricts unquoted characters to

So does -03.  I think 7f should be changed to ff.

-- 
Hallvard

---

From: "Steven Legg" <[email-masked]>
Date: Tue, 26 Nov 2002 17:02:30 +1100

> details surrounding those older conversations.

The string representation of LDAP search filters is used in LDAP URLs.
The whole area of non-ASCII characters in URLs/URIs is murky, but to
quote from RFC 2396:

   A URI is a sequence of characters from a very limited set,
   i.e. the letters of the basic Latin alphabet, digits, and
   a few special characters.

and

   A URI may be transcribed from a non-network source, and thus
   should consist of characters that are most likely to be able to
   be typed into a computer, within the constraints imposed by
   keyboards (and related input devices) across languages and
   locales.

The escaping of octets > 0x7f in assertion values means that the
LDAP URL is an ASCII string and therefore resilient in the face of
transcription. We should probably go further and escape the
non-printable ASCII characters as well.

An LDAP URL with non-ASCII UTF-8 characters could only be faithfully
entered through an interface that supported UTF-8. We have to allow for
the possibility that it will be entered through an interface that
supports a different charset.

Regards,
Steven

---

From: "Steven Legg" <[email-masked]>
Date: Tue, 26 Nov 2002 17:21:57 +1100

Another data point ...

The string representation of LDAP search filters is used in LDAP URLs.
LDAP URLs are present in Referrals in components of type LDAPString.
LDAPString is required to contain UTF-8 encoded characters.
If octets in the range 0x80 to 0xff in arbitrary octet data in assertion
values are not escaped then they could potentially represent invalid
UTF-8 encodings. Escaping octets > 0x7f means that the filter string
is ASCII and therefore also valid UTF-8.

Regards,
Steven

---

From: Hallvard B Furuseth <[email-masked]>
Date: Tue, 26 Nov 2002 11:40:04 +0100

> non-printable ASCII characters as well.

Why?  URIs have their own escaping mechanism (%<hex><hex>).  Also, since
the URI RFC recommends to escape '\', which LDAP uses for quoting, a
non-ASCII character could become 5 characters instead of 3 if we require
LDAP-quoting as well.

> entered through an interface that supported UTF-8.

Only if non-ASCII characters were required to be neither LDAP-quoted nor
URI-escaped, which would be silly.  The Filter, DN and URI drafts all
require some characters to be quoted/escaped and allow additional
characters to be quoted.

-- 
Hallvard

---

From: [email-masked] (Mark C Smith)
Date: Tue, 26 Nov 2002 20:36:54 -0500

Regarding draft-ietf-ldapbis-filter-03.txt, it has been suggested to me that we should first agree on what is meant by "string format." If we answer that question, then some of the detailed questions about which octet sequences must be escaped will be much easier to answer.

My definition would go something like this:

   The string representation of an LDAP search filter
   is a UTF-8 string.

That is, it consists of a series of octets that form a series of valid UTF-8 characters.  Comments?

-Mark Smith
 Netscape

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Tue, 26 Nov 2002 17:54:18 -0800

>  The string representation of an LDAP search filter

I believe it is generally recommended to state exactly which
character set is being encoded using UTF-8.  So, I suggest:

  The string representation of an LDAP search filter
  is a string of UTF-8 encoded ISO 10646-1 characters.

---

From: "Steven Legg" <[email-masked]>
Date: Wed, 27 Nov 2002 16:41:49 +1100

Hallvard,

Hallvard B Furuseth wrote:
> Why?  URIs have their own escaping mechanism (%<hex><hex>).  

You're right. I missed the bit in draft-ietf-ldapbis-url-02.txt that
applied this escaping mechanism. The LDAP URL format isn't a reason for
escaping octets > 0x7f in string filters. A less compelling reason for
escaping octets > 0x7f in UTF-8 encoded assertion values would be to
make the string representation of a filter more easily transcribable
in its own right. The reason for escaping octets > 0x7f in arbitrary
octet data assertion values is to ensure the string filter is a UTF-8
character string (a reasonable goal IMO).

Regards,
Steven

---

From: Hallvard B Furuseth <[email-masked]>
Date: Wed, 27 Nov 2002 11:55:43 +0100

> is ASCII and therefore also valid UTF-8.

Good point.  For the LDAPURL draft, that is:-)
I think last paragraph of [LDAPURL] section 5 covers it, but maybe it
wouldn't hurt to end that pargaraph with something like "Thus, a
generated LDAP URL MUST be an IA5 string, though implementations SHOULD
be prepared to accept other valid UTF-8 strings.".

There is a bug in that paragraph, BTW:

>   Note that characters that are not safe (e.g., spaces) (as defined in
>   section 2.1 of [RFC2396]), (...)

RFC2396 doesn't define "safe", and section 2.1 certainly doesn't - that
is about characters vs. octets.  I think it should be something like:

    Note that characters that are not an "uric" character, as defined
    in {Appendix A or section 2} of [RFC2396], (...)

-- 
Hallvard

---

From: Hallvard B Furuseth <[email-masked]>
Date: Thu, 28 Nov 2002 11:45:24 +0100

I wrote:
> There is a bug in [last paragraph of [LDAPURL] section 5], BTW:
> 
                ^^^^^^^^^^
>     in {Appendix A or section 2} of [RFC2396], (...)

Octets, not characters, since LDAPURLs are described elsewhere as UTF-8
strings.  That is, UTF-8 multibyte characters need to be esacped with
several '%'s, not just one.

As far as I can tell, no other characters in the draft need to be
octets.

-- 
Hallvard