Subject: Stringprep Considered Harmful

From: Rici Lake <[email-masked]>
Date: Mon, 8 Nov 2004 13:14:37 -0500

draft-ietf-ldapbis-strprep-04.txt would define and require the use of a 
stringprep profile for many common LDAP attribute types. The stringprep 
algorithm may fail on certain input strings; if it fails, that input 
string becomes unmatchable.

If all such strings were obviously illegitimate, this would not be a 
problem, but many legitimate strings will fail, and this will create 
problems, some of them serious.

The clearest example of this is a string which fails the bidi check, 
which requires any string which contains right-to-left characters to 
both start and end with a right-to-left character. Consequently, a 
string which ends with a latin character and contains an arabic word 
will fail.

There are a number of potential examples of such strings which might 
usefully be found in a directory. These include:

-- a url starting with the latin string 'http' arabic domain-name or 
path component
-- an email address ending with a standard TLD which contains hebrew in 
the mailbox or in some domain component.
-- a full name in latin with a syriac nickname embedded in it.
-- a descriptive text field which contains some right-to-left words.

As I understand it, strprep's bidi rule was essentially designed for 
the nameprep profile, which is performed component by component on 
domain names and therefore will not fail a domain name which has both 
arabic and latin components (which, I believe, would be the normal case 
for a domain name including arabic components.) It is therefore 
suitable for the dc attribute, but would not be suitable for an 
attribute which could contain a sequence of domain components.

Aside from leading to the failure to return unique identifiers (such as 
the first three examples above), the use of the bidi-prohibition may 
cause substring matches to fail, even if the substring assertions pass. 
Consequently, a text description with a single arabic word in it would 
effectively be unmatchable even if the assertion and the substring to 
be matched were idential latin (or arabic) words.

Furthermore, it appears to me that unmatchable attribute values will be 
harder to Modify. If the attribute type were single valued, one could 
use replace rather than delete/add, but for a multi-valued attribute 
where the delete/add sequence would probably be the expected procedure, 
the delete will fail because the current attribute value cannot match 
the one specified in the delete operation. Also, an attempt to add a 
duplicate value -- that is a value which is octet-for-octet identical 
-- will silently and erroneously succeed.

While the bidi prohibition step seems to me to be the most likely 
source of problems, other prohibitions -- such as the prohibition of 
characters not in Unicode 3.2 -- may result in non-intuitive behaviour.
It would also prevent an enterprise from using private use area 
characters in an internal directory, although that would be low on my 
list of priorities.

The strprep document cites the importance to security of a predictable 
and consistent string comparison algorithm. This is certainly true, at 
least in the case of strings which are used in some security process 
(but it is worth noting that many strings entered in directories are 
not part of any security process). However, the strings in question 
are, in many cases, also being used and matched by systems outside of 
the directory, such as filesystems, email processors, web servers, and 
so on. It is necessary that all systems using a particular string for 
identification purposes perform the same matching algorithm on that 
string. If the string in question is a filepath on a Mac OS X HFS 
filesystem, for example, the canonicalization algorithm needs to be the 
one used (and extremely well documented) by HFS. Similarly, for NTFS. I 
don't believe that LDAP is (yet) in a position to impose a single 
canonicalization on every string used in every system in the world; 
this may indicate the need for more MatchingRules which correspond to 
those system's matching algorithms.

Rici

---

From: Hallvard B Furuseth <[email-masked]>
Date: Tue, 9 Nov 2004 11:53:26 +0100

> problems, some of them serious.

Yes.  I don't know much about your specific examples, but in general
I've never understood why people feel the advantages of error returns
from stringprep are more important than the problems they cause.

-- 
Hallvard

---

From: Rici Lake <[email-masked]>
Date: Tue, 9 Nov 2004 10:15:11 -0500

> from stringprep are more important than the problems they cause.

Let me be clear. I think that IDNA is a perfectly acceptable way to
internationalise domain names. However, it differs from LDAP in that
it defines a canonical comparison rule for a specific naming domains
(DNS domain names) whereas LDAP is intended to be a general purpose
protocol applicable to a wide variety of naming domains.

Here is a concrete example (which requires a unicode mail reader,  
sorry.)

I don't speak Arabic, so I hope it is not rude; the example word comes
from Simon Josefsson's example page; I simply modified it to be a full
domain name

Suppose that this domain has been registered:

Mohammed@ليهمابتكلموشعربي؟.com.nu

(I believe .nu accepts IDNA domains)

IBM's idna demo page:
<http://oss.software.ibm.com/cgi-bin/icu/idnademo? 
t=www.ليهمابتكلموشعربي؟.com.nu>

shows the ToAscii transform of this domain name as:

   www.xn--egbpdaj6bu4bxfgehfvwxn.com.nu

You can see that the transformation has been done component by component
(label by label, as IDNA says). The bidi rule succeeded on each  
component.
Changing the first . to a - makes the first label invalid:

   www-ليهمابتكلموشعربي؟.com.nu

<http://oss.software.ibm.com/cgi-bin/icu/idnademo?t=www- 
ليهمابتكلموشعربي؟.com.nu>

So the second one is not a valid domain name. (That's irrelevant, I  
just wanted to show what happens when the bidi rule fails.)

However, if the bidi rule is applied to the domain name *as a whole*,
both will fail. Use of this domain name in an LDAP directory would be  
fine

   cn=Mohammed,dc=www,dc=-ليهمابتكلموشعربي؟,dc=com,dc=nu

However, the above cannot itself be used as a (useful) DirectoryString  
because
it will not match even itself.

If the LDAP directory contained email addresses:

   email: Mohammed@ليهمابتكلموشعربي؟.com.nu

Now this attribute is almost completely useless to me. While I could  
search the directory for email addresses by suffix: (email=*com.nu), It  
would not show up in the list because the stored value fails the bidi  
test. I cannot search the directory for that particular domain name:  
(email=*@ليهمابتكلموشعربي؟.com.nu) because the assertion will fail the  
bidi test. If I get desperate and try to Modify the address to insert  
the idna-transformed using {delete  
email=Mohammed@ليهمابتكلموشعربي؟.com.nu, add  
email=[email-masked]}, the delete will fail  
because the bidi rule fails. (That might not be a disaster in this  
case, but it is easy to see where it might be.)

I hope that makes my concern clearer.

---

From: Rici Lake <[email-masked]>
Date: Tue, 9 Nov 2004 10:27:31 -0500

Oops. Either me or my email client got confused at all the 
right-to-lefting.
The first example should read:

  www.ليهمابتكلموشعربي؟.com.nu

instead of

On 9-Nov-04, at 10:15 AM, Rici Lake wrote:
...

If anyone cannot read this message (and wants to) let me know and I'll 
email you a pdf.

R.

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Thu, 11 Nov 2004 11:21:16 -0500

In revising the LDAP technical specification, the WG was
charged with resolving known interoperability problems,
including those in character string matching.  I believe
the WG understood (and understands) that fixing the general
character string matching problems requires invalidating
(or "breaking") some specific application uses of LDAP.  I
also believe the WG understands well the limitations of the
Stringprep solution and accepts those limitations as
reasonable given the situation.

Your comment "this may indicate the need for more MatchingRules
which correspond to those system's matching algorithms" is on
target.  Where the general-purpose matching rules (e.g.,
caseIgnoreMatch, caseExactMatch, etc.) are inappropriate in
some application-specific context, an application-specific
matching rule should be introduced OR the application should
specify that application-specific transformations (e.g., IDN
toASCII/toUnicode) be employed above the directory such that
matching with general purpose matching rules (e.g.,
octetStringMatch) in the directory behaves in a manner
appropriate for the application.

Kurt, as Editor.

---

From: Rici Lake <[email-masked]>
Date: Thu, 11 Nov 2004 16:40:04 -0500

> reasonable given the situation.

I find that acceptable for most of the little quirks of the
stringprep profile proposed. But I would beg the WG to
reconsider the application of the bidi-prohibition rule.

To try to make the problem as simple as possible, all of the
following strings would be unmatchable with bidi-prohibition:

1) A URL which contains an Arabic or Hebrew component, whether
a domain component, path component or query component, unless
%-quoted. (This might even be an ldap: url.) These will be
unmatcheable because a URL must start with a scheme name,
all of which currently are left-to-right characters (in fact
[a-z0-9]+)

2) An email address which contains an Arabic or Hebrew
domain name component or Arabic or Hebrew mailbox name,
unless the entire domain name include the TLD is in
Arabic or Hebrew. (I know that there is a proposal to
implement Arabic TLDs, but I feel that <arabicword>.com
is rather more likely in the forseeable future.)

3) A string not identified as having distinguished name
syntax which happens to be a distinguished name which
includes Arabic or Hebrew components

4) An absolute filepath which includes Arabic or Hebrew
components, even if all components are Arabic or Hebrew, since
the initial / will fail the bidi prohibition

5) And a long etc.

So it seems to me that the bidi prohibition is more the exception
than the rule; it should be applied to the syntax for the dc
attribute, but it is hard to think of other attributetypes where
it will not primarily serve block the use of right-to-left scripts.

> appropriate for the application.

That seems entirely appropriate to me. Consequently, I would humbly
suggest that the default general-purpose matching rule not include
the bidi prohibition rule, and that an application-specific rule
be introduced for attributes whose values will be normalised by
nameprep.

Rici

---

From: Steven Legg <[email-masked]>
Date: Fri, 12 Nov 2004 16:27:24 +1100

Rici,

>> reasonable given the situation.

I doubt the working group as a whole has the background to give
due consideration to the issues around bidi. I know I don't.
I think prior consensus here probably amounts to a lot of
"I dunno, whatever".

> reconsider the application of the bidi-prohibition rule.

The situations you have described seem to me to be quite reasonable
uses of the directory. It is unfortunate if LDAPprep + stringprep
is making those things illegal.

> it will not primarily serve block the use of right-to-left scripts.

Seems like a fair assertion to me.

> be introduced for attributes whose values will be normalised by

I agree that it is appropriate for the general purpose matching rules to
be able to support arbitrary changes of direction. Though it is not at all
clear to me what the implications would be for the substring matching rules.

Regards,
Steven

---

From: Hallvard B Furuseth <[email-masked]>
Date: Fri, 12 Nov 2004 22:29:48 +0100

> I think prior consensus here probably amounts to a lot of

If so, I suggest to ask the [email-masked] mailinglist to review
the draft.

I had kind of assumed that people in general knew what they were talking
about here, otherwise I'd have tried a bit harder to learn more about it
myself:-)

-- 
Hallvard

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Sun, 14 Nov 2004 22:31:35 -0800

(combining responses)
>I think prior consensus here probably amounts to a lot of

Well, I don't give much weight to "I dunno, whatever"
opinions in consensus determination.

In respect to LDAPprep, there seems to enough folks (not
just LDAPbis regulars) engaged in LDAPprep discussions who
understand Stringprep/Unicode issues well enough to form an
opinion as to which of the two BIDI options Stringprep should
be used.  I note that throughout the engineering of LDAPprep,
the WG has repeatedly and regularly consulted with area
experts.  The BIDI restrictions were, in fact, added after
discussions with various Stringprep/Unicode experts (as I
noted in a post last year).

>is making those things illegal.

But one has to consider whether those same things are reasonable
for directory strings in general.  In particular, one has to
carefully consider the Security Considerations outlined in 9.1
of [Stringprep] apply to directory strings.  It would seem
these considerations apply well to values commonly used for naming
and in security applications (such as PKIX), and even descriptive
text.

>>all of which currently are left-to-right characters (in fact

Regardless of the BIDI issue, caseIgnore*Match/caseExact*Match
are inappropriate rules for matching URLs as the rules are not
aware of the particulars of the URL syntax, such as escaping.

>>is rather more likely in the forseeable future.)

There is no standard way to use Arabic or Hebrew in
mailbox names.  The existing 'mail' attribute is restricted
to IA5

As for domain components, in LDAP, domain components
should be IA5 strings (constrained to the appropriate
LDH subset).  IDNA likely should be applied above the
directory for interoperability reasons.  Or new attributes
are introduced with a Unicode-allowing syntax, they can be
introduced with new matching rules.

>>includes Arabic or Hebrew components

Regardless of the BIDI issue, case{Ignore,Exact}*Match
are inappropriate rules for matching DN strings as the rules
are not aware of the particulars of the DN string syntax,
such as escaping.

>>the initial / will fail the bidi prohibition

Though you didn't detail what kind of file path you were
referring to, but I would suspect that case{Ignore,Exact}*Match
are inappropriate for most if not all common file path syntaxes
(due to significant space issues, escaping, and various other
issues).

>Seems like a fair assertion to me.

I view all of cases listed as being application-specific cases
where the general purposes matching rules (and possible the
directory string syntax) simply are inappropriate.  These special
cases would be better handled by application-specific matching
rules.

>>be introduced for attributes whose values will be normalised by

The BIDI restrictions are appropriate for most (if not all of)
the standard track attribute types using these matching rules,
especially in light of various security considerations (such
as those elaborated on in 9.1 of [Stringprep]).

Kurt

---

From: Rici Lake <[email-masked]>
Date: Mon, 15 Nov 2004 03:34:37 -0500

On 15-Nov-04, at 1:31 AM, Kurt D. Zeilenga wrote:

> of [Stringprep] apply to directory strings.

Indeed. I have read that section several times. It appears to
be primarily concerned with visual spoofing attacks, which are
obviously relevant for domain names. However, I note that the
bidi prohibition only deals with one, fairly small, class of
such attacks, and possibly does not deal with all of those
(viz. the discussion last year on the W3C IRI mailing list.)

The ldapbis strprep draft says in its abstract that the motivation
for the specification is interoperability; there is no mention of
security as a primary motivator. Interoperability does not require
bidi prohibition; Unicode strings are always stored in logical
order, which is precisely the order in which they should be compared,
regardless of how many directionality runs there are.
(And consequently, to answer a previous question, mixed directionality
text has no impact on substring matching.)

Other than the bidi prohibition rule, the remaining specifications,
although well-intentioned people may disagree about details (such
as which if any compatibility decompositions should be required)
are highly relevant to the goal of interoperability, and any standard
would be welcomed (at least by me) as better than no standard at all.

Bidi prohibition therefore stands out as an odd exception in the 
specification.

I don't know what consultation the working group carried out in the
decision to adopt the bidi prohibition rule but I hope you will not be
offended by my saying that the messages you refer to (I presume these
are http://www.openldap.org/lists/ietf-ldapbis/200312/msg00087.html
and http://www.openldap.org/lists/ietf-ldapbis/200401/msg00019.html)
do not appear to indicate a detailed consultation with a number of
experts ("after chatting with Paul Hoffman again") or careful 
consideration
by the working group (the fact that no concerns were expressed between
December 19 and January 5, which I believe are two weeks in which most
people in the western world are thinking about something other than
string normalization.)

It seems odd to me to propose to use bidi prohibition for strings
which might have multiple components. Both of the bidi prohibition
applications I am aware of, the nameprep profile and the IRI internet
draft (http://xml.coverpages.org/draft-duerst-iri-09.txt)
are quite clear that the bidi prohibition rule should be performed
component by component, and not on strings as a whole, where it would
obviously be counterproductive.

The IRI draft, in particular, provides a number of useful 
recommendations
for the display of IRIs in order to avoid visual spoofing, but even
after articulating the bidi prohibition rule, it notes (p. 18):

     The above restrictions are given as shoulds, rather than as musts.
     For IRIs that are never presented visually, they are not relevant.
     However, for IRIs in general, they are very important to insure
     consistent conversion between visual presentation and logical
     representation, in both directions.

IRIs (perhaps less so than FQDNs) are used in contexts where visual
spoofing attacks are high-profile. But I wonder if a similar argument
can be made for general-purpose directory entries.

In fact, it is not clear that LDAP as a protocol should concern itself
at all with what is essentially a question of how to visually render
information rather than how to store, transport, and retrieve it.
Of course, guidance on rendering such information would always be 
useful.

> It would seem

I'm afraid this doesn't seem to me to be a sufficient justification for
bidi prohibition. Descriptive text can be a complete lie, regardless of
whether it contains bidi characters. There are an enormous number of
visual spoofing attacks which have nothing to do with the bidi rendering
algorithm (for example, the use of Greek and Cyrillic characters which
render identically to Latin characters) which are both easier to 
accomplish
and do not depend on the victim of the spoof being fluent in two 
languages
with differing writing directions.

> aware of the particulars of the URL syntax, such as escaping.

I accept that. However, there does not appear to be any proposal
for a syntax useful for URLs, and it is likely that DirectoryString
will be used in real applications.

> mailbox names.  The existing 'mail' attribute is restricted

There are many people (at least in the directories I work with) whose
mailboxes are not in IA5. I think this is increasingly common, whether
or not there is a standard. I wonder how typical LDAP administrators
deal with this; the use of DirectoryString syntax seems highly probable
to me.

> LDH subset).

In that case, the bidi prohibition is not necessary for the dc
attribute. I stand corrected. (It is also true that distinguished
names SHOULD be stored in attributes with DN syntax. However, I
believe there are cases where they are not; for example, a DN might
appear as part of the value of a description attribute.)

> (due to significant space issues, escaping, and various other

This is true with the application of the LDAPprep profile. However,
the filesystems I most commonly use do not employ escape characters
(indeed, I am not aware of a filesystem which does) and could be
appropriately be matched by one of case{Ignore,Exact}Match aside
from the issue of insignificant spaces. I believe that the majority
of traditional Unix filesystems would be better matched with a pure
octet string match, but the Macintosh filesystem I use daily uses
case-folding followed by Unicode normalization; it differs in some
details from the LDAPprep proposal, but in most cases (barring the
bidi prohibition) the results will be quite similar. Insignificant
spaces in filepaths (and other strings) are another instance of
visual spoofing.

Again, since there do not appear to be clear alternative matching
rules and syntaxes, it is likely that administrators will make do
with what there is.

> cases would be better handled by application-specific matching

In some sense, all directory strings are application-specific cases.
Indeed, if URLs, email addresses, and filepaths are application
specific, one might wonder what a general case for a directory
might be, and what if any purpose would be served by applying
bidi prohibition to it.

> as those elaborated on in 9.1 of [Stringprep]).

Looking through those of the standard track attributes, I actually
see very few for which the bidi prohibition rule would be appropriate.
The rule is inapplicable to attributes which numeric or restricted to 
IA5,
and as you point out, distinguishedName already has a specific matching
rule.

Postal Addresses in countries which use right-to-left scripts are likely
to contain right-to-left characters and start or end with a number.
Such addresses would fail the bidi prohibition rule.

Names are more likely to be single-directional, but some people
do like to include both right-to-left and left-to-right nicknames
in their common names. Such entries would fail the bidi prohibition
rule, and I cannot see a compelling reason to tell my colleagues to
not do this.

Descriptive text (such as, in the case of a directory important
to me, the titles of development-assistance projects) is quite
likely to contain text with both directionalities.

Other attributes, such as country, location, and generationalQualifier
do not appear to be likely targets for bidi spoofing attacks.

Of course, I freely admit that I am not a recognized expert in any
technology; merely someone trying to implement LDAP directories
which might be useful to an international organization working
in countries with a variety of scripts. Perhaps I am missing
something obvious. But I feel it would be a shame if LDAPbis, which
contains many useful and well-articulated ideas, were to render
itself less useful in a multilingual world as a result of the
inappropriate application of bidi prohibition.

Rici

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Mon, 15 Nov 2004 16:41:08 -0800

I'm going to respond to a couple of items separately.

At 12:34 AM 11/15/2004, Rici Lake wrote:
>experts ("after chatting with Paul Hoffman again") or careful consideration

Well, I was only referencing the messages in support of my statement
that various experts where consulted in regards to proposal for LDAPprep
to employ Stringprep BIDI restrictions.  While the referenced messages
only indicates that I chatted with one expert, I didn't mean for those
referenced messages to be inclusive of all experts that were consulted.
Other experts were consulted.

Kurt

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Mon, 15 Nov 2004 16:51:45 -0800

At 12:34 AM 11/15/2004, Rici Lake wrote:
>for the specification is interoperability;

Actually, the Abstract says:
  This led to a number of usability and interoperability problems.

which implies LDAPprep was intended to resolve these problems.

>there is no mention of security as a primary motivator.

Yes.  But the lack of mention of security in the Abstract
does not imply that LDAPprep does not or should not attempt
to resolve applicable security considerations.

Kurt

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Tue, 16 Nov 2004 23:45:36 -0800

I've given this a new subject heading to indicate the scope
of the discussion has narrowed to the question of whether
LDAPprep should or not ignore BIDI (as discussed in Section
6 of [Stringprep]), in particular in the general-purpose
text matching rules, e.g., caseIgnoreMatch and friends.

There have been a number of good points made in this thread.

I believe these matching rules are generally applicable to
the matching text values, and in particular short descriptive
text (e.g., values used in naming).  I believe they are not
applicable to the matching values of non-text abstractions,
such as DNs, domain names, email addresses, and URIs.  Use
of text matching rules will lead to inappropriate matching
including rule(A,A') evaluating to FALSE even though A and A'
are represent the same abstract value and rule(A,B)
evaluating to TRUE even though A and B represent two
non-equivalent abstract values.  (I believe we need to
explicitly state the applicability of these rules)

It is recognized that the Standard Track does not yet
include matching rules appropriate to match
internationalized domain names (or domain components),
email addresses (or local-parts), or URIs.  This is an
area where future standardization is needed and, in
particular, something this WG likely could consider
undertaking after we deliver the revised LDAP TS.

It can be argued that the text rules are not well suited
(regardless of BIDI issue) for matching large values of
running text (e.g., anything more than a line of text).

With all of the above in mind, the question can be narrowed
to whether or not ignoring BIDI makes sense for matching
rules designed to support short descriptive text, such
as values used in naming objects.  For instance, common
names and postal addresses.

Rici detailed his belief that ignoring BIDI makes sense for
short descriptive text.  I find the most of compelling of
his arguments to be those with values of common names
and postal address lines.

And while I believe BIDI (and other) spoofing attacks are
applicable to naming values, I think we can address these
by stating appropriate security considerations.

Hence, I support removal of the BIDI restrictions from
LDAPprep.

-- Kurt

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Wed, 17 Nov 2004 09:10:22 -0800

I have received the following comment regarding removal
of the BIDI restrictions in LDAPPprep:
   If you're absolutely sure that these are strings that
   will not be compared visually by humans, that is OK.
   If humans are supposed to be involved, you are possibly
   creating a very dangerous situation.

Kurt

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Wed, 17 Nov 2004 09:43:11 -0800

>   creating a very dangerous situation.

This is was my response to the above comment.
 I suspect there are cases where humans may visually compare  
 these strings.  However, it's been noted that there are numerous
 other visual spoofing attacks which can be made.  It's also
 clear that even with BIDI restrictions, humans could be presented
 with strings to compare which do not adhere to the BIDI restrictions.
 This is because the BIDI restrictions impact how implementations
 do comparisons, they do not impact what Unicode string can or cannot
 be transferred by LDAP (or stored by LDAP implementations).  My
 suggestion is that visual spoofing (BIDI and other) concerns can
 be addressed through security considerations, namely by stating
 (much like IRI I-D does) guidelines for input and rendering of
 BIDI values.

Kurt

---

From: Rici Lake <[email-masked]>
Date: Wed, 17 Nov 2004 13:11:00 -0500

A small anecdote about application-specific directory protocols:

When I was in Greece many years ago (and I don't know if this is
still the case), vehicle licence plates were alphanumeric using
Greek capital letters -- but only those Greek capital letters
which are identical to Latin capital letters. This seems like
an appropriate protocol for licence plates, but would not be
generally applicable.

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Thu, 18 Nov 2004 10:14:47 -0800

Steve Hanna provided the following comment:
>I agree with removing the bidi check from LDAPprep.
>The purpose of LDAPprep and PKIXprep is to allow
>LDAP servers and certification path validators to
>compare Unicode strings in a reasonable and predictable
>manner. Human comparison of strings is not an
>important part of the use case so spoofing is not
>a big concern. The bidi checks outlaw useful Unicode
>strings without providing any benefit in this context.
>Therefore, we should remove them.

-- Kurt