Subject: Re: X.500, LDAP Considered harmful Was: OCSP/LDAP

From: [email-masked] (Peter Gutmann)
Date: Mon, 27 Jan 2003 17:19:03 +1300

[Warning: The previous reply was short and to the point.  This reply is a bit
 of a longer rant, although it should be entertaining :-).  At least the first
 half is mostly technical]

=?ISO-8859-1?Q?Michael_Str=F6der?= <[email-masked]> writes:
>I see no reason why
>
>SELECT cert FROM foo WHERE key = bar
>
>(note that you left out the host connection part)
>
>is less complex than
>
>ldap://host.domain/o=foo?userCertificate?sub?(key=bar)

Thankyou Michael, you've just agreed with me by expressing the exact
{key,value} lookup I described as an LDAP query (rather than the usual
hierarchical filter that you're supposed to use with LDAP).  The only
difference between doing it with a real {key,value} lookup system (e.g.
Berkeley DB, DBMS) and LDAP is that half the time the LDAP admin has it turned
off because the whole directory grinds to a halt if you run more than one
global query at a time.

Two points to note:

- LDAP is never used in this way.  It's always used with extraordinarily
  complex filters that no normal human can figure out.  When I was documenting
  use of LDAP in the cryptlib manual, I went around a variety of CAs (the ones
  that publish the filters you need to use on their web pages), and also asked
  users for contributions - I wanted to include some examples that people
  could cut and paste.  The filters ranged in size from two lines to complete
  paragraphs, and all were completely different and to all but a select few
  X.500 cogniscenti, utterly incomprehensible, making them essentially useless
  as examples.  In the end I inserted a few of the shorter ones to show what a
  typical filter would look like, but couldn't do much more with them.
  
  One user who sent in some filters suggested that CAs were using these as
  access control mechanisms, since only the people who knew the correct magic
  incantation could ever locate a cert, and the only way to do that was to
  have the CA tell you what the incantation was.

- If you're concerned about efficiency, an LDAP directory can never, ever be
  better than a {key,value} lookup (e.g. Berkeley DB or a DBMS, NDS is a bit
  of an exception here).  The reason for this is that the client takes a basic
  SELECT foo WHERE key = bar query, reformats it as a complex LDAP filter, and
  sends it to a server that has to then translate the LDAP filter back into
  the original SELECT foo WHERE key = bar form for the {key,value} lookup, and
  then submit it to the underlying database, typically Berkeley DB (small-
  scale LDAP) or a DBMS (industrial-strength LDAP).  No matter how efficient
  you make the LDAP process, the extra bloat and overhead always make it less
  efficient than a direct query to the underlying database without the LDAP
  four-year-old-eating-peas process getting in the way.

A data point on usability:

  cryptlib supports any kind of cert storage mechanism that you can think of
  (flat files, DBMS, Berkeley DB, LDAP, PGP keyrings, PKCS #11, etc etc etc,
  including some semi-proprietary backends like named segments in flash
  memory).  The most frequently-used one is the DBMS interface (that is, the
  {key,value} lookup model), although there's a fair bit of usage of LDAP as
  well.  In terms of difficulty encountered in using it, no-one has ever,
  *ever* reported a single problem with the {key,value} interface (beyond
  compiling/linking, e.g. using an old version of the Oracle libs or
  something).  In contrast LDAP is the most problematic cert-storage interface
  there is, with an almost continuous flow of questions about the magic
  formulae needed to locate a cert (see above), problems with storing certs
  when the DN doesn't agree with the directory schema, or schema A doesn't
  agree with schema B, and every other LDAP problem that people on this list
  are all too familiar with.  Some people never get cert storage with LDAP
  working at all, and give up.
  
  This is real-world experience talking here folks, from a worldwide user
  base: {key,value} lookup works with zero problems, LDAP has endless
  problems.  I don't care about database or X.500 ideology or theology or
  proctology, the fact is that in practice one approach works and the other
  doesn't, and no amount of whining about "Well in theory we could maybe make
  it work some of the time, and here's an imaginary problem where yours won't
  work and ours will" is going to change that.

>>The
>>analogy I've used is that it's like watching a friend of mine's four-year-old
>>eat peas with a fork, she uses her fingers to put the peas on the fork, moves
>>it up to her mouth, then uses her fingers to move the peas off the fork into
>>her mouth, and everyone gets to say how clever she is to use a fork at her
>>age.  Certs in X.500 directories are the same thing.
>
>Peter, normally you're certainly doing better. Sorry, I file your postings
>about this topic under "Peter Gutmann's personal aversions against X.500
>directories". ;-)

This isn't any specific aversion to X.500 ("I was molested by a hierarchical
database as a child, your honour") but an aversion to, well, I'm not sure if
there's a specific term that exists to describe avoiding the obvious,
straightforward solution and instead choosing a complex, suboptimal one
because your ideology demands it, but purely for easy identification purposes
I'll label it "braindamage" here.  I'd be equally opposed to it if you said we
should use DCE for certificate distribution, or DCOM, or trained chimpanzees.
So although I'm flattered that you've singled me out for personal attention,
what you should really refer to is "everyone who knows anything about
databases' aversion to hierarchical databases, for the simple reason that they
don't work very well".  Although you've singled me out for special attention,
I know I'm in very good company - everyone who knows database technology
agrees with me.  In fact, the world's second largest (or largest, depending on
how the stock stands) software company exists solely from selling non-
hierarchical databases to anyone with a chequebook or credit card.  Larry
Ellison currently has a yacht^H^H^Hsmall ocean liner tied up down at the
waterfront here and is funding a multimillion-dollar sailing yacht that we'll
hopefully beat on the water, and that wasn't funded from building hierarchical
databases (I assume the X.500 creators aspire to one day owning a dingy on a
duckpond somwhere :-).  I know which way the wind's blowing, and it's not a
coincidence that we're racing against a yacht called Oracle and not one called
X.500 (actually I believe there was a ship called X.500, built in about 1912
and rediscovered in 1985 just in time for inclusion in OSI).

Peter.

---

From: Richard Levitte - VMS Whacker <[email-masked]>
Date: Mon, 27 Jan 2003 12:29:05 +0100 (CET)

In message <[email-masked]> on Mon, 27 Jan 2003 17:19:03 +1300, [email-masked] (Peter Gutmann) said:

pgut001> [Warning: The previous reply was short and to the point.  This reply is a bit
pgut001>  of a longer rant, although it should be entertaining :-).  At least the first
pgut001>  half is mostly technical]

Lovely rant :-).

Peter, I've no trouble understanding what you say, and (as you mention
in another post) you're entirely correct, for (key,value) lookups,
there's really no difference between LDAP or RDBMs.

The argument that I'd like to make is another, and it's based on a
wish (I'm not ready to call it anything else, yet) to be able to do 
certification path discovery without having to store every single
existing certificate in my own database (meaning I'd need to be able
to find out where the certificate I'm looking for is stored, and I'd
need to know exactly how to extract the desired certificate from that
store):

- a minor point is that except for the LDAP schema drafts and your
  CERTSTORE draft, there's a lack of anything even remotely looking
  like a standard for certificate storage/extraction.  I would have no
  problem with yet another draft defining a schema for SQL-accessible
  RDBMs, but will be satisfied with what is there otherwise...
- more importantly, there's a lack of anything defining how discovery
  should work, except for the DC-style DNs.  There's talk of DPD, but
  that's a draft on how to put the problem on someone else, which is
  definitely not the same thing...

The only technology that I can think of would work for certificate
lookup based on DNs is LDAP (and I've written some thoughts on that in
an earlier post).  It seems to me that any other type of lookup will
be tied to whatever servers are configured with the application using
them, and that'll hardly cover all possible CAs in the world,
especially if mesh PKIs start to become more common-place...

-- 
Richard Levitte     | http://richard.levitte.org/ | Spannv. 38, I
Levitte Programming | http://www.lp.se/           | S-168 35 Bromma
T: +46-708-26 53 44 |                             | SWEDEN
     "Price, performance, quality...  choose the two you like"

---

From: Richard Levitte - VMS Whacker <[email-masked]>
Date: Mon, 27 Jan 2003 14:30:34 +0100 (CET)

In message <[email-masked]> on Mon, 27 Jan 2003 12:29:05 +0100 (CET), Richard Levitte - VMS Whacker <[email-masked]> said:

levitte> Peter, I've no trouble understanding what you say, and (as you mention
levitte> in another post) you're entirely correct, for (key,value) lookups,
levitte> there's really no difference between LDAP or RDBMs.

As someone reminded me privately: except for performance differences,
of course.

---

From: Michael Ströder <[email-masked]>
Date: Tue, 28 Jan 2003 01:34:11 +0100

Peter Gutmann wrote:
> {key,value} lookup I described as an LDAP query

Well, is that really news to you?

> (rather than the usual
> hierarchical filter that you're supposed to use with LDAP).

???

> LDAP is that half the time the LDAP admin has it turned
> off
> because the whole directory grinds to a halt if you run more than one
> global query at a time.

It seems brain-dead sys admins are maintaining your or your customer's 
systems. With misconfigured servers and/or implementing misbehaving 
applications you can grind any system to a halt, no matter which technology.

>   complex filters that no normal human can figure out.

I don't know how the directories and applications you are working with look 
like. But in the scenarios I know most filters are pretty simple, except the 
ones produced by some strange PKI products of leading vendors.

>  When I was documenting
>   could cut and paste.

I think your fault was to ask PKI people instead asking directory people. 
Usually PKI people tend to make things even more complicated than directory 
people... ;-)

>   problems.

Simply not true. You're working with the wrong directory people I guess.

> I know I'm in very good company - [..] Larry
> Ellison [..] multimillion-dollar sailing yacht

Uumh, yeah! That beats me, I give up here admitting that I can't fight your 
strong arguments...

Ciao, Michael.