Subject: [ldapext] [Fwd: LDAP C API gripes]

From: Howard Chu <[email-masked]>
Date: Tue, 13 Feb 2007 16:03:12 -0800

I've created this page  http://scratchpad.wikia.com/wiki/LDAP_C_API for 
putting together ideas for a new LDAP C API. The fact that the last 
LDAPv3 API doc never made it out of draft status really needs to be 
addressed, and there are so many flaws with the API (essentially 
inherited from the LDAPv2 API) that it's obvious a simple refresh is 
inadequate. The fact that groups like the Samba team have decided it was 
necessary to create their own LDAP API underscores the point. If anybody 
has comments on what is broken and how it can be fixed, please chime in.

-------- Original Message --------
Subject: 	LDAP C API gripes
Date: 	Mon, 12 Feb 2007 06:27:41 -0800

Probably this should just go out on the ldapext mailing list again. The 
main discussion I referred to started here
http://www.openldap.org/lists/openldap-devel/200312/msg00123.html

I posted my laundry list here
http://www.openldap.org/lists/openldap-devel/200312/msg00187.html

Kurt's followup adds some other considerations.

The list thus far:

1) Needs an explicit library_init/library_destroy and a library_handle. (No
static/global state, everything referenced by the library_handle.)

2) APIs for operations must have a function parameter for every operation
protocol parameter. (E.g., current ldap_search variations all omit DEREF, so
you can only specify this via ldap_set_option. Ugly.)
 2a) Functions with long parameter lists are difficult to manage. Perhaps
use request structures/parameter blocks, like 2.2 slapd.
 2b) Simplified APIs are still desirable, for apps that don't care about
controls/extensions/whatever.

3) Requests and results must be bound together. (E.g., ldap_get_option(
OPT_ERROR_NUMBER) has no notion of which result's error is being retrieved.)

4) char *'s should be avoided, use struct bervals more extensively
 4a) the API is too malloc-happy. result parsing etc. should be done in-place
as much as possible.

- "proper" error handling
       Any routine which can "fail" (e.g., most) should return
       a clear indication of success or the nature of the failure.
       ("fail" here refers to API failure, not a protocol failure.)

       Protocol errors are not to be treated as API errors.

       no ld_errno, no ld->ld_errno, etc.. (This relates to
       Howard's #3.)
- API designed for thread-free implementation
- Ability to provide filters in native (LBER) form
- Applications provided extensibility where ever the ASN.1 allows it
- Applications allowed to "register" new error codes, etc..
- ldap_* be a low-level LDAP API (one-to-one mapping of API
 to protocol).  Higher level "directory" API provided above
 to handle referral chasing and such.

Another gripe I noted was that a lot of APIs are poorly named, but
that should be an easy one to fix along the way.

Obviously from the Samba side we need APIs that allow an external event 
handler to interact with the API. This includes using an already 
connected socket with the library, and being able to trigger response 
processing based on an external trigger/callback/whatever.

If you have suggestions for who else to get involved, let me know, and 
just forward this on. Once we figure out the correct forum for the 
discussion we can go on from there. Perhaps we should use the 
FAQ-o-Matic or a wiki for maintaining the updated list and evolving the 
spec.

-- 
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  Chief Architect, OpenLDAP     http://www.openldap.org/project/

_______________________________________________
Ldapext mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ldapext

---

From: Howard Chu <[email-masked]>
Date: Wed, 21 Feb 2007 07:47:35 -0800

Howard Chu wrote:
> I've created this page  http://scratchpad.wikia.com/wiki/LDAP_C_API 
> for putting together ideas for a new LDAP C API. The fact that the 
> last LDAPv3 API doc never made it out of draft status really needs to 
> be addressed, and there are so many flaws with the API (essentially 
> inadequate. The fact that groups like the Samba team have decided it 
> was necessary to create their own LDAP API underscores the point. If 
> anybody has comments on what is broken and how it can be fixed, please 
> chime in.

Something else that comes to mind - the client library ought to be 
schema-aware. This is something that was always a given in X.500 
clients, but has never had any priority in LDAP clients. It might be 
worthwhile to implement syntax validation in the library, to catch bogus 
inputs before talking to any servers...

---

From: simo <[email-masked]>
Date: Wed, 21 Feb 2007 11:11:17 -0500

On Wed, 2007-02-21 at 08:01 -0800, Howard Chu wrote:
> simo wrote:
> > On Wed, 2007-02-21 at 07:47 -0800, Howard Chu wrote:
> >   
> > Does it mean making the client retrieve the schema from the server each
> > time? Might work on long lived apps, but an ldapadd would terribly
> > suffer ...
> >   
> 
> This would only be part of a higher level API. I would only expect GUI 
> tools to use it, although if you're ldapadding a 512MB LDIF it might be 
> worthwhile to use it on the commandline too.

Still the idea does not really appeal me, can you summarize very quickly
what advantage do you get in doing that?

Simo.

-- 
Simo Sorce
Samba Team GPL Compliance Officer
email: [email-masked]
http://samba.org

---

From: Kurt Zeilenga <[email-masked]>
Date: Wed, 21 Feb 2007 08:24:42 -0800

On Feb 21, 2007, at 7:47 AM, Howard Chu wrote:

> clients, but has never had any priority in LDAP clients. It might  
> be worthwhile to implement syntax validation in the library, to  
> catch bogus inputs before talking to any servers...

Yes.  A server is only going to return the most basic information about
the problem, if lucky, identify the particular value which is invalid.
A syntax validation library could identify what in that value was bad.
For instance, consider validation of a value of the uniqueMember
attribute.  A server might say value x is bad.  The library might
be able to say that the 1st AVA of the 2nd RDN of the DN component
is invalid.

Of course, this validation API should be optional.  In supporting
handheld devices, offloading validation to the server would be a
good thing.  Of course, a handheld device is probably better offloading
all of LDAP to a device service of some sort.

-- Kurt

---

From: Howard Chu <[email-masked]>
Date: Wed, 21 Feb 2007 08:01:19 -0800

---

From: simo <[email-masked]>
Date: Wed, 21 Feb 2007 15:52:19 +0000

Simo.

---

From: Michael B Allen <[email-masked]>
Date: Wed, 21 Feb 2007 12:52:26 -0500

On Wed, 21 Feb 2007 11:11:17 -0500

> what advantage do you get in doing that?

We just did an LDAP abstraction API that has "schema awareness". We're
doing a PHP extension and we wanted to make things "as simple as possible
but not simpler" for the sake of developers.

One of the main issues was converting strings from UTF-8 to the locale
encoding. If you don't know what attributes are strings you cannot
convert them automatically. You would have to some kind of 'get_string'
accessor function or just require the developer to either run their
web server in the UTF-8 locale or convert strings manually somehow. We
wanted to simply return a PHP array with all of the elements properly
typed. Not just strings but integers as integers and large integers as
double and boolean as 1 or 0, etc.

Another issue, as Howard mentions, are GUI tools that work with arbitrary
attributes. The classic example is trying to determine which values are
binary so that you don't print garbage at the user. IIRC ldapsearch does
this by looking at the value and trying to determine if it contains only
ascii characters. If it does not, then it base64 encodes it first.

As for retrieving the schema from the server at runtime, we did not do
that. Not just for performance reasons but more so because there is no
standard way to retrieve that information, it is awkward to do so or
you cannot do it at all (e.g. RootDSE attributes in AD have no schema
definitions).

Instead, we attack the problem at three levels:

1) By default all attribute definitions are multivalued binary. This
provides full comapibility with the current API.

2) Very common attributes are hardcoded into the library. Types
are string, boolean, binary, time, int32, and int64 with flags for
single-valued, case-exact and dn. We have a script that can query AD and
produce C declarations for all attributes on a set of DNs. Currently we
hard code about 150 attributes.

3) We allow the user to set attribute definitions at runtime using a
'plexcel_set_attrdef' function. These defs are associated with the
current context object and are released with it.

We think this strategy is quite reasonable in terms of performance and
complexity. It is backwards compatible by default. Someone has to have
"attribute intelligence". It can either be the developer or the API. In
the later case the developer can put that "attribute intellegence"
into a list of set_attrdef calls somewhere and not have to think about
it all the time.

Mike

-- 
Michael B Allen
PHP Active Directory SSO
http://www.ioplex.com/

---

From: Hallvard B Furuseth <[email-masked]>
Date: Wed, 21 Feb 2007 19:00:43 +0100

> be able to say that the 1st AVA of the 2nd RDN of the DN component

No reason a server couldn't say that too, in the diagnosticMessage
field.  OTOH a library can also provide detailed error info which
a program can parse.

-- 
Regards,
Hallvard

---

From: Kurt Zeilenga <[email-masked]>
Date: Wed, 21 Feb 2007 10:08:26 -0800

> field.  OTOH a library can also provide detailed error info which

Another good reason to provide client side validation is to facilitate
conformance with the LDAP specification... RFC 4511, Section 4.1.5
does say:
    Clients MUST only send attribute values in a request that are valid
    according to the syntax defined for the attributes.

Of course, it's just wishful thinking that clients will actually
adhere to this MUST, which is why there are server side validation
requirements as well.

-- Kurt

---

From: Pierangelo Masarati <[email-masked]>
Date: Wed, 21 Feb 2007 19:48:52 +0100

> you cannot do it at all (e.g. RootDSE attributes in AD have no schema

There __is__ a standard way, it's described in RFC4512 secton 4.4; the
fact that some implementors don't comply with it, or do not populate
values as indicated in the standard is a totally different business.

p.

Ing. Pierangelo Masarati
OpenLDAP Core Team

SysNet s.n.c.
Via Dossi, 8 - 27100 Pavia - ITALIA
http://www.sys-net.it
------------------------------------------
Office:   +39.02.23998309
Mobile:   +39.333.4963172
Email:    [email-masked]
------------------------------------------

---

From: simo <[email-masked]>
Date: Wed, 21 Feb 2007 14:07:08 -0500

> values as indicated in the standard is a totally different business.

Last I checked AD supported subschemaSubentry, I don't think that
changed overnight. I can't say the same of the std openLdap installation
you find on many distributions out there :)

I am not sure to what attributes Michael refer, but I guess that even if
they are not defined they can be easily hard coded eventually.

however, I am not really keen into entering a debate about standard
implementations, I got some insight on what a client side validation API
might be good at. I am still unsure if the benefits are enough to
justify the added complexity but at least there is some substance there.

Anyone have any other example of the benefits a client side API can
bring?

Simo.

---

From: Michael B Allen <[email-masked]>
Date: Wed, 21 Feb 2007 15:11:29 -0500

On Wed, 21 Feb 2007 14:07:08 -0500

> you find on many distributions out there :)

Actually I'm wrong. Last time I looked at this I didn't think ADs
subSchemaSubEntry had all of the necessary information. But looking at
it again it apparently does. I think I was orignally concerned about not
being able to filter by name. But an installer could download and store
the defs for the library to load locally at runtime. I think that would
perform pretty well especially if they were hash indexed by name.

> they are not defined they can be easily hard coded eventually.

The RootDSE attributes are not listed in subSchemaSubEntry or
CN=Schema,CN=Configuration,DC=X under AD. Not sure why. I think it has
something to do with bootstrapping the DIT.

Mike

-- 
Michael B Allen
PHP Active Directory SSO
http://www.ioplex.com/

---

From: Pete Rowley <[email-masked]>
Date: Wed, 21 Feb 2007 13:53:34 -0800

> adhere to this MUST, which is why there are server side validation

It would certainly help clients to be more portable and deal with 
incompatibilities between servers, for example, servers which redefine 
standard attributes to be single valued etc., or that disagree in other 
ways about schema. Currently clients tend to be written to assume things 
about schema that reality reveals are not safe things to assume.

-- 
Pete