Subject: [ldapext] UTF-8 full support in LDIF / LDIF v2

From: Yves Dorfsman <[email-masked]>
Date: Sat, 21 Mar 2009 01:32:09 -0600

Hi,

I'd like to re-start this thread.

I have been re-reading the earlier messages, and have put some thought into 
it as well. I have also been communicating with the RFC-editor and Alexey 
Melnikov (one of the Area Directors for LDAP). I have also taken a stab at 
the new text to jumpstart the discussion.

Scope:
I personally would prefer to address the UTF-8 support only, and possibly 
other minor issues, but no other major change. My reasoning is that:
-the change is relatively simple
-the lack of UTF-8 support is a problem that affects me directly
-I understand the problem
-adding other major functionalities will probably delay the RFC, while we 
could get this one out relatively fast, while working on the other 
functionalities for another version.

Alexey said that he'd like to have all the options on the table.

Version scheme:
Last year I wrote that the version number is limited to one digit. That was 
a mistake, I re-read RFC 2234 (the specification of Backus-Naur Form), and 
what is in the current RFC is one digit or more, so this is a non-issue.

Escaping UTF-8 characters:
RFC 2253 (UTF-8 String Representation of Distinguished Names
) allows for escaping characters with backslashes and hex numbers. I can see 
the point when working from the command line, and say your terminal is not 
set properly, or you don't have the appropriate keyboard, but I am not sure 
about files... What do you guys think ?

Authorship:
I have tried to contact Gordon Good, the original author of RFC 2849,
but have not heard from him yet (could be due to spam filters). The 
RFC-Editor and Alexey say there are procedures around that, and we can leave 
that as a last minute item.

Example in UTF-8:
The RFC-Editor is very clear on this, RFCs are ASCII only but we can add a 
postscript file with examples containing UTF-8 if we want. I am not sure 
there is much value, this is pretty trivial.

Removing some paragraphs:
Should we remove some paragraph that don't seem to be relevant any more, 
such as:
"   The application/directory MIME content-type [RFC2425] is a general
    framework and format for conveying directory information, and is
    independent of any particular directory service.  The LDIF format is
    a simpler format which is perhaps easier to create, and may also be
    used, as noted, to describe a set of changes to be applied to a
    directory.
"

Expired draft:
Refence [Armijo00] is a draft expired in 2001. It is used in example 7. Is 
this still relevant ?

RFC4525:
I noticed that RFC 4525 has updated the LDIF definition. Should this be 
included in this RFC ? I have created an extra file with its inclusion.

Here are the two versions I have created:
http://www.sollers.ca/hg/ldif-utf8/file/d307d875966f/proposal.txt#l1
and a side by side diff:
http://www.sollers.ca/projects/ldif-utf8/files/proposal-diff.html

and with the addition of rfc 4525:
http://www.sollers.ca/hg/ldif-utf8/file/213f3b5dcd86/proposal.txt#l1
side by side diff:
http://www.sollers.ca/projects/ldif-utf8/files/rfc4525addon.html

-- 
Yves.
http://www.sollers.ca/

---

From: Kurt Zeilenga <[email-masked]>
Date: Sat, 21 Mar 2009 13:30:27 -0700

On Mar 21, 2009, at 12:32 AM, Yves Dorfsman wrote:

> I have been re-reading the earlier messages, and have put some  
> thought into it as well. I have also been communicating with the RFC- 
> editor and Alexey Melnikov (one of the Area Directors for LDAP). I  
> have also taken a stab at the new text to jumpstart the discussion.
>
>
> Scope:
> I personally would prefer to address the UTF-8 support only, and  
> possibly other minor issues, but no other major change.

My view is that I rather not suffer the pain of updating the LDIF  
format multiple times, and certainly rather not suffer significant  
pain for just minor change.

Also, I might note that adding just UTF-8 is not totally a minor  
change, there are a few oddities to consider.  For instance, what  
would be allows for end-of-line characters?  Would we use Net-UTF8?   
(I hope not as Net-UTF8 implies normalization and that goes against  
interchanging arbitrary sequences of Unicode code points.)

LDIF primary purpose is for interchange of directory data.  Just  
adding UTF-8 support doesn't allow for any additional interchange of  
information.  The UTF-8 change is for secondary purposes, to allow  
humans to more easily see what they are interchanging, to allow humans  
to directly modify the LDIF.

I rather spend my time on adding features that do allow for the a  
interchange of more directory data.  Also LDIF ought to have similar  
extensibility as LDAP itself has.

> My reasoning is that:
> -adding other major functionalities will probably delay the RFC,  
> while we could get this one out relatively fast, while working on  
> the other functionalities for another version.
>

Other items include (this is not intended to be exhaustive, I suspect  
others have more items):

Extending LDIF to support all LDAP requests, e.g., an LDAP Transaction

Extending LDIF to support all LDAP responses, e.g., Entries,  
References, Intermediate Responses, and Result returned in response to  
an LDAP Search request.

Extending LDIF to support XML values.

There were also suggestions of adding a "charset" specification (yuk).

>
>
>
> Version scheme:
> Last year I wrote that the version number is limited to one digit.  
> That was a mistake, I re-read RFC 2234 (the specification of Backus- 
> Naur Form), and what is in the current RFC is one digit or more, so  
> this is a non-issue.

This, I think, is an error.  Generally errors should be fixed when  
revising a document.
> ) allows for escaping characters with backslashes and hex numbers. I  
> can see the point when working from the command line, and say your  
> terminal is not set properly, or you don't have the appropriate  
> keyboard, but I am not sure about files... What do you guys think ?

I rather avoid adding escaping.

> RFC-Editor and Alexey say there are procedures around that, and we  
> can leave that as a last minute item.

A non-issue.

> Example in UTF-8:
> The RFC-Editor is very clear on this, RFCs are ASCII only but we can  
> add a postscript file with examples containing UTF-8 if we want. I  
> am not sure there is much value, this is pretty trivial.

(Though I rather the IETF would change the RFC format from ASCII to  
UTF-8, that's another fight.)

There are various conventions for providing examples of UTF-8  
protocols (and formats).  IIRC, there's an RFC about how this (though  
I don't recall its number).   And I think at least once such example  
ought to be included.

> Should we remove some paragraph that don't seem to be relevant any  
> more, such as:

I would oppose removing this text.  Any replacement of the LDIF  
specification needs to consider the impact on use of LDIF in MIME  
encoded contents of the application/directory type.

>
>
>
> Expired draft:
> Refence [Armijo00] is a draft expired in 2001. It is used in example  
> 7. Is this still relevant ?

The revised document should provide more relevant examples.  In  
particular, examples should use standard track controls.  So this  
(informative) reference ought to be replaced accordingly based upon  
which standard track controls are used in examples.

> RFC4525:
> I noticed that RFC 4525 has updated the LDIF definition. Should this  
> be included in this RFC ?

Yes, as LDIFv1 is formally RFC 2829 as updated by RFC 4525 (that's  
what UPDATES means).

> I have created an extra file with its inclusion.
>
>
> _______________________________________________
> Ldapext mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ldapext

---

From: Yves Dorfsman <[email-masked]>
Date: Mon, 23 Mar 2009 23:33:58 -0600

Kurt Zeilenga wrote:
> My view is that I rather not suffer the pain of updating the LDIF format 
> multiple times, and certainly rather not suffer significant pain for 
> just minor change.

I like the idea of addressing one change at a time. You address that one, 
and spend all the time and energy on making sure you got it right, then when 
that's done, you move on to the next...

> Also, I might note that adding just UTF-8 is not totally a minor change, 
> there are a few oddities to consider.  For instance, what would be 
> allows for end-of-line characters?  Would we use Net-UTF8?  (I hope not 
> as Net-UTF8 implies normalization and that goes against interchanging 
> arbitrary sequences of Unicode code points.)

Agreed, this is not minor, what I meant was I'd like to do UTF-8 + 
*possibly* other minor changes, but I would rather not do UTF-8 + other 
major changes. Sorry if I wasn't clear.

Having said that, if we get to a consensus on other changes that we can add 
to my changes, or even if somebody wants to take my proposition and add 
other change to it, I'm fine with that. My issue is UTF-8, that's what drove 
me here, that's what I want to be fixed.

> LDIF primary purpose is for interchange of directory data.  Just adding 
> UTF-8 support doesn't allow for any additional interchange of 

Good point, but UTF-8 sure helps for debugging ! Diffing base64 is 
completely useless.

Also you could argue that the beauty of LDIF is its simplicity for humans. 
If we don't care about that aspect, we could completely give up LDIF, and go 
with something more complex like an XML format, or even a binary one.

> extensibility as LDAP itself has.

Let's start a list, and see who wants what in/out...

> Extending LDIF to support all LDAP responses, e.g., Entries, References, 
> Intermediate Responses, and Result returned in response to an LDAP 
> Search request.

> Extending LDIF to support XML values.

Really ?

> There were also suggestions of adding a "charset" specification (yuk).

Is there something we cannot do with Unicode ? I don't understand the 
advantage of a "charset" keyword. If anything, I'd rather look at an 
"encoding" keyword to support the different encodings of Unicode, for east 
asian scripts for example which apparently takes a lot less space when 
encoded in UTF-16. But, UTF-16 in itself is quite a bit more complex than 
UTF-8, requiring a byte order mark etc... I was thinking this could wait a 
future version.

>> That was a mistake, I re-read RFC 2234 (the specification of 
>> Backus-Naur Form), and what is in the current RFC is one digit or 
>> more, so this is a non-issue.
> 

No, *I* am the one who made a mistake, there is nothing wrong with the 
current spec. for the version number, back in december, I thought there was, 
but I was wrong. It's a non-issue.

> I rather avoid adding escaping.

Agreed, I'd just like to confirm this isn't a personal bias.

>> add a postscript file with examples containing UTF-8 if we want. I am 
>> not sure there is much value, this is pretty trivial.
> 

No kidding.

> 
> There are various conventions for providing examples of UTF-8 protocols 
> (and formats).  IIRC, there's an RFC about how this (though I don't 
> recall its number).   And I think at least once such example ought to be 
> included.

Yes, I believe you need to add a PDF or PS version to the original. I can 
look at that. Is there a tradition on how to pick the names for the example, 
or is it completely random ?

> encoded contents of the application/directory type.

It seems to me that this paragraph is giving historical information and 
justifying some of the reasons behind LDIF... Am I wrong here.

If I am making an error and it's worth keeping, than that's fine.

> which standard track controls are used in examples.

I could not find any new version of this document, of a replacement for it. 
As this just become standard practice ? Or is there a written standard for it ?

> Yes, as LDIFv1 is formally RFC 2829 as updated by RFC 4525 (that's what 
> UPDATES means).
> 

Ok. I'll just merge the two files an drepost then.

-- 
Yves.
http://www.sollers.ca/

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 24 Mar 2009 16:03:35 -0700

> Kurt Zeilenga wrote:

 [...]

> "encoding" keyword to support the different encodings of Unicode, for 
> east asian scripts for example which apparently takes a lot less space 
> when encoded in UTF-16. But, UTF-16 in itself is quite a bit more 
> complex than UTF-8, requiring a byte order mark etc... I was thinking 
> this could wait a future version.

I think Unicode is pretty much won the charset war and UTF-8 would be 
the best thing to use.
But I think Kurt was just mentioning this as an idea that was suggested.

Just my 2p (as an individual contributor).
 [...]

>> UTF-8, that's another fight.)

Another debate for another place and time ;-)

> Yes, I believe you need to add a PDF or PS version to the original. I 
> can look at that. Is there a tradition on how to pick the names for 
> the example, or is it completely random ?

There is no tradition, you are free to use whatever you like.
 [...]

---

From: Kurt Zeilenga <[email-masked]>
Date: Tue, 24 Mar 2009 17:04:51 -0700

On Mar 23, 2009, at 10:33 PM, Yves Dorfsman wrote:
>> adding UTF-8 support doesn't allow for any additional interchange  
>> of information.  The UTF-8 change is for secondary purposes, to  
>> allow humans to more easily see what they are interchanging, to  
>> allow humans to directly modify the LDIF.
>
> Also you could argue that the beauty of LDIF is its simplicity for  
> humans. If we don't care about that aspect, we could completely give  
> up LDIF, and go with something more complex like an XML format, or  
> even a binary one.

One of the problems with using UTF-8 directly is that the UTF-8 form  
used within the human's editor might be quite different that the UTF-8  
form the human desires to store in the directory.  Or the human's  
editor cannot display any and all sequences of UTF-8 characters, some  
of which might not even be well-formed text, as stored in the directory.

There are other issues, such as how to deal with BIDI values.

While I am sure there are persons in the IETF that have the expertise  
to get a UTF-8 LDIF right, I am quite unsure of time/energy/ 
willingness to contribute to this effort.

>> interchange of more directory data.  Also LDIF ought to have  
>> similar extensibility as LDAP itself has.
>
>> References, Intermediate Responses, and Result returned in response  
>> to an LDAP Search request.
>

See http://www.xmled.info/drafts/draft-sciberras-xed-eldif-05.txt

> Is there something we cannot do with Unicode ? I don't understand  
> the advantage of a "charset" keyword. If anything, I'd rather look  
> at an "encoding" keyword to support the different encodings of  
> Unicode, for east asian scripts for example which apparently takes a  
> lot less space when encoded in UTF-16. But, UTF-16 in itself is  
> quite a bit more complex than UTF-8, requiring a byte order mark  
> etc... I was thinking this could wait a future version.

As Alexey noted, I was only listing what someone else suggesting.  I  
don't support adding a charset option.

>
> Yes, I believe you need to add a PDF or PS version to the original.  
> I can look at that. Is there a tradition on how to pick the names  
> for the example, or is it completely random ?

I think the ASCII I-D/RFC needs to have examples that use some sort of  
escaping mechanism (for the purposes of the example)

	value: \C3\BC

	where C3 BC is the 2 octet UTF-8 encoding of the Unicode LATIN SMALL  
LETTER U WITH DIAERESIS (U+00FC) codepoint.

	value: \75\CC\88

	where 75 is the 1 octet UTF-8 encoding of Unicode LATIN SMALL LETTER  
U (0075) codepoint and CC 88 is the 2 octet UTF-8
         encoding of the Unicode COMBINING DIAERESIS (0308) codepoint.

	Note: The hex escaping above is used only for illustration purposes.   
The LDIF content should contain the actual UTF-8
         encoded Unicode codepoints.

The PDF/PS version would not use the escaping.

> It seems to me that this paragraph is giving historical information  
> and justifying some of the reasons behind LDIF... Am I wrong here.
>
>>> Refence [Armijo00] is a draft expired in 2001. It is used in  
>>> example 7. Is this still relevant ?
> I could not find any new version of this document, of a replacement  
> for it. As this just become standard practice ? Or is there a  
> written standard for it ?

My point is that this example should be replaced with one that uses a  
standard track control, such as the LDAP Assertion Control [RFC4528].

>
>
>
>>> RFC4525:
>>> I noticed that RFC 4525 has updated the LDIF definition. Should  
>>> this be included in this RFC ?

---

From: Kurt Zeilenga <[email-masked]>
Date: Tue, 24 Mar 2009 18:58:42 -0700

After thinking about this some more I've come to the conclusion that  
the best one could do with UTF-8 and LDIF is to allow the value to  
contain any (but a few) code points.  The file might not be always  
well-formed text, and hence may not always be displayable.  I do not  
think it feasible to create an LDIF representing arbitrary Unicode  
attribute values using a off-the-shelf Unicode-aware text editor.  For  
instance, you could have one value that requires composed characters  
but the editor only produces decomposed characters, or vice versa.

I note my intent is not to discourage you from putting a draft  
together.  I'm just concerned that the result might not adequately  
address the issue of concern.  I suggest you be try to describe the  
problem you are trying to solve in the I-D.

Regards, Kurt

---

From: Yves Dorfsman <[email-masked]>
Date: Wed, 25 Mar 2009 23:14:10 -0600

> But I think Kurt was just mentioning this as an idea that was suggested.

I went back through the mailing list archive, "charset" came up, but I can't 
make sense of who started with it. Assuming we go ahead with UTF-8, does 
anybody here have a problem with NOT implementing "charset", to use Unicode 
/ UTF-8 only ?

> There is no tradition, you are free to use whatever you like.

Thanks.

-- 
Yves.
http://www.sollers.ca/

---

From: Yves Dorfsman <[email-masked]>
Date: Wed, 25 Mar 2009 23:48:41 -0600

> problem you are trying to solve in the I-D.

Thanks Kurt, I actually appreciate that you come up with these issues, as I 
had not thought of them all. This also makes me think that we should really 
focus on UTF-8 only for this revision, and make sure we look at all the angles.

> After thinking about this some more I've come to the conclusion that the 
> best one could do with UTF-8 and LDIF is to allow the value to contain 
> any (but a few) code points.  The file might not be always well-formed 
> text, and hence may not always be displayable.  I do not think it 
> feasible to create an LDIF representing arbitrary Unicode attribute 
> values using a off-the-shelf Unicode-aware text editor.  For instance, 
> you could have one value that requires composed characters but the 
> editor only produces decomposed characters, or vice versa.

I really do think the issue here has to do with editors, and to a certain 
extent, ldap client. But then, I expect the people who have a need for these 
characters will be equiped with the right software.

I have just tried to import these two LDIF entries (at separate time) into 
openldap:
##################################################################
## Ali Baba
dn:: Y2492LnZhNmKINio2KfYqNinLG91PVBlb3BsZSxkYz16aW91cCxkYz1jb20=
cn:: 2LnZhNmKINio2KfYqNin
givenName:: 2KjYp9io2Kc=
objectclass: person
objectclass: organizationalPerson
objectclass: inetOrgPerson
rfc822Mailbox: [email-masked]
sn:: 2LnZhNmK
##################################################################
dn: cn=علي بابا,ou=People,dc=zioup,dc=com
cn: علي بابا
givenName: علي
objectclass: person
objectclass: organizationalPerson
objectclass: inetOrgPerson
rfc822Mailbox: [email-masked]
sn: بابا
##################################################################

I then accessed openldap with Mozilla Thunderbird, and in both cases, 
Thunderbird displayed the name properly, from right to left. Note that the 
vim editor displays the arabic characters from left to right (wrong, there 
might be a way to make vim work, but it's not the issue here), and the gnome 
editor gedit displays them from right to left, so properly (openldap is 
happy to import non base64 encoded characters, but it follows the RFC in a 
strict manner when exporting and base64 encode those).

I have not looked at composed characters yet, but will.

If we leave the editor issue on the side, is there any reason why UTF-8 
would give us problems ? LDAP itself supports UTF-8, has there be any 
problem with it ?

If anybody has a more practical experience with bidirectional and composed 
characters in LDAP, I'd appreciate their help to do some more testing.

-- 
Yves.
http://www.sollers.ca/

---

From: Yves Dorfsman <[email-masked]>
Date: Wed, 25 Mar 2009 23:57:02 -0600

> See http://www.xmled.info/drafts/draft-sciberras-xed-eldif-05.txt

Interesting, I think this is a valid issue, although I think XML is the 
wrong solution. Any reason why the draft never made it to an RFC ?

What other solutions could be used to make multi-line entries ?
Would quoting a la python work, or would that add too much complexity, too 
much of a change from the current LDIF ?

> LETTER U WITH DIAERESIS (U+00FC) codepoint.

Thanks, I'll look at that. Anybody knows of an RFC that use something like 
this in their example, from the top of your head ?

-- 
Yves.
http://www.sollers.ca/

---

From: Steven Legg <[email-masked]>
Date: Thu, 26 Mar 2009 17:30:52 +1100

Hi Yves,

Yves Dorfsman wrote:
> wrong solution.

We created a solution for presenting directory attribute values that are
arbitrary UTF-8 character strings including line breaks. That accommodates
UTF-8 encoded XML documents, but would also be useful for GSER encodings
if the whitespace separator is extended to include line breaks.

> Any reason why the draft never made it to an RFC ?

It may yet. We've just been busy with other things and there hasn't been
external pressure to move it along.

> What other solutions could be used to make multi-line entries ?

Escaping the line break characters in the value and using the existing
LDIF line folding method would work. The output function can optionally
fold lines immediately after an escaped line break so that the number
of lines in the value matches the number of lines in the LDIF output.

Another solution is to drop into a literal mode until a special character
sequence is encountered (e.g., END-OF-VALUE). One must then take care to
escape the special sequence if it happens to occur in the attribute value.

Both of the above solutions require messing about with escape sequences,
whereas the simple prefix method we settled on doesn't need to.

> Would quoting a la python work, or would that add too much complexity, 
> too much of a change from the current LDIF ?

Can you present an example of what you mean ?

Regards,
Steven

>>     where C3 BC is the 2 octet UTF-8 encoding of the Unicode LATIN 
>> SMALL LETTER U WITH DIAERESIS (U+00FC) codepoint.
>>
>>     value: \75\CC\88
>>
> 
> Thanks, I'll look at that. Anybody knows of an RFC that use something 
> like this in their example, from the top of your head ?
>

---

From: Ludovic Poitou <[email-masked]>
Date: Thu, 26 Mar 2009 11:03:39 +0100

On Mar 26, 2009, at 6:14 AM, Yves Dorfsman wrote:

>>> at an
>> I think Unicode is pretty much won the charset war and UTF-8 would  
>> be the best thing to use.
> I went back through the mailing list archive, "charset" came up, but  
> I can't make sense of who started with it.

I probably did.
In europe there are lots of directory users and administrators that  
have non ascii data they need to transform to LDIF.
With simple scripts, turning the data to UTF-8 and then to base64  
encoded is a pain.
Allowing to specify a charset and then letting the tools doing the  
conversion to UTF-8 automatically could simplify their life.

I think this is getting less of an issue with modern desktops and OS,  
even though UTF-8 is not the default setting for a machine in French  
or German.

> Assuming we go ahead with UTF-8, does anybody here have a problem  
> with NOT implementing "charset", to use Unicode / UTF-8 only ?
No problem.

Ludovic.

>
>
>>> Yes, I believe you need to add a PDF or PS version to the  
>>> original. I can look at that. Is there a tradition on how to pick  
>>> the names for the example, or is it completely random ?

---
Ludovic Poitou                                    Sun Microsystems Inc.
OpenDS Community Lead               Directory Services
http://blogs.sun.com/Ludo/         Grenoble Engineering Center - France

Join OpenDS, https://opends.dev.java.net/servlets/ProjectMembershipRequest

Sun Microsystems requires the following notice:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NOTICE:  This email message is for the sole use of the intended
recipient(s) and may contain confidential and privileged information.
Any unauthorized review, use, disclosure or distribution is prohibited.
If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

---

From: Jim Willeke <[email-masked]>
Date: Thu, 26 Mar 2009 07:35:52 -0400

Is the suggestion to utilize and indicator similar to the HTTP charset
parameter (Content-Type: text/html; charset=utf-8) ?

-jim
Jim Willeke

2009/3/26 Ludovic Poitou <[email-masked]>

> I think Unicode is pretty much won the charset war and UTF-8 would be the
> best thing to use.
>
> I went back through the mailing list archive, "charset" came up, but I
> can't make sense of who started with it.
>
>
> I probably did.
> In europe there are lots of directory users and administrators that have
> non ascii data they need to transform to LDIF.
> With simple scripts, turning the data to UTF-8 and then to base64 encoded
> is a pain.
> I think this is getting less of an issue with modern desktops and OS, even
> though UTF-8 is not the default setting for a machine in French or German.
>
> Assuming we go ahead with UTF-8, does anybody here have a problem with NOT
> implementing "charset", to use Unicode / UTF-8 only ?
>
> No problem.
>
> Ludovic.
>
>
>

---

From: Yves Dorfsman <[email-masked]>
Date: Thu, 26 Mar 2009 07:38:04 -0600

> parameter (Content-Type: text/html; charset=utf-8) ?

I don't think we ever got it that well defined, but I guess, yes this would 
be one way of doing it.

I still think it is a bad idea. It make sense for http, because most html 
out there, even today, is not UTF-8, the migration is happening, but it has 
not happened. On the other hand, UTF-8 has been supported by LDAP for a long 
time, and is typically the default. As a matter of fact, LDIF already uses 
UTF-8 by default today, but the issue is that it has to be base64 encoded.

Jim: Do you have specific cases where using non-Unicode charsets would be an 
asset ?

PS: As mentioned earlier, I could see value in adding "encoding:" if it 
means an encoding of Unicode, but we need to keep in mind that the other 
encodings are more complicated to implement.
--
Yves.
http://www.sollers.ca/

---

From: Yves Dorfsman <[email-masked]>
Date: Thu, 26 Mar 2009 07:49:50 -0600

> external pressure to move it along.

Yes I was wondering about that, do we need multi-line values as work around 
because schemas aren't precise enough ?

> Can you present an example of what you mean ?

Python style would be usint a special character that then needs to be 
escaped, something like:
#####################################
organizationName: Smith, Smith
&
Smith
givenName: John
#####################################

or
#####################################
organizationName: 'Smith, Smith
&
Smith'
givenName: John
#####################################

But now that I think of it, it does look messy in the middle of an LDIF 
file. What about a mix, like:
#####################################
organizationName: 'Smith, Smith\' \
                   '&\' \
                   'Smith'
givenName: John
#####################################

I find it a lot more LDIFic...

Again, before we get into something like that, I'd like to have a better 
feeling for how much of a need we have for it ?

-- 
Yves.
http://www.sollers.ca/

---

From: Jim Willeke <[email-masked]>
Date: Thu, 26 Mar 2009 10:25:35 -0400

We, doing LDAP consulting, often run into these type of issues, but I
believe they could be covered utilizing UTF8.

Although I love the discussions, regardless of how individuals feel, these
discussions have already been and the decision was unicode.

The decision of the today's environment was also to use XML.

Not saying that either is right or wrong, but this is today's reality.

Why not use DSML or perhaps go wild and use protocol buffers. (
http://code.google.com/p/protobuf/)
I believe DSML would also handle the concern for LDAP transactions as is or
with some minor tweaks.

-jim
Jim Willeke

---

From: Steven Legg <[email-masked]>
Date: Fri, 27 Mar 2009 09:29:50 +1100

Hi Yves,

Yves Dorfsman wrote:
> Yes I was wondering about that, do we need multi-line values as work 
> around because schemas aren't precise enough ?

No, we need them because sheets of paper, computer screens and RFCs are
not infinitely wide. :-) Human-readability, line breaks and indenting tend
to go hand-in-hand.

>>> Would quoting a la python work, or would that add too much 
>>> complexity, too much of a change from the current LDIF ?
>>

And is starting to obscure what the value actually is, which any scheme
using escapes will do to greater or lesser degrees. The method Andrew
and I chose puts a '>' down the first column, but everything to the right
of that is literally the attribute value. It's easy to hand edit correctly,
and easy to process on input and output.

> feeling for how much of a need we have for it ?

We defined an extended LDIF because we need it, even if for now we are
only talking to ourselves. We are already storing XML documents in
directory attribute values so we need a convenient way to dump, edit
and load such values.

Regards,
Steven

---

From: Steven Legg <[email-masked]>
Date: Fri, 27 Mar 2009 09:55:09 +1100

Hi Jim,

Jim Willeke wrote:
> Although I love the discussions, regardless of how individuals feel, 
> these discussions have already been and the decision was unicode.
> 

Yep.

> Why not use DSML

The problem with DSML is that controls and extended operations have to
appear as the base64 encoding of the BER encoding, and directory attribute
values that contain XML have to have the markup escaped. I invented XLDAP
(http://www.xmled.info/drafts/draft-legg-xed-protocols-05.txt) so that
controls, extended operations and directory attribute values would all
appear as markup, without fiddly escaping and obscuring encodings.

 > or perhaps go wild and use protocol buffers.
> (http://code.google.com/p/protobuf/)

It seems to be doing much the same thing as what we can already do with
ASN.1 and GSER encodings.

> I believe DSML would also handle the concern for LDAP transactions as is 
> or with some minor tweaks.

If you don't mind base64 encodings of BER encodings. XLDAP does it better.

Regards,
Steven

> 
> -jim
> Jim Willeke
> 
> 
> On Thu, Mar 26, 2009 at 9:49 AM, Yves Dorfsman <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>     Steven Legg wrote:
> 
>                 See
>                 http://www.xmled.info/drafts/draft-sciberras-xed-eldif-05.txt
> 
> 
>     Python style would be usint a special character that then needs to
>     be escaped, something like:
>     But now that I think of it, it does look messy in the middle of an
>     LDIF file. What about a mix, like:
>     Again, before we get into something like that, I'd like to have a
>     better feeling for how much of a need we have for it ?
> 
>     -- 
>     Yves.
>     http://www.sollers.ca/
> 
>     [email-masked] <mailto:[email-masked]>
> ------------------------------------------------------------------------
>