Subject: RE: LDUP Working Group Agenda - Summary of objections to requirem ents draft

From: "Lloyd, Alan" <[email-masked]>
Date: Sun, 30 Jul 2000 18:51:07 +1000

This is good and should be discussed.

Lets put the goals down first - can the LDUP and URP standards make
scaleable, reliable directory information systems for business use?

I aggree and have voiced the opinion that once we go "sub atomic" on
directory objects/entries - then that is implementation - not
standardisation. For instance we dismantle in every DSA a complete DSAs
NameSpace into RDB tables (32 patents) - so we can automatically index, 2
phase commit, etc.. we can do database replication (sub atomic - product
feature) and ,DISP and DSP writethrough (standards feature).

Can any vendor/person  on this list confirm their desire to do directory
entry sub - atomic interoperability testing.. and waht are the tests when
they can do that.

The rules of the IETF (IMHO) state that 2 or more implementations must
interwork - to make an RFC.
 Can those involved say so.. otherwise -  please consider..

Something which is "lightweight" cannot be "Unpredictable" and
"Preposterously complex".

regards alan

-----Original Message-----
From: Albert Langer [mailto:[email-masked]]
Sent: Thursday, July 27, 2000 4:16 PM
To: 'Christopher Apple'; [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: RE: LDUP Working Group Agenda - Summary of objections to
requirements draft

[Chris]
I. WG Deliverables Review

"LDAP Replication Requirements"
   http://www.ietf.org/internet-drafts/draft-ietf-ldup-replica-req-03.txt
   Editor(s): Russ Weiser, Ellen Stokes

[...]

III. Discussion of LDUP Requirements Document

[Albert]
I am also sending this to LDAPEXT as the direction taken by LDUP could have
major consequences for LDAPEXT (eg the current proposals for ACL simply will
not replicate correctly and future support for transactions will be
difficult if not impossible).

Above link shows:

***
This Internet-Draft has expired and is no longer available.

Unrevised documents placed in the Internet-Drafts directories have a
maximum life of six months. After that time, they must be updated, or
they will be deleted. This document was deleted on July 17, 2000.
***

A copy of the "final" version, which expired on 21 April 2000, is in the
LDUP email archive at:

http://www.imc.org/ietf-ldup/mail-archive/msg00471.html

Despite assurances from at least one person who should know, I remain quite
convinced that members of the LDUP WG have NOT read it carefully. It is
especially unlikely that anybody else has read it recently, since it expired
two months ago and has now been deleted.

In view of agenda item III, I strongly urge that you do read it again,
carefully.

As I cannot attend the discussion, I have included a summary of my formal
objection to it below.

SUMMARY OF OBJECTIONS TO REQUIREMENTS DRAFT

The three key points are:

1) There is no requirement for convergence or "eventual consistency".

This looks like just poor expression, but in fact the LDUP architecture and
Update Reconciliation Procedures do specify proposed standards that
guarantee long term divergence by relying on timestamps and allowing DSAs to
transmit changes out of order and drop changes when clocks are out of sync.
This is easily fixed, once a requirement to fix it is agreed on.

Details on how to fix it based on the Coda replication protocols also
adopted by Active Directory, and a semi-formal proof that the fix would be
robust in the face of DSAs crashing and being restored from backups, network
partitioning etc etc, is included in my draft below. That fix is also
consistent with the rest of the current architecture and URP and would not
require major re-work of existing drafts.

2) There is no requirement for atomic operations.

Again this is obscured by poor expression and nonsensical definitions, but
in fact the architecture and URP drafts merge changes to individual
attribute values made concurrently at different replicas. The fact that this
obviously breaks the ACL standards being developed in LDUP, despite an
overlap in authorship between the two documents, strongly confirms that the
consequences for existing applications are simply not understand and should
be studied through a requirements analysis by actively explaining the
implications and soliciting input from other areas (operations etc) that may
be affected.

Fixing this would require substantial changes to the current architecture
and URP. I have sketched one possible way to do so in the draft below.

3) There is no requirement to support mandatory operational attributes of
LDAP.

The operational attribute "modifiersName" cannot be supported meaningfully
as nobody in particular can be said to be responsible for a change that has
in fact been merged from two or more concurrent changes made independently
and without knowledge of each other. This severely complicates system
administration as the first thing anybody would want to know after receiving
a problem report is "who changed what".

I believe that point 1) would be taken for granted by anybody thinking about
LDUP requirements. Until somebody presents an argument against convergence
or "eventual consistency" I see no point in even trying to present an
argument for it. The current approach is simply absurd.

Point 3) is also pretty obvious and is just one of many consequences of
point 2.

On point 2, there are plausible arguments (which I disagree with) for
breaking the current LDAP/X.500 data model. If the WG does intend to do
that, it has an obligation to clearly explain its intentions in a
requirements draft and actively solicit input.

I cannot express the argument against doing so more clearly than has already
been done, long ago, without adequate response, in the following comments
"Re: LDUP warmup exercise: atomicity in LDAPv3", from Tim Howes (16 December
1998):

***
"Each LDAP operation (add, modify, delete, moddn) as
a whole is atomic. The whole operation either happens
or it doesn't. Changes cannot be half-applied to any
single LDAP server.

The replication consistency model must assume and
build on this basic fact to define how multiple LDAP
replicas converge to the same state over time, in
the absence of additional changes. This kind of loose
consistency model is pretty fundamental to the notion
of a directory.

My two cents on what's important in a replication
consistency model are that it must be 1) predictable,
and that it should 2) make some kind of sense to
people using the system.

All this talk of consistency at different levels
(e.g., between applications using the directory at
the same time) is a red herring. Our job is to define
a consistency model for the directory itself. Some
applications may find this model sufficient for their
needs. Others may have to build more elaborate models
on top. But let's start with the basics.    -- Tim"

http://www.imc.org/ietf-ldup/mail-archive/msg00214.html

***
In my view, both an explicit requirement for atomic operations and a
requirement that the results be 1) predictable and 2) make some kind of
sense to users, should be in the final requirements draft.

The consistency model of URP, was summarized in Alison's "Contribution to
Profiles Document (Consistency Discussion)":

http://www.imc.org/ietf-ldup/mail-archive/msg00548.html

The attached Word version, with more easily read tables, is:

http://www.imc.org/ietf-ldup/mail-archive/doc00000.doc

In my view it is:

1) Unpredictable. See the explanation of "Extraordinary States" and
"Transient Extraordinary States" above.

2) Preposterously complex. See the extensive use of procedural pseudo code
for a protocol that simply defies plain description, in the URP draft:

http://www.ietf.org/internet-drafts/draft-ietf-ldup-urp-03.txt

In reviewing the LDUP archives I noted that a number of people had expressed
various reservations, especially concerning the consistency model for
replication,
but subsequently ceased active participation. Hopefully some of them may
still be
involved in LDAP-EXT and could therefore see this and decide to take part in
the
discussion.

See also my individual submission to LDUP:

http://www.ietf.org/internet-drafts/draft-langer-ldup-mdcr-00.txt

and my response to the WG chair's request for status reports, which contains
links to all relevant discussion of that objection prior to the
document expiring:

http://www.imc.org/ietf-ldup/mail-archive/msg00561.html

Subsequent discussion can be seen by following from the thread of
that message in the archive.

---

From: "Albert Langer" <[email-masked]>
Date: Sun, 30 Jul 2000 22:52:01 +1000

[Alan]

[Albert]
Thanks for the agreement. However I would like to focus on user requirements
rather than vendor desires. When reviewing the LDUP archives I noticed your
views and others expressing various "reservations", however the previous
discussion was focussed on detail design and in the context of what
vendors desired to implement. I believe that focus on vendor desires rather
than
user requirements is what led to the fundamental technical errors in the
current
architecture and URP drafts.

Now that the requirements draft is up for "final call" (regrettably AFTER
extensive architecture and design work) a decision MUST be made as to
whether
these ARE the requirements that users of the directory service have, that
must be satisfied by the future proposed standards for meeting them.

Let's focus on that REQUIREMENTS issue. It's what is on the agenda NOW.

The basic problem, underlying all three of the key issues I summarized,
is that the requirements draft simply contains NO requirements whatsoever
relevant to multi-master replication.

It does start from similar goals to your "scaleable, reliable directory
information systems for business use". But then it simply does not state
what is required from the standards in order to achieve those goals.

All it requires of multi-master standards is:

      5.6  The replication model MUST support both master-slave and
           authoritative multi-updateable replica relationships.

Perhaps that's why others didn't find anything to object to? They may
have only been looking for requirements that could cause difficulties
in their vendor implementations.

But even that is completely botched by the meaningless definition:

Updateable Replica - A Non-authoritative read-writeable copy of the
      replicated information. Such that during conflict resolution a
      authoritative master takes precedents in resolving conflicts.

Simple proof reading is clearly required before final call.

But how could ANYONE have actually READ that and thought that either the
document or the intentions behind it, was ready for "final call"?

There is obviously no possibility of an "authoritative master" taking
precedence in resolving conflicts, since the definition of multi-master
replication, copied from the WG charter correctly specifies:

Multi-Master Replication - A replication model where entries can be
      written and updated on any of several updateable replica copies
      without requiring communication with other updateable replicas
      before the write or update is performed.

There is no such thing as a "non-authoritative read-writeable copy"
in multi-master.

Nobody with the slightest understanding of what the WG is actually
supposed to be working on could have such a complete misconception about
updateable replicas, regardless of their expertize in other areas.

Of course its rather hard for people to read the draft, since it is being
discussed while expired and deleted, and is only available from the
LDUP email archives:

[Alan]

regards alan

[Albert]
My understanding is that the current architecture and design approach is the
result of a "vendor bake-off", so there must be sufficient vendor support
for interoperability testing. Presumably Telstra is planning an
implementation
since the URP design came from there and they already have a simulator.
Likewise
Novell looks strongly committed together with Netscape and Sun. Even Oracle
seems to
be involved despite knowing a thing or two about the consequences of
non-atomicity
in databases. Of the majors, only Microsoft seems to have stayed well clear,
perhaps laughing quietly on the way to the bank.

Despite the complexities, I don't think the URP draft is unimplementable at
all.

Presumably there would have been loud screams from the various vendors if it
was.

The criteria for any testing would presumably be based on whether an
implementation
complies with the proposed standards. Acceptance of those proposed standards
would
presumably depend on whether they satisfy the requirements in the
requirements
document. Since that does not actually state ANY meaningful requirement for
multi-master replication, it would not be difficult not to meet those
requirements.

Actually, as currently drafted, with not even convergence clearly required,
an
implementation could pass ANY test suite by simply dropping all updates and
returning:
the error code serverClocksOutOfSync (72). See p40 of:

http://www.ietf.org/internet-drafts/draft-ietf-ldup-model-04.txt

That's what happens when you let implementors define their own requirements,
though
it is rather more spectacular than usual in this case, since they have come
up
with something simply absurd.

It is the users, including system administrators, who will be hit by the
complexities
resulting from non-convergence, inconsistency and no audit trail.

There is no shortage of vendor products that are "unpredictable" and
"preposterously
complex" to use. Sticking an IETF label of approved LDAP standard on such
products
won't help them as much as they seem to think.

The question is whether what the vendors plan to implement will in fact meet
user
requirements for LDAP replication (whether "business" or other) and whether
it will
negatively impact existing LDAP standards.

If there is a user requirement and LDAP standard for guaranteed eventual
convergence
it doesn't matter a damn if every potential implementor is ready, willing
and able to
do something else - the result would be useless to users.

Likewise if there is a user requirement for atomic operations or
modifiersName. Given
the LDAP standards, anything that doesn't provide that just isn't LDAP.

BTW a "proposed standard" from a WG does not require 2 or more
implementations.
That is only essential at a later stage of draft standard.

An Informational RFC need not have any implementations at all. It can just
be a poem
provided it offers some information of interest to the internet community.

The requirements draft now being considered for WG "final call" is naturally
intended as
an "Informational" RFC and would normally go through without much external
review.

Unfortunately it does not provide ANY useful information as to what
requirements the LDUP
WG intends to meet for its main focus on multi-master LDAP replication,
despite considerable
detail on matters common to single-master and multi-master replication.

If unchallenged, that would result in the WG continuing to focus on vendor
desires
instead of user requirements in completing their proposed standards.

Since a requirements draft is intended to guide the development of
standards, and this
one does not, it should be published only with simultaneous application of
the status
"Historic" (your spelling may vary).