Subject: [ldapext] nfsv4 vs the ldap consistency model

From: Leif Johansson <[email-masked]>
Date: Fri, 21 Nov 2008 21:38:54 +0100

So I had a talk with the nfsv4 wg today (and on their list earlier). The are 
trying to build AFS-style volume location for NFS using LDAP and I think they 
may have a problem with the LDAP consistency model: i.e "eventually
consistent" for a couple of their usecases.

I won't go into details unless people particularly care but I think we may
need an I-D (something for a bof in SF perhaps) describing a control which
asks for the result of an update operation to be  delayed until a certain 
criterion (eg in the form of a compare condition) is fulfilled on all 
"slaves". Implementation details would differ depending on the precise 
way replication is done for that vendor.

Does that sound like something you would

a) barf at
b) laugh at
c) ignore
d) review given that someone wrote it up?

	Cheers Leif
_______________________________________________
Ldapext mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ldapext

---

From: Howard Chu <[email-masked]>
Date: Fri, 21 Nov 2008 12:49:30 -0800

> d) review given that someone wrote it up?

Sounds like what's really wanted is a distributed transaction. Why not just 
call it that?

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/
_______________________________________________
Ldapext mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ldapext

---

From: "Liben, Michael \(A&E\)" <[email-masked]>
Date: Fri, 21 Nov 2008 15:53:58 -0500

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Howard Chu
Sent: Friday, November 21, 2008 3:50 PM
To: Leif Johansson
Cc: Ldapext list
Subject: Re: [ldapext] nfsv4 vs the ldap consistency model

> d) review given that someone wrote it up?

Sounds like what's really wanted is a distributed transaction. Why not
just 
call it that?

--------------------------------------------------------

This message w/attachments (message) may be privileged, confidential or proprietary, and if you are not an intended recipient, please notify the sender, do not use or share it and delete it. Unless specifically indicated, this message is not an offer to sell or a solicitation of any investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Merrill Lynch. Subject to applicable law, Merrill Lynch may monitor, review and retain e-communications (EC) traveling through its networks/systems. The laws of the country of each sender/recipient may impact the handling of EC, and EC may be archived, supervised and produced in countries other than the country in which you are located. This message cannot be guaranteed to be secure or error-free. This message is subject to terms available at the following link: http://www.ml.com/e-communications_terms/. By messaging with Merrill Lynch you consent to the foregoing.
--------------------------------------------------------
_______________________________________________
Ldapext mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ldapext

---

From: Leif Johansson <[email-masked]>
Date: Fri, 21 Nov 2008 21:59:01 +0100

On Friday 21 November 2008 21:49:30 Howard Chu wrote:
> Leif Johansson wrote:
> > So I had a talk with the nfsv4 wg today (and on their list earlier). The
> > are trying to build AFS-style volume location for NFS using LDAP and I
> > think they may have a problem with the LDAP consistency model: i.e
> > "eventually consistent" for a couple of their usecases.
> >
> > I won't go into details unless people particularly care but I think we
> > may need an I-D (something for a bof in SF perhaps) describing a control
> > which asks for the result of an update operation to be  delayed until a
> > certain criterion (eg in the form of a compare condition) is fulfilled on
> > all "slaves". Implementation details would differ depending on the
> > precise way replication is done for that vendor.
> >

Not really - I don't think they need commit/rollback or the ability to modify
multiple entries in a transaction.

	Cheers Leif

---

From: Howard Chu <[email-masked]>
Date: Fri, 21 Nov 2008 13:18:11 -0800

> multiple entries in a transaction.

Then my answer is (a) and (b).

You cannot guarantee that the update will succeed at all slaves, so 
abort/rollback is essential.

This is 2 phase commit, whether they think they want to handle multiple 
entries or not. In reality they are modifying multiple entries anyway, even if 
it's the "same" entry on each slave. This is a transaction, plain and simple.
-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/
_______________________________________________
Ldapext mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ldapext

---

From: simo <[email-masked]>
Date: Fri, 21 Nov 2008 16:28:03 -0500

> it's the "same" entry on each slave. This is a transaction, plain and simple.

This is a "distributed" transaction, something that is even more complex
to guarantee then just a transaction at single server level.

Can we get first the reason why they want to do something like that ?

It seem to me that way too many premises are wrong on a request like
this, from thinking only in term of "slaves", to thinking that all
"slaves" can always be reached by a specific master and so on.

Simo.

-- 
Simo Sorce
Samba Team GPL Compliance Officer <[email-masked]>
Senior Software Engineer at Red Hat Inc. <[email-masked]>

---

From: Kurt Zeilenga <[email-masked]>
Date: Fri, 21 Nov 2008 23:57:36 -0800

> d) review given that someone wrote it up?

At present, I'm kind of at b).

But I don't see how that changes the replication consistency.  In  
particular, replication of the data for which the condition depends  
upon would still only eventually consistent.

If what they are after is transactional consistency, there may be  
existing LDAP implementations which provide this.  I recall something  
be presented in this area at LDAPcon.

-- Kurt

---

From: Leif Johansson <[email-masked]>
Date: Sun, 23 Nov 2008 16:06:49 +0100

> abort/rollback is essential.

So how is this different from the model we have today - only today you
will never find out if updates fail on any of your slaves!?

	Cheers Leif

---

From: Leif Johansson <[email-masked]>
Date: Sun, 23 Nov 2008 16:12:56 +0100

> upon would still only eventually consistent.

It requires that the server does the work of waiting for the update
operation to finish on all slaves. If the update failed on one of the 
slaves the operation would return an error immediately. 

> be presented in this area at LDAPcon.

Transactional consistency would solve the problem but I think what
they need is somewhat less than transactional consistency. Not sure
if it is less enough to warrant a standard extension.

	Cheers Leif

---

From: simo <[email-masked]>
Date: Sun, 23 Nov 2008 14:39:53 -0500

> if it is less enough to warrant a standard extension.

Leif,
may I insist in asking what do they need this feature for ?

Simo.

---

From: Howard Chu <[email-masked]>
Date: Sun, 23 Nov 2008 14:21:28 -0800

> will never find out if updates fail on any of your slaves!?

Don't confuse the issue. The model we have today is irrelevant to the 
discussion; you started this thread with the criterion that the result of an 
update be delayed until it is completed on all slaves. My point is that you 
cannot fulfill this requirement without abort/rollback capability.

Whether or how any particular server can actually determine the definitive 
outcome of the update is a completely separate question. Simo already noted 
some problems in that area; the entire question is too poorly thought out to 
even begin addressing it here. (Cart before the horse...)

If you acknowledge that the feature you're asking for is in fact a distributed 
transaction (which it very obviously is) then we may at least know that you 
(generic "you" / NFS wg) understand the problem well enough to start 
discussing solutions. Without that basic level of understanding, it's simply a 
waste of time.
-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/
_______________________________________________
Ldapext mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ldapext

---

From: Kurt Zeilenga <[email-masked]>
Date: Sun, 23 Nov 2008 16:31:46 -0800

> slaves the operation would return an error immediately.

This implies not only transactional consistency (all shadow replicas  
are updated at once), but a distributed transaction (all master and  
shadow replicas updated as at once).

I would suggest if that is what the client is expecting that it always  
specify the dontUseCopy control on read operations.  dontUseCopy  
requires the operation be against an authoritative copy.  While  
generally, that's THE master copy, it would extend to all copies  
updated transactionally by the request.

> if it is less enough to warrant a standard extension.

dontUseCopy is not yet a standard extension (in LDAP).

---

From: Steven Legg <[email-masked]>
Date: Mon, 24 Nov 2008 11:33:40 +1100

Leif,

Leif Johansson wrote:
 > Kurt Zeilenga wrote:

Under the eventually-consistent replication model, the update presumably
succeeded on the master but has failed to immediately propagate to all
the shadows (either because a shadow was unavailable, too busy right now,
or had tried the update and failed). The response should be success with
an indication that propagation to the shadows was incomplete at this time.
We assume that one way or another (possibly after administrator
intervention) the shadows would eventually become consistent with the
master, but it could take an indefinitely long time. The nsfv4 folks either
have to be prepared to wait an indefinitely long time for a response or
else be prepared to handle a relatively immediate response that indicated
incomplete propagation of the update. It would certainly help to know
what they are trying to achieve. If a relatively prompt response is
desired, then the control imposes a requirement on the master to propagate
the update immediately, despite what its replication schedules might say.

Since a shadow may have its own shadows (i.e., secondary shadows) that
the master is unaware of, the requirement to not respond until the update
is fully propagated would have to be applied by a shadow to its
interactions with the secondary shadows. However, the protocol used by
the master to communicate with the shadow and/or the protocol used by the
shadow to communicate with the secondary shadows might not provide the
means to indicate the requirement (X.500 DISP, for example).

Regards,
Steven

> ------------------------------------------------------------------------
>

---

From: Kurt Zeilenga <[email-masked]>
Date: Sun, 23 Nov 2008 21:06:28 -0800

On Nov 23, 2008, at 4:33 PM, Steven Legg wrote:

>> operation to finish on all slaves. If the update failed on one of  
>> the slaves the operation would return an error immediately.
>

Or it could be a cache copy.

The control, I think, needs to be the read, not the write.

-- Kurt

---

From: Howard Chu <[email-masked]>
Date: Sun, 23 Nov 2008 22:02:33 -0800

> The control, I think, needs to be the read, not the write.

OK, if we're going to actually discuss this seriously... This is going about 
the problem the completely wrong way. It is impractical in most cases and 
impossible in many cases for a particular master to determine when an update 
has reached every shadow copy.

Also, for the most part, answering the question "has the update reached every 
copy" is unnecessary. The only question that matters is, "have all relevant 
updates reached the server I'm currently talking to?"

The dontUseCopy control is too restrictive in its functionality. What you 
really want is a "AreYouUpToDate?" control. And *that* is something you can 
actually implement, because any shadow will know who to ask to get the answer 
to that question. And, you can implement it without forcing all of the servers 
in a replication setup to pay the cost of distributed locks and two-phase commit.

I would suggest an "AreYouUpToDate?" control with a timestamp or CSN argument. 
The question being asked would be "do you have content equal to or newer than 
the specified CSN?" i.e., "have you already received this update?" The value 
to supply in the CSN argument can be obtained using a PostRead control on a 
previous update operation.

Alternatively, the control could specify an interval of acceptable time 
difference (e.g. 5 seconds) and the contacted server would poll any masters it 
knows about to see if it is sync'd to within the specified interval. (And if 
the contacted server is the master in a single-master setup, it can obviously 
just answer directly.)

Additionally the control could force the current operation to wait until the 
specified sync interval constraint is fulfilled.

The main point being, the query that started this thread, and distributed 
transactions in general, force every server (and consequently, every client) 
to wait before progress can be made. That's foolish, because the progress of a 
particular update isn't important to every server and every client; only a few 
agents in the system know about and care about a given update. Only the agents 
that actually care about a particular operation should pay the cost of 
overseeing the progress of the operation.
-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/
_______________________________________________
Ldapext mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ldapext

---

From: Steven Legg <[email-masked]>
Date: Mon, 24 Nov 2008 17:25:03 +1100

Howard,

Howard Chu wrote:
> OK, if we're going to actually discuss this seriously... This is going 
> about the problem the completely wrong way.

You're right there! We need to know the actual requirement or use cases
that motivated Leif's original posting. :-)

Regards,
Steven

---

From: Leif Johansson <[email-masked]>
Date: Mon, 24 Nov 2008 13:30:01 +0100

> may I insist in asking what do they need this feature for ?

They want to use LDAP as a volume location service for NFS (aka NSDB). 
LDAP would be used to keep track of location replicated copies of file-sets. 
The LDAP consistency model implies that a two clients could see different 
views of the filesystem depending on which LDAP replica the client was
looking at.

In AFS this function is implemented by something called VLDB which is 
backed by a distributed databased 'ubik'. By comparison ubiq uses a 
strict consistency model where if an update doesn't reach one of the
replicas further updates are forbidden until the replica is brought back
in sync with the others (this is called that the cell has lost quorom).
	
It could be that "federated nfsv4" doesn't actually need the type of 
consistency model that is needed by other distributed fs:en (eg AFS)
in which case this may be an unnecessary discussion... 

	Cheers Leif

---

From: Leif Johansson <[email-masked]>
Date: Mon, 24 Nov 2008 13:33:22 +0100

> the update immediately, despite what its replication schedules might say.

Again (cf my other mail just now) comparing with AFS waiting an indeterminate
amount of time may actually be ok - that is more or less how AFS works today.

	Cheers Leif

---

From: Leif Johansson <[email-masked]>
Date: Mon, 24 Nov 2008 13:36:56 +0100

> about the problem the completely wrong way. It is impractical in most cases
> and impossible in many cases for a particular master to determine when an
> update has reached every shadow copy.

That is valuable feedback - if major vendors aren't able to implement 
waiting for shadow copies then the nfsv4 folks must either use something
other than LDAP for volume location or change their requirements.

	Cheers Leif

---

From: simo <[email-masked]>
Date: Mon, 24 Nov 2008 08:26:36 -0500

> in which case this may be an unnecessary discussion... 

A system that enforce all components to be online 100% of the time does
not seem very useful in the general case.
For LDAP servers, the point of having multiple masters and replicas is
also that most applications can continue to work even when the network
splits.

I think many users of a "federated nfsv4" setup would like the same
characteristics if at all possible. But it is up to them.

The problem with requiring LDAP servers to synchronize is that you have
no assurance all LDAP servers are working in unison. In some
organization LDAP proxies are used to separate networks or to filter
information, and they sometimes use caches to manage the load. In this
case even if the "core" set of master and slaves is synchronized, these
proxies may be not.
Although, I guess, you can put the restriction that the nfsv4 deployment
must point only to servers that can communicate to each other.

In any case putting the burden to check for synchronization on the
reader is certainly a much easier approach than trying to modify all
existing LDAP server implementations, unless you want to corner the
users of nfsv4 to be able to use only some specific implementation
potentially for many years.

Simo.

---

From: Leif Johansson <[email-masked]>
Date: Mon, 24 Nov 2008 15:28:35 +0100

> must point only to servers that can communicate to each other.

This is exactly the requirement placed on AFS database servers btw.

> potentially for many years.

That is why I brought it up here! If major vendors (and it certainly sounds
like it) don't like the idea it is dead in the water.

	Cheers Leif

---

From: Andrew Findlay <[email-masked]>
Date: Tue, 25 Nov 2008 10:24:17 +0000

> be presented in this area at LDAPcon.

Yes - the CA product has something like this, which Hilla Reynolds
talked about in Cologne last year. There is a cost to it though, as
all writes block until the slowest replica responds. There are also
some potentially nasty problems if the network gets partitioned.

There is no magic pixie dust for this: *any* distributed replication
system has to make compromises between consistency, timely updates,
and behaviour in the face of various failure modes. The Three-Army
Problem tells us that we cannot design a perfect protocol. DNS, X.500
and LDAP have made similar choices here: favour the availability of
*something* rather than blocking everything to achieve global
consistency.

Andrew
-- 
-----------------------------------------------------------------------
|                 From Andrew Findlay, Skills 1st Ltd                 |
| Consultant in large-scale systems, networks, and directory services |
|     http://www.skills-1st.co.uk/                +44 1628 782565     |
-----------------------------------------------------------------------
_______________________________________________
Ldapext mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ldapext