Subject: [ldapext] Request Progress Reporting

From: Howard Chu <[email-masked]>
Date: Thu, 02 Apr 2009 07:44:05 -0700

This discussion began on the OpenLDAP-Devel mailing list, talking about 
returning a progress indicator to a client for a long running search operation.

http://www.openldap.org/lists/openldap-devel/200903/msg00082.html

It was suggested that progress reporting may be useful in general, not just 
for search requests. Moving the discussion here to get more input.

The idea currently is to attach a "ProgressReport" control to any request. The 
control will specify a time interval to indicate how often progress updates 
should be returned.

ProgressReport updates may be attached in Response controls accompanying 
Search Response messages, Result messages, and also in Intermediate Response 
messages.

The ProgressReport update will contain two integers of arbitrary units: one 
representing the total amount of work, and one representing the current amount 
already executed.

E.g., for a Search request, the total value may represent the number of 
candidate entries to be evaluated in the search, and the current value may 
represent the number of entries already evaluated.

Typically a write request should execute quickly, but if a write has other 
side-effects, it may take longer. E.g., if a server is configured with 
synchronous replication, where the client write does not complete until some 
number of replicas have received the update, then the progress report may be 
useful to indicate how many replicas have been updated.

There are some security concerns here regarding the control giving out 
information that clients wouldn't otherwise have access to. E.g., access 
controls may limit a client to seeing only a subset of entries in a directory; 
a ProgressReport may allow them to discover the true number of entries. It's 
not clear to me that this is critical or useful to an attacker, but it would 
make sense for implementations to also control who is authorized to use the 
ProgressReport control, and to fudge the units used in the update data so that 
they don't correspond 1-to-1 with any actual counts of anything meaningful.

Comments?
-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

---

From: Howard Chu <[email-masked]>
Date: Thu, 02 Apr 2009 08:02:03 -0700

> control will specify a time interval to indicate how often progress updates

I forgot to mention - time intervals should probably be in units smaller than 
seconds, but probably not smaller than milliseconds. Human factors tells us 
that most users don't perceive lag until intervals exceed 100ms.

---

From: Hallvard B Furuseth <[email-masked]>
Date: Thu, 2 Apr 2009 17:54:06 +0200

Howard Chu writes:
> I forgot to mention - time intervals should probably be in units
> smaller than seconds, but probably not smaller than milliseconds.
> Human factors tells us that most users don't perceive lag until
> intervals exceed 100ms.

"Surely only interactive programs will find a use for this feature."
Hmm.

BTW, we'll also need to define a dummy Intermediate Response, so
servers can send this control when they have nothing else to say.

-- 
Hallvard

---

From: Andrew Findlay <[email-masked]>
Date: Thu, 2 Apr 2009 17:24:59 +0100

On Thu, Apr 02, 2009 at 08:02:03AM -0700, Howard Chu wrote:

> I forgot to mention - time intervals should probably be in units smaller 
> than seconds, but probably not smaller than milliseconds. Human factors 
> tells us that most users don't perceive lag until intervals exceed 100ms.

I would suggest that report messages should include some
representation of wallclock time at the server, so that clients can
allow for network latency and jitter when interpreting them. This is a
common strategy when making SNMP queries: a client will usually
request the values of some counters and also the value of the server
agent's "time since boot" counter.

>> Search Response messages, Result messages, and also in Intermediate Response

Clients must understand that an indication of progress is not an
indication that the operation will ever finish, nor that it will be
successful if it does.

Andrew
-- 
-----------------------------------------------------------------------
|                 From Andrew Findlay, Skills 1st Ltd                 |
| Consultant in large-scale systems, networks, and directory services |
|     http://www.skills-1st.co.uk/                +44 1628 782565     |
-----------------------------------------------------------------------

---

From: Howard Chu <[email-masked]>
Date: Thu, 02 Apr 2009 19:59:07 -0700

> "Surely only interactive programs will find a use for this feature."

That's hardly relevant. Much smaller intervals will also place an unreasonable 
burden on the server. Also, you have platforms like Windows where the system 
clock period is only 55ms anyway; there'd be no point in asking for anything 
smaller.

> servers can send this control when they have nothing else to say.

I was suggesting we would define a ProgressReport Intermediate Response, which 
would be sent in those situations. A dummy Intermediate Response could be more 
useful in general, but I'm not sure whether it's within the scope of this 
discussion. Certainly that would make processing a lot more uniform.

---

From: Kurt Zeilenga <[email-masked]>
Date: Wed, 8 Apr 2009 18:25:19 -0700

> servers can send this control when they have nothing else to say.

After consideration of whether it would be better to define an  
Intermediate Response specifically for this extension, or define a  
general "dummy" Intermediate Response to which one can attach this and  
other controls, I've concluded the latter is better (though it can be  
defined in the same document).

-- Kurt