Subject: [nfsv4] Spec issue - CB_NOTIFY size and callback transport size limit

From: "Talpey, Thomas" <[email-masked]>
Date: Tue, 11 Dec 2007 09:38:26 -0500

I just realized there's a serious issue with the CB_NOTIFY operation
relating to transport size. This op is documented as:

- "The server will send an array of notifications for all changes that
- might have occurred in the directory."

<http://www.nfsv4-editor.org/draft-17/draft-ietf-nfsv4-minorversion1-17.html#OP_CB_NOTIFY>

The problem is, this is a large array, and sending it in a single operation
is dependent on the RPC transport size negotiated for the callback
channel. Typically, this is rather small - a few KB or less - and, in fact
the smaller the better, because clients must generally allocate fixed
resources for receiving it.

Unline READDIR/READDIRPLUS, there is no provision for continuations
across multiple CB_NOTIFY ops. As such, the server may not be able
to comply with the requirement to send "all" the changes.

At a minimum it's clear that there needs to be some guidance as to
what the server (sender) should do if the notification does not fit in
the negotiated callback RPC size. (Such text might be worthwhile for
all callback ops, in fact, though I believe only this is likely to hit it).
Recall that the callback transport size is explicitly negotiated during
session setup, generally in relation to the limits of the underlying
transport, and the client's implementation limits.

If the callback change-array does become truncated to fit in the op,
what should we dictate? I can imagine a series of CB_NOTIFYs, using
a cookie like READDIR, but I am not certain that's the right thing, since
a) it may lead to a significant cascade and
b) it's possible the client will simply perform a re-scan of the directory
after receiving the first one, making the following ops out-of-order.

Another alternative is to provide an "all changes" indicator which, if false,
indicates to the client that there are other unsent changes, and a rescan
is required. This too raises the concern that clients may get "stuck". For
example, if an rm -rf is in progress, the session may see a continuous
stream of notifications and rescans until all three (rm, CB_NOTIFY, rescan)
are done.

A third conceivable alternative is we could remove the "all changes"
requirement on the server, truncate the op, and let the client sort it out.
This will greatly lower the usefulness of CB_NOTIFY to support negative
caching however, which is one of its strongest points.

Saving my actual suggestion for last...

Finally, we could require the server to *recall* the directory delegation,
by issuing CB_RECALL with the directory delegation stateid, instead of
the CB_NOTIFY. This is the safest method in my view, because it squelches
any further notifies, and therefore prevents the cascade effect by forcing
the client back into sync. The client would simply flush its cache for the
directory, and start over. And, this option doesn't require changing the
protocol at this late date.

Other ideas? 

Tom.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Rick Macklem <[email-masked]>
Date: Tue, 11 Dec 2007 12:07:43 -0500 (EST)

On Tue, 11 Dec 2007, Talpey, Thomas wrote:

[good stuff snipped]
> caching however, which is one of its strongest points.
This sounds confusing for the client (or maybe the poor client implementor
that has to figure out how to get this right).

> protocol at this late date.
Since I haven't tried to implement any of this stuff, it's just a hunch,
but I suspect a CB_NOTIFY of a large number of changes isn't very useful
for a typical client implementation. As such, the above sounds reasonable
to me. So long as it's clearly stated that servers will CB_RECALL if the
CB_NOTIFY would be too big a reply, then clients can try and negotiate
an appropriate reply size, so they'll get the CB_NOTIFYs that the can
effectively use.

Just my $0.00 worth, rick

---

From: Benny Halevy <[email-masked]>
Date: Thu, 13 Dec 2007 09:04:25 +0200

> that has to figure out how to get this right).

IIRC, there's no hard requirement for synchronization between the actual
changes to the directory on the server and CB_NOTIFY.  That tells me
that, modulo removing "all changes", the server can just push the changes
over multiple CB_NOTIFY calls.  An "End of stream" flag seems to be a
good addition to the protocol as it tells the client that the directory
is still mutating and it should expect to get more notifications
(the server must serialize these of course to avoid races)

It's up to the server to decide if that economical, and if not, it can
always CB_RECALL the dir delegation.  It's up to the client to stop
waiting for the server for more updates and it can always return the
delegation and send a READDIR to refresh its cache.

Benny

---

From: Mike Eisler <[email-masked]>
Date: Wed, 12 Dec 2007 23:21:24 -0800 (PST)

Benny's interpretation matches mine.

I don't see what the "end of stream" flag buys uis, because even if
it is the end of stream, nothing prevents another client from issuing
an operation that causes a new stream a microsecond later.

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 13 Dec 2007 07:41:20 -0500

> (the server must serialize these of course to avoid races)

That's the issue here.  Is the server responsible for not sending notify
two until it has received a reply to notify one?  If so, the spec should
say that.   

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Thursday, December 13, 2007 2:04 AM
To: Rick Macklem; Talpey, Thomas
Cc: [email-masked]
Subject: Re: [nfsv4] Spec issue - CB_NOTIFY size and callback transport
sizelimit

On Dec. 11, 2007, 19:07 +0200, Rick Macklem <[email-masked]>
wrote:
>> This will greatly lower the usefulness of CB_NOTIFY to support 
>> negative caching however, which is one of its strongest points.
>>
> This sounds confusing for the client (or maybe the poor client 
> implementor that has to figure out how to get this right).

IIRC, there's no hard requirement for synchronization between the actual
changes to the directory on the server and CB_NOTIFY.  That tells me
that, modulo removing "all changes", the server can just push the
changes over multiple CB_NOTIFY calls.  An "End of stream" flag seems to
be a good addition to the protocol as it tells the client that the
directory is still mutating and it should expect to get more
notifications (the server must serialize these of course to avoid races)

Benny

>> Finally, we could require the server to *recall* the directory 
>> delegation, by issuing CB_RECALL with the directory delegation 
>> stateid, instead of the CB_NOTIFY. This is the safest method in my 
>> view, because it squelches any further notifies, and therefore 
>> prevents the cascade effect by forcing the client back into sync. The

>> client would simply flush its cache for the directory, and start 
>> over. And, this option doesn't require changing the protocol at this
late date.
>>
> Since I haven't tried to implement any of this stuff, it's just a 
> hunch, but I suspect a CB_NOTIFY of a large number of changes isn't 
> very useful for a typical client implementation. As such, the above 
> sounds reasonable to me. So long as it's clearly stated that servers 
> will CB_RECALL if the CB_NOTIFY would be too big a reply, then clients

> can try and negotiate an appropriate reply size, so they'll get the 
> CB_NOTIFYs that the can effectively use.
>

---

From: Benny Halevy <[email-masked]>
Date: Thu, 13 Dec 2007 16:08:19 +0200

> two until it has received a reply to notify one?  If so, the spec should

The server MUST serialize CB_NOTIFYs in case there could be conflicts between
a CB_NOTIFY callback and any outstanding CB_NOTIFY to the same directory at the same
client.

If the server can't determine such conflicts it MUST assume that this conflict
could occur and serialize all CB_NOTIFYs to the same directory at the same client.

[off the top of my head I thought that an atomic update to the directory cannot
contain conflicts so the server can send multiple CB_NOTIFYs if they represent
an atomic change to the dir (i.e. the sequence of changes from one point-in-time
snapshot of the directory to another point-in-time snapshot of it), but I think
that might not be true if a name A is deleted and then another name B is renamed
onto A]

Benny

---

From: "Talpey, Thomas" <[email-masked]>
Date: Thu, 13 Dec 2007 09:20:01 -0500

I think it depends on whether the client has requested synchronous or
asynchronous notify behavior in its initial GET_DIR_DELEGATION. But we
need to be really careful about using the word "atomic". There is never
a guarantee that this all happens atomically. The key is consistenly, and
in an ordered fashion. The client can never be sure he's seen "all" the
updates, ever.

My issue going in to this is that the spec said blankly that the server
has to send "all" the updates in one message. This is patently impossible.
So, I think we need to tell the server what to do when it can't.

I agree that it may feel free to send multiple notifications. But there
are cases where this is inconvenient, or can lead to cascades, as my
original mail mentioned.

I disagree with your suggested text, because it doesn't make sense
to require the server to "serialize" these callbacks. It introduces a huge
bottleneck into the server, for no good reason. Is the server supposed
to prevent all directory changes until the client is done ACKing all the
callbacks the server might queue? Ouch!

I say, let the server recall whenever it needs to, for whatever definition
of "needs to" it cares to decide. The alternative is to mandate that
directory notifies are perfectly sequenced and perfectly consistent,
that way lies madness.

Tom.

I think we should pick one basic

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 13 Dec 2007 10:18:35 -0500

> The key is consistenly, and in an ordered fashion. 

That's the critical point.  There are cases in which misordering would
result in the client being confused about the state of the directory or
rather the client would think the directory had been in a state that it
was never in.

> The client can never be sure he's seen "all" the updates, ever.

Agree.

> So, I think we need to tell the server what to do when it can't.

Right.

> bottleneck into the server, for no good reason. 

There is a good reason.  See "consistently, and in an ordered fashion".
I assume "ordered" here means "correctly ordered" ;-)

> Is the server supposed
> callbacks the server might queue? 

No.  He just has to make sure that he records the changes in order and
sends them in order.  If he has some limitation as to the amount he can
buffer (and there always will be some limitation), he would recall the
delegation when he started to get anywhere close to that limit.

One thing that the spec should make clear is that whenthe client gets a
recall he should not be able to assume that he got all pending notifies
until he returns the delegation.  The server must be able to say, by
making the recall, that the client may not have a full history up to
that point.

> Ouch!

No pain, no gain.

> The alternative is to mandate that

They should be sequenced and consistent or else they are of limited use.

How is "sequenced and consistent" different from "perfectly sequenced
and perfectly consistent"?

The way to avoid madness is to make clear that when the server cannot
meet this mandate, it is free to, and should recall the delegation, at
which point its obligations are cancelled.

To avoid a directory locked up while waiting for a slow client to
respond to notifies and/or recalls, the spec needs to make clear that if
you have requested async notifications, then the server does not have to
hold up directory changes while the delegation is being recalled.  It
can let things that you consented to async notify for go on, even if you
are not going to get those notifies, because of buffering issues.  Also,
I think the server should be free to revoke the directory delegation
without necessarily waiting the lease period.  If there is a revoke, the
client will get a notification in the status bits in the next SEQUENCE
op and will find out that the directory delegation is gone through using
TEST_STATEID.  That should be fine for someone who is using the
delegation as a way to get async notifications.

-----Original Message-----
From: Talpey, Thomas 
Sent: Thursday, December 13, 2007 9:20 AM
To: Benny Halevy; Noveck, Dave
Cc: [email-masked]
Subject: Re: [nfsv4] Spec issue - CB_NOTIFY size and callback transport
sizelimit

a guarantee that this all happens atomically. The key is consistenly,
and

My issue going in to this is that the spec said blankly that the server
has to send "all" the updates in one message. This is patently
impossible.
So, I think we need to tell the server what to do when it can't.

to require the server to "serialize" these callbacks. It introduces a
huge

I say, let the server recall whenever it needs to, for whatever
definition
of "needs to" it cares to decide. The alternative is to mandate that
directory notifies are perfectly sequenced and perfectly consistent,
that way lies madness.

Tom.

---

From: Mike Eisler <[email-masked]>
Date: Thu, 13 Dec 2007 07:35:53 -0800 (PST)

On the misordering issue, we have the technology. The
delegation stateid is in the arguments. Mandate the use (incrementing)
of the seqid in the stateid, and all should be well. The client can
use the seqid to detect the temporaral order of two unserialized
notifications.

If one notification adds a file name, and another remove the file name,
the one with the higher seqid applies.

I now see why the "end of stream" indicator might be useful. More
useful might be for the client and server to negotiate how
"unserialized" notifications can be. I.e. the maximum gap in seqid
between what the client thinks the stateid is and the what server can
send in a recall. This could be useful for all stateids associated
with recallable objects.  

--- "Noveck, Dave" <[email-masked]> wrote:

> > to prevent all directory changes until the client is done ACKing
> all
> the
> >The server MUST serialize CB_NOTIFYs in case there could be
> conflicts
> between
> >If the server can't determine such conflicts it MUST assume that
> this
> conflict
> >could occur and serialize all CB_NOTIFYs to the same directory at
> the 
> >same client.
> >
> >snapshot of the directory to another point-in-time snapshot of it),
> but
> I think
> >that might not be true if a name A is deleted and then another name
> B 
> >is renamed
> >onto A]
> >
> >Benny
>

---

From: "Talpey, Thomas" <[email-masked]>
Date: Thu, 13 Dec 2007 11:06:08 -0500

>the one with the higher seqid applies.

I guess so. While it helps the client, it doesn't actually solve the problem
for the server. So I think it's a useful tweak, assuming client implementers
agree.

>send in a recall. This could be useful for all stateids associated

I can imagine it being useful, but I don't think we have any reason to
spend time on it. We don't have any implementation feedback at all,
so why tune the spec? It's more important to finish.

MHO.

Tom.

---

From: Mike Eisler <[email-masked]>
Date: Fri, 14 Dec 2007 08:09:14 -0800 (PST)

--- "Talpey, Thomas" <[email-masked]> wrote:

> for the server. So I think it's a useful tweak, assuming client

The requirements are:

- no serialization of callbacks
- unambiguous indication of what the state of the directory is

The solution I proposed forces the server to indicate 
the order of which the directory changing event occured 
without serializing the callback responses. The client can 
use the notification with
the higher seqid to determine the state of the directory
How is that not a solution that meets
the requirements? 
 
> implementers

Agreed.

---

From: Mike Eisler <[email-masked]>
Date: Sat, 15 Dec 2007 08:17:57 -0800 (PST)

For now I've checked in changes that remove the offending "all",
and note servers SHOULD serialize multiple CB_NOTIFY requests in
order to prevent the client from having a fatal cache inconsistency.

I've also added text that explains by CB_NOTIFY_DEVICEID does
not have this problem (because device IDs, once deleted,
stay deleted).

--- Mike Eisler <[email-masked]> wrote:

> > I guess so. While it helps the client, it doesn't actually solve
> the
> > problem