Subject: [nfsv4] Clarifications requested for session dynamic slot management as described in RFC5661

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 21 Nov 2012 15:27:22 +0000

Hi,

I'm finally getting round to implementing the session slot management
scheme as outlined in Section 2.10.6.1 in RFC5661. A couple of questions
have surfaced, that I'd like to raise for general discussion. The issues
raised are about "retiring slots" and what that means, and about how we
should go about in practice updating the slot table parameters via the
SEQUENCE operation.

1) If the intention is that the client and server no longer have to
cache information about slots that are retired, then presumably slot
sequence numbers are among the information that is lost?

2) If a slot with a retired slotid is later added back to the table,
then should the sequence number start from the value '1' again?

3) Exactly when should the client and server both agree that the slot is
retired? As far as I can see, the only reply that makes sense is that
the client MUST have sent a SEQUENCE op with sa_highest_slotid < 'foo',
and MUST have received a sr_highest_slotid reply from the server that is
also less than 'foo'. Is this correct?

4) How does the server ensure that it isn't sending conflicting values
of sr_target_highest_slotid (or of sr_highest_slotid for that matter) to
the client?
The problem is due to the lack of ordering between replies to SEQUENCE
ops that are being sent on different slotids. If the server is changing
the value of sr_target_highest_slotid too quickly, then the client may
not be able to figure out which is the most recent value.
The only solution that I can see is that the server MUST NOT change the
value of sr_target_highest_slotid or sr_highest_slotid until it has seen
a new SEQUENCE request on at least _one_ of the slots on which the
server has sent a reply with the current value of
sr_target_highest_slotid. In other words, there must be serialisation of
the slot table parameter updates on at least one slot. Does this make
sense to all?

Cheers,
  Trond

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 21 Nov 2012 15:31:58 +0000

On Wed, 2012-11-21 at 15:27 +0000, Myklebust, Trond wrote:
> retired?

Sorry. That should read: "Exactly when should the client and server both
agree that the slot with slotid 'foo' is retired?"

>  As far as I can see, the only reply that makes sense is that

NetApp

---

From: Tom Talpey <[email-masked]>
Date: Wed, 21 Nov 2012 16:54:00 +0000

In general, the slot table size management is not a normative ("MUST") kind of thing, but yes, retiring entries needs a bit of care. Responses below.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Myklebust, Trond
> Sent: Wednesday, November 21, 2012 10:32 AM
> To: nfsv4 list
> Subject: Re: [nfsv4] Clarifications requested for session dynamic slot
> management as described in RFC5661
> 
> > scheme as outlined in Section 2.10.6.1 in RFC5661. A couple of
> > questions have surfaced, that I'd like to raise for general
> > discussion. The issues raised are about "retiring slots" and what that
> > means, and about how we should go about in practice updating the slot
> > table parameters via the SEQUENCE operation.
> >

Yes, when retired, all state for a given slot is discardable.

> > then should the sequence number start from the value '1' again?

In practice, the sequence can start at any value, but since the protocol recommends starting at '1' this would be a good choice.

> >
> > 3) Exactly when should the client and server both agree that the slot
> > is retired?
> 

I'd recommend not thinking about slots as belonging to "client" and "server". Because the backchannel has slots too, it might be best to consider "sender" and "receiver" or similar. Also, the state of a slot changes differently for each end of the connection. The sender cannot retire a slot until it receives an appropriate acknowledgement that the peer has also done so. Your next questions address that.

> 
> >  As far as I can see, the only reply that makes sense is that the
> > client MUST have sent a SEQUENCE op with sa_highest_slotid < 'foo',
> > and MUST have received a sr_highest_slotid reply from the server that
> > is also less than 'foo'. Is this correct?

Correct. The receiver can actually retire all inactive slots which are greater than sr_highest_slotid. It will want to verify the inactive state of these slots before doing so, of course. A new message in that slot might have been sent. Any slots less than this active high-water should not be retired yet, although they perhaps can be marked as do-not-use (otherwise, a busy client might never comply with the downpressure).

> > of sr_target_highest_slotid (or of sr_highest_slotid for that matter)
> > to the client?

It doesn't really matter whether the sr_target_highest_slotid conflicts, since that value is only a hint. If the receiver is using a slot outside the range, it can't honor the target at that time. So "conflicting" doesn't really apply to the target value.

For sr_highest_slotid, only the sender can tell if misordering is happening. If sr_highest_slotid is larger than the currently highest active slot when received, it needs to be raised to that value.

> > ops that are being sent on different slotids. If the server is
> > changing the value of sr_target_highest_slotid too quickly, then the
> > client may not be able to figure out which is the most recent value.
> > The only solution that I can see is that the server MUST NOT change
> > the value of sr_target_highest_slotid or sr_highest_slotid until it
> > has seen a new SEQUENCE request on at least _one_ of the slots on
> > which the server has sent a reply with the current value of
> > sr_target_highest_slotid. In other words, there must be serialisation
> > of the slot table parameter updates on at least one slot. Does this
> > make sense to all?

It does make sense, and it's good advice to not adjust these values "too often". But as long as slots are thought of as representing an active range, without holes, the behavior is consistent even with misordered updates. Think of the slotids as a window of requests and not a sparse array. The window can be opened at any time, but is only narrowed when no active requests are in flight to the gap.

Tom.

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 21 Nov 2012 17:36:12 +0000

Hi Tom,

Thanks for responding!

> Correct. The receiver can actually retire all inactive slots which are greater than sr_highest_slotid. It will want to verify the inactive state of these slots before doing so, of course. A new mes…

Thanks!

> It does make sense, and it's good advice to not adjust these values "too often". But as long as slots are thought of as representing an active range, without holes, the behavior is consistent even…

Where can I find text that supports the statement that the window can
not be shrunk when there are active requests?

If the sr_highest_slotid and sr_target_highest_slotid can only grow the
window size in response to a SEQUENCE request, then that will trivially
solve the ordering issues, however I can't remember seeing anything that
forbids its use to shrink the window.

Cheers
  Trond

NetApp

---

From: Tom Talpey <[email-masked]>
Date: Wed, 21 Nov 2012 18:59:22 +0000

I'm going to re-quote your question to avoid butchering the message too badly with my email client.

> Where can I find text that supports the statement that the window can not
> be shrunk when there are active requests?

I don't think there's such a statement, it's more of a conclusion which is drawn based on the sequencing rules. We could make one of course.

> window size in response to a SEQUENCE request, then that will trivially solve
> the ordering issues, however I can't remember seeing anything that forbids
> its use to shrink the window.

Here's a concrete example, maybe I can be clearer this way. 

Say the client has a slot table of size 5, and has sent 3 requests into slots 1, 2 and 3.

Say the server starts completing these requests, and sends replies to 2, 3 and 1 because they complete in that order.

Further, when completing 1, the server decides to ask for some slots back, so it sends sr_target_highest_slotid of 3, but even though it completed all the active requests it doesn't yet know that the client has seen them; it therefore maintains its sr_highest_slotid at 5. In other words, while it is requesting that the window be shrunk, it can't actually shrink it - i.e. retire any slots.

In the meantime, before seeing any of these replies, the client sent an additional request, which of course used slot 4.

When the client receives the reply on slot 1, it cannot honor the sr_target_highest_slotid, because it is less than 4. But, it can mark slots 1, 2 and 3 as complete, can mark slot 4 as do-not-use (eligible for future retirement), and can retire slot 5 immediately. The server doesn't know any of this yet.

Once the server completes the operation in slot 4, it will again send sr_highest_slotid of 5, and (probably) also send sr_target_highest_slotid of 3. When it's received, the client can retire slot 4, and reduce the slot table size to the server's target of 3. The server still doesn't know this.

Finally, the client's *next* request will now carry sa_slotid 1 and sa_highest_slotid 3. Once it sees this, the server can start a new request in slot 1, mark slots 2 and 3 as inactive, and can retire slots 4 and 5. Its next reply will carry sr_highest_slotid of 3, upon which the client can finally retire slot 4.

Mainly, it's important to remember that the two sides are making these moves independently, and at different times. If one side detects the other side not responding to slot reducton, and needs the resources back, the solution is to close the connection - starting things over from zero.

One thing to consider is that if the server does shrink the slot window, it should be sensitive to the client's need for increased slots. There's no forward-pressure (a so-called sa_target_highest_slotid) so if the server sees the client using up all its slots, it may want to consider growing the window, by increasing the slot table and sr_highest_slotid. As mentioned, it can do this direction at any time.

Tom.

> From: Myklebust, Trond [mailto:[email-masked]]
> Sent: Wednesday, November 21, 2012 12:36 PM
> To: Tom Talpey
> Cc: nfsv4 list
> > In general, the slot table size management is not a normative ("MUST")
> kind of thing, but yes, retiring entries needs a bit of care. Responses below.
> >
> > > From: [email-masked] [mailto:[email-masked]] On
> > > Behalf Of Myklebust, Trond
> > > Subject: Re: [nfsv4] Clarifications requested for session dynamic
> > > slot management as described in RFC5661
> > >
> > > > I'm finally getting round to implementing the session slot
> > > > management scheme as outlined in Section 2.10.6.1 in RFC5661. A
> > > > couple of questions have surfaced, that I'd like to raise for
> > > > general discussion. The issues raised are about "retiring slots"
> > > > and what that means, and about how we should go about in practice
> > > > updating the slot table parameters via the SEQUENCE operation.
> > > >
> > > > 1) If the intention is that the client and server no longer have
> > > > to cache information about slots that are retired, then presumably
> > > > slot sequence numbers are among the information that is lost?
> >
> > > > 2) If a slot with a retired slotid is later added back to the
> > > > table, then should the sequence number start from the value '1' again?
> >
> > In practice, the sequence can start at any value, but since the protocol
> recommends starting at '1' this would be a good choice.
> >
> > > >
> > > > 3) Exactly when should the client and server both agree that the
> > > > slot is retired?
> > >
> > > Sorry. That should read: "Exactly when should the client and server
> > > both agree that the slot with slotid 'foo' is retired?"
> >
> > I'd recommend not thinking about slots as belonging to "client" and
> "server". Because the backchannel has slots too, it might be best to consider
> "sender" and "receiver" or similar. Also, the state of a slot changes
> differently for each end of the connection. The sender cannot retire a slot
> until it receives an appropriate acknowledgement that the peer has also
> done so. Your next questions address that.
> >
> > >
> > > > client MUST have sent a SEQUENCE op with sa_highest_slotid <
> > > > 'foo', and MUST have received a sr_highest_slotid reply from the
> > > > server that is also less than 'foo'. Is this correct?
> >
> > Correct. The receiver can actually retire all inactive slots which are greater
> than sr_highest_slotid. It will want to verify the inactive state of these slots
> before doing so, of course. A new message in that slot might have been sent.
> Any slots less than this active high-water should not be retired yet, although
> they perhaps can be marked as do-not-use (otherwise, a busy client might
> never comply with the downpressure).
> 
> Thanks!
> 
> > >
> > >
> > > > 4) How does the server ensure that it isn't sending conflicting
> > > > values of sr_target_highest_slotid (or of sr_highest_slotid for
> > > > that matter) to the client?
> >
> > It doesn't really matter whether the sr_target_highest_slotid conflicts,
> since that value is only a hint. If the receiver is using a slot outside the range,
> it can't honor the target at that time. So "conflicting" doesn't really apply to
> the target value.
> >
> > For sr_highest_slotid, only the sender can tell if misordering is happening.
> If sr_highest_slotid is larger than the currently highest active slot when
> received, it needs to be raised to that value.
> >
> > > > The problem is due to the lack of ordering between replies to
> > > > SEQUENCE ops that are being sent on different slotids. If the
> > > > server is changing the value of sr_target_highest_slotid too
> > > > quickly, then the client may not be able to figure out which is the most
> recent value.
> > > > The only solution that I can see is that the server MUST NOT
> > > > change the value of sr_target_highest_slotid or sr_highest_slotid
> > > > until it has seen a new SEQUENCE request on at least _one_ of the
> > > > slots on which the server has sent a reply with the current value
> > > > of sr_target_highest_slotid. In other words, there must be
> > > > serialisation of the slot table parameter updates on at least one
> > > > slot. Does this make sense to all?
> >
> > It does make sense, and it's good advice to not adjust these values "too
> often". But as long as slots are thought of as representing an active range,
> without holes, the behavior is consistent even with misordered updates.
> Think of the slotids as a window of requests and not a sparse array. The
> window can be opened at any time, but is only narrowed when no active
> requests are in flight to the gap.
> 
> Where can I find text that supports the statement that the window can not
> be shrunk when there are active requests?
> 
> window size in response to a SEQUENCE request, then that will trivially solve
> the ordering issues, however I can't remember seeing anything that forbids
> its use to shrink the window.
> 
> Cheers
>   Trond
> 
> --
> Trond Myklebust

---

From: "Noveck, David" <[email-masked]>
Date: Mon, 26 Nov 2012 08:38:05 -0500

Tom Talpey said:
> Finally, the client's *next* request will now carry sa_slotid 1 and 
> sa_highest_slotid 3. Once it sees this, the server can start a new 
> request in slot 1, mark slots 2 and 3 as inactive, and can retire slots 
> 4 and 5. Its next reply will carry sr_highest_slotid of 3, upon which 
> the client can finally retire slot 4.

This is all very sensible.  One problem is that it doesn't agree with RFC5661.

Note that RFC5661 says: 

>  The sa_highest_slotid argument is the highest slot ID for which the
>  client has a request outstanding; it could be equal to sa_slotid.

So, in Tom's example, sa_highest_slotid would have varied from 1 to 3 and then gone to four and then to one once the request on slot four completed.  If that were the case, the server could not conclude anything from the change in the received value of sa_highest_slotid. 

I think Tom is assuming some other definition of sa_highest_slotid.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Tom Talpey
Sent: Wednesday, November 21, 2012 1:59 PM
To: Myklebust, Trond
Cc: nfsv4 list
Subject: Re: [nfsv4] Clarifications requested for session dynamic slot management as described in RFC5661

> 
> If the sr_highest_slotid and sr_target_highest_slotid can only grow 
> the window size in response to a SEQUENCE request, then that will 
> trivially solve the ordering issues, however I can't remember seeing 
> anything that forbids its use to shrink the window.

Tom.

> > > > and what that means, and about how we should go about in 
> > > > practice updating the slot table parameters via the SEQUENCE operation.
> > > >
> > > > to cache information about slots that are retired, then 
> > > > presumably slot sequence numbers are among the information that is lost?
> >
> > > Sorry. That should read: "Exactly when should the client and 
> > > server both agree that the slot with slotid 'foo' is retired?"
> >
> "server". Because the backchannel has slots too, it might be best to 
> consider "sender" and "receiver" or similar. Also, the state of a slot 
> changes differently for each end of the connection. The sender cannot 
> retire a slot until it receives an appropriate acknowledgement that 
> the peer has also done so. Your next questions address that.
> >
> > >
> > > > the client MUST have sent a SEQUENCE op with sa_highest_slotid < 
> than sr_highest_slotid. It will want to verify the inactive state of 
> these slots before doing so, of course. A new message in that slot might have been sent.
> Any slots less than this active high-water should not be retired yet, 
> although they perhaps can be marked as do-not-use (otherwise, a busy 
> client might never comply with the downpressure).
> 
> Thanks!
> 
> > >
> > >
> since that value is only a hint. If the receiver is using a slot 
> outside the range, it can't honor the target at that time. So 
> "conflicting" doesn't really apply to the target value.
> >
> If sr_highest_slotid is larger than the currently highest active slot 
> when received, it needs to be raised to that value.
> >
> > > > change the value of sr_target_highest_slotid or 
> > > > sr_highest_slotid until it has seen a new SEQUENCE request on at 
> > > > least _one_ of the slots on which the server has sent a reply 
> > > > with the current value of sr_target_highest_slotid. In other 
> > > > words, there must be serialisation of the slot table parameter 
> > > > updates on at least one slot. Does this make sense to all?
> >
> often". But as long as slots are thought of as representing an active 
> range, without holes, the behavior is consistent even with misordered updates.
> Think of the slotids as a window of requests and not a sparse array. 
> The window can be opened at any time, but is only narrowed when no 
> active requests are in flight to the gap.
> 
> If the sr_highest_slotid and sr_target_highest_slotid can only grow 
> the window size in response to a SEQUENCE request, then that will 
> trivially solve the ordering issues, however I can't remember seeing 
> anything that forbids its use to shrink the window.
> 
> Cheers
>   Trond
> 
> --
> Trond Myklebust
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Tom Talpey <[email-masked]>
Date: Mon, 26 Nov 2012 14:55:36 +0000

> From: Noveck, David [mailto:[email-masked]]
> Sent: Monday, November 26, 2012 8:38 AM
> To: Tom Talpey; Myklebust, Trond
> Cc: nfsv4 list
> Subject: RE: [nfsv4] Clarifications requested for session dynamic slot
> So, in Tom's example, sa_highest_slotid would have varied from 1 to 3 and
> then gone to four and then to one once the request on slot four completed.
> If that were the case, the server could not conclude anything from the
> change in the received value of sa_highest_slotid.

The server concludes that the client has seen all responses, in particular, the response to the request using slotid 4. Can you be more specific as to your concern?

Tom.

> Of Tom Talpey
> I don't think there's such a statement, it's more of a conclusion which is
> drawn based on the sequencing rules. We could make one of course.
> 
> >
> Further, when completing 1, the server decides to ask for some slots back, so
> it sends sr_target_highest_slotid of 3, but even though it completed all the
> active requests it doesn't yet know that the client has seen them; it
> therefore maintains its sr_highest_slotid at 5. In other words, while it is
> requesting that the window be shrunk, it can't actually shrink it - i.e. retire
> any slots.
> 
> In the meantime, before seeing any of these replies, the client sent an
> additional request, which of course used slot 4.
> 
> When the client receives the reply on slot 1, it cannot honor the
> sr_target_highest_slotid, because it is less than 4. But, it can mark slots 1, 2
> and 3 as complete, can mark slot 4 as do-not-use (eligible for future
> retirement), and can retire slot 5 immediately. The server doesn't know any
> of this yet.
> 
> Once the server completes the operation in slot 4, it will again send
> sr_highest_slotid of 5, and (probably) also send sr_target_highest_slotid of
> 3. When it's received, the client can retire slot 4, and reduce the slot table
> size to the server's target of 3. The server still doesn't know this.
> 
> sa_highest_slotid 3. Once it sees this, the server can start a new request in
> slot 1, mark slots 2 and 3 as inactive, and can retire slots 4 and 5. Its next reply
> will carry sr_highest_slotid of 3, upon which the client can finally retire slot 4.
> 
> Mainly, it's important to remember that the two sides are making these
> moves independently, and at different times. If one side detects the other
> side not responding to slot reducton, and needs the resources back, the
> solution is to close the connection - starting things over from zero.
> 
> One thing to consider is that if the server does shrink the slot window, it
> should be sensitive to the client's need for increased slots. There's no
> forward-pressure (a so-called sa_target_highest_slotid) so if the server sees
> the client using up all its slots, it may want to consider growing the window,
> by increasing the slot table and sr_highest_slotid. As mentioned, it can do
> this direction at any time.
> 
> Tom.
> 
>

---

From: "Noveck, David" <[email-masked]>
Date: Mon, 26 Nov 2012 10:44:15 -0500

> The server concludes that the client has seen all responses, in particular, 
> the response to the request using slotid 4. 

If the client did send an sa_highest_slotid of 1, then the server could conclude that 
the only outstanding request was the one it just received.

> Can you be more specific as to your concern?

OK.

> sa_highest_slotid 3. 

But, according to RFC5661, the value of sa_highest_slotid in this case would be
1.   

> Once it sees this, the server can start a new 
> request in slot 1, mark slots 2 and 3 as inactive, 

If it did get that, it would have to conclude that the client still had not
received the response to the request on slot 3.

But as this was presented, we were told that 3 was already compete, but maybe
I'm mistaken about that.

> and can retire 
> slots 4 and 5. Its next reply will carry sr_highest_slotid of 3, upon 
> which the client can finally retire slot 4.

True.

-----Original Message-----
From: Tom Talpey [mailto:[email-masked]] 
Sent: Monday, November 26, 2012 9:56 AM
To: Noveck, David; Myklebust, Trond
Cc: nfsv4 list
Subject: RE: [nfsv4] Clarifications requested for session dynamic slot management as described in RFC5661

> > 4 and 5. Its next reply will carry sr_highest_slotid of 3, upon 
> > which the client can finally retire slot 4.
> 
> So, in Tom's example, sa_highest_slotid would have varied from 1 to 3 
> and then gone to four and then to one once the request on slot four completed.

Tom.

> > Where can I find text that supports the statement that the window 
> > can not be shrunk when there are active requests?
> 
> I don't think there's such a statement, it's more of a conclusion 
> which is drawn based on the sequencing rules. We could make one of course.
> 
> >
> Say the server starts completing these requests, and sends replies to 
> 2, 3 and
> 1 because they complete in that order.
> 
> Further, when completing 1, the server decides to ask for some slots 
> back, so it sends sr_target_highest_slotid of 3, but even though it 
> completed all the active requests it doesn't yet know that the client 
> has seen them; it therefore maintains its sr_highest_slotid at 5. In 
> other words, while it is requesting that the window be shrunk, it 
> can't actually shrink it - i.e. retire any slots.
> 
> In the meantime, before seeing any of these replies, the client sent 
> an additional request, which of course used slot 4.
> 
> sr_target_highest_slotid, because it is less than 4. But, it can mark 
> slots 1, 2 and 3 as complete, can mark slot 4 as do-not-use (eligible 
> for future retirement), and can retire slot 5 immediately. The server 
> doesn't know any of this yet.
> 
> sr_highest_slotid of 5, and (probably) also send 
> sr_target_highest_slotid of 3. When it's received, the client can 
> retire slot 4, and reduce the slot table size to the server's target of 3. The server still doesn't know this.
> 
> request in slot 1, mark slots 2 and 3 as inactive, and can retire 
> slots 4 and 5. Its next reply will carry sr_highest_slotid of 3, upon which the client can finally retire slot 4.
> 
> moves independently, and at different times. If one side detects the 
> other side not responding to slot reducton, and needs the resources 
> back, the solution is to close the connection - starting things over from zero.
> 
> One thing to consider is that if the server does shrink the slot 
> window, it should be sensitive to the client's need for increased 
> slots. There's no forward-pressure (a so-called 
> sa_target_highest_slotid) so if the server sees the client using up 
> all its slots, it may want to consider growing the window, by 
> increasing the slot table and sr_highest_slotid. As mentioned, it can do this direction at any time.
> 
> Tom.
> 
> 
> > > > Subject: Re: [nfsv4] Clarifications requested for session 
> > > > dynamic slot management as described in RFC5661
> > > >
> > > > > management scheme as outlined in Section 2.10.6.1 in RFC5661. 
> > > > > A couple of questions have surfaced, that I'd like to raise 
> > > > > for general discussion. The issues raised are about "retiring slots"
> > > > > 1) If the intention is that the client and server no longer 
> > > > > have to cache information about slots that are retired, then 
> > > > > presumably slot sequence numbers are among the information 
> > > > > that is
> lost?
> > >
> > > > > 3) Exactly when should the client and server both agree that 
> > > > > the slot is retired?
> > > >
> > consider "sender" and "receiver" or similar. Also, the state of a 
> > slot changes differently for each end of the connection. The sender 
> > cannot retire a slot until it receives an appropriate 
> > acknowledgement that the peer has also done so. Your next questions address that.
> > >
> > > >
> > > > > the client MUST have sent a SEQUENCE op with sa_highest_slotid 
> > > > > < 'foo', and MUST have received a sr_highest_slotid reply from 
> > > > > the server that is also less than 'foo'. Is this correct?
> > >
> > these slots before doing so, of course. A new message in that slot 
> > might
> have been sent.
> > Any slots less than this active high-water should not be retired 
> > yet, although they perhaps can be marked as do-not-use (otherwise, a 
> > busy client might never comply with the downpressure).
> >
> > Thanks!
> >
> > > >
> > > >
> > > > > 4) How does the server ensure that it isn't sending 
> > > > > conflicting values of sr_target_highest_slotid (or of 
> > > > > sr_highest_slotid for that matter) to the client?
> > >
> > If sr_highest_slotid is larger than the currently highest active 
> > slot when received, it needs to be raised to that value.
> > >
> > > > > sr_highest_slotid until it has seen a new SEQUENCE request on 
> > > > > at least _one_ of the slots on which the server has sent a 
> > > > > reply with the current value of sr_target_highest_slotid. In 
> > > > > other words, there must be serialisation of the slot table 
> > > > > parameter updates on at least one slot. Does this make sense to all?
> > >
> > often". But as long as slots are thought of as representing an 
> > active range, without holes, the behavior is consistent even with 
> > misordered
> updates.
> > Where can I find text that supports the statement that the window 
> > can not be shrunk when there are active requests?
> >

---

From: Tom Talpey <[email-masked]>
Date: Mon, 26 Nov 2012 18:33:21 +0000

> Sent: Monday, November 26, 2012 10:44 AM
> > The server concludes that the client has seen all responses, in
> > particular, the response to the request using slotid 4.
> 
> If the client did send an sa_highest_slotid of 1, then the server could
> conclude that the only outstanding request was the one it just received.

Actually, because of the ordering property of the TCP (or RDMA) connection, the server can conclude the same thing either way. When the client reuses a slot, it implicitly acknowledges having seen the previous response on that slot, and therefore any earlier ordered responses for other slots. But, it's perhaps unlikely that servers will bother to remember the order in which they sent their responses.

> But, according to RFC5661, the value of sa_highest_slotid in this case would

Hmm. You are correct, I misstated. (At one point, I think sa_highest_slotid was the client's current view of the server's sr_highest_slotid, but that was later changed, to improve slot retirement efficiency.) The client needs to send sa_highest_slotid of 1 for the final exchange. This changes the scenario slightly, here's a corrected version of my example:

When the client receives the reply on slot 1, it cannot honor the sr_target_highest_slotid (3), because it is less than 4. But, it can mark slots 1, 2 and 3 as complete, can mark slot 4 as do-not-use (eligible for future retirement), and can retire slot 5 immediately. The server doesn't know any of this yet.

Finally, the client's *next* request will now carry sa_slotid 1 and sa_highest_slotid 1. Once it sees this, the server can start a new request in slot 1, mark slots 2 and 3 as inactive, and can retire slots 4 and 5. Its next reply will carry sr_highest_slotid of 3, indicating to the client that it can continue with 3 slots.

> 
> > Once it sees this, the server can start a new request in slot 1, mark
> > slots 2 and 3 as inactive,
> 
> But as this was presented, we were told that 3 was already compete, but
> maybe I'm mistaken about that.
> 
> > and can retire
> > > slots
> The server concludes that the client has seen all responses, in particular, the
> response to the request using slotid 4. Can you be more specific as to your
> concern?
> 
> Tom.
> 
> 
> >