Subject: [dhcwg] draft-ietf-dhc-dhcpv6-bulk-leasequery in IESG review

From: Jari Arkko <[email-masked]>
Date: Fri, 12 Dec 2008 00:03:08 +0200

Authors, WG,

I wanted to report on the status of this draft after tonight's IESG 
telechat. Everything else is in order, but Lars Eggert has a 
transport-related Discuss. Two issues were raised:

First, he wants to put in some text to recommend against going lower 
than the current timeout values, as too frequent reconnection attempts 
or giving up too fast can interfere with the TCP's mechanisms and cause 
congestion or even make the application work slower than it would with TCP.

This should be easy to do. Here's some suggested text:

   OLD:
     Implementations MAY make these values configurable.
   NEW:
     Implementations MAY make these values configurable.
     However, configuring too small timeout values may lead
     to harmful behavior both to this application as well
     as to other traffic in the network. As a result,
     timeout values smaller than the default values are
     NOT RECOMMENDED.

The other issue relates to how the draft has specified timeout behavior 
for various stages of the protocol. I just had a long chat with Lars and 
we think we understand the situation better now (authors, please ignore 
previous e-mail on this topic).

The high-level question we have is whether all this is necessary. For 
instance, the protocol includes retry attempts and timeouts for opening 
the connection. Or for maximum waiting time for blocking on the socket 
when sending the request.

TCP will do all this for you. I'm not sure it is worthwhile to try 
again, if TCP cannot establish a connection. It already tried very hard. 
And if the request/response cannot be sent, TCP will eventually timeout. 
Specifying your own timeout behavior will involve some pain (more on 
this below). Why do we need to do that? Are we expecting that this 
application is special in the sense that its timeliness requirements are 
more stringent than TCP's? Are there current implementations, and do 
they do this?

After a lengthy discussion with Lars, we think that your current scheme 
works. However, there are a few caveats that may or may not be obvious:

By default, send(2) only blocks if the *socket buffer* is full, and this 
is not the same as TCP itself being stuck. The fact that you 
successfully used the system call to send a small number of bytes, for 
instance, does not imply that TCP is making progress. In the bulk 
leasequery protocol this would very likely happen for requests. The end 
result would be that you'd never block on request. If TCP had trouble 
getting the request to the server, you'd eventually timeout on waiting 
for the response.

There are different ways to interpret the word "blocked" in the 
specification. Are we using it in the EAGAIN sense i.e., with 
non-blocking mode set, or simply in the sense of the send/write 
blocking? The latter would be incorrect. For instance, if the server 
sends a very large response with one call, it may take a while to 
transport everything. But TCP is still making progress, so it would seem 
to be inappropriate to fail just because you had a lot of data to send.

Finally, behavior at the end of a response becomes interesting depending 
on whether or not the socket is closed. If you write some amount of data 
to the socket buffer, send/write succeeds but you don't actually know 
how long it takes to transport the data. You'll get an error if TCP 
fails, but that would be much later. You can do a blocking close call, 
however. But I do not know if the server even cares about whether the 
data went through... maybe it is enough that the client waits for all 
the data to arrive, and eventually gives up if it does not.

If we decide to keep the timeout processing in the draft, some 
clarification regarding the above should probably be added.

What do you want to do?

Jari

_______________________________________________
dhcwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/dhcwg

---

From: Mark Stapp <[email-masked]>
Date: Fri, 12 Dec 2008 11:35:41 -0500

Thanks for the explanation and clarification - as you know, I wasn't 
entirely clear about what issue Lars was concerned about when he first 
raised his "DISCUSS".

I will make the suggested change about the configurable timeout values.

and yes, we were trying to specify a way to be somewhat more 
deterministic than TCP. the goal of the protocol is to support 
infrastructural use within an operator's network. if one device inside 
the network can't get a response from another device within 30 seconds, 
for example, that suggests that something fairly dramatic is happening, 
and waiting around for the stack's TCP timers isn't necessarily worthwhile.

perhaps the choice of the word "block" was a poor one - that'd be my 
fault. I was intending to mean flow of data, liveness, in the 
connection, not necessarily referring to a specific posix return status. 
do you think it's that one word that should be ... universally replaced 
(with "live" or "liveness", for example)? or should I simply add 
something that explains better what I mean by it?

Thanks,
Mark

Jari Arkko wrote:
> or giving up too fast can interfere with the TCP's mechanisms and 
> cause congestion or even make the application work slower than it 
> would with TCP.
>
> The other issue relates to how the draft has specified timeout 
> behavior for various stages of the protocol. I just had a long chat 
> with Lars and we think we understand the situation better now 
> (authors, please ignore previous e-mail on this topic).
>
> instance, the protocol includes retry attempts and timeouts for 
> opening the connection. Or for maximum waiting time for blocking on 
> the socket when sending the request.
>
> again, if TCP cannot establish a connection. It already tried very 
> hard. And if the request/response cannot be sent, TCP will eventually 
> timeout. Specifying your own timeout behavior will involve some pain 
> (more on this below). Why do we need to do that? Are we expecting that 
> this application is special in the sense that its timeliness 
> requirements are more stringent than TCP's? Are there current 
> implementations, and do they do this?
>
> After a lengthy discussion with Lars, we think that your current 
> scheme works. However, there are a few caveats that may or may not be 
> obvious:
>
> By default, send(2) only blocks if the *socket buffer* is full, and 
> this is not the same as TCP itself being stuck. The fact that you 
> leasequery protocol this would very likely happen for requests. The 
> end result would be that you'd never block on request. If TCP had 
> trouble getting the request to the server, you'd eventually timeout on 
> waiting for the response.
>
> transport everything. But TCP is still making progress, so it would 
> seem to be inappropriate to fail just because you had a lot of data to 
> send.
>
> Finally, behavior at the end of a response becomes interesting 
> depending on whether or not the socket is closed. If you write some 
> amount of data to the socket buffer, send/write succeeds but you don't 
> actually know how long it takes to transport the data. You'll get an 
> error if TCP fails, but that would be much later. You can do a 
> blocking close call, however. But I do not know if the server even 
> cares about whether the data went through... maybe it is enough that 
> the client waits for all the data to arrive, and eventually gives up 
> if it does not.
>

---

From: Jari Arkko <[email-masked]>
Date: Fri, 12 Dec 2008 23:13:20 +0200

Mark,

> connection, not necessarily referring to a specific posix return 
> status. do you think it's that one word that should be ... universally 
> replaced (with "live" or "liveness", for example)? or should I simply 
> add something that explains better what I mean by it?
The change is not that simple. The capabilities of APIs matter; you 
can't necessarily observe these things directly. The words themselves do 
not matter as much as explaining what you mean and providing sufficient 
guidance to implement this on commonly available platforms.

I think we should actually discuss the overall strategy a bit before 
make any changes. Can you speak about why you believe a more 
deterministic model than what TCP has is needed in this case? (And note 
that its hard to make the system entirely deterministic.)

Jari

---

From: Mark Stapp <[email-masked]>
Date: Fri, 12 Dec 2008 17:17:09 -0500

I guess I thought we'd supplied pretty explicit guidance already. if 
you've made a request but you're not receiving any data, you apply the 
timeout specified. if you're a server and you accept a connection, you 
apply the timeout if you don't receive a request. I'm not sure there's 
more "overall strategy" than that: we wanted  to make it possible to 
bound situations where limited resources were tied up. that's not 
"entirely deterministic" - it's just "be able to bound".

if you can't get a 200-byte request message up from an edge router to a 
more-central server within an operator's network in 30 seconds, there's 
no point asking the server to wait around for its TCP stack's keepalive 
timer. at least, that's the default. if an operator _wants_ to raise the 
timeouts to the multi-minute level, that's permitted by the draft, 
because the timeouts are configurable. is there a "commonly available 
platform" where what's described in the text can't be implemented?

it's not clear to me now what the issue is. I saw Lars's DISCUSS that 
said that we should be clear that it's the draft's timeout values that 
implementations should be applying, because they may not get indications 
from TCP. I agree, but when I looked at the text I saw that the words 
already said that - as I was reading them. I asked for a more specific 
suggestion, to help me address the DISCUSS. but this email, Jari, 
doesn't seem clear at all about what the objection is. we use TCP. we 
offer a configurable way to make determinations of progress or liveness 
that are, by default, more timely than (most) TCPs. an operator who 
wants to use the 2-hour TCP keepalive can, with the text that's in the 
draft now.

Thanks,
Mark

Jari Arkko wrote:
>> status. do you think it's that one word that should be ... 
>> universally replaced (with "live" or "liveness", for example)? or 
>> should I simply add something that explains better what I mean by it?
> can't necessarily observe these things directly. The words themselves 
> do not matter as much as explaining what you mean and providing 
> sufficient guidance to implement this on commonly available platforms.
>
> deterministic model than what TCP has is needed in this case? (And 
> note that its hard to make the system entirely deterministic.)
>
> Jari
>

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Sun, 14 Dec 2008 10:03:24 -0500

I think one point to keep in mind (for the IESG review) is that this is
not a general purpose Internet application protocol to be used across
the Internet or by lots of users at any given time.

This is designed for use in an operators backbone network (between DHCP
relay agents and DHCP servers) and if messages (TCP connections) are
delayed for large periods of time, the data itself that is being
communicated is unlikely to be current or of significant value.

Time is generally of the essence and that is why much shorter timeouts
are being applied here -- if the relay agent (well, router that is
usually co-located with the relay agent) can not recover the prefix
delegations rapidly, the operator's customers' TCP connections will
fail. 

- Bernie 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Mark Stapp (mjs)
Sent: Friday, December 12, 2008 5:17 PM
To: Jari Arkko
Cc: Dhcwg; IESG; Lars Eggert
Subject: Re: [dhcwg] draft-ietf-dhc-dhcpv6-bulk-leasequery in IESG
review

Thanks,
Mark

---

From: Thomas Narten <[email-masked]>
Date: Tue, 16 Dec 2008 12:53:52 -0500

At the risk of stepping in here without sufficient context, it seems
to me that the goal of terminating connections  earlier than TCP
normally does would be fine -- so long as certain constraints are met.

I understand that terminating connections earlier than the normal TCP
defaults is the motivation for all this, so corrective action can be
taken more quickly. 

That said, it seems to me to be poor protocol design to terminate the
TCP connection early, and then immediately try to reopen a new
connection to the same destination to try again (and the current
document seems to allow that, via "MAY" language). Doing that risks
bypassing TCP's congestion control, etc. And, it's not at all apparent
to me why doing that has any real benefit. (I.e., why that would work
any better than just waiting for the stalled TCP connection to recover
on its own.)

I.e., if you just terminated the TCP connection because it wasn't
responsive, immediately opening it again and trying again isn't likely
to work any better. Yet, the document says:

   A requestor attempts to establish a TCP connection to a DHCPv4 server
   in order to initiate a Leasequery exchange.  The requestor SHOULD be
   prepared to abandon the connection attempt after
   BULK_LQ_CONN_TIMEOUT.  If the attempt fails, the requestor MAY retry.
   Retries MUST use an exponential backoff timer, increasing the
   interval between attempts up to BULK_LQ_MAX_CONN_RETRY.

Which certainly suggests that just trying again immediately is
OK. That suggestion is not something we should suggest, IMO.
   
So, I'm curious as to what the real intention here is, with respect to
having the timers. I.e., what is an implementation supposed to do when
the timer goes off.

Thomas

---

From: Mark Stapp <[email-masked]>
Date: Tue, 16 Dec 2008 15:19:59 -0500

Hi Thomas,

so adding text that makes it clear that immediate connection retries is 
not permitted would address this? that is, the intent has always been to 
allow periodic connection attempts, with a configurable bound.

Thanks,
Mark

---

From: Thomas Narten <[email-masked]>
Date: Tue, 16 Dec 2008 17:24:20 -0500

> allow periodic connection attempts, with a configurable bound.

Actually, I might express this differently.

I think that implementations have always been free to terminate
connections whenever they want. We don't need text in any spec to say
that.

Going back to text in the document:

> >    BULK_LQ_CONN_TIMEOUT.

First,, if the timeout is configurable, then I don't understand why this
isn't a MUST (but that is a separate issue)

> >    If the attempt fails, the requestor MAY retry.

This text is what seems out of place to me. An implementation is free
to do a number of (possibly ill-advised) things. Why does the spec
need to explicitely say this? This sounds to me like someone trying to
give a particular implementation explicit permission to do
something. I would prefer to just see the text deleted.

Looking at additional text (and what probably made Lar's nervous...)

> 8.3.  Processing Bulk Replies
> 
>    The requestor attempts to read a DHCPv4 Leasequery message from the
>    TCP connection.  If the stream of replies becomes blocked, the
>    requestor SHOULD be prepared to terminate the connection after
>    BULK_LQ_DATA_TIMEOUT, and MAY begin retry processing if configured to
>    do so.

This seems iffy to me. There are many reasons why a connection might
block. Maybe the network is now down. Maybe the server is over loaded.
Why terminate the connection under such circumstances? The above could
actually make the application LESS robust than one would normally
expect.

Well, OK, if you want to terminate go ahead. But, it would now be
completely inappropriate (IMO) to retry the query immediately. If you
are going to retry, you really should not have terminated the
connection in the first place. That is the standard IETF way of doing
things (for good reasons) and IMO, if you want to deviate from that
model, you need to explan what problem this is trying to improve upon.

So the real question above is, what happens after terminating the
connection? Do you expect the transaction to fail and that is it? Or
do you expect the application to retry? I have to sort of guess what
it is that you expect the requestor to do at this point. What is the
actual requirement of the USER of this protocol in this area?

> 9.2.  Replying to a Bulk Leasequery
> 
>    If the connection becomes blocked while the server is attempting to
>    send reply messages, the server SHOULD be prepared to terminate the
>    TCP connection after BULK_LQ_DATA_TIMEOUT.

This seems even more iffy to me. There are many reasons why a
connection might block. One reason could be that the server sent so
much data, the client can't consume it. Or the client was suspended
(put in background) and isn't reading data anymore. Why terminate the
connection under such circumstances? Again, the above could actually
make the application LESS robust than one would normally expect (e.g.,
if the requestor and server have different values configured for the
timeout).

Thomas

---

From: Jari Arkko <[email-masked]>
Date: Wed, 17 Dec 2008 06:27:25 +0800

Mark,

> so adding text that makes it clear that immediate connection retries 
> is not permitted would address this? that is, the intent has always 
> been to allow periodic connection attempts, with a configurable bound.
That particular problem would be easy to correct with that. (There were 
a few things were the document should be more specific or give better 
guidance, e.g., the nature of blocking with the usual APIs that you have.)

However, lets think about what you really achieve with the strategy in 
the case that Thomas brought up. For an initial period of X seconds you 
will let TCP handle the opening of the session. Then you will close it 
(causing a RST to be sent), and stay quiet for Y seconds. At Y seconds 
you will send a SYN as you re-attempt opening the TCP session.

If on the other hand, you simply let TCP deal with the whole thing, you 
would have had exactly the same behaviour up until X seconds. From there 
onwards you would have had SYNs sent, attempting to set up the 
connection. A small number of SYNs are sent at this stage because TCP 
has already slowed down quite a bit by X seconds if there was no 
response. And then finally at Z seconds you would give up. You could 
then attempt re-opening the connection or simply signal failure.

What did we achieve with our own timer scheme? AFAICT, in bulk 
leasequery we can keep trying to contact a server, but if it does not 
answer, we have no useful alternate strategy. There's typically no other 
DHCP server to ask for the same information, so all we can do is either 
keep trying or give up... and alerting management that something is 
wrong can be done independently of what is going on with the connection 
opening (if after 10 s you do not have a connection, you could raise an 
alarm).

Also, the strategy of killing the TCP connection attempt sooner does not 
really reduce the number of packets in the network that much, as TCP has 
already slowed down. But if we kill the connection, there is no way you 
can recover between X and Y seconds.

Jari

---

From: Jari Arkko <[email-masked]>
Date: Wed, 17 Dec 2008 06:35:17 +0800

Thomas, Mark,

> actual requirement of the USER of this protocol in this area?
I think this is the crux of the issue. A device can open one or multiple 
bulk leasequery connections to DHCP servers. However, if one of these 
connections fail, under normal conditions it is not clear to me that 
there is a useful alternate. (Unless you have failover DHCP servers and 
you go to the other one). The session data that you were expecting to 
see is only available from the DHCP server that you are attempting to 
connect to.

So if we don't seem to be making progress, what else can we do than to 
keep trying? We can certainly tell the human user (irrespective of what 
is done with the connection) that this is taking a long time. But giving 
up too early... why? And rerunning the entire query from start, 
immediately upon failure seems like the wrong thing from congestion etc. 
perspective. Finally, give up + stay quiet for a while + rerun the query 
a bit later seems to be less robust than simply letting TCP attempt to 
make progress. You can't recover during the "stay quiet" period.

Jari

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Tue, 16 Dec 2008 23:30:01 -0500

What human user?

This protocol is meant to be used by relay agents. There's usually no
human involved!!

Most DHCP servers and relay agents will NOT want to use normal TCP
connection time out times as those might allow connections to be open
for hours. As I previously stated, if the data isn't available fairly
quickly, it is likely to be stale and not usable. We could argue for
years whether 30 second, 120 seconds, 10 minutes or whatever are the
right values.

If the information isn't available, the relay agent will attempt to
recover what it can and go one -- possibly doing individual UDP
leasequery as needed to recover leases.

Perhaps the solution is just to remove the text regarding connection
handling and let implementers do what they want (the text can point out
the issues).

- Bernie 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Jari Arkko
Sent: Tuesday, December 16, 2008 5:35 PM
To: Thomas Narten
Cc: Dhcwg; Mark Stapp (mjs); Lars Eggert; IESG
Subject: Re: [dhcwg] draft-ietf-dhc-dhcpv6-bulk-leasequery in IESG
review

Thomas, Mark,

Jari

---

From: Ted Lemon <[email-masked]>
Date: Tue, 16 Dec 2008 23:03:39 -0600

Why would a broken TCP connection stay up for 10 minutes?   Shouldn't  
it die within 90 seconds if no ACKs are forthcoming from the remote end?

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Wed, 17 Dec 2008 00:46:53 -0500

Oh, no way!

If the both ends enable keep-alives (some stacks do by default), the
there will be periodic probes but they generally don't time out for
minutes (I think for most BSD systems it works out to 9 or more if I
remember correctly).

And, if no keep-alives are sent, if the server crashes, the relay may be
waiting forever.

While everyone mentions non-fatal failures -- such as slow servers --
there are lots of other conditions that could cause a significant delay
in detecting a downed server or relay agent.

These connections are mostly a one-way data flow - relay agent sends a
request, server sends lots of responses.

TCP is great, don't get me wrong, but sometimes you need faster response
than it offers.

- Bernie 

-----Original Message-----
From: Ted Lemon [mailto:[email-masked]] 
Sent: Wednesday, December 17, 2008 12:04 AM
To: Bernie Volz (volz)
Cc: Jari Arkko; Thomas Narten; Dhcwg; Mark Stapp (mjs); Lars Eggert;
IESG

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Wed, 17 Dec 2008 08:26:24 -0500

There are two cases:

1. Responses (from the server to relay) are blocked because relay isn't
reading data fast enough.

2. Responses (from the server to relay) have stopped being sent because
the server is not there.

In case 1, the relay would never know about connection issues since it
isn't reading. This can also mean that there is a communication problem.
Or, it could mean (at least for a short time) that the relay is gone --
if the window was full when the relay went down, it will take a periodic
probe to discover that.

In case 2, the relay would be waiting for data that never arrives.

We're worried more about case 2.

- Bernie 

-----Original Message-----
From: Lars Eggert [mailto:[email-masked]] 
Sent: Wednesday, December 17, 2008 2:44 AM
To: Bernie Volz (volz)
Cc: Ted Lemon; Jari Arkko; Thomas Narten; Dhcwg; Mark Stapp (mjs); IESG
Subject: Re: [dhcwg] draft-ietf-dhc-dhcpv6-bulk-leasequery in IESG
review

> minutes (I think for most BSD systems it works out to 9 or more if I

TCP keep-alives only kick in when there is no data to send in both  
directions, which isn't the case you're worried about. In your case,  
one end is trying to send data, but the throughput it is achieving  
isn't considered sufficient. The timeouts for failing to make send  
progress are usually on the order of minutes in typical stacks and  
often configurable via setsockopt. (For some more background  
information, see draft-ietf-tcpm-tcp-uto.)

> And, if no keep-alives are sent, if the server crashes, the relay  

Not if it is trying to send data and not making progress.

Lars

PS: And yes, Thomas note pretty much matches my concerns with the  
document.

---

From: Thomas Narten <[email-masked]>
Date: Wed, 17 Dec 2008 08:51:00 -0500

Maybe I'm dense, but I still don't get it.

> leasequery as needed to recover leases.

This scares me. If TCP doesn't work, retrying to the *SAME* server
using UDP isn't the right way to fix the problem. Indeed, why would
UDP work if TCP doesn't? Exactly what type of failure/scenario are you
concerned about?

Can you please describe a scenario where retrying using UDP actually
makes sense?

As others have indicated, I still don't get the actual problem, and
how the suggested workarounds actually address the problem.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Wed, 17 Dec 2008 08:49:05 -0500

Thomas

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Wed, 17 Dec 2008 12:09:21 -0500

Thomas:

The TCP connection (bulk leasequery) is designed to recover all of the
leases quickly.

The UDP leasequery is used when a packet arrives that a router
(typically CMTS which is both a router and relay agent) has no
information for (ie, it doesn't know which client behind which CM is
using that address). In this case, it asks the DHCP server (via a UDP
leasequery) about that individual lease (query by address). This
happens, on demand, when traffic arrives. This is the ONLY recovery
technique that exists today. The rate at which these requests are
generated is limited by traffic arrival (and hopefully rate limiting).
Once information is known (either via TCP bulk LQ or UDP LQ), no further
queries are needed.

The problem with DHCPv6 is that if prefix delegation is being used and
the CMTS (router) is also advertising those routes in order for traffic
to get there, no traffic will arrive until the PD data is recovered and
the routers advertised. Therefore, the UDP based leasequery can't be
used since the relay agent/router has no idea what to request.

The general model is that likely BOTH TCP (for bulk and PD) and UDP (on
demand when no information exists) leasequery would be used. While bulk
(TCP) is loading the configuration, traffic may arrive and thus generate
a UDP LQ as well. TCP is only used to seed the information initially --
after successful bulk LQ, future information is learn by monitoring
DHCPv6 traffic.

The UDP LQ are extremely efficient -- after all, the DHCP server is all
about address management and looking up an address is very easy and
quick for it.

If the server is down, what difference does it make if TCP or UDP are
being used? Sure, traffic is being generated, but it isn't going to be
answered. In which case, the relay will not have information and be
forced to drop packets on the floor. If the relay only relied on TCP, it
would be forced to wait until potentially ALL data was downloaded (it
may be the last lease that is the one needed "now"). And, remember that
bulk LQ does not yet exist -- so UDP is being used today.

Also, remember that this is all between components (relay agents/routers
and DHCP servers) and not thousands of users on the Internet. Traffic
provision in an operators' network needs to accommodate this.

- Bernie

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Thomas Narten
Sent: Wednesday, December 17, 2008 8:49 AM
To: Bernie Volz (volz)
Cc: Dhcwg; Mark Stapp (mjs); Lars Eggert; IESG
Subject: Re: [dhcwg] draft-ietf-dhc-dhcpv6-bulk-leasequery in IESG
review

Thomas

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Wed, 17 Dec 2008 12:50:25 -0500

Sorry to add DHC WG and a few others but I think this is important to
this discussion ...

---

If enabled.

I also think that the time frame is much longer than 180 seconds -- more
like 9 minutes on most Unix implementations, but read on!

My experience with TCP (I actually implemented a stack in the 1980s -
1990s -- it was NOT BSD derived as most implementations are, but home
grown) does date back about 10 years now. But as I recall, keepalives
tended to keep connections alive for long periods as they were designed
to survive short term network outages -- here we are taking minutes, not
seconds or days.

And, RFC 1122 has:

         4.2.3.6  TCP Keep-Alives

            Implementors MAY include "keep-alives" in their TCP
            implementations, although this practice is not universally
            accepted.  If keep-alives are included, the application MUST
            be able to turn them on or off for each TCP connection, and
            they MUST default to off.

            Keep-alive packets MUST only be sent when no data or
            acknowledgement packets have been received for the
-->         connection within an interval.  This interval MUST be
-->         configurable and MUST default to no less than two hours.

So, we have no less than 2 HOURS as the MUST default!!

A relay agent would not want to wait 2 hours to recover from this
condition. A DHCP server also may well chose not to leave open a
connection for 2 hours. Neither of these actions are banned.

- Bernie 

-----Original Message-----
From: Ted Lemon [mailto:[email-masked]] 
Sent: Wednesday, December 17, 2008 12:15 PM
To: Bernie Volz (volz)
Subject: Re: [dhcwg] draft-ietf-dhc-dhcpv6-bulk-leasequery in IESG
review

On Dec 17, 2008, at 11:10 AM, Bernie Volz (volz) wrote:
> Client sends request ... Received by server. Server crashes before
> sending a single byte as a response. No data is flowing.

Hm, okay.   So in this case wouldn't keepalives kill the connection  
after a maximum of 180 seconds?

---

From: Mark Stapp <[email-masked]>
Date: Wed, 17 Dec 2008 16:11:29 -0500

The motivation for offering administrative controls has been to
allow the client (a dhcp relay) to notice a condition in a
deterministic way. Having noticed the condition - which might be a
failure to connect in a timely way, or a failure to receive a timely
answer to a query - the relay can do _something_. The objection seems
to be about what it can do, as the draft is written now - correct?

The concrete objections seem to be:

1. allowing a relay to terminate a connection attempt, and then attempt
to reconnect. the objection is that this fails to take advantage of
TCP's own retry mechanisms.

2. allowing a relay to attempt UDP, one-at-a-time queries if it
determines that the TCP, bulk query is not succeeding.

I do think case 2 is valid. it's possible that a server may be under
TCP DOS attack for example, or it may have reached a limit in the
number of TCP lease queries it's willing to accept and process in
parallel. in either of those cases, a server may be willing and able to
process a lighter-weight, single query. I don't think that's a
highly-likely case, but I think it's legitimate, and there's no reason
to "forbid" it.

I think case 1 is a little subtle, but I'm willing to try to change the
text about connection attempts. I'd like to say that there's no benefit
to terminating a TCP connection attempt and retrying it. A relay may
take action after the draft's connection timeout - it may try to
connect to another server, or it may just signal the condition or write
to its log. but the relay shouldn't abandon the connection attempt if
TCP hasn't signalled a connection failure and if it intends
to retry the same server again.

Would that help - Thomas? Lars?

Thanks,

Mark

Thomas Narten wrote:

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Wed, 17 Dec 2008 17:05:27 -0500

Bernie, thanks again for the long explanation. I have no issue with
the scenario you describe.

> Once information is known (either via TCP bulk LQ or UDP LQ), no further

So, the usage of TCP vs. UDP is completely unrelated. That is, you
don't start with TCP and then if that fails, try UDP or anything like
that.

Rather, on system restart, you use TCP, because you are trying to
restore lost state. If, while you are waiting, you happen to get a
packet you don't know what to do with, you issue a UDP query just for
the one address. That seems fine, since this is what you do normally,
independent of whether any TCP query is happening in background.  An
important point here is that there is no need to tie the UDP & TCP
behavior together - they are logicially independent, invoked at 
mostly unrelated times.

My recommendation would be to just keep the two usages separate and
avoid saying things that make people think that if TCP fails, you need
to failover to using UDP. You don't. (An earlier message suggested
that.)

But, that does again raise the question of what do you do if TCP
"fails"? You presumably do either nothing (in which case terminating
the TCP connection early seems pointless), or you restart the TCP
query (because you still have to restore all that missing state, and
UDP queries can't do that... you only use UDP for specific, narrow
queries). Again, IMO, you should only restart the TCP connection if
you have good reason to think that retrying would help. It would go
against long-standing IETF practice to try to restart more
aggressively than TCP already does. IMO, you need a compelling
argument to go down this path. So I just don't think you need to
terminate a TCP connection early. I don't see how that helps the relay
agent at all.

I would just describe the two usages separately and not tie them
together.

> after successful bulk LQ, future information is learn by monitoring

Works for me.

> answered.

Well, I would argue that you might well want to think about rate
limiting UDP lease queries generally. Consider a DOS attack using
addresses in a packet that the CMTS gets. Do you really want the CMTS
pounding on the server for *each* of those packets? Or, consider when
it gets a train of legitimate TCP packets from a (pre-existing)
bittorrent connection.. Do you want each packet to cause a UDP query
to the server? I would think not... I.e., the same issue happens with
ARP/ND, and they are typically rate limited to prevent storms in this
scenario...

> In which case, the relay will not have information and be

So one thing I don't quite get. You said earlier that in some cases,
UDP is insufficient because traffic won't arrive at the CMTS to
generate the query. Hence, Bulk Query is needed. But here you say,
Bulk Query isn't deployed yet. Does that mean that current deployed
technology/standards has real holes (i.e,. lead to permanent
failures)?

> provision in an operators' network needs to accommodate this.

I hear you, but that argument has the problem that no technology
controls where it gets used in practice, so its not good for our
protocols to rely on this for proper behavior.

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Wed, 17 Dec 2008 17:17:06 -0500

> 2. Responses (from the server to relay) have stopped being sent because

Well, this seems like a pretty narrow edge case. I.e., you just opened
a connection to the server, then before the server actually starting
sending data, it crashes. This can happen with *ANY* protocol
today. But it doesn't happen very often, and I can think of few
protocols that actually worry about this case. I.e., we are talking of
a window of a second or less, in most cases.

Is it *really* that important to handle this case in your scenario?
And how quickly would you need to detect this? (And the question of
*how* would you respond to this event after detection is still
unclear, IMO.)

Anyway, a better way to deal with that is to (at the application
layer) add a specific "are you still there" ping/reply liveness test
(across TCP). You could do that at the application level pretty
easily.  But it seems like overkill to me.

Thomas

---

From: Ted Lemon <[email-masked]>
Date: Wed, 17 Dec 2008 16:33:38 -0600

> easily.  But it seems like overkill to me.

The problem with doing this is that in the scenario Bernie's talking  
about, you've already sent a request, and there's no protocol for  
sending another one until the response to the first request comes  
back.   And if your platform doesn't support keepalives, or doesn't  
allow you to control the keepalive interval, then the protocol is  
simply wedged until such time as the relay is rebooted.   So even  
though it's an unlikely case, it matters because it causes a fairly  
serious problem.

It seems to me that in this case it makes sense to propose that if  
after a request has been transmitted, if no response is heard for some  
period of minutes, the requester should break the connection and try  
again.   But there's no need to set the tolerances shorter than the  
TCP timeout interval, at least as far as I can see.

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Wed, 17 Dec 2008 18:09:40 -0500

>to failover to using UDP. You don't. (An earlier message suggested

I think the earlier message was just trying to convey this. I don't
believe the I-D actually says any such thing (I took a quick look and
didn't see anything like this, but perhaps I missed it).

>limiting UDP lease queries generally. Consider a DOS attack using
 
Rate limiting is always a good idea (for UDP). This would be an issue
for RFC 5007 (and 4388 for DHCPv4).

In the RFC 5007 in the security considerations we have:

   Since even trusted access concentrators may generate LEASEQUERY
   requests as a result of activity external to the access concentrator,
   access concentrators SHOULD minimize potential denial-of-service
   attacks on the DHCPv6 servers by minimizing the generation of
   LEASEQUERY messages.  In particular, the access concentrator SHOULD
   employ negative caching (i.e., cache the fact that a particular
   recent query failed to return client data) and address restrictions
   where possible (i.e., don't send a LEASEQUERY message for addresses
   outside the range of the attached broadband access networks).
   Together, these mechanisms limit the access concentrator to
   transmitting one LEASEQUERY message (excluding message retries) per
   legitimate broadband access network address after a reboot event.

   Packet-flooding denial-of-service attacks can result in the
   exhaustion of processing resources, thus preventing the server from
   serving legitimate and regular DHCPv6 clients as well as legitimate
   DHCPv6 LEASEQUERY requestors, denying configurations to legitimate
   DHCPv6 clients as well lease information to legitimate DHCPv6
   LEASEQUERY requestors.  While these attacks are unlikely when only
   communicating with trusted LEASEQUERY requestors, the possibility
   always exists that the trust is misplaced, security techniques are
   compromised, or even trusted requestors can have bugs in them.
   Therefore, techniques for defending against packet-flooding denial of
   service are always a good idea, and they include good perimeter
   security, as mentioned earlier, and rate limiting DHCPv6 traffic by
   relay agents, other network elements, or the server itself.

So, I think this issue is pretty well covered there?

>technology/standards has real holes (i.e,. lead to permanent

We've worked on this issue with the RAAN proposal
(draft-droms-dhc-dhcpv6-agentopt-delegate which has expired). This is
really only an issue with Prefix Delegation (RFC 3633), and when
non-aggregated prefixes are delegated. This probably hasn't been common
yet since it is a missing piece to recover these leases. UDP LQ (RFC
5007) can be used to recover the prefix delegations, but only if traffic
is received that has an address contained by a delegated prefix. But, if
the prefix needs to be advertised before traffic is received ... There's
a catch-22. Hence, one of the key reasons Bulk LQ is required. An
alternative is to use some routing protocol between the requesting and
delegating router.

I think there really are two key issues for Bulk LQ with respect to the
TCP connections:

1. Requestor (relay agent) - if a request was initiated but responses
are not furthcoming in a "timely" fashion.

There are three primary causes:
A) The server is unable to communicate -- it is down.
B) The network prevents communication.
C) The server is "slow" (busy) and not responding in a timely fashion.

A is clearly the one that is important to detect. But, it is often
difficult to distinguish between the three.

Clearly a timeout is appropriate here as TCP could allow a long period
of time (hours) to go by before reporting a problem. A user would
generally give up (of course, a user would likely use recent experience
to determine whether the wait is appropriate or not).

2. Server - if a connection is open but no outstanding request exists on
it.

After some period of time, a server is again within its right to close
the connection to "conserve" resources.

I believe the main concern is not about disallowing the above actions,
but what is a reasonable time period to allow TCP "to do its thing" and
avoid re-doing work by reopening a connection and repeating a request
(perhaps with the same result). In particular the following parameters
are the issue:

   Parameter             Default  Description
   ------------------------------------------
   BULK_LQ_CONN_TIMEOUT  30 secs  Bulk Leasequery connection timeout
   BULK_LQ_DATA_TIMEOUT  30 secs  Bulk Leasequery data timeout

I suspect that removing BULK_LQ_CONN_TIMEOUT would be fine as TCP
already handles this (usually with a 90 second timeout). The
BULK_LQ_MAX_RETRY parameter (60 seconds) I don't think should be an
issue -- waiting at most a minute before retrying to connects seems
reasonable, as recovering the data quickly is a goal and thus waiting
too long prevents that.

For BULK_LQ_DATA_TIMEOUT, 30 seconds is likely too short. Making this
several minutes would be far better. Perhaps a value of 5 minutes would
be a reasonable default, with recommendations not to make it too short
as it prevents TCP from doing its thing. While larger values are also
possible, I think if it takes 5 minutes to get data through, something
is very wrong and there are bigger issues (retrying the connection with
the same result is not going to cost a lot every 5 minutes). (If a user
where initiating this, I suspect they would long have given up -- do you
wait 5 minutes to a web page to load??)

- Bernie

-----Original Message-----
From: Thomas Narten [mailto:[email-masked]] 
Sent: Wednesday, December 17, 2008 5:05 PM
To: Bernie Volz (volz)
Cc: Dhcwg; Mark Stapp (mjs); Lars Eggert; IESG
Subject: Re: [dhcwg] draft-ietf-dhc-dhcpv6-bulk-leasequery in IESG
review

Works for me.

Thomas

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Wed, 17 Dec 2008 20:06:36 -0500

>a connection to the server, then before the server actually starting 

Perhaps you copied and pasted the wrong case, but that particular one is
not such a narrow edge case. If there are 100K leases to be
communicated, the server could "stop" sending at any time during those
100K leases. If the server crashed, how would the receiver know. (I
don't know what the scale is likely to be for most cases, but having
tens of thousands of DHCPv4 leases on a CMTS in Cable is not unrealistic
today -- with IPv6 there's no reason not to expect the same level of
scale once deployed.)

>easily.  But it seems like overkill to me.

Agreed; overkill. It also means that the server must be able to handle
multiple requests from a client (the draft allows for the server to
support this, but does not require it).

I think for this case, waiting for a number of minutes should work
reasonably well because not being able to send anything in several
minutes is unlikely for a server and it addresses the ability for TCP to
ride out short term connectivity issues (and for retransmissions to
occur). And, again, this is likely to be used in environments where
guarantees of service are much more likely -- and if they aren't met,
there will be bigger issues than retrying TCP connections ever few
minutes.

- Bernie

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Thomas Narten
Sent: Wednesday, December 17, 2008 5:17 PM
To: Bernie Volz (volz)
Cc: Mark Stapp (mjs); IESG; Lars Eggert; Dhcwg; Ted Lemon
Subject: Re: [dhcwg] draft-ietf-dhc-dhcpv6-bulk-leasequery in IESG
review

Thomas

---

From: Lars Eggert <[email-masked]>
Date: Wed, 17 Dec 2008 09:43:32 +0200

Lars

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Thu, 18 Dec 2008 11:47:15 -0500

Lars:

One of the events I'm worried about is when the server system becomes
unreachable and thus no data is flowing.

The other case is when the server is waiting for the first or "next"
request after having completed one - such as when the relay agent
becomes unreachable either after opening the connection or after having
received all of the data.

In neither of these cases is data flowing. Hence, keep-alives are
required or an application timeout to handle the case when no data is
received over some period of time.

- Bernie

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Lars Eggert
Sent: Wednesday, December 17, 2008 2:44 AM
To: Bernie Volz (volz)
Cc: Thomas Narten; Mark Stapp (mjs); IESG; Dhcwg; Ted Lemon
Subject: Re: [dhcwg] draft-ietf-dhc-dhcpv6-bulk-leasequery in IESG
review

Lars

---

From: Lars Eggert <[email-masked]>
Date: Fri, 19 Dec 2008 16:37:21 +0200

Hi,

> unreachable and thus no data is flowing.

there are two cases here:

Case 1: the connection is idle, i.e., neither end has data queued for  
transmission in the socket buffers. This is the case where TCP  
keepalives help. They let you detect that the other end has  
disappeared, which without keepalives you'd only notice when you'd try  
to send to or receive from the other end. Question: if your end isn't  
trying to send to or receive from the other end, is there any use in  
tearing down the connection via a keealive?

Case 2: the connection is not idle, i.e., at least one end has data  
queued for transmission in its socket buffer. TCP keepalives aren'a a  
factor here - the TCP stack on the sending side will tear down the  
connection after a few sending attempts fail to generate ACKs. On  
typical stacks, this happens after a few minutes.

Is there any need for an additional mechanism in either of these two  
cases?

Lars

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Fri, 19 Dec 2008 10:25:20 -0500

For case 1, keepalives help but the application is left to the stack to
eventually signal the timeout. An application should NOT be required to
rely solely on that -- an application should be able to shut down the
connection earlier if it feels it isn't making sufficient progress.
Also, keepalives may not always be available (?) or there are other
issues that prevent keepalives from being completely effective. And, the
connection may be idle because it just is - neither side has any data to
send. Thus, again, a server (or client) should be able to close the
connection. FTP does this.

I think none of these cases are a problem *IF* the "timeout" is
reasonable.

For case 2, there are two causes - one is that the receiver isn't
reading data from the socket and hence the sender is blocked from
sending more data; the other is that the receiver is no longer there (or
is unreachable for an extended period). The first would "never" abort
the connection unless the one end takes some action. The second will
eventually result in the connection being shut down (but note that in
this case, the sender detects this but the receiver end may not -- that
depends on your case 1.

Again, an application should be able to have its own time out as long as
it is reasonable.

In any case, it is NOT appropriate to rely ONLY on TCP to provide these
signals. Clients and servers that do will sometimes end up waiting for
action for long periods of time (something usually gives in the end but
it could be days, weeks, or months and is often the result of some
operator or 'external' (ie loss of power) action).

If an application timeout does not exist, the cost is consuming some
system (and application) resources. It may not be high for each
connection, but if one were to have a bug in a relay agent software
release that causes it to stop reading and one were to have 100s of
those relays ... I think you get the point.

This is NOT new. FTP servers and other applications have had application
level timeouts. I think the orignal concerns were what we need to go
back to:
A) The timeout times were very short; they need to be longer to allow
TCP to "do its thing".
B) The recovery techniques, especially if the timeouts were short, were
flawed because they recommended to retry the connection/operation. If
the timeout were 10 minutes and the recovery was as stated, I don't
think this would be that big an issue because trying once every 10
minutes is not a major concern.

RFC 1123 states:

         4.1.3.2  Idle Timeout

            A Server-FTP process SHOULD have an idle timeout, which will
            terminate the process and close the control connection if
            the server is inactive (i.e., no command or data transfer in
            progress) for a long period of time.  The idle timeout time
            SHOULD be configurable, and the default should be at least 5
            minutes.

            A client FTP process ("User-PI" in RFC-959) will need
            timeouts on responses only if it is invoked from a program.

            DISCUSSION:
                 Without a timeout, a Server-FTP process may be left
                 pending indefinitely if the corresponding client
                 crashes without closing the control connection.

So, if the draft had specified a BULK_LQ_DATA_TIMEOUT of 5 minutes, we
would NOT be having this discussion, would we?

- Bernie

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Lars Eggert
Sent: Friday, December 19, 2008 9:37 AM
To: Bernie Volz (volz)
Cc: Thomas Narten; Dhcwg; Ted Lemon; Mark Stapp (mjs); IESG
Subject: Re: [dhcwg] draft-ietf-dhc-dhcpv6-bulk-leasequery in IESG
review

Hi,

Lars

---

From: Lars Eggert <[email-masked]>
Date: Fri, 19 Dec 2008 17:41:40 +0200

Hi,

> connection earlier if it feels it isn't making sufficient progress.

I agree that an application is free to stop a connection when it feels  
that it doesn't make sufficient progress, and assuming that mechanism  
are in place to establish some quiet time after that has happened.

But I'm confused to hear you talk about "progress" here. Case 1 is  
when neither end has any data to send - there isn't anything on which  
to make progress on. Which is why I'm wondering why it is considered  
important to tear such connections down.

> connection. FTP does this.

Generally agreed, but not in case 1.

> I think none of these cases are a problem *IF* the "timeout" is

Yup.

> Again, an application should be able to have its own time out as  

I agree - my discuss position isn't saying that you shouldn't do this,  
it's saying that the current description of the timeout scheme isn't  
sufficiently well described.

It contains statements like "is blocked", "makes progress", etc. that  
simply aren't visible via the sockets API. Even when that's changed to  
talk about having been able to send or receive within some amount of  
time, the only thing the application knows is that the socket API call  
returned, which esp. for sending does NOT correspond to progress  
happening at the connection level, at least for short sends.

> minutes is not a major concern.

Yes. And the description of the timeout scheme needs to be written in  
a way that's actually implementable based on the information that the  
TCP API provides to the application.

> would NOT be having this discussion, would we?

We wouldn't have the part of the discussion about the length of the  
timeouts. We would still be having the discussion about the clarity of  
the description of the proposed timeout scheme.

Lars

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Fri, 19 Dec 2008 10:53:26 -0500

>important to tear such connections down.

Huh? This is exactly what the FTP idle timeout text in RFC 1123 was all
about!

---

>simply aren't visible via the sockets API. Even when that's changed to

"progress" and "blocked" are VISIBLE from the socket API.

Progress means that you're able to issue additional send() or recv()
calls and get them completed (before whatever timeout, if any).

Blocked means that you aren't. If the socket is set to non-blocking
mode, EWOULDBLOCK would be returned.

You can easily write code to do this by using select to limit how long
you wait to send or receive (you don't need to use non-blocking
sockets). Before reading, select on the socket being readable with a
timeout. Before sending, select on the socket being writeable with a
timeout (though in this case, there is a risk of blocking on the send if
the socket is non-blocking).

- Bernie

-----Original Message-----
From: Lars Eggert [mailto:[email-masked]] 
Sent: Friday, December 19, 2008 10:42 AM
To: Bernie Volz (volz)
Cc: Thomas Narten; Dhcwg; Ted Lemon; Mark Stapp (mjs); IESG
Subject: Re: [dhcwg] draft-ietf-dhc-dhcpv6-bulk-leasequery in IESG
review

Hi,

Yup.

Lars

---

From: Thomas Narten <[email-masked]>
Date: Fri, 19 Dec 2008 10:59:06 -0500

Thanks again Bernie for helping clarify.

It seems that the real issue is that the client wants to detect the
situation where it has already opened a TCP connection, has sent a
request to the server, but sometime before the server sends its last
response, the server dies. Outside some external timer, the connection
hangs for a long time, possibly foerever (if no keepalives are
available).

We seem to have agreement that terminating a TCP connection open
attempt prior to TCP concluding on its own that the connection cannot
be established is not necessary. I.e,

>    BULK_LQ_CONN_TIMEOUT  30 secs  Bulk Leasequery connection timeout

Is not necessary and can be removed.  Is there agreement on this
point?

Second, I think it is fine to have application level timeouts. There
are justifiable reasons to do that. The only issue that needs
discussing (IMO) is reasonable values for such timeouts. What we want
to avoid is having the application have such short timeouts that it
works at cross-purposes with TCP's underlying reliability/congestion
avoidance mechanisms. If we had minimum timeouts of a few minutes,
that would be fine (IMO). But the document has 30 second timeouts for
this case. That seems low to me, because IMO, you are more likely to
react (hastily) to network congestion (or a temporary routing problem)
then because the server has died. I would expect server crashes to be
much less common than temporary network outages. Right?

And, I'm OK with terminating the connection even after just 30
seconds, so long as the recovery is NOT to retry the connection
again. The
current document suggests doing just that via a MAY.

So, bump the timeout to something longer (e.g., 2 or 3 minutes) and
IMO that would be OK.

And, if someone still wants to argue for a shorter timeout in this
case, I still haven't recieved an answer to my question of *why*
terminating early in this case and *not* retrying makes sense.

Does this clarify the key issues needed to get closure on this
document?

Slightly longer explanation.

We seem to have a classic example of the end-to-end argument at play
here:

      The function in question can completely and correctly be
      implemented only with the knowledge and help of the application
      standing at the end points of the communication
      system. Therefore, providing that questioned function as a
      feature of the communication system itself is not
      possible. (Sometimes an incomplete version of the function
      provided by the communication system may be useful as a
      performance enhancement.)

Keepalives are such an enhancement, but are not sufficient in this
case to correctly implement the application.

Indeed, the only way to detect a failed server is send periodic probe
packets to test the liveness of both the connection *and* the
application at the other end. Note that even TCP keepalives aren't
sufficient (in theory). The TCP stack (in the kernel) could be
working, the server could still be running, but the application itself
may have wedged and be unable to respond. You can only detect that by
sending the equivalent of a "are-you-still-there" to the server, with
the server returning a
"yes-I-am-and-I'm-still-working-on-your-outstanding-request-so-no-need-to-resend". And
even here, there are probably edge cases that can't be detected.

Thomas

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Fri, 19 Dec 2008 11:05:32 -0500

Mark is the author on this draft, so it is ulimately up to him.

But, yes, I think BULK_LQ_CONN_TIMEOUT should be removed.

And, BULK_LQ_DATA_TIMEOUT should be 5 minutes (as the default) -- per
RFC 1123 4.1.3.2. I see little reason to make it any different.

- Bernie

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Thomas Narten
Sent: Friday, December 19, 2008 10:59 AM
To: Bernie Volz (volz)
Cc: Dhcwg; Mark Stapp (mjs); Lars Eggert; IESG
Subject: Re: [dhcwg] draft-ietf-dhc-dhcpv6-bulk-leasequery in IESG
review

"yes-I-am-and-I'm-still-working-on-your-outstanding-request-so-no-need-t
o-resend". And

Thomas

---

From: Mark Stapp <[email-masked]>
Date: Fri, 19 Dec 2008 11:10:40 -0500

Thanks, Bernie for helping to clear up the specifics of the issues:
I'll issue a new revision.

Regards,

Mark

Bernie Volz (volz) wrote:

- Bernie

-----Original Message-----

Thomas