Subject: [nfsv4] [3530bis] Driving another nail into the UDP coffin

From: Thomas Haynes <[email-masked]>
Date: Sun, 25 Sep 2011 16:25:14 -0500

I'm proposing taking the fine text in 5661 back into 3530bis to
state that UDP is dead:

KinMage:3530bis thomas$ git diff
diff --git a/nfsv4_middle_rpc_sec.xml b/nfsv4_middle_rpc_sec.xml
index d20572f..ce620c3 100644
--- a/nfsv4_middle_rpc_sec.xml
+++ b/nfsv4_middle_rpc_sec.xml
@@ -25,13 +25,52 @@
 
     <t>
       Where an NFSv4 implementation supports operation over the IP
-      network protocol, the supported transports between NFS and IP MUST be
-      among the IETF-approved congestion control transport protocols,
-      which include TCP and SCTP.  To enhance the possibilities for
-      interoperability, an NFSv4 implementation MUST support
-      operation over the TCP transport protocol, at least until such time
-      as a standards track RFC revises this requirement to use a different
-      IETF-approved congestion control transport protocol.
+      network protocol, the transport must have the following
+      attributes:
+    </t>
+
+    <t>
+      <list style='symbols'>
+        <t>
+          The transport supports reliable delivery of data, which NFSv4
+          requires but neither NFSv4 nor RPC has facilities for ensuring
+          <xref target="Chet" />.
+        </t>
+
+        <t>
+          The transport delivers data in the order it was sent.  Ordered
+          delivery simplifies detection of transmit errors, and simplifies
+          the sending of arbitrary sized requests and responses via the
+          record marking protocol <xref target="RFC5531" />.
+        </t>
+      </list>
+    </t>
+
+
+    <t>
+      Furthermore, any transport used between NFS and IP MUST be among
+      the IETF-approved congestion control transport protocols.  At the
+      time this document was written, the only two transports that had the
+      above attributes were TCP and the Stream Control Transmission
+      Protocol (SCTP).  To enhance the possibilities for interoperability,
+      an NFSv4 implementation MUST support operation over the TCP
+      transport protocol.
+    </t>
+
+    <t>
+      Even if NFSv4 is used over a non-IP network protocol, it is
+      RECOMMENDED that the transport support congestion control.
+    </t>
+
+    <t>
+      It is permissible for a connectionless transport to be used under
+      NFSv4; however, reliable and in-order delivery of data combined
+      with congestion control by the connectionless transport is REQUIRED.
+      As a consequence, UDP by itself MUST NOT be used as an NFSv4
+      transport.  NFSv4 assumes that a client transport address and
+      server transport address used to send data over a transport together
+      constitute a connection, even if the underlying transport eschews the
+      concept of a connection.
     </t>
 
     <t>

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 27 Sep 2011 06:39:01 +0000

Are there any objections to Tom's proposal?  Please speak up if you don't
believe this is a reasonable change to the 3530bis document?

Spencer

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Thomas Haynes
> Sent: Sunday, September 25, 2011 2:25 PM
> To: [email-masked]
> Subject: [nfsv4] [3530bis] Driving another nail into the UDP coffin
> 
> I'm proposing taking the fine text in 5661 back into 3530bis to state that
> UDP is dead:
> 
> diff --git a/nfsv4_middle_rpc_sec.xml b/nfsv4_middle_rpc_sec.xml index
> d20572f..ce620c3 100644
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Nico Williams <[email-masked]>
Date: Tue, 27 Sep 2011 14:39:41 -0500

> +      concept of a connection.

I have a hard time imagining a connection-less Internet transport that
provides reliable, in-order delivery of data -- enough state is
required to detect missed messages that it amounts to a "connection".

I can imagine a connection-less transport that provides congestion
control, but I'm not sure that congestion control is necessary in all
networks that one can imagine NFSv4 deployed in.  Perhaps that's just
me thinking of a priori bandwidth allocation as not congestion control
(which is probably wrong).

I think what you should say is that datagram-oriented transports are
OK, provided that they provide reliable and ordered datagram delivery.
 That's enough to rule out UDP without saying anything about whether
the transport provides connections.

Nico
--

---

From: <[email-masked]>
Date: Tue, 27 Sep 2011 15:43:22 -0400

Nico,

Ruling out UDP explicitly is an improvement in clarity over the current language that
does so implicitly.  Had 3530 originally done this, we would have avoided the entire
discussion over whether NFVv4.0 is allowed over UDP ;-).

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Nico Williams
> Sent: Tuesday, September 27, 2011 3:40 PM
> To: Thomas Haynes
> Cc: [email-masked]
> Subject: Re: [nfsv4] [3530bis] Driving another nail into the UDP coffin
>

---

From: Nico Williams <[email-masked]>
Date: Tue, 27 Sep 2011 14:52:20 -0500

> discussion over whether NFVv4.0 is allowed over UDP ;-).

Go for it.  I was picking a nit though, with the logic set out for
ruling out UDP.  Yes, totally a nit, I know.

---

From: Chuck Lever <[email-masked]>
Date: Tue, 27 Sep 2011 16:01:03 -0400

On Sep 25, 2011, at 5:25 PM, Thomas Haynes wrote:

> +      NFSv4;

Aren't there some small number of error cases that require the server to drop its connection to the client, and vice versa?  How would an equivalent connection drop be communicated via a connectionless transport?

> however, reliable and in-order delivery of data combined

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 27 Sep 2011 19:13:31 -0400

> the transport provides connections.

Tom and Nico,

I'm fine with 'reliable' and 'congestion control', since that's already
in RFC3530, but I'm not sure I understand the need for 'ordered'. That
looks like a new requirement, and as such demands an explanation.

Cheers
  Trond
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 27 Sep 2011 19:23:01 -0400

> looks like a new requirement, and as such demands an explanation.

The reason why I ask is because IIRC, SCTP allows for multi-pathing and
therefore does not guarantee ordered delivery of messages.
It does however guarantee reliable delivery and congestion control.

Cheers
  Trond

NetApp

---

From: Nico Williams <[email-masked]>
Date: Tue, 27 Sep 2011 18:33:15 -0500

> It does however guarantee reliable delivery and congestion control.

Also, I think NFSv4 doesn't need reliable either -- isn't it the case
that it can retransmit on its own?  Because, after all, NFSv4 has its
own transport, and RPCs can get dropped even if delivered.

In any case, we want to stop using UDP.  Congestion control is
probably the foremost reason.  It might also be the only reason.  I
don't think more than that is necessary.

As for SCTP, it is sure is tempting.

Nico
--

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 27 Sep 2011 18:49:10 -0500

> It does however guarantee reliable delivery and congestion control.

Straight cut and paste from 4.1, which probably has more requirements
due to sessions?

Anyway, drop-able if that is what the consensus is.

---

From: "Myklebust, Trond" <[email-masked]>
Date: Tue, 27 Sep 2011 17:15:47 -0700

> From: Nico Williams [mailto:[email-masked]]
> Sent: Tuesday, September 27, 2011 7:33 PM
> To: Myklebust, Trond
> Cc: Haynes, Tom; [email-masked]
> >> I'm fine with 'reliable' and 'congestion control', since that's
> >> already in RFC3530, but I'm not sure I understand the need for
> >> 'ordered'. That looks like a new requirement, and as such demands an
> explanation.
> >
> > The reason why I ask is because IIRC, SCTP allows for multi-pathing
> > and therefore does not guarantee ordered delivery of messages.
> Also, I think NFSv4 doesn't need reliable either -- isn't it the case that it can
> retransmit on its own?  Because, after all, NFSv4 has its own transport, and
> RPCs can get dropped even if delivered.

Right, but the 'reliable' bit is what allows the client and server to assume that the RPC call doesn't need to be retransmitted unless the connection breaks (see section 3.1.1 in rfc3530).

 Cheers
  Trond

---

From: "Myklebust, Trond" <[email-masked]>
Date: Tue, 27 Sep 2011 17:24:08 -0700

> From: Haynes, Tom
> Sent: Tuesday, September 27, 2011 7:49 PM
> To: Myklebust, Trond
> Cc: Nico Williams; [email-masked]
due
> to sessions?

I don't think that sessions add any requirements to the transport. In
fact they were originally designed in order to enable the use of a
transport that doesn't have congestion control (i.e. RDMA).

> Anyway, drop-able if that is what the consensus is.

Please do. AFAICS, UDP is excluded due to the existing requirements of
reliable+congestion control and doesn't require any further restrictions
on the allowed types of transport. It is useful to clarify why it is not
allowed, though, so we don't have to get into these little discussions
every few years.

Cheers
  Trond

---

From: <[email-masked]>
Date: Tue, 27 Sep 2011 20:46:57 -0400

> Straight cut and paste from 4.1, which probably has more requirements

If it is a valid requirement, then it is probably because of sessions.

If it isn't because of sessions, what would make v4.1 have additional 
Transport requirements?

On the face of it, you'd think that there would be fewer transport
requirements with 4.1, given that sessions gives you some facilities 
that you might rely on the transport for, in the absence of sessions.

> Anyway, drop-able if that is what the consensus is.

I think this transport requirement should be added to v4.0 only if there
is a consensus that it is required.  I don't see one, but perhaps we should
ask Spencer if he does.

Is there is anyone who understands why this transport requirement is in 5661?
I don't remember it.  Spencer?  Mike?

---

From: Tom Talpey <[email-masked]>
Date: Wed, 28 Sep 2011 07:54:35 -0400

On 9/27/2011 4:01 PM, Chuck Lever wrote:
>>...
>> +<t>

Strictly speaking, the proper requirement is that the server drop the
*request*, which on a reliable transport implies that the server must
also drop the *connection* to signal data loss.

On an unreliable, or more accurately perhaps stateless transport, the
server would simply drop the request and continue processing. This would
require the server to implement significant congestion, denial of
service and replay protection, which are currently provided by TCP in
the NFSv4.x stack.

Allowing UDP would, IMO, require a significant respecification effort
to re-achieve the reliability and robustness currently present in
NFSv4.x/TCP. Let's not go there.

It's my understanding that the Linux NFSv4 client and server permitted
use of UDP as a transport. Is that still the case?

Tom
ttalpey(at)microsoft(dot)com

---

From: Tom Talpey <[email-masked]>
Date: Wed, 28 Sep 2011 08:06:07 -0400

I think it does have an explanation in RFC5661:

> 2.9.  Transport Layers
>
> 2.9.1.  REQUIRED and RECOMMENDED Properties of Transports
>
>    NFSv4.1 works over Remote Direct Memory Access (RDMA) and non-RDMA-
>    based transports with the following attributes:
>
>
>
>
>
>
> Shepler, et al.              Standards Track                   [Page 37]
>
> RFC 5661                         NFSv4.1                    January 2010
>
>
>    o  The transport supports reliable delivery of data, which NFSv4.1
>       requires but neither NFSv4.1 nor RPC has facilities for ensuring
>       [34].
>
>    o  The transport delivers data in the order it was sent.  Ordered
>       delivery simplifies detection of transmit errors, and simplifies
>       the sending of arbitrary sized requests and responses via the
>       record marking protocol [3].
>

If the transport is not ordered, then it must be atomic - i.e. all
requests and responses must be delivered intact without being fragmented
across network messages. This is effectively an RPC requirement, and
removing it in turn would impose severe restrictions on the NFS RPC
sizes, including r/w data size and even compounding.

There are no such provisions in the protocol today, so relaxing the
ordering would ripple right back up the stack and require significant
upper layer rework/rewording.

>>> and therefore does not guarantee ordered delivery of messages.

Correct, but SCTP does guarantee ordering within each stream, and
therefore satisfies the requirement.

> I don't think that sessions add any requirements to the transport.

Correct.

>In

Incorrect. The IETF standard iWARP spec is layered on TCP, in part for
this exact reason - to obtain congestion protection.

In addition, it's worth remembering that in NFSv4.1, the session
enforces a slot count, which limits the number of messages that
can be in flight and therefore affords an additional level of congestion
protection.

Tom.

---

From: Chuck Lever <[email-masked]>
Date: Wed, 28 Sep 2011 11:07:48 -0400

> also drop the *connection* to signal data loss.

The language I'm concerned with is Section 3.1.1 of 3530bis15 (see below).

If we say that "(i)t is permissible for a connectionless transport to be used under NFSv4," wouldn't we need some clarification of S3.1.1?  At the very least S3.1.1 implies that a transport cannot be reliable unless it is connection oriented, which contradicts the language Tom H. is proposing.

If language is introduced that explicitly discusses the possible use of reliable connectionless transports, I would prefer to see S3.1.1 expanded to discuss required retransmission behavior for such transports.  Or it should simply state clearly that the stated retransmission requirements do not apply to reliable connectionless transports.

If I read Tom T.'s e-mail correctly, our relevant implementation experience for reliable connectionless transports appears to be RDMA.  Perhaps when referencing an example transport in the new language, RDMA can be explicitly cited.

> 3.1.1.  Client Retransmission Behavior
> 
>    When processing a request received over a reliable transport such as
>    TCP, the NFSv4 server MUST NOT silently drop the request, except if
>    the transport connection has been broken.  Given such a contract
>    between NFSv4 clients and servers, clients MUST NOT retry a request
>    unless one or both of the following are true:
> 
>    o  The transport connection has been broken
> 
>    o  The procedure being retried is the NULL procedure
> 
>    Since reliable transports, such as TCP, do not always synchronously
>    inform a peer when the other peer has broken the connection (for
>    example, when an NFS server reboots), the NFSv4 client may want to
>    actively "probe" the connection to see if has been broken.  Use of
>    the NULL procedure is one recommended way to do so.  So, when a
>    client experiences a remote procedure call timeout (of some arbitrary
>    implementation specific amount), rather than retrying the remote
>    procedure call, it could instead issue a NULL procedure call to the
>    server.  If the server has died, the transport connection break will
>    eventually be indicated to the NFSv4 client.  The client can then
>    reconnect, and then retry the original request.  If the NULL
>    procedure call gets a response, the connection has not broken.  The
>    client can decide to wait longer for the original request's response,
>    or it can break the transport connection and reconnect before re-
>    sending the original request.

---

From: Tom Talpey <[email-masked]>
Date: Wed, 28 Sep 2011 11:49:46 -0400

> If we say that "(i)t is permissible for a connectionless transport to be used under NFSv4," wouldn't we need some clarification of S3.1.1?  At the very least S3.1.1 implies that a transport cannot…

I agree that the text in 3.1.1 conflates the notions of "reliable" and
"connection oriented". They are not the same, although in the cited TCP
example this is true.

> If language is introduced that explicitly discusses the possible use of reliable connectionless transports, I would prefer to see S3.1.1 expanded to discuss required retransmission behavior for suc…

It my be risky to make a normative statement of NFS behavior atop a
reliable connectionless transport. It might be better to state that
any such specification in the future must make the statement. After
all, a port number would need to be assigned, which would require an
IANA action, which would require an RFC specification.

> If I read Tom T.'s e-mail correctly, our relevant implementation experience for reliable connectionless transports appears to be RDMA.  Perhaps when referencing an example transport in the new lang…

The only IETF-defined RDMA transport is connection oriented. There do
exist non-CO reliable RDMA transports, but they are out of scope here,
have some distinctly different semantics, and no NFSv4.x protocol
layering is defined for them (see RFC5666 and RFC5667, for example).
We should not attempt to do so at this time, IMO.

Tom.

---

From: <[email-masked]>
Date: Wed, 28 Sep 2011 13:24:32 -0400

Tanks for pointing us to his explanatory text, but it does raise some questions.

>     requires but neither NFSv4.1 nor RPC has facilities for ensuring

I think everyone is fine with pulling this into RFC3530, turning the "NFSv4.1"
reference into "NFSv4.0".  This doesn't change the transport requirements already
in RFC3530.

With that change, the statement will be true, while in RFC5661, it isn't true.
I realize the spec ship has sailed here but just to maintain our clarity of 
understanding...  In RFC5661, there, is  big section on EOS and it is not clear
how or why it wouldn't work with UDP.  UDP is forbidden and I don't want it back.
But if we want a requirement to justify this (assuming, "nobody wants to use that POS"
doesn't fly), congestion control meets the need.

>       record marking protocol [3].

So it sounds from that the new (v4.1?) requirement  is for sending 
arbitrary-sized request and responses.  If so, does it apply to v4.0?
I think we would have to be really sure that it did to pull this into RFC3530.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Tom Talpey
Sent: Wednesday, September 28, 2011 8:06 AM
To: [email-masked]
Subject: Re: [nfsv4] [3530bis] Driving another nail into the UDP coffin

Correct.

Tom.

---

From: Chuck Lever <[email-masked]>
Date: Wed, 28 Sep 2011 13:25:53 -0400

On Sep 28, 2011, at 11:49 AM, Tom Talpey wrote:

> It may be risky to make a normative statement of NFS behavior atop a

Yep, that's what I'm getting at.  If our community doesn't have implementation experience with NFSv4 over reliable connectionless transports, I'm hesitant to provide a lot of verbiage about them in 3530bis15, whether or not unadorned UDP is explicitly mentioned as a MUST NOT.

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 28 Sep 2011 13:46:02 -0400

> >       record marking protocol [3].

Meh... As long as I can reliably receive complete RPC messages, why do I
care how the transport implements this at the protocol level? Why
shouldn't I be able to use something that guarantees fast and reliable
delivery of the messages but doesn't guarantee that ordering between
messages? 

> upper layer rework/rewording.

As you said, those are RPC requirements, and not a NFS requirements.

> therefore satisfies the requirement.

The RFC5661 wording that you quoted makes no mention of streams. It says
'data must be delivered in the order in which it was sent' which is not
quite the same thing as saying that ordering by stream is OK: if I send
2 messages, and the SCTP protocol ships them over 2 different streams,
they may get reordered at delivery.

> this exact reason - to obtain congestion protection.

Yes, but the original design in NetApp's DAFS was for supporting
infiniband-based RDMA.

> can be in flight and therefore affords an additional level of congestion

Agreed, and it also provides stream-like ordering guarantees. This is
another reason why I'm confused by all these extra requirements on the
transport layer.

Cheers
   Trond

NetApp

---

From: Tom Talpey <[email-masked]>
Date: Wed, 28 Sep 2011 13:58:25 -0400

> doesn't fly), congestion control meets the need.

Well, I think it would be better to confront the issue head-on instead
of tripping UDP on a technicality (no congestion).

The main reasons UDP doesn't work for NFSv4 are:

1) The 64K-epsilon length restriction. This impacts the ability to
    marshal upper layer data successfully, and potentially prevents
    NFSv4 COMPOUNDs from working at all, to say nothing of inhibiting
    the proven use of large reads and writes.

2) The lack of protection against misordered and/or duplicate delivery,
    which if not provided by the transport make the reply cache
    mandatory for correctness. This is not an issue for NFSv4.1, because
    of the session. But it is an issue for NFSv4.0, which provides no
    protocol behavior for it, and relies on what is fundamentally
    unspecified RPC behavior.

The lack of congestion control could be implemented in another layer,
for instance moving to DCCP, or implementing congestion in RPC. These
would not satisfy the above issues.

> I think we would have to be really sure that it did to pull this into RFC3530.

I believe that it does. v4.0 has the same basic COMPOUND and read/write
operations, and the issues apply equally. Or as I hinted, perhaps even
more, because v4.0 provides no upper layer facilities of its own to
recover from the transport errors.

Tom.

---

From: Tom Talpey <[email-masked]>
Date: Wed, 28 Sep 2011 14:25:34 -0400

> delivery of the messages but doesn't guarantee that ordering between

Well, I was just quoting text from RFC5661. Are you arguing that it
should not be there, and that one or both of 5661 or 3530bis should
support UDP?

> As you said, those are RPC requirements, and not a NFS requirements.

Correct. But NFSv3 and NFSv4.0 depend on RPC semantics inherently for
correctness. NFSv3 uses it additionally for endpoint mapping. It's not
possible to write RPC out of the stack.

With NFSv4.1, it approaches the realm of possibility. RPC is used mainly
for network representation, apart from endpoint sharing (protocol
number and version, plus identification of request and response). So
it is not *solely* an RPC issue. There is upper layer involvement.

> they may get reordered at delivery.

Fair. That might be a deficiency in the existing NFS transport
requirement.

> Yes, but the original design in NetApp's DAFS was for supporting

DAFS who? :-)

My point was that the NFS and NFS/RDMA specs are only normative to IETF
specs. While they can and do "work" on other RDMA transports, the
IETF standard iWARP lower layer implements full congestion control, by
design. The congestion behavior of Infiniband is out of scope.

Tom.

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 28 Sep 2011 16:52:49 -0400

> should not be there, and that one or both of 5661 or 3530bis should

I'm not arguing for UDP support. I'm merely arguing that we should
beware of adding in transport requirements that might come back to haunt
us later.
I agree that we should ban transports that don't meet basic requirements
for reliable high-speed bi-directional RPC messaging between clients and
servers.
I agree that UDP doesn't meet the existing requirements that are spelled
out in RFC3530.

I have yet to be fully convinced that adding ordering to the existing
requirements is a good idea. All NFS clients and servers that I'm aware
of try to parallelise RPC calls as much as possible in order to improve
throughput. Restricting the ability of the transport layer to do the
same thing, and demanding ordered message delivery would seem to need
more justification.

> it is not *solely* an RPC issue. There is upper layer involvement.

Yes, but the job of the NFS RFC should not be to over-proscribe lower
layer behaviour. It should aim to set stack requirements at the minimal
level that allows for reliable operation.

> Fair. That might be a deficiency in the existing NFS transport

My point is that it's a deficiency in the RFC5661 requirements, but not
RFC3530. Unless we can justify the data ordering requirement, my
preference would rather be to loosen the requirements in RFC5661 instead
of tightening those in RFC3530.

> design. The congestion behavior of Infiniband is out of scope.

Sure, but my point was that in this case the design demonstrably applies
beyond the minimal scope of RDMA transports that are currently
sanctioned by the IETF, irrespective of what our actual interests may
be. We have proven that we don't need all the features those transports
provide, and don't have to enumerate those features as actual
requirements.

Who knows, maybe the IETF will invent new transport protocols in the
future; maybe even ones that prove to be useful? :-)

Cheers
  Trond

NetApp

---

From: Tom Talpey <[email-masked]>
Date: Wed, 28 Sep 2011 20:05:32 -0400

> beware of adding in transport requirements that might come back to haunt

Well, here's an experiment. Here are some semi-relevant transports
and their properties in tabular form. "IB RC" is Infiniband reliable
connected mode, which is the mode that NFS/RDMA, among others, is
able to use.

It's quite easy to see the odd-protocol-out. But it does indicate
that the presence of ordering in the transports is universal.
As previously stated, to RPC, NFSv3 and NFSv4.0 it's a very important
property to have, because of the crude support in these upper layers
for dealing without it. I grant that it is less important for NFSv4.1,
because of the slot/sequence support afforded by the session.

Transport
    Protocol ->	TCP	UDP	SCTP	RDDP	IB RC(out scope)
Property
    v
Connected	 y	 n	 y	 y	 y
Reliable	 y	 n	 y	 y	 y
Ordered		 y	 n	opt	 y	 y
At-most-once	 y	 n	 y	 y	 y
Datagram	 n	 y	 y	 y	 y
Bytestream	 y	 n	opt	 n	 n
Congestion	 y	 n	 y	 y	 n
Large xfer	 y	 n	 y	rdma    rdma

So, I now feel that a requirement for ordering should be added to
3530bis, but agree the text be removed/relaxed for post-4.1 work,
to enable future innovation.

As for 4.1 itself, we can perhaps decide to update RFC5661 separately.

Thoughts?

Tom.

---

From: Nico Williams <[email-masked]>
Date: Wed, 28 Sep 2011 19:43:09 -0500

> Thoughts?

Mainly this: Lustre, I believe, gets this right.  Lustre expects at
most congestion control and integrity protection (i.e., error
detection/correction) out of a transport.  This is useful because a
dropped TCP connection, for example, says nothing about what RPCs were
also dropped.  But that's a nother story.  (Oh, and Lustre uses TCP to
run over IP.)

Nico
--

---

From: Tom Talpey <[email-masked]>
Date: Wed, 28 Sep 2011 21:46:05 -0400

On 9/28/2011 8:43 PM, Nico Williams wrote:
> On Wed, Sep 28, 2011 at 7:05 PM, Tom Talpey<[email-masked]>  wrote:
>> Thoughts?
>

Ok, sure we'd like to require no more than tin cans and string, but
what technical reasons does the NFSv4.0 protocol (this thread is about
3530bis after all) offer for operation over a minimal transport? To do
so requires it to protect itself, which, unfortunately, it doesn't. Are
we going to retroactively require a DRC? Even then it won't solve
the problem.

Besides, emulate Lustre? Really? :-)

Tom.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 29 Sep 2011 00:02:18 -0400

> Large xfer	 y	 n	 y	rdma    rdma

...and 6/10 households prefer Persil detergent over competing brands...
I wasn't asking for a poll: I really do want a justification.

Cheers
   Trond

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 29 Sep 2011 00:25:54 -0400

> I wasn't asking for a poll: I really do want a justification.

What I'm saying (flippantly, perhaps - I apologize for that) is that the
above shows that our current rfc3530 requirements clearly do what we
intended: they eliminate UDP as a candidate. However I don't see how
they justify the stronger requirements in rfc5661.

Cheers
  Trond

NetApp

---

From: Thomas Haynes <[email-masked]>
Date: Thu, 29 Sep 2011 00:09:37 -0500

> they justify the stronger requirements in rfc5661.

So I also believed RFC 3530 forbade UDP. 

The authors of RFC 5661 thought that wording was not strong enough.

In reviewing the FEDFS documents, David Black believes it is not strong
enough.

Trond is right in that we only want to introduce the minimum set of changes
necessary to convey UDP is forbidden. We don't want to introduce anything
else that can be misconstrued.

Here is what we currently have:

   Where an NFSv4 implementation supports operation over the IP network
   protocol, the supported transports between NFS and IP MUST be among
   the IETF-approved congestion control transport protocols, which
   include TCP and SCTP.  To enhance the possibilities for
   interoperability, an NFSv4 implementation MUST support operation over
   the TCP transport protocol, at least until such time as a standards
   track RFC revises this requirement to use a different IETF-approved
   congestion control transport protocol.

   If TCP is used as the transport, the client and server SHOULD use
   persistent connections.  This will prevent the weakening of TCP's
   congestion control via short lived connections and will improve
   performance for the WAN environment by eliminating the need for SYN
   handshakes.

We could state this as:

   Where an NFSv4 implementation supports operation over the IP network
   protocol, the supported transports between NFS and IP MUST be among
   the IETF-approved congestion control transport protocols, which
   include TCP and SCTP.  To enhance the possibilities for
   interoperability, an NFSv4 implementation MUST support operation over
   the TCP transport protocol, at least until such time as a standards
   track RFC revises this requirement to use a different IETF-approved
   congestion control transport protocol. These requirements dictate 
  UDP by itself MUST NOT be used as an NFSv4 transport

   If TCP is used as the transport, the client and server SHOULD use
   persistent connections.  This will prevent the weakening of TCP's
   congestion control via short lived connections and will improve
   performance for the WAN environment by eliminating the need for SYN
   handshakes.

It states a little bit more clearly no UDP and doesn't introduce the ordering
argument.

---

From: <[email-masked]>
Date: Thu, 29 Sep 2011 08:48:36 -0400

My concern is clarity, which is about "strength of language" not "strength
of requirements".  A single sentence of the form:

      Because UDP does not meet the above requirements, UDP MUST NOT be used as
      an NFSv4 transport protocol.

would get the point across clearly without introducing new requirements.  Part
of the clarity problem is that for an RFC that forbids use of UDP, there's a
surprising amount of UDP support defined elsewhere in RFC 3530.

The FedFS DNS draft will need to state that the underlying 4.0 and 4.1 RFCs
prohibit use of UDP as part of the explanation for why it contains no support
for UDP.

Thanks,
--David

From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Haynes
Sent: Thursday, September 29, 2011 1:10 AM
To: Trond Myklebust
Cc: Tom Talpey; [email-masked]
Subject: Re: [nfsv4] [3530bis] Driving another nail into the UDP coffin

We could state this as:

---

From: Tom Talpey <[email-masked]>
Date: Thu, 29 Sep 2011 11:12:19 -0400

On 9/29/2011 8:48 AM, [email-masked] wrote:
> My concern is clarity, which is about “strength of language” not “strength
> of requirements”. A single sentence of the form:
>
> of the clarity problem is that for an RFC that forbids use of UDP, there’s a

So, the nuclear approach would be to remove any and all mention of UDP
from 3530bis, and be done with it? I'm ok with that. The MUST NOT
sentence becomes optional at that point btw, but it's good for nailing
the coffin closed.

Tom.

 > prohibit use of UDP as part of the explanation for why it contains no
 > support for UDP.
>
> Thanks,
> --David
>
> *From:[email-masked] [mailto:[email-masked]] *On Behalf
> Of *Thomas Haynes
> *Sent:* Thursday, September 29, 2011 1:10 AM
> *To:* Trond Myklebust
> *Cc:* Tom Talpey; [email-masked]
> *Subject:* Re: [nfsv4] [3530bis] Driving another nail into the UDP coffin
>

---

From: Thomas Haynes <[email-masked]>
Date: Thu, 29 Sep 2011 10:30:15 -0500

>> an NFSv4 transport protocol.

5661 is pretty clear that you can use UDP if and only if you pair it with something else
that will meet the requirements.

I.e., the sentence above would have 3530bis be stronger than 5661.

> sentence becomes optional at that point btw, but it's good for nailing

UDP is only used to describe Network Identifiers:

KinMage:3530bis thomas$ grep UDP draft-ietf-nfsv4-rfc3530bis-14.txt 
   For TCP over IPv4 and for UDP over IPv4, the format of r_addr is the
   For TCP over IPv4 the value of r_netid is the string "tcp".  For UDP
   For TCP over IPv6 and for UDP over IPv6, the format of r_addr is the
   as with universal addresses for TCP and UDP over IPv4.  The prefix,
   For TCP over IPv6 the value of r_netid is the string "tcp6".  For UDP
   "udp":  UDP over IP version 4
   "udp6":  UDP over IP version 6

Does it need to be there? No.

Is it a strong case that UDP is allowed in 3530bis? No.

I'd prefer to keep these mentions of UDP, they describe an IANA registry.

---

From: <[email-masked]>
Date: Thu, 29 Sep 2011 11:30:47 -0400

Having missed it the first time, I favor "nailing the coffin closed." ;-)

Thanks,
--David

> Sent: Thursday, September 29, 2011 11:12 AM
> To: [email-masked]

---

From: <[email-masked]>
Date: Thu, 29 Sep 2011 11:33:35 -0400

> I.e., the sentence above would have 3530bis be stronger than 5661.

In that case, the FedFS DNS draft would have to explain why such pairings of
UDP with something else aren't supported.

Thanks,
--David

> Sent: Thursday, September 29, 2011 11:30 AM
> To: Tom Talpey
> Cc: [email-masked]

---

From: Tom Talpey <[email-masked]>
Date: Thu, 29 Sep 2011 11:51:54 -0400

> UDP with something else aren't supported.

Tom H, I think you're referring to the only sentence in RFC5661 that
mentions UDP:

-- As a consequence, UDP by itself MUST NOT be used as an NFSv4.1
-- transport.

I don't think it's inconsistent with a statement that UDP is not
allowed, it still disallows UDP, right? The words "by itself" are
certainly more than a little strange though.

So, I still say that deleting all mention of UDP from 3530bis is a
solution.

Tom.

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 29 Sep 2011 17:34:59 +0000

> Tom Talpey
> Sent: Thursday, September 29, 2011 8:52 AM
> To: [email-masked]
> >>>> Because UDP does not meet the above requirements, UDP MUST NOT be
> >>>> used as an NFSv4 transport protocol.
> >>
> >>
> >> 5661 is pretty clear that you can use UDP if and only if you pair it
> >> with something else that will meet the requirements.
> >>
> > In that case, the FedFS DNS draft would have to explain why such
> > pairings of UDP with something else aren't supported.
> 
> I don't think it's inconsistent with a statement that UDP is not allowed,
> it still disallows UDP, right? The words "by itself" are certainly more
> than a little strange though.
> 

This is my goal.  When NFSv4 minor version share functionality or have
common dependencies (as in this case with transport usage), the statements
in the RFCs should be congruent.  Obviously, there can be exceptions if
a minor version has varying functionality (slot table, etc. in the case
of 4.1).

So, the two things I would like to be congruent between 4.0 and 4.1 is
that UDP MUST NOT be used.  The exception would be that if UDP were
combined with an IETF "approved" congestion control mechanism.

This was the requirement and intent of the original 4.0 language and
my personal desire is to make the requirement very clear.  Removal
of mention of the UDP statements are inadequate in my opinion.
I want to avoid the case where an implementer reads the RFC and 
does not interpret the subtlety correctly.  Let's make it painfully clear.

The other thing I would like to see mirrored in the 4.0bis is the
requirement that if the NFSv4 client needs to retransmit a request
because of lack of response, the client MUST NOT use the same TCP
connection that was previously used.  The client should rely on 
the transport/TCP to deliver the data as it was intended.

BTW: This issue does need to be resolved but I am not willing to hold
the 3530bis for full resolution.  If we can not come to closure by
the time I am ready to shepherd this to our AD, then we will go without 
it and deal with it as an errata or fold it in during the IETF review
process.

Spencer

---

From: <[email-masked]>
Date: Fri, 30 Sep 2011 10:43:46 -0400

> certainly more than a little strange though.

It certainly interferes with clarity, and I think David is right that 
clarity here is of the greatest importance.

My interpretation of the "by itself" was that you could essentially build
your own transport layered on top of UDP, and if that higher-level transport 
met the requirements, you could use it, if you could convince people that there
was a good reason to do so.

Is that other people's interpretation?

If that is the correct interpretation, I think David's proposed
MUST NOT language is still OK.  If someone were to do this, he would still 
not be using UDP as the transport.

As long as we didn't make this possibility explicit, I take it the FedFS 
document would be OK without dealing with this theoretical possibility.

So the idea would be that "by itself" goes in the coffin as well.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Tom Talpey
Sent: Thursday, September 29, 2011 11:52 AM
To: [email-masked]
Subject: Re: [nfsv4] [3530bis] Driving another nail into the UDP coffin

Tom.

---

From: Daniel Ellard <[email-masked]>
Date: Fri, 30 Sep 2011 10:56:32 -0400

I'm worried that the original problem that spawned this thread might
get lost in the "should we support UDP?" discussion (which is also
important).

If there's more than one transport permitted, then the dnssrv record
either must be able to specify which transport to use, or else there
must be some other way for the client to detect the proper protocol.
(I think that the former is preferable.)

Alternatively, we could just say "we only support TCP, so the
transport protocol can be implicit", but I can imagine valid
objections to assuming that nobody will ever want to use anything
else.

-Dan

---

From: Nico Williams <[email-masked]>
Date: Fri, 30 Sep 2011 10:15:20 -0500

> objections to assuming that nobody will ever want to use anything

+1.

---

From: Tom Talpey <[email-masked]>
Date: Fri, 30 Sep 2011 11:18:41 -0400

> (I think that the former is preferable.)

I think you should start a new thread for that discussion. :-)

But I can say that it seems odd to not specify SCTP and RDMA forms
of the record, since those protocols are arguably in scope, and have
predefined netid's.

Tom.

> On Fri, Sep 30, 2011 at 10:43 AM,<[email-masked]>  wrote:

---

From: Tom Talpey <[email-masked]>
Date: Fri, 30 Sep 2011 11:23:10 -0400

> So the idea would be that "by itself" goes in the coffin as well.

That's certainly my opinion. It says nothing, after all; it merely makes
a hypothetical statement of some unspecified layering. If there were
such a layering, we would not call it UDP.

So to summarize, we should:
1) Delete all existing mention of UDP from 3530bis
2) Add a sentence to 3530bis equivalent to the statement in 5661,
    without the words "by itself"
3) Queue a change for 5661 to delete "by itself" at some future time.

Tom.

---

From: <[email-masked]>
Date: Fri, 30 Sep 2011 11:30:51 -0400

> It says nothing, after all; 

So it says nothing and adds a lot of confusion.

Other than that, what have you got against it?

> it merely makes

I guess it would be called VUDP.

That's what comes from nailing the coffin shut without driving a stake
through its heart.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Tom Talpey
Sent: Friday, September 30, 2011 11:23 AM
To: [email-masked]
Subject: Re: [nfsv4] [3530bis] Driving another nail into the UDP coffin

Tom.

---

From: "Haynes, Tom" <[email-masked]>
Date: Fri, 30 Sep 2011 11:53:50 -0400

Then bring it up in a different thread. I.e., this one does not hijack
the
original problem.

There is an existing precedent you can follow from 3530:

  To enhance the possibilities for

So simply state that TCP is the only supported transport protocol until
such
time as a standards track RFC revises that requirement.

-----Original Message-----
From: Daniel Ellard [mailto:[email-masked]] 
Sent: Friday, September 30, 2011 9:57 AM
To: [email-masked]
Subject: Re: [nfsv4] [3530bis] Driving another nail into the UDP coffin

-Dan

---

From: "Mike Eisler" <[email-masked]>
Date: Fri, 30 Sep 2011 14:04:41 -0700

"by itself" refers to Transport Area Advisor advice that UDP was
acceptable as long as we would include a congestion control feature. No
one in WG stepped up to design one, perhaps because previous attempts at
this have been a failure and/or perhaps because TCP and SCTP do a good job
of this already.

>>> that will meet the requirements.

Actually iff:

- you pair it with something else that will meet the requirements.
- IESG approves

In an earlier message, Tom Talpey notes that NFSv4 is specified to work
over SCTP. This is true (I think I might have said otherwise in a previous
note, and if so that was in error). RFC 5665 states
"    The Stream Control Transmission Protocol (SCTP) (described in RFC
   4960 [7]) is a connection-oriented protocol that supports both byte-
   streamed and record-oriented data transfer.  When the "sctp" and
   "sctp6" netids are used, the Open Network Computing (ONC) RPC Record
   Marking standard (see Section 11 of RFC 5531 [4]) is not used;
   instead, SCTP's native record-oriented data transfer is used. "

IOW, use RPC/SCTP but don't prefix requests and responses with the 32 bit
record marking header; you MUST use SCTP's native record marking.

So if fedfs is going to mention tcp, it needs to mention sctp.

On Thu, September 29, 2011 8:51 am, Tom Talpey wrote:
>> pairings of
>>>>> would get the point across clearly without introducing new
>>>>> requirements.
>>>>> Part

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/