Subject: Re: rfc2292bis text proposal

From: Erik Nordmark <[email-masked]>
Date: Tue, 16 Apr 2002 04:11:47 +0200 (CEST)

>      x == -1: use kernel default.  That is, perform path MTU discovery
>               for unicast, and disable it for multicast.
>      x ==  0: always perform path MTU discovery.
>      x >=  1: always disable path MTU discovery and send packets at
>               the minimum MTU.
>      (values less than -1 are invalid.)

I'd restate the text for -1 to say something like:
	Perform path MTU discovery for unicast destinations but do not
	perform it for multicast destinations. Packets to multicast
	destinations are therefor sent with the minimum MTU.
and separately state that
	The default value of this option is -1.

Why the x >=1 as opposed to x == 1? To be compatible with existing
applications?
The reason I'm asking is because it looks a bit odd to disallow less than -1
when greater than 1 is allowed.

  Erik

--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: JINMEI Tatuya / 神明達哉<[email-masked]>
Date: Tue, 16 Apr 2002 23:06:20 +0900

>>>>> On Tue, 16 Apr 2002 04:11:47 +0200 (CEST), 
>>>>> Erik Nordmark <[email-masked]> said:

> 	The default value of this option is -1.

Okay.

> when greater than 1 is allowed.

I was afraid that there may be an existing implementation which tries
to set a positive but !=1 value to set the IPV6_USE_MIN_MTU option.
But, this is probably an imaginary fear...actually all implementations
that I know of use the value 1 for this purpose.  So, I basically
agree that it is more natural to restrict the positive value (to 1) as
well.

If there's no other opinion on this point, I'll restrict the positive
value in the revised text.

					JINMEI, Tatuya
					Communication Platform Lab.
					Corporate R&D Center, Toshiba Corp.
					[email-masked]
--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: Robert Elz <[email-masked]>
Date: Tue, 16 Apr 2002 21:51:56 +0700

Date:        Tue, 16 Apr 2002 23:06:20 +0900
    From:        JINMEI Tatuya <[email-masked]>
    Message-ID:  <[email-masked]>

  | If there's no other opinion on this point, I'll restrict the positive
  | value in the revised text.

This is probably not important - but is there any real need to allow
the value -1 to actually be set?

That is, I fully understand why it is the default value, but if an
application decides it wants to use the minimium MTU (for unicast) or
that it doesn't (for multicast), is it really likely that it will
want to ever say (on one socket) "use it for multicast but not for
unicast" ?

Having that as the default makes sense, as there's no way to know in
the stack whether unicast or multicast packets will be sent, until
after they're sent.   But the application sure knows...   Do applications
really send both unicast and multicast packets out one socket?

I guess my point is that all that might need to be altered is the
default setting, no need to add the -1 value as a possible arg to the
syscall (if an app really needs to return to the default values, it
could just close the socket, open a new one, and bind it to the same
address).

If we do need the new value, then are we 100% sure that we're not also
going to need "use min mtu for unicast, but not for multicast" (the
4th possible case).   I know that one seems crazy, but are we so certain
that it will never be needed that it shouldn't be provided?   If we
assume that apps don't send unicast & multicast out the same socket,
then clearly it isn't needed (but nor is the ability to set -1).  If
we assume that apps never send multicast other than min mtu, then it
isn't needed - but then IPV6_USE_MIN_MTU could simply be defined as
unicast only.   And we already know that apps want to use min mtu for
unicast packets (that's why the thing exists in the first place).
So, from this, it seems to be, that if -1 exists (as a value that can
be set), -2 (or something) needs to exist as well...

kre

---

From: JINMEI Tatuya / 神明達哉<[email-masked]>
Date: Wed, 17 Apr 2002 02:31:25 +0900

Thanks for the careful consideration,

>>>>> On Tue, 16 Apr 2002 21:51:56 +0700, 
>>>>> Robert Elz <[email-masked]> said:

> really send both unicast and multicast packets out one socket?

Sometimes yes.  In fact I have a couple of examples of such
applications.

> could just close the socket, open a new one, and bind it to the same

It could, but it is not always a good coding style (at least I believe
not).  This way we'll have to figure out the type and the protocol of
the socket, the bound address and port for the socket, all socket
option values previously set for the socket, etc..., and then recover
all the values for the new socket.  This is weird, and sometimes even
almost impossible.

Also, we usually provide an application with a way to go back to the
default, as an API convention.  So I think the existence of -1 here
makes sense, even though the knob is usually unused.

> be set), -2 (or something) needs to exist as well...

As I said above, there *are* applications that use a single socket for
both unicast and multicast applications.  However, through my
experiences, all such applications do not care about the path MTU;
they are just happy with the default.  Meanwhile, if a multicast
application ever wanted to enable path MTU discovery for multicast,
I'm quite sure that the application would use a dedicated socket for
multicast only.  Thus, from a practical point of view, I believe we do
not need the 4th argument.

As a consequence, I don't see a need to change the current proposal.
However, I guess we could add an explicit note that:
  - enabling path MTU discovery for multicast application is basically
    not recommended.
  - if an application really wants to enable it, the application
    should use a dedicated socket for multicast.

Does this make sense?

					JINMEI, Tatuya

---

From: Erik Nordmark <[email-masked]>
Date: Wed, 17 Apr 2002 20:28:58 +0200 (CEST)

> want to ever say (on one socket) "use it for multicast but not for

From an API design perspective it's odd if there are options that
can be "set" but not "unset".
In this particular case it would mean that an application which wants to
revert back to the default behavior would need to close the socket and
open a new one.

> be set), -2 (or something) needs to exist as well...

I agree it sounds crazy...

Looking at assumed applications that send both unicast and multicast out 
the same socket it seems like there is a scale of performance issues
around whether to do PMTU discovery or not. I think this scale has
three levels:
 - In some cases there are very few packets are sent to the same destination
   thus the overhead associated with getting the ICMP packet too big (and
   the data packet dropped) outweights the benefits of being able to send
   larger packets.
 - In other cases there might be several packets sent to a given destination
   but the multicast destinations have many widely spread members. Thus
   while the packet loss involved in the PMTU discovery is ok the ICMP
   'too big' implosion would mean that you don't want to do PMTU discovery
   for multicast.
 - Finally the number of members in the multicast group is small enough
   so that the ICMP implosion is not a significant concern (and multiple
   packets are sent to each unicast and multicast destination to
   amortize the cost of the dropped packet and ICMP error) so
   that PMTU discovery makes sense for both unicast and multicast.

I can't think of a place on this scale where it would make sense to
use PMTU discovery for multicast destinations but not unicast destinations.

Of course, one can envision applications where PMTU discovery makes sense
to destination A (whether unicast or multicast) but not to destination B
(whether unicast or multicast). Such applications either need to toggle the
option between sending such packets on the same socket, or use different
sockets for the two classes of destinations. But that level of
flexibility seems to be orthogonal to unicast vs. multicast.

So I think 3 values is what's needed - not 4.

  Erik

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 17 Apr 2002 11:48:27 -0700

Erik, your thoughts are all valid, i just wonder to what degree an API
should try to predict all possible applicationr equirements when it's
just the question about wether or not to restrict the API alternatives
from all possible options down to what we today may consider to be likely
alternatives. I personally think it's better not to try to predict the
future and simply keep it flexible. You're just calling for a chance
later on to bang your head because you overlooked something coming up.

Cheers
	Toerless

-- 
Thanks
   Toerless Eckert

---

From: JINMEI Tatuya / 神明達哉<[email-masked]>
Date: Thu, 18 Apr 2002 16:01:29 +0900

>>>>> On Tue, 16 Apr 2002 23:06:20 +0900, 
>>>>> JINMEI Tatuya <[email-masked]> said:

> agree that it is more natural to restrict the positive value (to 1) as

I've attached a revised version of the proposed text according to
discussions after the proposal, including very recent results.
I'll soon submit a new I-D with this change, but if you see something
missing in the attached text, please let me know.

					JINMEI, Tatuya

11.1.  Sending with the Minimum MTU

   Unicast applications should usually let the kernel to perform path
   MTU discovery, as long as the kernel support it, and should not care
   about the path MTU.  Some applications, however, might not want to
   incur the overhead of path MTU discovery, especially if the
   applications only send a single datagram to a destination.  A
   potential example is a DNS server.

   Also, path MTU discovery for multicast has severe scalability
   limitations and should thus be avoided.

   This specification defines a mechanism to avoid path MTU discovery by
   sending at the minimum IPv6 MTU [RFC-2460].  If the packet is larger
   than the minimum MTU and this feature has been enabled the IP layer
   will fragment to the minimum MTU.  To control the policy about path
   MTU discovery, applications can use the IPV6_USE_MIN_MTU socket
   option.

   As described above, the default policy should depend on whether the
   destination is unicast or multicast.  For unicast destinations path
   MTU discovery should be performed by default.  For multicast
   destinations path MTU discovery should be disabled by default.  This
   option thus takes the following three types of integer arguments:

       -1: Perform path MTU discovery for unicast destinations but do
           not perform it for multicast destinations. Packets to multicast
           destinations are therefor sent with the minimum MTU.
       0: always perform path MTU discovery.
       1: always disable path MTU discovery and send packets at the
          minimum MTU.

   The default value of this option is -1.  Values other than -1, 0, and
   1 are invalid, and an error EINVAL will be returned for those values.

   As an example, if a unicast application intentionally wants to
   disable path MTU discovery, it will add the following lines:

       int  on = 1;
       setsockopt(fd, IPPROTO_IPV6, IPV6_USE_MIN_MTU, &on, sizeof(on));

   Note that this API intentionally excludes the case where the
   application wants to perform path MTU discovery for multicast but to
   disable it for unicast.  This is because such usage is not feasible
   considering a scale of performance issues around whether to do path
   MTU discovery or not.  When path MTU discovery makes sense to a
   destination but not to a different destination, regardless of whether
   the destination is unicast or multicast, applications either need to
   toggle the option between sending such packets on the same socket, or
   use different sockets for the two classes of destinations.

   This option can also be sent as ancillary data.  In the cmsghdr
   structure containing this ancillary data, the cmsg_level member will
   be IPPROTO_IPV6, the cmsg_type member will be IPV6_USE_MIN_MTU, and
   the first byte of cmsg_data[] will be the first byte of the integer.
--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 18 Apr 2002 08:16:24 -0700

>    limitations and should thus be avoided.
                                            ... by default.

Not sure if you want to spend rfc space on tis, but this would be the
rationale i would give:

... Applications sending multicast traffic should explicitly enable
path MTU discovery only when they understand that the benefit of 
possibly larger MTU usage outweights the possible impact of MTU
discovery for active sources across the delivery tree(s).

And here is an outlook you may want to add (or not):

... This default behavior is based on todays available MTU path discovery
mechanism and may change in the future once better scalable mechanisms are
sufficiently ubiquitously available.

(As in: We do well understand what technology must be defined/deploy to
 allow for efficient and scalable path MTU discovery for ip multicast
 (like: define an appropriate GRA option and ensure that everybody and
 his routers implement this), but practice has shown that it won't happen
 because it is not there yet, it's not even defined, so at best we can start
 to work on it, but i wouldn't bet any major wagers on a chance to see it 
 in a mayority of routers within 5 years).

Cheers
	Toerless

---

From: JINMEI Tatuya / 神明達哉<[email-masked]>
Date: Fri, 19 Apr 2002 02:35:28 +0900

>>>>> On Thu, 18 Apr 2002 08:16:24 -0700, 
>>>>> Toerless Eckert <[email-masked]> said:

> sufficiently ubiquitously available.

I'm okay with the addition, but let me check, is the following your
intended text?

   ...A

   Also, path MTU discovery for multicast has severe scalability
   limitations and should thus be avoided by default.  Applications
   sending multicast traffic should explicitly enable path MTU
   discovery only when they understand that the benefit of possibly
   larger MTU usage outweights the possible impact of MTU discovery
   for active sources across the delivery tree(s).  This default
   behavior is based on todays available MTU path discovery mechanism
   and may change in the future once better scalable mechanisms are
   sufficiently ubiquitously available.

   This specification defines a mechanism to avoid path MTU discovery by
   sending at the minimum IPv6 MTU [RFC-2460]....

					JINMEI, Tatuya

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 18 Apr 2002 10:40:07 -0700

>    sufficiently ubiquitously available.

Yes, that's the paragraph i would propose.

Thanks a lot.