Subject: rfc2292bis text proposal

From: Toerless Eckert <[email-masked]>
Date: Fri, 5 Apr 2002 10:26:05 -0800

I would like to propose another section to rfc2292bis for the issue
of MTU path discovery with ipv6 multicast. The issue with the current
rfc2292bis draft is that path discovery is enabled by default on
ipv6 sockets and with ipv6 multicast this is something that applications
should shy away from much more than unicast fragmentation because of the
implosion of IGMP messages sent back to the source.

The most critical case from what we see in IPv4 is the
case of applications sending large UDP MTU (4500 bytes or the like)
that are getting fragmented at the source into 1500 byte fragments,
and then hit some PPPoE/L2TP/GRE/MPLS-VPN/etc.. links (in practice these
are hundreds of links) and you have the issue of either re-fragmentation
(v4 only) or implosion of ICMP messages. And then add the pathetic case of
the broken network with stupid access control policies, and the ICMP messages
may not make it back at all and the application breaks completely in some
subtrees of it's delivery tree, with routers continuously sending rate-limited
ICMP messages, and receivers not getting traffic.

For this reason, i would like to propose the following, and i am not
sure wether or not this could entirely be put into rfc2292bis or if
this would also be considered to be an architectural issue (i hope it
can really be claimed to be a socket behavior only issue and thus
rfc2292bis):

11.5 Fragmentation and IPv6 multicast packets

  Path MTU discovery for multicast has severe scalability limitations
  and should thus be avoided. To ease porting of ipv4 applications and
  reduce the risk of application developers overlooking this issue,
  originating of ipv6 multicast packets must behave in the following way:

  Originating ipv6 multicast packets will insert a fragmentation header
  when the data is larger than the minimum IPv6 MTU [RFC-2460] or the
  interface MTU - whichever is smaller.  Setting of the IPV6_USE_MIN_MTU
  option on the socket has no effect on originating ipv6 multicast packets.

  If an application wants to originate larger ipv6 multicast fragments than
  the above default behavior, it must use the IPV6_MULTICAST_MTU_DISC 
  socket option:

       int  on = 1;
       setsockopt(fd, IPPROTO_IPV6, IPV6_MULTICAST_MTU_DISC, &on, sizeof(on));
  
  By default, this option is disabled. Enabling it will allow for the
  application to originate packets as large as the interface MTU without
  insertion of fragment headers (fragment headers will still be inserted
  when the packet is larger than the interface MTU - unless the IPV6_DONTFRAG
  option has been set of course).

  The IPV6_DONTFRAG and IPV6_RECVPATHMTU options apply to ipv6 multicast
  packets as they apply to ipv6 unicast packets.

Thanks
        Toerless
--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: JINMEI Tatuya / 神明達哉<[email-masked]>
Date: Sat, 06 Apr 2002 16:49:20 +0900

>>>>> On Fri, 5 Apr 2002 10:26:05 -0800, 
>>>>> Toerless Eckert <[email-masked]> said:

> implosion of IGMP messages sent back to the source.
               ^^^^should be ICMP

(snip)

> can really be claimed to be a socket behavior only issue and thus

I see your point.  However, I'm not sure if we really need a new
option to deal with this issue.  An application can in fact avoid ICMP
implosion by setting IPV6_USE_MIN_MTU.  This only requires 4 (or a bit
more) additional lines like:

{
	int on = 1;
	if (setsockopt(sock, IPPROTO_IPV6, IPV6_USE_MIN_MTU, &on, sizeof(on)))
		/* error handling */;
}

Is it really a big deal to let programmers omit the additional lines,
comparing to the complexity of adding a new socket option?  My
personal feeling is that it is enough to add some note which suggests
setting IPV6_USE_MIN_MTU for sockets used to send multicast packets.

Secondly, the proposed option "IPV6_MULTICAST_MTU_DISC" seems to me a
bit ad-hoc.  If we wanted to go with this path, I'd rather generalize
the MTU-related option like as follows:

  - add a new socket option, say, "IPV6_PMTU_DISC", which takes an
    integer as an argument
  - the semantics of the argument is:
    + 0: let the kernel make a decision whether to perform path MTU
         discovery.  the recommended way to make the decision is:
         fragment packets at the minimum MTU if the destination is a
         multicast address.  otherwise, perform path MTU discovery.
    + 1: force the kernel to perform path MTU discovery
    + 2: specify the kernel to fragment packets at the minimum MTU
         (equivalent to IPV6_USE_MIN_MTU)
    + 3: specify the kernel not to fragment packets
         (equivalent to IPV6_DONTFRAG)
  - this option can also be used as an ancillary data item for
    per-packet control.
  - IPV6_USE_MIN_MTU and IPV6_DONTFRAG will be deprecated by the new
    IPV6_PMTU_DISC option.

(There should have been a similar proposal before, but I don't recall
the result of the discussion.)

					JINMEI, Tatuya
					Communication Platform Lab.
					Corporate R&D Center, Toshiba Corp.
					[email-masked]
--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: JINMEI Tatuya / 神明達哉<[email-masked]>
Date: Mon, 08 Apr 2002 23:56:32 +0900

>>>>> On Sat, 6 Apr 2002 09:28:39 -0800, 

> Well, i've just seen bad and worse ip multicast applications over the
> last 10 years, programmers have been ignorant about the issues at hand
> and people who deploy the applications have been ignorant about the
> network issues they cause.

Out of curiosity, do you have concrete examples of the bad
applications?  I tend to agree with you, but I'm still not fully
convinced that application programmers are that silly and we need to
add the complicated intelligence to the kernel.

I'd also like to hear from others.  We cannot make a decision just
from the "local" conversation...

If we reach consensus on adding something to deal with this issue,
then we'll definitely need another revision of the draft (and another
working group last call?).

					JINMEI, Tatuya

---

From: Steve Deering <[email-masked]>
Date: Mon, 8 Apr 2002 08:29:19 -0700

>from the "local" conversation...

Jinmei-san,

There is at least some precedent for having different default behavior
for IP multicast than unicast, in order to reduce the chance of
accidental "misuse" of multicast:  in the IPv4 multicast socket support,
we had TTL default to 1 for multicast, so that someone wanting to send
multi-hop multicasts had to explicitly say so.  I think Toerless's
suggestion is in the same spirit, ensuring that multicast MTU discovery
doesn't happen unless explicitly requested.  I don't think it takes that
much complicated intelligence in the kernel to have different defaults
for multicast and unicast.

It's not so much judging application programmers as "silly" as reminding
them of consequences they may have forgotten or never realized.

Steve

---

From: Toerless Eckert <[email-masked]>
Date: Sat, 6 Apr 2002 09:28:39 -0800

Well, i've just seen bad and worse ip multicast applications over the
last 10 years, programmers have been ignorant about the issues at hand
and people who deploy the applications have been ignorant about the
network issues they cause. So my point in face of all this is those
two recommendations in my original mail:

  - Change the default behavior (1280 max MTU) for ip multicast
    packets so all those lazy programmers and users get the desired
    default behavior. Don't trust them to do/program the right thing by
    default.
  
  - Ignore the setting of IPV6_USE_MIN_MTU. Again, this is for reasons of
    avoiding that programmers who don't really think about multicast
    but use it will get the wrong default behavior. It would be completely
    confusing to have the default behavior for sending ip multicast packets
    to be 1280, but then if somebody explicitly set's IPV6_USE_MIN_MTU to
    OFF, we would change - that is, why i am very much persuaded, that
    this option should only apply for unicast.
  
> the MTU-related option like as follows:

Yes, the proposed option was ad-hoc and i am happy with trying to make
it more generic, but again:

  - The main key to all the API issues is the fact that we should have
    two different defaults for unicast and multicast: Path MTU discovery
    is cool and wonderfull for unicast, it is bad and silly for multicast
    (especially for non-ssm style multicast where you may have groups
     with thousands of hosts starting to send few packets amongst themselves
     and routers really don't want to keep track of them individually).

  - Because of the different default behaviors that we want for unicast
    and multicast, a more generic option should always differentiate
    wether it is applied to unicast and/or multicast !

>   - IPV6_USE_MIN_MTU and IPV6_DONTFRAG will be deprecated by the new

So, logically i want a second argument: wether or not this option should
apply to unicast and/or multicast. Now we can still make this all one
integer by using a simple bitmask:

   #define PMTU_UNICAST     1
   #define PMTU_MULTICAST   2
   #define PMTU_ALLPAKETS   3

   #define PMTU_DEFAULT     0 /* default behavior: ucast:disc, mcast:min */
   #define PMTU_MTUDISC     4  
   #define PMTU_MINMTU      8
   #define PMTU_DONTFRAG   12

   (eg: setsockopt(s, IPPROTO_IPV6, IPV6_PMTU_DISC,
				    PMTU_ALLPAKETS | PMTU_DONTFRAG))

or use bitfield definitions if that's more conformant with policies.

And if that option would replace USE_MIN_MTU and DONTFRAG, then i would
like that very much !

Cheers
	Toerless

---

From: JINMEI Tatuya / 神明達哉<[email-masked]>
Date: Wed, 10 Apr 2002 03:42:29 +0900

>>>>> On Mon, 08 Apr 2002 23:56:32 +0900, 
>>>>> JINMEI Tatuya <[email-masked]> said:

> add the complicated intelligence to the kernel.

I've got an example from a private message, and I now agree that we
need some changes on the current API spec.

Considering the tradeoff between the advantage of safe API
specification and the disadvantage of the loss of compatibility and/or
the complexity of the additional API knobs, I'd like to propose the
following approach:

  allow IPV6_USE_MIN_MTU 3 arguments, -1, 0, and 1.  the semantics of
  the 3 values are as follows:
  -1: if the destination is unicast, regard the path MTU and perform
      path MTU discovery if necessary.  if the destination is
      multicast, fragment the packet at the minimum MTU (and disable
      path MTU discovery as a result)
      ***This is the default value of the option, including the case
      where the application does not explicitly specify this option***
   0: always regard the path MTU, regardless of the destination
      (unicast or multicast).
   1: always fragment packets at the minimum MTU, regardless of the
      destination.

This way the default behavior will be optimal while keeping
compatibility to existing implementations that use IPV6_USE_MIN_MTU
and avoiding additional socket options.

This may still be regarded as an ad-hoc solution, but I believe this
is practically the best way.

Is this change acceptable?

					JINMEI, Tatuya

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 9 Apr 2002 13:54:19 -0700

If we would change the default for applications that don't explicitly
configure any value of IPV6_USE_MIN_MTU to now be the value of "-1",
then this extension would give us something, otherwise not (the
main incentive is to change the default behavior for multicast packets).

Even with -1 as a default i think that i would prefer a cleaner API and quite
frankly i don't think that we are so short into the life of IPv6 that it
doesn't really make too much sense to obfuscate such rather exotic
API pieces for the sake of backward compatibility (i.e.: there's so little
application already written using that API compared to what we in the
future want to see being written but using a clean API).

For this reason i proposed to add a new API call that would allow to
set the MTU behavior of unicast and multicast individually or together
and simply consider IPV6_USE_MIN_MTU to only apply to unicast - after all,
it's quite obvious that multicast wasn't really much thought off when that
API call was designed, so going along this path not only gives us a clean
new API call, but also allows us to keep backward compatibility of
IPV6_USE_MIN_MTU for that scope of applications (unicast) that were
obviously only considered for it in the first place - i.e.: sufficient
backward compatiblity AND a clean new API call (API call along the lines
of the last emails proposal i sent).

Cheers
	Toerless

-- 
Thanks
   Toerless Eckert

---

From: JINMEI Tatuya / 神明達哉<[email-masked]>
Date: Wed, 10 Apr 2002 17:06:37 +0900

>>>>> On Tue, 9 Apr 2002 13:54:19 -0700, 

> main incentive is to change the default behavior for multicast packets).

I'm not sure if I understand the nuance of the sentence above, but I
intended in my proposal that "we do change the default for
applications that don't explicitly configure any value of
IPV6_USE_MIN_MTU to now be the value of -1".

> future want to see being written but using a clean API).

I'm sorry, but I have a quite different view about the deployment
status of IPv6 and the advanced API.  (As you may have known) many
major OSes (including Windows, Solaris, Linux and BSDs) have officially
supported IPv6.  Some major network vendors have already shipped
IPv6-ready routers.  We'll see (or even have seen) nation-wide IPv6
commercial services here in Japan within this year.  Also, I know at
least one OS (BSD/OS) has already shipped with the rfc2292bis API.  I
also hear that Solaris has (at least partly) adopted rfc2292bis.  I
don't know the situation about other OSes, but I guess BSD/OS and
Solaris are not the only instance.  BIND 9 uses the IPV6_USE_MIN_MTU
socket option in its release versions since more than a year before.

Unfortunately, rfc2292bis has lost compatibility to its ancestor, RFC
2292, in many points.  The current situation where some OSes provide
RFC 2292 and some provide rfc2292bis is really confusing for
application programmers.  This is another reason why we want to fix
and publish the API spec as soon as possible.  Adding another socket
option surely introduces a delay of the process.  Frankly, I really
want to keep the delay as short as possible, as long as the API spec
provides the "correct" behavior (in this case, it means the default
behavior for multicasted packets will fragment the packets at the
minimum MTU).

Also, I don't think the approach I proposed obfuscates the API spec if
we describe the background well, and I'd rather prefer the advantage to
keep the number of options than introducing the "clean" API.  (This is
just my thoughts.  I know you may have a different opinion, on which
we can perhaps not reach consensus.)

Considering all of this above, I'd very much like to go with the
approach I proposed.  I'll hear from others for while, but could you
agree with the decision if there is no other strong objection?

Thanks,

					JINMEI, Tatuya

---

From: Erik Nordmark <[email-masked]>
Date: Thu, 11 Apr 2002 18:26:32 +0200 (CEST)

Is there a middle ground where the API can be made more clear
than using "-1" without having to replace IPV6_USE_MIN_MTU?

We already have IPV6_option and IPV6_MULTICAST_option in some cases.
Thus would it make sense to 
 1) declare the IPV6_USE_MIN_MTU only applies to unicast.
 2) define a new IPV6_MULTICAST_USE_MIN_MTU whose default would be "true".

   Erik

---

From: JINMEI Tatuya / 神明達哉<[email-masked]>
Date: Fri, 12 Apr 2002 20:48:00 +0900

>>>>> On Thu, 11 Apr 2002 18:26:32 +0200 (CEST), 
>>>>> Erik Nordmark <[email-masked]> said:

>  2) define a new IPV6_MULTICAST_USE_MIN_MTU whose default would be "true".

This is essentially the same idea of the original proposal...the
problem of this approach is that programmers would wonder "why
does IPV6_USE_MIN_MTU not contain "UNICAST" but the use of it is
limited to unicast?".  I don't think this makes the API "more clear".
Either way we'll need to describe the background and its limitation
clearly, and if so, I'd prefer to keep the number of options.

Actually I can make a compromise with either approach *if we can make
a quick consensus*.  To make the approach I proposed clearer, I've
attached concrete text for the revised version of IPV6_USE_MIN_MTU
according to my proposal.  If we can live with this (though you may
prefer the other one), please let us go.

Thanks,

					JINMEI, Tatuya

===
 11.1.  Sending with the Minimum MTU

   Unicast applications should usually let the kernel to perform path
   MTU discovery, as long as the kernel support it, and should not
   care about the path MTU.  Some applications, however, might not
   want to incur the overhead of path MTU discovery, especially if the
   applications only send a single datagram to a destination.  A
   potential example is a DNS server.

   Also, path MTU discovery for multicast has severe scalability
   limitations and should thus be avoided.

   This specification defines a mechanism to avoid path MTU discovery by
   sending at the minimum IPv6 MTU [RFC-2460].  If the packet is larger
   than the minimum MTU and this feature has been enabled the IP layer
   will fragment to the minimum MTU.  To control the policy about path
   MTU discovery, applications can use the IPV6_USE_MIN_MTU socket
   option.

   As described above, the default policy should depend on whether the
   destination is unicast or multicast.  For unicast destinations path
   MTU discovery should be performed by default.  For multicast
   destinations path MTU discovery should be disabled by default.
   This option thus takes the following three types of integer
   arguments,

     x == -1: use kernel default.  That is, perform path MTU discovery
              for unicast, and disable it for multicast.
     x ==  0: always perform path MTU discovery.
     x >=  1: always disable path MTU discovery and send packets at
              the minimum MTU.
     (values less than -1 are invalid.)

   For example, if a unicast application intentionally wants to
   disable path MTU discovery, it will add the following lines:

       int  on = 1;
       setsockopt(fd, IPPROTO_IPV6, IPV6_USE_MIN_MTU, &on, sizeof(on));

   This option can also be sent as ancillary data.  In the cmsghdr
   structure containing this ancillary data, the cmsg_level member
   will be IPPROTO_IPV6, the cmsg_type member will be
   IPV6_USE_MIN_MTU, and the first byte of cmsg_data[] will be the
   first byte of the integer.
--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: JINMEI Tatuya / 神明達哉<[email-masked]>
Date: Fri, 12 Apr 2002 21:06:20 +0900

>>>>> On Fri, 12 Apr 2002 20:48:00 +0900, 

> prefer the other one), please let us go.

I forgot to mention this:
  some parts of the text was derived from the original proposal from
  Toerless.  Thanks!

					JINMEI, Tatuya

---

From: JINMEI Tatuya / 神明達哉<[email-masked]>
Date: Fri, 12 Apr 2002 23:08:06 +0900

>>>>> On Fri, 12 Apr 2002 06:49:54 -0700, 

> My concern was to change the default. Your text below seem
> to indicate this, so it would solve my concern. As far as your
> text is concerned, I would love to see more explicitly the sentence 
> "the default setting of this option for a socket is -1". In addition,
> i would strongly recommend to call this the "API default" instead of the
> "kernel default", because "kernel default" sounds as if it is up to the
> kernel to decide what the default is (i.e.: i am worried that some kernel
> developer could think that he can decide what to do in the -1 case). But
> that's probably just me being paranoid.

Okay, I don't have any problem with using the term "API default."

> Beside this, i would just argue that psychologically i'd love to see
> the differentiation UNICAST/MULTICAST to show up in the API parameters
> explicitly to help developes think about the difference, but that's just
> cosmetic. I agree with you that just a _MULTICAST_ option like Eric sugested
> is a bit confusing, but i have no strong opinions here. The real advantage
> of NOT using IPV6_USE_MIN_MTU is to catch the hopefully small/non-existing
> number of applications who already think that the value 0 to it is the
> default and thus set it.

Then you seem to agree with my proposal (thanks).  What about you,
Erik?  If you're okay too, I'll soon update the draft according to
the proposal.

					JINMEI, Tatuya

---

From: JINMEI Tatuya / 神明達哉<[email-masked]>
Date: Mon, 15 Apr 2002 15:11:29 +0900

>>>>> On Fri, 12 Apr 2002 23:08:06 +0900, 

> Erik?  If you're okay too, I'll soon update the draft according to

Though I've not seen an explicit response, I guess we've reached a
consensus.  So my next question is, what should I do with this in
terms of the standardization procedure?  Should I submit a new draft
and ask chairs a new wg last call?

					JINMEI, Tatuya

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 12 Apr 2002 06:49:54 -0700

Jinmei,

Thanks
	Toerless

-- 
Thanks
   Toerless Eckert