Subject: [GROW] ixp jumbo frames doc

From: Christopher Morrow <[email-masked]>
Date: Thu, 17 Nov 2011 03:09:11 -0500

We had a bit of a lively discussion in the meeting today about:
  <http://tools.ietf.org/html/draft-mlevy-ixp-jumboframes-00>

Some of the topics covered were:
  o Is there a problem with CRC problems with 9k ?
  o is this something that the IEEE reigns supreme on?
  o should there be other methods of deployment?
  o is this a BCP instead of Informational doc?
  o should this be adopted for WG work?

We'll pass along a separate call for adoption as well, but keep these
points in mind (and hopefully let's chat some about these as well)

-chris
(co-chair)

---

From: "iLya" <[email-masked]>
Date: Thu, 17 Nov 2011 09:37:16 +0100

I've read the draft but couldn't attend the session due to clash in 
schedule.

One point which is currently missing in the draft and which I consider very 
important is effect of transition to "jumbo" on BGP. Many BGP 
implementations today group updates to similar peers and format them only 
once then replicate to multiple peers. Formatting is arguably more 
resource-consuming compare to replication. When MTU is the same for all IXP 
customers, then most of UPDATE messages will be formatted only once. During 
transition period there will be peers with different MTU, and depending on 
BGP implementation more than one formatting will be required. It would be 
nice if the draft could be updated to address this subject.

Kind regards,
iLya

--------------------------------------------------
From: "Christopher Morrow" <[email-masked]>
Sent: Thursday, November 17, 2011 9:09 AM
To: <[email-masked]>; <[email-masked]>; "Martin J. Levy" 
<[email-masked]>
Subject: [GROW] ixp jumbo frames doc

> _______________________________________________
> GROW mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/grow
>
>

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 17 Nov 2011 04:59:22 -0500

> nice if the draft could be updated to address this subject.

I had heard from John Scudder (and Dave Ward actually) at some point
in the near past that essentially the idea of 'peer-groups' is just
syntactic sugar at this point in time. There is some attempt to keep
like-peers together in update groups, but over time drift in state
happens and peers are separated from the herd.

In the end, this isn't super important in at least 2 bgp
implementations in the field today...

-chris

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 17 Nov 2011 10:08:58 -0800

Hi Chris,

See below for some follow-up: 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Christopher Morrow
> Sent: Thursday, November 17, 2011 12:09 AM
> To: [email-masked]; [email-masked] [email-masked]; 
> Martin J. Levy

Please note that the concern is specifically for CRC-32.
Several works have discussed the error characteristics
of CRC-32 in relation to data set sizes:

    Koopman, P., "32-Bit Cyclic Redundancy Codes for
    Internet Applications", Dec. 2002.

    Stone, J. & Partridge, C., "When the CRC and TCP
    Checksum Disagree", Aug. 2000.

    Jain, R., "Error Characteristics of Fiber Distributed
    Data Interface (FDDI), August 1990.

Those works tend to support the assertion that CRC-32
is adequate for data set sizes up to "about 9k or even
a little bit more".

That said, I can easily imagine something stronger than
CRC-32, and it is the responsibility of the link layer
to provide adequate error checking if it intends to
support larger MTUs. 

>   o is this something that the IEEE reigns supreme on?

It is the link layer's job to ensure that it performs
adequate error checking for packets of various sizes;
it is not the network layer's place to guess at how well
the link layer is doing its job. So, the link layer's
advertised MTU is a certification that suitable error
detection will be performed for packets no larger than
the MTU; otherwise, it will become known as a "bad link".

>   o should there be other methods of deployment?

Other methods of deployment than what?

>   o is this a BCP instead of Informational doc?

The document is proposing jacking up the Internet cell
size from 1500 to 9000, where 9000 seems like a good
fit for the vast majority of link paths anticipated
for the near future. That seems like BCP to me.

At the same time, at some point down the road we may
need to turn the crank again and jack up from 9000
to something still larger. Hence, the BCP for the
near term should leave open the door for an update
at some point in the future.

>   o should this be adopted for WG work?

Yes.

Thanks - Fred
[email-masked]

---

From: Tony Li <[email-masked]>
Date: Thu, 17 Nov 2011 10:15:50 -0800

> for the near future. That seems like BCP to me.

It should again be noted that the IETF lacks an enforcement arm or any legal standing to dictate operations whatsoever.

Thus, even as a BCP, this basically has the effect of saying "we think this is a really good idea, would you please do this?"

Tony

---

From: Jakob Heitz <[email-masked]>
Date: Thu, 17 Nov 2011 18:46:12 -0500

BGP uses TCP. Differing MTU has no impact on update generation in BGP. BGP today has a maximum message size of 4096 bytes. TCP slices and dices that into IP packets of its own choosing.

--
Jakob Heitz.

---

From: john heasley <[email-masked]>
Date: Thu, 17 Nov 2011 23:51:19 +0000

> BGP uses TCP. Differing MTU has no impact on update generation in BGP. BGP today has a maximum message size of 4096 bytes. TCP slices and dices that into IP packets of its own choosing.

i was about to reply with the same, then it occured to me that an
implementation may have closer ties to the underlying mtu for rfc2385
or similar.  so the question becomes where implementation-specific
limitation belong in the draft.

---

From: Tony Li <[email-masked]>
Date: Thu, 17 Nov 2011 15:55:14 -0800

> limitation belong in the draft.

They don't.

Tony

---

From: "Martin J. Levy" <[email-masked]>
Date: Thu, 17 Nov 2011 16:23:03 -0800

I've been waiting for the feedback (which has been great) before commenting or editing up a revision to the document; however this is one point that's worth noting.

We did a test on a Jumbo Frame enabled IX (NETNOD). We looked at all the peers and listed the MSS against the Jumbo or non-Jumbo VLAN. By doing a simple "show ip bgp nei" and "show ipv6 bgp nei" command, collecting the data and looking at it; I find:

    MSS     IP ADDRESS          HARDWARE  VLAN
    ----    -----------------   --------  ----

    1240    ***.***.**.189      Cisco     1500
    ...
    1440    ***.***.**.26       Cisco     1500
    1420    ****:***:*:**::26   Cisco     1500
    ...
    1440    ***.***.**.34       Cisco     4470
    1420    ****:***:*:**::34   Cisco     4470
    ...
    1460    ***.***.**.172      Brocade   4470
    ...
    4410    ****:***:*:**::19   Juniper   4470
    4430    ***.***.**.19       Juniper   4470
    ...
    4410    ****:***:*:**::24   Juniper   4470
    4430    ***.***.**.24       Juniper   4470
    ...
    4430    ***.***.**.143      Cisco     4470
    4430    ***.***.**.181      Juniper   4470

I only included the interesting peers. There's tons of peers within the 1,4xx MSS range; but there are ONLY six with ~4,400 Byte MSS.

The hardware column is the hardware of the peer (we run Brocade).

Hence ... There is MSS negotiation; hence there is "value"; but I believe I should only note this fact within the document and not make in any way a requirement to tune or change the BGP or TCP knobs to take advantage of the large MTU path.  The fact that some sessions enable this and some don't is perfectly OK. It's being decided by the routers for whatever reason they choose. 

NETNOD's Jumbo Frame setup has been working for many many years (over various h/w platforms) and has stood the test of time.

Does this help?

Martin

PS: I'll respond to other points in a bit.

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 18 Nov 2011 00:48:20 -0500

> Other methods of deployment than what?

Sorry, I was rushing :( One of the proposals was to use a secondary
VLAN at the exchange for 'larger' packets, one was to forklift the
exchange, I believe there was a third discussed in the meeting as
well.

> for the near future. That seems like BCP to me.

Err, it's proposing setting the IXP interfaces to 'larger than 1500'.
One of the problems is that some networks already show up at exchanges
with 'larger than 1500 on their backbone/core facing interfaces, and
likely on most other interfaces in the network as well, so these IXP
interfaces are one-off situations :(

I don't see this proposal as jacking up the whole of the Internet,
there are still a vast number of edge interfaces which are not capable
of changing MTU, but making the one-off problems go away seems like a
good idea.

>>   o should this be adopted for WG work?

thanks!

-chris

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 18 Nov 2011 08:30:00 -0800

Hi Chris, 

> From: Christopher Morrow [mailto:[email-masked]] 
> Sent: Thursday, November 17, 2011 9:48 PM
> To: Templin, Fred L
> Cc: [email-masked]; [email-masked] [email-masked]; 
> Martin J. Levy
> Subject: Re: [GROW] ixp jumbo frames doc
> 

Thanks for the clarification; I have no further comment
on this.

> interfaces are one-off situations :(

Right, but it is asking for a specific minimum MTU of
9000 in the core and is hence striving for a new
Internet cell size. IMHO, this is a necessary first
step toward migration to larger MTUs.

Don't get me wrong, though - the Internet will still
need to accommodate MTU diversity since there is no
way to legislate a sweeping change of this nature
that can be deployed all at once. That means PMTUD
will still be necessary, and should be made as robust
as possible. 
 
> of changing MTU, but making the one-off problems go away seems like a

Right; there are certainly edge interfaces for which
larger MTUs are not practical. Most wireless devices
fall under that category. However, migrating the core
to 9000+ allows edge devices that can do better than
1500 to discover larger MTUs.

> >>   o should this be adopted for WG work?

Fred

---

From: john heasley <[email-masked]>
Date: Fri, 18 Nov 2011 17:31:28 +0000

> Hence ... There is MSS negotiation; hence there is "value"; but I believe I should only note this fact within the document and not make in any way a requirement to tune or change the BGP or TCP kno…

enable what?  it's TCP that is concerned with the MTU (& MSS), not BGP.
a vendor might provide data about the underlying tcp connection in its
bgp session output, but that doesn't mean that BGP is concerned with its
value - its just handy debugging information for the operator.  I suppose
an implementation could use the MTU or MSS for something, but it shouldn't
need to do that if its TCP stack were implemented properly - implementation
specifc.