Subject: Re: [RRG] A data point on transit MTU size

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 23 Sep 2008 17:50:15 +0200

On 22 sep 2008, at 23:49, Brian E Carpenter wrote:

> We've argued here about whether it's reasonable to assume >1500 byte
> MTU on transit links, when running a LISP-type solution.

> Here's a data point about the real world, as far as Internet exchange
> points at the southern end go:

> http://list.waikato.ac.nz/pipermail/nznog/2008-September/014471.html

Note that they only talk about the capabilities of their switches.

Obviously jumbo capability there is a prerequisite, but this is easily  
met by pretty much all gigabit and faster gear. The next problem,  
which is harder to crack, is finding a way to send larger packets even  
though some routers on the shared infrastructure don't support this.

I've been working for a while now on a draft that would solve this  
through automatic discovery and testing of the jumboframe capability  
between two systems on a subnet two systems can use the max MTU they  
both support between them without being limited by less capable  
systems on the same subnet:

http://www.muada.com/drafts/draft-van-beijnum-multi-mtu-02.txt

However, for the internet exchange case this would probably be much  
easier to solve through static configuration. Something like static  
ARP entries with an MTU attached that can overrule the default MTU.  
Would any of our favorite vendors be interested in implementing  
something like that?

--
to unsubscribe send a message to [email-masked] with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 23 Sep 2008 15:45:39 -0700

Hi Brian, 

>-----Original Message-----
>From: Brian E Carpenter [mailto:[email-masked]] 
>Sent: Tuesday, September 23, 2008 2:32 PM
>To: Templin, Fred L
>Cc: Dino Farinacci; RRG
>Subject: Re: [RRG] A data point on transit MTU size
>
>Fred,
>
>On 2008-09-24 03:25, Templin, Fred L wrote:
>>  
>> 
>>> -----Original Message-----
>>> From: Dino Farinacci [mailto:[email-masked]]
>>> Sent: Tuesday, September 23, 2008 8:11 AM
>>> To: Templin, Fred L
>>> Cc: Brian E Carpenter; RRG
>>> Subject: Re: [RRG] A data point on transit MTU size
>>>
>>>>> -----Original Message-----
>>>>> From: Dino Farinacci [mailto:[email-masked]]
>>>>> Sent: Monday, September 22, 2008 3:13 PM
>>>>> To: Brian E Carpenter
>>>>> Cc: RRG
>>>>> Subject: Re: [RRG] A data point on transit MTU size
>>>>>
>>>>>> We've argued here about whether it's reasonable to assume
>>>> 1500 byte
>>>>> And if the assumption doesn't hold, then you fragment before 
>>>>> encapsulate. That is specified in the main LISP spec.
>>>> 1) You can't use IPv4 fragmentation if DF=1.
>>> Well, you could.
>> 
>> And break RFC791?
>
>Why would DF be set on the outer (LISP-generated) header?
>LISP can simply forbid that by specification.
>
>It's irrelevant whether DF is set on the inner 
>(end-host-generated) header. LISP is acting as a layer 2 as 
>far as that header is concerned.
>There's no violation of 791, since the packet will be 
>reassembled before it's decapsulated.

This is in fact the very principle employed by SEAL, i.e.,
it is OK to do *outer* fragmentation even if the *inner*
packet has DF=1, because it is the ETR and not the final
destination that has to reassemble. (The final destination
therefore gets the inner packet in its original form.)

However, LISP talks about fragmenting the *inner* packet
(even if DF=1) before encapsulating as two *outer* packets,
in which case it is the final destination that gets to
reassemble. (But, the LISP ITR cannot know the size of
the final destination's reassembly buffer, nor whether
the final destination is even capable of reassembly.)
 
>>>> 2) You can't send at high data rates if you use
>>>>   IPv4 fragmentation.
>>> Yes, you can. That is implementation dependent.
>> 
>> But sub-optimal implementations risk data corruption, which could be 
>> very bad.
>
>Yes, we agree that sub-optimal implementations are a bad idea.
>We also agree, I trust, that writing perfect reassembly code 
>is fairly hard (although I speak from 20 year old experience).
>In particular the RFC 4459 and 4963 issues have to be handled.

But, LISP does not propose reassembly at the ETR; it
proposes reassembly at the final destination and with
no coordination from the ITR.

>I don't have an opinion whether SEAL is necessary for that, or 
>whether LISP can just rely on basic frag. 

Hopefully from the above it can be seen that basic frag
per the LISP spec is optimistic at best and potentially
dangerous as well.

Fred
[email-masked] 

>
>   Brian
>> 
>>>> IMHO, LISP should be SEALed. I will be here and ready to
>>> talk about it
>>>> whenever you are.
>>> There is no reason to change our position.
>> 
>> In the lisp-interest archives, it was shown how 
>incorporation of SEAL 
>> into LISP would in fact be better from an efficiency standpoint in 
>> that the header size remains the same plus you gain 6 additional 
>> Locator Reach Bits. That is in addition to the advantage 
>that the path 
>> MTU issues are put to rest.
>> 
>>> Bigger fish to fry,
>> 
>> Eh?
>> 
>> Fred
>> [email-masked]
>> 
>>> Dino
>>>
>>>>
>>>> Fred
>>>> [email-masked]
>>>>
>>>>> Dino
>>>>>
>>>>>>   Brian
>>>>>> to unsubscribe send a message to [email-masked] 
>with the word 
>>>>>> 'unsubscribe' in a single line as the message text body.
>>>>> to unsubscribe send a message to [email-masked] with 
>the word 
>>>>> 'unsubscribe' in a single line as the message text body.

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 23 Sep 2008 15:56:26 -0700

Dino, 

>Sent: Tuesday, September 23, 2008 2:53 PM
>To: Brian E Carpenter
>Cc: Templin, Fred L; RRG
>is fairly 
>> hard (although I speak from 20 year old experience).
>> I don't have an opinion whether SEAL is necessary for that, 
>or whether 
>> LISP can just rely on basic frag.
>
>The code for reassembly isn't hard, it's allocating the 
>buffers to store the arrived fragments. That's nearly 
>impossible in hardware.

Then, consider any link with linkMTU < ~2KB as "marginal"
and in need of replacement. As long as the ITR can detect
that there is a marginal link in the path, it can still
push packets through using SEAL reassembly at the ETR
while a trouble ticket is called in on the link.

Once all of the < ~2KB links are eradicated, there will
be no more reassembly required at ETRs. Then, the same
approach can be reiterated for more robust definitions
of "marginal", and eventually everything smaller than ~9KB
will be weeded out and we will converge on a jumbo-clean
Internet.

Sound like a plan?

Fred

---

From: "Tony Li" <[email-masked]>
Date: Wed, 24 Sep 2008 14:23:28 -0700

Hi Fred, Dino,

|>> Patently nonsense.  Packet reassembly in hardware is very much
|>> straightforward if one is willing to burn large buffers to do so.   
|>> Imagine
|>
|>The willing part is the impossible part.

Well, that sounds like a personal problem.  ;-)

|I'm not convinced as to how "large" the buffers need to
|be? AFAICT, the size depends on the number of simultaneous
|reassemblies in progress plus the degree to which the
|network mis-orders packets plus the degree to which stale
|reassemblies are garbage-collected. This, plus SEAL puts
|a firm cap of 2KB as the largest size an ETR will ever
|have to reassemble.

Obviously you can optimize, but even a naïve implementation has to have a
bound on the number of reassemblies that it can have in progress at any
given time.  Again a trivial implementation is to provide a max sized buffer
for each such reassembly.

If folks want to do a more sophisticated implementation, then it wouldn't be
too hard to use sparse matrix techniques and/or cell techniques to segment
incoming fragments into cells and then reassemble the fixed sized cells.
This reduces the problem to one that's previously been solved.

Tony

---

From: Stephen Sprunk <[email-masked]>
Date: Wed, 24 Sep 2008 18:15:00 -0500

Tony Li wrote:
> |The code for reassembly isn't hard, it's allocating the buffers to  
> |store the arrived fragments. That's nearly impossible in hardware.
>
>
> Patently nonsense.  Packet reassembly in hardware is very much straightforward if one is willing to burn large buffers to do so.  Imagine taking an implemention in C and simply converting it to Verilog.  Not at all out of the question.
>
> In fact, a subset of this problem has been solved for a very long time. Remember ATM?  Remember a SAR chip?  That's reassembly of fixed size packet fragments.
>   

I remember building ATM networks (not voluntarily) and that whole 
movement seemed to burn out when it was discovered that SAR chips 
couldn't keep up above OC-12 speeds -- and that was with easy fixed-size 
cells.  I also remember months of my life around the same time eaten up 
by tuning a certain vendor's routers for optimum use of various-sized IP 
buffers so that they wouldn't drop packets even without reassembly being 
an issue.

Now, I'm not a hardware guy, but I'd like to see some assurance from 
those that are that things have changed enough that we can ask routers 
now and in the foreseeable future will be able to reassemble the 50% of 
packets that will be at the Ethernet MTU (whatever that may be) + LISP 
header size at line rate -- including for customers who have GE, 10GE, 
and (in a few years) 100GE uplinks to their ISP(s).

S

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 25 Sep 2008 09:26:11 -0700

Tony,

>Obviously you can optimize, but even a naïve implementation 
>has to have a bound on the number of reassemblies that it can 
>have in progress at any given time.  Again a trivial 
>implementation is to provide a max sized buffer for each such 
>reassembly.

A Linux box is a far cry from a carrier-grade router, but
Linux sets high/low water marks of 256K/192K for its cache
of outstanding IPv4 reassemblies. It also provides a tunable
parameter for how aggressively to purge stale reassemblies.
With SEAL, I believe the tuning parameters for reassembly of
IPv4-fragmented SEAL packets can and should be clamped down
very tightly indeed even for high-end routers (i.e., even to
the point of dropping all IPv4-fragmented SEAL packets) since
SEAL will very quickly tune out all IPv4 fragmentation.

Greater care must be taken with SEAL-layer reassembly, since
SEAL-segmented packets need to be reassembled and forwarded
with high assurance. However, SEAL-layer reassembly in itself
is an indication that there are marginal links in the path
that can be immediately identified and a trouble ticket called
in to their owners. The interdomain routing community peer
pressure alone would serve as ample motivation for upgrading
obsolete network gear to new and relatively inexpensive
equipment that supports jumbo-clean links. The eventual end
state is a jumbo-clean interdomain core, with a transition
timeframe that parallels the rollout of LISP itself.

Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 25 Sep 2008 09:32:38 -0700

>From: Stephen Sprunk [mailto:[email-masked]] 
>Sent: Wednesday, September 24, 2008 4:15 PM
>To: [email-masked]
>Cc: Routing RG
>> Patently nonsense.  Packet reassembly in hardware is very 
>much straightforward if one is willing to burn large buffers 
>to do so.  Imagine taking an implemention in C and simply 
>converting it to Verilog.  Not at all out of the question.
>>
>> In fact, a subset of this problem has been solved for a very 
>long time. Remember ATM?  Remember a SAR chip?  That's 
>reassembly of fixed size packet fragments.
>>   
>
>I remember building ATM networks (not voluntarily) and that 
>whole movement seemed to burn out when it was discovered that 
>SAR chips couldn't keep up above OC-12 speeds -- and that was 
>with easy fixed-size cells.  I also remember months of my life 
>around the same time eaten up by tuning a certain vendor's 
>routers for optimum use of various-sized IP buffers so that 
>they wouldn't drop packets even without reassembly being an issue.
>
>Now, I'm not a hardware guy, but I'd like to see some 
>assurance from those that are that things have changed enough 
>that we can ask routers now and in the foreseeable future will 
>be able to reassemble the 50% of packets that will be at the 
>Ethernet MTU (whatever that may be) + LISP header size at line 
>rate -- including for customers who have GE, 10GE, and (in a 
>few years) 100GE uplinks to their ISP(s).

But, that's just it. We're not talking about steady-state
line-rate reassembly for the long term. We are talking about
very short term identification and eradication of marginal
links in the interdomain core that probably should have
been replaced years ago anyway - and for relatively low
cost. As the obsolete links go away, reassembly at ETRs
dissipates and the Internet eases into a comfort zone that
includes all-jumboframe links everywhere. IMHO, it is well
worth the (very) short-term transition.

Fred