Subject: Re: [lisp] IPv6 UDP checksum issue

From: Brian E Carpenter <[email-masked]>
Date: Mon, 03 Aug 2009 10:15:58 +1200

Lars,

It seems to me that it would not violate the spirit of RFC2460 if
we added a rule that stacks MUST follow the RFC2460 rule by default
but MAY deviate from it for duly configured tunnel end points
in routers (where "router" is strictly as defined in section 2
of 2460 and the Note in that section). That would fully preserve
the requirement as far as hosts and applications go.

In fact, if draft-ietf-6man-node-req-bis is to become an
Applicability Statement as we've discussed, that would be a fine
document to define such a rule.

    Brian

---

From: Marshall Eubanks <[email-masked]>
Date: Sun, 2 Aug 2009 18:31:08 -0400

Dear Brian;

> the requirement as far as hosts and applications go.

This was exactly the intention of

http://tools.ietf.org/html/draft-eubanks-chimento-6man-00

We intend to rev this shortly and comments would be appreciated.

Regards
Marshall

> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------
>

---

From: Lars Eggert <[email-masked]>
Date: Mon, 3 Aug 2009 11:31:53 +0300

Hi,

> the requirement as far as hosts and applications go.

agreed, and this is what I've discussed with Dino and Joel in the  
hallway in Stockholm. My objection was about the LISP spec to  
unilaterally saying "we do something that conflicts with RFC2460"  
instead of getting community consensus for this variant.

Lars

---

From: Margaret Wasserman <[email-masked]>
Date: Mon, 3 Aug 2009 16:18:54 -0400

Hi Dino,

On Jul 30, 2009, at 4:25 PM, Dino Farinacci wrote:

>> Hi, Dino,
>>
>> On 2009-7-29, at 14:02, Dino Farinacci wrote:
>>> From a practical perspective, we prefer that a LISP encapsulator  
>>> (ITR
>>> and PTR) not incurred additional work when encapsulating packets.
>>
>> could you share some data on how much of a performance impact we're  
>> talking about here? I was under the (maybe naive) impression that  
>> checksum offloading was practically ubiquitous these days.
>
> One of the problems with IPv6 is that is so similar to IPv4 but  
> different enough to cause pain for implementations. So since we can  
> use 0 UDP checksums for IPv4 in UDP in IPv4, we would want to build  
> (or use existing hardware) that does IPv6-or-IPv4 in UDP in IPv6.
>
> Reasons being (1) cost in building hardware, (2) chip area, (3)  
> power, (4) complexity, and (5) inconsistency from IPv4.
>
> This is only for encapsulated packets. This is for the outer header  
> only.

The problem, as I see it, is that when you set the UDP checksum to 0  
in IPv4 and IPv6, you aren't doing the same thing in both cases...

In IPv4, there is an IP header checksum that protects the IP source  
and destination addresses.  So, when you turn off UDP checksums in  
IPv4, you are only turning off checksums for the UDP header and the  
UDP payload (in this case, an encapsulated packet).  You still need to  
calculate the IP header checksum.

In IPv6, the only checksum for the IPv6 source and destination  
addresses is included in the UDP checksum (via the IPv6 pseudo-header  
checksum).  So, when you turn off UDP checksums in IPv6, you are also  
turning off checksums for the IPv6 source and destination addresses.   
For cases where the payload is loss-tolerant and/or protected by a  
different mechanism, we have defined UDP-Lite, which allows you to  
calculate a checksum on the header fields only, not on the entire  
payload.  This is not much more computational expensive than computing  
the IP header checksum for IPv4-encapsulated packets.

So, we have two standards-compliant alternatives in IPv6 that would  
not require you to calculate a UDP checksum on the entire payload:

(1) UDP-Lite:  Is there a reason why UDP-Lite isn't a reasonable  
choice for LISP encapsulation?  When we looked into this for CAPWAP  
(another IP-in-UDP/IP tunneling case), we found that UDP-Lite would  
meet our needs for IPv6, so we did not need to specify the use of zero  
UDP checksums.

(2) IP-in-IPv6:  Why do you need the UDP encapsulation at all in  
IPv6?  In IPv4, you may need it for NAT traversal, but it is not clear  
that NATs will work the same way in IPv6.  Or are there other reasons  
why you need the UDP encapsulation?  When you encapsulate IP in IPv6,  
you do not have a checksum that covers the outer IP header, so you  
would need to make sure that LISP is robust against corruption of the  
source and destination addresses.  However, you do not have to worry  
as much about confusing non-LISP applications and/or corrupting their  
data, as there won't be a transport header there to misdeliver LISP  
packets to an unsuspecting application if the addresses and/or ports  
are corrupted.

I personally think it is important to consider the standards-compliant  
choices before we decide to do something non-standards-compliant.

FWIW, I agree that practical needs are very important.  I am just not  
sure that we can't meet the practical needs with a standards-compliant  
solution.

Margaret

---

From: Margaret Wasserman <[email-masked]>
Date: Mon, 3 Aug 2009 16:17:10 -0400

Hi Dino,

Margaret

---

From: "Rémi Denis-Courmont" <[email-masked]>
Date: Tue, 4 Aug 2009 00:15:42 +0300

Le lundi 3 août 2009 23:17:10 Margaret Wasserman, vous avez écrit :
(...)
> meet our needs for IPv6, so we did not need to specify the use of zero

You would expect 64-translators to convert UDP-Lite/IPv6 into UDP-Lite/IPv4 
and back. Converting UDP-Lite/IPv6-with-8-bytes-coverage into UDP/IPv4-
without-checksum sounds hackish :(

> why you need the UDP encapsulation?
(...)

Same problem. 64-translators would translate IP-in-IPv6 to IPIP and vice 
versa. IPIP does not go through NATs.

That's why I was trying to suggest defining a new UDP-Lite-like transport 
protocol that would only be allowed over IPv6 and that 64-translator would 
convert into checksum-less UDP/IPv4.

-- 
Rémi Denis-Courmont
http://www.remlab.net/

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 4 Aug 2009 13:24:20 +0200

On 30 jul 2009, at 18:49, Margaret Wasserman wrote:

> We need to consider what will happen if one of these packets is  
> received by a non-LISP node.  Are you assuming that non-LISP stacks  
> will simply throw away these packets, because they have zero (and  
> therefore invalid) UDP checksums?  That's only a good assumption if  
> we assume that LISP is the _only_ protocol that will allow the use  
> of zero checksums.

The logic behind the UDP header in LISP is to allow effective ECMP  
load balancing. In IPv6 you don't need a UDP header for that, because  
you can hash on the flow label rather than port numbers. So no need  
for a UDP header -> no 0 checksums with LISP over IPv6.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 4 Aug 2009 13:28:19 +0200

On 31 jul 2009, at 9:06, Rémi Denis-Courmont wrote:

> Sounds like this would require a third datagram protocol number, that
> is basically UDP except the checksum would only covering the IPv6 and
> transport header. Hmmph, this is just like UDP-Lite really but it  
> seems
> like there is opposition to overloading UDP-Lite. Then the 64- 
> translators
> would convert it to normal UDP/IPv4... ?

UDP ultra-lite: no length, no checksum, just port numbers.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 4 Aug 2009 13:39:17 +0200

I would be sorry to waste so much bandwidth, but it's your fault for  

>> choice for LISP encapsulation?

As I explained before, you don't need port numbers to load balance in  
IPv6 so no need for UDP here, light or otherwise.

> without-checksum sounds hackish :(

Then don't run LISP through an IPv4/IPv6 translator. It's not going to  
work anyway.

---

From: Sam Hartman <[email-masked]>
Date: Tue, 04 Aug 2009 09:08:32 -0400

>>>>> "Marshall" == Marshall Eubanks <[email-masked]> writes:

    Marshall> Dear Brian;
    Marshall> On Aug 2, 2009, at 6:15 PM, Brian E Carpenter wrote:

> Lars,
    >> 
    >> It seems to me that it would not violate the spirit of RFC2460
    >> if we added a rule that stacks MUST follow the RFC2460 rule by
    >> default but MAY deviate from it for duly configured tunnel end
    >> points in routers (where "router" is strictly as defined in
    >> section 2 of 2460 and the Note in that section). That would
    >> fully preserve the requirement as far as hosts and applications
    >> go.
    >> 

    Marshall> http://tools.ietf.org/html/draft-eubanks-chimento-6man-00

    Marshall> We intend to rev this shortly and comments would be
    Marshall> appreciated.

Margaret brought up a set of questions for LISP if it's going to send
0 UDP checksums, basically surrounding what happens when a packet on
such a tunnel is corrupted and gets received by a node that either
does or does not understand the tunneling protocol.  One of these
questions hinged on the expected behavior of receivers seeing a 0 UDP
checksum.

I suggest that your draft

1) Indicate whether receivers should be specially configured to accept
0 checsums or whether all stacks should accept 0 checksums.

2) Adapt her questions as questions that IETF specs considering this
exception need to answer to make sure that their protocol will work
correctly in this mode.

--Sam

---

From: Shane Amante <[email-masked]>
Date: Tue, 4 Aug 2009 08:24:48 -0600

Iljitsch,

On Aug 4, 2009, at 05:24 MDT, Iljitsch van Beijnum wrote:
> load balancing. In IPv6 you don't need a UDP header for that,  
> because you can hash on the flow label rather than port numbers. So  
> no need for a UDP header -> no 0 checksums with LISP over IPv6.

The above assumes that a host actually imposes a flow-label on  
outgoing IPv6 packets.  My understanding is that the flow-label isn't  
widely (if at all?) implemented, by default, in hosts; however, I  
would welcome corrections in my understanding with a list of host  
implementations that have the flow-label enabled by default.

Second, I'll check around some more, but with respect to most routers  
I'm familiar with (particularly, MPLS LSR's that will be carrying IPv6  
over MPLS inside a SP's ASN), they *also* don't use the IPv6 flow- 
label as an input key to calculate a hash to determine the outgoing  
component-link.  However, they *do* currently use the Source and  
Destination Ports of the Transport Layer protocol as input keys to  
calculate a hash-key to determine the outgoing component-link/path.   
Since I don't work at these router vendors I can't explain the  
rationale for not using the flow-label as an input key to an ECMP or  
LAG load-balancing hash; however, I would suspect that if the flow- 
label isn't well-defined nor is it widely implemented in hosts, then  
it makes no sense for router vendors to try to include it in their  
hash-algorithms for ECMP/LAG load-balancing.

So, in summary, I support UDP/IPvX encapsulation for LISP, (and other  
similar protocols that may appear as "macro-flows" in the core of a  
network), as it's the only well-known method, available today, for  
achieving load-balancing across ECMP or LAG paths that are widely  
deployed in today's networks and will continue to exist forever, (even  
with 100GE "just around the corner").

-shane

---

From: Marshall Eubanks <[email-masked]>
Date: Tue, 4 Aug 2009 10:30:11 -0400

> 0 checsums or whether all stacks should accept 0 checksums.

I personally think that receivers SHOULD require special configuration  
to
accept UDP 0 checksums. Receivers that are acting as tunnel end points  
are
likely to require special optimizations anyway.

> exception need to answer to make sure that their protocol will work

OK, thanks, will look at this.

Marshall

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 4 Aug 2009 16:52:19 +0200

On 4 aug 2009, at 16:24, Shane Amante wrote:

> outgoing IPv6 packets.

It does not, it only requires the encapsulation boxes to do this.  
Since these don't exist yet, it's trivial to specify that they do this.

I am unfortunately IPv6-less as of late, with first the university and  
then my ISP stopping their IPv6 service. As such I can't do a quick  
tcpdump to see how many sessions have a non-zero flow label. But I  
remember it's the vast majority. What I don't know is the behavior of  
the different operating systems, I'm mainly familiar with MacOS and  
FreeBSD, not sure what Linux and Windows do.

> Second, I'll check around some more, but with respect to most  
> routers I'm familiar with (particularly, MPLS LSR's that will be  
> carrying IPv6 over MPLS inside a SP's ASN), they *also* don't use  
> the IPv6 flow-label as an input key to calculate a hash to determine  
> the outgoing component-link.

Why not?

> I would suspect that if the flow-label isn't well-defined nor is it  
> widely implemented in hosts, then it makes no sense for router  
> vendors to try to include it in their hash-algorithms for ECMP/LAG  
> load-balancing.

The obvious solution in this case would be to use the flow label if  
available, and (either if the flow label isn't available or always)  
use the ports if those are available. That only leaves the situation  
where there is neither a flow label nor ports.

> So, in summary, I support UDP/IPvX encapsulation for LISP

It's still the wrong decision.

---

From: Margaret Wasserman <[email-masked]>
Date: Tue, 4 Aug 2009 10:54:15 -0400

Hi Dino.
>
>> We also need to consider the possibility that a packet will be  
>> received by a different LISP node than the one for which it was  
>> intended, or that it will arrive at the correct LISP destination  
>> with the wrong source address in the external header.  What happens  
>> in those cases?
>
> This doesn't happen since the outer IP header is checksum.

Not if the outer header is IPv6, as the IPv6 header does not include a  
header checksum.  This is why it is more important in IPv6 for the UDP  
header to have a checksum that includes the IP pseudo header checksum.

If you have been missing this point, it is no wonder that you did not  
understand my concerns about simply setting the UDP checksum to zero.

Margaret

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 04 Aug 2009 11:08:31 -0400

It has become clear with the passage of time that the description of the 
flow label in the original IPv6 specs served only to convince everyone 
not to use that field for anything.  Even now, no one is sure what to do 
with it.

To propose that encapsulators should use this field to mark the flows 
would seem reasonable.  The problem is that since that was not done in 
the first place, as I understand it, router vendors concluded that the 
only sensible thing to do was to ignore the field, and do load balancing 
on ports just like we did on IPv4.

Saying that we should have done it differently requires that the vendors 
be accurate fortune tellers, able to realize that the fields was going 
to turn out to be useful, and able to guess what usage they could safely 
make that would correctly match an eventual usage :-)

And given the deployment assumptions, if we hope for LISP to be usable 
over IPv6, it can not depend for correct operation ona router feature 
that is not yet being delivered.

(I will happily eat crow and apologize if it turns out routers already 
do what Iljitsch is asking for.)

Yours,
Joel

Iljitsch van Beijnum wrote:
> It does not, it only requires the encapsulation boxes to do this. Since 
> these don't exist yet, it's trivial to specify that they do this.
> 
>> over MPLS inside a SP's ASN), they *also* don't use the IPv6 
>> flow-label as an input key to calculate a hash to determine the 
>> outgoing component-link.
> 
> Why not?
> 
>> widely implemented in hosts, then it makes no sense for router vendors 
>> to try to include it in their hash-algorithms for ECMP/LAG 
>> load-balancing.
> 
> available, and (either if the flow label isn't available or always) use 
> the ports if those are available. That only leaves the situation where 
> there is neither a flow label nor ports.
> 
> _______________________________________________
> lisp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lisp
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 4 Aug 2009 17:46:29 +0200

On 4 aug 2009, at 17:08, Joel M. Halpern wrote:

> And given the deployment assumptions, if we hope for LISP to be  
> usable over IPv6, it can not depend for correct operation ona router  
> feature that is not yet being delivered.

I am really starting to lose my patience here!!!

Even IF existing implementations don't put any value in the flow  
label, the LISP spec can specify that this MUST be done for LISP and  
then there will be flow labels in all LISP packets.

Obviously it will take some time for the router implementations to  
start hashing on the flow label, but then again, it will also take  
some time for the traffic from a single ITR to a single ETR (or the  
other way around, whatever) to need more than 1 Gbps so the load  
balancing issue becomes more than academic.

Come on, people! If we can't even get stuff right in fairly green  
fields, how can we expect to get ANYTHING right EVER??

---

From: Sam Hartman <[email-masked]>
Date: Tue, 04 Aug 2009 12:47:09 -0400

>>>>> "Joel" == Joel M Halpern <[email-masked]> writes:

    Joel> It has become clear with the passage of time that the
    Joel> description of the flow label in the original IPv6 specs
    Joel> served only to convince everyone not to use that field for
    Joel> anything.  Even now, no one is sure what to do with it.

    Joel> To propose that encapsulators should use this field to mark
    Joel> the flows would seem reasonable.  The problem is that since
    Joel> that was not done in the first place, as I understand it,
    Joel> router vendors concluded that the only sensible thing to do
    Joel> was to ignore the field, and do load balancing on ports just
    Joel> like we did on IPv4.

IPV6 deployment is fairly low.  It seems likely that most of the
equipment that will be used to forward IPV6 traffic has not yet been
built, and that while we definitely need to work with today's
equipment, I think it is safe to say that 10, 15, 20 years from now,
the things we optimize for will be different.  so, if we find new best
practices for encapsulators today, we should write them down, try them
out and build consensus.

That way, 3-4 years from now we may be in a position to recommend
forwarders do something different and 10-15 years from now, rely on
that

Naturally, any such discussion should have a sufficiently broad
audience to be useful.

---

From: Rémi Després <[email-masked]>
Date: Tue, 4 Aug 2009 19:57:07 +0200

Le 4 août 09 à 17:08, Joel M. Halpern a écrit :

> It has become clear with the passage of time that the description  
> of the flow label in the original IPv6 specs served only to  
> convince everyone not to use that field for anything.  Even now, no  
> one is sure what to do with it.

RFC 3697, which is on standards track, specifies how to use flowlabels.
I would personally have no objection to it's deprecation, but it's here.

RD

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 4 Aug 2009 20:39:04 +0200

> I would personally have no objection to it's deprecation, but it's  

Since apparently nobody looks at it today, I'm tempted to attempt to  
scavenge some header bits there. For re-ecn we need one bit and I  
could use three or so as a path selector for path selection in  
multipath TCP...

---

From: Margaret Wasserman <[email-masked]>
Date: Tue, 4 Aug 2009 15:01:01 -0400

> would convert it to normal UDP/IPv4... ?

Why do you think that this would be "overloading UDP-Lite"?  And what  
is the opposition?

We used UDP-Lite in a very similar situation in CAPWAP for very  
similar reasons.  We needed to be able to checksum-protect the outer  
headers in an IP-in-IPv6/UDP(-ish) encapsulation, and we did not want  
the encapsulator to be required to (re-)checksum the entire payload.   
IMO, that is a very sound reason to use UDP-Lite, and one that has   
IETF precedent in at least one standards-track RFC.

Margaret

---

From: "Rémi Denis-Courmont" <[email-masked]>
Date: Tue, 4 Aug 2009 22:13:27 +0300

Le mardi 4 août 2009 22:01:01 Margaret Wasserman, vous avez écrit :
> Why do you think that this would be "overloading UDP-Lite"?

Gorry Fairhurst does, not me. UDP-Lite was specified for damage-tolerant 
payloads, not to avoid redumdant checksums. Personally, I don't care.

> And what is the opposition?

The only (valid) argument that I remember against UDP-Lite was lack of 
compatibility with IPv4 NAT/firewall in case the flow would go through a 
NAT64. This was a comment from Dave Thaler in SF. Admittedly, it was about 
AMT, not LISP.

-- 
Rémi Denis-Courmont
http://www.remlab.net/

---

From: Margaret Wasserman <[email-masked]>
Date: Tue, 4 Aug 2009 15:37:58 -0400

> We intend to rev this shortly and comments would be appreciated.

If you do rev this document, I would like to see:

(1) An explanation of the difference in applicability between this  
mechanism
       and UDP-Lite.

(2) A specific applicability statement for when this mechanism can  
(and can't)
       be used.  Would we be allowing this mechanism _only_ for cases  
where
       IP is tunneled inside UDP/IP?   What about the AMT case?

(3) A section listing considerations that must be documented for  
protocols
       that use this mechanism, including:

     (3a) What happens when the destination IP address is corrupted in  
transit?
             (Note:  This isn't a concern in IPv4, as the IP header  
checksum will result
             in this packet being discarded by the receiving IP stack).

         (3a.i) What if a node that does not implement this protocol  
receives the
                    packet?
         (3a.ii) What if a node that does implement this protocol  
receives a packet
                    destined for another node?

     (3b) What happens when the source IP address is corrupted in  
transit?
             (Note:  This isn't a concern in IPv4, as the IP header  
checksum will result
             in this packet being discarded by the receiving IP stack).

	(3b.i) What happens when a node that does not implement this protocol
                    receives the ICMP "port unreachable" error?
         (3b.ii) What happens when a node that does implement this  
protocol
                    receives an ICMP "port unreachable" error for a  
packet it didn't send?

     (3c) What happens if one or both of the UDP ports are corrupted  
in transit?
             (Note:  This can happen in IPv4 in the zero checksum  
case, too, but not
             with UDP checksums turned on and/or UDP-Lite).

     (3d) What types of middleboxes does the protocol need to cross  
(routers,
              NAT boxes, firewalls, etc.), and how will those  
middleboxes deal with
              these packet

I don't really have enough knowledge to answer these questions for  
LISP.  I
tried to go down this path in an earlier message, and Dino didn't  
understand what
I was saying, but I'll try to do it once again.  Please correct me if  
I am missing
something.

I am making the assumption that (unlike AMT), all LISP packets will be  
sent to
a specific UDP port assigned to LISP.  Also, I am making the  
assumption that
if we write this document, some other protocols (besides LISP) will  
make use of
it, so some IP stacks on non-LISP nodes will process these packets.

(3a.i)  If a node that doesn't implement LISP receives this packet, it  
may be
            passed up to UDP (if zero checksums are supported).  At  
that point,
            though, it will be thrown away because there is no process  
listening
            on the LISP port and an ICMPv6 "port unreachable" error  
will be
            generated.  (This is different than the IPv4 case, where  
the packet
            will never make it to UDP).

(3a.ii) If a node that does implement LISP receives this packet, it  
will be
            sent to the LISP process, which will recognize that it is an
            encapsulated LISP packet.  What then?  (Note:  Again, this
            wouldn't happen in IPv4, as the packet would be discarded by
            the receiving IP stack.).

(3b.i) If a node that doesn't implement LISP receives an ICMP port
           unreachable for the LISP port, it may have a process that is
           using the local port from the original LISP packet.  Will  
that
           process receive the destination unreachable?  Or will the
           message be thrown away because the destination port doesn't
           match?  I think this might depend on how the process was
           bound to its socket, or something.  Can someone help me
           out here?  (Note:  This won't happen in IPv4, because the
           packet would have been discarded by IP, not by UDP).

(3b.ii) If a node that does implement LISP receives an ICMP port
            unreachable, it still might not have a process that is using
            the corresponding source port (right?).  In that case, I  
think
            that this is the same as 3b.i...  Would another process
            receive this error?  And, if so, how would it respond?

(3c) If the ports were corrupted in transit, we might get packets
         delivered to the wrong process (on the right machine)
         and/or responses or errors sent to the wrong process (on
         the right machine).  I think this all would have been covered
         by the above cases.

(3d) I don't know how middleboxes will deal with this...  What
         do IPv6 routers do today with zero-checksum UDP packets?
         What other IPv6 middleboxes exist today, and what would
         they do?

Personally, I think this is something we would need to understand
pretty fully before we could decide that turning off IPv6 UDP checksums
is a superior choice to using IP-in-IPv6 encapsulation (no UDP),
IP-in-IPv6/UDP-Lite encapsulation or UDP with checksums.

These are also areas that we should explore in a general sense before
we decide to publish Marshall's document as an alternative to using
IP-in-IPv6 encapsulation, UDP-Lite, or UDP with checksums.

Margaret

---

From: Margaret Wasserman <[email-masked]>
Date: Tue, 4 Aug 2009 15:42:12 -0400

> without-checksum sounds hackish :(

Are we actually expecting 64-translators to exist between LISP routers?

In the more general sense, this is something you need to deal with,  
because there are things out there that use UDP-Lite.  I don't see the  
transform you have suggested as any worse than just dropping the IP  
header checksum when converting from IPv4 to IPv6 in cases where it is  
not covered by a transport layer checksum.

> versa. IPIP does not go through NATs.

So, now we're running encapsulation protocols behind an IPv4 NAT _and_  
a 64-translator?  Ick.
> convert into checksum-less UDP/IPv4.

Why is the current UDP-Lite protocol not okay for this?

Margaret

---

From: Margaret Wasserman <[email-masked]>
Date: Tue, 4 Aug 2009 15:44:14 -0400

> UDP ultra-lite: no length, no checksum, just port numbers.

This is sort of like the UDP-TT proposal, except for the part where  
UDP-TT overloads the UDP next-header type, hard-codes the length to 8  
and includes a checksum of the IP and UDP headers (which is fixed  
value per flow)...

Margaret

---

From: "Rémi Denis-Courmont" <[email-masked]>
Date: Tue, 4 Aug 2009 22:58:48 +0300

> Are we actually expecting 64-translators to exist between LISP routers?

No idea - apparently not. But it seemed like it would be the case for AMT (so 
was it implied at the SF IPv6 wg meeting).

> not covered by a transport layer checksum.

The transform is bad. What if you actually want to convert UDP-Lite/IPv6 into 
UDP-Lite/IPv4? This seems like the natural thing to do. Then how would NAT64s 
know whether to translate UDP-Lite/IPv6 with 8-bytes coverage into checksum-
less UDP/IPv4 or into UDP-Lite/IPv4 with 8-bytes coverage?

If we actually except valid use cases of checksum-less UDP/IPv4 to hit NAT64s, 
then I'd argue a new over-IPv6-only transport protocol is needed. UDP-Lite 
does not fit precisely because it already exists over IPv4. It would not be 
the first non-IPv4 next header number.

Sure, if we expect that neither LISP, nor AMT, nor any other user of checksum-
less UDP/IPv4 to be protocol-translated, then this is a non-issue. Then LISP 
and AMT and whatever can use UDP-Lite.

> So, now we're running encapsulation protocols behind an IPv4 NAT _and_

Well, I definitely expect that, if NAT64 become widely deployed, we will see 
IPv4 firewalls or even IPv4 NATs on the IPv4 side of those.

> Why is the current UDP-Lite protocol not okay for this?

I assume that NAT64s are used. Not that I plan to use them myself. But that 
"reason" was invoked against UDP-Lite when the troll started in IPv6 wg 
meeting in San Francisco, when Marshall presented his draft.

The reasoning was that UDP-Lite/IPv6 would be 64-translated to UDP-Lite/IPv4, 
yet UDP-Lite/IPv4 is too poorly supported by the IPv4 infrastructure (firewall 
and NATs). If everybody now agrees that this is a non-issue, fine with me.

-- 
Rémi Denis-Courmont
http://www.remlab.net/

---

From: Marshall Eubanks <[email-masked]>
Date: Tue, 4 Aug 2009 16:57:59 -0400

Dear Margaret;

Thank you for this long list of issues/questions. They
will be addressed.

Regards
Marshall

On Aug 4, 2009, at 3:37 PM, Margaret Wasserman wrote:

> I was saying, but I'll try to do it once again.  Please correct me  
> if I am missing
> something.
>
> I am making the assumption that (unlike AMT), all LISP packets will  
> be sent to
> (3a.i)  If a node that doesn't implement LISP receives this packet,  
> it may be

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 05 Aug 2009 10:09:52 +1200

On 2009-08-05 06:39, Iljitsch van Beijnum wrote:
> scavenge some header bits there. For re-ecn we need one bit and I could
> use three or so as a path selector for path selection in multipath TCP...

That's not so easy. The bits aren't up for grabs and they aren't microcoded
like the Traffic Class bits (which is why ECN was possible in the first
place). It may be entirely possible to devise a use case that falls within
RFC3697 where you could encode some bits; that RFC was very carefully
written to allow a very wide variety of uses. In fact the only real
rule, when you come down to it, is that the 20 bits must be delivered
exactly as sent. Pretty much everything else is allowed if you use the
MAYs and SHOULDs creatively.

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 05 Aug 2009 10:24:28 +1200

Hi Joel,

> not to use that field for anything.  Even now, no one is sure what to do

If you're referring to the lame appendix to RFC2460, that's exactly
why we wrote RFC3697. My understanding is that some stacks do set the
flow label according to the SHOULD in RFC3697, but I'm not aware
of any deployed use cases (such as load balancing routers). The gap
is not in the basic specs but in exploitation. It's very similar to
what happened with the IPv4 TOS byte, and that took 20 years before
we (a) defined it properly and (b) began to see limited deployment,
as corporate VoIP took off.

My expectation has always been that exploitation of the flow label
would stay on the back burner until IPv6 deployment reached a
significant level, because there isn't much benefit in optimising
flow handling for <1% of the traffic. So I don't really see anything
odd in the current situation.

That said, I can't see any reason why ITRs and ETRs can't use
the flow label among themselves, in a way completely compatible
with RFC3697. But if their hardware engines can't include the
flow label in the n-tuple, that's a problem.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 05 Aug 2009 14:09:52 +1200

On 2009-08-05 11:26, Vince Fuller wrote:
> The issue isn't whether the "hardware engines can't include the flow label
> in the n-tuple" on the ITRs/ETRs; since LISP is implemented on ITRs and ETRs,
> we have no problem specifying their behavior.
> 
> The issue is how to effect load-splitting across Link Aggregation Groups
> (LAGs) in all of the non-LISP-aware transit routers on the Internet through
> which LISP-encapsulated traffic will flow.
> 
> Current operational reality is that the installed base of transit routers
> on the Internet uses a hash of source/dest addres/port to split traffic
> across LAGs so LISP uses an encapsulation that is compatible with that
> reality.

Sure. But IMHO the LISP design should not be restrictive. It should be possible
by design to change the fields used for the hash if the operational reality
changes in the future. (In fact, one of the motivations for the exact choices
made in RFC3967 was to make it possible to include the flow label in such
a hash without knowing what sort of usage, if any, was being made of the
flow label. And I see no particular issue with a network where some
LAG-aware routers do include the flow label in the hash and others don't.)

> 
> Specifying some alternate reality and hoping that the operational world will
> modify its behavior to match doesn't seem very practical, particularly since
> one of LISP's virtues is that it requires no changes to the transit routing
> system.

Well understood. But future-proofing is rarely an error.

   Brian

---

From: Vince Fuller <[email-masked]>
Date: Tue, 4 Aug 2009 16:26:48 -0700

--Vince

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 04 Aug 2009 22:51:50 -0400

The problem is not what the ITRs and ETRs use the field for.  They could 
/ can use the field.
The problem is that the UDP header was introduced specifically so that 
different flows would be different in a place that the routers would 
look when doing ECMP calculations.
And the router to date do not use the flow label.
Preliminary indications are that routers also won't use UDP-lite, 
because it is a different protocol, and they don't know it has port numbers.

The simple path forward is to allow UDP with 0 checksum.
If that is incorrect, then finding the correct path forward is going to 
be hard.

Joel

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 5 Aug 2009 13:44:19 +0200

On 5 aug 2009, at 1:26, Vince Fuller wrote:

> modify its behavior to match

Isn't that the business the IETF is in?

> doesn't seem very practical, particularly since

If we get router vendors on board with flow label ECMP for IPv6 now,  
by the time we have those multi-gigabit IPv6 flows between a single  
ITR/ETR pair, we'll be in reasonable shape.

I would be very interested to hear what the status of ECMP for IPv6 is  
today, and especially if the implementations that can do this on the 5- 
tuple today are flexible enough to be able to look at the flow label  
without hardware changes.

The problem with all of this in IPv6 is that there can be routing and  
fragmentation headers that make the port numbers appear in different  
places. When I tested this in 2005, the Cisco routers that I used  
wouldn't accommodate for this in their access lists, I so I can easily  
see IPv6 ECMP not looking at the 5-tuple but the 3-tuple, not be able  
to look at the 5-tuple consistently or simply not exist at all.

---

From: Margaret Wasserman <[email-masked]>
Date: Wed, 5 Aug 2009 10:12:58 -0400

Hi Joel,

On Aug 4, 2009, at 10:51 PM, Joel M. Halpern wrote:

> The problem is not what the ITRs and ETRs use the field for.  They  
> could / can use the field.
> The problem is that the UDP header was introduced specifically so  
> that different flows would be different in a place that the routers  
> would look when doing ECMP calculations.
> If that is incorrect, then finding the correct path forward is going  
> to be hard.

We're talking about the ECMP calculations in _IPv6_ routers here,  
right?  Do you really believe that enough IPv6 routers have shipped  
with this sort of ECMP behavior in _hardware_ that we need to consider  
that legacy deployment?  I'm somewhat skeptical that this could be the  
case...  If we're going to make design compromises to deal with  
currently-deployed hardware, I'd like to see some evidence that this  
hardware actually exists in numbers that are worth considering.

Note, I'm not asking how many routers have been shipped that support  
IPv6, nor am I asking whether those routers support ECMP (in  
software).  I'm not even asking whether some router vendors have taped  
out versions of silicon that are so inflexible that they will only  
support IPv6 ECMP using UDP ports (not the flow label or UDP-Lite)...   
What I am asking is whether IPv6 routers containing that silicon exist  
in real-world deployments in large enough numbers that they should be  
considered in our design choices.

Thoughts?

Margaret

---

From: Margaret Wasserman <[email-masked]>
Date: Wed, 5 Aug 2009 10:16:27 -0400

Hi Joel,

Thoughts?

Margaret

---

From: Pekka Savola <[email-masked]>
Date: Wed, 5 Aug 2009 17:24:12 +0300 (EEST)

On Wed, 5 Aug 2009, Margaret Wasserman wrote:
> Do you really believe that enough IPv6 routers have shipped with 
> this sort of ECMP behavior in _hardware_ that we need to consider 
> that legacy deployment? I'm somewhat skeptical that this could be 
> the case...  If we're going to make design compromises to deal with 

FWIW, Juniper docs say the following.  Not sure what else they could 
do in hardware or what are the caveats.

========================
By default, IP version 6 (IPv6) packets are automatically 
load-balanced based on the following Layer 3 and Layer 4 information:

     * Source IP address
     * Destination IP address
     * Protocol
     * Source port number
     * Destination port number
     * Incoming interface index
     * Traffic class

http://www.juniper.net/techpubs/en_US/junos9.5/information-products/topic-collections/config-guide-policy/policy-configuring-per-packet-load-balancing.html

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Shane Amante <[email-masked]>
Date: Wed, 5 Aug 2009 08:29:31 -0600

Margaret,

On Aug 5, 2009, at 08:12 MDT, Margaret Wasserman wrote:
>> If that is incorrect, then finding the correct path forward is  
>> going to be hard.
>
> with this sort of ECMP behavior in _hardware_ that we need to  
> consider that legacy deployment?  I'm somewhat skeptical that this  
> could be the case...  If we're going to make design compromises to  
> deal with currently-deployed hardware, I'd like to see some evidence  
> that this hardware actually exists in numbers that are worth  
> considering.
>
> software).  I'm not even asking whether some router vendors have  
> taped out versions of silicon that are so inflexible that they will  
> only support IPv6 ECMP using UDP ports (not the flow label or UDP- 
> Lite)...  What I am asking is whether IPv6 routers containing that  
> silicon exist in real-world deployments in large enough numbers that  
> they should be considered in our design choices.

Take a look at the following URL:
http://www.sixxs.net/faq/connectivity/?faq=ipv6transit
(Note, I can't vouch for the accuracy of the entire list, but it's  
about the best publicly available list I've seen).

That list includes some very large networks.

-shane

---

From: Sam Hartman <[email-masked]>
Date: Wed, 05 Aug 2009 11:01:17 -0400

>>>>> "Shane" == Shane Amante <[email-masked]> writes:

    Shane> Take a look at the following URL:
    Shane> http://www.sixxs.net/faq/connectivity/?faq=ipv6transit
    Shane> (Note, I can't vouch for the accuracy of the entire list,
    Shane> but it's about the best publicly available list I've seen).

    Shane> That list includes some very large networks.

Thanks for the data!  Naturally that's only part of the answer to
Margaret's question.  We know which companies offer native IPV6
transit service.
We don't know:

1) How much V6 traffic they have 2) Whether they have V6
infrastructure that would scale to their current V6 traffic or say
their current V4 traffic 3) How much is done in hardware for their V6
infrastructure.  4) How much of their network supports V6/how
widespread their V6 works.  Any information you can provide on these
sorts of questions would be very useful.

---

From: Shane Amante <[email-masked]>
Date: Wed, 5 Aug 2009 10:50:32 -0600

Sam,

> sorts of questions would be very useful.

I think question #1 is a little impractical, given that data is highly  
sensitive/confidential to each network.  However, one likely can  
Google/Bing around and find public Exchange points with graphs that  
show current v6 traffic (as compared to v4 traffic).  Of course, I  
would highlight that these are *public* exchange points, thus they  
will not show v6 traffic going through private peering connections as  
well as staying Intra-provider, (i.e.: customer-to-customer inside a  
single provider) -- see first sentence about sensitivity/ 
confidentiality.  That said, there have been presentations at the last  
couple of IETF's (I forget if it was in 'behave' or 'v6ops' WG's, or  
both) surrounding Free Telecom's rollout of '6rd' that show their v6  
traffic, which has some astounding growth and traffic rates.   
Regardless, one should consider the following graphs a absolute  
"floor" of v6 traffic and there is definitely more, in absolute terms,  
v6 traffic than is visible in these graphs.
Here's one example from AMS-IX showing v6 traffic:
http://www.ams-ix.net/technical/stats/sflow/?type=ipv6
And, here are graphs showing Total traffic load, (I believe this  
includes the v6 traffic in the graph above, but am not 100% certain on  
that point):
http://www.ams-ix.net/technical/stats/
According to these graphs, v6 traffic is just a fraction of v4  
traffic, but it is definitely growing and, FWIW, it's a lot more than  
Inter-AS multicast traffic as seen by other graphs at the AMS-IX Web  
site.  :-/

With respect to #2, SP's have been mandating that they only buy v6- 
capable HW for the last /several years/ as part of the normal growth/ 
replacement cycle of network equipment.  So, yes, this equipment  
should scale to support v6 traffic at the same, (or nearly the same),  
rates as v4 traffic today.

#3: All v6 forwarding is done in HW for larger, "PE" or "P" class  
devices, which are the subject of this thread.

#4: Depends on where the carrier is at in their deployment of v6.   
However, I would refer you back to question #2, which is even if it's  
not technically offered at a given location today, it is only a matter  
of time given that the HW is already deployed to do so (and, has been  
deployed for several years).

To bring this back up a level, while it's /possible/ to encourage  
vendors to adopt the IPv6 flow-label as input-keys to their hash- 
calculations for LAG/ECMP, it takes [a lot] of time to see that  
materialize in the field.  Practically, you're probably looking at  
somewhere between, at best, a 3  - 5 year window, before it will  
actually appear in live, production networks, given that it has to be  
prioritized for development at the vendor, tested, released in  
software, then re-tested by the SP and, finally, deployed.  That's not  
an "easy" process that happens in the blink-of-an-eye.  That's not to  
say that we (SP's) should not "encourage" vendors to do this, anyway,  
(we are/will) however if LISP (or other protocols like it that depend  
on tunneling) quickly gain traction, we need a way to deal with these  
traffic flows in our networks today, without telling customers: "turn  
off protocol <FOO>, because we can't deliver your packets".  Perhaps  
one way to satisfy the parties in this conversation would be something  
along the following lines:
- LISP, and other protocols that wish to use tunneling, adopt UDP-lite  
(or UDP w/ 0 checksums) as a MUST for near-term deployment;
- LISP, and other protocols that wish to use tunneling, adopt IP-in- 
IPv6 tunneling with a flow-label required in the outer IPv6 header as  
a "SHOULD" for medium- to long-term deployments.
... assuming vendors successfully adopt the IPv6 flow-label as input- 
keys in their hash calculations at some point in the future, we come  
back and deprecate the UDP/IPv6 tunnel method and elevate IP-in-IPv6  
w/ flow-labels to a MUST.

It likely wouldn't hurt to go back and "tighten up" RFC 3697, as well,  
in order that it offers more prescriptive behavior in several places,  
not least of which would be:
1)  How to deal with regular IP-in-IPv6 encapsulation, (i.e.: likely  
by creating a flow-label based on a "hash" of the incoming, to-be- 
encapsulated L3 & L4 header fields), since the RFC only discusses  
IPsec tunneling; and,
2)  Removing other "gems" (or clarifying them) like the second  
sentence in the following:
---cut here---
IPv6 nodes MUST NOT assume any mathematical or other properties of the  
Flow Label
values assigned by source nodes.  Router performance SHOULD NOT be  
dependent on the
distribution of the Flow Label values. Especially, the Flow Label bits  
alone make
poor material for a hash key.
---cut here---
... specifically, if "router performance" is meant to imply the  
behavior of load-balancing on ECMP/LAG paths, then router performance  
*is* heavily dependent on the distribution of flow-label values ...

I've said all I'm going to say on this thread ...

-shane

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 5 Aug 2009 13:34:24 -0400

On Wed, Aug 5, 2009 at 12:50 PM, Shane Amante<[email-masked]> wrote:
> sensitive/confidential to each network.  However, one likely can Google/Bing
> around and find public Exchange points with graphs that show current v6
> traffic (as compared to v4 traffic).  Of course, I would highlight that
> these are *public* exchange points, thus they will not show v6 traffic going
> through private peering connections as well as staying Intra-provider,
> (i.e.: customer-to-customer inside a single provider) -- see first sentence
> about sensitivity/confidentiality.  That said, there have been presentations
> at the last couple of IETF's (I forget if it was in 'behave' or 'v6ops'
> WG's, or both) surrounding Free Telecom's rollout of '6rd' that show their
> v6 traffic, which has some astounding growth and traffic rates.  Regardless,
> one should consider the following graphs a absolute "floor" of v6 traffic
> and there is definitely more, in absolute terms, v6 traffic than is visible
> in these graphs.
> And, here are graphs showing Total traffic load, (I believe this includes
> the v6 traffic in the graph above, but am not 100% certain on that point):
> According to these graphs, v6 traffic is just a fraction of v4 traffic, but
> it is definitely growing and, FWIW, it's a lot more than Inter-AS multicast
> traffic as seen by other graphs at the AMS-IX Web site.  :-/
>
> With respect to #2, SP's have been mandating that they only buy v6-capable
> HW for the last /several years/ as part of the normal growth/replacement
> cycle of network equipment.  So, yes, this equipment should scale to support
> v6 traffic at the same, (or nearly the same), rates as v4 traffic today.
>
> #3: All v6 forwarding is done in HW for larger, "PE" or "P" class devices,
> which are the subject of this thread.
>
> #4: Depends on where the carrier is at in their deployment of v6.  However,
> I would refer you back to question #2, which is even if it's not technically
> offered at a given location today, it is only a matter of time given that
> the HW is already deployed to do so (and, has been deployed for several
> years).
>
>
> To bring this back up a level, while it's /possible/ to encourage vendors to
> adopt the IPv6 flow-label as input-keys to their hash-calculations for
> LAG/ECMP, it takes [a lot] of time to see that materialize in the field.
>  Practically, you're probably looking at somewhere between, at best, a 3  -
> 5 year window, before it will actually appear in live, production networks,

s/networks/hardware-shipped-by-vendors/

You may see 2-3 year cycle on new asics for this feature to appear...
given 1-2 years for haggling/bugs/blah it's safe to say 3-5 yrs before
hardware is on the shelf to purchase.

Look at jason schiller's presos at IETF/RAWS/RRG that show 5-7 year
lifecycle of equipment in a network, so add some years before shipped
and deployed hardware is in general coverage in a network. Call it 10+
before it's ubiquitous...

> given that it has to be prioritized for development at the vendor, tested,
> released in software, then re-tested by the SP and, finally, deployed.

and hopefully deployed in the hardware, since that's where this
decision is made today (in v4 and v6 by 5-tuple). Software routers are
dead in any large SP for an serious workload. (P or PE devices)

>  That's not an "easy" process that happens in the blink-of-an-eye.  That's
> not to say that we (SP's) should not "encourage" vendors to do this, anyway,
> (we are/will) however if LISP (or other protocols like it that depend on
> tunneling) quickly gain traction, we need a way to deal with these traffic
> flows in our networks today, without telling customers: "turn off protocol
> <FOO>, because we can't deliver your packets".  Perhaps one way to satisfy
> the parties in this conversation would be something along the following
> lines:
> - LISP, and other protocols that wish to use tunneling, adopt UDP-lite (or
> UDP w/ 0 checksums) as a MUST for near-term deployment;
> - LISP, and other protocols that wish to use tunneling, adopt IP-in-IPv6
> tunneling with a flow-label required in the outer IPv6 header as a "SHOULD"
> for medium- to long-term deployments.
> ... assuming vendors successfully adopt the IPv6 flow-label as input-keys in
> their hash calculations at some point in the future, we come back and
> deprecate the UDP/IPv6 tunnel method and elevate IP-in-IPv6 w/ flow-labels
> to a MUST.
>
> It likely wouldn't hurt to go back and "tighten up" RFC 3697, as well, in
> order that it offers more prescriptive behavior in several places, not least
> of which would be:
> 1)  How to deal with regular IP-in-IPv6 encapsulation, (i.e.: likely by
> creating a flow-label based on a "hash" of the incoming, to-be-encapsulated
> L3 & L4 header fields), since the RFC only discusses IPsec tunneling; and,
> 2)  Removing other "gems" (or clarifying them) like the second sentence in
> the following:
> ---cut here---
> IPv6 nodes MUST NOT assume any mathematical or other properties of the Flow
> Label
> values assigned by source nodes.  Router performance SHOULD NOT be dependent
> on the
> distribution of the Flow Label values. Especially, the Flow Label bits alone
> make

'flow label bits alone make a poor material for a hash key'... isn't
this the reverse of saying that we'll (operators) require vendors to
use flow-label for hashing on ECMP/LAG? If so, then... I don't think
flow-label's going to cut it.

-chris

> ... specifically, if "router performance" is meant to imply the behavior of
> load-balancing on ECMP/LAG paths, then router performance *is* heavily
> dependent on the distribution of flow-label values ...
>

---

From: Sam Hartman <[email-masked]>
Date: Wed, 05 Aug 2009 13:43:24 -0400

>>>>> "Shane" == Shane Amante <[email-masked]> writes:
    Shane> With respect to #2, SP's have been mandating that they only
    Shane> buy v6- capable HW for the last /several years/ as part of
    Shane> the normal growth/ replacement cycle of network equipment.
    Shane> So, yes, this equipment should scale to support v6 traffic
    Shane> at the same, (or nearly the same), rates as v4 traffic
    Shane> today.

I don't see how this follows at all.  I can sell you a box that
supports V6 and that provides much lower performance for V6 than V4,
but that still meets your requirement of supporting V6.

You may be trying to tell me that SP's are unwilling to buy such boxes
and explicitly want boxes where V6 and V4 performance is the same.  If
that is what you are saying, it is very interesting, but it goes far
beyond simply saying that SPs are buying V6-supporting devices.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 05 Aug 2009 13:42:16 -0400

Reading this discussion, there seem to be a small number of practical 
choices.

If the vendor hardware that is / will be handling IPv6 can handle the 
flow label as part of the ECMP / LAG calcualtion, than an I-D / 
direction to use the flow label seems sensible.  (This is about what 
will be used, not what off-the-shelf parts might support in some other 
box.)  While hardware upgrades take a long time, software upgrades are 
easier, particularly if it makes the operators traffic handling work better.

Conversely, if the hardware we have to count on will not be able to 
handle flow labels for ECMP / LAG hash calculations, then it probably 
also won't be able to handle alternate protocols (like UDP-lite).  As 
such, everything that needs ECMP / LAG handling will need to have TCP or 
UDP encapsulations, and given that many of those are protected, we will 
want to allow UDP with 0 checksum.

Yours,
Joel

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 5 Aug 2009 14:06:50 -0400

> and explicitly want boxes where V6 and V4 performance is the same.  If

an SP can't know a-priori what protocol traffic is using on a transit
device (presuming it supports both popular protocols of course). So,
the only viable answer for an SP today who has dual-stacked their
network is to demand performance parity.

(which I think all large sp's are doing, which is one of the reasons
that large SP deployments of v6 aren't moving along at a pace that
some folks want. There are still significant numbers of devices not
capable of doing line-rate v6 forwarding.)

-chris

---

From: Margaret Wasserman <[email-masked]>
Date: Wed, 5 Aug 2009 14:33:11 -0400

Hi Shane,

On Aug 5, 2009, at 12:50 PM, Shane Amante wrote:
> actually appear in live, production networks, given that it has to  
> be prioritized for development at the vendor, tested, released in  
> software, then re-tested by the SP and, finally, deployed.  That's  
> not an "easy" process that happens in the blink-of-an-eye.  That's  
> not to say that we (SP's) should not "encourage" vendors to do this,  
> anyway, (we are/will) however if LISP (or other protocols like it  
> that depend on tunneling) quickly gain traction, we need a way to  
> deal with these traffic flows in our networks today, without telling  
> customers: "turn off protocol <FOO>, because we can't deliver your  
> packets".

ECMP of LISP flows is an optimization that only matters when there is  
a large amount of LISP traffic, right?  Do you think there is a  
reasonable chance that LISP would be widely-enough to deployed to  
require this optimization in less than 3-to-5 years?

> Perhaps one way to satisfy the parties in this conversation would be  
> something along the following lines:
> - LISP, and other protocols that wish to use tunneling, adopt UDP- 
> lite (or UDP w/ 0 checksums) as a MUST for near-term deployment;
> IPv6 tunneling with a flow-label required in the outer IPv6 header  
> as a "SHOULD" for medium- to long-term deployments.

I would agree to this statement if one of two things were to happen:   
(1) we were to remove "(or UDP w/0 checksums) -- making UDP-Lite a  
MUST in the near term", or (2) we were to satisfy ourselves that using  
zero checksums will not represent an operational problem for LISP, or  
for other applications that need to co-habitate with LISP, and we were  
to add a normative dependency on a document (like Marshall's, with  
edits) that would allow the use of UDP w/0 checksums over IPv6 in some  
cases, with LISP being an example of a case where this was appropriate.

Margaret

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 5 Aug 2009 14:54:21 -0400

On Wed, Aug 5, 2009 at 2:33 PM, Margaret Wasserman<[email-masked]> wrote:
>> To bring this back up a level, while it's /possible/ to encourage vendors
>> to adopt the IPv6 flow-label as input-keys to their hash-calculations for
>> <FOO>, because we can't deliver your packets".
>
> ECMP of LISP flows is an optimization that only matters when there is a
> large amount of LISP traffic, right?  Do you think there is a reasonable
> chance that LISP would be widely-enough to deployed to require this
> optimization in less than 3-to-5 years?

in the extreme it's only needed for +1g flows, but there are lots of
places where this may be necessary across smaller links ecmp'd
together... Planning for no ECMP is not a plan that's sensible. There
must be a way to sensibly hash traffic across more than one path in a
network (not just local ecmp on a single LAG or set of links, think
about multple lsp's between 2 distant endpoints).

>> ... assuming vendors successfully adopt the IPv6 flow-label as input-keys
>> in their hash calculations at some point in the future, we come back and
> I would agree to this statement if one of two things were to happen:  (1) we
> were to remove "(or UDP w/0 checksums) -- making UDP-Lite a MUST in the near

What was the original reason for removing the ability to do zero
checksums on udp in v6? Are we sure that that decision is still
sensible/appropriate in today's internet/world?

-Chris

> term", or (2) we were to satisfy ourselves that using zero checksums will
> not represent an operational problem for LISP, or for other applications
> that need to co-habitate with LISP, and we were to add a normative
> dependency on a document (like Marshall's, with edits) that would allow the
> use of UDP w/0 checksums over IPv6 in some cases, with LISP being an example
> of a case where this was appropriate.
>
> Margaret
>

---

From: Margaret Wasserman <[email-masked]>
Date: Wed, 5 Aug 2009 15:08:09 -0400

Hi Joel,

On Aug 5, 2009, at 1:42 PM, Joel M. Halpern wrote:

> Reading this discussion, there seem to be a small number of  
> practical choices.
>
> If the vendor hardware that is / will be handling IPv6 can handle  
> the flow label as part of the ECMP / LAG calcualtion, than an I-D /  
> will be used, not what off-the-shelf parts might support in some  
> other box.)  While hardware upgrades take a long time, software  
> upgrades are easier, particularly if it makes the operators traffic  
> handling work better.
>
> handle flow labels for ECMP / LAG hash calculations, then it  
> probably also won't be able to handle alternate protocols (like UDP- 
> lite).  As such, everything that needs ECMP / LAG handling will need  
> to have TCP or UDP encapsulations, and given that many of those are  
> protected, we will want to allow UDP with 0 checksum.

Could you explain what you mean by "given that many of those are  
protected"?

I have this uncomfortable feeling that many of us (including me) are  
missing important details of what is being discussed here, and that  
the answer (or at least the problem) might become clearer if we could  
pool our knowledge...

For example, I really don't understand how LISP, as currently defined,  
gains any significant LAG/ECMP benefits by including a UDP header in  
the outer packet...  I understand that you want to be able to load- 
balance flows, even though they've been encapsulated by LISP.  And I  
understand that current load balancers can only do this based on a few  
fields:
src/dest IP addresses (two RLOCs), the IP traffic class, the IP  
protocol field (UDP=17) and the src/dest UDP ports (xxxx and LISP=4341).

I just don't see how this works very well with LISP, though...  The  
two RLOCs are likely to be constant for all traffic between the same  
LISP routers, and ITR/ETR pair (right?).  The protocol will always be  
UDP (as currently defined), and LISP doesn't do anything unusual with  
the traffic class.  So, are we expecting all of the load balancing for  
LISP flows between a specific ITR/ETR pair to be based on the single  
UDP port that _isn't_ 4341?  Wouldn't this end-up with a bi- 
directional flow being broken into two different flows for load  
balancing, resulting in asymmetric routing, because the load balancer  
wouldn't see the same two UDP ports in both directions?

I guess I really need to read that ECMP draft again...

Margaret

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 05 Aug 2009 15:18:47 -0400

The basic model is that ECMP works by hashing all the fields.
Therefore, you need two related properties:
1) You need those fields to be stable across all the packets in a flow 
in one direction.  (There is absolutely no need for the answer to be the 
same in the reverse direction, as there is no order preservation issue 
between the two direcitons.
2) You need to be able to produce (with high probability) different 
answers for different flows, so that the hash logic can spread the 
different flows across different paths / links.

By varying the source port in the UDP header LISP allows the balancing 
logic to spread flows.  (The exactly logic it chooses to use to pick the 
source port is a local matter.  If there are enough different traffic 
sources, it can use the inner source address (EID).  Or it can use the 
EID, and where available source and dest port numbers.

As you implied, there is no RFC on this.  That is because it is a local 
behavior.  (different devices can do different things, and it still 
works.)  The problem we have surfaced is that other systems actually do 
depend upon this behavior, and therefore it probably needs to be documented.

It has been a very long time since I actually worked with this aspect of 
router (forwarding) logic.

Joel

Margaret Wasserman wrote:
>> such, everything that needs ECMP / LAG handling will need to have TCP 
>> or UDP encapsulations, and given that many of those are protected, we 
>> will want to allow UDP with 0 checksum.
> 
> missing important details of what is being discussed here, and that the 
> answer (or at least the problem) might become clearer if we could pool 
> our knowledge...
> 
> gains any significant LAG/ECMP benefits by including a UDP header in the 
> outer packet...  I understand that you want to be able to load-balance 
> flows, even though they've been encapsulated by LISP.  And I understand 
> that current load balancers can only do this based on a few fields:
> src/dest IP addresses (two RLOCs), the IP traffic class, the IP protocol 
> field (UDP=17) and the src/dest UDP ports (xxxx and LISP=4341).
> 
> I just don't see how this works very well with LISP, though...  The two 
> RLOCs are likely to be constant for all traffic between the same LISP 
> routers, and ITR/ETR pair (right?).  The protocol will always be UDP (as 
> currently defined), and LISP doesn't do anything unusual with the 
> traffic class.  So, are we expecting all of the load balancing for LISP 
> flows between a specific ITR/ETR pair to be based on the single UDP port 
> that _isn't_ 4341?  Wouldn't this end-up with a bi-directional flow 
> being broken into two different flows for load balancing, resulting in 
> asymmetric routing, because the load balancer wouldn't see the same two 
> UDP ports in both directions?
>

---

From: Margaret Wasserman <[email-masked]>
Date: Wed, 5 Aug 2009 15:32:25 -0400

> sensible/appropriate in today's internet/world?

I have not been around long enough to have been there when that  
decision was made. However, it has been revisited several times, so  
I've heard the re-hashed reasons...

The removal of the IPv6 header checksum was done so that routers would  
not have to updated it on a hop-by-hop basis when they changed the IP  
TTL.  Also, the IP header checksum calculation was seen as redundant  
for most traffic (TCP and UDP with checksums enabled), and people  
wanted to avoid the extra processing.

However, there was concern that the removal of the IP header checksum  
in IPv6 would lessen the protection of the source/destination IP  
addresses and result in a significant (a multiplier of ~32,000)  
increase in the number of times that a UDP packet was accidentally  
delivered to the wrong destination address and/or apparently sourced  
from the wrong source address when UDP checksums were set to zero --  
at the time there were vendors who shipped their IP stacks with UDP  
checksums turned off by default.  There was concern that this would  
result in misdelivery of data to UDP applications (dropped connections  
or even corrupted data -- we all saw this when NFS data was corrupted  
on the wire), in replies sent to nodes that didn't send a request  
(perhaps interrupting valid exchanges), and/or in ICMP errors sent to  
nodes that didn't send the packet that generated the error (perhaps  
resulting in dropped communications or unintelligible user errors).

The solution for this concern was to mandate UDP checksums for IPv6,  
so that the IP source and destination addresses would be protected by  
the UDP pseudo-header checksum.

There might be other solutions that we could implement in LISP that  
would eliminate these concerns, as well.

Margaret

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 5 Aug 2009 15:55:01 -0400

On Wed, Aug 5, 2009 at 3:32 PM, Margaret Wasserman<[email-masked]> wrote:
> I have not been around long enough to have been there when that decision was
> made. However, it has been revisited several times, so I've heard the
> re-hashed reasons...
>
> The removal of the IPv6 header checksum was done so that routers would not
> have to updated it on a hop-by-hop basis when they changed the IP TTL.
>  Also, the IP header checksum calculation was seen as redundant for most
> traffic (TCP and UDP with checksums enabled), and people wanted to avoid the
> extra processing.

The 'extra processing' is done today in v4 though, right? and
apparently not a huge problem for routers even at line-rate 10/40G?

> However, there was concern that the removal of the IP header checksum in
> IPv6 would lessen the protection of the source/destination IP addresses and
> result in a significant (a multiplier of ~32,000) increase in the number of
> times that a UDP packet was accidentally delivered to the wrong destination
> address and/or apparently sourced from the wrong source address when UDP
> checksums were set to zero -- at the time there were vendors who shipped
> their IP stacks with UDP checksums turned off by default.  There was concern
> that this would result in misdelivery of data to UDP applications (dropped

Does this happen today with any appreciable frequency? (on packets
without udp checksum, say that come from edonkey/emule or other
sources of zero/no checksum on udp)

> connections or even corrupted data -- we all saw this when NFS data was
> corrupted on the wire), in replies sent to nodes that didn't send a request
> (perhaps interrupting valid exchanges), and/or in ICMP errors sent to nodes
> that didn't send the packet that generated the error (perhaps resulting in
> dropped communications or unintelligible user errors).

This I don't recall at all... I think part of my question is we (as a
group) are assuming that the reasons for requiring ipv6 udp checksums
as stated +10 years ago are still valid, I don't see data supporting
that fact.

-Chris

> The solution for this concern was to mandate UDP checksums for IPv6, so that
> the IP source and destination addresses would be protected by the UDP
> pseudo-header checksum.
>
> There might be other solutions that we could implement in LISP that would
> eliminate these concerns, as well.

---

From: Margaret Wasserman <[email-masked]>
Date: Wed, 5 Aug 2009 17:07:41 -0400

> as stated +10 years ago are still valid, I don't see data supporting

There are some classic papers on this topic.  The most recent I could  
find is from 2000:

http://conferences.sigcomm.org/sigcomm/2000/conf/abstract/9-1.htm

In a quick re-read of this paper, I didn't see anything that is  
obviously dated about it.  So, I'd assume its error rates and analysis  
are still pertinent, unless there is a more recent study that says  
otherwise.

Margaret

---

From: Hesham Soliman <[email-masked]>
Date: Thu, 06 Aug 2009 08:10:30 +1000

On 5/08/09 8:24 AM, "Brian E Carpenter" <[email-masked]> wrote:

> of any deployed use cases (such as load balancing routers).

=> One deployment case can be found in: draft-ietf-mext-flow-binding
And the accompanying filter description draft-tsirtsis-mext-binary-filters

Hesham

The gap

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 06 Aug 2009 16:19:53 +1200

On 2009-08-06 05:34, Christopher Morrow wrote:
...
> flow-label's going to cut it.

Please note the word "alone" in the above extract from RFC3697. That RFC
specifically does not forbid flows from different sources that happen
to go through the same router having the same flow label. That
would have required some massive unscaleable flow-label-distribution
protocol. (MPLS avoids this problem by having the downstream switch tell
the upstream switch what label to use.) So even if the flow labels are
pseudo-random, it's essential to include other material such as the
addresses in the hash. But clearly a pseudo-random flow label will
always improve the distribution of the resulting hash. Just don't use
it *alone*.

Our hidden fear when drafting those words was that lazy implementors
would allocate flow labels starting at 1 and counting up. That would
guarantee collisions in flow label space, strengthening the "poorness"
of the material. And it would also mess up any router whose performance
relied on the absence of flow label collisions.

I hope this clarifies, as requested. There's no reason to remove these
words, since they're true. The WG consensus at the time was to trim RFC3697
down to the normative text, so a lot of rationale was removed.

    Brian

---

From: Pekka Savola <[email-masked]>
Date: Thu, 6 Aug 2009 08:03:15 +0300 (EEST)

On Thu, 6 Aug 2009, Brian E Carpenter wrote:
> Please note the word "alone" in the above extract from RFC3697. [...]

And from the POV of LISP, this is not relevant.  In IPv6, the hash key 
should probably consist of four-tuple {srcIP, dstIP, traffic class, 
flow label}.  As such, the addition of flow label would make a great 
addition to the total hash key for the purposes of this discussion.

Even though the flow label is set to zero, hashing would still work 
just fine but on IP/tclass granularity.  By adding flow label, you 
could get finer granularity for flows that do set it.

---

From: Shane Amante <[email-masked]>
Date: Thu, 6 Aug 2009 07:40:28 -0600

Brian,

On Aug 5, 2009, at 22:19 MDT, Brian E Carpenter wrote:
>>> dependent
> Please note the word "alone" in the above extract from RFC3697.

I think Chris may have read that a little too fast.  :-)  I wasn't  
concerned with the third sentence in the RFC, that makes sense and is  
clear.  However, my concern was pertaining to the 2nd sentence,  
specifically: "Router performance SHOULD NOT be dependent on the  
distribution of the Flow Label values".  Specifically, if Flow Label  
values ARE used as one of the input-keys (in addition to src/dst IPv6  
addresses), then the "distribution" of Flow Label values matters /a  
lot/ in order to achieve "good" load-balancing over LAG/ECMP paths  
and, consequently, good router performance.  However, perhaps I'm  
misreading/misunderstanding that 2nd sentence.  Can you clarify it's  
intent?

-shane

---

From: Margaret Wasserman <[email-masked]>
Date: Thu, 6 Aug 2009 10:11:14 -0400

Hi Joel,

I think I understand both sides of the UDP checksum issue now...

We (or at least some of us) believe that it is a hard requirement to  
support ECMP through legacy routing equipment.  This equipment will  
only identify flows using the 5-tuple described in the draft.  These  
devices do not know about UDP-Lite, and they couldn't usefully load  
balance LISP traffic that was tunneled IP-in-IP, because there would  
be no source port to use as a hash value for identification of the  
original flow.

We (or at least some of us) are concerned about using UDP zero  
checksums in IPv6, because of possible problems caused by corruption  
of the source/destination IP addresses or the next header field.   
These fields are protected in IPv4, even when UDP checksums are  
disabled.

I have an idea that might work to satisfy both sets of concerns...

What about putting a checksum field in the LISP header?  We could  
checksum across the IP pseudo-header (at least for IPv6), and the LISP  
header.  LISP nodes that receive these packets could check the  
checksum and discard packets that have been corrupted.  In addition to  
resolving the concerns about corruption in the IPv6 header fields,  
this checksum would protect the LISP header, which contains some one- 
bit fields and a nonce that would be sensitive to corruption.  We  
could also specify that LISP nodes that receive an ICMP error should  
check this checksum in the returned packet before acting on the error.

We would still have to make sure that we understand the impacts of non- 
LISP nodes receiving corrupted LISP packets (which will probably be  
that the packet is dropped, which is fine), but we wouldn't have to  
worry about what happens when a LISP packet arrives at the wrong LISP  
node and/or when a response is sent back to the wrong LISP node.

Thoughts?

Margaret

On Aug 4, 2009, at 10:51 PM, Joel M. Halpern wrote:

>>> of the

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 06 Aug 2009 10:32:58 -0400

I must be slow this morning.

I am not sure which of two possible (similar) problems this checksum is 
supposed to help.

1) Due to in-network corruption, the LISP packet arrives at some 
non-LISP entity which has something listening at whatever UDP port the 
packet now ha as a destination.  Clearly, a checksum in the LISP header 
won't help, since the receiver won't know there is a LISP header.

2) Due to in-network corruption, the LSP packet arrives at the wrok LISP 
receiver.  He understands it is LISP, and does not realize it was not 
for him.  So what?  He decapsulates, looks at the contents, and says "I 
can't do anything with that."  The LISP spec already tells us what it 
will do, and there is no problem.  If LISP were not using UDP, we would 
still have this "risk".  I, at least, don't see this as a problem to be 
solved.
2') There is some discussion of using the IPv 6 flow label instead of 
the UDP header for ECMP effects.  If we do that, I can not see why we 
would suddenly decide that LISP itself needed a checksum.

Yours,
Joel

Margaret Wasserman wrote:
> support ECMP through legacy routing equipment.  This equipment will only 
> identify flows using the 5-tuple described in the draft.  These devices 
> do not know about UDP-Lite, and they couldn't usefully load balance LISP 
> traffic that was tunneled IP-in-IP, because there would be no source 
> port to use as a hash value for identification of the original flow.
> 
> We (or at least some of us) are concerned about using UDP zero checksums 
> in IPv6, because of possible problems caused by corruption of the 
> source/destination IP addresses or the next header field.  These fields 
> are protected in IPv4, even when UDP checksums are disabled.
> 
> header.  LISP nodes that receive these packets could check the checksum 
> and discard packets that have been corrupted.  In addition to resolving 
> the concerns about corruption in the IPv6 header fields, this checksum 
> would protect the LISP header, which contains some one-bit fields and a 
> nonce that would be sensitive to corruption.  We could also specify that 
> LISP nodes that receive an ICMP error should check this checksum in the 
> returned packet before acting on the error.
> 
> We would still have to make sure that we understand the impacts of 
> non-LISP nodes receiving corrupted LISP packets (which will probably be

---

From: Luigi Iannone <[email-masked]>
Date: Thu, 6 Aug 2009 16:22:42 +0200

On Aug 6, 2009, at 16:11 , Margaret Wasserman wrote:

> checksum across the IP pseudo-header (at least for IPv6), and the  
> LISP header.  LISP nodes that receive these packets could check the  
> checksum and discard packets that have been corrupted.  In addition  
> to resolving the concerns about corruption in the IPv6 header  
> fields, this checksum would protect the LISP header, which contains  
> some one-bit fields and a nonce that would be sensitive to  
> corruption.  We could also specify that LISP nodes that receive an  
> ICMP error should check this checksum in the returned packet before  
> acting on the error.
>
> non-LISP nodes receiving corrupted LISP packets (which will probably  
> be that the packet is dropped, which is fine), but we wouldn't have  
> to worry about what happens when a LISP packet arrives at the wrong  
> LISP node and/or when a response is sent back to the wrong LISP node.
>
> Thoughts?

Adding a checksum in the LISP header would mean either to shrink some  
other fields or to enlarge the existing LISP header. I am not sure  
that these are good ideas.

Another approach is to use the UDP source port number. So far in LISP  
the source port number is selected using an hash function and ignored  
on reception. This can be changed in a checksum to be checked on the  
ETR.

Not sure if such an approach breaks something or goes against any RFC.

just my random thoughts....

Luigi

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 6 Aug 2009 08:50:29 -0700

I told myself I would stay out of this, but I can't help
but point out that if the SEAL-FS header were used instead
of the UDP/LISP headers there would only be 4 bytes exposed
to corruption instead of 16. And, if one of the SEAL-FS
header fields is corrupted there is no danger of causing
unpredictable behavior.

The SEAL-FS header is formatted as follows:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|VER|I|                     Identification                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

If the I bit is flipped, the end result is that the ETR
could send a spurious Information Request message which
should cause no harm whatsoever to the ITR. If the
Identification field is corrupted, the end result is that
the ETR may send a control message back to the ITR with
an unrecognized Identification - this would be dropped
silently by the ITR. If the VER field is corrupted, there
is a chance that the ETR could try to process the packet
according to a different SEAL protocol version. But, LISP
ITRs would only be expected to implement VER=0 anyway so
the packet would simply be dropped.

All this, plus a tidy 12 bytes per-packet saved over what
the existing UDP/LISP encapsulation gives.

Fred
[email-masked]

---

From: Margaret Wasserman <[email-masked]>
Date: Thu, 6 Aug 2009 12:15:25 -0400

Hi Fred,

There are three things we are trying to address here:

- We want to support load balancing through legacy systems that only  
support load balancing based on the 5-tuple of IP src/dest address,  
protocol/next header and UDP or TCP src/dest ports. To meet this goal,  
we need a UDP (or TCP) header, so that we can prevent all LISP packets  
between a single ITR/ETR pair from being seen as a single flow.

- But, we don't want to pay the cost of checksumming the entire UDP  
payload (including the LISP header and the encapsulated packet).  To  
avoid this, the current proposal states that we will use UDP with zero  
checksums.

- However, we are concerned about using UDP with zero checksums in  
IPv6, because there would be no checksum protection for the IPv6  
header fields (src/dest port and next header).  These fields are  
protected by the IP header checksum in IPv4.  Some people are also  
concerned about making sure that LISP is compatible with other IETF  
RFCs, which currently say that there won't be zero UDP checksums in  
IPv6.

So, we need to figure out what to do.  Once we have all of the facts  
on the table, we'll probably have to make a hard decision and  
compromise on some of the above goals.

It seems to me that our proposal to move to a different LISP header  
format is a non-starter, because it doesn't address any of these  
issues/tradeoffs.

Margaret

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 6 Aug 2009 10:42:10 -0700

Margaret,

> -----Original Message-----
> From: Margaret Wasserman [mailto:[email-masked]]
> Sent: Thursday, August 06, 2009 9:15 AM
> To: Templin, Fred L
> Cc: [email-masked]; [email-masked]
> Subject: Re: [lisp] Flow label redux [Re: IPv6 UDP checksum issue]
> 
> 
> Hi Fred,
> 

How are non-TCP/UDP flows handled by these legacy systems
today? For example, 6to4 uses ip-proto-41.

Thanks - Fred

---

From: Margaret Wasserman <[email-masked]>
Date: Thu, 6 Aug 2009 16:09:37 -0400

Hi Fred,

> today? For example, 6to4 uses ip-proto-41.

My understanding is that these flows will not be handled well...   
Since ECMP load balancers will have limited information available (the  
two IP address and the next header value), they will group all of the  
traffic between two 6to4 tunnel endpoints into a single flow.  If it  
becomes common for pairs of tunnel endpoints to transmit large amounts  
of data across ISP networks, this could present a significant  
operational issue for the Internet.

This problem (and the LISP problem) could potentially be alleviated by  
hardware that performs load balancing based on the IP address, the  
next header and the IPv6 flow label. But, according to folks who ought  
to know, most of the IPv6 routers they have deployed can't handle load  
balancing based on anything but a UDP or TCP 5-tuple, even in IPv6 (no  
IPv6 flow label, no UDP-Lite, no SCTP, etc.).  This is rather  
distressing, but I have no reason to doubt their statements.

Margaret

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 07 Aug 2009 09:32:02 +1200

Shane,

On 2009-08-07 01:40, Shane Amante wrote:
> addresses), then the "distribution" of Flow Label values matters /a lot/
> in order to achieve "good" load-balancing over LAG/ECMP paths and,
> consequently, good router performance.  However, perhaps I'm

Pekka just did that, I think - the flow label will add value if it happens
to increase the randomness of the hash, but this can't be relied on
so we shouldn't depend on it.

     Brian

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 6 Aug 2009 15:30:13 -0700

Margaret,

> Sent: Thursday, August 06, 2009 1:10 PM
> To: Templin, Fred L

I guess add IPSec to the list of flows that would not be
handled well in the future? Are you saying that we need
to accept that the Internet is ossified, and all new
protocols will need to continue to do things the old way?

Fred

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 7 Aug 2009 11:47:25 +0200

On 5 aug 2009, at 16:16, Margaret Wasserman wrote:

> What I am asking is whether IPv6 routers containing that silicon  
> exist in real-world deployments in large enough numbers that they  
> should be considered in our design choices.

The real question is how many of these routers use parallel links  
towards other IPv6 routers where balancing traffic on the 3-tuple  
(addresses + next header) wouldn't provide sufficient balancing of the  
traffic over the links.

Assuming a reasonable number of different hosts communicating through  
the link bundle, this will only be problematic when any two hosts are  
responsible for a significant fraction of the bandwidth of an  
individual link, but communicate using multiple sessions. If we assume  
these are at least 1 Gbps, we'd only have trouble when a single host  
has several 200 Mbps+ session to another IPv6 host through the link  
bundle.

Considering the fact that the total native IPv6 traffic through the  
AMS-IX currently peaks at around 1.5 Gbps I manage to not lose sleep  
over the risk of not being able to load balance more granularly than  
on the 3-tuple in IPv6 currently.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 7 Aug 2009 11:52:03 +0200

> hardware is on the shelf to purchase.

You assume this requires new hardware. Although that's not  
inconceivable, I would think that in most cases a software or  
microcode update would be sufficient.

> flow-label's going to cut it.

Huh?

By the way, if we want good load balancing in IPv6 it's always  
possible to use more source/destination address pairs, as addresses  
are cheap in IPv6.

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 7 Aug 2009 09:49:33 -0400

On Fri, Aug 7, 2009 at 5:52 AM, Iljitsch van Beijnum<[email-masked]> wrote:
> You assume this requires new hardware. Although that's not inconceivable, I
> would think that in most cases a software or microcode update would be
> sufficient.

I look forward to your microcode update ... I'm not sure that
classifiers (on juniper or cisco devices) that work at 40g+ are going
to be microcode upgradable. My point is that just saying: "Well, now
use X. See all fixed!" isn't reality. There is atleast a 3-5 year lag.

>> flow-label's going to cut it.

brian sufficiently cleared up the wording/reasoning.

> By the way, if we want good load balancing in IPv6 it's always possible to
> use more source/destination address pairs, as addresses are cheap in IPv6.

awesome, how do I do that dynamically and on-demand? you're imposing a
restriction on the end users in order to fix this core problem,
outside of LISP I mean, I'm not sure that's going to work out either
(shifting costs and such).

-chris

---

From: Francis Dupont <[email-masked]>
Date: Fri, 07 Aug 2009 17:24:23 +0200

In your previous mail you wrote:

   And I understand that current load balancers can only do this based
   on a few fields:
   src/dest IP addresses (two RLOCs), the IP traffic class, the IP  
   protocol field (UDP=17) and the src/dest UDP ports (xxxx and LISP=4341).
   
   I just don't see how this works very well with LISP, though...  The  
   two RLOCs are likely to be constant for all traffic between the same  
   LISP routers, and ITR/ETR pair (right?).

=> in fact the IPv6 addresses don't need to be the same when xTRs are
attached to regular links with /64 prefixes. So IMHO most of this
discussion is insane:
 - if we need to vary things between a pair of IPv6 xTRs it should
  be enough (and simple/easy) to vary the addresses
 - we already have a "waste" of 2^64 addresses per links so this should
  never become a resource problem (:-)
 - the O UDP checksum proposal obsoletes all the today deployed nodes
  which check them (so all hosts I know and perhaps a lot of routers too),
  so if we believe something has to be fixed some routers should be fixed
  and not all BSD/Linux/MacOS/Windows boxes (software and hardware).
  To summary not only it is better to encourage application of RFCs
  than to twist them because they don't reflect the last idea, but
  in this case this is a total economical non-sense...

Thanks

[email-masked]

PS: Margaret, the insane is not for you: your messages in this thread
are clearly the most reasonnable/rational.
PPS: to use the flow label (not alone) is not stupid, the 5-tuple is
not always easy to extract in particular with IPv6.

---

From: Shane Amante <[email-masked]>
Date: Fri, 7 Aug 2009 11:02:11 -0600

Francis,

> attached to regular links with /64 prefixes. So IMHO most of this

:-)

>  be enough (and simple/easy) to vary the addresses

The above was considered at the initial design stages of LISP, (I  
think Dino may have considered this approach?); however, it was shot  
down, because this would require the installers/administrators of the  
LISP ITR to both know the reasons why the above is required *and*  
configure the IPv6 ITR's with /64's to get the load-splitting benefits  
across LAG/ECMP paths.  (Remember the adage: "My network, my rules").   
Instead, it's more 'straightforward' to just have the ITR's / 
automatically/ calculate a hash of the incoming IP packet and stuff  
that into the UDP source port before transmitting it out its uplinks  
toward the ETR.  The key advantage of this approach is that ITR's / 
will/ be owned & operated by downstream customer's (i.e.: it's their  
CE's) who are likely unaware of and, therefore, may not care about the  
(extremely large) prevalence of LAG/ECMP paths through their upstream  
providers ... until congestion occurs in their upstream providers and  
they get upset.  When in doubt, automatic == good.

-shane

---

From: Shane Amante <[email-masked]>
Date: Fri, 7 Aug 2009 12:05:30 -0600

Hi Margaret,

Apologies for the delay, but it took some time to follow-up with some  
vendors.  See below.

On Aug 5, 2009, at 12:33 MDT, Margaret Wasserman wrote:
>> not to say that we (SP's) should not "encourage" vendors to do  
>> this, anyway, (we are/will) however if LISP (or other protocols  
>> like it that depend on tunneling) quickly gain traction, we need a  
>> way to deal with these traffic flows in our networks today, without  
>> telling customers: "turn off protocol <FOO>, because we can't  
>> deliver your packets".
>
> ECMP of LISP flows is an optimization that only matters when there  
> is a large amount of LISP traffic, right?

ECMP _and_ LAG.  But, in short, you're correct.  However, given the  
dominant link speed deployed in today's networks is 10G, (which are  
then lashed together in N x 10G LAG and/or ECMP paths), then we  
clearly don't want anything near that size of single flows on these  
links otherwise this will have a detrimental effect on the performance  
of other "well-behaved" traffic that is being evenly spread over  
parallel links.

> Do you think there is a reasonable chance that LISP would be widely- 
> enough to deployed to require this optimization in less than 3-to-5  
> years?

My crystal ball is out-of-service.  In other words, that's difficult  
to say, given that LISP is currently pursuing a "Experimental" track.   
However, assuming the experiment is successful, then it's pretty safe  
to assume folks would want to quickly turnaround (with the methods &  
protocols already developed and tested during the Experiment) and turn  
that straight into a production service.  So, IMO, it's best if we try  
to get this "good enough" from the beginning and not hamper its  
deployment by requiring networks to adapt before LISP, for example,  
could really take off (at least, in the short-term).

Furthermore, given the LISP folks believe this is the first  
"iteration" of LISP, there may be 'natural' points in the LISP  
evolution path (for example, to support better mapping functions)  
where it makes sense to consider evolving the encapsulation (for  
example, to support flow-labels instead of UDP encaps), as well.  I  
would argue, though, to get to those 'natural' points people will need  
to experiment with LISP to find out what works, and doesn't work, and  
bring that back to the IETF community to figure when & how to evolve  
LISP.

>> Perhaps one way to satisfy the parties in this conversation would  
>> be something along the following lines:
>> ... assuming vendors successfully adopt the IPv6 flow-label as  
>> input-keys in their hash calculations at some point in the future,  
>> we come back and deprecate the UDP/IPv6 tunnel method and elevate  
>> IP-in-IPv6 w/ flow-labels to a MUST.
>
> I would agree to this statement if one of two things were to  
> happen:  (1) we were to remove "(or UDP w/0 checksums) -- making UDP- 
> Lite a MUST in the near term", or (2) we were to satisfy ourselves  
> that using zero checksums will not represent an operational problem  
> for LISP, or for other applications that need to co-habitate with  
> LISP, and we were to add a normative dependency on a document (like  
> Marshall's, with edits) that would allow the use of UDP w/0  
> checksums over IPv6 in some cases, with LISP being an example of a  
> case where this was appropriate.

Based on my conservations with several vendors, flow-based load- 
balancing over LAG/ECMP paths is currently NOT accounting for the  
following in their input-keys to LAG/ECMP hash algorithms:
1)  IPv6 flow-label; or,
2)  UDP-lite (since, UDP-lite uses a different protocol ID [137]  
compared to standard UDP [17]).  In other words, if routers see UDP- 
lite (Protocol ID: 137), the router will revert back to just using Src/ 
Dst IPv6 Addresses and will not have the benefit of being exposed to  
the finer granularity of micro-flows that are denoted by varying the  
Src/Dst Layer-4 (UDP-lite) ports.

Therefore, I'll have to revise my original recommendation in the first  
bullet above that we only consider UDP with 0 checksums as the  
preferred short-term solution when IPv6 is being used as the outer  
encapsulation, (unless someone has a better idea that haven't been  
brought forth, yet).

-shane

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 7 Aug 2009 20:21:15 +0200

Shane, thanks for infusing this discussion with some data.

On 7 aug 2009, at 20:05, Shane Amante wrote:

> Therefore, I'll have to revise my original recommendation in the  
> first bullet above that we only consider UDP with 0 checksums as the  
> encapsulation,

I don't see that. Currently, there isn't that much IPv6 traffic in the  
first place, and certainly not between the same source/destination  
addresses. So the lack of a fine-grained optimal solution to the load  
balancing issue is not a problem in practice. This affords us the  
relative luxury of being able to ignore current problems in  
implementations and do the right thing, rather than be forced to do  
something ugly and difficult (that would be UDP with no checksum).

And if we were to consider an ugly solution, making LISP use several  
IPv6 addresses per ITR/ETR would be a better solution.

---

From: Shane Amante <[email-masked]>
Date: Fri, 7 Aug 2009 12:43:27 -0600

On Aug 7, 2009, at 12:21 MDT, Iljitsch van Beijnum wrote:
>> first bullet above that we only consider UDP with 0 checksums as  
>> the preferred short-term solution when IPv6 is being used as the  
>> outer encapsulation,
>
> I don't see that.  Currently, there isn't that much IPv6 traffic in  
> the first place, and certainly not between the same source/ 
> destination addresses. So the lack of a fine-grained optimal  
> solution to the load balancing issue is not a problem in practice.  
> This affords us the relative luxury of being able to ignore current  
> problems in implementations and do the right thing, rather than be  
> forced to do something ugly and difficult (that would be UDP with no  
> checksum).

In that case, I guess we'll have to agree to disagree and in doing so,  
I'd like to take the liberty of quoting Vince Fuller's message to this  
list back on 8/4/2009, where he says it better than I could:
---snip---

-shane

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 7 Aug 2009 14:59:41 -0400

On Wed, Aug 5, 2009 at 5:07 PM, Margaret Wasserman<[email-masked]> wrote:
> There are some classic papers on this topic.  The most recent I could find
> is from 2000:

If by 'classic' you mean 'not relevant to today's networking
technologies', sure.

> http://conferences.sigcomm.org/sigcomm/2000/conf/abstract/9-1.htm

This study captured data on:

1) a web crawl server on a 10mb hub
2) a dorm on a broadcast 10base2 network
3) 2 locations that don't give enough information about the
connections/tech used

The dataset analyzed is not relevant to today's networking
connectivity or technologies. Looking very quickly at a small set of
data I have access to (servers serving web content to the internet
users):

32,945,810,591 packets received, 0 dropped due to bad checksum (ip
header checksum)

1,004,728,008 datagrams received, 0 bad checksum, 15886 with no
checksum (udp datagram stats)

(collected from some unix hosts, via netstat -s or netstat -s -p udp output)

Given this set of data I don't think that having a checksum matters
for UDP or IP-header on today's internet, since there are zero errors
out of ~34B packets.

-chris

> In a quick re-read of this paper, I didn't see anything that is obviously
> dated about it.  So, I'd assume its error rates and analysis are still
> pertinent, unless there is a more recent study that says otherwise.
>
> Margaret
>
>
>
>

---

From: Marshall Eubanks <[email-masked]>
Date: Fri, 7 Aug 2009 15:42:48 -0400

> checksum (udp datagram stats)

Just polling the routers here, I see a similar scenario, e.g.,

4,166,900,871 packets 0 dropped due to bad checksum

However, this is over good clean fiber links. What I would worry about  
are RF links, such as 802.11 or P2P microwaves. These generally have  
link layer redundancy / checksums, etc., which I think are pretty good  
at detecting corrupted packets, but
it might not be wise to rely on that.

I wonder if we could dig up similar numbers for the IETF 802.11  
network. I will make inquiries.

Regards
Marshall

>> obviously

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 7 Aug 2009 16:06:16 -0400

On Fri, Aug 7, 2009 at 3:42 PM, Marshall Eubanks<[email-masked]> wrote:
>>> find

neat! (I'm also going to see if I can get some stats from a wider set
of hosts, but....)

> However, this is over good clean fiber links. What I would worry about are
> RF links, such as 802.11 or P2P microwaves. These generally have link layer
> redundancy / checksums, etc., which I think are pretty good at detecting
> corrupted packets,
>
> I wonder if we could dig up similar numbers for the IETF 802.11 network. I
> will make inquiries.
>
> Regards
> Marshall
>
>
>>
>>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 7 Aug 2009 22:10:52 +0200

CCing the IAB because I think we are reaching a slippery architectural  
slope. Hopefully they can help us out.

>> forced to do something ugly and difficult (that would be UDP with  
>> no checksum).

> In that case, I guess we'll have to agree to disagree

Well, we still either have UDP encapsulation with a 0 checksum for  
LISP over IPv6 or not.

> and in doing so, I'd like to take the liberty of quoting Vince  
> Fuller's message to this list back on 8/4/2009, where he says it  
> better than I could:
> ---snip---

The problem with adopting that logic and adopting a UDP header for  
LISP even though that would require breaking the IPv6 spec just  
because router vendors haven't gotten around to implementing ECMP  
based on the flow label is that it cements us into a TCP/UDP view of  
the world with IPv6 just like NATs did for IPv4.

In my opinion, the possible short term pain of less than optimal load  
balancing in a few places with high IPv6 LISP traffic is worth it if  
that allows us to proceed in an architecturally clean way.

---

From: Dino Farinacci <[email-masked]>
Date: Sat, 8 Aug 2009 17:34:00 -0700

> 0 checsums or whether all stacks should accept 0 checksums.

The spec says ETRs MUST ignore the UDP checksum field. This is what  
the LISP authors intended and has been implemented this way.

The spec says ITRs MUST set the UDP checksum field to 0.

Dino

---

From: David Malone <[email-masked]>
Date: Sun, 9 Aug 2009 21:02:09 +0100

> neat! (I'm also going to see if I can get some stats from a wider set

If routers check the IPv4 header checksum (which I think they are
supposed to for IPv4), then I don't think you will see packets with
IP checksum errors unless you are adjecent to a router/host that
is generating bad checksums. For example, on one router I have
access to that is exposed to a networks worth of random hardware,
I've seen 2 bac IP checksums from 333128929 IP packets. On another
home router I see 4 from 16089085.

However, in IPv6, these errors will be forwarded on and show up as
UDP or TCP checksum errors on end hosts.

	David.

---

From: Francis Dupont <[email-masked]>
Date: Mon, 10 Aug 2009 00:14:17 +0200

across LAG/ECMP paths.

=> I can't see the difference with varying the UDP source port.

=> it is not more straightforward, you can put the hash where you'd like.

=> you have just to put at least one of the end on a virtual link
to get up to 64 bits of path selection. IMHO it is more than one
could ever need...   
   
Thanks

PS: IMHO this is an example of IPv6 misunderstanding: the solution
was developed for IPv4 and as it doesn't fit exactly into IPv6
in place of adjusting the solution you propose to adjust IPv6.

---

From: Christopher Morrow <[email-masked]>
Date: Sun, 9 Aug 2009 22:29:52 -0400

> in place of adjusting the solution you propose to adjust IPv6.

ugh... Thanks for listening to operators and device implementors
providing input to the process Francis. Always nice to see that we are
appreciated in this arena.

-Chris

---

From: Margaret Wasserman <[email-masked]>
Date: Mon, 10 Aug 2009 22:41:23 -0400

>  be enough (and simple/easy) to vary the addresses

This is an interesting idea, and I think it is worth exploring.   
However, there are a couple of issues to overcome...

Would you perform DAD on these addresses before you use them?  Or  
would you somehow delegate a prefix (perhaps longer than a /64) to  
each xTR for this purpose?
>  and not all BSD/Linux/MacOS/Windows boxes (software and hardware).

I think that the idea is that the IPv6 UDP headers with zero checksums  
will only be sent between the ITR and ETR, so it is only LISP nodes  
that need to change.

However, I personally would like to be able to implement a compliant  
LISP node (ITR and ETR) on a desktop operating system like Windows or  
Unix, and I don't think that one can expect deployed desktop operating  
systems to be updated (to allow zero UDP checksums in IPv6) any faster  
than the estimates given for routing updates (3-5 years).

Also, the last time I dealt with zero UDP checksums (mid 1990s), it  
was only possible to disable UDP checksums (for IPv4) in the unix  
kernel, so they were on or off for the entire box.  Have things  
evolved since then?  Is it possible to have IPv4 UDP checksums enabled  
(both calculating them and checking them) for some sockets while they  
are disabled (set to zero, and not checked on receipt) for other  
sockets?  If so, what sockets option is commonly used for this in Unix  
operating systems?

Thanks,
Margaret

---

From: Margaret Wasserman <[email-masked]>
Date: Mon, 10 Aug 2009 22:53:04 -0400

> 4,166,900,871 packets 0 dropped due to bad checksum

Marshal, do your routers actually calculate the checksums of the  
packets they process?

Shane, are you sure that checksums aren't be calculated lower down in  
the hardware than your counters are counting?  I have seen cases where  
error counters showed up as zero, because the packets that would have  
invoked those errors were already discarded at a lower level.

Margaret

---

From: Margaret Wasserman <[email-masked]>
Date: Mon, 10 Aug 2009 23:22:32 -0400

> The spec says ITRs MUST set the UDP checksum field to 0.

Could you tell us how to achieve this on commonly deployed desktop  
software/hardware that has two properties:

- There is no means to turn off the checking of UDP checksums.  They  
will be checked if they are non-zero in IPv4, and they will be checked  
in all cases in IPv6 with zero always being considered invalid.

- Networking Interface Cards that automatically calculate TCP/UDP  
checksums on transmission if a checksum of zero is sent to the card.

Systems with these properties are far more widely deployed than LAG/ 
ECMP routers, and they are no easier/quicker to update.

Margaret

---

From: Luigi Iannone <[email-masked]>
Date: Tue, 11 Aug 2009 09:58:00 +0200

On Aug 11, 2009, at 4:41 , Margaret Wasserman wrote:

> I think that the idea is that the IPv6 UDP headers with zero  
> checksums will only be sent between the ITR and ETR, so it is only  
> LISP nodes that need to change.
>
> LISP node (ITR and ETR) on a desktop operating system like Windows  
> or Unix, and I don't think that one can expect deployed desktop  
> operating systems to be updated (to allow zero UDP checksums in  
> IPv6) any faster than the estimates given for routing updates (3-5  
> years).

If you want LISP on a desktop OS you need to update that OS, hence at  
the same time you can patch it to handle the 0 UDP checksum  
consequently.  I do not see any real issue here.

Luigi

> evolved since then?  Is it possible to have IPv4 UDP checksums  
> enabled (both calculating them and checking them) for some sockets  
> while they are disabled (set to zero, and not checked on receipt)  
> for other sockets?  If so, what sockets option is commonly used for  
> this in Unix operating systems?
>
> Thanks,
> Margaret
>
>
>

---

From: [email-masked]
Date: Tue, 11 Aug 2009 10:11:23 +0200 (CEST)

> >> checksum (udp datagram stats)

Checking six of our name servers here (some pure recursive, some pure
authoritative):

- 11,125,766,153 IP packets received, with 0 bad header checksums
-  9,311,954,730 UDP datagrams received, with 1,300,228 bad checksums

Thus IP header checksum really looks like it is very close to zero,
probably because any IP packets with bad header checksums would be
dropped by the closest router.

However, the UDP bad checksum rate is very definitely not zero. I
get a rate of 1.39e-4 for my dataset.

Steinar Haug, Nethelp consulting, [email-masked]

---

From: Margaret Wasserman <[email-masked]>
Date: Tue, 11 Aug 2009 08:23:31 -0400

On Aug 11, 2009, at 3:58 AM, Luigi Iannone wrote:
>
> If you want LISP on a desktop OS you need to update that OS, hence  
> at the same time you can patch it to handle the 0 UDP checksum  

So, if I want LISP on a non-open-source desktop, you think I should  
just wait until the OS vendor gives it to me?

Why is this any more reasonable than telling ISPs that if they want to  
support LAG/ECMP for LISP flows, they need to buy new routers?

Margaret

---

From: Margaret Wasserman <[email-masked]>
Date: Tue, 11 Aug 2009 08:28:36 -0400

Hi Steinar,

> dropped by the closest router.

If LISP uses UDP checksums of zero with IPv6, packets with IP header  
corruption will not be dropped by the next router, they will be  
forwarded all the way to the ETR (or, in the case of destination  
address corruption) all the way to the wrong ETR or a non-LISP end-node.
> get a rate of 1.39e-4 for my dataset.

This is potentially an argument for using UDP (or UDP-Lite) checksums  
in LISP for both IPv4 and IPv6, so that the corruption in the outer  
UDP header and the LISP header will be detected before these packets  
are processed on the remote end.

Margaret

---

From: Luigi Iannone <[email-masked]>
Date: Tue, 11 Aug 2009 14:44:02 +0200

> just wait until the OS vendor gives it to me?

You have to wait only if you want the solution proposed by the vendor  
of the OS.

>
> Why is this any more reasonable than telling ISPs that if they want  
> to support LAG/ECMP for LISP flows, they need to buy new routers?
>

I did not claim that (or anything alike).

Luigi

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 11 Aug 2009 08:37:24 -0700

> software/hardware that has two properties:

We don't want this rule on commonly deployed desktop hosts.

Dino

> will be checked if they are non-zero in IPv4, and they will be  
> checked in all cases in IPv6 with zero always being considered  
> invalid.
>

---

From: Margaret Wasserman <[email-masked]>
Date: Tue, 11 Aug 2009 11:44:20 -0400

Hi Dino,

On Aug 11, 2009, at 11:37 AM, Dino Farinacci wrote:

>>> The spec says ETRs MUST ignore the UDP checksum field. This is  
>>> what the LISP authors intended and has been implemented this way.
>>>

I am not sure what you are saying here.  I can think of two  
possibilities:

(1) You don't want LISP to be implemented on commonly deployed desktop  
hosts.

(2) You want to change the document to state a different rule, so that  
LISP can be implemented in a compliant manner on desktop hosts.

Could you elaborate?

Margaret

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 11 Aug 2009 08:47:31 -0700

> (1) You don't want LISP to be implemented on commonly deployed  
> desktop hosts.

Right, it is a network based solution.

> (2) You want to change the document to state a different rule, so  
> that LISP can be implemented in a compliant manner on desktop hosts.

That is not what I am suggesting.

Dino

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 11 Aug 2009 12:01:00 -0400

Given that LISP ITRs work by intercepting packets that are not addressed 
to them, a host implementation would need to be able to intercept 
packets "in the stack".  That is going to need some ability to modify 
kernel behavior.

(Yes, I think we will see LISP enabled hosts.  I don't think mobility is 
the only case where this will happen.  But it will mean someone has 
modified the kernel to do so.)

Yours,
Joel

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 11 Aug 2009 12:20:22 -0400

I believe that saying "ITRs don't receiving packets." is a linguistic 
step that only confuses people.  ITRs receive unencapsulated packets 
from the site, and encapsulate them in LISP headers (assuming mappings 
are already available.)  Now, one can argue that the ITR function in the 
router does not "receive" packets, since it was the router that received 
the packet.  (One can even argue that the router "intercepted" the 
packet rather than "received" it.)  But functionally it is the same thing.

Yours,
Joel

Dino Farinacci wrote:
>> Given that LISP ITRs work by intercepting packets that are not 
>> addressed to them, a host implementation would need to be able to 
>> intercept packets "in the stack".  That is going to need some ability 
>> to modify kernel behavior.
> 
> (1) ITRs don't receive packets. They encapsulate packets.

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 11 Aug 2009 09:10:18 -0700

> intercept packets "in the stack".  That is going to need some  
> ability to modify kernel behavior.

(2) ETRs receive encapsulated packets addressed to one of their RLOC  
addresses.

(3) For Map-Requests, there are 3 forms:
     i.  Encapsulated Map-Requests which are originated by the map- 
server the ETR is registered to
         and is addressed to the ETR.
     ii. Map-Requests which are addressed to EIDs but are traveled  
over the ALT. Only when an ETR is
         attached to the ALT (not a standard configuration  
recommendation), that it must process a
         packet that is not addressed to it. But this is a packet  
destined to port 4342 and not 4341.
         UDP port 4342 is the control port.
     iii. Data Probes (which is not default and probably will be  
deprecated behavior) are addressed
          to EIDs which ETRs must intercept. They are addressed to UDP  
port 4341. Data Probes can
          be sent on the underlying network (LISP variant 1) or sent  
on the ALT which means the
          requirements in ii must be adhered to.

> (Yes, I think we will see LISP enabled hosts.  I don't think  
> mobility is the only case where this will happen.  But it will mean  
> someone has modified the kernel to do so.)

I would not recommend it and I believe won't be accepted for the same  
reasons that shim6 wasn't accepted. You will use locator  
aggregatability for EID-prefixes if you do. The mapping database will  
be unnecessarily large. IMHO, a non-starter.

Where you can aggregate EIDs, you should. In a roaming case where you  
may not be able to, then you don't.

Dino

>>>>> Could you tell us how to achieve this on commonly deployed  
>>>>> desktop software/hardware that has two properties:
>>>>

---

From: Sam Hartman <[email-masked]>
Date: Tue, 11 Aug 2009 12:38:45 -0400

Joel> Given that LISP ITRs work by intercepting packets that are
    Joel> not addressed to them, a host implementation would need to
    Joel> be able to intercept packets "in the stack".  That is going
    Joel> to need some ability to modify kernel behavior.

I'm trying to figure out how an ITR does anything different than a
router with a tunneled interface.

Every host I'm aware of has a facility for setting up an interface
that routes some set of packets--including potentially the default
route--through a tunnel interface that then passes the packet to
userspace for processing.

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 11 Aug 2009 09:42:44 -0700

> I believe that saying "ITRs don't receiving packets." is a  
> linguistic step that only confuses people.

You are right, but your text wasn't clear if the packets were coming  
from the site or from the core. So I assumed you were referring to the  
Map-Request or Data-Probe case.

> ITRs receive unencapsulated packets from the site, and encapsulate  
> them in LISP headers (assuming

Right, just like any router does today.

> mappings are already available.)  Now, one can argue that the ITR  
> function in the router does not "receive" packets, since it was the  
> router that received the packet.  (One can even argue that the  
> router "intercepted" the packet rather than "received" it.)  But  
> functionally it is the same thing.

I am not following you, but don't bother to explain. You are making a  
subtle point about forwarding versus consuming packets? Where an ETR  
would consume an encapsulated packet because its addressed to itself  
and when an ITR receives packets from site hosts, that it is forwarding?

Dino

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 11 Aug 2009 12:44:10 -0400

Maybe I was missing a bet.  You would have to direct all the packets 
from the host back to user space to be processed, since it is only the 
LISP logic that can decide whether the packet should be tunneled or not.
But if they support that concept, then it can be done.

And yes, that does seem to mean that if we could use the UDP stack right 
out of the box, then this would require no kernel modifications to add 
it.  Which is a nice benefit.

Given what has been discussed, this leads me to looking more closely at 
using the IPv6 flowID to enable load balacning, and not using UDP at 
all.  However, I do not know how to make this work with the deployed 
boxes that do hardware ECMP / LAG in the field already for IPv6.

Yours,
Joel

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 11 Aug 2009 09:46:14 -0700

> router with a tunneled interface.

It doesn't. I mean there are subtleties about if a map-cache entry  
exists and what to do when it doesn't. But if a map-cache entry exists  
it is similar to a packet arriving on an interface and is sent on a  
GRE tunnel. There are implementation details that make it different,  
but from a conceptual level, they are the same.

> userspace for processing.

We call "LISP tunnels" as "dynamic encapsulating tunnels" where an  
implementation must not implement the tunnel as a logical interface.  
The implementation cannot scale if it does this. You get the level of  
indirection by doing another lookup in another data structure called  
the "map-cache".

Dino

---

From: Luigi Iannone <[email-masked]>
Date: Tue, 11 Aug 2009 19:18:16 +0200

> ability to modify kernel behavior.

We already did it for FreeBSD. (http://inl.info.ucl.ac.be/softwares/openlisp 
)

The latest version (not yet publicly available) supports IPv6, and yes  
it ignores the checksum (will be changed if the WG reaches a different  
consensus).

This is possible in OpenLISP since we capture the packet at IP level,  
before actually forwarding it to the UDP level.

The same applies for outgoing packets. They are captured at IP level  
just before the routing table lookup. This allows to encap (if  
necessary) no matter if the packet is coming from the upper layers or  
from the forwarding module.

Luigi

---

From: Margaret Wasserman <[email-masked]>
Date: Tue, 11 Aug 2009 13:48:16 -0400

On Aug 11, 2009, at 12:46 PM, Dino Farinacci wrote:
> The implementation cannot scale if it does this. You get the level  
> of indirection by doing another lookup in another data structure  
> called the "map-cache".

I don't think I understand this...

Why couldn't LISP be implemented as a logical interface that  
encapsulates or not based on the contents of the LISP Mapping cache  
and the results of mapping lookups?

Margaret

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 11 Aug 2009 11:05:06 -0700

>> implementation must not implement the tunnel as a logical  
>> interface. The implementation cannot scale if it does this. You get  
>> the level of indirection by doing another lookup in another data  
>> structure called the "map-cache".
>

Because you could have 100K of them. Interface data structures come  
with all kinds of other stuff that doesn't apply here. And guess what  
if you had 100K map-cache entries, the number of logical interfaces is  
equal to the sum of all locators for all 100K entries.

Dino

---

From: Sam Hartman <[email-masked]>
Date: Tue, 11 Aug 2009 14:18:56 -0400

>>>>> "Dino" == Dino Farinacci <[email-masked]> writes:

    Dino> We call "LISP tunnels" as "dynamic encapsulating tunnels"
    Dino> where an implementation must not implement the tunnel as a
    Dino> logical interface.  The implementation cannot scale if it
    Dino> does this. You get the level of indirection by doing another
    Dino> lookup in another data structure called the "map-cache".

I see no text in draft-ietf-lisp-03 that says that lisp tunnels cannot
be logical interfaces and would object strongly to including such an
implementation detail in the spec.  I see text that says that they
must not be statically configured.

I also don't see the scaling argument you're making.  Perhaps you're
trying to say is that you want all traffic considered for lisp
encapsulation and do not want to flood the mapping database into the
local routing table.  That's probably true, but seems quite unrelated
to whether something is a logical interface at least with several host
routing implementations I'm familiar with.

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 11 Aug 2009 11:25:52 -0700

> must not be statically configured.

Because it is an implementation issue.

If an ITR decided to encapsulate all its packets to a recursive  
encapsulator, it could build an implementation with a single logical  
interface.

> routing implementations I'm familiar with.

This is all implementation dependent. You can talk to dozens if not  
100s of IOS engineers about this next time you are in an IETF hallway.  
They will talk your ear off.  ;-)

Dino

---

From: Margaret Wasserman <[email-masked]>
Date: Tue, 11 Aug 2009 14:28:57 -0400

Hi Dino,

On Aug 11, 2009, at 2:05 PM, Dino Farinacci wrote:
> with all kinds of other stuff that doesn't apply here. And guess  
> what if you had 100K map-cache entries, the number of logical  
> interfaces is equal to the sum of all locators for all 100K entries.

I was talking about running an ITR as a logical interface on a LISP- 
aware end-node or a home gateway, so I'm not talking about something  
that would need to scale to handle 100K simultaneous connections.

This could be done in a way that _didn't_ involve a logical interface  
per map-cache entry, by having the code in the logical interface  
"driver" maintain and access the map-cache.

Margaret

---

From: Luigi Iannone <[email-masked]>
Date: Tue, 11 Aug 2009 20:35:58 +0200

On Aug 11, 2009, at 20:05 , Dino Farinacci wrote:

>>> interface. The implementation cannot scale if it does this. You  
>>> get the level of indirection by doing another lookup in another  
>>> data structure called the "map-cache".
>>

I fully support Dino. When we started to implement LISP, our first  
choice was to create a new logical interface, but we soon realized  
that it will never work.

Luigi

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 11 Aug 2009 11:45:45 -0700

I think this is off topic. If you want to continue the discussion,  
send me email privately.

>>> encapsulates or not based on the contents of the LISP Mapping  
>>> cache and the results of mapping lookups?
>>

Doesn't matter. You can still talk to 100s or 1000s of places, meaning  
you'd have a map-cache that large.

> This could be done in a way that _didn't_ involve a logical  
> interface per map-cache entry, by having the code in the logical  
> interface "driver" maintain and access the map-cache.

Sure, you are using the logical interface for a different reason.

Dino

---

From: Margaret Wasserman <[email-masked]>
Date: Tue, 11 Aug 2009 14:56:25 -0400

On Aug 11, 2009, at 2:45 PM, Dino Farinacci wrote:
>> aware end-node or a home gateway, so I'm not talking about  
>> something that would need to scale to handle 100K simultaneous  
>> connections.
>
> Doesn't matter. You can still talk to 100s or 1000s of places,  
> meaning you'd have a map-cache that large.

Or not.  The whole point of a cache is that it is an optimization...   
I don't connect to more than a couple of dozen hosts regularly, and I  
certainly don't connect to more than a few hundred hosts in the course  
of a few minutes.  So, I could be pretty happy with a cache that had a  
few thousand entries.  I'd just have to do a mapping lookup if my  
destination EID wasn't one of the last (or most frequent, depending on  
my cache aging algorithm) few thousand EIDs I'd visited.

I'm not arguing that this model would work for all ITRs.  Obviously if  
my ISP (Verizon) deploys an ITR that has to handle all of the hosts in  
Massachusetts, it will need to be _considerably_ more capable than my  
Mac or Windows PC.  I'd just like to make sure we also define  
something that can scale down to desktop PCs and to low cost home  
routers, etc.

Margaret

---

From: Luigi Iannone <[email-masked]>
Date: Tue, 11 Aug 2009 20:41:19 +0200

On Aug 11, 2009, at 20:28 , Margaret Wasserman wrote:

> interface per map-cache entry,

Yes you can do that. _But_ this means that your routing table has to  
include entries in order to "route" the packet to the logical  
interface in order to be encapsulated. I do not think this will help  
to solve the scalability issue.

Luigi

> by having the code in the logical interface "driver" maintain and  
> access the map-cache.
>
> Margaret
>

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 11 Aug 2009 12:49:19 -0700

> I'm not arguing that this model would work for all ITRs.  Obviously  
> if my ISP (Verizon) deploys an ITR that has to handle all of the  
> hosts in Massachusetts, it will need to be _considerably_ more  
> capable than my Mac or Windows PC. I'd just like to make sure we  
> also define something that can scale down to desktop PCs and to low  
> cost home routers, etc.

We are. There is nothing prohibiting this usage.

Dino

---

From: Francis Dupont <[email-masked]>
Date: Thu, 13 Aug 2009 16:34:58 +0200

=> this is a typical IPv6 idea (:-)...

=> IMHO the second solution (what I called a virtual link) is simpler.
Of course, it wastes resources so I agree with the "longer than a /64".
In fact, this depends on how many addresses are needed (and don't
forget one can play on both source and destination addresses).

   
=> you know it is not reasonable to introduce a third kind of
IPv6 nodes (hosts, routers and LISP-capable routers :-).

=> I share this concern.

   evolved since then?

=> as far as I know the answer is no. The only different in modern code
is the handling of checksum offloading in NIC hardware.

   Is it possible to have IPv4 UDP checksums enabled  
   (both calculating them and checking them) for some sockets while they  
   are disabled (set to zero, and not checked on receipt) for other  
   sockets?  If so, what sockets option is commonly used for this in Unix  
   operating systems?
   
=> BSDs have only a system-wide sysctl, Linux offers a setockopt()
(SO_NO_CHECK). I don't know for Windows. IMHO, it is a bad idea to
disable UDP checksums, UDP-lite should be used instead.

Thanks

PS: about the original thread (0 checksums vs IPv4 to IPv6 translation),
DNS-OARC DITL should have the data.

---

From: Francis Dupont <[email-masked]>
Date: Thu, 13 Aug 2009 16:56:12 +0200

consequently.

=> I disagree, it is easy to implement LISP in user mode (using
the tun/tap interface/device driver for instance).

Regards

---

From: Luigi Iannone <[email-masked]>
Date: Thu, 13 Aug 2009 17:41:50 +0200

On Aug 13, 2009, at 16:56 , Francis Dupont wrote:

> at

It is also easy to do it with pcap and then you do what you want with  
the packet.

Luigi

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 20 Aug 2009 11:29:35 +0100

I'd be very concerned if the IETF consensus was to introduce a change to 
the UDP checksum without fully evaluating the implications for the 
network and before considering the procedures by which new protocols 
could access a zero-checksum mode.

As I understand, the proposal to update RFC 2460 is NOT just concerning 
LISP routers, there would be host uses, at least for AMT, and likely 
future other applications too, their would be implications on 
middleboxes and possibly on existing applications.

I think the list below is a good summary of key issues. I'd like to use 
this as the basis for a summary of the tradeoffs. I am aware there's 
been quite a lot of discussion on the list while I was on vacation (both 
before and after this email), and also some more quite some months ago 
elsewhere. I'm going to try to compile what I understand under these 
headings... and get back soon.

Gorry

Margaret Wasserman wrote:
> (2) A specific applicability statement for when this mechanism can (and 
> can't)
>       be used.  Would we be allowing this mechanism _only_ for cases where
> (3) A section listing considerations that must be documented for protocols
>     (3b) What happens when the source IP address is corrupted in transit?
>         (3b.ii) What happens when a node that does implement this protocol
>     (3c) What happens if one or both of the UDP ports are corrupted in 
> transit?
>             (Note:  This can happen in IPv4 in the zero checksum case, 
> too, but not
>              NAT boxes, firewalls, etc.), and how will those middleboxes 
> deal with
>              these packet
> 
> I don't really have enough knowledge to answer these questions for LISP.  I
> I was saying, but I'll try to do it once again.  Please correct me if I 
> am missing
> something.
> 
> a specific UDP port assigned to LISP.  Also, I am making the assumption 
> that
> if we write this document, some other protocols (besides LISP) will make 
> use of
>            passed up to UDP (if zero checksums are supported).  At that 
> point,
>            on the LISP port and an ICMPv6 "port unreachable" error will be
>            generated.  (This is different than the IPv4 case, where the 
> packet
> (3a.ii) If a node that does implement LISP receives this packet, it will be
>           using the local port from the original LISP packet.  Will that
>            the corresponding source port (right?).  In that case, I think