Subject: Hop-by-Hop Extension Header processed in Slow Path?

From: "Hing-Kam (Kam) Lam" <[email-masked]>
Date: Fri, 4 Feb 2011 06:39:51 +0530

The below white paper from Cisco asserts that most vendors including
Cisco process Hop-by-Hop extension headers in CPU (slow path). Is this
correct?

http://www.cisco.com/en/US/technologies/tk648/tk872/technologies_white_paper0900aecd8054d37d.html

If Yes, then we should not add support for more sub options with HBH header.

Kam

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 3 Feb 2011 21:57:43 -0500

> If Yes, then we should not add support for more sub options with HBH header.

yes, see notes from me (in particular) about this from the last 2+
years... no more HBH header options pls... OR understand that these
may/will get dropped (probably the whole packet actually) at some
provider edges.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 03 Feb 2011 22:11:16 -0500

Lets be a little careful here:
1) If we say "No Extension Headers for intermediate processing", and "No 
Hop By Hop Options", then we are saying that we do not want any 
extensions intended to be processed in intermediate routers.  While I 
personally like that, I want to make really sure that we understand that 
is what we are saying.

2) I have been told that many large rotuers have a setting that causes 
them to ignore the hop by hop options field.  To the degree that is both 
true and turned on, then we need to allow for that in our think (both 
positive and negative.)

Yours,
Joel

On 2/3/2011 9:57 PM, Christopher Morrow wrote:
> On Thu, Feb 3, 2011 at 8:09 PM, Hing-Kam (Kam) Lam<[email-masked]>  wrote:
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------
>

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 3 Feb 2011 22:17:06 -0500

On Thu, Feb 3, 2011 at 10:11 PM, Joel M. Halpern <[email-masked]> wrote:
> 1) If we say "No Extension Headers for intermediate processing", and "No Hop
> By Hop Options", then we are saying that we do not want any extensions
> intended to be processed in intermediate routers.  While I personally like
> that, I want to make really sure that we understand that is what we are
> saying.

My feeling is this:
hop-by-hop processing will happen in slow-path (if you permit it to
happen at all), you can't build a router today with an asic that'll
know how to handle options which are created in the future :(

This means that the gear in the network is subject to failure at
relatively low packet rates (as compared to interface rates), it also
means that the failure is likely not just at the interface level (the
RP/RE/PRP/etc will be doing this processing), much of this box will
fail.

This is an unacceptable outcome.

> 2) I have been told that many large rotuers have a setting that causes them
> to ignore the hop by hop options field.  To the degree that is both true and
> turned on, then we need to allow for that in our think (both positive and
> negative.)

I've mostly seen the 'throw the packet away' option... I agree that if
I can ignore these in all of my routers I don't care what  you do with
them before you see my router or after I pass this off to my friendly
neighbor router. This means that if you expect all intermediate
networks to do "something" for you with this option, you will not get
it, your application and other things should never rely upon this
being functional.

if we get to that then... why are we making these exist anyways?

-chris
(I agree that being clear here is a good plan)

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Fri, 4 Feb 2011 09:35:22 +0530

If there are folks who ignore the hop by hop options field then it makes the entire discussion about not supporting http://tools.ietf.org/html/draft-ietf-6man-exthdr-01 moot. Given this, it makes all the more sense to either proceed with this draft as it is or change it to define a standard format for all IPv6 extension headers that get defined in future (which is precisely what it did in its first version).

Cheers, Manav 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Joel M. Halpern
> Sent: Friday, February 04, 2011 8.41 AM
> To: Christopher Morrow
> Cc: [email-masked]
> Subject: Re: Hop-by-Hop Extension Header processed in Slow Path?
>

---

From: Vishwas Manral <[email-masked]>
Date: Thu, 3 Feb 2011 21:23:42 -0800

Hi Joel,

Options are really bad for fast path processing. To find an option,
the entire list of TLV's needs to be parsed, which is really hard
especially when you have a limited number of clock cycles.

For end-to-end processing however options can be considered better
(especially if the traffic is CPU bound and not BFD or some other
application). Though there are issues even there.

Assume a case where we need to prioritize OSPFv3 packets to the CPU,
we need to put multiple such filters for the same, as the OSPFv3
header can come after a bunch of IPv6 headers.

Thanks,
Vishwas

---

From: Shane Amante <[email-masked]>
Date: Thu, 3 Feb 2011 22:31:38 -0700

Chris,

I think we may want to be a little careful here.  At a minimum, maybe take a deep breath and think about this some, before making a final decision.  :-)  See below.

> know how to handle options which are created in the future :(

Agreed.

> This is an unacceptable outcome.

I agree that's an unacceptable outcome; however, we have come up with solutions for this in the past.  Namely, implement 'proper' rate-limiting at the linecards, in order to prevent overwhelming the RE/RP/PRP, etc. (think: ICMP rate-limiting).  Of course, the key constraint this would place on any "applications" that attempt to use HbH options would be that it couldn't carry any user data traffic at any meaningful bit-rate as it WILL, most definitely, get dropped.  OTOH, if we're talking about periodic/sporadic OAM protocols that are only used on a best-effort basis, (probably by operators, not end-users), then this "limitation" may be acceptable.

> it, your application and other things should never rely upon this

That's certainly another approach.

> if we get to that then... why are we making these exist anyways?

Actually, that latter part is probably the best question.  It would probably be a useful thought exercise to revisit why they were originally created (anybody got pointers to source material?), whether those applications still hold water in today's world and/or if there are new, _practical_ (i.e.: could really get deployed) uses for them.

Of course, there's always the argument that folks may want to use HbH headers in a private network (Enterprise network?) context, but "good luck" on getting them to be recognized or acted upon (at all) by SP routers.  However, it seems like too much work to document even that in a BCP as the boilerplate would likely overwhelm the actual content of said draft/RFC.  (Or, at least I'm not volunteering to do so :)

-shane

---

From: Fernando Gont <[email-masked]>
Date: Fri, 04 Feb 2011 02:24:03 -0300

Manav,

On 04/02/2011 01:05 a.m., Bhatia, Manav (Manav) wrote:
> If there are folks who ignore the hop by hop options field then it
> makes the entire discussion about not supporting
> http://tools.ietf.org/html/draft-ietf-6man-exthdr-01 moot. Given
> this, it makes all the more sense to either proceed with this draft
> as it is or change it to define a standard format for all IPv6
> extension headers that get defined in future (which is precisely what
> it did in its first version).

Huh? I don't follow.

Could you please explain how the current discussion on Hop-by-Hop
extensions relate to the aforementioned I-D?

Thanks,
-- 
Fernando Gont
e-mail: [email-masked] || [email-masked]
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 4 Feb 2011 01:01:36 -0500

> I think we may want to be a little careful here.  At a minimum, maybe take a deep breath and think about this some, before making a final decision.  :-)  See below.

 I don't disagree with being careful.

> I agree that's an unacceptable outcome; however, we have come up with solutions for this in the past.  Namely, implement 'proper' rate-limiting at the linecards, in order to prevent overwhelming th…

You probably also want this to be rate-limited separately from your
example ICMP rate-limit, correct? Do you want to lose functionality or
features because hop-by-hop is over-running the limiter? ICMP limits
(error limits) are often there, because generate of unreachables
happens in hardware in many platforms, to protect OtherPeoplesNetworks
not your device. Limits set against transit traffic (icmp
echo/echo-reply rate-limits) generally require a filter on the
interface (or interface level configuration), that too is sub-optimal.

I could see a limiter for HBH, in a cisco example, at or before the
linecard-CPU (from interface -> LC-CPU -> RP for the example here)
that is built into the hardware and is different from what exists
today for COPP.  Again, this is something that will impact transit
traffic, so how reliable is it going to be? how reliable is it
expected to be?

> Actually, that latter part is probably the best question.  It would probably be a useful thought exercise to revisit why they were originally created (anybody got pointers to source material?), whe…

I'd also be interested in the purpose of these...

> Of course, there's always the argument that folks may want to use HbH headers in a private network (Enterprise network?) context, but "good luck" on getting them to be recognized or acted upon (at…

Yup, the 'but these are only supposed to be used in private networks'
worked well for .. NetBios :( or many, many other protocols. Hoping
that this 'bad thing' will only be used on the 'inside' has a horrible
track record.

-chris

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Fri, 4 Feb 2011 12:23:56 +0530

Fernando,

One of the major reasons given for not accepting this was that no new extension headers need to be *ever* defined in future because you MUST either use hop-by-hop ext header or the destination options ext header.

Cheers, Manav

> From: Fernando Gont 
> [mailto:[email-masked]] On Behalf Of 
> Fernando Gont
> Sent: Friday, February 04, 2011 10.54 AM
> To: Bhatia, Manav (Manav)
> Cc: Joel M. Halpern; Christopher Morrow; [email-masked]

---

From: Fernando Gont <[email-masked]>
Date: Fri, 04 Feb 2011 04:01:26 -0300

Hi, Joel,

> personally like that, I want to make really sure that we understand that

This is the case with v4 already, and there's no reason to expect the
v6-equivalent case to be any different.

> them to ignore the hop by hop options field.  

Yes. Some event drop packets with options in v4. So, again, there's no
reason to expect the v6 equivalent to be any different.

Thanks,

---

From: Fernando Gont <[email-masked]>
Date: Fri, 04 Feb 2011 04:08:44 -0300

Manav,

On 04/02/2011 03:53 a.m., Bhatia, Manav (Manav) wrote:
> One of the major reasons given for not accepting this was that no new
> extension headers need to be *ever* defined in future because you
> MUST either use hop-by-hop ext header or the destination options ext
> header.

What type of options are you envisioning that would not fit in any of
the existing extension headers? -- That was the main argument against
the publication of the aforementioned I-D.

Thanks,

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Fri, 4 Feb 2011 12:53:59 +0530

Fernando,

If you deprecate HbH then if you need an extension that requires such a behavior then you would need a new header. Routers that don't implement that will skip over it, while the ones that understand that will process it. This is different from bundling everything inside HbH where all routers will have to process this packet in slow path.

Cheers, Manav 

> Sent: Friday, February 04, 2011 12.39 PM

---

From: Thomas Narten <[email-masked]>
Date: Fri, 04 Feb 2011 08:35:31 -0500

Gawd, I love these sorts of discussions.

Remind me. How many HBH options has the IETF approved in the last ten
years?

How many proposed HBH options are in the pipeline for approval?

Could we please skip this silly discussion until such a time as
someone actually proposes a HBH option that makes sense, and for which
alternate approaches to solving the problem (that don't use HBH) are
worse? And when we have such a compelling proposal, discuss the
pros/cons of whether it can be deployed, whether routers will actually
implement it, etc.? 

These entirely theoretical discussions are mostly a complete waste of
time.

And to be clear, I suspect we will not be approving any HBH options
any time soon. We know they are generally a bad idea. It is unlikely
that the reasons that HBH are a last resort approach will change
anytime soon.  But can we please just leave things alone as they are
now? The possibility that a HBH may be defined in the future is not
something that is causing us problems today and needs fixing.

Thomas

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 04 Feb 2011 09:00:10 -0500

Your reasoning does not follow, Manav.

Given what we know about packet processing, any extension that believes 
it will be processed by every router along the path is probably fooling 
itself.  Therefore, documenting something that claims to provide that 
functionality is probably a mistake.

At that point, we go to the question of whether, for those packets that 
want hop-by-hop processing by some routers distinct from the end point 
to which the packet is addressed, how should we do it.
1) Don't.
2) Extensions: Note that our underlying documents specifically say not 
to do that, for very good reasons.  Even the new document on extension 
headers was only describing extension headers for end-to-end processing.
3) Use HbH anyway.  There is hardware that can ignore it.  There is 
hardware that can parse it.  We coudl certainly couple that with 
discouraging all such use.

My own preference is (1).  We have found, repeatedly, that pretending we 
can direct packets to every router along a path is just not a good idea. 
  Note that even our OAM fanatics (MPLS-TP) have given up on trying that 
trick.

That of course would bring us back to discussing where extensions with 
end-to-end semantics should go.  I am still unclear why the end-to-end 
options header is not sufficient for those cases.  However, if the WG 
things it can not or should not ban end-to-end extension headers, then 
having a document giving them a more regular format still seems worth 
while to me.

Yours,
Joel

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 4 Feb 2011 10:23:32 -0500

> Gawd, I love these sorts of discussions.
<snip>
> something that is causing us problems today and needs fixing.

fair enough, though having some documented (at least in email)
reasoning that "Oh, a new HBH will be cool!" is bad, certainly isn't
bad, right?

---

From: Thomas Narten <[email-masked]>
Date: Fri, 04 Feb 2011 10:29:45 -0500

> reasoning that "Oh, a new HBH will be cool!" is bad, certainly isn't

Well, it's been documented before in email, and I have little hope
that documenting it again in email will be remembered six months from
now. :)

Perhaps we should take:

                  The case against Hop-by-Hop options
                    draft-krishnan-ipv6-hopbyhop-05

And turn it into (if it doesn't do this enough already) a list of all
the reasons why HBH is problematical, and then suggest that anyone
proposing a HBH in the future include a checklist of how the HBH
approach is still compelling given all the issues listed in the doc.

But I don't think we need a protocol statement banning HBH. The
reasons they are problematical should speak for themselves.

Thomas

---

From: Suresh Krishnan <[email-masked]>
Date: Fri, 4 Feb 2011 11:16:46 -0500

Hi Thomas,

> approach is still compelling given all the issues listed in the doc.

I wrote this because of a frustrating implementation experience I had. I 
periodically republish this draft to serve as a reminder to people but I 
have not put any real effort into this for a few years. I would 
appreciate any help you and Chris could give to set a more positive 
direction for this.

> reasons they are problematical should speak for themselves.

+1.

Thanks
Suresh

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Fri, 4 Feb 2011 22:12:55 +0530

> having a document giving them a more regular format still seems worth 

Which is more or less what that draft does. If you require a draft to document a regular standardized format then that means that you are open to the idea of defining new extension headers in the future. If that's the case then why not use GIEH?

Cheers, Manav

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 04 Feb 2011 18:45:09 +0100

Hi Thomas,

I would like to hear your take on how we should encode conex information 
in IPv6 packets

(http://datatracker.ietf.org/wg/conex/charter/)

This is critical for conex and we are not clear on how we could do this 
in a way that could be deployed.

Thanks in advance for any opinions you can provide on this to us.

Regards, marcelo

---

From: Thomas Narten <[email-masked]>
Date: Fri, 04 Feb 2011 13:05:23 -0500

Marcelo,

Right off, I don't have enough understanding of what kind of
architecture conex envisions to have any useful thoughts, but if you
are thinking of using HBH options, the effort is certainly faces
significant challenges.

But rather than saying "don't use HBH" I'll suggest that you make the
case for why HBH is the least worse way of designing
conex. Understanding that use of HBH for anything implies serious
questions about whether anyone will actually implement/deploy it.

Thomas

---

From: Suresh Krishnan <[email-masked]>
Date: Fri, 4 Feb 2011 17:51:44 -0500

Hi Thomas,

> questions about whether anyone will actually implement/deploy it.

Fully agree with you. Hop-by-hop options are not really suitable for 
conex at all, just like you said. Before the last IETF, we went through 
the alternatives for performing conex markings on IPv6 datagrams. These 
were the requirements we were working with

R1: The information needs to be visible to all interested nodes on the path.
R2: The information needs to be able to traverse nodes that do not 
understand the markings.
R3: The presence of the marking mechanism should not significantly alter 
the processing of the packet.

There were no real suitable mechanisms for IPv6 that met these 
requirements other than using some free bits in the header. We would 
greatly appreciate any hints you can provide us in this regard.

Thanks
Suresh

---

From: John Leslie <[email-masked]>
Date: Fri, 4 Feb 2011 18:49:10 -0500

>> are thinking of using HBH options, the effort is certainly faces

   I don't believe anyone on the ConEx list _likes_ the idea of using
hop-by-hop options -- it's just hard to grok the alternatives.

>> case for why HBH is the least worse way of designing conex.

   IMHO, that's a null case. ;^)

>> Understanding that use of HBH for anything implies serious

   We don't even rise to the level of discussing implementation in
routers.

>     the processing of the packet.

   That's pretty much where we stand.

   Hop-by-hop is pretty-much ruled out. Destination-Options is _so_
confusing that we hestitate to mention it.

   To add to the confusion, our primary algorithm talks of piggy-backing
on the ECN bits, and the most serious proposal so far (developed for
IPv4-land) talks of sort-of-redefining one of the four values of ECN
to show what I'll call "congestion-predicted" and adding one additional
bit to make sure all cases are covered.

   (I don't think it would help at this stage to go into more detail
of that proposal: there clearly are other ways we could cover-the-cases,
and actually it's premature to say we wouldn't want more than one-bit
precision on "congestion-predicted" -- however that proposal _does_
seem adequate for useful experiments. We are, after all, chartered to
produce an Experimental spec.)

> greatly appreciate any hints you can provide us in this regard.

   It's early enough in the ConEx process that we're open to some quite
fundamentally different approaches. For example, we have discussed how
to expose congestion (along the path, at the IP layer) in cases where
ECN isn't deployed. We're exploring the possible -- and not really
expecting to like what we find. ;^)

--
John Leslie <[email-masked]>

---

From: james woodyatt <[email-masked]>
Date: Mon, 07 Feb 2011 11:04:34 -0800

> the publication of the aforementioned I-D.

Once again, I will repeat that I-D.woodyatt-ald envisioned a profile of ICMPv6 messages that need to be processed by firewall devices.

I would have preferred to specify a new class of header extension that is neither hop-by-hop nor destination oriented that would include a new Firewall Alert option.  I suppose a HBH extension would suffice, but it would mean passing a lot of packets through the HBH path in routers that are not also firewalls.  Another option would be to expect firewalls to snoop all the ICMPv6 traffic looking for ALD packets, but that would have negative performance effects as well.  What to do what to do what to do...

In any case, there isn't currently a class of extension headers intended for processing only at some middleboxes and not all routers.  I appreciate that IETF would like to keep the middleboxes from getting too uppity, but that goal is in conflict with IETF recommendations to implement security functions in packet filters integrated with routers, c.f. RFC 6092.  It would be helpful if more participants would bear this in mind, so that we don't have to keep having discussions in which we pretend that the conflict isn't happening when it clearly is an ongoing problem.

--
james woodyatt <[email-masked]>
member of technical staff, communications engineering