Subject: [v6ops] Roman Danyliw's No Objection on draft-ietf-v6ops-ipv6-ehs-packet-drops-06: (with COMMENT)

From: Roman Danyliw via Datatracker <[email-masked]>
Date: Wed, 21 Apr 2021 08:37:32 -0700

Roman Danyliw has entered the following ballot position for
draft-ietf-v6ops-ipv6-ehs-packet-drops-06: No Objection

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-v6ops-ipv6-ehs-packet-drops/

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

** Section 1.
However, common implementation limitations suggest
   that EHs present a challenge for IPv6 packet routing equipment and
   middle-boxes, and evidence exists that IPv6 packets with EHs are
   intentionally dropped in the public Internet in some network
   deployments.

Can you clarify a “common implementation limitations” -- is that the same as
saying vendors aren’t supporting EHs?

** Section 4.  Are there more updated references for the trends identified
in[PMTUD-Blackholes], [Linkova-Gont-IEPG90] and [RFC7892] as the freshest of
these is 6 year old data?

** Sections 6.1 and 6.2 don’t have clear text like Section 6.4 and 6.5 that
explains how layer-4 inspections leads to dropped packets.

** Section 6.4
Use of extension headers can be problematic for NIDS/IPS, since:

   o  Extension headers increase the complexity of resulting traffic,
      and the associated work and system requirements to process it.

   o  Use of unknown extension headers can prevent an NIDS/IPS from
      processing layer-4 information.

   o  Use of IPv6 fragmentation requires a stateful fragment-reassembly
      operation, even for decoy traffic employing forged source
      addresses (see e.g., [nmap]).

-- Per (1), this seems analogous to the argumentation in Section 5 but in that
section there were details around the limitation of hardware support.  Is there
an analogous reference here?  That is, asserting that computation requirements
to process EH is a significant additional load?

-- How is the core issue of (3) different than handling fragments in IPv4
(i.e., operators dropping them at the border and NIPS/NIDS also have to
reassembly them to avoid evasion)?

** Section 6.5.  Thanks for including the reference to [Zack-FW-Benchmark]. 
Due to the age of the benchmark, another sentence explaining how 2013-era
firewalls are still quite similar to those used now would be helpful

---

From: Nick Hilliard <[email-masked]>
Date: Thu, 22 Apr 2021 23:20:39 +0100

Hi Roman,

thanks for the review.

> saying vendors aren’t supporting EHs?

Supporting EHs isn't a binary thing.  You can have different types of 
EHs and lots of EHs. The more complexity attached to inspecting a packet 
to pull out things that are relevant to processing or forwarding it, the 
more likely you're going to run into problems.  The current ipv6 spec 
allows for enough EHs which will fit into a single packet (i.e. the 
first fragment), so a device could be hit with 9000 bytes of EHs on a 
network with large frame support.  Would processing them all at 400Gbit 
line rate be workable across all ports on a device?  Probably not.  Is 
that worth being concerned about?  Definitely not.

> in [PMTUD-Blackholes], [Linkova-Gont-IEPG90] and [RFC7892] as the freshest of

Yep, there are subsequent works listed in the references, for example 
[Huston-2017], [Huston-2020] and [RFC8900].  The problems noted in 
references you mentioned are still outstanding, which is why they're listed.

> explains how layer-4 inspections leads to dropped packets.

Right, statements but no conclusions.  We'll add in at the end of the 
two sections:

6.1: If a forwarding router cannot determine consistent n-tuples for 
calculating flow hashes, data streams are more likely to end up being 
distributed unequally across ECMP and load-shared links.  This may lead 
to packet drops or reduced performance.

6:2: If a network cannot deploy infrastructure ACLs, then the security 
of the network may be compromised due to having more potential attack 
vectors open.

Does this work for you?

> to process EH is a significant additional load?

The figures from [Zack-FW-Benchmark] would be applicable here, as 
NIDS/IPS devices will usually encompass the sort of packet processing 
that a regular stateful firewall would need to do anyway.  We can add 
this in as a reference here too, if you feel it would add to the 
document?  This ref is mentioned in 6.5 already though.

> reassembly them to avoid evasion)?

For example - and we didn't go into this in the draft, but - nested 
fragments are a thing in ipv6:

> https://blog.quarkslab.com/analysis-of-a-windows-ipv6-fragmentation-vulnerability-cve-2021-24086.html

It would be a good idea to take some aspirin before reading this.

> firewalls are still quite similar to those used now would be helpful

Ok good point, and we'll add a note in there:  "Note that although the 
data presented in [Zack-FW-Benchmark] were several years old at the time 
of publication of this document, many contemporary firewalls use 
comparable hardware and software architecture, and consequently the 
conclusions of this benchmark are still relevant, despite its age."

Proposed xml commit to be reviewed by other authors at:

> https://github.com/nickhilliard/ipv6-ehs-packet-drops/commit/1f708b8e96415451aca254e815b69f143e9d4286

Nick

---

From: Tom Herbert <[email-masked]>
Date: Thu, 22 Apr 2021 15:48:05 -0700

> that worth being concerned about?  Definitely not.

Nick,

That then begs the obvious question: what is reasonable number of bytes in
HBH options that we can expect routers to forward? (some number greater
than zero and less than 9K) Unfortunately, this draft doesn't help in
answering that question. This basically my concern with one reason for
drops being characterized as EH makes header chains too big where "too big"
isn't quantified.

Tom

> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops
>

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 23 Apr 2021 00:25:36 +0100

Hi Tom,

Tom Herbert wrote on 22/04/2021 23:48:
> That then begs the obvious question: what is reasonable number of bytes 
> in HBH options that we can expect routers to forward?

The issue of EH recommendations was finalised, I believe, by Rob Wilton 
a couple of weeks ago:

https://mailarchive.ietf.org/arch/msg/v6ops/FvtIsfARK7KOWDeB5HF9YnM5Uvo/

The relevant paragraphs of his email are here:

> Whilst I entirely understand your desire for end hosts to have better
> information about how extension headers may be used, I don't think
> that is the purpose of this document, nor am I willing to send it
> back to the WG to ask them to rework it into something that would be
> a significant larger scope, and would take a long time to collect.
> 
> I'm satisfied that this issue has been discussed and that the
> document represents rough consensus.  I will include a note regarding
> this discussion in my IESG writeup so that the other ADs are aware of
> it when they ballot on the document.

As the authors have been consistently very clear about, your questions 
about sizes / numbers of EHs is work for a future document.

Nick

---

From: Tom Herbert <[email-masked]>
Date: Thu, 22 Apr 2021 20:12:04 -0700

Nick,

I only commented on this because you did mention a specific value in your
response to Roman, namely 9K bytes of EH that you suggest is outside of a
reasonable limit. Regardless of the fact that the draft doesn't suggest
limits, it is obvious people are going to make guesses and conjectures as
you just did.

Tom

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 23 Apr 2021 23:40:34 +0100

Hi Tom,

[punting this back to v6ops, as it is not relevant to iesg last-call]

Tom Herbert wrote on 23/04/2021 04:12:
> I only commented on this because you did mention a specific value in
> your response to Roman, namely 9K bytes of EH that you suggest is
> outside of a reasonable limit. Regardless of the fact that the draft
> doesn't suggest limits, it is obvious people are going to make
> guesses and conjectures as you just did.

Roman made a comment which posited that EH support was a yes/no issue. 
I plucked a hypothetical case out of the air, which was intended to be 
pathological, i.e. 9k jumbo frames (because that's a common jumbo size), 
400g ports on current tech, line rate on all ports, packets containing 
nothing but EHs instead of payload.

The reason was to point out that while technically this should be 
supported by all ipv6 implementations, it's not a configuration that a) 
would ever happen on production networks or b) would ever be useful to 
anyone even if someone produced a box which supported it.

So just to be completely clear, hypothetical cases to demonstrate a 
point like this aren't "guesses and conjectures":  they're thought 
experiments to help us understand what we talk about, and nothing more.

Nick

> 
> Tom
> 
> On Thu, Apr 22, 2021, 7:25 PM Nick Hilliard <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>     Hi Tom,
>

---

From: Tom Herbert <[email-masked]>
Date: Sat, 24 Apr 2021 07:02:10 -0700

Nick,

I'm not sure a thought experiment helps much to understand what is
happening in real operations, actual data is much better. The existing
data, as old as it is, doesn't seem to reflect the scenario you're
describing.

RFC 7872 data shows that 11-15%  of packets with minimal size Destination
Options EH are dropped. This isn't happening because the routers the header
chain length is too long, not because of some nobel cause of security, and
not because a router needs to process options in slow path (routers don't
process DestOpts per RFC8200). The reason is simply that there are devices
don't allow next header for EH, and probably only allow TCP and UDP and no
other protocol numbers. This is not due to the operators intentionally
configuring it, but because some router vendors that made a unilateral
decision that only TCP and UDP (only after be compelled by a behemoth to
allow QUIC) are needed by users and least cost. This drives Least Common
Denominator down to the point that hosts can only send the simplest and
most common protocols into the Internet and hence is the epitome of how the
Internet has become ossified.

Tom

---

From: Nick Hilliard <[email-masked]>
Date: Sat, 24 Apr 2021 16:10:12 +0100

Tom Herbert wrote on 24/04/2021 15:02:
> happening in real operations

ok sure, but backtracking for a moment:

- You're talking about a set of recommendations which would applicable 
to real world operations, so that you have a baseline reference against 
which to write code.

- I was answering a specific question in a review of a document which 
explicitly doesn't address the issue of recommendations by using a 
specific example of something that is outlandish from a practical point 
of view.

I totally get what you're interested in, and why, but can you please 
accept that draft-ietf-v6ops-ipv6-ehs-packet-drops aims only to describe 
a set of symptoms, and not prescribe the cure?

Nick

---

From: Tom Herbert <[email-masked]>
Date: Sun, 25 Apr 2021 06:20:03 -0700

Nick,

Yes, this draft does not prescribe cures, but describes a problem. My
issue with the draft is that the description is inadequate as input to
addressing the issue, specifically the draft describes an operational
problem in entirely qualitative terms. A quantitative analysis of
operational data from which logical conclusions are drawn would be
useful. For instance, in in my last email I did that:

Based on the available data (in RFC7872), there is a ten to twenty
percent probability that packets sent on the Internet containing
extension headers are dropped by an intermediate node whose
implementation disallows extension headers as an IPv6 next protocol.

Do you disagree with this analysis or conclusion?

Tom

---

From: Nick Hilliard <[email-masked]>
Date: Sun, 25 Apr 2021 14:51:38 +0100

Tom Herbert wrote on 25/04/2021 14:20:
> useful.

Yep, I get that + your rationale + that it would be useful.  This point 
was addressed, several times in fact, and the tl;dr response is 1. 
measurements have been addressed in another document (rfc7872), 2. 
specific data regarding the behaviour of specific equipment can be 
notoriously sensitive to many things, including but not limited to: 
hardware type, hardware component revision, software type, software 
revision, microcode revisions, and not least, configuration.  Creating a 
quantitative analysis of this would be expensive and ephemeral, and it 
wouldn't change the outcome or conclusions of either this draft or even 
rfc7872.

The canonical example would be a large chassis based router:  one 
hardware configuration will produce one outcome; a different 
configuration will produce another;  a software revision + microcode 
update might produce a third; swapping out a line card for the same 
model but for HW Rev B01 instead of A01 will produce yet another result. 
  I.e. any specific result would need to be flagged with caveats to say: 
"in this extraordinarily specific set of circumstances, the output is 
X".  Notably, the conclusions of draft-ietf-v6ops-ipv6-ehs-packet-drops 
wouldn't change.

> For instance, in in my last email I did that:
> 

Yep, as you say, that level of conclusion is already reached in rfc7872. 
  We don't need to re-state it in this draft.

fyi, the steps we're broadly aiming at are:

rfc7872: measurements of a problem
draft-ietf-v6ops-ipv6-ehs-packet-drops: analysis of causes
future draft: prescription about what to do

Nick

---

From: Tom Herbert <[email-masked]>
Date: Mon, 26 Apr 2021 05:10:45 -0700

> X".  Notably, the conclusions of draft-ietf-v6ops-ipv6-ehs-packet-drops
As Eric pointed out in his review "Please also note that many high
speed routers are now VMs with normal generic processors in a data
center (of course not in the public Internet).". If you add in all the
servers that run a virtual router to serve VM, it may very well be
that the majority of routers are actually implement in CPU and not in
fixed hardware logic. The processing cost of extension headers (in
both hardware and software implementations), particulary those EH with
TLVs has been noted, and in fact that is why the limits on host
processing of extension headers with TLVs in RFC8504. Given that a
host has to process all extension headers, if it does apply a limit to
what it is willing to accept then that is essentially a de facto limit
on what intermediate nodes will accept. For instance, declaring
default maximum limit of eight non-padding effectively limits HBH to
only have eight options.

>   We don't need to re-state it in this draft.
Can you point out in RFC7872 where that conclusion is articulated or
where it is mentioned in this draft that one of the major reasons that
packets with extension headers are dropped is because some router
implementations don't support EH and disallow all packets with an
extension header as IPv6 next header.

Thanks,
Tom