Subject: What to improve? BCP-38/SAC-004 anyone?

From: "Patrik Fältström" <[email-masked]>
Date: Thu, 31 Dec 2015 06:16:40 +0100

Jari,

Thank you for the blog post, and of course as co-chair of the ICG 
working on the IANA Transition I fully support and want to emphasize 
your last bullet, that we need to finalize the IANA transition, finally.

But, I want to mention one detail you might have hidden in one of your 
other bullets, although not explicitly, and that is one we in the 
Security and Stability Advisory Committee of ICANN where I am chair have 
been struggling with for years, and that is source address validation 
(for IP addresses that is). In the IETF there is the well known BCP-38 
that for various reasons is questioned, although "implement BCP-38" is a 
statement used for "do whatever is needed". In SSAC we already in 2004 
created SAC-004 
<https://www.icann.org/en/system/files/files/sac-004-en.pdf> about 
"Securing the Edge". The main author of SAC-004, Paul Vixie, and others 
have after that written numerous other documents on the same topic.

Baseline is, we must do something about it. For some definition of "we" 
and some definition of "something".

This is of course related to what Fred Baker brought up, that we do have 
some tools, but they are not deployed. And if they are deployed, they 
are not deployed to the degree and quality needed to give the intended 
effect.

We do have internationalized email addresses, we do have DNSSEC, we do 
have IPv6, we do have...but how much of this is deployed?

I am sorry to say I see even here on this list many people not using 
technologies they argue about. If I take things I have some clue about, 
I think, DNS, and check the domain names used for the conversation, 
neither DNSSEC, nor IPv6 or any other by the IETF after RFC1035 invented 
technology that is DNS related is in use. By the very same engineers 
discussing how to move standards forward. Is that a sign, or at least 
indication?

Back to the source address validation issues. That the edge can source 
IP packets with fake source IP addresses is a problem. It is, I claim, 
the by far largest problem we have today.

Is this connected to the fact that not even people developing standards 
use very same standards?

Has the IETF ended up being too academic and lost connection to what is 
actually deployed?

Sure, this gap between what is developed in the IETF and what is 
deployed has always existed. Existed when I started be wg chair, 
continued when I was an AD, continued even more when I was on IAB, and 
later ISOC BoT, and now SSAC Chair. And some of the RFCs I have written 
has been excellent examples of standards never taking off(!).

So yes, I blame myself for not having answers to my own questions. If I 
had, I would have pushed for the answers. I have at least myself 
working(?) PGP, DNSSEC, IPv6, NAPTR and many other things. I.e. I am, I 
claim, eating my own dog food.

But, is the taste of our own food so bad we do not eat it ourselves?

If so, how can we make others eat it?

At least some flavor of BCP-38?

Because that is really really the largest issue we have today.

With this, all the best for a successful 2016!

    Patrik Fältström

---

From: Leif Johansson <[email-masked]>
Date: Thu, 31 Dec 2015 11:00:59 +0100

> Is this connected to the fact that not even people developing standards

The problem is that we don't have enough dogs on our tasting panels: we
need to get more ops folks directly & actively involved in the IETF.

Examining some of our success stories I suspect we'd find extensive
involvement from operations in every case.

---

From: tom p. <[email-masked]>
Date: Thu, 31 Dec 2015 11:31:49 +0000

----- Original Message -----
From: "Patrik Fältström" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, December 31, 2015 5:16 AM
Subject: What to improve? BCP-38/SAC-004 anyone?

> the by far largest problem we have today.

I agree that source address validation is by far the largest problem we
have today, but not for IP addresses.  My own personal nightmare is that
90% of the phone calls  I get are fake, phony, fraudulent, phishing and
such like.  I used to be able to block most of them with a single filter
for 'withheld' but then the UK government, perhaps EU-driven, decided
that 'withheld' calls should be deprecated, so the fraudsters now use
SIP etc to generate fake source phone numbers so most of the calls get
through and have to be blocked individually.  The BBC blamed this on the
IETF for failing to introduce checks on the source phone addresses in
the protocol.  Myself I blame the phone companies for not having the
likes of RPF for phone numbers.

I understand that not many will have this precise problem at the top of
their list of nightmares but many will have a closely related one, that
there is little source address validation for e-mails which then
provides a ready vector for fraud.  (I note that I-D announcements have
trebled in size recently because of X- headers added by ESPs but I doubt
that they bring any benefit in this area that offsets the greater cost).

IP addresses are only meaningful to geeks like us.  Users at large see
and use e-mail addresses and phone numbers and, when they have yet to
learn that these can be faked, are susceptible to fraud.  This brings
real pain to real people and could provide a lever to get things done.

Tom Petch

---

From: "Patrik Fältström" <[email-masked]>
Date: Thu, 31 Dec 2015 15:28:43 +0100

> real pain to real people and could provide a lever to get things done.

You are correct on the difference between what is an issue to "normal people" and what is an issue "to us".

And yes, the issues with untrust in SS7, similar to the breakdown I see in X.509 CA environment, is hurting users. Makes it quite hard to set up a 2-factor authentication with recovery mechanisms being secure and trusted (i.e. hint: do not use the same telephone number for your 2-factor authentication as your fallback mechanism).

But not at all hurting the Internet as much as source IP address validation.

Given the depletion of IPv4 address space, we now see increased number of real use of allocated but not announced IPv4 address space. Simply because trust in routing make it easier to just use some space that no one else is using than pay $12/address or whatever it is.

This reuse of already allocated (by others) IPv4 address space has grown faster than I expected.

In the next phase, unless people use IPv6, people will just use whatever IPv4 addresses they can find. And that will hurt the ones having IPv4 addresses in the outskirts of the net. In developing countries. What do an organization in Sweden care if they just reuse whatever some organization in Rwanda use? If announced from Sweden, the addresses will probably work quite well for the Swedish organisation, but so much less for the one in Rwanda which is the real holder of the address space. Creation of local caches from Akamai, Google, Apple, Microsoft, Cloudflare and DNS operators as us as Netnod that moves content closer to Rwanda will for this scenario make the impact smaller. When it hurts. Not if.

Without proper source address validation, it will be worse than a cold on New Years Eve (which I understand many have), and we can never recuperate the situation we sort of have today -- that we can trust the uniqueness and correct (well) route announcement of IPv4.

And THIS is for me the largest driver for IPv6.

That we can not guarantee for how long IPv4 will actually work. For some definition of "we" and some definition of "work".

I am scared.

Really scared.

And if we loose this, then our fight against spam and other for the consumer visible things does not matter.

At all.

   Patrik

---

From: Kathleen Moriarty <[email-masked]>
Date: Thu, 31 Dec 2015 10:57:29 -0500

Just a note on an effort by ISOC that is getting some good traction...

Sent from my iPhone

> On Dec 31, 2015, at 12:16 AM, Patrik Fältström <[email-masked]> wrote:
> 
> Jari,
> 
> Thank you for the blog post, and of course as co-chair of the ICG working on the IANA Transition I fully support and want to emphasize your last bullet, that we need to finalize the IANA transition, finally.
> 
> But, I want to mention one detail you might have hidden in one of your other bullets, although not explicitly, and that is one we in the Security and Stability Advisory Committee of ICANN where I am chair have been struggling with for years, and that is source address validation (for IP addresses that is). In the IETF there is the well known BCP-38 that for various reasons is questioned, although "implement BCP-38" is a statement used for "do whatever is needed". In SSAC we already in 2004 created SAC-004 <https://www.icann.org/en/system/files/files/sac-004-en.pdf> about "Securing the Edge". The main author of SAC-004, Paul Vixie, and others have after that written numerous other documents on the same topic.

ISOC's MANRS effort is helping to make BCP38 recommendations a reality and actively increasing the number of service providers involved.  This might not be the best link, but you can find out a little here and search for more information from there.

https://www.routingmanifesto.org/news/

Kathleen 

> 
> Baseline is, we must do something about it. For some definition of "we" and some definition of "something".
> 
> This is of course related to what Fred Baker brought up, that we do have some tools, but they are not deployed. And if they are deployed, they are not deployed to the degree and quality needed to give the intended effect.
> 
> We do have internationalized email addresses, we do have DNSSEC, we do have IPv6, we do have...but how much of this is deployed?
> 
> I am sorry to say I see even here on this list many people not using technologies they argue about. If I take things I have some clue about, I think, DNS, and check the domain names used for the conversation, neither DNSSEC, nor IPv6 or any other by the IETF after RFC1035 invented technology that is DNS related is in use. By the very same engineers discussing how to move standards forward. Is that a sign, or at least indication?
> 
> Back to the source address validation issues. That the edge can source IP packets with fake source IP addresses is a problem. It is, I claim, the by far largest problem we have today.
> 
> Is this connected to the fact that not even people developing standards use very same standards?
> 
> Has the IETF ended up being too academic and lost connection to what is actually deployed?
> 
> Sure, this gap between what is developed in the IETF and what is deployed has always existed. Existed when I started be wg chair, continued when I was an AD, continued even more when I was on IAB, and later ISOC BoT, and now SSAC Chair. And some of the RFCs I have written has been excellent examples of standards never taking off(!).
> 
> So yes, I blame myself for not having answers to my own questions. If I had, I would have pushed for the answers. I have at least myself working(?) PGP, DNSSEC, IPv6, NAPTR and many other things. I.e. I am, I claim, eating my own dog food.
>

---

From: Jared Mauch <[email-masked]>
Date: Thu, 31 Dec 2015 12:04:07 -0500

> On Dec 31, 2015, at 5:00 AM, Leif Johansson <[email-masked]> wrote:
> 

Sadly that’s not the case here.

The reason we (as an operator) can’t use BCP-38 is the vendor hardware can’t do it at line-rate and the performance hit is too much to sustain.

Sorry to disappoint.  We even dropped it as a requirement in 2015 because it was clear the gap was getting wider not narrower.

Happy to discuss in person in BA or anywhere else we end up at the same time.

- Jared

---

From: Leif Johansson <[email-masked]>
Date: Thu, 31 Dec 2015 18:13:52 +0100

Skickat från min iPhone

> 31 dec. 2015 kl. 18:04 skrev Jared Mauch <[email-masked]>:
> 
> 

I dunno if that is true or not but in either case you just proved my point.

---

From: Jared Mauch <[email-masked]>
Date: Thu, 31 Dec 2015 12:27:36 -0500

> On Dec 31, 2015, at 12:13 PM, Leif Johansson <[email-masked]> wrote:
> 
> 
> 
> Skickat från min iPhone
> 

That’s not to say I’m not in favor of BCP-38, it’s just that we are often the wrong place to do the drops as well, spoofed packets should get addressed.  It’s often hard to describe topologies and while RPSL might be a technique to do it, and it’s a standards based way to describe these things, it’s certainly not commonly used.

- Jared

---

From: joel jaeggli <[email-masked]>
Date: Thu, 31 Dec 2015 09:51:49 -0800

> have after that written numerous other documents on the same topic.

Strict URPF filtering relied on to two convenient fictions:

One that the forward and reverse path, both, carry the route for the
originating prefix. Cases where where this is not the case are both
ubiquitous and totally normal.

Two that URPF is simple to implement in hardware, scales well on all
architectures, and has little cost with respect to forwarding. Simply
put it does not.

While we might choose to restrict our definition of edge filtering to
the single-homed customers of retail ISPs, there are rather a lot of
those. There is on the other hand a plurality of transit ISPs that
cannot implement BCP 38 as envisioned or even something close to it.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 1 Jan 2016 07:54:51 +1300

On 01/01/2016 06:04, Jared Mauch wrote:
...
> The reason we (as an operator) can’t use BCP-38 is the vendor hardware can’t do it at line-rate and the performance hit is too much to sustain.

That seems worth a bit more discussion. I'd always naively assumed that BCP38 was
scalable since all it appears to need is a prefix match, and routers are very
good at matching prefixes; it's just that they don't normally match the source
prefix. Could some router-vendor person comment on this?

There's another issue here, though. BCP-38 and uRPF are also a potential cause of
connectivity problems: https://tools.ietf.org/html/draft-ietf-6man-multi-homed-host

   Brian

---

From: joel jaeggli <[email-masked]>
Date: Thu, 31 Dec 2015 11:15:21 -0800

> prefix. Could some router-vendor person comment on this?

Not all routers use ternary cams , and some that do employ them
algorithmically, so it's not one and done in either case. if you have
multiple depedant memory accesses associated with a match, those need to
be serialized.

If the maximum pps of your linecard drops by say 50% when you enable a
feature that's a bit of a problem.

---

From: Steve Crocker <[email-masked]>
Date: Thu, 31 Dec 2015 14:21:25 -0500

I’ve always assumed the proper place to implement BCP 38 is at the edge of the network.  I’ve also assumed that part of the knowledge one network ought to have about another is whether the other network implements BCP 38 at its edge and requires its peers to do so at their edges and require their peers, etc.  Thus, there ought to be an ever growing collection of networks that have agreed that packets entering their networks have had their source addresses checked at their source.  Checking the source addresses as packets traverse from one tier one network to another or from a tier two network to a tier one network should not be necessary, and I can well understand why this would be a performance issue.

Steve

---

From: Jared Mauch <[email-masked]>
Date: Thu, 31 Dec 2015 14:36:47 -0500

> On Dec 31, 2015, at 1:54 PM, Brian E Carpenter <[email-masked]> wrote:
> 

The problem is you have to lookup on the SRC part of packet in addition with DST part.

These are now two lookups in the forwarding path.  Then you have to decide the policy
(drop|pass|count).  I’ve told vendors I’d like to be able to remark packets that
are a lookup miss.  This means I can easier count/track them in flow data, etc.

But for the small percentage of spoofed packets, the cost on the rest is so high when
we are often PPS limited on even the largest routers.  The 40-byte packet benchmark of
the late 90s isn’t seen today.

> connectivity problems: https://tools.ietf.org/html/draft-ietf-6man-multi-homed-host

Yes, this is a problem, there’s no good way to communicate the “feasible paths” policy
amongst all the devices involved.  Knowing what’s connected or downstream to me is easy,
knowing 1 hop away gets much harder and increases with each node/edge traversed.

- Jared

---

From: "John Levine" <[email-masked]>
Date: 31 Dec 2015 19:41:23 -0000

>The reason we (as an operator) can’t use BCP-38 is the vendor hardware can’t do it at line-rate and
>the performance hit is too much to sustain.

A year or two ago I was talking to some largish providers who told me
that their problem was multihomed customers who have PA space from
multiple providers and for varions reasons (not all silly ones) don't
always send traffic to the port corresponding to the address.  The
usual customer response when asked to clean up their routing is "if
you don't want our $<large> per month, we're sure we can find other
providers that do."

This strikes as a problem that could be solved by providing some way
for routers that don't do BGP to tell their upstreams what the other
address ranges are, but of course the real issue is whether such a
hack could get implemented in routers that people actually buy and
use.

R's,
John

---

From: Jared Mauch <[email-masked]>
Date: Thu, 31 Dec 2015 14:42:02 -0500

> On Dec 31, 2015, at 2:21 PM, Steve Crocker <[email-masked]> wrote:
> 

Sure.  The edge of my network may have a /19 behind it though, or it could be a /126.  I also don’t know if a customer later turns up as another providers backup-path and forgets to or doesn’t inform me.  Customers are often reluctant to describe who are in that downstream cone in advance, either because of poor planning, fear or something else.

I may want to drop packets that I can’t return (eg: 1918 type sources).  This is where I’ve tried to focus, to forward fewer ‘bad’ packets.

Instead we have ended up with a policy to have certain port or protocol numbers enter a ‘junk’ queue that gets policed.  I may not need Chargen or NTP to be more than 1% of my network.

- Jared

---

From: Michael Richardson <[email-masked]>
Date: Thu, 31 Dec 2015 17:59:06 -0500

> need to get more ops folks directly & actively involved in the IETF.

We've said that for years.
My take from what Patrik said is different:
   - we need to demand that people who participate in the IETF process use
     more of the specifications that we write.

What if [email-masked] and all of the [email-masked] WG lists simply didn't
accept email over IPv4?

What if they rejected email that was text/plain with lines longer than 80
characters long? (or 72 or 77, or..)

As the owner of a technology business, I regularly get emails (often with
resume attached), offering me the service to redesign my web site.
Strangely, few of them actually *have* a web site.

I think that Patrik is saying the same thing of the IETF.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Michael Richardson <[email-masked]>
Date: Thu, 31 Dec 2015 18:01:32 -0500

> prefix. Could some router-vendor person comment on this?

It's also really really really cheap to do in the CMTS or PPP concentrator,
where for IPv4, it's often not even a "prefix" machine, but a /32 match.

IPv6 with PD makes it potentially a list...

---

From: Michael Richardson <[email-masked]>
Date: Thu, 31 Dec 2015 18:50:58 -0500

Jared Mauch <[email-masked]> wrote:
    > But for the small percentage of spoofed packets, the cost on the rest
    > is so high when we are often PPS limited on even the largest routers.
    > The 40-byte packet benchmark of

Tragedy of the commons...  the cost here is balanced by the root name server
operators dealing with regular multi-Gb/s attacks.

(The last one, which seems to have been the largest to date, it is unclear to
me if it was with forged source address)

http://www.root-servers.org/news/events-of-20151130.txt

---

From: Jared Mauch <[email-masked]>
Date: Thu, 31 Dec 2015 19:17:59 -0500

> On Dec 31, 2015, at 6:01 PM, Michael Richardson <[email-masked]> wrote:
> 
> 

These often aren’t the devices that are a problem.  The majority of cable/DSL
networks do not permit spoofing.  There are external ways to measure this with
traceroute and data from things like the OpenResolverProject stuff which I worked
on.

I can get a $5/mo server or a $2/mo so-called-booter service to launch attacks from.

What I often need are better tools to trace back spoofed packets or mark them.  The
nice thing about most of these attack networks is they respond faster than I can trace
and most attacks we see are sub-15 minutes.  The incentives are all wrong here and
I’d love to talk to people about how to change them.  Some locations, eg: Finland
have a regulator that does not accept spoofing from the entities they supervise.

It’s one approach, but perhaps doesn’t scale to other markets.

- Jared

---

From: Jared Mauch <[email-masked]>
Date: Thu, 31 Dec 2015 19:20:11 -0500

> On Dec 31, 2015, at 6:50 PM, Michael Richardson <[email-masked]> wrote:
> 
> 

Yup, not news to me (at least).  We have a lot of DNS providers, including
root servers behind our network.  It’s often cheaper to throw more servers
and bandwidth at the problem.

- Jared