Subject: Re: What to improve? BCP-38/SAC-004 anyone?

From: "Patrik Fältström" <[email-masked]>
Date: Fri, 01 Jan 2016 09:04:10 +0100

On 31 Dec 2015, at 18:51, joel jaeggli wrote:

> While we might choose to restrict our definition of edge filtering to
> the single-homed customers of retail ISPs, there are rather a lot of
> those. There is on the other hand a plurality of transit ISPs that
> cannot implement BCP 38 as envisioned or even something close to it.

Completely agree, so why not start with the single home customers. What about look at default configuration of CPEs and alike? What about...I just do not know. Something just must be done.

   Patrik

---

From: Randy Bush <[email-masked]>
Date: Fri, 01 Jan 2016 17:25:22 +0900

> Completely agree, so why not start with the single home customers.
> What about look at default configuration of CPEs and alike?  What
> about...I just do not know. Something just must be done.

go to isoc manrs, sign up to have your logo on the glossies, and then go
home and make major blunders.  after all, a number of big isps are doing
it.

and, in the meantime, ietf idealists can continue to blame operators,
operators can continue to blame vendors (and ietf idealists), and
vendors can ask where the cash comes from.  and therein lies the
disconnect.  the pain is far removed from the basic causes.  this
generally does not work out very well.

randy, who should have stayed out of this silliness

---

From: "Patrik Fältström" <[email-masked]>
Date: Fri, 01 Jan 2016 10:48:39 +0100

On 1 Jan 2016, at 0:50, Michael Richardson wrote:

> (The last one, which seems to have been the largest to date, it is unclear to
> me if it was with forged source address)
>
> http://www.root-servers.org/news/events-of-20151130.txt

For one root server operator view (Verisign) see this post:

<http://www.circleid.com/posts/20151215_verisign_perspective_on_recent_root_server_attacks/>

  Patrik Fältström
  Root server operator...and NTP operator...and IX operator...

---

From: "Christian Huitema" <[email-masked]>
Date: Fri, 1 Jan 2016 09:51:37 -1000

On Thursday, December 31, 2015 10:25 PM, Randy Bush wrote:
> ...

It is pretty clear that BCP 38 is not being deployed because the incentives
are not there. Implementing ingress filtering on the current hardware
doubles the per packet processing time, and that's certainly a disincentive
for operators. It also creates new failure modes for ISP serving multi-homed
customers, and that too is a serious disincentive for operators. In short,
BCP 38 requires operators to increase their cost of operation in order to
protect "the whole Internet" against some forms of attacks. We can call it
tragedy of the commons or whatever, but the reality is that this kind of
mandate almost never gets deployed.

I can think of only one example of such mandates actually being enforced -
the fight against "open mail relays" a dozen years ago. The self-appointed
Internet police, or vigilantes, detected SMTP relays that could forward
spam, shamed them, and blacklisted them until their fixed their setup. The
relay operators could fix their operation, or face customer complaints that
their mail was being rejected. It was bitter, but there are very few open
mail relays left operating now, so in a sense we could say that vigilantism
did work. On the other hand, it is not like spam disappeared.

I shudder at the idea of vigilantes trying to enforce BCP-38 that way. Randy
gently pointed out the disconnect between operators and idealists. An
enforcement campaign complete with blacklists and BGP blocks would do
wonders for that disconnect! Besides, it would only work if we could also
secure BGP, another interesting problem. And even if it "worked," it would
probably not stop denial of service attacks, just like shutting down open
mail relays did not fix spam.

The realist view is thus to deprecate BCP-38. We tried, and we now know that
it cannot be deployed, and certainly cannot be relied on to stop attacks. We
already design new protocols with the assumption that the source IP address
can be forged. Let's fix the old ones. And in particular, let's fix DNS
implementations so they cannot be used as DDOS amplifiers!

-- Christian Huitema

---

From: "John Levine" <[email-masked]>
Date: 1 Jan 2016 21:49:15 -0000

>did work. On the other hand, it is not like spam disappeared.

Actually, the biggest reason open relays went away is that popular
MTAs changed the default configuration to be closed.  The various open
relay testers, one of which I used to run, were mostly used by system
managers to check that their config was really closed.

The various blacklists had some effect and certainly provoked loud
self-righteous complaints, but they've all gone away (except of course
for John Gilmore) since people realize the operational benefits of not
allowing criminals to use their systems to send spam.

I'd hate to think that the relative volume of spoofed packets would
have to approach the relative volume of spam to get people to do
something about it.

R's,
John

---

From: Jari Arkko <[email-masked]>
Date: Mon, 4 Jan 2016 17:37:36 +0200

Address spoofing indeed continues to be a scary problem, particularly
with the v4 space pressures that Patrik mentions.

Still, a couple of useful things to keep mind when looking at this problem.
First, break it down. It isn’t just one problem. For instance, my perception
is that in the consumer subscriber interface situation is pretty good. (But
stats would be useful, does anyone have data?) As you go towards
more complex cases, the cost and false positive problems start
to become significant, which has lead to the problems listed in
this thread. We may still usefully improve some aspects of the
problem. And different tools may apply to the different parts.

Second, I think we need to be realistic with deployment expectations.
I have a very hard time thinking of anything that would be universally
deployed everywhere. No matter how good it is. Deal with it.
This isn’t an excuse for bad solutions, but you have to set your
expectations differently, this isn’t your grandfather’s Internet
any more :-) it is big and not everyone wants to spend their
time updating. I also think some of our canonical examples
of deployment difficulties may need updates, see for instance
IPv6 connectivity for Google US customers [1] :-) I know
this isn’t the end state, but it also very significant, and with
a clear trend.

Third, this whole business reminds me of Marshall Rose’s note

   "once the cable is cut you don't need more retransmissions,
   you need a *lot* more voltage.”

If other people are sending junk your way, *you* don’t need a
new protocol, you need to make those people *care*.

I think the issues are more fundamental and about networking
incentives. I’m also in agreement with Christian’s point about
attacks moving up in the stack if you plug a particular hole at
a particular layer.

I wanted to mention also that for a long time we’ve had some
people in the IETF who wanted to go beyond what current
ingress filtering tools do. SAVI WG, for instance. But once
you have done the simple things, the set of people willing
invest further, e.g., to consider the more complex solutions
will be smaller. I’d love to see more work on this (as long
as everybody understands what they are getting, and the
tradeoffs.)

So what can we do? We can address some subparts of the
problem, e.g., below some ideas from the discussion. Also,
perhaps there’s some completely new idea that is obviously
going to solve this problem. If anybody has that, let the rest
of us know :-)

Jared wrote:

> What I often need are better tools to trace back spoofed packets or mark them.  The
> nice thing about most of these attack networks is they respond faster than I can trace
> and most attacks we see are sub-15 minutes.  The incentives are all wrong here and
> I’d love to talk to people about how to change them.  Some locations, eg: Finland
> have a regulator that does not accept spoofing from the entities they supervise.

Christian wrote:

> We already design new protocols with the assumption that the source IP address

Patrik wrote:

> why not start with the single home customers. What about look at default configuration of CPEs and alike? What about...I just do not know. Something just must be done.

Jari

[1] https://www.google.com/intl/en/ipv6/statistics.html

---

From: Donald Eastlake <[email-masked]>
Date: Mon, 4 Jan 2016 10:48:28 -0500

On Mon, Jan 4, 2016 at 10:37 AM, Jari Arkko <[email-masked]> wrote:

> ....

See draft-ietf-dnsop-cookies.

Thanks,
Donald
=============================
 Donald E. Eastlake 3rd   +1-508-333-2270 (cell)
 155 Beaver Street, Milford, MA 01757 USA
 [email-masked]

...

---

From: Jim Gettys <[email-masked]>
Date: Mon, 4 Jan 2016 11:28:10 -0500

On Mon, Jan 4, 2016 at 10:37 AM, Jari Arkko <[email-masked]> wrote:

>
> Patrik wrote:
>
> > why not start with the single home customers. What about look at default
> configuration of CPEs and alike? What about...I just do not know. Something
> just must be done.
>
>
​Certainly CeroWrt (Dave Taht's version of OpenWrt where much of the
bufferbloat work was done) implements BCP38​. And a home router has to know
what address ranges it is responsible for routing; it makes sense to
delegate the home part of the problem to the home router.

Dave may be able to comment as to whether BCP 38's requirements cause any
compute issues in a home router, given the processors/software on those
devices. It was implemented using the usual Linux packet filtering
mechanism.

The bigger headache is the previously unsolved problem: the very slow
uptake from upstream sources and brokenness of home router market.   I
typically find a minimum of *four years* old firmware packages even on
*brand new *devices on the market, with little sign of security software
updates/fixes.

Here, ISP's that provide home routers could have leverage; but only if
ISP's are willing to make it a hard requirement on purchasing decisions
they make, rather than the currently observed behavior of buying from the
lowest vendor the junk they typically buy today.  The technical side of the
ISP's need to educate the business people that they are encouraging a "race
to the bottom" with possibly catastrophic consequences; BCP 38 is the least
of the problem. I'll take ongoing prompt security updates for the life of
devices such as home routers over BCP 38 any day, and if the devices
continue insecure, BCP 38 is moot, as an attacker will just take over the
router first.

As an industry, this is the bigger challenge.

For more information on the dysfunctional embedded market, see my Berkman
Center talk:
https://cyber.law.harvard.edu/events/luncheon/2014/06/gettys

Jim

---

From: dave taht <[email-masked]>
Date: Mon, 4 Jan 2016 10:26:29 -0800

I note that I am now off all ietf mailing lists - I am trying to focus 
on make-wifi-fast, and while I'd like help on that from any org willing, 
it leaves me with no time to spare on anything else for at least the 
next year. Please do feel free to cc me on items of interest, please 
note my new email address.

I note also that I have also flipped all my email servers to encrypt via 
starttls *always*. Perhaps the ietf could consider that as a weak 
protest in the post-snowden, post CISA era.

After 10+ years of "may" -

It turned out that I had issues with email interchange with 3 out of 100 
of my regular correspondents, which I emailed via a separate channel to 
notify them of the problem, and I am now down to 1 out of hundred that I 
cannot swap mail with - which is a small price to pay compared the total 
silence of the spams, thus far, AND getting a reliable message back when 
something fails to transfer (unlike silent dropping of messages which 
happens far too often these days)

Anyway, moving on below.

On Mon, Jan 4, 2016 at 10:37 AM, Jari Arkko <[email-masked]> wrote:
> Certainly CeroWrt (Dave Taht's version of OpenWrt where much of the
> bufferbloat work was done) implements BCP38. And a home router has to
> know what address ranges it is responsible for routing; it makes sense
> to delegate the home part of the problem to the home router.
>
> Dave may be able to comment as to whether BCP 38's requirements cause
> any compute issues in a home router, given the processors/software on
> those devices. It was implemented using the usual Linux packet
> filtering mechanism.
... BCP38 on openwrt ...

We implemented bcp38 with the powerful ipset mechanism built into linux. 
The package is now part of mainline openwrt, but it is not the default. 
Toke has had a writeup on how it worked that has long needed a venue to 
publish in (circleid? isoc? ). Technically, however, that package is an 
extension of bcp38 - not only does it prevent invalid real source 
addresses from being used from the router, it also stops rfc1918 
addresses from escaping.

Although the package contains a method to handle simple double-natted 
situations, (leveraging the dhcp supplied by upstream), we ran into 
enough places where it wouldn't work without manual intervention for it 
to not be the default for a customer supplied device. An ISP-supplied 
device could probably do it right.

An example of this is that cable modems - even when presenting a real IP 
- have a fixed configuration IP address of 192 dot 168 dot X dot Y. 
(100.1? can't remember)  which you can only get to if you allow that ip 
address through the router otherwise doing bcp38.

(ipset is often used as a means to block bad actors also - for example, 
I am using it right now to block a botnet that insists on registering a 
mailman account of [email-masked] every few seconds. There are 
thus far 20+ different ips so flooding my mailman server alone, there 
must be hundreds or thousands more under attack - totally saturating the 
target accounts at gmail. I have no idea why this thing exists, I could 
be exceptionally paranoid and suspect that this is also a known 
plaintext attack along the starttls channel between me and gmail.)

Dynamic and flexible responses to all sorts of attack vectors should get 
built into everything.

...

Secondly, a lot of my involvement in the ietf was intended to push along 
those actors using other hardware and OSes besides Linux. There are 
plenty of commercial products out there that hopefully have similar 
methods, but I see a remarkable lack of public documentation on how to 
do it right: http://www.bcp38.info/index.php/HOWTO:Juniper for example.

I miss the days when whois actually could get you a sysadmin responsible.

Thirdly, I pushed source specific routing for ipv6 as a default so as to 
avoid the bcp38 problem in the future there - but of course, with 
attackers having 2^64 addresses to choose from or more, it's uglier 
there to start with, and I go back again, to automated, shared, 
distributed defenses.

I would like to see far more 3 way handshakes in key protocols - like 
what rfc6013 promised for dns - in the future.

> *brand new *devices on the market, with little sign of security
> software updates/fixes.
>
> ISP's are willing to make it a hard requirement on purchasing
> decisions they make, rather than the currently observed behavior of
> buying from the lowest vendor the junk they typically buy today.  The
> technical side of the ISP's need to educate the business people that
> they are encouraging a "race to the bottom" with possibly catastrophic
> consequences; BCP 38 is the least of the problem.
Other forms of DDOS attack exist, even with perfect bcp38 coverage, I 
doubt that linode's
recent survival of a massive onslought over the holidays would have been 
made any easier.

http://status.linode.com/

>   I'll take ongoing
> prompt security updates for the life of devices such as home routers
> over BCP 38 any day, and if the devices continue insecure, BCP 38 is
> moot, as an attacker will just take over the router first.

Specifically on the CPE front, I agree with jim's position, that the 
edges have got to get hardened, and regularly updated, and bcp38 is the 
least of the problems there compared to the common wholesale takeovers 
possible.

And even then, well, devices inside the firewall have rather major 
issues also:

http://www.wired.com/2015/07/hackers-remotely-kill-jeep-highway/

To me, bcp38 is something that ISPs should be doing to contain their 
problems inside
their own networks - from their interchange points, to containing the 
edge (bras/dslam/cmtses) to just their allocations. The ipset facility 
already mentioned scales to 10s of thousands of ips/subnets on
really cheap hardware.

> As an industry, this is the bigger challenge.
In the recent savewifi campaign, and the mandates laid out therein, we 
tried to raise the bar for the industry to build better - and 
maintainable, regularly updated stuff. Many of the good actors already 
come close to our suggestions.

http://fqcodel.bufferbloat.net/~d/fcc_saner_software_practices.pdf

We didn't go into legal things - like revising liability law, or 
bringing insurers into the equation that would help enforce those 
mandates, in that piece, but I'm pretty sure at this point that only 
legal/financial threats - like what has hit vw for cheating emissions 
standards - will make for ongoing technical investments in user safety, 
security, and privacy from those companies "responsible".

In context with this, bcp38 compliance could be made a legal or 
insurer's requirement for an ISP to function.

> For more information on the dysfunctional embedded market, see my
> Berkman Center talk:

---

From: "George, Wes" <[email-masked]>
Date: Tue, 5 Jan 2016 20:09:59 +0000

On 12/31/15, 10:57 AM, "ietf on behalf of Kathleen Moriarty"
<[email-masked] on behalf of [email-masked]>
wrote:

>Just a note on an effort by ISOC that is getting some good traction...

http://www.internetsociety.org/doc/addressing-challenge-ip-spoofing is
also a good read, comes out of previous discussion involving some
operators.

Thanks,

Wes

Anything below this line has been added by my company’s mail server, I
have no control over it.
-----------

________________________________

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.