Subject: [tsvwg] draft-byrne-opsec-udp-advisory

From: Ca By <[email-masked]>
Date: Mon, 20 Jul 2015 17:44:17 -0700

Dear transport,

Given the number of udp related developments, I created this I-d to express
concern  about further udp work and expound on some operational practices
that conflict with udp growth.

Your feedback is welcome.

=========

A new version of I-D, draft-byrne-opsec-udp-advisory-00.txt
has been successfully submitted by Cameron Byrne and posted to the
IETF repository.

Name:        draft-byrne-opsec-udp-advisory
Revision:    00
Title:        Advisory Guidelines for UDP Deployment
Document date:    2015-07-20
Group:        Individual Submission
Pages:        5
URL:
https://www.ietf.org/internet-drafts/draft-byrne-opsec-udp-advisory-00.txt
Status:
https://datatracker.ietf.org/doc/draft-byrne-opsec-udp-advisory/
Htmlized:
https://tools.ietf.org/html/draft-byrne-opsec-udp-advisory-00

Abstract:
  User Datagram Protocol (UDP) is commonly used as a volumetric attack
  transport on the internet.  Some network operators experience surges
  of UDP attack traffic that are multiple orders of magnitude above the
  baseline traffic rate for UDP.  Application developers should be
  advised that UDP is being rate-limited on a bits-per-second and
  packet-per-second basis by network operators to enforce known good
  baseline traffic levels for UDP. UDP has been abused to such an
  extent that legitimate use may become collateral damage and
  application and protocol developers should avoid using UDP as a
  transport when possible.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

---

From: Joe Touch <[email-masked]>
Date: Tue, 21 Jul 2015 18:08:23 -0700

I disagree with the recommendations as follows:

- avoiding use of UDP as a general recommendation isn't helpful

	UDP should never be used to subvert protections available
	in other transports, e.g., TCP, SCTP, or DCCP

	UDP has its uses and in particular has no direct replacement
	for multicast discovery (whether using mDNS or service-specific
	capability discovery)

- IPsec is appropriate for only some uses
	Notably it cannot easily be used with NAT (you'd end up
	needing to run IPsec over UDP)

- I'm not sure what you mean by "IANA assigned protocol number" for
things like QUIC:
	if you mean port number, I disagree, as it would serve only
	to move the problem to the new port and only consume a new
	port assignment without benefit

	if you mean transport protocol number, that won't be compatible
	with NATs and firewalls that translate or block non-TCP/UDP
	protocols

	in general, assigning new things to new numbers doesn't
	protect it from abuse

I agree with the most of the advice to network operators. IMO, this
document would be better served focusing there.

The exception is to rate-limit UDP. There's no basis for doing so UNTIL
their resources become recipients of an attack (where they could limit
traffic to themselves only) or they experience network congestion. The
latter could be handled with preferential dropping of non-responsive
traffic, and need only be done when resources are limited.

I.e., large UDP flows are not themselves an attack until they are.
Please don't jump to conclusions and act in the absence of need.

Joe

On 7/20/2015 5:44 PM, Ca By wrote:
> Dear transport,
> 
> 
> Given the number of udp related developments, I created this I-d to
> express concern  about further udp work and expound on some operational
> practices that conflict with udp growth. 
> 
> until the htmlized version and diff are available at tools.ietf.org
> <http://tools.ietf.org/>.
> 
> The IETF Secretariat

---

From: Ca By <[email-masked]>
Date: Tue, 21 Jul 2015 19:11:35 -0700

Thanks Joe, for providing feedback

>         for multicast discovery (whether using mDNS or service-specific
Ack.

>         needing to run IPsec over UDP)
Seems the ephemeral existence of  NAT has ossified  your view of the stack

> things like QUIC:

I mean transport protocol

>         with NATs and firewalls that translate or block non-TCP/UDP
Yes, NATs are a challenge for many things. The ietf has specified many
things that pose a challenge to NATs.  Every month that goes by, fewer and
fewer of my subscribers require NAT. Please take note of this reality.

>         in general, assigning new things to new numbers doesn't
But it does help us classify and treat traffic.  This is critical for
network operation. Failure to do so is immediate doom.

In the case of udp, the vast majority of transmitted udp is only ever
destine for the bit bucket.  I speak from a great deal of experience
in dropping udp packet in the name of tactical  network availability.  This
is especially the case in the last 2 years.  It really is war out there to
keep the internet up, and the folks who brought us ntp, snmp, ssdp, chargen
and so on  .... Did not foresee the evil that would be done

> document would be better served focusing there.
In fact, the document is focused at app and protocol developers, but for
good measure i paid dues to bcp38, but it is only address the trigger of a
DRDOS. The napalm is udp and its harmfully deployed protocol.

> traffic, and need only be done when resources are limited.
Joe, as a network operator, i can assure you that 100% of the time, there
is service disrupting udp flows on the internet. Here is a cool visual, i
am not sure it is grounded in fact .... Somebody made it

http://www.digitalattackmap.com/

> Please don't jump to conclusions and act in the absence of need.
I would not dare act in the absence of need.  As a network operator
performing the outlined tactics in the I-D, i have been called to action to
advise the ietf that further udp development efforts will not be met well.
It simply not possible to restore udp to be a first class citizen on the
'net.

I am all for innovation in transport, but not with protocol 17 which is
home to nearly all ddos  modes and vectors. It is simply unconscionable to
pretend udp is a blank slate for transport innovation when it is in fact a
sewer of downtime engendering crime.

I created a visual of my concern
https://twitter.com/theipv6guy/status/623535325915168768

I see certain people grabbing on to udp because they think it is an easy
way to deal with NAT. This is not particularly innovative and
transfers cost to network operators who will now have to work harder to
parse noise from signal on this crazy network of networks.  It would be
better for these ietf innovators to carry the full cost of their innovation
with a new transport protocol number, and thusly separating themselves from
the abomination that is udp reflecting home routers.

Best regards,

Cameron

---

From: Szilveszter Nadas <[email-masked]>
Date: Wed, 22 Jul 2015 10:24:29 +0000

Hi,

I am just wondering, what keeps the “DOS guys” from adopting any new wire format introduced? E.g. the wire format of QUIC or SPUD even if those are not over UPD?

Of course it might take a while before they adapt, but not that long, probably. Is it a strong enough argument that we have a few more months or a year?

Cheers,
Szilveszter

From: tsvwg [mailto:[email-masked]] On Behalf Of Ca By
Sent: Wednesday, July 22, 2015 4:12
To: Joe Touch
Cc: [email-masked]; [email-masked]
Subject: Re: [tsvwg] draft-byrne-opsec-udp-advisory

On Tuesday, July 21, 2015, Joe Touch <[email-masked]<mailto:[email-masked]>> wrote:

Ack.

Yes, NATs are a challenge for many things. The ietf has specified many things that pose a challenge to NATs.  Every month that goes by, fewer and fewer of my subscribers require NAT. Please take note of this reality.

But it does help us classify and treat traffic.  This is critical for network operation. Failure to do so is immediate doom.

In the case of udp, the vast majority of transmitted udp is only ever destine for the bit bucket.  I speak from a great deal of experience in dropping udp packet in the name of tactical  network availability.  This is especially the case in the last 2 years.  It really is war out there to keep the internet up, and the folks who brought us ntp, snmp, ssdp, chargen and so on  .... Did not foresee the evil that would be done

In fact, the document is focused at app and protocol developers, but for good measure i paid dues to bcp38, but it is only address the trigger of a DRDOS. The napalm is udp and its harmfully deployed protocol.

Joe, as a network operator, i can assure you that 100% of the time, there is service disrupting udp flows on the internet. Here is a cool visual, i am not sure it is grounded in fact .... Somebody made it

I would not dare act in the absence of need.  As a network operator performing the outlined tactics in the I-D, i have been called to action to advise the ietf that further udp development efforts will not be met well. It simply not possible to restore udp to be a first class citizen on the 'net.

I am all for innovation in transport, but not with protocol 17 which is home to nearly all ddos  modes and vectors. It is simply unconscionable to pretend udp is a blank slate for transport innovation when it is in fact a sewer of downtime engendering crime.

I created a visual of my concern https://twitter.com/theipv6guy/status/623535325915168768

I see certain people grabbing on to udp because they think it is an easy way to deal with NAT. This is not particularly innovative and transfers cost to network operators who will now have to work harder to parse noise from signal on this crazy network of networks.  It would be better for these ietf innovators to carry the full cost of their innovation with a new transport protocol number, and thusly separating themselves from the abomination that is udp reflecting home routers.

Best regards,

Cameron
Joe

On 7/20/2015 5:44 PM, Ca By wrote:
> until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org>

---

From: Ca By <[email-masked]>
Date: Wed, 22 Jul 2015 05:54:56 -0700

On Wednesday, July 22, 2015, Szilveszter Nadas <
[email-masked]> wrote:

>  Hi,
>
>
>
> I am just wondering, what keeps the “DOS guys” from adopting any new wire
> format introduced? E.g. the wire format of QUIC or SPUD even if those are
> not over UPD?
>
>
>
> Of course it might take a while before they adapt, but not that long,
> probably. Is it a strong enough argument that we have a few more months or
> a year?
>
>
>
>

Good question.

The short answer is that udp's massive install base of dangerously insecure
software is the fundamental problem. This problem is unique to 30+ years of
udp deployment. So, no, the wire format is not the issue. The issue is
insecure apps and negligent stewardship and deployment of nodes

Cheers,
>
> Szilveszter
>
>
>
> *From:* tsvwg [mailto:[email-masked]
> <javascript:_e(%7B%7D,'cvml',[email-masked]');>] *On Behalf Of *Ca
> By
> *Sent:* Wednesday, July 22, 2015 4:12
> *To:* Joe Touch
> *Cc:* [email-masked]
> <javascript:_e(%7B%7D,'cvml',[email-masked]');>;
> [email-masked] <javascript:_e(%7B%7D,'cvml',[email-masked]');>
> *Subject:* Re: [tsvwg] draft-byrne-opsec-udp-advisory
>
>
>
> On Tuesday, July 21, 2015, Joe Touch <[email-masked]
> <javascript:_e(%7B%7D,'cvml',[email-masked]');>> wrote:
>

---

From: Brian Trammell <[email-masked]>
Date: Wed, 22 Jul 2015 14:56:38 +0200

hi Szilveszter, Cameron, all,

(First, Cameron, thanks for the draft; I'm drafting my own response thereto now).

I can't speak to QUIC, and note that SPUD is a prototype (in the sense of "experimental test article", not "alpha version of a future proposed standard")

However I will note that both SPUD and GUE use a magic number at a fixed point in the header. This magic number was in both cases chosen not to collide with valid packet content at the same offsets for a variety of known UDP-based protocols. This is of course completely useless for keeping *direct* sources of UDP attack traffic from being passed by a QUIC or SPUD-"friendly" baseline UDP rate filter: the magic number (and other details of the packet structure) are documented. So yes, useless for attack mitigation against an army of bots.

It does, however, do a lot for reducing *reflected* attack traffic, and I assume the majority of the garbage packets you see in operation are reflected. (Indeed, have you got data for this?)

In a reflection attack. Note that the attacker can only induce the reflectors to generate reply traffic with bit patterns within the set of possible patterns for the protocol used. The "echo" service can be used to reflect packets with valid SPUD magic, but I doubt there's much echo out there (and echo in any case has a reflection factor of 1x, making it not much more attractive that just spoofing source addresses).

We need a better answer for how the UDP-based encapsulations we're designing operate in a high-garbage-packet environment in general. This protection against reflection attack is a part of that answer IMO.

Cheers,

Brian

> On 22 Jul 2015, at 12:24, Szilveszter Nadas <[email-masked]> wrote:
> 
> Hi,
> 
> 
> 
> I am just wondering, what keeps the “DOS guys” from adopting any new wire format introduced? E.g. the wire format ofQUIC or SPUD even if those are not over UPD?
> 
> 
> 
> In the case of udp, the vast majority of transmitted udp is only ever destine for the bit bucket.  I speak from a great folks who brought us ntp, snmp, ssdp, chargen and so on  .... Did not foresee the evil that would be done
> 
> 
> 
> I see certain people grabbing on to udp because they think it is an easy way to deal with NAT. This is not particularlythe full cost of their innovation with a new transport protocol number, and thusly separating themselves from the abomination that is udp reflecting home routers.
> 
> 
> 
> 
> 
> Best regards,
> 
> 
> 
> Cameron
> 
> Joe
> 
> 
> 
>

---

From: Ca By <[email-masked]>
Date: Wed, 22 Jul 2015 07:55:55 -0700

Thanks for the response Brian

On Wed, Jul 22, 2015 at 5:56 AM, Brian Trammell <[email-masked]> wrote:

> I can't speak to QUIC, and note that SPUD is a prototype (in the sense of
> "experimental test article", not "alpha version of a future proposed
> standard")
>
> However I will note that both SPUD and GUE use a magic number at a fixed
> point in the header. This magic number was in both cases chosen not to
> collide with valid packet content at the same offsets for a variety of
> known UDP-based protocols. This is of course completely useless for keeping
> *direct* sources of UDP attack traffic from being passed by a QUIC or
> SPUD-"friendly" baseline UDP rate filter: the magic number (and other
> details of the packet structure) are documented. So yes, useless for attack
> mitigation against an army of bots.
>
> It does, however, do a lot for reducing *reflected* attack traffic, and I
> assume the majority of the garbage packets you see in operation are
> reflected. (Indeed, have you got data for this?)
>
>
yes.

> In a reflection attack. Note that the attacker can only induce the
> reflectors to generate reply traffic with bit patterns within the set of
> possible patterns for the protocol used. The "echo" service can be used to
> reflect packets with valid SPUD magic, but I doubt there's much echo out
> there (and echo in any case has a reflection factor of 1x, making it not
> much more attractive that just spoofing source addresses).
>
> We need a better answer for how the UDP-based encapsulations we're
> designing operate in a high-garbage-packet environment in general. This
> protection against reflection attack is a part of that answer IMO.
>
> Cheers,
>
> Brian
>
>
That is all good to know.  But, the genesis
of draft-byrne-opsec-udp-advisory is trying to create "a light at the end
of the tunnel."  We need a path out of this "DDoS Hell".  Right now, UDP is
largely attack traffic.  And, it is of such a scale that it causes network
meltdowns.  The only acceptable path is to rate-limit the entire
protocol... as unsavory as that might sound, i have been doing this for
years as a last ditch effort to keep the lights on, and it works.  Without
it, we careen from outage to outage.

Loading more types of legitimate traffic into transport protocol 17 makes
my life harder because you are mixing important and legitimate signal
(youtube!) with a sewer of destructive criminal noise (ssdp!)

I can only think of 2 paths forward.  Remove the noise from UDP (aka,
upgrade the internet to stop DRDoS [home routers with SSDP, open DNS and
NTP, booter services...]) or stop assuming your UDP packets will be
transported at scale (aka, the ietf needs to not load new things into UDP
and accept the mess that grew out of UDP... learning from it in new
transport protocols that are not transport protocol 17).

The former is impossible and outside the capabilities of anyone reading
this email.  The latter is inconvenient for some yet-to-be-standardized or
broadly deployed technologies, and it is within the circle of influence
here at the IETF to wind down UDP.

Finally, i would like to touch a 3rd option.  The DRDoS situation is so bad
that it clogs the pipes such that the traffic never reaches the on-premise
"middle-boxes".  Thus we have a new category of hell called the
"middle-clouds" which scrubs traffic in the cloud.  These services inflict
at 60x surchage over what one normally pays for ISP access on a usage
basis, and their marketing puffery is based around how large of a ddos they
have fought off.

You can tell them by the home page buttons that say "I AM UNDER ATTACK!"
that we lowly souls are compelled to click on it in desperation to keep our
networks up.  This market structure means more attacks generate more $$ for
those in the mitigation business.  It's a classic mafia structure where you
pay for protection or get shaken down.

There is even an IETF WG called DOTS that has resigned itself to
perpetually dealing with and signalling the DDoS situation to local and
cloud based middle boxes.  I cannot feel good about the long term vision of
DOTS being broadly deployed and relied on for internet operations.

We need a better solution than constant attacks and escalating tiers of
middle-boxen.

For context, in your favorite search engine, query "network booter" and
"hosted ddos protection".  These market structures are unsettling.  The
IETF needs to steer us out of this, which means "cutting bait" on UDP and
apply lessons-learned to new transport protocols.

Regards,
CB

> > I am just wondering, what keeps the “DOS guys” from adopting any new
> wire format introduced? E.g. the wire format ofQUIC or SPUD even if those
> are not over UPD?
> >
> >
> >
> destine for the bit bucket.  I speak from a great folks who brought us ntp,
> snmp, ssdp, chargen and so on  .... Did not foresee the evil that would be
> done
> >
> >
> >
> > Joe, as a network operator, i can assure you that 100% of the time,
> there is service disrupting udp flows on the internet. Here is a cool
> visual, i am not sure it is grounded in fact .... Somebody made it
> >
> >
> >
> way to deal with NAT. This is not particularlythe full cost of their
> innovation with a new transport protocol number, and thusly separating
> themselves from the abomination that is udp reflecting home routers.
> >
> >
> >
> >
> >
> > Best regards,
> >
> >
> >
> > Cameron
> >
> > Joe
> >
> >
> >
> >

---

From: Joe Touch <[email-masked]>
Date: Wed, 22 Jul 2015 10:34:15 -0700

Hi, Cameron,

On 7/21/2015 7:11 PM, Ca By wrote:
> <mailto:[email-masked]>> wrote:
> 
...
> Seems the ephemeral existence of NAT has ossified  your view of the stack

Ephemeral? All home wifi routers default to NAT and CGNs are becoming
more, not less, ubiquitous as well.

> I mean transport protocol

Thanks for the clarification.

...
> things that pose a challenge to NATs.  Every month that goes by,
> fewer and fewer of my subscribers require NAT. Please take note of this
> reality.  

Very few customers *require* NATs. That was thrust upon us by ISPs as a
way to enforce their pricing model and stretch the IPv4 address space
(depending on who you believe).

I've never been offered non-NAT service for the same price as NAT'd, but
I - like most customers who can tell the difference - would probably
jump at the chance.

> network operation. Failure to do so is immediate doom. 

I don't understand this - it sounds a lot like NIMBY-ism*, i.e., if we
get only legitimate users to shift over to the new protocol then can
dump all the UDP attack traffic.

The flaw in that logic is that attack traffic will just follow the
transport protocol use.

(*https://en.wikipedia.org/wiki/NIMBY)

> destine for the bit bucket. 

A lot of valid UDP traffic uses doesn't make it to the ISP because it's
used for local multicast discovery.

> I speak from a great deal of experience
> in dropping udp packet in the name of tactical  network availability. 
> This is especially the case in the last 2 years.  It really is war out
> there to keep the internet up, and the folks who brought us ntp, snmp,
> ssdp, chargen and so on  .... Did not foresee the evil that would be done

You're arguing for port blocking, which is widely used and can be very
helpful.

> In fact, the document is focused at app and protocol developers, 

OPS isn't the best group in which to accomplish that; if you want the
protocols changed or deprecated, you need to take that issue to the
standards WGs (e.g., TSVWG in this case).

> but for
> good measure i paid dues to bcp38, but it is only address the trigger of
> a DRDOS. The napalm is udp and its harmfully deployed protocol.

You are confusing the application with the transport. UDP is no more the
problem than is IP.

> http://www.digitalattackmap.com/

Interesting; it clearly depends on the date on which you look, which
basically tells me that attackers are wise to the kind of shift you're
proposing - the only thing that will be accomplished by using a new
protocol number is to shift a lot of attack traffic to displaying as
"unknown" in these monitors until they too catch up.

...
> home to nearly all ddos  modes and vectors. It is simply unconscionable
> to pretend udp is a blank slate for transport innovation when it is in
> fact a sewer of downtime engendering crime.

By your logic we should dismantle the Web, DNS, and TCP too - they too
are sources of ongoing attack as well. Frankly, it's all IP, so let's
just shut that down too.

Unfortunately, the real answer involves filtering the attack from
non-attack, not merely labeling everything you either don't understand
or don't expect as malicious.

...
> parse noise from signal on this crazy network of networks.

I.e., to "do their job" :-)

> It would be
> better for these ietf innovators to carry the full cost of their

Even if I accepted your premise (which I do not), the result of your
efforts is only to shift the work back to you - the network operator -
in filtering the attack traffic out of whatever protocol number we shift to.

You won't solve the problem, only move it. That's not helpful.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 22 Jul 2015 10:36:12 -0700

On 7/22/2015 5:54 AM, Ca By wrote:
> 
> 
> On Wednesday, July 22, 2015, Szilveszter Nadas
> <[email-masked] <mailto:[email-masked]>>
> wrote:
> 
...
>     wire format introduced? E.g. the wire format of QUIC or SPUD even if
>     those are not over UPD?____
...
> The short answer is that udp's massive install base of dangerously
> insecure software is the fundamental problem. This problem is unique to
> 30+ years of udp deployment.

And IPv4. And TCP. So let's shut those off too.

I do wish we could simply fix things, but every new OS and application
has only proven an opportunity to repeat our past mistakes (buffer
overruns? seriously?)

So this won't help any more than did Linux relieve us all from the bugs
of Windows and BSD.

Joe

---

From: Ca By <[email-masked]>
Date: Wed, 22 Jul 2015 10:50:37 -0700

> And IPv4. And TCP. So let's shut those off too.
I am working on shutting off IPv4, i think i have made some meaningful
progress, if you are familiar with my work

> So this won't help any more than did Linux relieve us all from the bugs

Here is an important point.  Linux does relieve us from bugs in Windows and
BSD since it helps us avoid a very dangerous mono-culture.

The TCP:80 and TCP:443 and UDP:xyz internet is another dangerous
mono-culture that we need to work against.

---

From: Joe Touch <[email-masked]>
Date: Wed, 22 Jul 2015 11:00:04 -0700

On 7/22/2015 10:50 AM, Ca By wrote:
> 
> On Wed, Jul 22, 2015 at 10:36 AM, Joe Touch <[email-masked]
...
> I am working on shutting off IPv4, ...

Is that helping relieve attack traffic? Or are you finding that attack
traffic on IPv6 is picking up at the same rate as legitimate uses?

> Here is an important point.  Linux does relieve us from bugs in Windows
> and BSD since it helps us avoid a very dangerous mono-culture.   

"Culture" is why some people wear penguins on their tee-shirts, others
wear wobbly-windowframes, etc.

How exactly has this avoided the attacks you're actually concerned with,
though? Code gets reused and ported all over the place. A UDP
vulnerability will just become the same vulnerability on a different
protocol.

> mono-culture that we need to work against.

I hear that the Spain has a few targets for your list*

(*https://en.wikipedia.org/wiki/Tilting_at_windmills)

You appear to be confusing a transport protocol issue with an attack on
your revenue model - which involves doing as little work as possible to
isolate specific attack streams and pointing the finger at transport
protocol designers instead.

And again, IMO making recommendations on how to write apps or design
protocols is out of scope for OPS anyway.

Joe

---

From: Ca By <[email-masked]>
Date: Wed, 22 Jul 2015 12:36:12 -0700

> traffic on IPv6 is picking up at the same rate as legitimate uses?
No, since i am the recipient of IPv4 attack traffic.  The problem will
persist until i can disconnect from the IPv4 internet, which is not on the
horizon.

Separately, i have never seen an IPv6 ddos attack.

> vulnerability will just become the same vulnerability on a different
The hope is that something was learned along the path and applied

> protocol designers instead.

Not sure what you are saying there.

> protocols is out of scope for OPS anyway.
I believe it is in line with the charter opsec.

CB

---

From: Joe Touch <[email-masked]>
Date: Wed, 22 Jul 2015 12:53:47 -0700

On 7/22/2015 12:36 PM, Ca By wrote:
...
> Separately, i have never seen an IPv6 ddos attack.

That might be biasing your logic, FWIW.

...
> The hope is that something was learned along the path and applied 

We have far too much evidence that we don't learn. Buffer overruns have
been around for decades yet they persist in new code.

...
> Not sure what you are saying there.

The test "Proto == UDP" is not itself a signal of attack.

There are plenty of valid UDP uses and UDP applications that either do
not generate substantial traffic or use application-layer congestion
control.

It would certainly be appropriate to limit large amounts of traffic to
ports you know do not serve UDP (e.g., 80, 443) or to known problems
(e.g., chargen). Or flows that are not responsive or otherwise consume
too much resources.

But having app designers cease using UDP won't solve the problem except
in the very short term.

All attack traffic migrates to the ports where there are vulnerabilities.

> I believe it is in line with the charter opsec.

Please review it and let us know why you think that is the case.

Here is my evidence:

Scope:

 The scope of the OPSEC WG includes the protection and secure operation
 of the forwarding, control and management planes. Documentation of
 operational issues, revision of existing operational security practices
 documents and proposals for new approaches to operational challenges
 related to network security are in scope.

and later:

  The OPSEC WG is will not write or modify protocols.

Application design and protocol specification are not operational issues.

Joe

---

From: Ca By <[email-masked]>
Date: Wed, 22 Jul 2015 13:05:47 -0700

> more, not less, ubiquitous as well.
Some perspectives shows 20% penetration of IPv6 in the USA, which is a
great growth from 2010.  NAT  is not dead, but waning.

https://www.google.com/intl/en/ipv6/statistics.html#tab=per-country-ipv6-adoption

> The flaw in that logic is that attack traffic will just follow the
My position is that security is more on the mind of folks now of devs and
operators

> used for local multicast discovery.
The I-D does not have recommendations for this area

> You're arguing for port blocking, which is widely used and can be very
No, the I-D refers to transport protocol rate-limiting.  Whack-a-mole with
UDP ports is not an acceptable approach.

> standards WGs (e.g., TSVWG in this case).
I am not trying to have UDP changed or deprecated.  I am trying to guide
folks away from it and describe what i believe is the operational practice.

> You are confusing the application with the transport. UDP is no more the
They are tied to together.  Apps have done bad things with UDP that have
sunk the UDP boat.

> You won't solve the problem, only move it. That's not helpful.
I have already solved the problem for my network for the current traffic
trajectory. Other networks operators have done the same.   The goal of the
I-D is to provide a final exit strategy and guide against the unwise choice
of doubling down on UDP.

CB

---

From: Joe Touch <[email-masked]>
Date: Wed, 22 Jul 2015 13:43:19 -0700

On 7/22/2015 1:05 PM, Ca By wrote:
...
> great growth from 2010.  

Lars started tracking this when he was my student:

https://eggert.org/meter/ipv6

> NAT is not dead, but waning.

Don't get me wrong; I'll throw a party when that happens, but CGN issues
have become only more frequent in the IETF and I haven't yet seen a home
wifi that didn't default to NAT, so I have no idea why you would believe
this.

...
> No, the I-D refers to transport protocol rate-limiting. 

Agreed; I should have said "you should be arguing for port blocking".

> Whack-a-mole with UDP ports is not an acceptable approach.

Well, the way the Internet works is that communication is negotiated by
mutual consent of the endpoints, which means that you can't merely open
the "safe ports" and shut what you don't understand.

If you did that to me, I'd claim you aren't selling me Internet service
(that argument has been used by others and held up).

> I am not trying to have UDP changed 

Well, you want it basically moved to a new protocol number.

> or deprecated. 

>From your doc:

1. Application and protocol developers should avoid using UDP.

> I am trying to guide

RFC5405 already guides designers to use other protocols where they are
more appropriate, e.g., where UDP is used only as an end-run around
safety, state, or other protection afforded by other transports.

Again, making recommendations to protocol designers isn't within OPS scope.

> They are tied to together.  Apps have done bad things with UDP that have

Attackers have done bad things. There are reasonable UDP apps.

...
> trajectory. Other networks operators have done the same.

So if I try to use UDP over your network for DNS, you'll drop it? E.g.,
if I want to use openDNS instead of the DNS you provide?

> The goal of
> the I-D is to provide a final exit strategy and guide against the unwise
> choice of doubling down on UDP.

Burying UDP inside IPsec is your solution?

Again, all you're asking is for a cheap, dumb fix (block proto=17) which
will be useful only until everyone - app designers and attackers - end
up on another transport.

That's not best practice. That's just practice.

Joe

---

From: Michael Welzl <[email-masked]>
Date: Thu, 23 Jul 2015 08:04:54 +0200

Everybody who knows me knows what’s coming… I’ve been repeating this so often that it makes me feel like a robot   :-(     Joe, Brian, all you folks, you can stop reading here.

> That is all good to know.  But, the genesis of draft-byrne-opsec-udp-advisory is trying to create "a light at the end of the tunnel."  We need a path out of this "DDoS Hell".  Right now, UDP is largely attack traffic. 

There was talk about the feasibility of using a new IP protocol number vs. running everything over UDP. So, is there a way to do it? Is there no way?

=>

It’s not realistic to assume that application developers will (want to) take care of trying to do “the right thing” by not running something over UDP that doesn’t need to run over UDP. You can pray for it, preach it, put it in I-Ds or try to prescribe it: that won’t solve anything. The incentives are wrong.

In case of failure, there needs to be a fall-back (nobody wants to write an application that, for reasons of trying to do “the cleanest approach”, simply cannot communicate). So we’re talking about a piece of code that someone must write, that isn’t as easy as it sounds, that even requires root privileges to run (actually, that may be the most important immediate showstopper), and that will not yield a significant benefit to the application itself. You may be thinking of addressing the latter element in this list by rate-limiting UDP, but this addresses only one part of the problem and harms your existing customers until you do have enough benefit to maybe motivate future app developers to incorporate that. This doesn’t sound like a recipe for success to me  :)

The solution is to break the tie between applications and the protocol that they use, such that the application programmer is no longer the party that has to deal with the onus of trying things "the right way”, figuring out the best protocol and configuration for its purposes. Underneath the application, there should be a transport system that just does its best to act reasonable in the network and provide a good service to the app. That system can be in user space or kernel space - preferably the latter, if only for the necessity to use a different IP protocol number (but other reasons too). This also creates diversity: so far, TCP/IP stacks just try to do the correct thing, and rather minor optimizations make up the difference between one OS or another. In a better world, where applications would never be tied to protocols, my OS could do a far better job at realizing one of the services that an application needs than your OS, because it uses other, different, more … protocols. There are simply not many degrees of freedom available in today’s stacks - they move up to applications that run over UDP.

Yes, trying to fix that will take a long time - but I do believe that there really is no quicker solution for your problem. At least we have a WG now to begin to address this matter:
http://datatracker.ietf.org/wg/taps/charter/ <http://datatracker.ietf.org/wg/taps/charter/>

Yes it sounds academic, but I do hope it can make a real impact - I guess that most of all, this will depend on code. We need folks who want (and can) implement a TAPS system (but the work of the group has only just begun, we’re still far away from this most exciting bit). Join the list and contribute and help us get there.

Cheers,
Michael

---

From: Brian Trammell <[email-masked]>
Date: Thu, 23 Jul 2015 15:50:15 +0200

hi Cameron, all,

(Copying the IAB Stack Evolution program as well...)

> On 22 Jul 2015, at 16:55, Ca By <[email-masked]> wrote:
> 

Cool. So, if we assume that we can find a way to quickly and easily distinguish bad old UDP ("all" of which is reflected crap, for some high value of all) from UDP-as-layer-boundary-reinforcement -- a well-selected magic number being the most obvious way (to me) to do this statelessly.

> That is all good to know.  But, the genesis of draft-byrne-opsec-udp-advisory is trying to create "a light at the end of the tunnel."  We need a path out of this "DDoS Hell".  Right now, UDP is largely attack traffic.  And, it is of such a scale that it causes network meltdowns.  The only acceptable path is to rate-limit the entire protocol... as unsavory as that might sound, i have been doing this for years as a last ditch effort to keep the lights on, and it works.  Without it, we careen from outage to outage.
> 
> Loading more types of legitimate traffic into transport protocol 17 makes my life harder because you are mixing important and legitimate signal (youtube!) with a sewer of destructive criminal noise (ssdp!)

Reductio ad absurdum, you could say the same thing about IPv4. :) But I get your point...

So, there are broadly two things I mean when I say UDP:

(1) Packets with a protocol identifier of 17 on them. This is what you want to filter on because most of them are crap.

(2) What happens when you open a SOCK_DGRAM socket. This service is basically "nonprivileged userspace raw sockets". When we're talking about making it possible to deploy new transport protocols, the "userspace" is IMO crucial here (and the "nonprivileged" equally so if you're talking about the new world where everything happens on devices where the user has no root).

If we had a way to get this second thing without throwing a UDP header on it, that'd be great, but whatever it is you need port numbers on it, more because you can have multiple userspace processes on each eventual endpoint than you need to get around NAT.

> I can only think of 2 paths forward.  Remove the noise from UDP (aka, upgrade the internet to stop DRDoS [home routers with SSDP, open DNS and NTP, booter services...]) or stop assuming your UDP packets will be transported at scale (aka, the ietf needs to not load new things into UDP and accept the mess that grew out of UDP... learning from it in new transport protocols that are not transport protocol 17).

I see the use of UDP here as a transition technology: that is, something that will be deployed for the next twenty years or so. The eventual goal (at least mine) is to be able to do all of this over a new protocol. The stack looks something like this:

   |  transport services API (TAPS)  |
   +---------------------------------+
   | shiny new transport protocol(s) |  encrypted
==========================================
   |   explicit cooperation layer    |  cleartext
   +-----------------+               |
   |       UDP       |               |
   +----------+------+---------------+
   |    IPv4  |           IPv6       |
   +----------+----------------------+

    ------------ time -------------->

> The former is impossible and outside the capabilities of anyone reading this email.

I'd submit that if a good-UDP / bad-UDP discriminator, as well as return routability proof and state exhaustion resistance, can be designed into future good-UDP protocols, that's a start.

> The latter is inconvenient for some yet-to-be-standardized or broadly deployed technologies, and it is within the circle of influence here at the IETF to wind down UDP.

This is tantamount to saying the future of the Internet is over TCP, which is not a particularly useful future. The inability to get new transport protocols deployed isn't just an issue with NATs, it's an issue with the kernelspace/userspace boundary in the endpoints. This has more than NAT to do with why we failed to deploy SCTP.

One thing that has changed since we failed to deploy SCTP: machines (kernel instances) are silly cheap on the server side now, so building new transports into your servers, which used to be kind of an operational pain, is actually pretty easy. Another thing that has changed: most people are connected to the Internet on machines they don't even have administrative privilege: even if they want to, they can't add new protocols to the kernels.

Something that probably hasn't: default firewall rules.

<shameless plug>But this is something we should measure. Interested people, come to HOPSRG on Friday morning.</shameless plug>

Cheers,

Brian

> Finally, i would like to touch a 3rd option.  The DRDoS situation is so bad that it clogs the pipes such that the traffic never reaches the on-premise "middle-boxes".  Thus we have a new category of hell called the "middle-clouds" which scrubs traffic in the cloud.  These services inflict at 60x surchage over what one normally pays for ISP access on a usage basis, and their marketing puffery is based around how large of a ddos they have fought off.
> 
> You can tell them by the home page buttons that say "I AM UNDER ATTACK!" that we lowly souls are compelled to click on it in desperation to keep our networks up.  This market structure means more attacks generate more $$ for those in the mitigation business.  It's a classic mafia structure where you pay for protection or get shaken down.
> 
> There is even an IETF WG called DOTS that has resigned itself to perpetually dealing with and signalling the DDoS situation to local and cloud based middle boxes.  I cannot feel good about the long term vision of DOTS being broadly deployed and relied on for internet operations.
> 
> We need a better solution than constant attacks and escalating tiers of middle-boxen.
> 
> For context, in your favorite search engine, query "network booter" and "hosted ddos protection".  These market structures are unsettling.  The IETF needs to steer us out of this, which means "cutting bait" on UDP and apply lessons-learned to new transport protocols.
> 
> Regards,
> CB
> 
> 
> 
> 
>

---

From: Ca By <[email-masked]>
Date: Thu, 23 Jul 2015 22:04:06 -0700

On Thursday, July 23, 2015, Brian Trammell <[email-masked]> wrote:

> > On 22 Jul 2015, at 16:55, Ca By <[email-masked] <javascript:;>>
> wrote:
> >
> > On Wed, Jul 22, 2015 at 5:56 AM, Brian Trammell <[email-masked]
> <javascript:;>> wrote:
> >> I can't speak to QUIC, and note that SPUD is a prototype (in the sense
> of "experimental test article", not "alpha version of a future proposed
> standard")
> >>
> >> However I will note that both SPUD and GUE use a magic number at a
> fixed point in the header. This magic number was in both cases chosen not
> to collide with valid packet content at the same offsets for a variety of
> >> It does, however, do a lot for reducing *reflected* attack traffic, and
> I assume the majority of the garbage packets you see in operation are
> Cool. So, if we assume that we can find a way to quickly and easily
> distinguish bad old UDP ("all" of which is reflected crap, for some high
> value of all) from UDP-as-layer-boundary-reinforcement -- a well-selected
> magic number being the most obvious way (to me) to do this statelessly.
>
>
I would simply call it udp2 with a new transport protocol number.

But I imagine you want to preserve some way to blast through legacy nats,
which is not a good strategic goal / constraint

> > That is all good to know.  But, the genesis of
> draft-byrne-opsec-udp-advisory is trying to create "a light at the end of
> the tunnel."  We need a path out of this "DDoS Hell".  Right now, UDP is
> > Loading more types of legitimate traffic into transport protocol 17
> makes my life harder because you are mixing important and legitimate signal
> (1) Packets with a protocol identifier of 17 on them. This is what you
> want to filter on because most of them are crap.
>
>
Yes

> (2) What happens when you open a SOCK_DGRAM socket. This service is
> basically "nonprivileged userspace raw sockets". When we're talking about
> making it possible to deploy new transport protocols, the "userspace" is
> IMO crucial here (and the "nonprivileged" equally so if you're talking
> about the new world where everything happens on devices where the user has
> no root).
>
>
Udp2

> If we had a way to get this second thing without throwing a UDP header on
> it, that'd be great, but whatever it is you need port numbers on it, more
> because you can have multiple userspace processes on each eventual endpoint
> than you need to get around NAT.
>
>
I am optimistic about IPv6 deployment and the removal of nats and middle
boxes.  This is tied to LTE capacity  and latency improvements outpacing
middle box utility and capacity. I find it strange that the ietf is trying
now to talk to middle boxes while they are waning. YMMV.

> I see the use of UDP here as a transition technology: that is, something
> that will be deployed for the next twenty years or so. The eventual goal
> (at least mine) is to be able to do all of this over a new protocol. The
> stack looks something like this:
>
>
> this email.
> I'd submit that if a good-UDP / bad-UDP discriminator, as well as return
> routability proof and state exhaustion resistance, can be designed into
> future good-UDP protocols, that's a start.
>
>
But it does not solve change anything in the I-D at hand, as the I-D is
tactical and operational in nature

> > The latter is inconvenient for some yet-to-be-standardized or broadly
> deployed technologies, and it is within the circle of influence here at the
> IETF to wind down UDP.
>
> This is tantamount to saying the future of the Internet is over TCP, which
> is not a particularly useful future. The inability to get new transport
> protocols deployed isn't just an issue with NATs, it's an issue with the
> kernelspace/userspace boundary in the endpoints. This has more than NAT to
> do with why we failed to deploy SCTP.
>
> TCP is not a useful future?

>From where I sit, udp is less than 5% of traffic at peak, except during
attacks when it goes north of 50%.  This is why policing udp is low risk
and high reward in real networks.

Put the good udp in IPSec. Better, put sctp on the wire or in IPSec. Xbox
is doing this with IPv6

https://www.nanog.org/sites/default/files/wed.general.palmer.xbox_.47.pdf

> One thing that has changed since we failed to deploy SCTP: machines
> (kernel instances) are silly cheap on the server side now, so building new
> transports into your servers, which used to be kind of an operational pain,
> is actually pretty easy. Another thing that has changed: most people are
> connected to the Internet on machines they don't even have administrative
> privilege: even if they want to, they can't add new protocols to the
> kernels.
>
>
Failed to deploy sctp ?!  SCTP is just needs a chance.

The good news is, people don't have admin privilege so sctp can be brought
down from on high and deployed without any intervention from people who
don't know / care

Google and Apple can push sctp to a billion devices and boom, it is good
(some assembly required)

But, they won't even deploy edns0  ...

>
> <shameless plug>But this is something we should measure. Interested
> people, come to HOPSRG on Friday morning.</shameless plug>
>
Cheers,
>
> Brian
>
> > Finally, i would like to touch a 3rd option.  The DRDoS situation is so
> bad that it clogs the pipes such that the traffic never reaches the
> on-premise "middle-boxes".  Thus we have a new category of hell called the
> > > On 22 Jul 2015, at 12:24, Szilveszter Nadas <
> [email-masked] <javascript:;>> wrote:
> > >
> > > Hi,
> > >
> > >
> > >
> > > From: tsvwg [mailto:[email-masked] <javascript:;>] On Behalf
> Of Ca By
> > > Cc: [email-masked] <javascript:;>;
> [email-masked] <javascript:;>
> > > On Tuesday, July 21, 2015, Joe Touch <[email-masked] <javascript:;>>
> wrote:
> > >
> > > In fact, the document is focused at app and protocol developers, but
> for good measure i paid dues to bcp38, but it is only address the trigger
> of a DRDOS. The napalm is udp and its harmfully deployed protocol.
> > >
> > >
> > >
> > > I am all for innovation in transport, but not with protocol 17 which
> is home to nearly all ddos  modes and vectors. It is simply unconscionable
> to pretend udp is a blank slate for transport innovation when it is in fact
> a sewer of downtime engendering crime.
> > >
> > >
> > >
> > > I see certain people grabbing on to udp because they think it is an
> easy way to deal with NAT. This is not particularlythe full cost of their

---

From: Brian Trammell <[email-masked]>
Date: Fri, 24 Jul 2015 15:00:40 +0200

[...and now copying the newly-created stackevo-discuss list, just created as a place for public discussion on IAB stack evolution program related topics...]

> On 24 Jul 2015, at 07:04, Ca By <[email-masked]> wrote:
> 

<snip>

> I would simply call it udp2 with a new transport protocol number.

If we're going to do that we might as well fix other things with UDP. I see this as an excellent way forward for the second stage of this transition.

> But I imagine you want to preserve some way to blast through legacy nats, which is not a good strategic goal / constraint

It'd be nice, yes, but it's not the main concern.

On the NAT question, I'd have to see data to convince me that the proportion -- internet-wide -- of connections from end-user / that are NATted (in a way that a *new* UDP that has port numbers in the same place in the header won't make it through, which is I assume today all NAT devices). V6 takes a lot of address pressure away, true, and careful engineering of V6-enabled CPE can reduce NATting. But V6 only access connectivity to services that are V4 only itself requires NAT. So the slope points downward but I don't see NAT being an insignificant part of the deployed architecture for a long time. We want something that works tomorrow, not someday, with negligible risk to connectivity. Shiny new features that break the delivery of cat videos they promise to improve get switched off.

More important is that basically every programming environment on every internet connected platform can give applications access to UDP/17. It will take a long time for that to be the case with UDP2/144 (or whatever number this protocol would be assigned). I hope that the curve for getting kernel support for these things is faster than it was with e.g. SCTP: endpoint operating system maintenance has matured a lot in the past two decades. But it's still a long curve.

> I am optimistic about IPv6 deployment and the removal of nats and middle boxes.

I'm far more optimistic about the former than the latter. Just because one enables the other doesn't mean that it will.

>  This is tied to LTE capacity and latency improvements outpacing middle box utility and capacity.
> I find it strange that the ietf is trying now to talk to middle boxes while they are waning. YMMV.

For my part, I see this effort as an attempt to make interactions with middleboxes *explicit* as opposed to implicit; this can happen even as more paths get more transparent, and the diagnostics enabled by this layer can accelerate this trend.

> But it does not solve change anything in the I-D at hand, as the I-D is tactical and operational in nature

It would, were it deployed: there would be an easy way to determine reflected crap from nearly-certainly-not-reflected not-crap. The I-D at hand is a good collection of observations about why UDP might not be appropriate here. It is not clear that generalizing these observations to the wider Internet is useful guidance in and of itself ("deprecate UDP"), though it can be input to efforts to improve the situation on UDP ("reduce reflected crap until you can deprecate UDP").

> TCP is not a useful future?

TCP *alone* is not a useful future. A future where all traffic runs over TCP is a missed opportunity. Most network applications don't necessarily fit on a transport whose API was designed to make the network look like a tape drive, and wedging them into a stream API is inefficient and brittle. Loss-based TCP congestion control algorithms (basically all of the ones deployed for general use) don't play particularly nicely with link layers designed without a recognition that TCP *must* induce loss as a congestion control signal.

> From where I sit, udp is less than 5% of traffic at peak, except during attacks when it goes north of 50%.  This is why policing udp is low risk and high reward in real networks.
> 
> Put the good udp in IPSec. Better, put sctp on the wire or in IPSec. Xbox is doing this with IPv6
> 

Indeed, the diagram above looks a lot like a super-simplified version of the diagram in the IPsec RFC that says where you stick various IPv6 extension headers when using ESP, if you put the explicit cooperation stuff in hop by hop and destination options. But it's not clear to me how I would ship an app over SCTP over transport mode ESP when I'm not also the administrator *and* the vendor of the OS running on the endsystem, and it's not clear to me how I would handle keying for such a thing without a closed management framework.

So I still think we need something else here.

Cheers,

Brian

---

From: Joe Touch <[email-masked]>
Date: Fri, 24 Jul 2015 10:13:32 -0700

I'm left with the following questions:

- if you want a new UDP protocol number, why not just us 33 (DCCP)?

- if NATs would not be an issue to a new transport, why has it inhibited
all the recent attempts? (DCCP, SCTP)

Although I appreciate the implications on TAPS for "do what I mean"
negotiation of alternate transports, TAPS doesn't solve either of the
issues above.

AFAICT, they both highlight:

a) why this is hard

b) why this doesn't actually need a new IANA transport protocol codepoint

Joe

---

From: Dave Taht <[email-masked]>
Date: Fri, 24 Jul 2015 19:43:58 +0200

I note that udplite is widely available, understood by bsd and linux,
passed by at least a few firewalls, and if you don't use the reduced
checksum feature, indistinguishable from udp... and one line of app
code to use...

... and widely underused. I did quite a bit of testing of it on ipv6
paths, and even converted a few tools like mosh and netperf to use it
for a while. Attempting to at least measure how well udplite gets
around would be a starting place for a new protocol, and at least
probing for the ability to use it in, say, quic might make some
headway.

My overall objection, however, to a new connectionless protocol for
"good udp applications" is that we had no idea how many crap udp
applications we had at the time we wrote them.

-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

---

From: Joe Touch <[email-masked]>
Date: Fri, 24 Jul 2015 10:52:14 -0700

Hi, Dave,

Yes, I should have included UDPLite in the list below and it's a very
direct replacement for UDP when used as you indicate.

However, one goal of a transition out of UDP might be to include
integrated congestion control, for which either SCTP or DCCP would be a
better alternative.

It would be interesting to know if you've tested UDPLite through
consumer-grade NATs or across any CGNs - i.e., network devices that need
to keep transport-based state, not just firewalls.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sat, 25 Jul 2015 22:17:44 -0400

> This is tantamount to saying the future of the Internet is over TCP, which is not a particularly useful future. The inability to get new transport protocols deployed isn't just an issue with NATs,…

Hi Brian,

Can you provide any more context to support this statement? SCTP has
been around a long time, and also there is SCTP-over-UDP which should
already facilitate userspace implementation of SCTP (in fact that
there is a publicly available userspace SCTP stack
(https://docs.google.com/presentation/d/190sBrVsLICDn6ayni9bYTjZNRUgpOjlvUhbIVhyjfpA/edit#slide=id.p).
When I Googled "why isn't SCTP deployed", I get
http://stackoverflow.com/questions/1171555/why-is-sctp-not-much-used-known
which seems to mostly imply that SCTP is not deployed because home
routers can deal with it (at least as of 2011). I suppose the demise
of SCTP deployment could be a classic chicken-and-the-egg problem: OS
vendors won't invest in a protocol that can be reliably be transported
over the Internet, and router vendors are probably unwilling invest in
support for protocols that aren't widely being used.

In any case, for any proposed transport protocol, the questions of why
existing protocols (e.g. SCTP, DCCP) aren't sufficient or are
chronically non-deployable will be undoubtably raised. IMO, we should
separate out transport protocol definition from the delivery mechanism
intended to facilitate the short term or "transitional" goal of
deployment on the current Internet (e.g. encapsulation in UDP). SCTP
already demonstrates this model since it is both a native IP protocol
(132) and can be encapsulated in UDP (RFC6951). Encapsulation of
pretty much any IP protocol (foo-over-UDP for TCP, DCCP, etc.) is
straightforward, and in fact GUE already defines a common and
extensible method that allows encapsulation of any IP protocol over
UDP without needing a port number for each protocol.

As for the desire to run transport protocols in userspace, I would
point out that transport layer stacks are *not* implemented in a
vacuum. For instance, a fully functional and performant kernel TCP
stack will directly interact with the IP layer protocols, driver
layer, the device, and other sub-systems in the kernel. Interactions
with IP protocol layer are needed to support PMTU discovery, explicit
congestion notification, and network QoS via diff-serv. For PMTU
discovery, we need an interface to set DF for IPv4 and to get ICMP PTB
messages, for ECN we need to get bits from the IP header of marked
packets, for diff-serv we need a mechanism to set DSCP bits. I would
be surprised if support for these were not requirements of a new
transport protocol intended for Internet scale.

Other than that, there are probably >50 places (at least in Linux
stack) where TCP interacts with other layers. This includes the
routing layer (like re-routing for failing connections), saving of
routing path metrics such as ssthresh and ICW which can be applied to
new connections to a common destination, interactions with queuing
layer to prioritize and optimize transmission (TCP sockets, unlike UDP
sockets can be directly flow controlled by local egress queues). There
is a rich set of protocol offloads in devices to provide checksum
offload, LSO, LRO, as well as software analogues in GSO and GRO. The
stack interacts with packet steering mechanisms (deliver packets to
right CPU), pacing mechanisms to mitigate the effect of packet bursts
within a flow, has interactions with DOS mitigation mechanisms,
interactions with memory management for network buffers. There are
interfaces to set IP options or extension headers, appropriately set
IPv6 flow labels. There's also newer interactions with mechanisms to
fight buffer bloat such as TCP small queues and Byte Queue Limits
(BQL). Kernels also implement protocol aware NAT and packet filtering
per TCP flags and fields. For UDP specifically, there are newer socket
APIs which can be used improve efficiency of the UDP data path
(sendmmsg, recvmmsg). If a new transport protocol is intended to serve
high throughput (like for serving video) then support for sendfile
might be important
(https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf is interesting
reading on where kernel support is likely headed for that).

For many of these items there are already kernel interfaces that a
user space stack could use to get of set the appropriate information
(will vary between OSes), for some newer features new interfaces will
need to be developed, for a few extending support into userspace might
not be feasible. Support for some features, such as LSO or LRO, would
require the kernel or device to understand the an encapsulated
transport protocol as well.

Along these lines, for anyone who is planning on attending Linux
Plumber's Conference next month we're planning a BOF to discuss kernel
support for user space transport protocols. We are posing two
questions: 1) is there anything we can do from the kernel side to
address the perception that OS development and deployment impedes
advancements in transport protocols 2) assuming user space transport
protocols become popular, what sort of support and interfaces can we
provide to optimize them (like for above interactions).

Thanks,
Tom

---

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 27 Jul 2015 13:30:54 +0200

Hi CB,

> Am 24.07.2015 um 07:04 schrieb Ca By <[email-masked]>:
> 

Please note that they encapsulate IPsec/IPv6 over UDP over IPv4 for IPV4 support. Yet another example where UDP is used.

Mirja

---

From: Brian Trammell <[email-masked]>
Date: Tue, 28 Jul 2015 10:27:48 +0200

Hi, Tom,

Thanks for your message. A partial reply for now, may revisit a few bits later once I've chewed through the post-IETF queue a bit...

> On 26 Jul 2015, at 04:17, Tom Herbert <[email-masked]> wrote:
> 

First, I should have used a different tense in my statement above:

"This has more than NAT to do with why we *have* failed to deploy SCTP." We haven't deployed SCTP widely *yet*, but history isn't over yet. If I wasn't optimistic, I wouldn't be posting to a list called "stack evolution". :)

I agree with you that it's chicken-and-egg. But it would be easier to conjure chickens from the vacuum without needing to hack the kernel to do so.

As someone who was introduced to SCTP by Randall's chapter in the third edition of Stevens, I was excited about using the features it described (at the time, transport-layer multihoming and multistreaming) in my applications. I then had occasion to build a compliant IPFIX implementation in 2006 (for which SCTP is the MTI), and suddenly found myself wedged between a maze of kernel patches and userland implementations on the one side and the depths of the OpenSSL codebase trying to wedge DTLS atop SCTP on the other -- for that was the way you were supposed to do security for SCTP before we decided reversing the stack was the way to go and put SCTP on DTLS on UDP. I was not a good enough programmer to get it to work reliably. In any case, the maze of kernel patches meant I could never get the compliance guys to sign off on deploying it; I suspect there's still a fair deployed base of that particular application running IPFIX over TLS over TCP for exactly that reason.

This is admittedly a relatively specific anecdote. I'll observe that with the move to mobile this problem is getting worse, not better, though.

Or tl;dr: even an open source kernel doesn't fix all your problems with requiring kernel changes to get an app deployed on a new stack.

> UDP without needing a port number for each protocol.

I've skimmed the draft but need to look into GUE more deeply. Is there code I can play with somewhere?

In any case I agree completely that the "evolutionary" requirement ("support userspace implementation and deployment of new transports in the Internet") should be separated from the "transitional" requirement (UDP encaps).

> transport protocol intended for Internet scale.

I agree with you here: we need a standard kernel interface for getting PMTU and ECN information back from the IP stack into userspace, and a bidirectional interface for DSCP -- and the abstract API should be reasonably portable across platforms. These are not merely necessary for transport protocols in userspace; this information is often useful up at the application layer, as well.

> stack) where TCP interacts with other layers.

I'm only passingly familiar with the situation here -- I doubt any of the TCP/IP stack code from the kernel that was around the last time I dug into it (1997) is still there. But I will observe that an interface with fifty hardpoints isn't an interface, it's an arbitrary line drawn through what is essentially an integrated design.

At least for Linux, then, there would need to be a design effort to determine which of these interfaces are essential to expose, and which are essentially accidents of localized optimization decisions. But some of the particular interfaces you mention sound like exactly what would be needed to put a "userspace transport layer" atop IP in Linux, especially the "wider" interfaces (sendmmsg, recvmmsg, sendfile).

> This includes the
> offload, LSO, LRO, as well as software analogues in GSO and GRO.

Protocol offloading is pretty much the opposite of what we're trying to do here -- offloads only work for ossified protocols pretty much by definition, so we'd have to give these up in the near term for new protocols.

> The
> right CPU),

This may be exposing my ignorance, but packet steering works on (a function of) the 5-tuple, no? One of the points we're conceding here is that the ports are de facto part of the network layer, and appear at fixed offsets from the end of the network header.

Thanks again, cheers,

Brian

> pacing mechanisms to mitigate the effect of packet bursts

---

From: Tom Herbert <[email-masked]>
Date: Tue, 28 Jul 2015 09:07:47 -0700

Hi Brian,

Some replies inline...

> I've skimmed the draft but need to look into GUE more deeply. Is there code I can play with somewhere?
http://lwn.net/Articles/615044/ for GUE in Linux
https://lwn.net/Articles/614348/ for FOU which allows direct
encapsulation of any IP protocol over UDP (GUE and FOU share code
base)

> I'm only passingly familiar with the situation here -- I doubt any of the TCP/IP stack code from the kernel that was around the last time I dug into it (1997) is still there. But I will observe tha…
For the most part the interfaces are clean. SCTP and DCCP similarly
interact with the rest of the stack although probably not as intensely
since TCP tends to be the first focus for optimizations.

> Protocol offloading is pretty much the opposite of what we're trying to do here -- offloads only work for ossified protocols pretty much by definition, so we'd have to give these up in the near ter…
Not really, many good protocol offloads are generic mechanisms that
permit broad application. For instance, if implemented properly HW
checksum offload can work on any transport layer protocol checksum
even if device does not know how to parse the the transport or even
network layer or any used encapsulation protocols. CRC could be
similarly offloaded (like for SCTP) at least on transmit. For UDP
encapsulations, we are requesting that vendors implement offloads that
are agnostic to (the many :-) ) different encapsulation protocols (see
http://people.netfilter.org/pablo/netdev0.1/papers/UDP-Encapsulation-in-Linux.pdf).

> This may be exposing my ignorance, but packet steering works on (a function of) the 5-tuple, no? One of the points we're conceding here is that the ports are de facto part of the network layer, and…
The 5-tuple hash for TCP and UDP is what most devices have
implemented, however IMO there is a better solution moving forward. As
I pointed out before, not all packets contain port numbers (IP
fragments, IPIP, GRE/IP, MPLS/IP, etc.). Also, for ECMP and RSS
hardware needs to parse the transport protocol so currently we are
effectively limited from using anything other than TCP or UDP in the
DC for high packet rate applications. I have come to believe that the
real fix for these problems (as opposed to just blindly encapsulating
everything in UDP) is to use IPv6 flow labels for load balancing (a la
RFC6438). We've implemented the necessary kernel support, and are
actively requesting HW vendors to use the field as input to ECMP and
RSS (several already do). We intend to enable automatic flow label
generation by default in Linux after due diligence, so in the not too
distant future expect to see a lot of IPv6 packets on the Internet
with non-zero flow labels!

Tom