Subject: [Spud] Bare-minimum PLUS (was Re: Thoughts on the privacy concerns expressed at the BoF)

From: Erik Nygren <[email-masked]>
Date: Thu, 28 Jul 2016 11:02:54 -0400

On Mon, Jul 25, 2016 at 4:20 PM, Mirja Kühlewind <
[email-masked]> wrote:

> As mentioned by Ted, this is actually not what we propose. The mechanism
> we propose is exposing information under endpoint control which is a very
> important point here. The endpoint has to add an option to send (or
> request) a specific bit of information (which also must be registered by
> IANA with IESG approval). This is just the same as TCP options or IP option
> headers. The only difference is that PLUS has a MAC which allows detection
> of middlebox manipulation (which is not the case for other existing
> protocols such as TCP options). This is another a very big and important
> aspect of what we propose with PLUS as our goal is to encrypt everything
> above PLUS including the TCP header and TCP options in future. Especially
> encrypting TCP option is important because TCP options provide much less
> control than what we propose with PLUS today which is a problem (especially
> as currently still unto 90% of the traffic is TCP). So I actually though we
> are on the same page here and have a common goal.
>

I'm not sure that we get enough value out of extensibility or being overly
generic here, at least for practical non-research purposes.  If there is a
tightly defined set of things allowed, we'll quickly ossify on these
anyways.  As firewall vendors will just bake in those registered by IANA
when they ship their product and then we'll be right back where we are
today with IPv6 extension headers.

I'd propose that it may make more sense to define a fixed header prefix
format that contains the right set of features needed to solve the problems
of QUIC and WebRTC (and other future UDP protocols).  Along with this, also
define a common set of requirements for protocols using this header prefix
(eg, around DDoS reflection prevention).

Having a toolkit for things like handling connection ids in a way that
support multipath mobility, stateless load-balancers, DDoS resiliance, and
which minimize linkability would also be a nice-to-have to prevent each UDP
protocol from having to come up with subtly different ways to solve this
set of problems.

For example, a bare minimum set of features might include:

* Opportunistic signalling to middle-boxes (NATs and Firewalls) for when
they can discard connection state.  Unless we want to declare that new UDP
protocols only work with IPv6 (worth seriously considering, but perhaps not
realistic yet), there just aren't enough {IPv4,port} tuples available for
NATs to support large numbers of users with large numbers of UDP
connections without having crazy-short keep-alive lifetimes.  Middle-boxes
will always need to be aware that this is best-effort (the FINs can be
dropped or lost or not sent by malicious clients, especially during
multipath transitions) but this could make life much better for
well-behaved clients and things like the existing NATs which use port
ranges per client might encourage clients to be well-behaved here on
average.

* Negotiating initial PMTU/MSS.  While clients will need to be able to
probe up/down and respond to routing changes that change the PMTU, the TCP
MSS mechanism and ability for middle boxes to decrement/clamp this value
makes a big difference in how quickly clients behind tunnels can converge
without losing round-trips or having very complicated probing.  Especially
if we want to be able to deploy jumbo-frames it will be important to have a
mechanism here.

* Sending signals in both directions that can be correlated to
connections.  Using ICMP/ICMPv6 may be a fine way to do this but having a
common prefix for connection information that can be included in the
message makes some things such as routing the messages through a
load-balancer much more viable).

* ... there may be a few others from the use-cases doc.

Having a fixed header prefix format might also make it much more viable to
implement with today's router/firewall hardware with just firmware/software
updates which might make a big difference for medium-term uptake.
It might be we can get away with something like:

   [MAGIC+VERSION] [CONNECTION_ID] [BITS_FOR_SYN_FIN]
   [MSS_DURING_NEGOTIATION_WHEN_SYN_IS_SET]
   [EXTENSION_SPACE_OSSIFICATION_MAY_NOT_PASS]

It might be reasonable to allow for some padding space for future extension
and research, but with the expectation that this will often get
dropped/filtered and thus would be something clients would need to probe
for (and is something people could find a way to do today regardless).

    Erik

---

From: Dave Dolson <[email-masked]>
Date: Thu, 28 Jul 2016 15:23:56 +0000

Regarding the state kept by middle-boxes such as firewalls:
The timeout should be short (a few seconds) until a three-way handshake is complete. This is required to defend against attacks on state memory of the firewall.

In the three-way handshake:
- The server’s SYN-ACK indicates the server is real and willing to talk to the client.
- the client’s ACK of the server’s SYN-ACK provides evidence the client is real (vs. spoofed) and has witnessed the SYN-ACK

The random sequence numbers and ack numbers are valuable in this regard. E.g., a spoofing device cannot simply send the first and third packets because the third packet must correctly ACK the second packet from the server.

Once a three-way handshake is witnessed, TCP state timeout can be several minutes.

The proposal below doesn’t seem to contain enough information for a firewall to know the client is real.
Consider adding a random number that each side can echo back? Perhaps a different connection-ID per direction?

-Dave

From: Spud [mailto:[email-masked]] On Behalf Of Erik Nygren
Sent: Thursday, July 28, 2016 11:03 AM
To: Mirja Kühlewind
Cc: Kyle Rose; [email-masked]
Subject: [Spud] Bare-minimum PLUS (was Re: Thoughts on the privacy concerns expressed at the BoF)

On Mon, Jul 25, 2016 at 4:20 PM, Mirja Kühlewind <[email-masked]<mailto:[email-masked]>> wrote:
As mentioned by Ted, this is actually not what we propose. The mechanism we propose is exposing information under endpoint control which is a very important point here. The endpoint has to add an option to send (or request) a specific bit of information (which also must be registered by IANA with IESG approval). This is just the same as TCP options or IP option headers. The only difference is that PLUS has a MAC which allows detection of middlebox manipulation (which is not the case for other existing protocols such as TCP options). This is another a very big and important aspect of what we propose with PLUS as our goal is to encrypt everything above PLUS including the TCP header and TCP options in future. Especially encrypting TCP option is important because TCP options provide much less control than what we propose with PLUS today which is a problem (especially as currently still unto 90% of the traffic is TCP). So I actually though we are on the same page here and have a common goal.

I'm not sure that we get enough value out of extensibility or being overly generic here, at least for practical non-research purposes.  If there is a tightly defined set of things allowed, we'll quickly ossify on these anyways.  As firewall vendors will just bake in those registered by IANA when they ship their product and then we'll be right back where we are today with IPv6 extension headers.
I'd propose that it may make more sense to define a fixed header prefix format that contains the right set of features needed to solve the problems of QUIC and WebRTC (and other future UDP protocols).  Along with this, also define a common set of requirements for protocols using this header prefix (eg, around DDoS reflection prevention).
Having a toolkit for things like handling connection ids in a way that support multipath mobility, stateless load-balancers, DDoS resiliance, and which minimize linkability would also be a nice-to-have to prevent each UDP protocol from having to come up with subtly different ways to solve this set of problems.

* Opportunistic signalling to middle-boxes (NATs and Firewalls) for when they can discard connection state.  Unless we want to declare that new UDP protocols only work with IPv6 (worth seriously considering, but perhaps not realistic yet), there just aren't enough {IPv4,port} tuples available for NATs to support large numbers of users with large numbers of UDP connections without having crazy-short keep-alive lifetimes.  Middle-boxes will always need to be aware that this is best-effort (the FINs can be dropped or lost or not sent by malicious clients, especially during multipath transitions) but this could make life much better for well-behaved clients and things like the existing NATs which use port ranges per client might encourage clients to be well-behaved here on average.
* Negotiating initial PMTU/MSS.  While clients will need to be able to probe up/down and respond to routing changes that change the PMTU, the TCP MSS mechanism and ability for middle boxes to decrement/clamp this value makes a big difference in how quickly clients behind tunnels can converge without losing round-trips or having very complicated probing.  Especially if we want to be able to deploy jumbo-frames it will be important to have a mechanism here.

* Sending signals in both directions that can be correlated to connections.  Using ICMP/ICMPv6 may be a fine way to do this but having a common prefix for connection information that can be included in the message makes some things such as routing the messages through a load-balancer much more viable).

Having a fixed header prefix format might also make it much more viable to implement with today's router/firewall hardware with just firmware/software updates which might make a big difference for medium-term uptake.

It might be reasonable to allow for some padding space for future extension and research, but with the expectation that this will often get dropped/filtered and thus would be something clients would need to probe for (and is something people could find a way to do today regardless).

    Erik

---

From: Erik Nygren <[email-masked]>
Date: Thu, 28 Jul 2016 11:56:54 -0400

This is exactly the sort of thing that I think demonstrates the value of
defining something here so we can have one mechanism that a bunch of UDP
protocols can use to allow for several minute state timeouts.

With TCP you need the sequence/ack numbers to get enough entropy due to the
port number being only 16 bits.  If the connection-id is 64 bits (or there
are two 64-bit connection-ids in the packet, one contributed by each
end-point) does this contribute enough entropy to prevent passive adversary
close/reset attacks?  It seems like active adversary close/reset attacks
aren't possible to reliably defend against (without crypto between middle
boxes and endpoint).

A pattern that has been discussed that seems quite valuable would be to
distribute additional connection-ids for mobility and resumption within the
encrypted channel.  This would allow for resumption, mobility, and
periodically cranking the connection-id without linkability.

Having a server-contributed component of the connection-id (perhaps an
initial one in clear-text and later ones through the encrypted channel)
could allow for stateless packet distribution by server-side load balancers
and could also form part of a proof-of-address-ownership
three-way-handshake mechanism to mitigate against DDoS reflection and
spoofed source address attacks.

      Erik

On Thu, Jul 28, 2016 at 11:23 AM, Dave Dolson <[email-masked]> wrote:

> The timeout should be short (a few seconds) until a three-way handshake is
> complete. This is required to defend against attacks on state memory of the
> firewall.
>
>
>
> - the client’s ACK of the server’s SYN-ACK provides evidence the client is
> real (vs. spoofed) and has witnessed the SYN-ACK
>
>
>
> The random sequence numbers and ack numbers are valuable in this regard.
> E.g., a spoofing device cannot simply send the first and third packets
> because the third packet must correctly ACK the second packet from the
> server.
>
>
>
> *From:* Spud [mailto:[email-masked]] *On Behalf Of *Erik Nygren
> *Sent:* Thursday, July 28, 2016 11:03 AM
> *To:* Mirja Kühlewind
> *Cc:* Kyle Rose; [email-masked]
> *Subject:* [Spud] Bare-minimum PLUS (was Re: Thoughts on the privacy
> concerns expressed at the BoF)
>
>
>
> It might be reasonable to allow for some padding space for future
> extension and research, but with the expectation that this will often get

---

From: Tom Herbert <[email-masked]>
Date: Thu, 28 Jul 2016 09:22:55 -0700

On Thu, Jul 28, 2016 at 8:02 AM, Erik Nygren <[email-masked]> wrote:
> On Mon, Jul 25, 2016 at 4:20 PM, Mirja Kühlewind
> <[email-masked]> wrote:
>>
>> important point here. The endpoint has to add an option to send (or request)
>> a specific bit of information (which also must be registered by IANA with
>> IESG approval). This is just the same as TCP options or IP option headers.
>> The only difference is that PLUS has a MAC which allows detection of
>> middlebox manipulation (which is not the case for other existing protocols
>> such as TCP options). This is another a very big and important aspect of
>> what we propose with PLUS as our goal is to encrypt everything above PLUS
>> including the TCP header and TCP options in future. Especially encrypting
>> TCP option is important because TCP options provide much less control than
>> what we propose with PLUS today which is a problem (especially as currently
>> still unto 90% of the traffic is TCP). So I actually though we are on the
>> same page here and have a common goal.
>
>
> when they ship their product and then we'll be right back where we are today
> with IPv6 extension headers.
>
> protocol from having to come up with subtly different ways to solve this set
> of problems.
>
> NATs to support large numbers of users with large numbers of UDP connections
> without having crazy-short keep-alive lifetimes.  Middle-boxes will always
> need to be aware that this is best-effort (the FINs can be dropped or lost
> or not sent by malicious clients, especially during multipath transitions)
> but this could make life much better for well-behaved clients and things
> like the existing NATs which use port ranges per client might encourage
> clients to be well-behaved here on average.
>
>From an application provider point of view, I am still very dubious of
purposely exposing connection state so that anonymous middleboxes can
track my connections. As pointed out at the mic during the PLUS BOF,
firewalls currently assume that all packets go through their device--
if packets are routed to some random middlebox then my transport layer
connection can be dropped by the device just because the middlebox
doesn't have state. This breaks the E2E model of the Internet and
kills our ability to use multipath and multihoming. Any device that
assumes a singular path for a connection is therefore the problem;
this is not a model that we want to perpetuate in PLUS.

Tom

---

From: Dave Dolson <[email-masked]>
Date: Thu, 28 Jul 2016 17:20:37 +0000

Tom,
Yes, firewalls do assume all packets go through them. This is how they
protect a (stub) network. They must be deployed such that this is true.

The E2E principle, I think, is intended to provide application developers
with a very useful abstraction, to make it easier to reason about the network.

But for network operators, the network isn't an abstract thing.
It isn't a cloud that provides infinite bandwidth between any pair of nodes.
There have been many creative ideas about how to build such a network,
which have been beneficial but led to ossification.

I think we need to understand what is to be the new contract between
endpoints and the network, to limit the "creativity".

-Dave

-----Original Message-----
From: Spud [mailto:[email-masked]] On Behalf Of Tom Herbert
Sent: Thursday, July 28, 2016 12:23 PM
To: Erik Nygren
Cc: Kyle Rose; Mirja Kühlewind; spud
Subject: Re: [Spud] Bare-minimum PLUS (was Re: Thoughts on the privacy concerns expressed at the BoF)

Tom

_______________________________________________
Spud mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/spud

---

From: Tom Herbert <[email-masked]>
Date: Thu, 28 Jul 2016 11:17:28 -0700

> with a very useful abstraction, to make it easier to reason about the network.
The abstraction is only useful up to the point that networks preserve
the model. If the network breaks the model in ad hoc ways that forces
the application to try to compensate with more ad hoc mechanisms or
just stop innovating altogether as we see happened with protocol
ossification.

> But for network operators, the network isn't an abstract thing.

I think that would more appropriately be stated that for network
operators *their* network isn't an abstract thing. We haven't
particularly seem a lot of consistency how different operators run
their networks.

> which have been beneficial but led to ossification.
Some of this creativity is beneficial, but it is also true that some
has also been malevolent. The problem from an application point of
view is that we have no insight as to what is being done with the data
being exposed. The only recourse we have to eliminate malicious use
and stop protocol ossification is to hide as much as possible from the
network-- this is why we want to encrypt the transport layer in the
first place.

> endpoints and the network, to limit the "creativity".
"contract" is a key word here. If there is an explicit trust
relationship with guarantees established between end nodes and devices
in the attached network, then there is a much better chance we
(application providers) would be willing to share transport layer
information (this the philosophy in the mobile guidance). Volunteering
arbitrary transport layer information to the network on the off chance
that some anonymous device we don't even know exists might find it
useful still seems like a big stretch to me.

Tom

---

From: Eliot Lear <[email-masked]>
Date: Fri, 29 Jul 2016 14:18:19 +0200

Hi Tom,

Something that did not get brought up at the BOF is nagging at me...

> just stop innovating altogether as we see happened with protocol

You write that as if it's a bad thing ;-)

Some amount of protocol ossification is a GOOD thing.  In particular,
the calling interface for transports needs to be stable or source code
will have no shelf life.  The fact that TCP's calling interface has been
SO stable for SO long has meant that a large code base did not require
substantial maintenance just to keep doing the same thing it was
doing.   I see this as an exercise to determine what needs ossification
and what does not, and there are extreme positions at both ends.

Eliot

---

From: Brian Trammell <[email-masked]>
Date: Fri, 29 Jul 2016 14:35:49 +0200

Greetings, all,

Oops, somehow GPGTools decided that message needed to be encrypted. Here it is so that everyone can read it:

> On 29 Jul 2016, at 12:57, Brian Trammell <[email-masked]> wrote:
> 
> hi Erik, all,
> 
> Still catching up on threads here. One point inline:
> 
>> On 28 Jul 2016, at 17:56, Erik Nygren <[email-masked]> wrote:
>> 
>> This is exactly the sort of thing that I think demonstrates the value of defining something here so we can have one mechanism that a bunch of UDP protocols can use to allow for several minute state timeouts.
>> 
>> With TCP you need the sequence/ack numbers to get enough entropy due to the port number being only 16 bits.  If the connection-id is 64 bits (or there are two 64-bit connection-ids in the packet, one contributed by each end-point) does this contribute enough entropy to prevent passive adversary close/reset attacks?  It seems like active adversary close/reset attacks aren't possible to reliably defend against (without crypto between middle boxes and endpoint).
> 
> See Luckie et al in IMC2015 "Resilience of Deployed TCP to Blind Attacks" at http://www.caida.org/~mjl/pubs/blind.pdf. Summary: the situation is pretty bleak for everything except BGP. The dominance of relatively short TCP sessions in Internet traffic mitigates this somewhat.
> 
> An explicit separation between integrity-protected, path-visible, and confidentiality-protected, path-invisible communication in the transport layer is necessary to remedy this situation. This is of course what we're trying to do here. :)
> 
> Cheers,
> 
> Brian
> 
>> A pattern that has been discussed that seems quite valuable would be to distribute additional connection-ids for mobility and resumption within the encrypted channel.  This would allow for resumption, mobility, and periodically cranking the connection-id without linkability.
>> 
>> Having a server-contributed component of the connection-id (perhaps an initial one in clear-text and later ones through the encrypted channel) could allow for stateless packet distribution by server-side load balancers and could also form part of a proof-of-address-ownership three-way-handshake mechanism to mitigate against DDoS reflection and spoofed source address attacks.
>> 
>>      Erik
>> 
>> 
>> 
>> On Mon, Jul 25, 2016 at 4:20 PM, Mirja Kühlewind <[email-masked]> wrote:
>>

---

From: Kyle Rose <[email-masked]>
Date: Fri, 29 Jul 2016 08:39:30 -0400

> and what does not, and there are extreme positions at both ends.

I think this conflates two issues. I think of "ossification" as unintended
structure that precludes development within the protocol stack, something I
think is broadly accepted as a problem by people who are trying to improve
the experience of users. What you are talking about here seems to be
stability of the API for a specific protocol. Frankly, if we apply
"ossification" to that then the term becomes overly broad and therefore
meaningless. I don't think anyone is suggesting that all APIs should be
randomly perturbed over time just to see what happens.

No one is going to break the sockets API anytime soon, but having to
develop a new API for modern protocols would be a nice problem to have.
First, though, we need to figure out how to route those protocols through
the boxes that right now assume the entire internet runs over a snapshot of
TCP from 20 years ago.

Kyle

---

From: Eliot Lear <[email-masked]>
Date: Fri, 29 Jul 2016 15:01:12 +0200

Kyle,

On 7/29/16 2:39 PM, Kyle Rose wrote:
> develop a new API for modern protocols would be a nice problem to
> have. First, though, we need to figure out how to route those
> protocols through the boxes that right now assume the entire internet
> runs over a snapshot of TCP from 20 years ago.
>

We now have platforms that lose backward compatibility in as little as a
year.  And so if a transport interface is being rev'd, a developer may
be left to only guess at which point it will be stable.  In as much as
the semantics are changing, that's a problem.

Eliot

---

From: Kyle Rose <[email-masked]>
Date: Fri, 29 Jul 2016 09:08:04 -0400

On Fri, Jul 29, 2016 at 9:01 AM, Eliot Lear <[email-masked]> wrote:

> year.  And so if a transport interface is being rev'd, a developer may be
> left to only guess at which point it will be stable.  In as much as the
> semantics are changing, that's a problem.
>

Are you referring to QUIC? What you highlight is precisely why Google did
*not* take it to IETF too quickly: because they wanted to retain the
ability to revise it quickly during a period of intense experimentation. No
one here doubts that standardization necessarily brings stability in terms
of backward compatibility to the API and wire protocol, but I imagine one
of the things we'll be focusing on in the QUIC WG is how to enable future
evolution within the constraint of interoperability with older versions.

Kyle

---

From: Brian Trammell <[email-masked]>
Date: Fri, 29 Jul 2016 15:16:43 +0200

hi Kyle,

> On 29 Jul 2016, at 14:39, Kyle Rose <[email-masked]> wrote:
> 
> I think this conflates two issues. I think of "ossification" as unintended structure that precludes development within the protocol stack, something I think is broadly accepted as a problem by people who are trying to improve the experience of users. What you are talking about here seems to be stability of the API for a specific protocol. Frankly, if we apply "ossification" to that then the term becomes overly broad and therefore meaningless. I don't think anyone is suggesting that all APIs should be randomly perturbed over time just to see what happens.

Some of it is stability of the API, some of it is the stability of the what i'll call the "wire image": what the headers and behaviors look like to an on-path device.

I agree with Eliot that what we're trying to do here is re-ossify the wire image *on purpose*, given the things we've learned since the 1980s from the less intentional ossification that's happened since then. There is a careful line to draw when doing this. It should be generic enough that all the transport innovation we think we might want to do between now and 2040 fits inside it. It should be restrictive enough that it presents an improvement on the present state of the world with respect to the types of shenanigans described in https://tools.ietf.org/html/draft-trammell-privsec-defeating-tcpip-meta-00. But if it's too restrictive, people will just find other ways to implement the kinds of in-network functions they want to: cf. using the QUIC version number as a QUIC magic number, for instance.

> No one is going to break the sockets API anytime soon

*ahem* https://www.ietf.org/proceedings/96/slides/slides-96-taps-2.pdf *cough* -- I'm not saying we're going to be successful in breaking it, but it *is* the other half of the problem we have today.

Cheers,

Brian

> but having to develop a new API for modern protocols would be a nice problem to have. First, though, we need to figure out how to route those protocols through the boxes that right now assume the entire internet runs over a snapshot of TCP from 20 years ago.
> 
> Kyle

---

From: Eliot Lear <[email-masked]>
Date: Fri, 29 Jul 2016 15:27:09 +0200

On 7/29/16 3:08 PM, Kyle Rose wrote:
> On Fri, Jul 29, 2016 at 9:01 AM, Eliot Lear <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Kyle,
>
>>     No one is going to break the sockets API anytime soon, but having
>>     to develop a new API for modern protocols would be a nice problem
>>     to have. First, though, we need to figure out how to route those
>>     protocols through the boxes that right now assume the entire
>>     internet runs over a snapshot of TCP from 20 years ago.
>>
>
>     We now have platforms that lose backward compatibility in as
>     little as a year.  And so if a transport interface is being rev'd,
>     a developer may be left to only guess at which point it will be
>     stable.  In as much as the semantics are changing, that's a problem.
>
>
> Are you referring to QUIC?

No.  I am not.  I am reflecting on how people may really have overlooked
the value of ossification at the waist.  But there is such a thing as
overdoing it, and were I a transport developer I'd want things a little
easier to change too.  Yin/yang (no pun intended).

Eliot

---

From: Kyle Rose <[email-masked]>
Date: Fri, 29 Jul 2016 09:56:17 -0400

On Fri, Jul 29, 2016 at 9:16 AM, Brian Trammell <[email-masked]> wrote:

> Some of it is stability of the API, some of it is the stability of the
> what i'll call the "wire image": what the headers and behaviors look like
> to an on-path device.
>
> I agree with Eliot that what we're trying to do here is re-ossify the wire
> image *on purpose*, given the things we've learned since the 1980s from the
> less intentional ossification that's happened since then. There is a
> careful line to draw when doing this. It should be generic enough that all
> the transport innovation we think we might want to do between now and 2040
> fits inside it. It should be restrictive enough that it presents an
> improvement on the present state of the world with respect to the types of
> shenanigans described in
> https://tools.ietf.org/html/draft-trammell-privsec-defeating-tcpip-meta-00.
> But if it's too restrictive, people will just find other ways to implement
> the kinds of in-network functions they want to: cf. using the QUIC version
> number as a QUIC magic number, for instance.
>

Agreed. I think we're saying basically the same thing. I was merely
suggesting that "ossification" as we've been using it applies to what you
call the network image, not to the API for the protocols: those APIs are
under the full control of each endpoint independently, so long as the
underlying protocol can be expressed in terms of that API. (Extreme
counterexample: trying to use sigaction/kill as an API to TCP. Maybe it
could be done, but it sure would be limiting and pointless.)

> *ahem* https://www.ietf.org/proceedings/96/slides/slides-96-taps-2.pdf
> *cough* -- I'm not saying we're going to be successful in breaking it, but
> it *is* the other half of the problem we have today.
>

This isn't breaking it: this is offering an alternative. I'm all for that,
as long as telnet continues to work without source changes.

Kyle

---

From: Tom Herbert <[email-masked]>
Date: Fri, 29 Jul 2016 08:58:23 -0700

> and what does not, and there are extreme positions at both ends.
Note that I specifically referred to 'protocol' ossification not
'interface' ossification. In fact, the TCP sockets interface is
constantly being extended. For instance, in order to make TCP Fast
Open (TFO) work we needed to change the semantics of 'connect' call
and allow sendmsg on an unconnected socket.

Tom