Subject: [tcpinc] TCP's treatment of data in SYN packets

From: Kyle Rose <[email-masked]>
Date: Sun, 24 Jul 2016 19:45:46 -0400

In the absence of a TFO cookie, what is the behavior of TCP in the presence
of data in the SYN packet? The ability of ENO to send early data on session
resumption to servers that potentially don't understand ENO depends on the
servers throwing this data away so it can be replaced in the following ACK.

RFC 4987 section 3.5 (https://tools.ietf.org/html/rfc4987#section-3.5)
states that:

q( If data accompanies the SYN segment, then this data is not
   acknowledged or stored by the receiver, and will require
   retransmission. )

But I have read elsewhere that the server might queue it up and wait for
the 3WHS to complete (though if it is conformant, it will ACK only a single
byte for the SYN and have the blob retransmitted by the client anyway).

I guess I'm interested in both normative and observed behavior. I'm happy
to inquire elsewhere if someone has a suggestion. (tcpm?)

Kyle

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 26 Jul 2016 10:18:00 -0700

On Sun, Jul 24, 2016 at 4:45 PM, Kyle Rose <[email-masked]> wrote:
>
> In the absence of a TFO cookie, what is the behavior of TCP in the presence of data in the SYN packet? The ability of ENO to send early data on session resumption to servers that potentially don't understand ENO depends on the servers throwing this data away so it can be replaced in the following ACK.
>
> RFC 4987 section 3.5 (https://tools.ietf.org/html/rfc4987#section-3.5) states that:
>
> But I have read elsewhere that the server might queue it up and wait for the 3WHS to complete (though if it is conformant, it will ACK only a single byte for the SYN and have the blob retransmitted by the client anyway).

We did some tests with this when during TFO development. AFAIK Linux,
*BSD/iOS, Windows all return a SYN that acks only the ISN but not the
data. None of them queues the data hence the active side needs to
retransmit the data. But my knowledge is 5 years old.

FWIW I and other TFO developers in other stacks (cc'd) have all found
that some middle-boxes will crash or hang on receiving SYN-data
(regardless of TFO cookie option or not). For instance, I found a bug
in Linux netfliter that the conntrack module assumes SYN never carries
data, so the internal sequence was only incremented by one on SYN-data
packe, and led to rejecting every packet afterward. Others have found
FW blocking the IP completely (see last meetings material by Praveen).
We were talking about putting a TFO-bis on these issues.

>
> I guess I'm interested in both normative and observed behavior. I'm happy to inquire elsewhere if someone has a suggestion. (tcpm?)
>
> Kyle
>
>
> _______________________________________________
> Tcpinc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpinc

---

From: Kyle Rose <[email-masked]>
Date: Tue, 26 Jul 2016 18:10:43 -0400

> retransmit the data. But my knowledge is 5 years old.

Which naturally makes me curious if this behavior has changed since TFO was
implemented. What sort of tools do/did you have to do this kind of probing?

> We were talking about putting a TFO-bis on these issues.

This is really good information: thanks! I think if tcpinc decides to
pursue early data, we'll want to learn as much as possible from your
experience.

Kyle

---

From: David Mazieres <[email-masked]>
Date: Wed, 27 Jul 2016 10:35:09 -0700

> pursue early data, we'll want to learn as much as possible from your

So just to scope what we had in mind, "pursue early data" sounds a bit
aggressive.  What I think we had in mind is just saying something about
what receivers MUST do when receiving a SYN segment with data, not
encouraging people to do that.

Since TFO will eventually cause people to address these middlebox
issues, it would be nice to avoid creating a second round of problems
where TCP-ENO implementations also crash or do weird stuff.  And it
would also be nice if TCP-ENO didn't disable this possible optimization.

What I had in mind was something along the lines of saying that the last
TEP suboption in host A's SYN segment is the SYN TEP, and that host B
MUST ignore and MUST NOT ACK any data in SYN segments unless the
negotiated TEP is the SYN TEP and that TEP explicitly defines the use of
data in SYN segments.  Furthermore, SYN data MUST be discarded in the
event that TCP-ENO is disabled, even when such data has already been
ACKed.  TEPs MAY define the use of data in SYN segments for when they
are the SYN TEP, but SHOULD limit such usage to cases in which it is
known that the passive opener and network path both support ENO.

David

---

From: Kyle Rose <[email-masked]>
Date: Wed, 27 Jul 2016 14:11:28 -0400

> known that the passive opener and network path both support ENO.

It's interoperability with older stacks that drove my question: if there
are multiple machines behind an IP, some of which support ENO and some of
which don't, do we break the charter by sending *any* data in the SYN
(outside of TFO, which as we've established is unusable for tcpinc)?

The charter says: q( The protocol must gracefully fall-back to TCP if the
remote peer does not support the proposed extensions. )

If TCP's behavior in the absence of a TFO cookie is to discard data in the
SYN, normatively and (especially) observationally, then your second
sentence ("Furthermore, SYN data MUST be dscarded...") is true when a host
doesn't support ENO, and so we can safely put whatever we like in the SYN.

If not, then unexpected data in the SYN *might* be forwarded to the
application as garbage by some TCP stacks, resulting in mostly-undefined
behavior but almost certainly connection breakage; in that case, do we need
some explicit signaling from the server on a prior connection for when SYN
data on resumption attempts is okay, or is "...but SHOULD limit such
usage..." good enough?

Kyle

---

From: David Mazieres <[email-masked]>
Date: Wed, 27 Jul 2016 14:45:11 -0700

Kyle Rose <[email-masked]> writes:

> do we need some explicit signaling from the server on a prior
> connection for when SYN data on resumption attempts is okay, or is
> "...but SHOULD limit such usage..." good enough?

This is precisely the question on which some input from more experienced
members of the working group could be helpful.

Basically I'd like to ensure that ENO conflicts as little as possible
with future use of data in SYN segments.  So rather than just flat-out
prohibiting it (and having people not even test the corner case), I
think it would be good to specify that receivers explicitly ignore the
data in places where it couldn't possibly be useful.  And then once you
are there, why not say that TEPs MAY define data but probably shouldn't
send it for the time being in the remaining cases.

David

---

From: Derek Fawcus <[email-masked]>
Date: Thu, 28 Jul 2016 00:24:19 +0100

On Wed, Jul 27, 2016 at 02:45:11PM -0700, David Mazieres wrote:
> with future use of data in SYN segments.

My understanding of what TCP _allows_ (absent FO) is that one can send
data in the SYN,  it can be ACK'ed in the SYN-ACK,  but there may be
some text indicating it is not to be delivered to the receiving
application unless and until the 3whs completes.

Implying that a passive opener which chose to ack such data bytes would
have to buffer such data until the handshake completes;  but I'm not
aware of any stack which actually works that way. (Due to DOS attack).

So I'd expect it would probably be safe to start defining some new
expectations / constraints for use of such data in SYN segments.

However,  one could envisage say a user space TCP stack (for a SERVER)
which supported the following sequence:

        CLIENT                                         SERVER

   1)   SYN-SENT     --> <SYN, data1> -->             SYN-RECEIVED

   2)   ESTABLISHED  <-- <SYN, ACK(data1), data2> <-- ESTABLISHED

   3)   FIN-WAIT-1   --> <FIN, ACK(data2)> -->        CLOSE-WAIT

followed by the rest of the normal close sequence.

i.e. something slightly similar to T/TCP but without trying to optimise
the close sequence,  while still allowing a request-response exchange
to be done in 1-RTT if all goes well.

DF

---

From: Derek Fawcus <[email-masked]>
Date: Thu, 28 Jul 2016 07:37:54 +0100

On Thu, Jul 28, 2016 at 12:24:19am +0100, Derek Fawcus wrote:
> aware of any stack which actually works that way.

Except the old KA9Q NOS,  which was tickling my memory...

It did have the ability to send such SYN data.

DF

---

From: Joe Touch <[email-masked]>
Date: Thu, 28 Jul 2016 10:52:41 -0700

On 7/24/2016 4:45 PM, Kyle Rose wrote:
> In the absence of a TFO cookie, what is the behavior of TCP in the
> presence of data in the SYN packet? The ability of ENO to send early
> data on session resumption to servers that potentially don't
> understand ENO depends on the servers throwing this data away so it
> can be replaced in the following ACK.
>
> But I have read elsewhere that the server might queue it up and wait
> for the 3WHS to complete (though if it is conformant, it will ACK only
> a single byte for the SYN and have the blob retransmitted by the
> client anyway).

Data in the SYN can be:

    - ACKd, but then it MUST be cached and delivered to the application
at the completion of the 3WHS (and not before)
    - not ACKd (i.e., ACK only the SYN but not the data) but then it
MUST be discarded until it is retransmitted

Either behavior is valid in the absence of TFO. Dropping the SYN data
protects against DOS attacks that could overload servers, but a server
can easily protect itself by ACKing SYN data only until space becomes
limited. Keeping the data can be more efficient and reduce latency in
some cases.

TCP options should assume this behavior unless *they* redefine it (e.g.,
TFO).

Middleboxes MUST not interfere with this behavior (or any TCP behavior,
FWIW).

Joe

---

From: David Mazieres <[email-masked]>
Date: Thu, 28 Jul 2016 13:16:27 -0700

> TCP options should assume this behavior unless *they* redefine it (e.g.,

Thanks for this explanation.  This matches my understanding, and raises
a subtle point because of the elimination of the application-aware
mandatory bit on the wire.  Suppose you are in the following situation:

  * Host A and host B have previously established a bunch of TCP-ENO
    sessions successfully, so believe the network path is clear and so
    forth.

  * The application on host A suddenly sets the mandatory
    application-aware mode, and host B is not application aware.

  * Host A uses a TEP that allows for SYN data, and in fact tries to use
    SYN data on the next connection.

  * Host B acks the SYN data, because the SYN TEP is also the negotiated
    TEP.

  * Host A disables TCP-ENO because host B was not application aware, so
    the third leg of the handshake does not include an ENO option.

At this point host B would have to discard the data, since it includes
ciphertext.  If the data were a Diffie-Hellman parameter this might be
okay, though a redefinition of TCP's behavior since the data was ACKed.
I'm hesitant to redefine TCP's behavior in a way that does not closely
hue to other studied modifications such as TFO.

In particular, TFO suggests required buffering is not the best behavior.
In fact, I would imagine in the short term SYN data is most useful for
cases like session resumption, where you really want the data directly
delivered to the application (since it's cryptographically authenticated
anyway) and not buffered.  This is a modification of TCP I feel
comfortable with because TFO has essentially already vetted this model.

One possible solution is to say that you cannot send data in the SYN
segment if you set the application-aware mandatory bit.  Another
possible solution is to convey the application-aware mandatory bit by
going back to two a bits, the way we had a few drafts ago.

Reactions/opinions?

David

---

From: Joe Touch <[email-masked]>
Date: Thu, 28 Jul 2016 13:25:47 -0700

Hi, David,

>     sessions successfully, so believe the network path is clear and so

As with investment, past behavior is not a guarantee of future
performance. "Believe", but verify.

>     the third leg of the handshake does not include an ENO option.

IMO, the 3WHS supports offering capabilities and letting the other end
decide to confirm them. It does not allow for offers to be rescinded.
That would require another 3WHS, which can be done only inside a
specialized option (but it's a mess and should be avoided at all cost).

> ciphertext.
Strictly, whatever B ACKs MUST be delivered.

So the solution is simple - if you think this behavior needs to be
supported as part of the exchange, then (in the third step above) host B
MUST ACK only the SYN and discard the data.

>  If the data were a Diffie-Hellman parameter this might be
Within your option,  you can require that TCP's that support ENO MUST
discard SYN data.

But you need to allow for legacy receivers that behave otherwise, as
long as they also don't support ENO.

> going back to two a bits, the way we had a few drafts ago.

See above; I see only one way out here.

Joe

---

From: Kyle Rose <[email-masked]>
Date: Thu, 28 Jul 2016 18:01:30 -0400

> going back to two a bits, the way we had a few drafts ago.

If box B sends back an ENO option, we know it's ENO aware, so we can choose
the behavior here; we just need to make sure it's consistent and knowable
by both endpoints. Why can't the semantics here be that B throws away the
SYN data if it disagrees with A on app awareness? Are there situations
where the two endpoints could disagree on this determination?

If this works, then in the steady state (both endpoints are either
app-aware or not) clients will be able to send SYN data to known ENO-aware
servers, while in the rollout case (A and B don't agree on app awareness)
SYN data will be thrown away by B and known by A to have been thrown away.

I still don't have a good feel for what folks like Joe are recommending re:
dealing with a B that doesn't understand ENO. If B is suddenly not
ENO-aware, by the spec it could deliver garbage data to the application
upon completion of the 3WHS; is this okay under an assumption that B should
*not* suddenly be unable to understand ENO, or do we need explicit
signaling from a prior connection to the same IP saying "SYN data is okay
upon resumption", or is even that not good enough?

A lot of this would be resolved if we updated TCP to say that a server MUST
not ACK, buffer, or deliver SYN data unless this behavior is overridden by
some negotiated extension (like TFO or ENO), in which case the treatment of
such data is extension-specific. The ship has probably sailed on that one,
but it's interesting that this is essentially what 4987 states, as I posted
at the top of the thread:

Kyle

---

From: Joe Touch <[email-masked]>
Date: Thu, 28 Jul 2016 15:07:31 -0700

On 7/28/2016 3:01 PM, Kyle Rose wrote:
> I still don't have a good feel for what folks like Joe are
> recommending re: dealing with a B that doesn't understand ENO. If B is
> suddenly not ENO-aware, 
Not suddenly. During the SYN exchange.

Or are you not dealing with legacy endpoints that don't speak ENO? The
goal there is to fail gracefully - either fail to support ENO and
proceed, kill the connection, etc. But you should never start delivering
data to the application by accident - that's something you can't roll back.

BTW, you DO need to deal with an endpoint that spoke ENO before but does
not now (for two different connections). That could happen because of an
OS revision or because the connection configured differently anyway.

Joe

---

From: Kyle Rose <[email-masked]>
Date: Thu, 28 Jul 2016 18:31:04 -0400

> Not suddenly. During the SYN exchange.

Sorry, by "suddenly" I mean that an IP that formerly negotiated ENO gets
swapped out for one that doesn't understand ENO: this could for example be
the result of a load balancer sending the request to a machine with an
older kernel.

Note that it must be the case that the stack in question doesn't understand
ENO, not just that it is now unwilling to negotiate ENO. If it knows about
ENO but just doesn't want to negotiate it, it knows enough to throw away
the SYN data.

> OS revision or because the connection configured differently anyway.

Again, I would put the "machine understands but does not want to negotiate
ENO" case into the supportable category, because we can completely define
the behavior of endpoints that understand ENO. I'm worried about the
interoperability case with legacy kernels that don't even know what ENO is.

These two paragraphs do get to what I was asking. I agree we do need to
deal with them, but what is the space of reasonable solutions? When an
endpoint *could* deliver garbage to an application, as any legacy TCP stack
could according to the spec, we can never send encrypted data in the SYN if
we are unwilling to retain and/or assume some state about peers. I see four
main options:

(1) Don't ever send data in the SYN. This is easiest, but sacrifices ENO
support for SYN data forever for the sake of full interoperability with
legacy TCP stacks, even with hosts a user knows a priori support ENO. This
is not my preference.

(2) Retain knowledge of ENO support for an IP and assume it on resumption
attempts, throwing that state away if ENO is not negotiated. This case
means accepting that we'll have to retry the connection in cases when a
server supporting ENO and one knowing nothing about ENO share an IP.

(3) Require signaling from the server on a previous connection. This case
is a more explicit form of (2), and carries the same dangers, but puts the
configuration burden on the server administrator to assert that, yes, all
servers responding for this IP support ENO.

(4) Require signaling from the client application. This allows a client to
decide whether to support SYN data based on a priori knowledge of the
server's support for ENO.

Kyle

---

From: David Mazieres <[email-masked]>
Date: Thu, 28 Jul 2016 19:43:38 -0700

How about the following wording?

   A SYN segment containing an ENO option MUST NOT include a TCP Fast
   Open (TFO) option [RFC7413].  However, TEPs MAY specify the use of
   data in SYN segments to achieve similar benefits to TFO.

   The last TEP identifier suboption in host A's SYN segment is the _SYN
   TEP_.  The SYN TEP governs the use of data in A's SYN segment.  If
   the SYN TEP's specification does not define the use of such data,
   then host A's SYN segment MUST NOT contain data and host B MUST
   discard any such data.  Host B must also discard data in A's SYN
   segment if either the SYN TEP differs from the negotiated TEP or host
   B disables encryption.

   The use of data in B's SYN-ACK segment is governed by the negotiated
   TEP.  If the negotiated TEP's specification does not define the use
   of such data, then host B's SYN-ACK segment MUST NOT contain data and
   host A MUST discard any such data.  Host A MUST also discard any
   received SYN data if it disables encryption.

   When a host discards SYN data, it MUST NOT acknowledge the sequence
   number of the discarded data.  Rather, it MUST acknowledge the other
   host's initial sequence number as if the received SYN segment
   contained no data.

   Regardless of the SYN TEP and negotiated TEP, host A MUST NOT include
   data in a SYN-only segment when in mandatory application-aware mode.
   Moreover, in the event that host B is an active opener (because of
   simultaneous open), host B's SYN-only segment MUST NOT include data.

   Using data in SYN segments deviates from TCP semantics and can cause
   problems with middleboxes or non-compliant TCP hosts.  Hence, all
   TEPs SHOULD support a normal mode of operation that does not make use
   of data in SYN segments.  Moreover, implementations SHOULD employ SYN
   data only if explicitly requested by the application or in cases
   where such use is highly unlikely to pose problems.

David

---

From: Joe Touch <[email-masked]>
Date: Fri, 29 Jul 2016 11:47:33 -0700

FWIW, IMO the best wording would:

- start with the simplest case, i.e., NOT trying to optimize EDO to use
SYN data

- present the use of SYN data with EDO as an optimization
        that optimization might be a MAY or SHOULD, but not a MUST

Otherwise, you're needlessly complicating your spec for an optimization.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 29 Jul 2016 11:46:08 -0700

On 7/28/2016 3:31 PM, Kyle Rose wrote:
> On Thu, Jul 28, 2016 at 6:07 PM, Joe Touch <[email-masked]
> <mailto:[email-masked]>> wrote:
>
> Sorry, by "suddenly" I mean that an IP that formerly negotiated ENO
> gets swapped out for one that doesn't understand ENO: this could for
> example be the result of a load balancer sending the request to a
> machine with an older kernel.

Or a different connection. For all you know, there are multiple stacks
within the kernel (e.g., FreeBSD supports this) or per-connection settings.

IMO, it isn't a good idea to build a protocol around the default
assumption that past connections predict future ones. It's OK to guess,
but that guess has to be "unrollable". Which leads me to the concern
below...
>
> Note that it must be the case that the stack in question doesn't
> understand ENO, not just that it is now unwilling to negotiate ENO. If
> it knows about ENO but just doesn't want to negotiate it, it knows
> enough to throw away the SYN data.

The thing about TCP is that it doesn't say what to do with data that is
retransmitted.

E.g., let's say you send SYN-abc, and I ACK only the SYN (not the data
"abc").

Then you send "def". I ACK the data *at the position of 3*. There's
nothing that indicates which "position of 3" data I'm ACKing. For all
you know, I cached the data in the SYN but did not ACK It, but when the
new data came in I didn't overwrite it.

Although I know this is unusual behavior, I'm not entirely positive it's
prohibited by the spec. TCP doesn't say what happens to data that is
sent more than once at a given position - there's no rule that it MUST
be the variant seen in the most recently received segment.

So I can easily imagine compliant implementations that might end up
doing something you didn't want.

That's why, IMO, if you send ENO, you need to make sure the SENDER
doesn't put anything in the SYN data that has to be *changed*. Re-sent
is fine. But CHANGED is not.

...
To your point:

> I see four main options:
>
> (1) Don't ever send data in the SYN. This is easiest, but sacrifices
> ENO support for SYN data forever for the sake of full interoperability
> with legacy TCP stacks, even with hosts a user knows a priori support
> ENO. This is not my preference.
It is known safe, though.

> (2) Retain knowledge of ENO support for an IP and assume it on
> resumption attempts, throwing that state away if ENO is not
> negotiated. This case means accepting that we'll have to retry the
> connection in cases when a server supporting ENO and one knowing
> nothing about ENO share an IP.
This is safe only if you kill off the connection where you try ENO with
data and it fails.

> (3) Require signaling from the server on a previous connection. This
> case is a more explicit form of (2), and carries the same dangers, but
> puts the configuration burden on the server administrator to assert
> that, yes, all servers responding for this IP support ENO.

Dangerous. TCP shares state explicitly only within a single connection.
Across connections, you can share hints that you can undo (ala RFC2140),
but nothing that changes the semantics of TCP or can't be unrolled
completely.

> (4) Require signaling from the client application. This allows a
> client to decide whether to support SYN data based on a priori
> knowledge of the server's support for ENO.

I'm not sure what you mean here...

Joe

---

From: Wesley Eddy <[email-masked]>
Date: Fri, 29 Jul 2016 15:18:15 -0400

> Otherwise, you're needlessly complicating your spec for an optimization.

Just to be clear, this is actually talking about ENO, and not EDO, 
correct?  (I'm assuming a repeated typo in your message)

SYN data with EDO does not need any special discussion, because EDO only 
alters DO semantics for later (non-SYN) segments.  So, how a stack deals 
with data on SYN is un-related to EDO (but more critical for ENO).

---

From: David Mazieres <[email-masked]>
Date: Fri, 29 Jul 2016 12:24:07 -0700

> Otherwise, you're needlessly complicating your spec for an optimization.

I'm not sure I follow.  The issue here is about application data that
needs to be decrypted, acked, and delivered in order.  EDO would seem to
be a poor fit for such data, particularly since TFO is already doing the
heavy lifting of normalizing the use of data in SYN segments.  That's
not to say ENO TEPs couldn't significantly benefit from EDO,
particularly if EDO is later extended to grow SYN segment options, but I
think the obvious application for that is early exchange of DH
parameters to optimize away the fourth flow in protocols like tcpcrypt,
and NOT any kind of TFO-like immediate data exchange.  We've already
implicitly addressed EDO by the suboption length byte and the fact that
you can repeat TEP suboptions if you need more than 32 bytes for a
non-final suboption.

More to the point, ENO is kind of a meta-specification, so we aren't
trying to optimize anything so much as to prevent chaos should TEPs
later want to employ optimizations.  To do so, we have to require data
to be ignored in a bunch of cases, otherwise future TEPs risk running
into weird problems with ENO-compliant hosts.

Also, to clarify, the bulk of the document is about the simplest case,
but the wording I proposed if for a subsection specifically addressing
what to do if you receive SYN data--namely ignore it and don't ACK it in
a bunch of potentially problematic cases.

If this isn't clear, I can post a new draft so that everyone can see the
proposed language in context, and we can take it from there.

David

---

From: Joe Touch <[email-masked]>
Date: Fri, 29 Jul 2016 12:35:17 -0700

Yeah - ENO. 

> On Jul 29, 2016, at 12:18 PM, Wesley Eddy <[email-masked]> wrote:
> 
> Just to be clear, this is actually talking about ENO, and not EDO, correct?  (I'm assuming a repeated typo in your message)
> 
> SYN data with EDO does not need any special discussion, because EDO only alters DO semantics for later (non-SYN) segments.  So, how a stack deals with data on SYN is un-related to EDO (but more critical for ENO).
> 
> 
> 
>

---

From: Kyle Rose <[email-masked]>
Date: Fri, 29 Jul 2016 15:41:33 -0400

On Fri, Jul 29, 2016 at 2:46 PM, Joe Touch <[email-masked]> wrote:

> nothing that indicates which "position of 3" data I'm ACKing. For all you
> know, I cached the data in the SYN but did not ACK It, but when the new
> data came in I didn't overwrite it.
>
> prohibited by the spec. TCP doesn't say what happens to data that is sent
> more than once at a given position - there's no rule that it MUST be the
> variant seen in the most recently received segment.
>
> So I can easily imagine compliant implementations that might end up doing
> something you didn't want.
>
> That's why, IMO, if you send ENO, you need to make sure the SENDER doesn't
> put anything in the SYN data that has to be *changed*. Re-sent is fine. But
> CHANGED is not.
>

Right, I get that for interoperability with ENO-unaware stacks there is no
way to change data presented in the SYN. In the case where both ends
understand ENO but the server is not negotiating it, we can define TCP SYN
payload semantics differently: essentially, when there is an ENO option in
the SYN, an ENO-aware server will do one of two things:

(1) If it is negotiating ENO, it will hand the data off to the chosen TEP,
which will then indicate back to ENO whether to ACK the data or not.

(2) If it is not negotiating ENO, it will not ACK the data, and will in
fact throw it away and explicitly permit transmission of *different* data
for the same sequence number range.

This seems fine to me: since we're defining it in this case, we get to
choose the behavior. The problem is with interoperability with servers that
don't understand ENO, because they can't make an informed decision here,
and don't all throw away SYN data.

Allow the client to assume that a particular server supports ENO, and so
choose to send SYN data knowing a priori that the server will be able to
make the choice I outlined above. If you run a service on a particular
version of Linux that supports ENO, and distribute a phone app that
contacts only that service, it seems perfectly fine to tell your app to
assume the server knows ENO.

I'm not so sure about this case, as a middlebox unaware of ENO might be
breaking the connection: without SYN data, the connection simply wouldn't
negotiate ENO; with SYN data, the connection wouldn't negotiate ENO, and
the server might receive garbage data from the middlebox. Basically, cases
2-4 from my last email all carry similar potential for interoperability
problems.

Kyle

---

From: Joe Touch <[email-masked]>
Date: Fri, 29 Jul 2016 13:03:55 -0700

> needs to be decrypted, acked, and delivered in order.  EDO would seem to
s/EDO/ENO/

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 29 Jul 2016 13:09:22 -0700

On 7/29/2016 12:41 PM, Kyle Rose wrote:
> On Fri, Jul 29, 2016 at 2:46 PM, Joe Touch <[email-masked]
>     doesn't put anything in the SYN data that has to be *changed*.
>     Re-sent is fine. But CHANGED is not.
>
>
> Right, I get that for interoperability with ENO-unaware stacks there
> is no way to change data presented in the SYN.
Then you're basically stuck between the semantics of ENO (which
presumably are: deliver encrypted data if possible or unencrypted if
not) and TCP (never undo data).

I.e., as long as ENO is silently backward compatible, then ENO SYNs MUST
NOT include data.

> In the case where both ends understand ENO but the server is not
> negotiating it,

I have no idea what that means. TCP only *knows* what is assumed for all
endpoints (the specs) and what it negotiates. Everything else is wrong
unless verified (e.g., as with a cookie like in TFO).

> we can define TCP SYN payload semantics differently: essentially, when
> there is an ENO option in the SYN, an ENO-aware server will do one of
> two things:
>
> (1) If it is negotiating ENO, it will hand the data off to the chosen
> TEP, which will then indicate back to ENO whether to ACK the data or not.
>
> (2) If it is not negotiating ENO, it will not ACK the data, and will
> in fact throw it away and explicitly permit transmission of
> *different* data for the same sequence number range.
>
> choose the behavior.

Agreed, though I really think redefining the TCP data plane for control
is a bad idea...

> The problem is with interoperability with servers that don't
> understand ENO, because they can't make an informed decision here, and
> don't all throw away SYN data.

Exactly.

> Allow the client to assume that a particular server supports ENO, and
> so choose to send SYN data knowing a priori that the server will be
> able to make the choice I outlined above. If you run a service on a
> particular version of Linux that supports ENO, and distribute a phone
> app that contacts only that service, it seems perfectly fine to tell
> your app to assume the server knows ENO.

Clients have no business telling TCP where to put data IMO. They also
have no way of knowing when this *assumption* is wrong.

If you want this sort of "knowledge", you need to build in persistent
cross-connection state at the TCP layer.

TCP isn't "best effort".

Joe

---

From: Jeremy Harris <[email-masked]>
Date: Sat, 30 Jul 2016 16:18:02 +0100

> the SYN, an ENO-aware server will do one of two things:
[...]
> for the same sequence number range.

I don't think so.  Whatever the client is, if the server is not showing
support for different behaviour then the client must assume a standards
TCP-implementing server, meaning that any data sent cannot be altered
in a retransmission.  That applies whether or not the _initial_ ack for
a SYN-with-data acks that data; even if it did not, a later one might.

Unless you're dividing signalling of ENO-aware from ENO-negotiating?
-- 
Jeremy

---

From: David Mazieres <[email-masked]>
Date: Sat, 30 Jul 2016 10:20:39 -0700

> Unless you're dividing signalling of ENO-aware from ENO-negotiating?

No one is suggesting sending data in a SYN segment to a server that does
not support ENO.  In fact, no one is even suggesting sending data in a
SYN segment.  All we want is a minimum specification of what you do if
you receive such data, so that people can capitalize on it in the
future.  The proposal currently on the table is that you do whatever the
data TEP/negotiated TEP says, and if that TEP doesn't say anything, then
you discard the data and don't ACK it.  Moreover, we are saying that
even TEPs that support SYN data shouldn't send any unless requested to
do so by the application or unless they know it won't lead to connection
failure (through some future mechanism we don't currently anticipate).

The long-term plan here is to secure 100% of TCP traffic that isn't
secured by some other protocol such as TLS or SSH.  To get better than
opportunistic encryption, there needs to be logic in applications or
middleware to abort TCP connections that are not properly authenticated.
That logic may well want to permit SYN data, since it will scuttle a
non-ENO connection anyway.  Yes, this is very, very far in the future.
But if the TCPINC working group is successful, then the documents we are
working on today could be relevant in 10-20 years.  So without doing
anything to break TCP today, we want to ease the path to a more secure
Internet without trying to predict the future.  And mandating no
buffering of discarded SYN data is a good way to do that...

Incidentally, this discussion makes me think we need a section of the
non-normative rationale section on SYN data as well...

David

---

From: Joe Touch <[email-masked]>
Date: Sat, 30 Jul 2016 15:00:49 -0700

> On Jul 30, 2016, at 10:20 AM, David Mazieres <[email-masked]> wrote:
> 
> And mandating no

You can do that for ENO, but you cannot assert requirements for legacy stacks. 

The problem is that you keep saying you "know" when the other end supports ENO based on previous state, but if that were really true you wouldn't need to negotiate ENO in the 3WHS. 

Joe

---

From: David Mazieres <[email-masked]>
Date: Sat, 30 Jul 2016 16:28:24 -0700

Joe Touch <[email-masked]> writes:

> The problem is that you keep saying you "know" when the other end
> supports ENO based on previous state, but if that were really true you
> wouldn't need to negotiate ENO in the 3WHS.

No, I'm saying *if* the application knows the other end of a connection
supports ENO, and in particular *if* the application intends to abort an
unencrypted connection anyway, *then* it probably makes sense for the
application to send data in a SYN segment.

You are right that if we could have an internet flag day, we wouldn't
need TCP-ENO at all.  But getting from the status quo to our goal of
~100% encrypted TCP traffic requires multiple stages--first
opportunistic encryption, then shimming in some sort of endpoint
authentication, finally having applications or middleware drop
unauthenticated connections.  If TCPINC succeeds, people may, for
example, configure NFS servers not to talk to workstations over
unencrypted TCP.  By that point it makes to discuss TEPs supporting SYN
data, but not now.  Now the question before the working group is how ENO
should avoid preempting/complicating those future discussions.

So far I haven't heard any objections to anything before the last
paragraph of section 4.7, which means the discussion needs to focus on
that last paragraph, namely:

I think you've objected to the word "using" as too vague, and maybe
would prefer "standard TCP semantics" to make clear we aren't talking
about TFO.  You did not answer my question about whether the word
"consuming" is better than "using".  You also did not answer my question
about whether you would support a sentence saying active openers SHOULD
abort non-ENO connections if they have sent SYN data (though at this
point I think such a sentence would be good).  Your arguments seem
primarily directed at some hypothetical future straw-man TEP sending
encrypted SYN data to non-ENO aware hosts, but no existing TEP proposal
does that and more importantly that is not the question at hand.

So on the topic of TCP-ENO, do you have specific suggestions for
improving the last paragraph of section 4.7 of the draft?

David

---

From: Jeremy Harris <[email-masked]>
Date: Sun, 31 Jul 2016 00:33:33 +0100

On 30/07/16 18:20, David Mazieres wrote:
> SYN segment.

I'm confused.  Kyle was talking about it:

On 29/07/16 20:41, Kyle Rose wrote:
> way to change data presented in the SYN.
[...]

-- 
Jeremy

---

From: Joe Touch <[email-masked]>
Date: Sat, 30 Jul 2016 20:02:03 -0700

On 7/30/2016 4:26 PM, David Mazieres expires 2016-10-28 PDT wrote:
> supports ENO,
It doesn't until the SYN-ACK.

Let me put it this way - if you know the other end supports ENO, then
you clearly don't need to negotiate it in the SYN. Is that what you do?

>  and in particular *if* the application intends to abort an
> unencrypted connection anyway, 
TCP needs to be the one making that decision, because an "app" that
doesn't make that decision could be experiencing TCP semantics that are
incorrect.

> *then* it probably makes sense for the
An app doesn't make that decision - it can do things to prevent it from
happening (i.e., waiting for the connection to be open before writing
data), but it can't force this behavior. It's not in the user interface
defined in RFC793, so you can't assume it (unless you're creating this
definition solely for ENO, which seems far out of scope to me).

> unauthenticated connections. 

That's one possible future. Another is one where TCPINC dies off and is
replaced by something else, either at the TCP layer or IP.

>  If TCPINC succeeds, people may, for
It also needs to avoid adverse interactions with not using ENO at all.

> that last paragraph, namely:
First, just because I say the food is salty, doesn't mean it's great if
you fix the salt level. But yes, this part needs fixing.

> I think you've objected to the word "using" as too vague, and maybe
Yes..

> about TFO.  
Yes.

Then there's the part about middleboxes (it's useless to make "laws for
the lawless", i.e., if your goal is to modify TCP in ways that are
compliant with all middleboxes, you can stop now).

Then there's the part about non-compliant hosts - actually, "using" data
in the SYN violates the semantics of TCP for *compliant* hosts.

> You did not answer my question about whether the word
> "consuming" is better than "using". 
It's needlessly anthropomorphic.

The answer is simple - if the receiver (ENO or otherwise) uses the
information in the SYN data (i.e., it changes its behavior in any way
based on that content), then it MUST happen after TFO only (at this
point) or a true equivalent. There is *no* aspect of ENO that currently
supports that level of equivalence - i.e., *validated* by a SYN cookie.
You don't get to make that assumption based on any "belief" of the other
end's state

>  You also did not answer my question
> point I think such a sentence would be good). 
MUST, and it MUST occur inside TCP. You can't assume that happens at the
application (user) layer.

Note that I already said this when I gave you two choices:
a) never send anything you can't retract in the SYN-ACK
or
b) always abort - inside TCP - a connection where you would have needed
to do that retraction

> Your arguments seem
You seem to be operating under the mistaken belief that sending data to
the same IP address twice always reaches the same TCP context. I.e.,
when you send TCP-encrypted SYN data to a host, you have NO WAY OF
KNOWING whether that host is ENO-capable until AFTER the SYN-ACK.

Otherwise, if you did COULD already know that, you clearly wouldn't need
the ENO flag at all. Since you need that flag, then you need to consider
the case where it is wrong.

> improving the last paragraph of section 4.7 of the draft?

See above. Note that this does not imply that I consider the entire
draft otherwise correct.

Joe

---

From: Kyle Rose <[email-masked]>
Date: Sun, 31 Jul 2016 00:09:31 -0400

I'm not the author, so nothing to see here. :-) I was not considering
David's line of thinking about future mandatory application aware uses:
rather, I was speculating about how ENO could make use of SYN data in
general, which immediately led to the question in my OP about TCP's actual
(normative and observed) semantics regarding SYN data in the absence of TFO.

The thread has split, and I'm happy to let David represent his intentions.

Kyle

On Jul 30, 2016 7:33 PM, "Jeremy Harris" <[email-masked]> wrote:

> TEP,

---

From: David Mazieres <[email-masked]>
Date: Sat, 30 Jul 2016 23:59:32 -0700

Okay, guys.  This is getting long and feels a bit pedantic, but how
about the following?

David

4.7.  Data in SYN segments

   The last TEP identifier suboption in host A's SYN segment is the _SYN
   TEP_.  The use of data in host A's SYN segment is goverend by the SYN
   TEP.  If the SYN TEP's specification does not define the use of SYN
   data, then host A's SYN segment MUST NOT contain data and host B MUST
   discard any SYN data.  Host B MUST also discard SYN data if either
   the SYN TEP differs from the negotiated TEP or host B disables
   encryption.

   The use of data in B's SYN-ACK segment is governed by the negotiated
   TEP.  If the negotiated TEP's specification does not define the use
   of SYN data, then host B's SYN-ACK segment MUST NOT contain data and
   host A MUST discard any SYN data.  Host A MUST also discard SYN data
   if host A disables encryption.  In the event that host B is an active
   opener (because of simultaneous open), host B's SYN-only segment MUST
   NOT include data.

   A host MUST NOT act on discarded data under any circumstances.  In
   particular, after a host discards SYN data, if a later non-SYN
   segment contains different data for the same sequence numbers, the
   host MUST process the contents of the non-SYN segment only.
   Furthermore, when a host discards SYN data, it MUST NOT acknowledge
   the sequence number of the discarded data.  Rather, it MUST
   acknowledge the other host's initial sequence number as if the
   received SYN segment contained no data.

   If host A's SYN segment contains data, host B's SYN-ACK segment
   acknowledges that data, and either encryption is disabled or the
   negotiated TEP is not the SYN TEP, then host A MUST reset the TCP
   connection.  Under such circumstances, the acknowledged data is
   defined by the SYN TEP, which does not govern the connection.  Hence,
   there is significant risk of host B misinterpreting SYN data.

   If host A sends SYN data, host B does not acknowledge the SYN data,
   and host B's SYN-ACK segment does not include an ENO option, then
   host A SHOULD reset the connection.  The reason is that a legacy host
   B, if it does not implement ENO, could buffer and later act on SYN
   data even without first acknowledging that data.  Barring empirical
   evidence against the existence of unacknowledged data buffering, it
   is not safe for host A to assume its SYN data has been discarded.

   To avoid unexpected resets, TEPs SHOULD support a normal mode of
   operation that does not make use of data in SYN segments, and SHOULD
   enable data in host A's SYN segment only if explicitly requested by
   the application or in cases where such use is unlikely to cause
   connection failure.  Implementations MAY provide a per-connection
   mandatory-encryption mode that automatically resets a connection if
   ENO fails.  If supported by the TEP, host A MAY enable SYN data when
   the application requests mandatory-encryption mode.

---

From: Derek Fawcus <[email-masked]>
Date: Sun, 31 Jul 2016 22:26:46 +0100

> you clearly don't need to negotiate it in the SYN. Is that what you do?

If I am contacting a server I control from some arbitrary point within the
internet, then I do know what the far end supports.  It may well be that
a middlebox interfers with the connection,  but I can know a priori how
the other end would respond assuming no interfering middleboxes.

So for some scenario's:
 a) connecting to my home machines from my parents network - I know I will
    have an unadulterated connection between machines I control.
 b) connecting to my home machines from my office - I have a NAT'ed
    connection with known characteristics.
 c) connecting so my home machine from say an airport lounge - I almost
    certainly have a NAT'ed connection,  and with unknown characteristics.

So I can know the box I am connecting to should have a known set of capabilities.
As to why ENO still needs to be negotiated,  well because the box connecting
to my home machine may well be from one which does not support ENO.

> definition solely for ENO, which seems far out of scope to me).

I don't really view the model APIs mentioned in RFCs as perscriptive,
rather the over the wire formats are.  So if something can be sent over
the wire,  a user interface can be created to exercise it.

[snip]

> in the SYN violates the semantics of TCP for *compliant* hosts.

Well RFC793 implies that,  I've closest I/ come to spotting the bit stating
that is the text saying data in the syn is handled in the established state.

However,  other than strictly legalistic reading,  I can't see a valid objection.
A syn can contain data.  A syn-act can contain data.  The syn-ack data could be
perceived to be a function of the syn data,  but how does it harm any third party
if it is or is not?  The worst that can happen (in this case) would seem to be
a failure to establish an connection,  and a wasted RTT (or two).

> You don't get to make that assumption based on any "belief" of the other

But one does based upon knowledge of the other ends state;  so it won't
necessarily be available for opportunistic cases,  but for some cases
connecting to know end nodes it will.

> b) always abort - inside TCP - a connection where you would have needed

So in the scenario I give above,  if the host one know's about does not
support ENO,  then there is a MITM messing with things,  and I'd suggest
one should abort the connection.  As to if the stack should then attempt
an automatic retry w/o ENO,  I've not given enough though.

DF

---

From: Joe Touch <[email-masked]>
Date: Sun, 31 Jul 2016 16:47:52 -0700

> the other end would respond assuming no interfering middleboxes.

If you know as much as you claim, then you could avoid sending ENO
option completely.

...

> to my home machine may well be from one which does not support ENO.

That considers only the case where the call originator does not support
ENO. TCP extensions need to support all cases of interactions between
upgraded and legacy endpoints.

> the wire,  a user interface can be created to exercise it.
Protocols are defined by the following entire set:
- messages on the wire
- state at the endpoints
- the interface above (for TCP, that's called the "application" interface)
- the interface below (for TCP, that's IPv4 or IPv6 and its upper-layer
interface)
- timers
- the rules that describe the relationship between the items above as a FSM

The idea that  "if something can be sent on the wire, then the API needs
to be able to control it" fails to consider that upper layer behaviors
act on the FSM and the FSM generates the wire messages.

An API isn't simply a way to write packets. That would be true only for
a degenerate protocol whose FSM is the identity function (and that's not
TCP, in particular).

> that is the text saying data in the syn is handled in the established state.
It's there in the "SEGMENT ARRIVES" processing.

> However,  other than strictly legalistic reading,  I can't see a valid objection.
Protocol specifications aren't suggestions. They are there exactly for
this sort of legalistic reason.

The objection is simple - TCP ensures reliable delivery. Acting on SYN
data before the end of the 3WHS violates that reliability requirement.

TFO gets around this by using the establishment of a cookie from the
previous connection as a sort of "pending" new connection, thus the SYN
data in TFO is considered in the context of the end of this partially
completed handshake, rather than as the start of a completely new handshake.

> if it is or is not? 

Because an endpoint that follows RFC793 would NEVER make the SYN-ACK
data depend on the SYN data.

> The worst that can happen (in this case) would seem to be

The worst thing would be acting on data that has not been reliably
transmitted, which means you have TCP but it no longer is reliable.
That's no longer TCP.

> But one does based upon knowledge of the other ends state;
I'll try not to repeat this too much, but you KNOW NOTHING until the end
of the 3WHS or if you use TFO. Everything else you think you know can -
and will - be wrong.

>  so it won't
> one should abort the connection. 
It's not just MITM - the endpoint has a right to change (rollback its
OS, etc.).

But yes, aborting the connection *inside TCP* is the only correct
behavior if you send SYN data that turns out to be incorrect.

>  As to if the stack should then attempt

That would determine whether your solution is robust or not. Do you
really want to be the one to blame when (not if) this sort of thing happens?

Joe

---

From: Joe Touch <[email-masked]>
Date: Sun, 31 Jul 2016 17:05:28 -0700

Hi, David,

There are two problems with the text:
1) it's never clear whether you're talking about:
    - all hosts, including legacy ones
    - hosts that support ENO, whether the segment includes the ENO
option or not
    - hosts that support ENO and are processing a segment including the
ENO option
Wherever you say "host" below, you should be clear about the case you're
indicating.

2) the text is still incorrect about interaction with legacy hosts

Joe

On 7/30/2016 11:59 PM, David Mazieres wrote:
>    host A SHOULD reset the connection. 
MUST. There is no case where you can do otherwise and retain the
semantics of TCP to the legacy host.

>  The reason is that a legacy host
>    data even without first acknowledging that data.
That's incorrect - what it CAN do is buffer the SYN data and ACK it
later, and act on it once ACKd.  The issue is that TCP does not have a
strict rule for what happens if data is received but not ACKd and later
overwritten - there's no rule of "latest data is kept".

>   Barring empirical

The above is fine, FWIW, and what I was expecting - i.e., it's OK to
send data in the SYN when you think your bet is likely to win, and that
also means you're willing to take the RESET hit on legacy endpoints if
you're wrong. So the gamble is yours to win or pay the cost of.

Joe