Subject: [tcpm] More TCP option space in a SYN: draft-briscoe-tcpm-syn-op-sis-02

From: Bob Briscoe <[email-masked]>
Date: Mon, 22 Sep 2014 15:55:42 +0100

Joe,

I've just posted a revision here:
<http://tools.ietf.org/html/draft-briscoe-tcpm-syn-op-sis-02>

* I've taken more care to explain it. It was a bit terse before. So 
it's ready for the list to review properly now.

It would be useful if someone wrote a comparison with the other ideas 
on how to do this, including the ones we wrote jointly and SLO, LOIC 
and DBormam's 4-way HS. Better someone independent of all the authors 
does this. But I want to get syn-op-sis right first, a comparison 
won't be much use until its stable.

The underlying idea is still the same. The reasons I've updated it 
are many (the change log taken from the appendix is pasted at the end 
of this email for convenience).

* Mainly I've articulated that all these approaches that add TCP 
options after the Data Offset are not just hacks to get more space; 
they add a new architectural capability to TCP - a distinction 
between TCP options that might be relevant to middleboxes, and ones 
that are only for the destination (a bit like destopt in IPv6 - 
needing to specify which TCP options are for the destination only is 
ironic but necessary).

* I've also realised that this approach will ultimately return to a 
single SYN solution, whereas both the ideas in 
draft-touch-tcpm-tcp-syn-ext-opt will always require two SYNs.

Bob

Pasted from the change log:

    From briscoe...-01 to briscoe...-02:

       Technical changes:

       *  Defined the client behaviour dependent on which response
          arrives first.

       *  Allowed retransmission of either SYN or SYN-U if no response
          from either.

       *  Redefined EOO as an offset from the end of the packet, not from
          the beginning of the payload.

       *  Added section on Migration to a Single Handshake.  Reworded
          dual handshake so that it is not mandatory for the client to
          send dual SYNs simultaneously; only the relation between the
          SYNs and the response to either is mandatory, while parallel
          SYNs is purely for latency reduction.

       *  Added rules for writing TCP options, i.e. i) options like TFO
          MUST NOT be located in the TCP header and ii) add no-ops to
          align on 4-octet boundary.

       *  Added rules for forwarding TCP options, i.e. only the
          destination looks for TCP options after the Data Offset, not
          middleboxes.

       *  Moved the Explicit Handshake variant (SYN-L) into the body from
          the appendix, and recommended the choice could be down to
          implementers or apps.  Included section on corner cases.

       *  Introduced more normative language throughout the Protocol
          Spec.

       Editorial changes:

       *  Added temporary motivation section

       *  Added confusible terminology to Terminology section.

       *  Divided protocol spec into sub-sections.

       *  Handshake table: Clarified that the two columns under each
          server represent separate threads, that may run on separate
          servers, without co-ordination.  Represented message
          dependencies in the alignment of the rows.

       *  Explained the table.

       *  Explained why a legacy server won't ever pass SYN-U to the app.

       *  More precisely described loss as 'not arrived before a
          timeout', and explained the tradeoff between latency and extra
          TCP options.

       *  Gave reasoning for locating TCP options in three groups.

       *  Acknowledged Rob Hancock for the architectural idea of hiding
          an extension to a protocol in the layer above.

       *  Appendix about protocol alternatives now only presents the SYN-
          UD alternative, given the implicit/explicit handshake choice
          has been moved to the body.

       *  Rewrote appendix about comparing the choices to treat the two
          pairs of choices separately, rather than discussing all four
          combinations of pairs of choices.

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: John Leslie <[email-masked]>
Date: Tue, 23 Sep 2014 16:43:39 -0400

> it's ready for the list to review properly now.

   I won't claim to have read the whole thing (24 pages); but I think
I've read enough to justify a quick response (and I have a meeting
startig soon).

   Bob, as usual, has a very good idea -- if a bit over-complicated.

   Placing the additional options at the end of the packet has real
promise.

   I recommend that the response (acknowledging the SYN-U option)
include a checksum or hash to verify what additional options have
been received.

   I strongly recommend against trying to integrate this with the
already adopted after-the-initial-SYN draft.

   Much of the complexity is inevitable, alas, but a rewrite to
change the order of presentation and remove any differences in the
data on-the-wire would really help both comprehension and adoption,
IMHO.

   Also, in the interests of attracting adoption, I'd recommend
first presenting the case we hope to finally achieve -- a single
connection opened with the SYN-U -- immediately followed with a
list of problems we may see. Then in sections following, we can
introduce the overlapping-SYNs hack and other work-arounds.

   (IMHO there's nothing wrong with initial experiments which do NOT
use the overlappingSYNs.)

   More tomorrow...

--
John Leslie <[email-masked]>

---

From: Joe Touch <[email-masked]>
Date: Wed, 24 Sep 2014 15:25:30 -0700

Hi, Bob (et al.),

It's good to have a more detailed description of the proposal.

I still find a dual-SYN solution untenable, though, as it has been for
other upgrade paths in the past (e.g., IPv6).

Also, I don't particularly think that putting the options at the end of
the segment serves any useful purpose other than to make this option
handle the option space in a different way than the default DO header field.

I'm glad to hear what others think and agree that an unbiased comparison
of the approaches would be a useful place to for the discussion to start...

Joe

On 9/22/2014 7:55 AM, Bob Briscoe wrote:
> * I've taken more care to explain it. It was a bit terse before. So it's
> ready for the list to review properly now.
> 
> It would be useful if someone wrote a comparison with the other ideas on
> how to do this, including the ones we wrote jointly and SLO, LOIC and
> DBormam's 4-way HS. Better someone independent of all the authors does
> this. But I want to get syn-op-sis right first, a comparison won't be
> much use until its stable.
> 
> 
> The underlying idea is still the same. The reasons I've updated it are
> many (the change log taken from the appendix is pasted at the end of
> this email for convenience).
> 
> * Mainly I've articulated that all these approaches that add TCP options
> after the Data Offset are not just hacks to get more space; they add a
> new architectural capability to TCP - a distinction between TCP options
> that might be relevant to middleboxes, and ones that are only for the
> destination (a bit like destopt in IPv6 - needing to specify which TCP
> options are for the destination only is ironic but necessary).
>

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 25 Sep 2014 10:56:35 +0100

Joe,

>other upgrade paths in the past (e.g., IPv6).

That's why I prefer syn-op-sis because it only uses 2 SYNs for transition.

I've included a new section on ways to use only 1 SYN during 
transition (building a white-list) and ultimately moving to 1 SYN in 
the future, only falling back to the legacy SYN in series rather than 
parallel if you hit a legacy listener.

The other two in tcp-syn-ext-opt have to be 2-SYN for ever.

>handle the option space in a different way than the default DO header field.

That's the architectural point.
There's also a tactical point. It should get through every middlebox.

>of the approaches would be a useful place to for the discussion to start...

Yes, I'm already repeating myself - you're right we need wider opinions.

Bob

---

From: Joe Touch <[email-masked]>
Date: Thu, 25 Sep 2014 07:53:01 -0700

On 9/25/2014 2:56 AM, Bob Briscoe wrote:
> I've included a new section on ways to use only 1 SYN during transition
> (building a white-list) and ultimately moving to 1 SYN in the future,
> only falling back to the legacy SYN in series rather than parallel if
> you hit a legacy listener.
> 

But they don't both use two SYNs. One (that I prefer) uses an additional
out-of-band segment.

IMO, using two SYNs is problematic - regardless of whether the upgraded
system uses both or just one.

IMO, we're going to be dealing with legacy TCPs for a *very* long time.

Time for others to jump in ;-)

Joe

---

From: "Zimmermann, Alexander" <[email-masked]>
Date: Thu, 25 Sep 2014 15:13:46 +0000

Bob,

Am 25.09.2014 um 05:56 schrieb Bob Briscoe <[email-masked]>:

> I've included a new section on ways to use only 1 SYN during transition (building a white-list) and ultimately moving to 1 SYN in the future, only falling back to the legacy SYN in series rather than parallel if you hit a legacy listener.
> 
> There's also a tactical point. It should get through every middle box.

but do we have not exactly the same problem w/ TCPinc. With TCPinc we have
the problem too that a middle box doesn’t see any understandable after the
legacy TCP header (e.g. no HTTP header after TCP port 80). So, why it’s
acceptable for TCPinc but not for extending SYN option space? BTW any
tcpdump user will love to have the options in front of a segment and not
at the end ;-)

Alex

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 25 Sep 2014 16:59:15 +0100

Alex,

At 16:13 25/09/2014, Zimmermann, Alexander wrote:
>the problem too that a middle box doesn't see any understandable after the
>legacy TCP header (e.g. no HTTP header after TCP port 80). So, why it's

tcpinc is about encrypting payload. I'm interested in extending 
opportunistic encryption to TCP options, which is why I want to 
separate options into 'middlebox' and 'destination' (note: zero 
middlebox options would be valid).

Right now I'm working on a tcpcrypt-like approach, to see if I can 
get the cipher negotiation done in zero RTT (at least on a connection 
resume), so I can use it to encrypt TCP options after the payload (as 
well encrypting any payload, of course).

This will then /enforce/ the middlebox-v-destination categorisation of options.

I left any discussion of this out of syn-op-sis-02, because I'd 
rather not talk about possibilities until I know whether they are 
possible. (I guess that means I need to apologise for talking about 
possibilities on the list, but I couldn't help it, given your posting 
was so close to what I am thinking :|

Bob

---

From: Andrew Yourtchenko <[email-masked]>
Date: Thu, 25 Sep 2014 18:41:24 +0200

Bob, Joe, all,

some comments below, maybe with some obvious questions as I did not follow 
the latest discussions very closely.

> That's why I prefer syn-op-sis because it only uses 2 SYNs for transition.

Both syn-op-sis and tcp-syn-ext-opt talk about using
different port pairs - is this due to the explosion with the 
number of possibilities for the three-way handshake recovery ?

(Of course the application will anyway see only 1 file descriptor, which 
will assume the identity of the connection that succeeds, right ?)

> (building a white-list) and ultimately moving to 1 SYN in the future, only 
> falling back to the legacy SYN in series rather than parallel if you hit a 
> legacy listener.

The fundamental difference between the transition to the extended option 
space and IPv4->IPv6 transition is the absence of the DNS to signal the 
capabilities of the endpoint before the connection even starts - if the 
target host has A&AAAA you only have to deal with the failures on the 
path, while if the target host does not have AAAA record, it's worthless 
to try IPv6.

OTOH, when attempt to negotiate the extended option space, you do 
not have any hints about the remote end - besides maybe past memory.

In RFC6555 we tried to avoid dual-SYN as much as possible by providing a 
headstart that shrinks and inverses on failures - so the reference to 
RFC6555 being "parallel" is not correct, see section 5.5:

    The primary purpose of Happy Eyeballs is to reduce the wait time for
    a dual-stack connection to complete, especially when the IPv6 path is
    broken and IPv6 is preferred.  Aggressive timeouts (on the order of
    tens of milliseconds) achieve this goal, but at the cost of network
    traffic.  This network traffic may be billable on certain networks,
    will create state on some middleboxes (e.g., firewalls, intrusion
    detection systems, NATs), and will consume ports if IPv4 addresses
    are shared.  For these reasons, it is RECOMMENDED that connection
    attempts be paced to give connections a chance to complete.  It is
    RECOMMENDED that connection attempts be paced 150-250 ms apart to
    balance human factors against network load.

In today's conditions, a simple 300ms headstart for IPv6 if AAAA is 
advertised might be sufficient, as the subpar IPv6 connectivity
disappears, and more of the corner cases get into play for RTTs.

So, depending on how much of a drop rate we'd expect for a "new" SYN 
across the variety of network scenarios, the simpler approach of just 
using 300ms delay could be reasonable due to its simplicity - an 
experiment could show.

Doing this experiment on both address families at once could also reveal 
the drop rates to be different - I'd expect the IPv6 drop rates to be 
smaller - so maybe there is an opportunity to simplify things - if it 
"just works" in IPv6 using a single fancy-looking SYN in a sufficient %% 
of cases, that could be a great simplification *and* will give additional 
use case for IPv6 assuming there is enough demand for more space in the 
options.

> There's also a tactical point. It should get through every middlebox.

Help me understand this argument. If we were sending the "old" and "new" 
SYNs on the same 5-tuple - sure, the data must be identical.
But if source ports are different, then a typical middlebox is either 
happy about the "data in the SYN" or not ?

Or, are you saying that placing them at the end of the packet makes it 
difficult for the current architectures to "grab" them and mess around 
with them ? Sure, it does. Especially higher-performing boxes might look 
at the start of the packet.

But that's *today's* architectures - if someone pays one of my employer's 
BUs enough to create a box that fiddles with these options, that box 
will get created. Not an architecturally beautiful reality, but it is 
there.

> Yes, I'm already repeating myself - you're right we need wider opinions.

Even though I am nominally "on the list of alternatives" due to LOIC, I've 
disliked the idea shortly after, after trying to implement it - the linux 
kernel's accelerated NIC path showed that the invalid CRC would be 
dropped very early, before the software can see it - so I would argue 
against that approach today from implementation standpoint as well.

Hopefully arguing with my past self does buy me an "unbiased" ribbon, or 
at least a half of it :-)

--a

---

From: Joe Touch <[email-masked]>
Date: Thu, 25 Sep 2014 10:16:53 -0700

On 9/25/2014 9:41 AM, Andrew Yourtchenko wrote:
> Bob, Joe, all,
> 
> some comments below, maybe with some obvious questions as I did not
> follow the latest discussions very closely.
> 
> different port pairs 

tcp-syn-ext-opt describes two different approaches:

	SYN-EOS-OOB	uses one port pair, sends only one SYN

	SYN-EOS-DS	uses two port pairs

the DS variant is very similar to the one in syn-op-sis; both are Bob's
suggestion. Mine was SYN-EOS-OOB.

Bob - let me know if its useful to retain SYN-EOS-DS or if you want to
replace it with syn-op-sis.

> - is this due to the explosion with the number of
> possibilities for the three-way handshake recovery ?

Two ports are used only when two SYNs are sent, largely to avoid the
second SYN from affecting the progress of the first that arrives at a
legacy endpoint.

> will assume the identity of the connection that succeeds, right ?)

Yes.

> OTOH, when attempt to negotiate the extended option space, you do not
> have any hints about the remote end - besides maybe past memory.

You could have similar hints in SRV records, though those are used much
less frequently.

...
> using 300ms delay could be reasonable due to its simplicity - an

Sure, but it also has creates delay on either the legacy or upgraded
connection - depending on which SYN is stalled.

That's the advantage of the OOB method - there's no need to insert a
stall on the sending side (an upgraded receiver might cache OOBs that
arrive briefly before SYNs to avoid the need for an additional exchange)

>> The other two in tcp-syn-ext-opt have to be 2-SYN for ever.

They're two segments forever. Only one is two SYNs.

Joe

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 25 Sep 2014 19:42:28 +0100

Andrew,

At 17:41 25/09/2014, Andrew Yourtchenko wrote:
>different port pairs - is this due to the explosion with the number 
>of possibilities for the three-way handshake recovery ?

SYN-EOS OOB (within tcp-syn-ext-opt) uses the same src & dst ports.
SYN-EOS DS (within tcp-syn-ext-opt) and syn-op-sis use same dst, but 
different src.

Rationale for latter: too many middleboxes out there that would 
reject a SYN to/from the same ports, in their misguided attempts to 
be 'helpful' by removing SYNs that appear to be duplicate or same 
4-tuple but different initial sequence no. Not only many firewalls, 
but all split connections would not forward the second SYN.

>(Of course the application will anyway see only 1 file descriptor, 
>which will assume the identity of the connection that succeeds, right ?)

Correct.

>>transition (building a white-list) and ultimately moving to 1 SYN 
>>in the future, only falling back to the legacy SYN in series rather 
>>than parallel if you hit a legacy listener.
>
>The fundamental difference between the transition to the extended 
>option space and IPv4->IPv6 transition is the absence of the DNS to 
>signal the capabilities of the endpoint before the connection even 
>starts - if the target host has A&AAAA you only have to deal with 
>the failures on the path, while if the target host does not have 
>AAAA record, it's worthless to try IPv6.
>
>In RFC6555 we tried to avoid dual-SYN as much as possible by 
>providing a headstart that shrinks and inverses on failures - so the 
>reference to RFC6555 being "parallel" is not correct, see section 5.5:
>

The dual SYN approaches could use a delay between the SYNs like 6655 
does (not nec exactly the same numbers). However, in the short flow 
world, I would argue that 250ms is completely unacceptable unless 
you're fairly certain the first one will work, which is more-or-less 
the v4/v6 scenario that happy eyeballs addressed. Reasoning: multiply 
250ms by a few connections with serial dependencies in a sequence and 
you have delays of seconds.

I've made a note to change the description of 6555 in the next rev - 
clearly the word parallel is not strictly correct anyway - one serial 
(e.g. network) interface can't actually send two SYNs in parallel, 
the closest they can leave is back-to-back.

Regarding billable total traffic:
Damon Wischik once proposed to improve performance by duplicating the 
most sensitive packets at flow start:
         "three copies of the first packet of every message, two 
copies of the second and one copy of subsequent packets"
He calculated these 6 extra full-sized packets per flow would (in 
2005) increase total traffic by 2.7% (because elephants dominate 
total traffic).
<http://rsta.royalsocietypublishing.org/content/366/1872/1941.full>
Duplicating just the legacy part of the SYN (60B max) would have a 
much smaller total effect (I could calculate it if you want - 
probably sub-0.1%). Certainly too small for people to care about the 
increment on bills or even packet load.

>across the variety of network scenarios, the simpler approach of 
>just using 300ms delay could be reasonable due to its simplicity - 
>an experiment could show.
>
>Doing this experiment on both address families at once could also 
>reveal the drop rates to be different - I'd expect the IPv6 drop 
>rates to be smaller - so maybe there is an opportunity to simplify 
>things - if it "just works" in IPv6 using a single fancy-looking SYN 
>in a sufficient %% of cases, that could be a great simplification 
>*and* will give additional use case for IPv6 assuming there is 
>enough demand for more space in the options.

The delay between the pair should be implementation dependent anyway. 
No point arguing this to death in standards... but I will at least 
state the opposing view...

The world is changing. Latency is everything and soon it will be even 
more than everything. If one values latency, the delay between the 
two will be as small as possible.

My simple view of the world is that, dual SYN seems the /only/ way to 
extend TCP option space. I'm not proud of it. It's not nice. But it's 
necessary. If dual SYNs require more memory for connection state, 
it's OK as long as the need for more memory appears slowly enough to 
be able to deploy it. If this is the price of extra capabilities, and 
those capabilities are valued, then we invest in memory to provide 
the capabilities.

Whether the number of connections that need >40B TCP options will 
suddenly jump, or gradually grow, I can't predict tho.

>Help me understand this argument. If we were sending the "old" and 
>"new" SYNs on the same 5-tuple - sure, the data must be identical.
>But if source ports are different, then a typical middlebox is 
>either happy about the "data in the SYN" or not ?
>
>Or, are you saying that placing them at the end of the packet makes 
>it difficult for the current architectures to "grab" them and mess 
>around with them ? Sure, it does. Especially higher-performing boxes 
>might look at the start of the packet.
>
>But that's *today's* architectures - if someone pays one of my 
>employer's BUs enough to create a box that fiddles with these 
>options, that box will get created. Not an architecturally beautiful 
>reality, but it is there.

Two clarifications:

1) By putting options after the payload, I'm including the (currently 
common) case of a SYN with no payload. Then the extra options will 
start immediately at the Data Offset.

2) A longer clarification...

Let's separate middleboxes into two types (I'm trying to deal with both):
- (meddleboxes) those that want to mess with headers for security
- (muddleboxes) those that happen to mess with headers, because their 
developers never stopped to think that TCP might change.

Muddleboxes tend to forward the payload unaltered. Some muddleboxes 
look at HTTP headers and try to be clever. By putting extra options 
after the Data Offset, they will pass the former. By putting them at 
the end of the payload they will mostly pass the latter.

Meddleboxes might well evolve to find these new extra options at the 
end of the packet. But in the meantime, we've got over the transition 
of deploying a new protocol. Then the meddleboxes are on the back 
foot, and they have to justify removing these packets that are useful 
to someone. If they can - e.g. for security reasons - then fine. But 
they are not just removing packets because they are using a new rev 
of the protocol.

As I said, the most important thing is we will have separated TCP 
options into two sets:
- Those that middleboxes (past and future) can mess with
- Those that middleboxes (future) should think much harder about 
before messing with them

Then, this (maybe) enables the next step - for endpoints to encrypt 
those options they don't want middleboxes to interact with at all. 
(See my response to Alex Zimmerman's post in this thread.)

>Even though I am nominally "on the list of alternatives" due to 
>LOIC, I've disliked the idea shortly after, after trying to 
>implement it - the linux kernel's accelerated NIC path showed that 
>the invalid CRC would be dropped very early, before the software can 
>see it - so I would argue against that approach today from 
>implementation standpoint as well.
>
>Hopefully arguing with my past self does buy me an "unbiased" 
>ribbon, or at least a half of it :-)

Certainly, this seems to give you the role of unbiased /and/ educated 
by experience.

I'm also no longer like an idea that my past self proposed (SYN-EOS 
DS), for the same reasons Joe didn't like it - the two SYNs will 
often not end up in the same place to be re-united. But I still have 
a candidate I do prefer (syn-op-sis). That rules me out for the role 
of innocent bystander.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 25 Sep 2014 19:51:26 +0100

Joe,

>replace it with syn-op-sis.

In my mind syn-op-sis solves problems with SYN-EOS-DS and doesn't 
introduce any new problems (AFAIK). So it's a pure upgrade.

Let's have an author-huddle off-list about how we deal with these.

Sorry for my last post crossing your response below.

---

From: Andrew Yourtchenko <[email-masked]>
Date: Thu, 25 Sep 2014 21:21:54 +0200

> suggestion. Mine was SYN-EOS-OOB.

Thanks for the clarification!

> second SYN from affecting the progress of the first that arrives at a

Makes sense, if both are valid SYNs for the legacy endpoint, indeed.

> You could have similar hints in SRV records, though those are used much

They're used quite frequently outside of HTTP
(http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/dns-srv-record-use-by-clients.html)
though the "outside of HTTP" part makes it quite infrequent.

OTOH, the SRV record has just priority,weight,port,target-name - there is 
no space for the hint as such. OTOH maybe a new record type that could be 
returned in "additional" field within the reply?

Regardless, the DNS "hint"'s presence could change the game quite a lot - 
if you know that the remote end is supporting the new cool extension, you 
you might want to make a leap of faith and give it a chance first, with 
the quick enough fallback to make the impact minimal yet avoid two SYNs if 
possible.

> connection - depending on which SYN is stalled.

Exactly. The connecting side may decide it makes sense if it knows upfront 
from either the hint in DNS, or its own cache of previous connections to 
that host (which may not be valid, BTW - because the network inbetween 
might have changed)

> arrive briefly before SYNs to avoid the need for an additional exchange)

It's a tradeoff.

two-SYN methods stall one of the connections during the 
transition period at initiator's discretion (with maybe using DNS hints 
or its local cache to decide which one to stall and the protocol type to 
decide for how long to stall), *or* cause two SYNs race which is not very 
nice. Assuming the stall delay is X, the session establishment time in the 
worst case is ~(1.5RTT+X)

OOB method has a permanent decision to stall the connection if the 
upgraded server receives SYN-EOS OOB but no OOB segment (say, it's 
being blocked by the middleboxes?), "after a certain time". This time 
supposedly should be much shorter than the stall in the 
two-SYN case, ideally a function of jitter on the path, let's call it Y.

If the client then proceeds as if the connection was legacy, the session 
establishment time is ~(1.5RTT+Y) - so, better than two-SYNs.

But if the client decides to retransmit the OOB (the worse performing 
case of the, the overall connection establishment time will be 
~(2.5*RTT+Y). And there'd need to be logic to still fallback to the legacy 
behavior if the (Second, third?) OOB packet gets dropped.

If the above estimates make sense, then it's a tough call to make 
which one is better without trying in the wild, IMHO.

--a

---

From: Joe Touch <[email-masked]>
Date: Thu, 25 Sep 2014 13:17:08 -0700

On 9/25/2014 12:21 PM, Andrew Yourtchenko wrote:
...
...
> OTOH, the SRV record has just priority,weight,port,target-name - there
> is no space for the hint as such. OTOH maybe a new record type that
> could be returned in "additional" field within the reply?

SRV records sometimes define additional parameters in the TXT records.
See here:
www.iana.org/assignments/service-names-port-numbers/

(search for TXT)

...
> two-SYN methods stall one of the connections during the transition
> period at initiator's discretion (with maybe using DNS hints or its
> local cache to decide which one to stall and the protocol type to decide
> for how long to stall), *or* cause two SYNs race which is not very nice.
> Assuming the stall delay is X, the session establishment time in the
> upgraded server receives SYN-EOS OOB but no OOB segment (say, it's being
> blocked by the middleboxes?), "after a certain time". This time
> supposedly should be much shorter than the stall in the two-SYN case,
> ideally a function of jitter on the path, let's call it Y.

That can be zero if the OOB arrives first or during the processing of
the SYN.

> establishment time is ~(1.5RTT+Y) - so, better than two-SYNs.

A legacy client wouldn't wait for the jitter - they would ignore the
SYN-EOS flag in the SYN, and wouldn't know to wait. The OOB segment
would get silently dropped if/when it arrives.

> ~(2.5*RTT+Y). And there'd need to be logic to still fallback to the
> legacy behavior if the (Second, third?) OOB packet gets dropped.

The client need not wait for a RTT to retransmit the OOB; multiple OOBs
have no impact on the server. Clients can send an OOB before and after
the SYN, and if the OOB gets there and is cached it wins; if the OOB
arrives after, the delay is just the arrival delay.

Joe

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 26 Sep 2014 00:48:33 +0200

> www.iana.org/assignments/service-names-port-numbers/

geez! Thanks for this link. I thought the uses of TXT beyond "descriptive 
text" were always very much frowned upon.

> That can be zero if the OOB arrives first or during the processing of

Sure. It's not about reordering. It's about the worst case scenario.

Best case scenarios "when everything works" both OOB and 2-SYNs are 
~equivalent, the mild failure cases are nicer with OOB iff Y is smaller 
than X. OOB has the additional advantage of not consuming one more 
connection slot for a short period of time on the middleboxes (and the 
per-flow processing capacity).

> would get silently dropped if/when it arrives.

We seem to somehow miscommunicate the case description. Sorry, it's late 
here, maybe mea culpa.

The case under discussion, which is the worst case scenario is: new client 
talking to new server with an evilbox in the middle dropping the OOB 100% 
of the time.

> arrives after, the delay is just the arrival delay.

See above on the definition of the worst case - the OOB spec does not 
appear to be clear. Maybe it's better to leave more ambiguity so this can 
be experimented with, but this this worst case was certainly at least to 
me one of the big differences between the 2-SYN and OOB approaches.

--a

---

From: Joe Touch <[email-masked]>
Date: Thu, 25 Sep 2014 15:55:40 -0700

...
> per-flow processing capacity).

Also on legacy servers it opens only one connection.

> The case under discussion, which is the worst case scenario is: new
> client talking to new server with an evilbox in the middle dropping the
> OOB 100% of the time.

That's one reason why we need to experiment with OOB packets; I don't
know what will happen, except in the case when OOB comes too early
before the SYN through a NAT.

However, dual-SYN has the problem of potentially going to different
servers or through different NATs and of opening multiple connections on
legacy servers (thus impacting service rate).

I wouldn't be surprised if the best path forward were to document each
as a separate experiment and try them both.

Joe

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 26 Sep 2014 01:27:42 +0200

Bob,

On Thu, 25 Sep 2014, Bob Briscoe wrote:

>> different port pairs - is this due to the explosion with the number of 
> Rationale for latter: too many middleboxes out there that would reject a SYN 
> to/from the same ports, in their misguided attempts to be 'helpful' by 
> removing SYNs that appear to be duplicate or same 4-tuple but different 
> initial sequence no. Not only many firewalls, but all split connections would 
> not forward the second SYN.

oh, *different* ISNs. Yes. I remember having to file a bug in at least 
one of the middleboxes that did not like such a scenario, it got fixed, 
but probably there're more. An experiment would be definitely interesting.

>> target host has A&AAAA you only have to deal with the failures on the path, 
>> while if the target host does not have AAAA record, it's worthless to try 
>> IPv6.
>> 
>> OTOH, when attempt to negotiate the extended option space, you do not have 
>> any hints about the remote end - besides maybe past memory.
>> 
> The dual SYN approaches could use a delay between the SYNs like 6655 does 
> (not nec exactly the same numbers). However, in the short flow world, I would 
> argue that 250ms is completely unacceptable unless you're fairly certain the 
> first one will work, which is more-or-less the v4/v6 scenario that happy 
> eyeballs addressed. Reasoning: multiply 250ms by a few connections with 
> serial dependencies in a sequence and you have delays of seconds.
>
> I've made a note to change the description of 6555 in the next rev - clearly 
> the word parallel is not strictly correct anyway - one serial (e.g. network) 
> interface can't actually send two SYNs in parallel, the closest they can 
> leave is back-to-back.

True... you might start with some conservative values, but 
then as you get RTT estimate for the connection, an implementation might 
opt to save that RTT estimate in a cache 
(http://www.tedunangst.com/flak/post/2Q-buffer-cache-algorithm seems maybe 
appropriate for this type of use?) And then we're somewhere in 6555 
territory anyway. So much for simplicity! :-)

> Damon Wischik once proposed to improve performance by duplicating the most 
> sensitive packets at flow start:
>        "three copies of the first packet of every message, two copies of the 
> second and one copy of subsequent packets"
> He calculated these 6 extra full-sized packets per flow would (in 2005) 
> increase total traffic by 2.7% (because elephants dominate total traffic).
> Duplicating just the legacy part of the SYN (60B max) would have a much 
> smaller total effect (I could calculate it if you want - probably sub-0.1%). 
> Certainly too small for people to care about the increment on bills or even 
> packet load.

Thanks for the link! I will read it in detail tomorrow, for now just 
glanced, and there does not seem to be any discussion of what happens if 
the link is constrained ?

Some time ago I had to work a nontrivial amount of time over a 3G 
throttled down to 64Kbps, and a number of tricks were required to change 
the experience from "unusable" to "slow but eventually loaded": switch 
from Chrome to Firefox (primarily because of my lack of knowledge of deep 
tuning for the former, maybe there was a better way!), and reducing the 
number of simultaneous connections down to 4, from 32.

Maybe it's small enough and I am being paranoid, but the 64kbps 
environments are still real (unfortunately).

>> So, depending on how much of a drop rate we'd expect for a "new" SYN across 
>> the variety of network scenarios, the simpler approach of just using 300ms 
>> delay could be reasonable due to its simplicity - an experiment could show.
>> 
>> smaller - so maybe there is an opportunity to simplify things - if it "just 
>> works" in IPv6 using a single fancy-looking SYN in a sufficient %% of 
>> cases, that could be a great simplification *and* will give additional use 
>> case for IPv6 assuming there is enough demand for more space in the 
>> options.
>
> The delay between the pair should be implementation dependent anyway. No 
> point arguing this to death in standards... but I will at least state the 
> opposing view...
>
> The world is changing. Latency is everything and soon it will be even more 
> than everything. If one values latency, the delay between the two will be as 
> small as possible.

Agreed.

>
> My simple view of the world is that, dual SYN seems the /only/ way to extend 
> TCP option space. I'm not proud of it. It's not nice. But it's necessary. If 
> dual SYNs require more memory for connection state, it's OK as long as the 
> need for more memory appears slowly enough to be able to deploy it. If this 
> is the price of extra capabilities, and those capabilities are valued, then 
> we invest in memory to provide the capabilities.
>
> Whether the number of connections that need >40B TCP options will suddenly 
> jump, or gradually grow, I can't predict tho.
>

Again, makes sense.

>> But if source ports are different, then a typical middlebox is either happy 
>> about the "data in the SYN" or not ?
>> 
>> difficult for the current architectures to "grab" them and mess around with 
>> them ? Sure, it does. Especially higher-performing boxes might look at the 
>> start of the packet.
>> 
>> BUs enough to create a box that fiddles with these options, that box will 
>> get created. Not an architecturally beautiful reality, but it is there.
>
> Two clarifications:
>
> 1) By putting options after the payload, I'm including the (currently common) 
> case of a SYN with no payload. Then the extra options will start immediately 
> at the Data Offset.
>
> Muddleboxes tend to forward the payload unaltered. Some muddleboxes look at 
> HTTP headers and try to be clever. By putting extra options after the Data 
> Offset, they will pass the former. By putting them at the end of the payload 
> they will mostly pass the latter.
>
> Meddleboxes might well evolve to find these new extra options at the end of 
> the packet. But in the meantime, we've got over the transition of deploying a 
> new protocol. Then the meddleboxes are on the back foot, and they have to 
> justify removing these packets that are useful to someone. If they can - e.g. 
> for security reasons - then fine. But they are not just removing packets 
> because they are using a new rev of the protocol.
>
> As I said, the most important thing is we will have separated TCP options 
> into two sets:
> - Those that middleboxes (future) should think much harder about before 
> messing with them
>
> Then, this (maybe) enables the next step - for endpoints to encrypt those 
> options they don't want middleboxes to interact with at all. (See my response 
> to Alex Zimmerman's post in this thread.)
>

Thanks for the detailed explanation! This might be useful to just 
copypaste into the draft in the next rev, for the rationale.

Now, a trick question. We all know IPv4 is entrenched with middleboxes 
necessarily meddling with L7 because they have to do NAT in the process.
What about IPv6 ? I have a totally unfounded hope that there the situation 
might be better, at least for now ?

>> kernel's accelerated NIC path showed that the invalid CRC would be dropped 
>> very early, before the software can see it - so I would argue against that 
>> approach today from implementation standpoint as well.
>> 
>> Hopefully arguing with my past self does buy me an "unbiased" ribbon, or at 
>> least a half of it :-)
>
> Certainly, this seems to give you the role of unbiased /and/ educated by 
> experience.
>
> I'm also no longer like an idea that my past self proposed (SYN-EOS DS), for 
> the same reasons Joe didn't like it - the two SYNs will often not end up in 
> the same place to be re-united. But I still have a candidate I do prefer 
> (syn-op-sis). That rules me out for the role of innocent bystander.

Probably the best way to decide is to implement both, get them in parallel 
into the olde wilde internets and see what sticks better!

--a

>>>> it's
>>>> on
>>>> options

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 26 Sep 2014 10:56:42 +0100

Andrew,

At 00:27 26/09/2014, Andrew Yourtchenko wrote:
>>>some comments below, maybe with some obvious questions as I did 
>>>not follow the latest discussions very closely.
>>SYN-EOS DS (within tcp-syn-ext-opt) and syn-op-sis use same dst, 
>>but different src.
>>
>oh, *different* ISNs. Yes. I remember having to file a bug in at 
>least one of the middleboxes that did not like such a scenario, it 
>got fixed, but probably there're more. An experiment would be 
>definitely interesting.

To be absolutely clear:
a) SYN-EOS OOB client sends a SYN and an OOB segment to complement 
it, using same src & dst ports. The OOB segment has SYN=0, ACK=0 and 
the seqno is the same as the ISN of the SYN.

b) SYN-EOS DS client sends 2 complementary SYNs, using same dst port, 
but different src ports and different ISNs.

c) syn-op-sis client sends 2 alternative SYNs, using same dst port, 
but different src ports and different ISNs.

a & b use the word 'complement', which means an upgraded server needs 
to have received both segments to start an upgraded connection.

c uses the word 'alternative', which means a server can start a 
connection when it receives either, and the upgraded handshake only 
completes if it's an upgraded server.

>>>>in the future, only falling back to the legacy SYN in series 
>>>>rather than parallel if you hit a legacy listener.
>>>option space and IPv4->IPv6 transition is the absence of the DNS 
>>>to signal the capabilities of the endpoint before the connection 
>>>even starts - if the target host has A&AAAA you only have to deal 
>>>with the failures on the path, while if the target host does not 
>>>have AAAA record, it's worthless to try IPv6.
>>>providing a headstart that shrinks and inverses on failures - so 
>>>the reference to RFC6555 being "parallel" is not correct, see section 5.5:
>>>
>>The dual SYN approaches could use a delay between the SYNs like 
>>6655 does (not nec exactly the same numbers). However, in the short 
>>flow world, I would argue that 250ms is completely unacceptable 
>>unless you're fairly certain the first one will work, which is 
>>more-or-less the v4/v6 scenario that happy eyeballs addressed. 
>>Reasoning: multiply 250ms by a few connections with serial 
>>dependencies in a sequence and you have delays of seconds.
>>
>>I've made a note to change the description of 6555 in the next rev 
>>- clearly the word parallel is not strictly correct anyway - one 
>>serial (e.g. network) interface can't actually send two SYNs in 
>>parallel, the closest they can leave is back-to-back.
>
>True... you might start with some conservative values, but then as 
>you get RTT estimate for the connection, an implementation might opt 
>to save that RTT estimate in a cache 
>(http://www.tedunangst.com/flak/post/2Q-buffer-cache-algorithm seems 
>maybe appropriate for this type of use?) And then we're somewhere in 
>6555 territory anyway. So much for simplicity! :-)

Cache algos don't have to be standardised. The standard can be 
simple, and implementers can tradeoff simplicity for latency.

>>Damon Wischik once proposed to improve performance by duplicating 
>>the most sensitive packets at flow start:
>>probably sub-0.1%). Certainly too small for people to care about 
>>the increment on bills or even packet load.
>
>glanced, and there does not seem to be any discussion of what 
>happens if the link is constrained ?
>
>throttled down to 64Kbps, and a number of tricks were required to 
>change the experience from "unusable" to "slow but eventually 
>loaded": switch from Chrome to Firefox (primarily because of my lack 
>of knowledge of deep tuning for the former, maybe there was a better 
>way!), and reducing the number of simultaneous connections down to 4, from 32.
>

The number of simulataneous connections is certainly critical for a 
constrained environment. But the dual SYN approaches don't double the 
number of /connections/, they only double the number of SYNs (in the 
worst case when all connections need large TCP options at the start).

I haven't done the proper calculation, but a rough estimate based on 
Wischik would be 0.05% more bytes.

>>>In today's conditions, a simple 300ms headstart for IPv6 if AAAA 
>>>is advertised might be sufficient, as the subpar IPv6 connectivity
>>>So, depending on how much of a drop rate we'd expect for a "new" 
>>>SYN across the variety of network scenarios, the simpler approach 
>>>of just using 300ms delay could be reasonable due to its 
>>>simplicity - an experiment could show.
>>>things - if it "just works" in IPv6 using a single fancy-looking 
>>>SYN in a sufficient %% of cases, that could be a great 
>>>simplification *and* will give additional use case for IPv6 
>>>assuming there is enough demand for more space in the options.
>>
>>The delay between the pair should be implementation dependent 
>>anyway. No point arguing this to death in standards... but I will 
>>at least state the opposing view...
>>
>>The world is changing. Latency is everything and soon it will be 
>>even more than everything. If one values latency, the delay between 
>>the two will be as small as possible.
>
>Agreed.
>
>>
>>My simple view of the world is that, dual SYN seems the /only/ way 
>>to extend TCP option space. I'm not proud of it. It's not nice. But 
>>it's necessary. If dual SYNs require more memory for connection 
>>state, it's OK as long as the need for more memory appears slowly 
>>enough to be able to deploy it. If this is the price of extra 
>>capabilities, and those capabilities are valued, then we invest in 
>>memory to provide the capabilities.
>>

Especially to someone who sells memory ;)

>>>Or, are you saying that placing them at the end of the packet 
>>>makes it difficult for the current architectures to "grab" them 
>>>and mess around with them ? Sure, it does. Especially 
>>>higher-performing boxes might look at the start of the packet.
>>>options, that box will get created. Not an architecturally 
>>>beautiful reality, but it is there.
>>
>>Two clarifications:
>>
>>1) By putting options after the payload, I'm including the 
>>(currently common) case of a SYN with no payload. Then the extra 
>>options will start immediately at the Data Offset.
>>
>>- (muddleboxes) those that happen to mess with headers, because 
>>their developers never stopped to think that TCP might change.
>>
>>after the Data Offset, they will pass the former. By putting them 
>>at the end of the payload they will mostly pass the latter.
>>
>>Meddleboxes might well evolve to find these new extra options at 
>>the end of the packet. But in the meantime, we've got over the 
>>transition of deploying a new protocol. Then the meddleboxes are on 
>>the back foot, and they have to justify removing these packets that 
>>are useful to someone. If they can - e.g. for security reasons - 
>>then fine. But they are not just removing packets because they are 
>>using a new rev of the protocol.
>>

I've added a 'Design Rationale' section for the next rev.

>Now, a trick question. We all know IPv4 is entrenched with 
>middleboxes necessarily meddling with L7 because they have to do NAT 
>in the process.
>What about IPv6 ? I have a totally unfounded hope that there the 
>situation might be better, at least for now ?

Sorry to disappoint, but AFAIK the only middlebox function that is 
IPv4-specific is NAT. Firewalls, Web-filters, connection splitters, 
PEPs, WAN accelerators, DPI, app-layer load balancers, etc all serve 
functions that are nothing to do with address lengths. And even NATs 
serve an obfuscation function, not just address-space expansion.

>>>the invalid CRC would be dropped very early, before the software 
>>>can see it - so I would argue against that approach today from 
>>Certainly, this seems to give you the role of unbiased /and/ 
>>educated by experience.
>>
>>often not end up in the same place to be re-united. But I still 
>>have a candidate I do prefer (syn-op-sis). That rules me out for 
>>the role of innocent bystander.
>
>Probably the best way to decide is to implement both, get them in 
>parallel into the olde wilde internets and see what sticks better!

Before going to the trouble of implementation, some middlebox 
traversal testing of the proposed packet formats might weed out the 
field. I have a suspicion that the OOB segment won't get very far in 
many cases - it's unlikely that all firewalls and intrusion detection 
systems will allow the unusual ACK=0, SYN=0 and the same seqno as the SYN.

Cheers

Bob

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 26 Sep 2014 13:15:15 +0200

Bob,

On Fri, 26 Sep 2014, Bob Briscoe wrote:

>>> Rationale for latter: too many middleboxes out there that would reject a 
>>> SYN to/from the same ports, in their misguided attempts to be 'helpful' by 
>>> initial sequence no. Not only many firewalls, but all split connections 
>>> would not forward the second SYN.
>> 
>> oh, *different* ISNs. Yes. I remember having to file a bug in at least one 
>> of the middleboxes that did not like such a scenario, it got fixed, but 
>> probably there're more. An experiment would be definitely interesting.
>
> To be absolutely clear:
> a) SYN-EOS OOB client sends a SYN and an OOB segment to complement it, using 
> same src & dst ports. The OOB segment has SYN=0, ACK=0 and the seqno is the 
> same as the ISN of the SYN.
>
> b) SYN-EOS DS client sends 2 complementary SYNs, using same dst port, but 
> different src ports and different ISNs.
>
> c) syn-op-sis client sends 2 alternative SYNs, using same dst port, but 
> different src ports and different ISNs.
>
> a & b use the word 'complement', which means an upgraded server needs to have 
> received both segments to start an upgraded connection.
>
> c uses the word 'alternative', which means a server can start a connection 
> when it receives either, and the upgraded handshake only completes if it's an 
> upgraded server.

This is a very useful and succing comparison, thanks!

>>> (not nec exactly the same numbers). However, in the short flow world, I 
>>> would argue that 250ms is completely unacceptable unless you're fairly 
>>> certain the first one will work, which is more-or-less the v4/v6 scenario 
>>> that happy eyeballs addressed. Reasoning: multiply 250ms by a few 
>>> connections with serial dependencies in a sequence and you have delays of 
>>> seconds.
>>> 
>>> (e.g. network) interface can't actually send two SYNs in parallel, the 
>>> closest they can leave is back-to-back.
>> 
>> True... you might start with some conservative values, but then as you get 
>> RTT estimate for the connection, an implementation might opt to save that 
>> RTT estimate in a cache 
> Cache algos don't have to be standardised. The standard can be simple, and 
> implementers can tradeoff simplicity for latency.
>

True.

>>>        "three copies of the first packet of every message, two copies of 
>>> the second and one copy of subsequent packets"
>>> smaller total effect (I could calculate it if you want - probably 
>>> sub-0.1%). Certainly too small for people to care about the increment on 
>>> bills or even packet load.
>> 
>> Some time ago I had to work a nontrivial amount of time over a 3G throttled 
>> down to 64Kbps, and a number of tricks were required to change the 
>> experience from "unusable" to "slow but eventually loaded": switch from 
>> Chrome to Firefox (primarily because of my lack of knowledge of deep tuning 
>> for the former, maybe there was a better way!), and reducing the number of 
>> simultaneous connections down to 4, from 32.
>> 
> constrained environment. But the dual SYN approaches don't double the number 
> of /connections/, they only double the number of SYNs (in the worst case when 
> all connections need large TCP options at the start).
>
> I haven't done the proper calculation, but a rough estimate based on Wischik 
> would be 0.05% more bytes.

The effect I had in mind was twofold - first, the number of connections, 
of course, but then also what are the loss patterns for the train of 
proactive retransmissions.

Oh, let me argue against myself a bit again and tell me if this line of 
thinking has legs: IW10 should have a somewhat similar effect compared to 
sending multiple SYNs -> so we could use the results from 
https://www.ietf.org/proceedings/80/slides/iccrg-1.pdf to ballpark the 
effect - which would be the increase in probability of loss by 0.5%, which 
does not seem bad.

>>> than everything. If one values latency, the delay between the two will be 
>>> as small as possible.
>> 
>> Agreed.
>> 
>>> 
>>> necessary. If dual SYNs require more memory for connection state, it's OK 
>>> as long as the need for more memory appears slowly enough to be able to 
>>> deploy it. If this is the price of extra capabilities, and those 
>>> capabilities are valued, then we invest in memory to provide the 
>>> capabilities.
>>> 

*that* memory is used for RIB, hello deaggregation! :-)

>>> common) case of a SYN with no payload. Then the extra options will start 
>>> immediately at the Data Offset.
>>> 
>>> Muddleboxes tend to forward the payload unaltered. Some muddleboxes look 
>>> at HTTP headers and try to be clever. By putting extra options after the 
>>> Data Offset, they will pass the former. By putting them at the end of the 
>>> payload they will mostly pass the latter.
>>> 
>>> Meddleboxes might well evolve to find these new extra options at the end 
>>> of the packet. But in the meantime, we've got over the transition of 
>>> deploying a new protocol. Then the meddleboxes are on the back foot, and 
>>> they have to justify removing these packets that are useful to someone. If 
>>> they can - e.g. for security reasons - then fine. But they are not just 
>>> removing packets because they are using a new rev of the protocol.
>>> 
>>> options they don't want middleboxes to interact with at all. (See my 
>>> response to Alex Zimmerman's post in this thread.)
>> 
>> Thanks for the detailed explanation! This might be useful to just copypaste 
>> into the draft in the next rev, for the rationale.
>

Thanks!

> IPv4-specific is NAT. Firewalls, Web-filters, connection splitters, PEPs, WAN 
> accelerators, DPI, app-layer load balancers, etc all serve functions that are 
> nothing to do with address lengths. And even NATs serve an obfuscation 
> function, not just address-space expansion.
>

My speculation is based on the fact that IPv4 part of the middleboxes 
have a decades of sophistication in them, but IPv6 support is relatively 
young. So the "corner cases" that the approaches we are discussing 
essentially are, may likely get missed. But again I will argue with myself 
and say (1) the L4+ codepath could be common, (2) it could be argued the 
reverse way - that these corner cases might have bugs that make them worse 
than in IPv4.

>>>> Even though I am nominally "on the list of alternatives" due to LOIC, 
>>>> I've disliked the idea shortly after, after trying to implement it - the 
>>>> linux kernel's accelerated NIC path showed that the invalid CRC would be 
>>> I'm also no longer like an idea that my past self proposed (SYN-EOS DS), 
>>> for the same reasons Joe didn't like it - the two SYNs will often not end 
>>> up in the same place to be re-united. But I still have a candidate I do 
>>> prefer (syn-op-sis). That rules me out for the role of innocent bystander.
>> 
> Before going to the trouble of implementation, some middlebox traversal 
> testing of the proposed packet formats might weed out the field. I have a 
> suspicion that the OOB segment won't get very far in many cases - it's 
> unlikely that all firewalls and intrusion detection systems will allow the 
> unusual ACK=0, SYN=0 and the same seqno as the SYN.

Yes, this is a prerequisite.

--a

>>>>>> are

---

From: John Leslie <[email-masked]>
Date: Fri, 26 Sep 2014 10:50:37 -0400

Alas, I haven't found time to completely read any of these drafts;
but I think an overview would be valuable...

> the seqno is the same as the ISN of the SYN.

   This is draft-touch-tcpm-tcp-syn-ext-opt. Two packets are sent: one
is a SYN; the other isn't.

> but different src ports and different ISNs.

   This is also draft-touch-tcpm-syn-ext-opt. Two packets are sent:
both are SYNs. Joe asks us to work out which approach is better.

> but different src ports and different ISNs.

   This is draft-briscoe-tcpm-syn-op-sis. One packet is sent, but Bob
expects a totally separate packet to be sent proposing a separate
connection, and only one of these connections will completely open.

> to have received both segments to start an upgraded connection.

   I agree with Bob here.

> completes if it's an upgraded server.

   Likewise...

   And I note that there's no _need_ to specify anything about the
overlap between the two proposed connections.

   I like this a lot better, because I never trust that two packets
will always reach the same endpoint. Meddleboxes and muddleboxes are
too common and too varied in their world-views.

   I'd like to remind folks of two other drafts:

- draft-ietf-tcpm-fastopen "allows data to be carried in the SYN" packet.
  Something like this is inevitable; and _will_ put application-layer
  data in the inital SYN (to reduce experienced latency).

- draft-nishida-tcpm-apaws "can provide protection against old duplicates"
  without using timestamps. (Timestamps will eat up ten bytes in _every_
  packet.) This should reduce the demand for option space in the SYN.

====

   Hopefully I don't need to draw attention to draft-touch-tcpm-edo.
This seems to be moving towards consensus on how to increase the option
space on every packet _after_ the initial SYN.

   Thus, in most cases, for today's traffic there is little need to
worry about extending the option space in the initial SYN. There seems
to be plenty of time to experiment...

> systems will allow the unusual ACK=0, SYN=0 and the same seqno as the SYN.

   I agree with Bob that we don't need formal Experimental status to
test how often an OOB packet will fail to pass a middlebox.

   For any Experimental-status document, I recommend:

- that the various on-the-wire formats be kept as similar as possible;

- that any acknowledgment of extra option space in an initial SYN
  include a checksum or hash of the extra options to ensure that both
  ends have the same view of what options have been signaled;

- that any timing questions be specifically explored as part of the
  experiment. (This includes _whether_ there should be any overlap
  between the experimental SYN and the legacy SYN.)

- that an explicit allocation of one (or more, possibly) dedicated
  option number be requested from IANA.

====

  IMHO, Bob's SYN-U cases are best condensed into a single format
where the SYN sender explicitly notes the position in the payload to be
occupied by additional TCP options, and guarantees that no application-
layer data will follow these. The receiving endpoint would search for
the extra options whether or not it found the SYN-U option in the
initial SYN, and reply including a checksum/hash of the options it
found. (The receiving endpoint MUST find some confirmation that
these options were intended, or _it_ bears the responsibility to
reverse any action on the extended options and pass the payload where
it otherwise would have gone.)

   I'm not sure this entire middlebox-transit idea is worth the trouble;
but it probably deserves to be an optional part of the experiment.

   The SYN-U sender MUST deal with the danger of its additional options
being passed to the application as data. For legacy receivers it can do
this by aborting before the end of the three-way handshake. But there
must be coordination with tcpm-fastopen to protect against such data
being passed to the application before the SYN-U sender can abort.
(This is not a trivial consideration!)

   I believe that much deserves to be settled _before_ tcpm-fastopen
is published as an RFC. (NB that document currently has been approved
for publication, but is waiting on a Point-Raised writeup.)

   The fastopen mechanism calls for the SYN sender to request a
Fast Open Cookie, which the SYN receiver will generate and return in
the SYN-ACK. In a _future_ connection, the SYN sender can send that
cookie, and all payload data will go to the application _before_ the
three-way handshake completes. Thus there is no way to use both
fastopen and SYN-U in the same connection. This is unfortunate!!!

   (Note that fastopen will be Experimental; and deficiencies can be
fixed before it reaches Proposed Standard: it's just best to consider
overlaps like this before the RFC is published.)

====

   The question of separating payload data to go to the application
from extended TCP options clearly requires action by the SYN receiver.
This is fairly simple if the SYN sender can ensure that _none_ of the
payload data should go to the application (and thus not use fastopen
at all): of course the connection must still be aborted if the SYN
receiver doesn't acknowledge separating out the extended options.

   IMHO, the SYN sender should never use SYN-U unless extra option
space in the SYN is needed; and it's probably better to never use
SYN-U for connections where latency of setup is the over-riding factor.
But YMMV. I wonder whether such overlap is worth the trouble to try
to design into the protocol.

---

From: Joe Touch <[email-masked]>
Date: Fri, 26 Sep 2014 09:41:11 -0700

On 9/26/2014 7:50 AM, John Leslie wrote:
...
> to be plenty of time to experiment...

I don't know if the need is as small, but I agree that experimentation
is needed to proceed.

...
> test how often an OOB packet will fail to pass a middlebox.

Agreed; that's on our to-do list.

> - that the various on-the-wire formats be kept as similar as possible;

I disagree; there's no reason for that.

>   ends have the same view of what options have been signaled;

Disagree. Options are ACKd by confirming the options - not by merely
sending an ACK.

Yes.

>   option number be requested from IANA.

These can and IMO should use the experimental option IDs until we're
ready to move forward.

> receiver doesn't acknowledge separating out the extended options.

I don't think that's tenable. SYNs have always been able to include
data. Fastopen just helps that data get to the user faster.

Joe

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 26 Sep 2014 18:16:11 +0100

John,

>   data in the inital SYN (to reduce experienced latency).

This is referenced from both drafts.

You have to be careful using fast open as well as putting TCP options 
in the payload. I've written the following into 
draft-briscoe-syn-op-sis, and a similar rule will be needed in 
draft-touch-tcpm-tcp-syn-ext-opt:

"  If an upgraded TCP client includes the TCP Fast Open option
    [I-D.ietf-tcpm-fastopen] in the SYN, it MUST be placed with the extra
    TCP options after the end of the payload.  An upgraded TCP client
    MUST NOT place any TCP option in the TCP header of a SYN that might
    cause a TCP server to pass user-data directly to the application
    before the 3-way handshake completes.
"
See <http://tools.ietf.org/html/draft-briscoe-tcpm-syn-op-sis-02#section-2.4.1>

>   packet.) This should reduce the demand for option space in the SYN.

Most certainly agree. See quoted text under the next point...

>to be plenty of time to experiment...

Not sure I'm so relaxed. Quoting
<http://tools.ietf.org/html/draft-briscoe-tcpm-syn-op-sis-02#section-1.1>

"1.1. Motivation for Adoption (to be removed before publication)

    It is recognised that there could be potential for compressing
    together multiple options in order to mitigate the option space
    problem.  However, it seems inevitable that ultimately more option
    space will be needed, particularly given that many of the TCP options
    introduced recently consume large numbers of bits in order to provide
    sufficient information entropy, which is not amenable to compression.

    Extension of TCP option space on a SYN requires support from both
    ends.  This means it will take many years before the facility is
    functional for most pairs of end-points.  Therefore, given the
    problem is already becoming pressing, a solution needs to start being
    deployed now.
"

>   ends have the same view of what options have been signaled;

Not convinced on this one. Each TCP option has its own way for the 
'server' to confirm whether it supports an option requested by the 
client. And support for the option to extend the option space is 
confirmed by the server. So it seems redundant for the server to 
feedback yet another confirmation via a checksum of all the extra 
options requested.

>   between the experimental SYN and the legacy SYN.)

Good point.

>layer data will follow these.

Good point about the guarantee.

>The receiving endpoint would search for
>initial SYN,

This won't work. This would require every server to hold a list of 
valid TCP options and their valid length fields. This list is large, 
and the list of future options yet to be defined is even larger (and 
obviously unknown).

If the server has to check every sequence of bytes starting at every 
byte within the payload for a match with all possible TCP options, 
when searching through any arbitrary payload, it will continually 
return loads of false matches (and it will consume excessive cycles).

This also doesn't provide any path to add new options.

>and reply including a checksum/hash of the options it

No need for all this - see above and see drafts.

>but it probably deserves to be an optional part of the experiment.

The reason I came up with syn-op-sis is to be able to be near-certain 
that it will get through current middleboxes. I think the group will 
find such near-certainty to be liberating, compared to the minefield 
we currently have to tip-toe through.

The only case I can think of where there could be a blockage is where 
a split TCP connection holds back any payload on the SYN until the 
3WHS completes.

>(This is not a trivial consideration!)

Altho certainly not a trivial consideration, it is trivial to be safe 
against this - see the rule already in the syn-op-sis draft, quoted 
above, which is careful to be specific about TFO, as well as 
generalising to anything in the future that might be like TFO (e.g. 
tcpcrypt uses the TCP payload for control messages).

>fastopen and SYN-U in the same connection. This is unfortunate!!!

As above, syn-op-sis contains text solving this already. And the same 
approach can be used by the other drafts.

>to design into the protocol.

In the syn-op-sis spec there is a choice between two variants of the 
protocol (I recommend the choice can be made by implementers, not the 
IETF). The choice I've recommended removes any risk of extra latency 
on the upgraded connection. I've included a discussion of the 
tradeoff between them in Appendix B.1:
<http://tools.ietf.org/html/draft-briscoe-tcpm-syn-op-sis-02#appendix-B.1>

BTW, I spend most of my time these days working on a project to 
remove latency from protocols.

Cheers

Bob

---

From: John Leslie <[email-masked]>
Date: Fri, 26 Sep 2014 14:29:15 -0400

> I disagree; there's no reason for that.

   "As possible" to be intepreted liberally...

> Disagree. Options are ACKd by confirming the options - not by merely

   No thanks! _This_ option deserves its own confirmation. We're putting
options in a funny place -- any number of things could go wrong with
that. :^(

   (But, of course, the individual options in the extended option space
should get their own confirmations.)

> These can and IMO should use the experimental option IDs until we're

   Absolutely. I only meant when we issue a formal Experimental RFC.

> I don't think that's tenable.

   What specifically isn't tenable?

> SYNs have always been able to include data.

   Indeed! That by itself complicates things.

> Fastopen just helps that data get to the user faster.

   Specifically, fastopen (when used) removes the opportunity to abort
before the application sees the payload data.

   With 20/20 hindsight, it's obvious that fastopen would have been
better if it included a byte-count of data to pass to the application
before the three-way-handshake completes.

---

From: Joe Touch <[email-masked]>
Date: Fri, 26 Sep 2014 11:39:29 -0700

>    "As possible" to be intepreted liberally...

That imposes a goal that will require cycles to resolve, one that serves
no purpose unless the two options converge naturally.

> options in a funny place -- any number of things could go wrong with

If the extension works, then you have the extension place in which to
send option confirmation. The extension itself clearly needs to be
confirmed too - like any other options. I don't see the reason for a hash.

...
>    Absolutely. I only meant when we issue a formal Experimental RFC.

It depends on the experiment. The ExID is intended for the potential to
use an option in the public that we might not end up keeping.

>    What specifically isn't tenable?

I don't think we should declare that SYNs don't have data.

>    Indeed! That by itself complicates things.

But we should allow it.

> before the three-way-handshake completes.

TCP is a bytestream protocol. Even with the PSH bit, there's no
guarantee of that boundary anyway.

Joe

---

From: John Leslie <[email-masked]>
Date: Fri, 26 Sep 2014 15:53:38 -0400

> This is referenced from both drafts.

   Yes. I felt a need to emphasize it...

>" [I-D.ietf-tcpm-fastopen] in the SYN, it MUST be placed with the extra
>" TCP options after the end of the payload.

   Indeed, that is a good way to ensure that the SYN receiver is aware
of the sender's intent to place something _after_ the data intended to
be passed to the application.

>" An upgraded TCP client MUST NOT place any TCP option in the TCP header
>" of a SYN that might cause a TCP server to pass user-data directly to
>" the application before the 3-way handshake completes.

   This is correct, if a bit hard for the reader to grok...

>" 1.1. Motivation for Adoption (to be removed before publication)

   (I think the point deserves to be made in the published RFC.)

>"   It is recognised that there could be potential for compressing
>"   together multiple options in order to mitigate the option space
>"   problem.  However, it seems inevitable that ultimately more option
>"   space will be needed, particularly given that many of the TCP options
>"   introduced recently consume large numbers of bits in order to provide
>"   sufficient information entropy, which is not amenable to compression.

   I agree on the ultimate need. But it will necessarily be postponed
until we finish Experimental status and move to Proposed Standard.

>"   Extension of TCP option space on a SYN requires support from both
>"   ends.  This means it will take many years before the facility is
>"   functional for most pairs of end-points.  Therefore, given the
>"   problem is already becoming pressing, a solution needs to start being
>"   deployed now.

   (No objection to that text.)

> client.

  And this one should also have its own way. I see so many things that
could get in the way that I _strongly_ recommend some checksum/hash.

> And support for the option to extend the option space is confirmed by
> the server.

   One bit of information (support-yes-or-no) just isn't sufficient,
IMHO.

> So it seems redundant for the server to feedback yet another
> confirmation via a checksum of all the extra options requested.

   Redundancy is your friend when complexity multiplies.

>> The receiving endpoint would search for the extra options whether or
>> not it found the SYN-U option in the initial SYN,

   (Did I write that?? I was thinking MAY...)

> and the list of future options yet to be defined is even larger (and 

   But that's exactly what you solve in you "non-deterministic" version.

>...
>> and reply including a checksum/hash of the options it found. (The
>> receiving endpoint MUST find some confirmation that these options
>> were intended, or _it_ bears the responsibility to reverse any
>> action on the extended options and pass the payload where it
>> otherwise would have gone.)
> 

   Your non-deterministic version does this with a magic-number. That
is an obvious weakness. I intended to draw attention to the issue --
not propose a particular solution.

   But obviously: if the SYN sender sends a SYN-U which the receiver
doesn't get because a meddlebox removed it, the sender can learn of
the problem when a returned checksum fails -- but the receiver won't
know of the problem until the sender aborts.

   The receiver might choose to delay sending fastopen data in every
case where no SYN-U was received -- or there are other possibilities.
Pick your poison...

> we currently have to tip-toe through.

   But it does add complexity. I won't pre-judge what balance of
complexity the WG may chooose.

> a split TCP connection holds back any payload on the SYN until the 

   ... which _will_ happen... :^(

> tcpcrypt uses the TCP payload for control messages).

   I expect fastopen to see widespread use -- and I expect it to be
programmed by different folks that program syn-op-sis. Our warning
label (IMHO) needs to be rather prominent...

> IETF).

   Hmm...

   This is a dangerous idea. Experimental results get hopelessly cloudy
when implementations are _fundamentally_ different...

> The choice I've recommended removes any risk of extra latency 

   :^)

   (Can you spell ECN?)

---

From: Joe Touch <[email-masked]>
Date: Fri, 26 Sep 2014 14:09:07 -0700

...
>> remove latency from protocols.

Congestion is only one of a half-dozen fundamental causes of latency, FWIW.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 29 Sep 2014 08:25:16 -0700

> be passed to the application.

I'd like to understand the logic behind this claim.

Regardless of where signalling information is placed:

	- legacy servers will (irrevocably) accept it as
	application data; the only way to undo it is to
	terminate the connection*

	- upgraded servers will know where to find it

Except to make parsing more complex, what is the perceived benefit of
using trailer-based signalling?

Joe

*- TCP-FO is particularly dangerous when coupled with dual-SYN (DS)
variants when the connection uses a cookie from a previous connection;
non-DS FO servers can't prevent passing invalid data to the application
by terminating the connection when the SYN-ACK is received by the client.

---

From: John Leslie <[email-masked]>
Date: Mon, 29 Sep 2014 12:58:39 -0400

> I'd like to understand the logic behind this claim.

   Good!

> using trailer-based signalling?

   Bob, I believe, thinks he can outwit meddleboxes. I'll let him make
that case.

   Myself, I see it a a way to coexist with things to come.

   Joe, I suspect, is thinking how to do this in hardware.

   Indeed, to a middlebox, harder-to-find translates to "expensive",
possibly prohibitively expensive. To an actual endpoint, the expense
is bearable -- if you need to act on both anyway, it's a toss-up...

   But myself, I think in terms of how to coexist with future things
like FastOpen. I believe we've got a better shot at ensuring that
nothing _follows_ our extended options than ensuring that nothing
precedes them.

>>> " TCP options after the end of the payload.

and recalling that extended options _can't_ be parsed until the inital
TCP option invoking extended options is parsed, we see that the receiving
endpoint _must_ know about the overlap of extended-options and fast-open.

   That brings us to the tussle of whether either _must_ play with the
data the other wants to add. To me, the simplest case is that fast-open
does its job first (hopefully someday to include a byte-count), while
extended-options comes along and puts its data after fast-open's data
(and anything else which may come along), trusting the receiver to
remove it from anything to be passed to the application.

   I am convinced we will see cases where both fast-open and extended-
options need to be used.

   While this tussle could go either way, I prefer to place the
complexity in extended-options.

> by terminating the connection when the SYN-ACK is received by the client.

   I had to read this twice to get Joe's point. :^(

   Dual-SYN solutions would be expected to place fast-open in both SYNs,
probably with the same cookie. In the legacy case, nothing can prevent
the payload being passed to the application long before the sender
becomes aware of the state of the other SYN. This probably deserves
mention in any dual-SYN proposal.

   (Obviously, if the legacy SYN is postponed long enough, this ceases
to be a problem; but by then you've lost most of the latency advantage.)

---

From: Joe Touch <[email-masked]>
Date: Mon, 29 Sep 2014 11:11:52 -0700

>    Joe, I suspect, is thinking how to do this in hardware.

Not specifically. IMO, might create another opportunity to get things
wrong because it's now how we currently process options.

> remove it from anything to be passed to the application.

I'm not sure I understand this. User data can't be placed in (or removed
from) the segment until after all the options are handled anyway.

> complexity in extended-options.

It's clearly only the option processing that should ever care where the
options are...

> becomes aware of the state of the other SYN.

Right - the legacy case of FO + any dual-SYN approach has this problem.
It's one that the OOB method is not susceptible to, though.

> to be a problem; but by then you've lost most of the latency advantage.)

That would happen only on a DS-capable server. I'm also thinking of the
legacy server case - which might not only open two TCP connections but
(with TFO) would send data to the user too early in the upgraded-SYN
case - and it's corrupt data because the server is interpreting the
extended option as user data.

Joe

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 30 Sep 2014 15:45:22 +0100

Joe & John,

> >> terminate the connection*

As you know, syn-op-sis is designed so the client detects a legacy 
SYN/ACK and terminates that connection.

The reason for requiring a syn-op-sis client to only ever locate TFO 
beyond the Data Offset in a SYN with extra options is precisely so 
that it will never use TFO to ask a legacy server to pass the payload 
to the app at the same time as it is placing (other) options in the payload.

And the wording in the draft generalises this to any future option 
that might cause a server to pass payload to the app during the 3WHS, 
not just TFO.

> >> using trailer-based signalling?

The fact that you (Joe) are asking the same question that I've 
already answered (on the list and in the draft) implies you don't 
accept the answer. But unless I can understand what you don't accept 
about the answer, I can only answer it the same way as I already 
have. However, I will try to spell it out more fully...

a) Ultimately, locating options as a trailer is not intrinsic to 
syn-op-sis. At this pre-implementation stage, a quick edit to the 
draft could locate them straight after the Data Offset.

b) However, as the initial proposition, I have located the options as 
a trailer, because middleboxes exist that look at the dest port, and 
they assume that the app-layer protocol is that associated with the 
port (eg. http if the port is 80). [Acting on this assumption 
violates layering, but I'll come to that.]

Then they parse the first X bytes of the payload, where the choice of 
X depends on the expect protocol and how much they can do at line 
rate. If the payload does not start with the expected app-layer 
commands, what they do next depends on the box function:
* If it's an intrusion detection system, it will probably drop the packet.
* If it's a app-layer filter, it will probably drop the packet (e.g. 
a Web filter is looking for the requested Web page to determine 
whether to block it, and it becomes suspicious when it's not even http).
* If it's a performance accelerator, it will probably forward the 
packet without being able to use any of its app-specific acceleration 
capabilities, because it doesn't recognise the app-layer protocol.

I would be very happy if experiments showed such middleboxes are a 
corner-case, but I suspect they are not.

I know these boxes violate layering and we do not have to dance to 
their tune. However, my goal is simple:
  There is no point in providing extra option space in a SYN if it 
does not get delivered.

There is a tradeoff between the probability of getting delivered and 
the complexity introduced to the protocol.

My simple view is that I would rather do (limited) extra processing 
on all packets so that a greater proportion get delivered. What use 
is it if an implementation doesn't work but it's efficient?

> >    Myself, I see it a a way to coexist with things to come.

* Locating extra options beyond the Data Offset (whether before or 
after the payload) provides a distinction between middlebox and 
destination options for the future.
* Locating extra options after the payload is to avoid certain 
classes of existing muddleboxes, but with no intention to outwit 
meddleboxes (spelled out below).

>wrong because it's now how we currently process options.

Noted.

> > is bearable -- if you need to act on both anyway, it's a toss-up...

That's why I'm happy to evolve to a position where the current TCP 
options are intended for middleboxes, and it's at least acceptable if 
extra 'destination' options are 'harder-to-find'.

> > endpoint _must_ know about the overlap of extended-options and fast-open.

Correct, John (in the case of an upgraded receiving endpoint).

> > remove it from anything to be passed to the application.

I think/hope you have switched to talking about the processing order 
on the sender here, because of the word 'puts'.

I don't understand tho what you (John) mean by "extended-options 
comes along and puts its data". What do you mean by 'data' in this 
case? User-data related to an extended option? Or the option fields themselves?

>from) the segment until after all the options are handled anyway.

I'm not sure why the sender has any constraint on the order in which 
it serialises a segment. Surely it can construct the segment in 
memory in any order, then serialise it?

John and Joe, I'm actually unclear what the context is for this part 
of the conversation. As a general point, it's not healthy to have 
email conversations where all the participants start with the words, 
"I haven't properly read the draft but...", then proceed to say there 
are probably problems, and then try to solve those probable problems.

It would be more useful for participants to trust that the author(s) 
of a draft wrote it because they have a solution, and they have taken 
some care to write it up carefully. Then it would be useful to hear 
critique (or praise) if there are problems with the draft (or not).

>Right - the legacy case of FO + any dual-SYN approach has this problem.

Wrong. It is easy to solve for dual-SYN as explained in the draft 
(and above). As long as TFO is after the Data Offset, a legacy server 
won't pass the payload to the app during the 3WHS.

Correct.

>That would happen only on a DS-capable server.

It would not. As above.

>I'm also thinking of the

It would not. As above.

In summary, TFO is not a problem for all three dual-segment 
approaches on the table:
- SYN-EOS OOB
- SYN-EOS DS
- syn-op-sis.

Bob

---

From: Joe Touch <[email-masked]>
Date: Tue, 30 Sep 2014 08:15:03 -0700

> SYN/ACK and terminates that connection.

By the time the client receives the SYN/ACK, the legacy server would
have already forwarded FTO data to the user.

> beyond the Data Offset in a SYN with extra options is precisely so that
> it will never use TFO to ask a legacy server to pass the payload to the
> app at the same time as it is placing (other) options in the payload.

Ahh - I see.

> And the wording in the draft generalises this to any future option that
> might cause a server to pass payload to the app during the 3WHS, not
> just TFO.

The problem is that you can't necessarily make assertions about options
that already exist, and this might complicate TCP option processing --
you need to enforce it on both client and server, and legacy receivers
can't know that this is a problem.

> The fact that you (Joe) are asking the same question that I've already
> answered (on the list and in the draft) implies you don't accept the
> answer. But unless I can understand what you don't accept about the
> answer, I can only answer it the same way as I already have. However, I
> will try to spell it out more fully...
> 
> syn-op-sis. At this pre-implementation stage, a quick edit to the draft
> could locate them straight after the Data Offset.
> 
> b) However, as the initial proposition, I have located the options as a
> trailer, because middleboxes exist that look at the dest port, and they
> assume that the app-layer protocol is that associated with the port (eg.
> http if the port is 80). [Acting on this assumption violates layering,
> but I'll come to that.]
> 
> Then they parse the first X bytes of the payload, where the choice of X
> depends on the expect protocol and how much they can do at line rate.
...

I am skeptical that middleboxes parse SYN payloads, but even if they did
this might still not work. It depends on the size of the segment and the
size of the extension option space.

> * Locating extra options beyond the Data Offset (whether before or after
> the payload) provides a distinction between middlebox and destination
> options for the future.

That's necessary because the DO field is limited anyway.

> * Locating extra options after the payload is to avoid certain classes
> of existing muddleboxes, but with no intention to outwit meddleboxes
> (spelled out below).
...
> extra 'destination' options are 'harder-to-find'.

All TCP options are "destination" options.

However, the nature of dual-SYN requires that certain options always
come after the extension (as above) - but that also means it might
interfere with this partitioning by option type.

E.g., I would not be surprised at all to see TFO be of particular
interest to middleboxes - esp. to support load balancing while
supporting TFO.

> I'm not sure why the sender has any constraint on the order in which it
> serialises a segment. Surely it can construct the segment in memory in
> any order, then serialise it?

It can, but I don't understand the need to support more esoteric
middleboxes (including those I still consider meddlers) but not the
typical case where headers are built then data is added (or headers are
built in one place, data in the other, and the two combined).

Putting trailing options into that mix complicates lots of
implementations - and can be difficult to support in offload engines,
depending on how they handle buffers.

> John and Joe, I'm actually unclear what the context is for this part of
> the conversation. As a general point, it's not healthy to have email
> conversations where all the participants start with the words, "I
> haven't properly read the draft but...", then proceed to say there are
> probably problems, and then try to solve those probable problems.

I didn't start my email with that statement.
...

> It would be more useful for participants to trust that the author(s) of
> a draft wrote it because they have a solution, and they have taken some
> care to write it up carefully. Then it would be useful to hear critique
> (or praise) if there are problems with the draft (or not).

I disagree; it's useful to post directed questions and have direct
answers. Not everyone wants to try to infer intent from pages of text.

...
> Wrong. It is easy to solve for dual-SYN as explained in the draft (and
> above). As long as TFO is after the Data Offset, a legacy server won't
> pass the payload to the app during the 3WHS.

Agreed. As long as TFO is after the DO.

Joe

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 30 Sep 2014 16:54:43 +0100

John,

I've got a bit behind in this conversation - I'm on another part of 
my job at the mo. So apologies that my response to this old posting 
on this thread arrives out of order.

>    This is correct, if a bit hard for the reader to grok...

I wanted to state it both for the specific case of TFO and in general 
terms in case some other option is invented in future that passes 
data to the app before the 3WHS completes.

>    (I think the point deserves to be made in the published RFC.)

These paras are about what the IETF's wider agenda should be and 
where this draft fits in, which isn't appropriate in an RFC. And they 
are not timeless, which an RFC should also avoid.

>    One bit of information (support-yes-or-no) just isn't sufficient,

I believe you're trying to recover some determinism because of the 
non-determinism I've introduced. If so, I strongly believe that it's 
better to lessen the original non-determinism (e.g. by a larger magic 
number), rather than to add more message exchanges.

If both achieve the same higher level of determinism, I'd much rather 
use the one that avoids adding more branches to the protocol logic.

>    Redundancy is your friend when complexity multiplies.

I believe the opposite is true. If you provide two different ways to 
say the same thing, you create more complexity - to check they are 
both saying the same thing. And if they are not, you have to decide 
which one not to believe. This makes the spec and the code more complex.

>    But that's exactly what you solve in you "non-deterministic" version.

I think you've read something into it that isn't there.

The draft doesn't say an implementation has to "search for the extra 
options". Perhaps you didn't mean that? It specifies a deterministic 
location for the SYNOPSIS option that tells you where the other extra 
TCP options start.

The draft doesn't say or imply that an implementation has to know 
anything about any valid TCP options. Maybe you've wrongly jumped to 
that conclusion?

Was it the last bullet in the quoted list below that made you think this?:

"  an upgraded server MUST determine whether there is a
    SynOpSis TCP option at the end of the packet by checking all the
    following conditions:

    o  The Kind value is the SynOpSis Kind value;

    o  The length is 8;

    o  The next 4 octets match the magic number;

    o  The sum of the value of the EOO field, and all the length fields
       found by walking along the TCP options at the end of the payload
       exactly reaches the end of the packet.
"

Perhaps I need to spell out what that means...

First note that the implementation only needs to continue down this 
list of tests if all the previous tests have passed. So it will only 
have to do the last test if, at the end of the payload, there is 
arbitrary user-data that happens to contain exactly the same bytes as 
a SYNOPSIS TCP option (i.e. it has passed the first three tests).

The only assumption the fourth test makes about TCP options is that 
their length field is at the second octet, and that the number in 
that field specifies the length of the TCP option in octets. This 
should be true for all TCP options, past and future.

"Walking along the TCP options" means that the implementation:
* looks at the byte where the SYNOPSIS option says the first length 
field will be (one byte forward from where the extra options start),
* jumps forward by the value in this byte to the byte where the next 
length field would be if there was a TCP option there,
* and so on.

And if there really are TCP options there, stepping along the bytes 
where the length fields should be will land on the length field of 
the SYNOPSIS option at the end packet.

On the other hand, if it's just user-data that happens to have bytes 
that look like a SYNOPSIS option at the end, there will be some 
arbitrary byte in the location where the extended options offset 
(EOO) would be if it was a SYNOPSIS option. Even though that byte is 
not really an offset, there will be some number in that byte. If that 
number is taken to be an offset, it will point to somewhere else in 
the segment.

One byte forward from there will be another byte where the first 
length field seems to be, even though it's just an arbitrary byte in 
user-data. So jumping forward by the value in that byte will find 
another arbitrary byte and so on.

If this process eventually lands beyond the point where the length 
field of the SYNOPSIS option is, the implementation knows with 
certainty that there aren't really TCP options there; it's just been 
following arbitrary bytes in arbitrary user-data.

Altho this took a long time to say, these tests require minimal 
cycles. They just involve reading a byte at a location and using it 
as an offset to the next location, then checking whether that is 
still within the payload. It cannot continue for long, because it's 
always going in one direction (because an option length is defined as 
an unsigned int).

>know of the problem until the sender aborts.

In syn-op-sis the sender will learn of the problem without the 
checksum approach. If a middlebox removes the SYNOPSIS option, the 
server will respond to the SYN-U with a regular SYN/ACK, not a SYN/ACK-U.

This is why step 3 is needed in Table 1. And the required client 
behaviour is formalized in the five bullets at the end of section 2.1.

With the explicit variant of the dual-handshake is used (Section 6), 
because a middlebox might removing the SYNOPSIS option, the draft 
requires the client behaviour to be the same as for the implicit variant.

Search for the text "a middlebox might strip the SynOpSis TCP option" 
in the syn-op-sis draft.

>complexity the WG may chooose.

Fully agree.

> >> receiver doesn't acknowledge separating out the extended options.

I've already dealt with the points about TFO in the rest of your 
email in my other recent email on this thread.

> >> to design into the protocol.

You're allowed your humble opinion. However, if you analyse the 
SYNOPSIS protocol, you will conclude with absolute certainty that the 
upgraded SYNOPSIS connection has no latency cost. So if a SYN-U gets 
options through middleboxes, I predict it will be used whether or not 
the extra space is needed.

>when implementations are _fundamentally_ different...

I've put forward two pairs of variants. For the first pair of 
variants, I have asked the IETF to decide between them (so I put the 
one I least recommend in an Appendix). But for the other pair of 
variants, I realised that the choice of variants can be left to the 
implementer, so I put it in the body (Section 6).

I didn't come to these conclusions without a lot of thought (whereas 
up to draft-01 I hadn't fully thought this through)...

The only difference in the section 6 variant is the SYN-L. It became 
clear to me that this is not a fundamental difference when I realised 
that a middlebox could reduce the SYN-L to a SYN and it would turn 
the protocol into the other variant. So the rest of the protocol is identical.

The reason for allowing either at run-time is that the SYN-L variant 
is more likely to be preferred after most servers have been upgraded, 
but not at the start of deployment (explained in the Appendix linked below).

> > <http://tools.ietf.org/html/draft-briscoe-tcpm-syn-op-sis-02#appendix-B.1>

Regards

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 30 Sep 2014 17:36:58 +0100

Joe,

>can't know that this is a problem.

I can make this assertion, because it only applies to the behaviour 
of a client that has been upgraded for SYNOPSIS. This text requires 
such an update to also alter any pre-existing TFO code.

>size of the extension option space.

If a box does parse a SYN payload, it is much more likely to work if 
it works from the start, because usually the parsing stops once it 
has found what it wants.

If the parsing comes to the clean end of the app-layer commands, a 
really paranoid box (e.g. an intrusion detection system) might be 
programmed to be suspicious of odd inexplicable bytes afterwards. But 
I figure that's much less likely than tripping up on unrecognisable 
bytes before it's even got started.

>All TCP options are "destination" options.

I forgot to add an <irony> tag to the destination adjective.

>interest to middleboxes - esp. to support load balancing while

The critical point here is that there are two viewpoints on which 
options are for middleboxes:
- the sender's viewpoint
- the middlebox's viewpoint.

Enabling the sender to separating options into two sets allows it to 
express its viewpoint. A middlebox does not have to /express/ its 
viewpoint. It just /acts/ in whatever way it chooses.

Nonetheless, once the separation is in place, it might be possible 
for the sender to encyrypt the extra options. Then the power to act 
unilaterally swings the other way.

>I didn't start my email with that statement.

Sorry, I noticed John did and I thought I recalled that you had in 
the email before that, but, checking back, you actually said "I did 
not follow the latest discussions very closely"

>answers. Not everyone wants to try to infer intent from pages of text.

Certainly. But only after having read the draft under discussion.

>Agreed. As long as TFO is after the DO.

Great. We're making progress.

Cheers

Bob