Subject: TCP proxys in mobile networks

From: "Olle E. Johansson" <[email-masked]>
Date: Wed, 27 Jul 2016 14:41:36 +0200

Hi!

Spencer asked me to bring up this issue, that originated on the TRAM mailing list on the topic of STUN discovery of middle boxes.

In a few projects, we’ve seen strange things happening when using mobile networks. Wireshark on a server tells us a message has left
and is confirmed delivery by the TCP layer - but nothing appears using Wireshark on the other end.

Seeing a mail message on a SIP implementors mailing list that one vendor have started using UDP-style retransmits when using
TCP as a transport made me starting to worry that this was a more widespread problem.

If I understand it right, mobile carriers are implementing some sort of middlebox that act as a TCP proxy. I guess the idea is
that they want to disable retransmits on the radio, so they confirm reliable delivery prematurely. If something goes wrong
beyond the proxy, it’s too late to do anything about it.

I had a few discussions about this with fellows at IETF in Berlin and it seemed like a known problem. One developer
pointed me to a paper discussing this.

https://www.cs.montana.edu/mwittie/publications/Goel16Detecting.pdf

From the abstract:
"In current cellular networks, a myriad of middleboxes disregard the end-to-end principle to enable network operators to deploy services such as content caching, compression, and protocol optimization to improve end-to-end network performance.”

Good reading! (and a bit scary)

This mess caused me sadly to suggest that we need to discuss breaking the assumption that TCP delivery is always reliable
and implement retransmits even over TCP in the STUN protocol. STUN was designed to discover middleboxes
with a focus on NAT. This is just another middle box to discover.

The bigger picture is even more scary - what happens if our reliable transport suddenly no longer is reliable?

One developer from a well known mobile system vendor said “well, I guess that using TLS may help”…

Have you seen this behaviour in networks close to you? What are your experiences?

/O

---

From: Joe Touch <[email-masked]>
Date: Wed, 27 Jul 2016 08:18:10 -0700

Olle,

On 7/27/2016 5:41 AM, Olle E. Johansson wrote:
> ...
None of this is news. One of the "features" of middleboxes is
"transparent" TCP relaying. That device always destroys TCP reliable
delivery semantics.

This has been known since the mid 90s'.

The challenge with STUN has always been that many middleboxes *do not
want to be found*.

> One developer from a well known mobile system vendor said “well, I guess that using TLS may help”…

Ask them *how* they think TLS helps. TLS relies on TCP semantics.

Joe

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Wed, 27 Jul 2016 10:27:42 -0500

Hi, Joe,

On Wed, Jul 27, 2016 at 10:18 AM, Joe Touch <[email-masked]> wrote:

> > This mess caused me sadly to suggest that we need to discuss breaking
> the assumption that TCP delivery is always reliable

Right. IIRC, you and I were part of a number of conversations about this in
PILC, while working on https://www.ietf.org/rfc/rfc3135.txt.

My reason for asking Olle to bring this forward is that we're having a lot
of conversations (starting at the IAB with
https://www.iab.org/activities/workshops/marnew/ and headed toward IETF
working groups) with wireless carriers about encryption and about UDP-based
transports, and I wanted to level-set on what people are (still) seeing
these days.

Thanks,

Spencer

> > The bigger picture is even more scary - what happens if our reliable
> transport suddenly no longer is reliable?
> >

---

From: Joe Touch <[email-masked]>
Date: Wed, 27 Jul 2016 11:51:58 -0700

On 7/27/2016 8:27 AM, Spencer Dawkins at IETF wrote:
> Hi, Joe,
>
> On Wed, Jul 27, 2016 at 10:18 AM, Joe Touch <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Olle,
>
>     > This mess caused me sadly to suggest that we need to discuss
>     breaking the assumption that TCP delivery is always reliable
>     > and implement retransmits even over TCP in the STUN protocol.
>     STUN was designed to discover middleboxes
> Right. IIRC, you and I were part of a number of conversations about
> this in PILC, while working on https://www.ietf.org/rfc/rfc3135.txt.

Yup - I'm just observing that this (mis)behavior has been seen in the
wild since the mid 90s. It was the topic of much discussion at the Web
Caching Workshops of that era.
>
> My reason for asking Olle to bring this forward is that we're having a
> lot of conversations (starting at the IAB
> with https://www.iab.org/activities/workshops/marnew/ and headed
> toward IETF working groups) with wireless carriers about encryption
> and about UDP-based transports, and I wanted to level-set on what
> people are (still) seeing these days.

Sure - my point is that the term "transparent proxy" is common, and ALL
such animals break TCP semantics *by design*.

Yes, it's possible to recover TCP semantics at a higher layer using
transaction confirmations, but that just sets up a game of mutual
escalation - once you do that, someone will invent a transparent
transaction proxy and you'll be back where you started.

IMO, transparent proxies should be considered the errors the are,
detected, and removed.

Joe

>     > The bigger picture is even more scary - what happens if our
>     reliable transport suddenly no longer is reliable?
>     >

---

From: "Olle E. Johansson" <[email-masked]>
Date: Thu, 28 Jul 2016 09:20:32 +0200

> On 27 Jul 2016, at 17:18, Joe Touch <[email-masked]> wrote:
> 
> Olle,
> 
Even more sad - I just discovered them.
> The challenge with STUN has always been that many middleboxes *do not
Which is one reason to improve STUN - right?

> Ask them *how* they think TLS helps. TLS relies on TCP semantics.
I asked the very same question, got no answer. Will get back to them.

/O

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 28 Jul 2016 00:33:41 -0700

You may want to move this discussion to the spud mailing list. Thats IMHO the
"improve STUN" solution.

On Thu, Jul 28, 2016 at 09:20:32AM +0200, Olle E. Johansson wrote:
> >> One developer from a well known mobile system vendor said ???well, I guess that using TLS may help??????
> > 

-- 
---
Toerless Eckert, [email-masked]

---

From: "Diego R. Lopez" <[email-masked]>
Date: Thu, 28 Jul 2016 08:44:19 +0000

On 27 Jul 2016, at 20:51 , Joe Touch <[email-masked]<mailto:[email-masked]>> wrote:

On Wed, Jul 27, 2016 at 10:18 AM, Joe Touch <[email-masked]<mailto:[email-masked]>> wrote:
Olle,

Right. IIRC, you and I were part of a number of conversations about this in PILC, while working on https://www.ietf.org/rfc/rfc3135.txt.

Yup - I'm just observing that this (mis)behavior has been seen in the wild since the mid 90s. It was the topic of much discussion at the Web Caching Workshops of that era.

My reason for asking Olle to bring this forward is that we're having a lot of conversations (starting at the IAB with https://www.iab.org/activities/workshops/marnew/ and headed toward IETF working groups) with wireless carriers about encryption and about UDP-based transports, and I wanted to level-set on what people are (still) seeing these days.

Sure - my point is that the term "transparent proxy" is common, and ALL such animals break TCP semantics *by design*.

Yes, it's possible to recover TCP semantics at a higher layer using transaction confirmations, but that just sets up a game of mutual escalation - once you do that, someone will invent a transparent transaction proxy and you'll be back where you started.

IMO, transparent proxies should be considered the errors the are, detected, and removed.

Fully agree. And the best solution I can imagine for this removal, avoiding at the same time mutual escalation, is precisely the idea of a controlled endpoint-path collaboration framework a-la-PLUS.

Be goode,

--
"Esta vez no fallaremos, Doctor Infierno"

Dr Diego R. Lopez
Telefonica I+D
http://people.tid.es/diego.lopez/

e-mail: [email-masked]
Tel:    +34 913 129 041
Mobile: +34 682 051 091
----------------------------------

________________________________

Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.

The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.

Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição

---

From: Joe Touch <[email-masked]>
Date: Thu, 28 Jul 2016 10:38:13 -0700

> Which is one reason to improve STUN - right?

You can't fix something that doesn't *want* to be found.  So-called
"transparent" middleboxes (I call them hijacking attackers) do
everything possible to hide, which means they refused to participate in
any mechanism you create and try very hard not to be discovered at all.

All you can do is cause them visibly break so you can detect and
eradicate them.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 28 Jul 2016 10:40:05 -0700

> You may want to move this discussion to the spud mailing list. Thats IMHO the
If this were a situation SPUD or STUN could improve, I might agree.

I don't think that's the goal of either system, though - there is no
system that can interact with something that does everything possible to
NOT interact.

Joe

---

From: "Olle E. Johansson" <[email-masked]>
Date: Thu, 28 Jul 2016 21:57:44 +0200

> On 28 Jul 2016, at 19:38, Joe Touch <[email-masked]> wrote:
> 
> 
> 

If you check the paper I referred to they have detected the presence 
of TCP proxys, which may help us with setting protocol options
right in order to work. Or just fail.

/O

---

From: Joe Touch <[email-masked]>
Date: Thu, 28 Jul 2016 13:03:16 -0700

> right in order to work. Or just fail.
My point is that this logic is backwards.

The order of TCP options is whatever TCP wants them to be at the
endpoints. It should NEVER be constrained by these devices, which are
already clearly violating spec.

I don't want everyone's TCP to "just fail" when it hits these boxes -
perhaps it would be useful to create a custom TCP that can be used to
find these boxes, but once they're found we need to point them out and
demand they be removed (if you're paying for "Internet" behind such a
box, you might have the right to make such a demand).

I.e., we should NEVER use these boxes to govern how we build TCP for the
masses.

Joe

---

From: "Olle E. Johansson" <[email-masked]>
Date: Thu, 28 Jul 2016 22:07:55 +0200

> On 28 Jul 2016, at 22:03, Joe Touch <[email-masked]> wrote:
> 
> 
> 

That’s right - but we do need to find out what happens. Right now our
support is getting a lot of issues that is hard to explain and we’re not even
customers of these carriers. Telling the customer that the net is broken
doesn’t help us - “Facebook, Pokemon and the others work” 

We either break the assumption that TCP is reliable, find these boxes and refuse to work 
with them or will continue to get complains…

We are pressed into a corner and want to find a way out.

At some point we as a community need to find a good “INternet test tool”
for the masses that doesn’t just focus on download bandwidth - but also
quality of the Internet connectivity.

/O

---

From: Theodore V Faber <[email-masked]>
Date: Thu, 28 Jul 2016 20:11:50 +0000

On 7/28/16 13:04, Joe Touch wrote:

> I.e., we should NEVER use these boxes to govern how we build TCP
> for the masses.

To say that another way: vendors who produce such devices are failing
to follow the Postel principle. Whether the IETF says anything
explicitly or not, they're producing artifacts of less value to their
customers.  Standards-compliant stacks will emit TCP options in ways
that such vendors evidently don't expect.

The IETF standard reflects a consensus among designers and
implementers that there's no constraint on TCP option ordering.  The
time to argue about it is past; live with it or produce crappy products.

- -- 
Ted Faber <[email-masked]>
Engineering Specialist
Computer Systems Research Department
310-336-7373

---

From: Joe Touch <[email-masked]>
Date: Thu, 28 Jul 2016 13:14:13 -0700

> doesn’t help us - “Facebook, Pokemon and the others work” 

It might be useful to have tools that both detect these boxes and
"repair" their effects, e.g., by tunneling over them.

However, we need to be careful not to build in these temporary repairs
into our widely used protocols.

> with them or will continue to get complains…

The boxes have broken the assumption, not you. I agree you need to find
a way to undo what the box is, but corrupting TCP to be tolerant of
byzantine network behavior is a bad idea IMO.

> quality of the Internet connectivity.

Agreed.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 28 Jul 2016 13:17:40 -0700

On 7/28/2016 1:11 PM, Theodore V Faber wrote:
> to follow the Postel principle. 

The Postel Principle talks about what to do when the docs don't say
otherwise.

> Whether the IETF says anything
> explicitly or not,

Actually, they're quite directly contradicting explicit existing standards.

> they're producing artifacts of less value to their

Those vendors shouldn't be looking at TCP options or TCP at all. It's
none of their business.

Maybe we should just start using IPsec in BTNS mode (to avoid needing
keys) as a tunnel to get through all such devices.

> time to argue about it is past; live with it or produce crappy products.

This issue cannot be fixed merely by reacting to what vendors deploy.

The solution has been clear for a long time - *compliance verification*.
I assure you that vendors that get sued for saying "Internet compatible"
who are not would behave differently.

Joe

---

From: Theodore V Faber <[email-masked]>
Date: Thu, 28 Jul 2016 20:29:42 +0000

On 7/28/16 13:18, Joe Touch wrote:
>>> I.e., we should NEVER use these boxes to govern how we build
>>> TCP for the masses.
>> 
>> To say that another way: vendors who produce such devices are
>> failing to follow the Postel principle.
> 
> The Postel Principle talks about what to do when the docs don't
> say otherwise.

No.  The Postel Principle applies to networking in general.  "Be
simple in your behavior and tolerant of others" does not include any
proviso about an IETF position.

> 
>> Whether the IETF says anything explicitly or not,
> 

Which falls under "says anything," right?  Feeling crabby, are we?

> 
>> they're producing artifacts of less value to their customers.
>> Standards-compliant stacks will emit TCP options in ways that
>> such vendors evidently don't expect.
> 
> Those vendors shouldn't be looking at TCP options or TCP at all.
> It's none of their business.
> 
> Maybe we should just start using IPsec in BTNS mode (to avoid
> needing keys) as a tunnel to get through all such devices.
> 
>> 
>> implementers that there's no constraint on TCP option ordering.
>> The time to argue about it is past; live with it or produce
>> crappy products.
> 

"React?"  Bah.

How is my position unclear here?  I advocate that the IETF do nothing.

Perhaps in a just world, their competitors should pipe up and say
"Whoever's crappy product breaks stuff."  The IETF has said its piece:
"there is no constraint on the order of TCP options."

> 
> The solution has been clear for a long time - *compliance
> verification*. I assure you that vendors that get sued for saying
> "Internet compatible" who are not would behave differently.

We prefer different societal pressures, evidently.  I like the one
where I can be lazier.

- --

---

From: "Olle E. Johansson" <[email-masked]>
Date: Thu, 28 Jul 2016 22:40:55 +0200

> On 28 Jul 2016, at 22:17, Joe Touch <[email-masked]> wrote:
> 
> 
> 
Well, the question is what they are trying to solve by breaking TCP?
I  personally suspect that the latency on some radio networks cause
a lot of retransmits. By adding a TCP proxy they avoid retransmits 
wasting bandwidth and at the same time causing issues when it 
doesn’t work. Can we help them with a better solution if this is 
the problem?

Does anyone know what the problem they are trying to solve is?
(Still trying to find information)

> who are not would behave differently.

So where do we have a good specification that we agree on defining
“Internet compatible” ?

/O

---

From: Joe Touch <[email-masked]>
Date: Thu, 28 Jul 2016 13:45:22 -0700

> proviso about an IETF position.

"Be liberal in what you accept, and conservative in what you send."  -
RFC1122

RFC760 puts it into exactly the context I indiicated:

  The implementation of a protocol must be robust.  Each implementation
  must expect to interoperate with others created by different
  individuals.  While the goal of this specification is to be explicit
  about the protocol there is the possibility of differing
  interpretations.  In general, an implementation should be conservative
  in its sending behavior, and liberal in its receiving behavior.

I.e., applies when there is ambiguity.

> "there is no constraint on the order of TCP options."

That's not strictly true. RFC5925 requires that TCP-AO come first,
because it would be impossible to undo the effect of processing earlier
options when the AO option might indicate the packet is not authentic.

However, I'm only saying that the IETF should not be changing TCP specs
solely to transit devices that violate this order independence.

> We prefer different societal pressures, evidently.  I like the one

It's fine to say "just don't deal with middlebox option order issues",
but then we end up with broken behavior.

Is that a viable way to leave things?

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 28 Jul 2016 13:49:55 -0700

> doesn’t work. Can we help them with a better solution if this is 
They should be doing FEC or ARQ at the link layer or via a tunnel across
multiple radio links, but that requires more effort on their part
(inserting and coordinating multiple components, rather than just one).

> (Still trying to find information)
The one you cite above is one. Another intermittent connectivity.
Another is bandwidth or device capability, but in those cases the proxy
can be visible.
> So where do we have a good specification that we agree on defining

RFC1122/1123 for hosts.

RFC1812 for routers.

Joe

---

From: Theodore V Faber <[email-masked]>
Date: Fri, 29 Jul 2016 14:52:36 +0000

>> simple in your behavior and tolerant of others" does not include
>> any proviso about an IETF position.
> 
> "Be liberal in what you accept, and conservative in what you send."
> - RFC1122
> 
> The implementation of a protocol must be robust.  Each
> implementation must expect to interoperate with others created by
> different individuals.  While the goal of this specification is to
> be explicit about the protocol there is the possibility of
> differing interpretations.  In general, an implementation should be
> conservative in its sending behavior, and liberal in its receiving
> behavior.
> 

That's a remarkable interpretation of the "In general," prefix that I
simply disagree with.  (And for the record, I agree that the principle
applies when there's ambiguity;  I also think it applies when there is
none - it applies "In general".)

Beyond the textual analysis, I think restricting that idea in the
(IMHO narrow) context of resolving specification ambiguity sells it
short.

>>>> implementers that there's no constraint on TCP option
>>>> ordering. The time to argue about it is past; live with it or
>>>> produce crappy products.
>> 
>> Perhaps in a just world, their competitors should pipe up and
>> say "Whoever's crappy product breaks stuff."  The IETF has said
>> its piece: "there is no constraint on the order of TCP options."
> 
> because it would be impossible to undo the effect of processing
> earlier options when the AO option might indicate the packet is not
> authentic.
> 
> However, I'm only saying that the IETF should not be changing TCP
> specs solely to transit devices that violate this order
> independence.

That sounds like we agree. :-D

>>> verification*. I assure you that vendors that get sued for
>>> saying "Internet compatible" who are not would behave
>>> differently.
>> 
>> We prefer different societal pressures, evidently.  I like the
>> one where I can be lazier.
> 
> It's fine to say "just don't deal with middlebox option order
> issues", but then we end up with broken behavior.
> 

 I am not personally nor is the IETF collectively in a position to
enforce good behavior from network devices.  At the scale and
complexity of the Internet, I don't think one could enforce even
compliance with published IETF standards.

I think trying to turn the IETF from a group of engineers who publish
their consensus on interoperation issues (y'know mostly) to some kind
of implementation evaluation and enforcement body is a waste of time
at best and counterproductive at worst.

I understand that we disagree on this, and I'm a little sad we won't
get to do so over coffee.

- --

---

From: Joe Touch <[email-masked]>
Date: Fri, 29 Jul 2016 10:15:48 -0700

> (IMHO narrow) context of resolving specification ambiguity sells it

Can I then ask where you define the limit?

What happens when you get an IP packet with protocol type = 18 (not 17),
but it otherwise represents a well-formed UDP payload?

My interpretation suggests the following:

- If there is a valid behavior according to the spec, it MUST be
accepted even it it does not represent the typical interpretation of the
spec, or if is otherwise "unusual" (I've phrased this before as "a
packet doesn't represent an attack merely because it's not what you
expect").

- If a spec prohibits something, it MUST NOT be accepted (even if it
"could" be)

This covers all possible cases (spec is clear, spec is ambiguous). I
include "spec says nothing" as "spec is ambiguous" IF there are multiple
ways that the missing information could be interpreted *consistent with
the explicit limits of the spec*.

So I'm struggling to figure out an example where you think "in general"
means what you think above (FWIW, I think it is intended to refer to the
phrase - i.e., "put in a general way", not "this should be applied to
all cases, not just ambiguities").

Can you give an example?

Joe

---

From: Theodore V Faber <[email-masked]>
Date: Fri, 29 Jul 2016 17:35:13 +0000

On 7/29/16 10:16, Joe Touch wrote:
>>>> implementation must expect to interoperate with others
>>>> created by different individuals.  While the goal of this
>>>> specification is to be explicit about the protocol there is
>>>> the possibility of differing interpretations.  In general, an
>>>> implementation should be conservative in its sending
>>>> behavior, and liberal in its receiving behavior.
>>>> 
>> That's a remarkable interpretation of the "In general," prefix
>> that I simply disagree with.  (And for the record, I agree that
>> the principle applies when there's ambiguity;  I also think it
>> applies when there is none - it applies "In general".)
>> 
>> Beyond the textual analysis, I think restricting that idea in
>> the (IMHO narrow) context of resolving specification ambiguity
>> sells it short.
> 

My limit is 3.  In general.

The Postel Principle is an aphorism.  A principle.  Advice.  I would
not even try to quantify this good advice (my limit is completely
facetious).

I'm not going to advocate ignoring IETF standards based on the words
of Jon Postel.  If you look at my first message in this thread, you'll
notice that I castigated these implementers *based* *on* *that*
*principle*:

	To say that another way: vendors who produce such devices [that make
assumptions about TCP option ordering] are failing to follow the
Postel principle.

In general, being conservative in what one sends implies following a
standard when one exists.  Why are we trying to quantify our agreement?

- --