Subject: TCP, multiple addresses and soft errors when connecting

From: Pekka Savola <[email-masked]>
Date: Fri, 5 Mar 2004 04:52:24 +0200 (EET)

Hi all (mainly the TCP guys!),

v6ops WG is in the process of documenting issues which are caused or 
made more severe by the introduction of IPv6.  The document is 
draft-ietf-v6ops-v6onbydefault-01.txt

TCP timeouts when connecting to a destination, but when the 
destination is unreachable (e.g., because you don't have global IPv6 
routes, your first-hop router is not operational, the packets are sent 
on-link by default, etc.), TCP does not abort the connectivity (and 
try the next address quicker -- instead of waiting for the TCP 
timeout).

The critical observation to make here is as IPv6 addresses are tried
first (I'm simplifying; it's more complex than that), one must cycle
through all the addresses, before trying with IPv4.  However, IPv4
connections might still work! With TCP timeouts, this takes a lot of
time --- and should be made quicker.

There is going to be a WG last call on the document soon, aimed for 
Informational -- just for documenting the issues.  The actual fixes 
(or not) would have to be done elsewhere (e.g., in TCPM WG ;-).

Please provide feedback especially on the language used for TCP.  
(Below.)

draft-ietf-v6ops-v6onbydefault-01.txt:

==========
2.3.1 TCP Implications

   In the case of a socket application attempting a connection via TCP,
   it would be unreasonable for the application to block even after the
   system has received notification that the destination address is
   unreachable via an ICMPv6 Destination Unreachable message.

   Following are some ways of solving TCP related delays associated with
   destination unreachability when ICMPv6 errors are generated.

2.3.1.1 TCP Connection Termination

   One solution is for TCP to abort connections in SYN-SENT or
   SYN-RECEIVED state when it receives an ICMPv6 Destination Unreachable
   message.

   [HOSTREQS] document, in section 4.2.3.9., states that TCP MUST NOT
   abort connections when receiving ICMP Destination Unreachable
   messages that indicate "soft errors", where soft errors are defined
   as ICMP codes 0 (network unreachable), 1 (host unreachable), and 5 
   (source route failed), and SHOULD abort connections upon receiving 
   the other codes (which are considered "hard errors").  ICMPv6 didn't
   exist when that document was written, but one could extrapolate the 
   concept of soft errors to ICMPv6 Type 1 codes 0 (no route to
   destination) and 3 (address unreachable), and hard errors to the
   other codes. Thus, it could be argued that a TCP implementation that
   behaves as suggested in this section is in conflict with [HOSTREQS].

   When [HOSTREQS] was written, most applications would mostly only try
   one address when establishing communication with a destination.  Not
   aborting a connection was a sane thing to do if re-trying a single
   address was a better alternative over quitting the application
   altogether. With IPv6, and especially on dual stack systems,
   destinations are often assigned multiple addresses (at least one IPv4
   and one IPv6 address), and applications iterate through destination  
   addresses when attempting connections.

   Since soft errors conditions are those that would entice an
   application to continue iterating to another address, TCP shouldn't 
   make the distinction between ICMPv6 soft errors and hard errors when
   in SYN-SENT or SYN-RECEIVED state.  It should abort a connection in
   those states when receiving any ICMPv6 Destination Unreachable
   message.  When in any other state, TCP would behave as described in 
   [HOSTREQS].

   Many TCP implementations already behave this way, but others do not.
   This should be noted as a best current practice in this case.

   A tangential method of handling the problem in this way would be for
   applications to somehow notify the TCP layer of their preference in
   the matter.  An application could notify TCP that it should abort a
   connection upon receipt of particular ICMPv6 errors.  Similarly, it 
   could notify TCP that it should not abort a connection.  This would 
   allow existing TCP implementations to maintain their status quo at 
   the expense of increased application complexity.

2.3.1.2 Asynchronous Application Notification

   In section 4.2.4.1, [HOSTREQS] states that there MUST be a mechanism
   for reporting soft TCP error conditions to the application. Such a  
   mechanism (assuming one is implemented) could be used by applications
   to cycle through destination addresses.
===========

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Mark Allman <[email-masked]>
Date: Mon, 15 Mar 2004 23:31:23 -0500

(catching up)

> time --- and should be made quicker.

Speaking personally, I think what you have written is fine and would not
block the document on it.

> (or not) would have to be done elsewhere (e.g., in TCPM WG ;-).

Speaking as tcpm co-chair, yeah - we'd certainly be game to help out
here.

allman

--
Mark Allman -- ICIR -- http://www.icir.org/mallman/

---

From: Fernando Gont <[email-masked]>
Date: Fri, 19 Mar 2004 13:27:29 -0300

>try the next address quicker -- instead of waiting for the TCP

Not sure if I got it right. Do you mean TCP does not abort the conectivity, 
and thus the *application* cannot try the next address quicker?

>    SYN-RECEIVED state when it receives an ICMPv6 Destination Unreachable

But what if this scenario arises for all the addresses, because of a 
transient problem?
The application won't cycle again, and will return an error. And the 
*transient* problem could disappear perhaps 5 seconds later...

IMHO, if you had a "higher-level" connect(), this behavior would make 
sense, as you could handle the whole issue, in a transparent way to the 
application.

Does it really make sense to handle the whole connection establishment 
issue inside the app, instead of inside the API itself?

I think that if we're debating this, we're implicitly assuming that 
applications want to connect to say www.example.com, and not one specific 
IP address to which it maps to.

So why not implement such a "higher-level" connect(), and handle all the 
relevant issues inside the API?

>    addresses when attempting connections.

Doesn't the problem really have to do with having applications performing 
such a low-level action?

Just my two cents,

--
Fernando Gont
e-mail: [email-masked] || [email-masked]

---

From: Pekka Savola <[email-masked]>
Date: Sat, 20 Mar 2004 16:39:00 +0200 (EET)

> and thus the *application* cannot try the next address quicker?

Yep, that's what I meant (and by application, I mean both the app and
the APIs/libraries it's using).

> *transient* problem could disappear perhaps 5 seconds later...

Note that this applies to new connections only.

Yes, this is a trade-off.  The assumption is that with big transient
problems, all the addresses {at least of the same address family} are
unreachable, or none of them are.  

So, you either have to choose between having TCP connect() try
retrying for dozens of seconds (hoping the transient to go away) or
failing completely.

The tradeoff recommended was that it's better to fail completely; this
works better for transient and non-transient failures, except for the
case where the transient failure was so short that the TCP retry
mechanism could be the feasible failure recovery mechanism for that.  
I don't think it is..

Obviously, there could be a flag to indicate to the stack whether you 
want this behaviour or not.

> issue inside the app, instead of inside the API itself?

Those are the core APIs we have today.  We could have better ones, but
IMHO I think we need to fix problems in the current ones as long as we
don't have a widely-deployed alternative.
 
> relevant issues inside the API?

Defining such APIs is probably beyond the scope of the work we can do, 
and the things we can fix except in the 5+ year timeframe.. but if I 
get you right, you're proposing that instead of:

 1) getaddrinfo() or gethostbyname() and connect for one address, 
[where, arguably, TCP retry timeout could be warranted -- I don't 
think so myself, but you could argue for it]

 2) getaddrinfo() / gethostbyname() loop to connect to one address
[where TCP retry timeout could not be warranted]

You'd be proposing to create a new function to accomplish 2) in such a 
manner that it would also react to ICMP messages, etc?

> Doesn't the problem really have to do with having applications performing 

At some point, someone has to decide which behaviour is desirable.  
Isn't this only hiding the same policy decision inside an abstraction 
layer -- and the same could be achieved by e.g. a setsockopt() flag?

(I don't deny that better APIs would not hurt, but we still seem to 
require low-level C-like primitives, and unless there are additional 
ones implemented there, we still have to fix the problems somehow..)

---

From: Fernando Gont <[email-masked]>
Date: Mon, 22 Mar 2004 08:18:12 -0300

>don't have a widely-deployed alternative.

IMHO, there are two points to note:

* If you fix the current ones so that their behavior is "acceptable" to the 
application programmer, then it'll be harder to introduce a new API. People 
is reluctant to change, so they must have a reason for learning a new API. 
If you try to fix the current one, they will find fewer reasons for the change.

* You said "we don't have a widely-deployed alternative". Unless you meant 
"we don't have a well-tested alternative", I'd say that it's a 
chicken-and-egg problem. That means, for an API to be widely deployed, 
people would have to use it. And for people to use it, they should be able 
to find a good reason to do it.

Yes, I'm aware that my proposal my sound as a long-term solution. But, 
IMHO, it will certainly be harder to introduce a new API if you actually 
want to do it "in long term".
I guess "timing" is an issue, here.

>manner that it would also react to ICMP messages, etc?

Exactly.

Then, within that function, each individual connect() could react to ICMP 
messages a
as you proposed, but if all addresses fail because of ICMP errors, it could 
let TCP retry in the hope the transiet problem will dissappear.

[....]
>layer

Yes, but even when the difference may sound subtle, IMHO it's not.
Having that abstraction layer is what let's you choose a policy, without 
having the risk of causing any harm to existing apps.
That way, if you ever need to change the low-level-connect() behavoir for 
whatever reason, you could still do it. As far as the high-level connect() 
behaves the same, there would be no problem.

Think about it. Why should an application programmer care about TCP 
retries, ICMP erros, an the like?
In the same way, why should he care about setting "strange" options such as 
SO_REUSEADDR on his listening socket?
Should *he* really handle byte-ordering issues  by means of htons() and the 
like?

>-- and the same could be achieved by e.g. a setsockopt() flag?

Strictly speaking, you *could* do that. But then you'd have all programmers 
implementing themselves a function you could provide them.
You'd have all them implementing the getaddrinfo()/gethostbyname() loop, 
setting a strange socket option, etc. Why not provide them with a 
well-tested function, and let them spend their time implementing their 
application itself?

>ones implemented there, we still have to fix the problems somehow..)

Well, the low-level ones could be provided as part of the new API.

Or along with the providing a new API, the current one could be modified as 
you suggest. It wouldn't hurt too much if the *low-level* API aborted a 
connection establishment because an ICMP error is received. Being a 
*low-level* API, you'd be suppossed to know the details behind the function.

Best Regards,