Subject: [tcpm] Fwd: New Version Notification for draft-gont-tcpm-urgent-data-00

From: Fernando Gont <[email-masked]>
Date: Mon, 27 Oct 2008 20:54:57 -0300

Hello, folks,

FYI, we have published a new Internet-Draft entitled "On the 
implementation of TCP urgent data". The draft is available at: 
http://tools.ietf.org/id/draft-gont-tcpm-urgent-data-00.txt  (see the 
Abstract on other details bellow).

Any comments will be more than welcome.

Thanks!

Kind regards,
Fernando

>A new version of I-D, draft-gont-tcpm-urgent-data-00.txt has been 
>successfuly submitted by Fernando Gont and posted to the IETF repository.
>
>Filename:       draft-gont-tcpm-urgent-data
>Revision:       00
>Title:          On the implementation of TCP urgent data
>Creation_date:  2008-10-28
>WG ID:          Independent Submission
>Number_of_pages: 10
>
>Abstract:
>This document analyzes the current practices for handling TCP urgent
>data in the current Internet.  It describes how popular TCP
>implementations process urgent data, and also describes how a number
>of middle-boxes affect how urgent data is processed by end systems.
>Additionally, it includes a survey of the processing of urgent data
>by popular TCP implementations.  This document updates the relevant
>specifications such that they accommodate current practice in
>processing TCP urgent data.

--
Fernando Gont
e-mail: [email-masked] || [email-masked]
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

_______________________________________________
tcpm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tcpm

---

From: David Borman <[email-masked]>
Date: Mon, 27 Oct 2008 20:01:17 -0500

I've done a quick scan of this document, and it indicates to me a  
misunderstanding of urgent data.

First, despite what 793 and 1122 say, TCP does *not* have urgent data  
in the sense that people think.  TCP has an urgent *pointer*, that  
indicates an interesting point in the data stream.  *All* the data  
prior to, and including, the Urgent pointer is the urgent data.  It is  
up to the application to decide what that means, and which data  
relative to the urgent pointer is interesting, but typically it would  
be the data just after the urgent pointer.  The application can't say  
"these x bytes of data are the urgent data".

If a TCP data stream writes some data in normal mode and the data gets  
lost, then there is a successive write in urgent mode, then when the  
lost packet is re-transmitted it'll now be transmitted with the URG  
bit and the urgent pointer, turning that data into "urgent data" by  
the 793 definition.  That's not what people think when they first hear  
the term "urgent data", but that's how it works.

It is much easier to understand if you don't use the term "urgent  
data", and just think about the urgent pointer.  The reader goes into  
urgent mode reading data until it gets to the urgent pointer, at which  
point it goes back into normal mode.  All the data is delivered to the  
application in sequence.  TCP does not have an out-of-band data  
delivery mechanism, and this is where the original BSD code got into  
trouble, because it took one byte out of the data stream and delivered  
it to the application out-of-band.  To fix that, the SO_OOBINLINE  
socket option was introduced, and is still around today.  All  
applications that use urgent data should be using the SO_OOBINLINE  
option, there's no excuse for using the old, broken BSD interpretation  
about urgent data.

There isn't anything wrong with the 793/1122 definition of the Urgent  
Pointer.  It might be useful to clarify the description if people are  
still implementing and/or using it wrong, but the definition in  
793/1122 is still correct.

			-David Borman

On Oct 27, 2008, at 6:54 PM, Fernando Gont wrote:

> implementation of TCP urgent data". The draft is available at: http://tools.ietf.org/id/draft-gont-tcpm-urgent-data-00.txt 
>   (see the Abstract on other details bellow).
>

---

From: Fernando Gont <[email-masked]>
Date: Mon, 27 Oct 2008 22:58:08 -0300

Hello, David,

Thanks for your timely response. You'll find my comments inline....

>prior to, and including, the Urgent pointer is the urgent data.

Agreed. However, there are two issues here:

Firstly, the recv(2) with MSG_OOB only returns a single byt of urgent 
data, whereas according to your explanation it should return all data 
prior to, and including, the Urgent pointer.

Secondly, virtually all implementations intepret that the Urgent 
pointer point to the byte following the last byte of urgent data. And 
this is what RFC 1122 aims at correcting. (However, this correction 
didn't reflect into actual implementations.)

>It is

mmm... why couldn't it say, e.g.,  that all the bytes up to the 
urgent pointer are urgent data? (currently, only the byte before the 
Urgent Pointer is interpreted as Urgent data... which does not seem 
to agree with what the RFCs specify....)

>the term "urgent data", but that's how it works.

Agreed.

>option, there's no excuse for using the old, broken BSD interpretation

Well, this seems to be the default for all major TCP implementations 
(see our survey).

>Pointer.

The specific issue we are discussing is that the Urgent pointer is 
defined in RFC 793 as pointing "to the byte following the last byte 
of urgent data". RFC 1122 later corrected this definitation, stating 
that it points to "the last byte of urgent data".

>  It might be useful to clarify the description if people are

Section 4.2.2.4 of RFC 1122 states:
"The second sentence is in error: the urgent pointer points
to the sequence number of the LAST octet (not LAST+1) in a
sequence of urgent data. The description on page 56 (last
sentence) is correct."

This is where I'm saying that RFC 1122 differs from RFC 793.

Thanks!

Kind regards,

---

From: Andrew Yourtchenko <[email-masked]>
Date: Tue, 28 Oct 2008 04:46:36 +0100 (CET)

Hello David,

thanks a lot for the comments. Addition to Fernando's comments inline.

On Mon, 27 Oct 2008, Fernando Gont wrote:

> Well, this seems to be the default for all major TCP implementations (see our 
> survey).
>

Just to add - the code I created for testing was deliberately 
without the SO_OOBINLINE - under the "naive" premise that indeed
this protocol implementation bug would've been corrected by 
now and the "standards-compliant" behaviour would be the default.

>> All applications that use
>>urgent data should be using the SO_OOBINLINE option, there's no excuse for 
>>using the old, broken BSD interpretation about urgent data.

Again, the deliberate "naive expectation" was that the old and known 
broken implementation is *not* the default in the today's TCP stacks, but 
would require a configuration tweak to fallback to buggy behaviour used in 
the legacy applications.

> The specific issue we are discussing is that the Urgent pointer is defined in 
> RFC 793 as pointing "to the byte following the last byte of urgent data". RFC 
> 1122 later corrected this definitation, stating that it points to "the last 
> byte of urgent data".
>

So, we have three options:

- "old and buggy" semantics
   -> default with no additional effort 
- the incorrect semantics from the RFC793 - correct by the RFC1122
   -> default after adding the additional code to set
      the socket to SO_OOBINLINE to get away from the "old and buggy" one
- the "standard" RFC1122 semantics
   -> achievable with SO_OOBINLINE setsockopt *and* the
      host-global configuration switch.

The latter two are split because at least in one scenario within 
the set that we tested there was an observable difference in the 
behaviour - while again this can be classified as a "bug",
it pretty much leaves the reasonable answer to the question
"How to interoperably use the urgent pointer/data in TCP for my 
application?" to be "Don't" ?

thanks,
andrew

---

From: David Borman <[email-masked]>
Date: Tue, 28 Oct 2008 00:42:55 -0500

On Oct 27, 2008, at 8:58 PM, Fernando Gont wrote:

> Firstly, the recv(2) with MSG_OOB only returns a single byt of  
> urgent data, whereas according to your explanation it should return  
> all data prior to, and including, the Urgent pointer.

No, applications shouldn't be using recv() with MSG_OOB on TCP  
sockets.  That is the interface for the old, broken BSD interpretation  
of the urgent pointer.  TCP does not have out of band data.  The  
SO_OOBINLINE option was added so that new applications (new, circa  
1989, the date of RFC 1122) could turn this broken interface off and  
leave the data in the data stream where it belongs.  It was felt that  
we couldn't just change the BSD TCP implementation to the new  
behavior, because that would break all the existing applications.   
With SO_OOBINLINE, applications could switch over without there  
needing to be a flag day.

> pointer point to the byte following the last byte of urgent data.  
> And this is what RFC 1122 aims at correcting. (However, this  
> correction didn't reflect into actual implementations.)
>
>
>> It is

The RFC more or less does say that.  The moment you get a packet with  
the URG bit, you enter urgent mode for all data until you get past the  
urgent pointer, hence that is all urgent data.  The key things are:

1) The urgent pointer only tells you where the urgent data ends
2) Urgent pointers are cumulative, a new urgent pointer updates where  
you leave urgent mode.
3) The urgent pointer can point beyond the data in the packet in which  
it is sent.

The problem is that (most of) the urgent data, as defined in RFC 793,  
is typically the uninteresting data.  For example, in telnet the  
urgent data is the data you want to read and flush, until you get to  
the Telnet Sync command.

> Well, this seems to be the default for all major TCP implementations  

Ok, there are two issues here.  First, SO_OOBINLINE turns off the old,  
broken BSD interpretation that removed a byte of data from the data  
stream and presented it to the application with the recv()/MSG_OOB  
call.  The second issue is which byte is indicated in the packet with  
the urgent pointer.  If the application knows what the kernel does, it  
can adjust accordingly so that the correct byte gets marked in the TCP  
data stream.

> that it points to "the last byte of urgent data".

That's why I said 793/1122.  RFC 793 defined it both ways, and in RFC  
1122 we clarified that the definition on page 56 is correct, and the  
one on page 17 is wrong.

> This is where I'm saying that RFC 1122 differs from RFC 793.

Yes, of course.  RFC 1122 says that this definition is wrong.  In RFC  
793 the urgent pointer is also defined on page 56:

       If the urgent flag is set, then SND.UP <- SND.NXT-1 and set the
       urgent pointer in the outgoing segments.

which conflicts with the definition on page 17.  RFC 1122 clarifies  
that this definition on page 56 is the correct definition.

I'll also point out that RFC 961 also says that the definition on page  
17 is wrong; RFC 1122 re-enforces what was already documented.

I will say that using the recv()/MSG_OOB interface is the bigger  
issue.  You just have to look at the BSD Telnet implementation and the  
dances that it does if SO_OOBINLINE isn't defined to understand why  
that broken interface is such a bad idea.

			-David Borman

---

From: Fernando Gont <[email-masked]>
Date: Tue, 28 Oct 2008 23:21:04 -0300

At 02:42 a.m. 28/10/2008, David Borman wrote:

>leave the data in the data stream where it belongs.

By default, the byte pointer byte (UP-1) is till kept OOB, nowadays.

Even worse, it has been reported to me that some major operating 
system did things even worse: it would queue what it thought was 
urgent data until the application read it by a recv(")/MSG_OOBB-like 
call. If the app didn't recv() this "OOB", then the urg data would 
pile up, until a DoS condition....

>  It was felt that

I believe this should be clarified. Do you?

I will prepare some text to update our draft before the 
non-00-version deadline....

>call.

Agreed. I believe this should be clarified.

>The second issue is which byte is indicated in the packet with

Not sure what you mean. The issue here is how the *remote* TCP sets 
the urgent pointer. How could you know whether the remote TCP set the 
UP such that it points to the last byte of urgent data (RFC1122) or 
to the byte following the last byte of urgent data?

Considering that even SO_OOBINLINE doesn't change the semantics of 
the UP, I think RFC 1122 could be updated to reflect what current 
implementations are doing. (this is one of the "updates" that our 
draft was aiming)

>that broken interface is such a bad idea.

So we have to issues:

* Deprecate recv(2)/MSG_OOB and encourage SO_OOBINLINE. (with which 
you seem to agree)

* Update RFC 793/RFC1122 wrt where the UP points to, to reflect what 
everybody does and has been doing. (with which I don't know whether 
you agree or not).

Please let me know what you think, so that we can update our draft accordingly.

Thanks!

Kind regards,

---

From: Joe Touch <[email-masked]>
Date: Wed, 29 Oct 2008 08:29:26 -0700

Fernando Gont wrote:
...
> So we have to issues:
> 
> * Deprecate recv(2)/MSG_OOB and encourage SO_OOBINLINE. (with which you
> seem to agree)

That sounds like a Unix man page update, not an RFC.

> everybody does and has been doing. (with which I don't know whether you
> agree or not).

Either this is an implementation error or it updates our definition of PUSH.

If the former, it can be noted in the next version of the TCP
implementation errors doc. If the latter, it would update 1122 - NOT
793. As Dave noted, 1122 supercedes 793 on this issue.

I don't like the idea of continuing to document implementations that do
not comply with current specs without either stating
	a) this is BAD and should be fixed
	b) this is GOOD and we're changing our spec

FWIW, so far, this looks to me like an implementation error, not a
reason to change the spec.

Joe