Subject: [TLS] Backwards compatilbility problem with variant records

From: "Yngve N. Pettersen" <[email-masked]>
Date: Wed, 28 Mar 2012 17:53:25 +0200

Hi,

As mentioned during my presentation at the TLS WG meeting today, when  
implementing my multiple OCSP stapling specification I encountered a  
problem with the variant records structure in my specification (which was  
based on the corresponding structure in RFC 6066), when I was going to  
code the server side handling of unknown status request types.

Investigating, I found that the Server Name Indication and Trusted CA  
extensions have the same problem, and that this will seriously affect the  
future expandability by these extensions.

The problem occurs with records that can be used in vector records, that  
is sequences, when the client knows, and uses, variants of a record  
structure that the server does not know about, and that they cannot  
negotiate before sending.

When the server tries to read such an unknown variant record, it will not  
be able to tell how much to read from the vector in order to reach the  
next record in the vector.

As an example, in the following example structure, the client knows about  
Wha1 and the associated fields whatsths and whatstht, but the server does  
not. They both know about all the other fields and values.

enum{
   Foo, Bar, Wha1;
} Typ

struct {
   Typ rec_typ;
   Select(rec_typ)
   {
     case Foo: opaque food<1..2^8-1>;
     case Bar: opaque barge<1..2^16-1>;
     case Wha1:
        opaque whatsths<1..2^8-1>;
        opaque whatstht<1..2^16-1>;
   } payload;
}Rec;

Rec Recs<1..2^16-1>;

If the client sends a Recs vector that includes a Wha1 variant, the server  
will not be able to read the Rec.payload field, and will likely crash or  
abandon the connection.

Within the current framework the only way to work around this is to add a  
payload length field before the payload field. It is a matter of taste  
whether it is placed at the beginning of the record, or just before the  
payload, but with the placement of the payload, the entire length and  
payload combination can be treated as an entire opaque<> vector.

This results in the following example structure

enum{
   Foo, Bar, Wha1;
} Typ

struct {
   Typ rec_typ;
   uint16 payload_length;
   Select(rec_typ)
   {
     case Foo: opaque food<1..2^8-1>;
     case Bar: opaque barge<1..2^16-1>;
     case Wha1:
        opaque whatsths<1..2^8-1>;
        opaque whatstht<1..2^16-1>;
   } payload;
}Rec;

Rec Recs<1..2^16-1>;

The server will now be able to read the structure even if it does not know  
about the Wha1 alternative, since it knows how many bytes to ignore.

A more elaborate example of the above can be found in the Appendix (last  
half) of my presentation slides at  
<http://www.ietf.org/proceedings/83/slides/slides-83-tls-3.pdf>

Consequences for existing, deployed extensions?

IMO the currently deployed extensions, at least SNI and Trusted CA will  
have to be frozen in their current designs. If new functionality is  
needed, new extensions will have to be defined.

This may affect other extensions, too.

This might require an errata to be published where the problem is noted,  
and available actions outlined for future updates.

Long term, it might be an idea to consider adding a variable record format  
for the vector type in TLS, alternatively, update the "Select" encoding to  
include a length field. These changes would have to be done in new  
extensions, and/or a new TLS version, and would still not be applied to  
records that will must be understood by older servers.

Please note that I do not think that there is any such problem for the  
client side, *unless* the server can pick variants for vector records that  
are not negotiated in advance.

-- 
Sincerely,
Yngve N. Pettersen

********************************************************************
Senior Developer                     Email: [email-masked]
Opera Software ASA                   http://www.opera.com/
Phone:  +47 23 69 32 60              Fax:    +47 23 69 24 01
********************************************************************

---

From: Marsh Ray <[email-masked]>
Date: Wed, 28 Mar 2012 11:10:53 -0500

> needed, new extensions will have to be defined.

Would it be sufficient to declare that all new extension sub-types 
(i.e., the types selected by the cases) begin with a length count of a 
predefined length?

For example, SNI http://tools.ietf.org/html/rfc4366#section-3.1 :

>       struct {
>           NameType name_type;
>           select (name_type) {
>               case host_name: HostName;
>           } name;
>       } ServerName;
>
>       enum {
>           host_name(0), (255)
>       } NameType;
>
>       opaque HostName<1..2^16-1>;
>
>       struct {
>           ServerName server_name_list<1..2^16-1>
>       } ServerNameList;

So if new NameType is defined, it would be required to also have a 
length field of uint16.

- Marsh

---

From: "Yngve N. Pettersen" <[email-masked]>
Date: Wed, 28 Mar 2012 18:44:47 +0200

> (i.e., the types selected by the cases) begin with a length count of a  

I don't know.

The main question for how that would work is how a *currently* deployed  
server reacts when it reads an unknown variant determining enum?

My guess is that it will not try to parse the variant as a field it  
already know, or as a vector, but will terminate reading of the extension.  
How aggressively it will terminate its reading is a good question; worst  
case it might terminate the connection, best case it will just ignore the  
rest of the current extension and go to the next extension.

Also, reading it as a vector would require that all future additions are  
single vectors, not multiple sequential fields which may or may not be  
vectors (In the Certificate Status extension the OCSP variant is a  
multiple field). The only way to accomodate that would be that all sub  
structures must be wrapped in a vector.

Some variant fields may also be fixed length, such as ints and enum, but  
also fixed-size vectors like the SHA1 hashes in Trusted CA; of course that  
can be avoided in the future, but not for integer/enum values.

-- 
Sincerely,
Yngve N. Pettersen

---

From: Martin Rex <[email-masked]>
Date: Thu, 29 Mar 2012 15:13:05 +0200 (MEST)

> will not be able to read the Rec.payload field, and will likely crash or  

A reasonable implementation would simply skip to the next TLS extension
(ignoring the rest of what it didn't understand).  If it crashes,
it is probably an early beta version that should neither be distributed nor
used in production environments...

>         opaque whatstht<1..2^16-1>;

Shouldn't this more likely read

enum{
   Foo, Bar, Wha1, (255);
} Typ

struct {
   Typ rec_typ;
   Select(rec_typ)
   {
   } payload<1..2^16-1>;
}Rec;

Rec Recs<4..2^16-1>;

-Martin

---

From: "Yngve N. Pettersen" <[email-masked]>
Date: Thu, 29 Mar 2012 16:28:08 +0200

> used in production environments...

Counterargument: SSL v3 and TLS 1.0+ required servers to ignore Client  
Hello content following the compression field. As we all know, a large  
number of servers did not implement it that way, causing the handshakes to  
fail. Similar problem exists for the version indication.

Maybe I am being pessimistic, but I am concerned about the possibility  
that implementations of the affected variants records will have failed in  
similar fashions. Only an extensive and detailed review of what existing  
implementations do will reveal what has been implemented.

> Shouldn't this more likely read

The syntax below is AFAIK currently not supported by RFC 5246, and this is  
IMO clearly indicated by the fact that similar variant records in TLS,  
such as Handshake and the Record Protocol records, are not coded in that  
fashion, but the one I am using above.

Extending the syntax might be an option, but at present I don't think it  
is realistic.

-- 
Sincerely,
Yngve N. Pettersen

---

From: Marsh Ray <[email-masked]>
Date: Thu, 29 Mar 2012 09:39:50 -0500

> used in production environments...

So now there's an ordering dependency on the payload records. All 
records that the server doesn't understand must come after all the 
records that the server needs to process.

There are many other places where TLVs are defined in IETF protocols. 
Surely they require a consistent format for T and L for any given context?

If I were an implementor trying to code from this spec, I would probably 
look at the payload field and think "Well, the only way this is gonna 
work is if all future extensions have the same length to their payload 
length as well. I'll infer that to be the case and yet be sure to check 
if the length I interpret implies the end outside of the extension."

That's the number one rule of extensible record formats: at least fix 
the length field in advance so receivers can know how to skip the stuff 
they don't understand.

- Marsh

---

From: Peter Sylvester <[email-masked]>
Date: Thu, 29 Mar 2012 17:00:23 +0200

On 03/29/2012 04:28 PM, Yngve N. Pettersen wrote:
>> used in production environments... ually
With the ServerNameIndication this can create problems.
The current spec provides for a list of names, some of them
may not be recognizable.

If one wants to be able to skip unknown names and still recognize
good ones, it would be dangerous to put new names in the fronnt

Openssl actually assumes that any opaque servername value
has two octets  length. and skips through that ...

Peter

---

From: Peter Sylvester <[email-masked]>
Date: Thu, 29 Mar 2012 17:02:27 +0200

> that implementations of the affected variants records will have failed 
> in similar fashions. Only an extensive and detailed review of what 
> existing implementations do will reveal what has been implemented.
Yes.

---

From: Michael D'Errico <[email-masked]>
Date: Thu, 29 Mar 2012 11:18:45 -0700

This problem was discussed several years ago w.r.t. the SNI
extension.  Please see:

http://www6.ietf.org/mail-archive/web/tls/current/msg02663.html
http://www6.ietf.org/mail-archive/web/tls/current/msg02664.html

Perhaps we need to update our notation to add a length-prefixed
variant record to be used wherever needed in the future.  That
shouldn't affect backward-compatibility since only new protocol
elements and extensions would use it.

Mike

Yngve N. Pettersen wrote:
> problem with the variant records structure in my specification (which 
> was based on the corresponding structure in RFC 6066), when I was going 
> to code the server side handling of unknown status request types.
> 
> extensions have the same problem, and that this will seriously affect 
> the future expandability by these extensions.
> 
> When the server tries to read such an unknown variant record, it will 
> not be able to tell how much to read from the vector in order to reach 
> the next record in the vector.
> 
> As an example, in the following example structure, the client knows 
> about Wha1 and the associated fields whatsths and whatstht, but the 
> server does not. They both know about all the other fields and values.
> 
> enum{
>   Foo, Bar, Wha1;
> } Typ
> 
> struct {
>   Typ rec_typ;
>   Select(rec_typ)
>   {
> If the client sends a Recs vector that includes a Wha1 variant, the 
> server will not be able to read the Rec.payload field, and will likely 
> crash or abandon the connection.
> 
> Within the current framework the only way to work around this is to add 
> a payload length field before the payload field. It is a matter of taste 
> The server will now be able to read the structure even if it does not 
> know about the Wha1 alternative, since it knows how many bytes to ignore.
> 
> Long term, it might be an idea to consider adding a variable record 
> format for the vector type in TLS, alternatively, update the "Select" 
> encoding to include a length field. These changes would have to be done 
> in new extensions, and/or a new TLS version, and would still not be 
> applied to records that will must be understood by older servers.
> 
> client side, *unless* the server can pick variants for vector records 
> that are not negotiated in advance.
> 
> 
>

---

From: Martin Rex <[email-masked]>
Date: Thu, 29 Mar 2012 20:19:54 +0200 (MEST)

Yngve N. Pettersen wrote:
> >> server
> > it is probably an early beta version that should neither be distributed  
> > nor used in production environments...
> 

No, the problem here was that the SSLv3 spec was retroactively changed,
and neither was that change clearly communicated, nor was an updated
SSLv3 spec prominently published and all prior variants of the SSLv3
spec without that backwards-incompatible change killed!

> implementations do will reveal what has been implemented.

I agree that this ambiguity must be expected to cause interop problems
(in the fashion that new name types inserted by clients may cause servers
to entirely ignore the extension although the very same server would
understand a traditional TLS extension SNI.

There is a certain likelyhood that for servers that need SNI badly
to pick the right cert, this would likely cause clients to barf on the
server picking the wrong cert when not understanding fancy names
sent by the client.

What I do not understand is (a) servers crashing and (b) servers aborting
the handshake.  It just does not make sense.

What confuses me here is that so many folks claim to have implemented
and shipped TLS extension SNI (and the RFC went through 2 revs!!),
and you're the first implementor to point out that there is a problem
with that part of the spec.

If there is an ambiguity, then it is crystal clear from the spec and
it is impossible for a sensible implementor to MISS that fact and
request clarification.  I'm disappointed.  I would really appreciate
implementors to pay more attention and point out problems that they
encounter.

The IETF mantra is "running code", and that does not include "crashing code".

> Extending the syntax might be an option, but at present I don't think it  

Where does rfc5246 say that variants (section 4.6.1) being a
variation of Constructed type (section 4.6) can not be defined as variable
length vectors (section 4.3)?  Just because there is no current
example / usage of this in the spec, I do not see anything that could
be interpreted to preclude this.

-Martin

---

From: Martin Rex <[email-masked]>
Date: Thu, 29 Mar 2012 20:35:01 +0200 (MEST)

> elements and extensions would use it.

OK, I see.

There was text added to rfc6066 recognizing the problem and suggesting
the workaround that was proposed in 2008:

   http://tools.ietf.org/html/rfc6066#page-7    2nd paragraph:

   Currently, the only server names supported are DNS hostnames;
   however, this does not imply any dependency of TLS on DNS, and other
   name types may be added in the future (by an RFC that updates this
   document).  The data structure associated with the host_name NameType
   is a variable-length vector that begins with a 16-bit length.  For
   backward compatibility, all future data structures associated with
   new NameTypes MUST begin with a 16-bit length field.  TLS MAY treat
   provided server names as opaque data and pass the names and types to
   the application.

I agree with Yngve in respect to document updates not magically
fixing the installed base.  But the suggested workaround is really
an approach that a sensible implementation should be able to cope with,
and those quick'n'dirty hacks probably have more serious problems
than this, so that they will have to be regularly updated anyway...

-Martin

---

From: "Yngve N. Pettersen" <[email-masked]>
Date: Thu, 29 Mar 2012 21:17:05 +0200

> spec without that backwards-incompatible change killed!

Most of the TLS Extension intolerant servers support TLS 1.0 as their  
highest version, not SSL v3. TLS 1.0, at least, is clear on this point.

The most common failure scenario, as I recall, is that the connection is  
closed.

>> in

Turns out I wasn't the first to discover the problem space.

> The IETF mantra is "running code", and that does not include "crashing  

"Running code" does not necessarily mean that it has been tested with  
unexpected, but valid data.

As it was, the reason I (re)discovered the issue was that I was thinking  
specifically about how to parse and ignore an item with an unknown type  
when I was implementing a prototype decoder. If I had only followed the  
structure layout and coding exactly that, I might not have thought about  
future changes to the structure.

> be interpreted to preclude this.

The variant you suggest may make sense, but at present there is AFAIK no  
example of the Select construct being used in that fashion. I'd just  
prefer to have the example in the text of the protocol specification  
first, rather than adding it in a related specification.

Even struct foo vectors  are IIRC first declared as a structure type, then  
the typename is used to declare the vector. Select constructors are not  
type definitions, they are field definitions, which may be one reason why  
your construct have not been considered.

Another reason it may not have been considered is that the <> vector  
construct is meant for *repeated* structures of the same type, not a  
single item of of a structure with variant length.

To avoid confusion I would think that such a length determined variant  
construct should have a special name and/or textual representation that is  
different from the <> and [] constructs.

-- 
Sincerely,
Yngve N. Pettersen