Subject: Re: [TLS] Fwd: Clarification on interleaving app data and handshake records

From: Matt Caswell <[email-masked]>
Date: Tue, 13 Oct 2015 10:12:45 +0100

Hello,

Does anyone have any views on the below?

Thanks

Matt

On 30/09/15 11:06, Matt Caswell wrote:
> Hi all
>
> I have a question on how to interpret RFC 5246 with regards to the
> interleaving of app data and handshake records.
>
> RFC 5246 (and RFC 4346 before it) contains these words:
>
>    Note: Data of different TLS Record layer content types MAY be
>    interleaved.  Application data is generally of lower precedence for
>    transmission than other content types.  However, records MUST be
>    delivered to the network in the same order as they are protected by
>    the record layer.  Recipients MUST receive and process interleaved
>    application layer traffic during handshakes subsequent to the first
>    one on a connection.
>
> This wording seems to place no limits whatsoever on when it is valid to
> receive app data in the handshake. By the wording in the RFC it would be
> valid for app data to be received *after* the ChangeCipherSpec has been
> received but *before* the Finished has been processed.
>
> There is also this wording:
>
>    Note: If a rehandshake occurs while data is flowing on a connection,
>    the communicating parties may continue to send data using the old
>    CipherSpec.  However, once the ChangeCipherSpec has been sent, the
>    new CipherSpec MUST be used.  The first side to send the
>    ChangeCipherSpec does not know that the other side has finished
>    computing the new keying material (e.g., if it has to perform a
>    time-consuming public key operation).  Thus, a small window of time,
>    during which the recipient must buffer the data, MAY exist.  In
>    practice, with modern machines this interval is likely to be fairly
>    short.
>
> I think this means that as soon as the first party sends a CCS, they
> must not send any app data until they have received a CCS back - they
> must buffer it until the CCS is seen - but on reading it again I'm not
> sure! If that were the case then the second party should never expect to
> see app data between CCS and Finished. It doesn't tell you anything
> about what the first party can expect though, i.e. is the second party
> allowed to send app data between the CCS and Finished?
>
> Finally there is also this:
>
>    A Finished message is always sent immediately after a change
>    cipher spec message to verify that the key exchange and
>    authentication processes were successful.
>
> I suppose this trumps everything else, although as this section is
> specifically talking about handshakes you could interpret "immediately"
> as applying to the handshake sequence only without saying anything about
> interleaved app data records.
>
> I believe the intention is that app data should not be sent between the
> CCS and Finished but the RFC wording is not exactly crystal clear. This
> is the interpretation I have taken whilst fixing this OpenSSL bug:
>
> https://rt.openssl.org/Ticket/Display.html?id=3712&user=guest&pass=guest
>
> Can some confirm my interpretation is correct?
>
> Thanks
>
> Matt
>
>

---

From: Ilari Liusvaara <[email-masked]>
Date: Tue, 13 Oct 2015 17:02:32 +0300

> Does anyone have any views on the below?

Well, my interpretation (can be totally off-base) is that application
data can be interleaved with renegotiation, except between
ChangeCipherSpec and Finished, which always appear back to back.

My reading of that buffering thing was that in some conditions the
receiver needs to buffer as it is receiving traffic it hasn't computed
the decryption keys for (as the DH/key derivation is in progress).

-Ilari

---

From: Watson Ladd <[email-masked]>
Date: Tue, 13 Oct 2015 11:01:42 -0400

> Does anyone have any views on the below?

Yes. I think we need to have a much, much tighter definition of the
state machine. This has been a perennial source of bugs and
interoperability issues in real world implementations.

---

From: [email-masked] (Martin Rex)
Date: Wed, 14 Oct 2015 17:44:01 +0200 (CEST)

> Does anyone have any views on the below?

Yup.  Interleaving application & handshake records is a
highly dangerous idea (and fortunately some TLS implementations
will abort if you try).

http://www.ietf.org/mail-archive/web/tls/current/msg07648.html

http://www.ietf.org/mail-archive/web/tls/current/msg09743.html

-Martin

> _______________________________________________
> TLS mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tls

---

From: Matt Caswell <[email-masked]>
Date: Wed, 14 Oct 2015 21:04:59 +0100

> http://www.ietf.org/mail-archive/web/tls/current/msg09743.html

So is your recommendation that we should ignore the RFC and prevent
interleaving at all? The current OpenSSL behaviour is to abort, but this
is causing an interoperability problem. Since the RFC is clear that
OpenSSL is in the wrong we were seeking to fix it.

Matt

---

From: David Benjamin <[email-masked]>
Date: Wed, 14 Oct 2015 20:29:57 +0000

> OpenSSL is in the wrong we were seeking to fix it.

BoringSSL doesn't allow application/handshake interleave under any
circumstances and further requires that renegotiate be server-initiated and
at a quiescent point in the application protocol. That is, a point where
the client is done writing and will not write more until receiving more
data from the server, like between an HTTP/1.1 request and response. This
avoids a ton of complexity and state-space explosion. (OpenSSL's
in_read_app_data = 2 and the interaction with in_handshake are terrifying.)
It's meant to be the bare minimum needed to support the legacy
renego/client-auth mistake for unpipelined HTTP/1.1 as a client and nothing
more.

It's been shipping in Chrome for a while now and I've never had a report of
breakage. But I also don't have to deal with the particular interop
situation you came across, so this approach may not work as well for
OpenSSL.

If you really absolutely must support interleave and can't avoid it, I
think it being allowed everywhere except between CCS and Finished is
probably the closest you can get to coherent semantics. "Coherent" here is,
of course, all relative since renego is involved.

David

Matt

---

From: Martin Thomson <[email-masked]>
Date: Wed, 14 Oct 2015 13:42:49 -0700

On 14 October 2015 at 13:29, David Benjamin <[email-masked]> wrote:
> If you really absolutely must support interleave and can't avoid it, I think
> it being allowed everywhere except between CCS and Finished is probably the
> closest you can get to coherent semantics. "Coherent" here is, of course,
> all relative since renego is involved.

I think that it needs to be more than that.  I would say:

1. You should not interleave data with handshake messages from the same flight.
2. You should not report any change to handshake status until the
renegotiation is complete.  That means no callbacks/events about new
peer certificates, or anything of that nature until you have received
and validated the Finished.  You would have to exercise caution here
for the callbacks that are necessary during the process, like the
"please choose a certificate and private key to use" callback on
either side.

If you can somehow manage to do all of that, then you might be able
get away with not halting progress entirely.  Because - as far as the
application is concerned - application data is sent as though it were
before the renegotiation.  In essence, you are keeping the application
ignorant of any changes until the whole process is over.

I've heard it recommended that you add other stipulations, such as

3. The server cannot change certificate.

I'm ambivalent on that because I don't believe it to materially change
the security properties (assuming that we have a good handshake, i.e.,
at least R-I and session hash).  I don't have proof of that, of
course.

---

From: David Benjamin <[email-masked]>
Date: Wed, 14 Oct 2015 21:10:14 +0000

On Wed, Oct 14, 2015 at 4:42 PM Martin Thomson <[email-masked]>
wrote:

> think

Indeed. This is, IMO, one of the core reasons renegotiation was a horrible
mistake and one we should make sure never to repeat in the future. :-) I
didn't mention it since it's more an API concern than what's over the wire,
but that certainly doesn't make it any less a problem! What renego even
means for the application data is extremely weird. A lot of TLS APIs and
implementations I've seen don't do a very good job of scoping their private
state or even public APIs properly, so it's very unclear what, say,
querying the peer certificate even means at various points mid-renego.
(Mid-initial-handshake is weird too, but most consumers are sensible enough
not to query such stuff while some abstract "connect" operation is still
going on.)

This is especially problematic because handling renegotiations
transparently seems a fairly popular implementation strategy. (Both NSS and
OpenSSL do this. OpenSSL even allows them by default and doesn't have a
good way to turn them off. Not sure about NSS. BoringSSL makes it opt-in.)

Above all, by far the best way to handle renegotiation is "never do it,
period". I think it's most important that any attempt to hammer down
semantics in the "we already got it wrong" cases should be clear on this.
New protocols should implicitly forbid renego, implementations should turn
it off by default, and that at least gives a bounded body of existing
mistakes to consider.

David

---

From: Matt Caswell <[email-masked]>
Date: Wed, 14 Oct 2015 23:43:54 +0100

> I've heard it recommended that you add other stipulations, such as

That would be very challenging to implement. Almost certainly not
possible for the stable released versions. I would hesitate to do so for
the new development version too without explicit published advice in the
form of an RFC/errata...it would add significant complexity.

It seems my options in reality are:

1) Allow interleaved data everywhere except between CCS and Finished, as
per the (hopelessly unclear) RFC. This would leave us conformant, would
solve our interoperability problems, but is a "highly dangerous idea"
according to your advice.

or

2) Leave things as they are now where we abort on interleaved
application data. This would leave us unconformant and with an
interoperability problem which is causing real issues for users (who
will point the finger at us for failing to fix it).

"Rock" meet "hard place". :-(

Matt

---

From: Martin Thomson <[email-masked]>
Date: Wed, 14 Oct 2015 16:06:00 -0700

On 14 October 2015 at 15:43, Matt Caswell <[email-masked]> wrote:
> "highly dangerous idea"

Wrong Martin.  I agree that there is a need for caution, but in
reality, it's not like you can use renegotiation to hand-off to
someone else entirely.  The person you are talking to hasn't changed.
What is dangerous is making assertions about *new* things that the
renegotiation introduces.

---

From: Matt Caswell <[email-masked]>
Date: Thu, 15 Oct 2015 09:35:07 +0100

On 15/10/15 00:04, Watson Ladd wrote:
> On Wed, Oct 14, 2015 at 6:43 PM, Matt Caswell <[email-masked]> wrote:
>>
>>
> It takes two sides here to have an interop problem. What
> implementations are doing this?

The specific report I have is for Oracle JRE 1.7.0_71 and 1.7.0_75 but
my assumption is that this affects all Oracle JRE versions.

The problem ticket in question describes a scenario where a PostgreSQL
server (which uses OpenSSL) is talking to a PostgreSQL JDBC client. Due
to that fact that these connections are long running, PostgreSQL server
has a config parameter "ssl_renegotiation_limit" which triggers a
renegotiation after a certain amount of data has been transferred over
the connection.

Matt

---

From: Matt Caswell <[email-masked]>
Date: Thu, 15 Oct 2015 09:46:38 +0100

On 15/10/15 00:06, Martin Thomson wrote:
> Wrong Martin.

Oops. Sorry.

>  I agree that there is a need for caution, but in

OpenSSL will not itself use anything from the new handshake until the
CCS/Finished has been processed. I couldn't make the same guarantee
about applications using OpenSSL APIs/callbacks (although no doubt
applications could do all manner of dangerous things if they chose to).
So that leaves me unclear whether you are advising me to "fix" the "bug"
because its not that dangerous really, or whether the risk of
applications using info they shouldn't is too great so we should
maintain the status quo and leave things as they are (i.e. broken in
certain scenarios).

Matt

---

From: [email-masked] (Martin Rex)
Date: Thu, 15 Oct 2015 15:00:40 +0200 (CEST)

Is the particular interop problem that you want to address
caused by a necessity to really process application data and
handshake data with arbitrary interleave,

or is it rather a problem of getting back into half-duplex operation,
i.e. a client being able to continue receiving application data
up to a ServerHello when it has sent out ClientHello, or a server being
able to continue receiving application data up to a ClientHello
(or warning level no-renegotiation alert) after the server has sent
a ClientHelloRequest?

-Martin

---

From: Matt Caswell <[email-masked]>
Date: Thu, 15 Oct 2015 14:12:12 +0100

> (or warning level no-renegotiation alert) after the server has sent

The former. The existing code should cope with the half-duplex issue. In
the reported problem we (OpenSSL) are running as a server and we have
received application data from the Client *after* we have sent our
ServerHelloDone.

Matt

---

From: Hubert Kario <[email-masked]>
Date: Fri, 16 Oct 2015 11:50:30 +0200

> renegotiation introduces.

Also, we're talking with a peer that does implement RFC 5746, so we can 
be *sure* that we're talking to the same peer still.

So the problem happens when application is querying the library for 
connection information (certificates mainly) and getting info from new 
connection while still actually receiving application data from the old 
context.

The problem is, that we can verify the handshake only after we receive 
Finished message, until then, the server can present any certificate it 
wants and client has no way of verifying if it (for *DH it can even 
receive information sent by client after its Finished message). For 
server it's nicer, as the certificate can be verified much quicker (in 
the same flight), but the window still exists.

That makes it dangerous when going from low to high security context, 
not so much other way round.
-- 
Regards,
Hubert Kario
Quality Engineer, QE BaseOS Security team
Web: www.cz.redhat.com
Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno, Czech Republic

---

From: Watson Ladd <[email-masked]>
Date: Fri, 16 Oct 2015 09:16:01 -0400

> received application data from the Client *after* we have sent our

After thinking about this a bit this should be okay so long as you
properly present the authentication state associated with the data.
The hypothetical problem is using this to evade the protection of the
secure renegotiation extension. As a solution the new authentication
state should only be made visible to application code after receiving
a CSS/Finished. This is supposed to have exactly the same semantics as
pretending that the application data was sent before any handshake
data.

Unfortunately I don't know how to verify this. Can miTLS cover this case?

-- 
"Man is born free, but everywhere he is in chains".
--Rousseau.

---

From: Hubert Kario <[email-masked]>
Date: Fri, 16 Oct 2015 16:05:34 +0200

On Friday 16 October 2015 09:16:01 Watson Ladd wrote:
> >> up to a ServerHello when it has sent out ClientHello, or a server
> >> being able to continue receiving application data up to a
> >> ClientHello (or warning level no-renegotiation alert) after the
> >> server has sent a ClientHelloRequest?
> > 
> > The former. The existing code should cope with the half-duplex
> > issue. In the reported problem we (OpenSSL) are running as a server
> > and we have received application data from the Client *after* we
> > have sent our ServerHelloDone.
> 

you mean, you want an implementation that can insert application data in 
any place of the handshake?

we've been using my project for that: 
https://github.com/tomato42/tlsfuzzer

the specific test cases are:

https://github.com/tomato42/tlsfuzzer/blob/master/scripts/test-interleaved-application-data-and-fragmented-handshakes-in-renegotiation.py

https://github.com/tomato42/tlsfuzzer/blob/master/scripts/test-interleaved-application-data-in-renegotiation.py

https://github.com/tomato42/tlsfuzzer/blob/master/scripts/test-openssl-3712.py

you can run them by:

pip install tlslite-ng
git clone https://github.com/tomato42/tlsfuzzer.git
cd tlsfuzzer
PYTHONPATH=. python scripts/test-openssl-3712.py

(they do expect a HTTP server on the other side)
-- 
Regards,
Hubert Kario
Quality Engineer, QE BaseOS Security team
Web: www.cz.redhat.com
Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno, Czech Republic

---

From: "Short, Todd" <[email-masked]>
Date: Fri, 16 Oct 2015 19:11:04 +0000

Hi Matt:

I agree with your interpretation, in that there should be no records of any type between the CSS and the Handshake/Finished message, even during re-handshake. The full handshake (and subsequent key material) cannot be validated until the Handshake/Finished messages has been received and processed.

Any AppData received between a CSS and Handshake/Finished is potentially malicious data trying to sneak in before the Handshake is verified; forwarding this data to an upper layer would be potentially harmful. A MITM would hope that the data would be processed by an application before the TLS layer receives and verifies the Handshake/Finished message.

--
-Todd Short
// [email-masked]<mailto:[email-masked]>
// "One if by land, two if by sea, three if by the Internet."

On Oct 13, 2015, at 5:12 AM, Matt Caswell <[email-masked]<mailto:[email-masked]>> wrote:

Hello,

Thanks

Matt

Thanks

Matt

_______________________________________________
TLS mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/tls

---

From: Kurt Roeckx <[email-masked]>
Date: Fri, 16 Oct 2015 22:36:10 +0200

> any place of the handshake?

Have you tried running any of your tests against miTLS?

Kurt