Subject: [TLS] TLS 1.2 - is it allowed to strip the leading zero byte(s) in RSA signature in ServerKeyExchange?

From: M K Saravanan <[email-masked]>
Date: Wed, 12 Feb 2020 14:26:47 +0800

Hi,

I recently encountered the below issue:

TLS1.2
ECDHE_RSA
server certificate: 2048-bit RSA (= 256 bytes)
ServerKeyExchange hash/sign algorithm: rsa_pkcs1_sha1

The server was sending the ServerKeyExchange with 255 byte as length for
the RSA signature (i.e. the leading zero was stripped) instead of 256 like
this:

====================
Handshake Protocol: Server Key Exchange
    Handshake Type: Server Key Exchange (12)
    Length: 328
    EC Diffie-Hellman Server Params
        Curve Type: named_curve (0x03)
        Named Curve: secp256r1 (0x0017)
        Pubkey Length: 65
        Pubkey: 042206562efea8bd47bf014a9e650c42f27078643c553671…
        Signature Algorithm: rsa_pkcs1_sha1 (0x0201)
        Signature Length: 255
        Signature: d1bf915eca2ec0bcdda6f90a398fe5378d2028a22574d213…
====================

Is this allowed?  i.e. stripping the leading zero of the RSA signature and
marking the length as 255?   It is not clear to me from the RFC5246 whether
it is allowed or not.

(client was failing to verify the signature due to this).

with regards,
Saravanan

---

From: David Benjamin <[email-masked]>
Date: Wed, 12 Feb 2020 01:53:29 -0500

The signature is invalid. The client is correct to reject it, and the
server is incorrect to produce it.

RFC5246 cites PKCS1 (then RFC3447, now RFC8017). Both versions spell out
the signing and verifying operations explicitly. The signing operation must
produce a fixed-width output and the verification operation must reject
incorrectly-sized inputs:
https://tools.ietf.org/html/rfc3447#section-8.2.1
https://tools.ietf.org/html/rfc3447#section-8.2.2
https://tools.ietf.org/html/rfc8017#section-8.2.1
https://tools.ietf.org/html/rfc8017#section-8.2.2

On Wed, Feb 12, 2020 at 1:27 AM M K Saravanan <[email-masked]> wrote:

> _______________________________________________
> TLS mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tls
>

---

From: M K Saravanan <[email-masked]>
Date: Wed, 12 Feb 2020 15:13:54 +0800

Thanks David for the clarification.

with regards,
Saravanan

On Wed, 12 Feb 2020 at 14:53, David Benjamin <[email-masked]> wrote:

>> Is this allowed?  i.e. stripping the leading zero of the RSA signature
>> and marking the length as 255?   It is not clear to me from the RFC5246
>> whether it is allowed or not.
>>

---

From: Peter Gutmann <[email-masked]>
Date: Wed, 12 Feb 2020 16:10:00 +0000

>marking the length as 255?   It is not clear to me from the RFC5246 whether

It's not allowed according to the spec but a number of implementations do it
because their underlying bignum libraries perform leading-zero truncation, so
you're better off allowing it to avoid breakage.

Peter.

---

From: David Benjamin <[email-masked]>
Date: Wed, 12 Feb 2020 12:12:05 -0500

> you're better off allowing it to avoid breakage.

For web use cases, this does not appear to be necessary. BoringSSL and
Chrome do not accept such signatures and have not for around five years
now. (Possibly longer. I do not know off-hand what Chrome's behavior was
when it used NSS.) I don't think I've ever seen a report of problems with a
website, and the specification quite clearly says to reject those
signatures. The robustness principle sounds plausible at face value, but I
think we now have the experience to know otherwise.
https://tools.ietf.org/html/draft-iab-protocol-maintenance-04

Note that bignum libraries that perform leading-zero truncation are
unlikely to be suitable for cryptography anyway. Signatures are public
values, but if you're implementing RSA decryption and care about side
channel attacks, fixed-width in-memory representations and serialization
functions are mandatory.
http://archiv.infsec.ethz.ch/education/fs08/secsem/Manger01.pdf

---

From: Peter Gutmann <[email-masked]>
Date: Thu, 13 Feb 2020 15:42:50 +0000

David Benjamin <[email-masked]> writes:

>Note that bignum libraries that perform leading-zero truncation

like OpenSSL, and some PKCS #11 libraries

>are unlikely to be suitable for cryptography anyway.

OK, I'll keep that in mind: OpenSSL and PKCS #11 are unlikely to be suitable
for cryptography :-).

Peter.