Subject: [pkix] Clarification on "zero" hash value in SigPolicyHash (CAdES)

From: Niklas Matthies <[email-masked]>
Date: Fri, 12 Jul 2019 22:05:49 +0200

Dear all,

I hope that someone can shed some light on the definition and origin 
of the special-case "zero" hash value for SigPolicyHash in CAdES. The 
main question is what exactly constitutes a "zero" hash value on the 
ASN.1 level.

In the most current CAdES specification, ETSI EN 319 122-1 V1.1.1 
(2016-04), the zero hash value is described as follows (clause 
5.2.9.1, page 20): 
    
    The sigPolicyHash field shall contain the identifier of the hash 
    algorithm and the hash of the value of the signature policy.
    The hashValue within the sigPolicyHash may be set to zero to 
    indicate that the policy hash value is not known.

        NOTE: The use of a zero-sigPolicyHash value is to ensure 
        backwards compatibility with earlier versions of ETSI TS 101 733 [1].

The ASN.1 definition of SigPolicyHash is as follows:

    SignaturePolicyId ::= SEQUENCE {
        sigPolicyId SigPolicyId,
        sigPolicyHash SigPolicyHash,
        sigPolicyQualifiers SEQUENCE SIZE (1..MAX) OF 
            SigPolicyQualifierInfo OPTIONAL
    }

    SigPolicyHash ::= OtherHashAlgAndValue

    OtherHashAlgAndValue ::= SEQUENCE {
        hashAlgorithm   AlgorithmIdentifier,
        hashValue       OtherHashValue }

    OtherHashValue ::= OCTET STRING

The question is what constitutes a "zero" hash value. A zero-length 
octet string? Any octet string where all octets are zero? Is the 
length of the octet string relevant, e.g. can or should it match the 
length implied by the hash algorithm?

I spent some time going back the history of the standards to find the 
"earlier version" relevant for the note about backwards compatibility, 
however that search was inconclusive.

All versions of ETSI TS 101 733 from the latest V2.2.1 (2013-04) back 
to V1.7.4 (2008-07) include the following note, which had a second 
sentence added:

        NOTE: The use of a zero-sigPolicyHash value is to ensure 
        backwards compatibility with earlier versions of the
        current document. If sigPolicyHash is zero, then the hash 
        value should not be checked against the calculated hash value 
        of the signature policy.

Confusingly, this now talks about sigPolicyHash being zero (whatever that 
would mean), not about the hashValue field within sigPolicyHash.

The same text appears in RFC 5126 (February 2008).

The previous version ETSI TS 101 733 V1.7.3 (2007-01) is the first 
version to talk about a zero hash value, and the note is just one 
sentence again:

    The hashValue within the sigPolicyHash max [sic] be set to zero to 
    indicate that the policy hash value is not known.

        NOTE: The use of zero policy hash value is to ensure backward 
        compatibility with earlier versions of the current document.

In addition, the annex states the following:

    Annex K (informative): Changes from the previous version

        • if the hash of the signature policy is unknown, then, by 
        convention, the sigPolicyHash shall be set to all zero

Here again, the annex talks about sigPolicyHash being set to zero, 
not the hashValue within sigPolicyHash.

On the RFC side, the same note first appears in 
https://tools.ietf.org/html/draft-ietf-smime-cades-02 (May 2007), 
and the sentence above (without the note) first appears in 
https://tools.ietf.org/html/draft-ietf-smime-cades-01 (April 2006).

In the previous ETSI TS 101 733 V1.6.3 (2005-09), the sigPolicyHash 
field is optional (which is mandatory in all previous and all 
subsequent versions), and the following note is included:

        NOTE: In the previous version of TS 101 733 (i.e. version 1.5.1) 
        sigPolicyHash was mandatory. Implementations requiring to be 
        backward compatible with version 1.5.1 and previous versions 
        of the current document MUST include SigPolicyHash.

None of the previous versions, from ETSI TS 101 733 V1.5.1 (2003-12) 
back to ETSI TS 101 733 V1.2.2 (2000-12) (the earliest version available 
at https://www.etsi.org/deliver/etsi_ts/101700_101799/101733/), as 
well as RFC 3126 (September 2001), do mention anything about zero hash 
values.

So it remains unclear to which specification backwards compatiblity is 
intended. The point where the wording about zero hash values was 
introduced suggests that this may be about V1.6.3, where the
sigPolicyHash field was optional. However, it's not clear how a "zero" 
hash value would achieve compatibility with the case of a missing 
sigPolicyHash field.

I would appreciate any clarification on that topic.

Kind regards,
Niklas

---

From: Stefan Santesson <[email-masked]>
Date: Mon, 15 Jul 2019 10:21:51 +0200

Hi Niklas,

A very interesting question to which I have absolutely no answer. But I have forwarded your question to the ESI group in ETSI who wrote that standard as I'm still a member of that group.
I'll forward any answer I can get out of this group

Stefan Santesson 

On 2019-07-12, 22:07, "pkix on behalf of Niklas Matthies" <[email-masked] on behalf of [email-masked]> wrote:

    Dear all,

        

    

    

    

    

    

    

    

    

    Kind regards,
    Niklas
    
    _______________________________________________
    pkix mailing list
    [email-masked]
    https://www.ietf.org/mailman/listinfo/pkix

---

From: Stefan Santesson <[email-masked]>
Date: Tue, 16 Jul 2019 10:43:28 +0200

Hi Niklas,

I posted your question on the ETSI ESI list and it created some very interesting discussions.

Very briefly there are 3 possible interpretations:

1) Empty. (The value is a empty OCTET STRING of zero bytes)
2) Zero. (The value is the byte 0x00)
3) All zeros. (The value is an OCTET STRING of the intended hash value length containing only 0x00 bytes).

There are actually some evidence that suggests that the original intent was option 3.
However, option 3 is the absolute worst from an implementation perspective as it is the hardest to programmatically distinguish from a real hash value.

My conclusion is that the standard is so ambiguous that any receiving implementation should be able to handle all 3 alternatives.
The standard need to be clarified by ETSI. Until that is done my proposal for sending applications is to use option 1 (empty) because:

1) It is the only option that makes sense and it can be expected as the most probable future resolution
2) The standard is so ambiguous that validators anyway must expect all options.
3) This feature is probably not widely implemented. Doing it one way or the other probably has very little impact on interop.

I'll keep you posted if any opinion emerges that materially contradicts this conclusion.

Stefan Santesson 

    Dear all,

        

    

    

    

    

    

    

    

    

    Kind regards,
    Niklas

---

From: Peter Gutmann <[email-masked]>
Date: Tue, 16 Jul 2019 09:32:15 +0000

Stefan Santesson <[email-masked]> writes:

>However, option 3 is the absolute worst from an implementation perspective as
>it is the hardest to programmatically distinguish from a real hash value.

I would say it's the best from an implementation perspective, you don't need
to make any code changes, it's a normal looking hash value that's guaranteed
not to match anything.  Existing implementations that expect a hash there will
continue to work as normal.

>My conclusion is that the standard is so ambiguous that any receiving
>implementation should be able to handle all 3 alternatives.

A slightly different interpretation is that since that part of the standard is
essentially unimplementable, it's likely no-one has ever implemented it, so it
can be safely dropped.  Given earlier evidence that it's only there for
backwards compatibility with something no-one can identify, this enhances the
case for dropping it from the standard.

Peter.

---

From: Stefan Santesson <[email-masked]>
Date: Tue, 16 Jul 2019 15:29:51 +0200

To assign a specific meaning to one out of all possible but syntactically valid hash values, is exactly the type of specification work that leads to implementation errors and security vulnerabilities.

Stefan Santesson 

On 2019-07-16, 11:32, "Peter Gutmann" <[email-masked]> wrote:

    

    
    Peter.

---

From: Niklas Matthies <[email-masked]>
Date: Tue, 16 Jul 2019 15:42:15 +0200

Hi Stefano,

Thanks for the information.

I agree that option 3 is somewhat problematic. All-bits-zero can 
conceivably be a valid hash value (I believe for SHA-1 and SHA-2 it's 
actually unknown whether all-bits-zero is reachable), and the standard 
probably shouldn't start disallowing all-bits-zero as a valid hash 
value for all hash algorithms.

I also agree with Peter Gutmann that dropping the feature altogether 
should be a strong consideration, in particular if the original 
motivation can't be identified, and also given that XAdES does not 
have an equivalent feature. Implementations would still be free to 
support older versions of the standard (with an interpretation of 
their choosing) for backwards compatibility, depending on local policy 
and configuration.

Just FYI, I'm maintaining an implementation that (only) supports 
option 2, however it is probably only interoperable with itself 
reagrding that aspect.

Niklas

---

From: Niklas Matthies <[email-masked]>
Date: Tue, 16 Jul 2019 16:58:08 +0200

It now occurred to me that sigPolicyHash field being optional in ETSI 
TS 101 733 V1.6.3 (2005-09) probably was discovered to be a mistake 
because (1) it introduces a parsing ambiguity with 
sigPolicyQualifiers, as both are optional SEQUENCEs, and (2) because 
it broke compatibility with existing implementation.

So it would seem that the zero hash value was introduced as a 
workaround to still be able to have an optional hash, syntactically 
masquerading as a real hash, and thus maintaining compatibility with 
the earlier versions at least on the ASN.1 level. Then the note about 
backwards compatibility would not be referring to the support of 
"zero" hash values, but to the fact that keeping the sigPolicyHash 
field mandatory for backwards compatibility reasons leaves no other 
choice than to use a "compatible" encoding for a missing hash value.

This interpretation would favor option 3 (using the length implied by 
the hash algorithm), as this arguably causes the least disruption for 
pre-V1.6.3 (and RFC-3126-based) implementations.

Niklas

On Fri 2019-07-12 at 22:05h, Niklas Matthies wrote on pkix:
>5.2.9.1, page 20):        The sigPolicyHash field shall contain the 
>identifier of the hash    algorithm and the hash of the value of the 
>signature policy.
>Confusingly, this now talks about sigPolicyHash being zero (whatever 
>that would mean), not about the hashValue field within sigPolicyHash.
>
>Here again, the annex talks about sigPolicyHash being set to zero, not 
>the hashValue within sigPolicyHash.
>
>https://tools.ietf.org/html/draft-ietf-smime-cades-02 (May 2007), and 
>the sentence above (without the note) first appears in 
>       NOTE: In the previous version of TS 101 733 (i.e. version 
>1.5.1)        sigPolicyHash was mandatory. Implementations requiring 
>to be        backward compatible with version 1.5.1 and previous 
>versions        of the current document MUST include SigPolicyHash.
>
>back to ETSI TS 101 733 V1.2.2 (2000-12) (the earliest version 
>available at 
>https://www.etsi.org/deliver/etsi_ts/101700_101799/101733/), as well 
>as RFC 3126 (September 2001), do mention anything about zero hash 
>values.
>

---

From: Andrea Caccia <[email-masked]>
Date: Tue, 16 Jul 2019 17:21:33 +0200

Dear Niklas and all,
option 3 seems the original intention, also looking at the "shall be set to all zero" in Annex K. 
Being the correct interpretation of the text does not imply also to be the best option of course...

Regards,
Andrea

> Il giorno 16 lug 2019, alle ore 16:58, Niklas Matthies <[email-masked]> ha scritto:
> 
> It now occurred to me that sigPolicyHash field being optional in ETSI TS 101 733 V1.6.3 (2005-09) probably was discovered to be a mistake because (1) it introduces a parsing ambiguity with sigPolicyQualifiers, as both are optional SEQUENCEs, and (2) because it broke compatibility with existing implementation.
> 
> So it would seem that the zero hash value was introduced as a workaround to still be able to have an optional hash, syntactically masquerading as a real hash, and thus maintaining compatibility with the earlier versions at least on the ASN.1 level. Then the note about backwards compatibility would not be referring to the support of "zero" hash values, but to the fact that keeping the sigPolicyHash field mandatory for backwards compatibility reasons leaves no other choice than to use a "compatible" encoding for a missing hash value.
> 
> This interpretation would favor option 3 (using the length implied by the hash algorithm), as this arguably causes the least disruption for pre-V1.6.3 (and RFC-3126-based) implementations.
> 
> Niklas
> 
> 
>> I hope that someone can shed some light on the definition and origin of the special-case "zero" hash value for SigPolicyHash in CAdES. The main question is what exactly constitutes a "zero" hash value on the ASN.1 level.
>> 
>> In the most current CAdES specification, ETSI EN 319 122-1 V1.1.1 (2016-04), the zero hash value is described as follows (clause 5.2.9.1, page 20):        The sigPolicyHash field shall contain the identifier of the hash    algorithm and the hash of the value of the signature policy.
>>  The hashValue within the sigPolicyHash may be set to zero to    indicate that the policy hash value is not known.
>> 
>>      NOTE: The use of a zero-sigPolicyHash value is to ensure        backwards compatibility with earlier versions of ETSI TS 101 733 [1].
>> 
>>      sigPolicyQualifiers SEQUENCE SIZE (1..MAX) OF            SigPolicyQualifierInfo OPTIONAL
>>  }
>> 
>> The question is what constitutes a "zero" hash value. A zero-length octet string? Any octet string where all octets are zero? Is the length of the octet string relevant, e.g. can or should it match the length implied by the hash algorithm?
>> 
>> I spent some time going back the history of the standards to find the "earlier version" relevant for the note about backwards compatibility, however that search was inconclusive.
>> 
>> All versions of ETSI TS 101 733 from the latest V2.2.1 (2013-04) back to V1.7.4 (2008-07) include the following note, which had a second sentence added:
>> 
>>      NOTE: The use of a zero-sigPolicyHash value is to ensure        backwards compatibility with earlier versions of the
>>      current document. If sigPolicyHash is zero, then the hash        value should not be checked against the calculated hash value        of the signature policy.
>> 
>> Confusingly, this now talks about sigPolicyHash being zero (whatever that would mean), not about the hashValue field within sigPolicyHash.
>> 
>> The previous version ETSI TS 101 733 V1.7.3 (2007-01) is the first version to talk about a zero hash value, and the note is just one sentence again:
>> 
>>  The hashValue within the sigPolicyHash max [sic] be set to zero to    indicate that the policy hash value is not known.
>> 
>>      NOTE: The use of zero policy hash value is to ensure backward        compatibility with earlier versions of the current document.
>> 
>>      • if the hash of the signature policy is unknown, then, by        convention, the sigPolicyHash shall be set to all zero
>> 
>> Here again, the annex talks about sigPolicyHash being set to zero, not the hashValue within sigPolicyHash.
>> 
>> On the RFC side, the same note first appears in https://tools.ietf.org/html/draft-ietf-smime-cades-02 (May 2007), and the sentence above (without the note) first appears in https://tools.ietf.org/html/draft-ietf-smime-cades-01 (April 2006).
>> 
>> In the previous ETSI TS 101 733 V1.6.3 (2005-09), the sigPolicyHash field is optional (which is mandatory in all previous and all subsequent versions), and the following note is included:
>> 
>>      NOTE: In the previous version of TS 101 733 (i.e. version 1.5.1)        sigPolicyHash was mandatory. Implementations requiring to be        backward compatible with version 1.5.1 and previous versions        of the current document MUST include SigPolicyHash.
>> 
>> None of the previous versions, from ETSI TS 101 733 V1.5.1 (2003-12) back to ETSI TS 101 733 V1.2.2 (2000-12) (the earliest version available at https://www.etsi.org/deliver/etsi_ts/101700_101799/101733/), as well as RFC 3126 (September 2001), do mention anything about zero hash values.
>> 
>> So it remains unclear to which specification backwards compatiblity is intended. The point where the wording about zero hash values was introduced suggests that this may be about V1.6.3, where the
>> sigPolicyHash field was optional. However, it's not clear how a "zero" hash value would achieve compatibility with the case of a missing sigPolicyHash field.
>>

---

From: Peter Gutmann <[email-masked]>
Date: Wed, 17 Jul 2019 02:57:08 +0000

Niklas Matthies <[email-masked]> writes:

>All-bits-zero can conceivably be a valid hash value

If someone has a means of reaching not even all-zero-bits but just a
significant number of leading zero bits for a hash, please contact me off-
list, I have a very lucrative Bitcoin-based business proposition for you.

Peter.

---

From: Peter Gutmann <[email-masked]>
Date: Wed, 17 Jul 2019 03:28:10 +0000

Stefan Santesson <[email-masked]> writes:

>To assign a specific meaning to one out of all possible but syntactically
>valid hash values, is exactly the type of specification work that leads to
>implementation errors and security vulnerabilities.

How would it lead to implementation errors and security vulns?  It's a value
that's guaranteed to not match anything while requiring zero changes to
existing code, what sort of implementation error would it lead to?

Peter.

---

From: Ernst G Giessmann <[email-masked]>
Date: Wed, 17 Jul 2019 08:33:19 +0200

Stefan,
what about the implementation of EC-RDSA (RFC 7091), where exactly this
"one out of all" is required:
https://tools.ietf.org/html/rfc7091#section-6.1

-----
Step 2.
...
If e = 0, then assign e = 1.
-----

;-)

/Ernst.

Note, that in this case it is crystal clear said what to do, which is
missing in the TS.

Am 2019-07-16 um 15:29 schrieb Stefan Santesson:
> To assign a specific meaning to one out of all possible but
> syntactically valid hash values, is exactly the type of specification
> work that leads to implementation errors and security vulnerabilities.
> Stefan Santesson 
...

---

From: Stefan Santesson <[email-masked]>
Date: Wed, 17 Jul 2019 09:46:28 +0200

The spec says: Don't match hash to a real document if the hash value is A, but ignore matching for any hash values other than A.
(A = all zero in this case).

That (quite obviously) may lead to security vulnerabilities if matching is important for security and implementers don't get it right.
Example: An implementer gets this wrong and only test if the first byte of the hash (he forgot to increment the index counter). I can then cause this implementation to skip matching if I produce a hash with first byte set to 0x00.

All I'm saying is that a safer design is to say "Don't match if value is absent". Implementers are more likely to get that right, but there are no guarantees.

Stefan Santesson 

On 2019-07-17, 05:28, "Peter Gutmann" <[email-masked]> wrote:

    
    
    Peter.

---

From: Stefan Santesson <[email-masked]>
Date: Wed, 17 Jul 2019 09:53:39 +0200

Niklas,

Thanks for your analysis, which quite possibly is very true.

However, I would just add that an empty OCTET STRING value would also be "syntactically correct" on the ASN.1 level.

Stefan Santesson 

On 2019-07-16, 16:58, "pkix on behalf of Niklas Matthies" <[email-masked] on behalf of [email-masked]> wrote:

    
    
    Niklas