Subject: [Sidrops] what to do when the CRL is hosed?

From: Job Snijders <[email-masked]>
Date: Mon, 24 Feb 2020 15:15:32 +0000

Hi group,

It seems we need guidance and consensus on what to do when the CRL is
hosed in some way or shape. We have two implementation discrepancies pop
up recently:

https://github.com/NLnetLabs/routinator/issues/274
RIPE NCC's top level CRL expired this weekend
(https://www.ripe.net/support/service-announcements/rpki-infrastructure-issues) 

https://lists.nlnetlabs.nl/pipermail/rpki/2019-December/000109.html

OpenBSD's rpki-client uses the x509 certificate validation functions
that come from libressl, which doesn't have a button to turn off only CRL
timestamp verification. I was told that some nasty code would be
required to work around that, so one can argue that rolling things by
hand in X509 handling rarely is a great idea.

One could also argue that a softer landing is needed, unavailability of
the CRL should mean that only the CRL itself is not available and
proceed to validate the tree without the revocation list. I can see how
that is helpful in some circumstances.

So, what to do? Whatever it is, ideally all validators follow a similar
process.

Kind regards,

Job

---

From: Job Snijders <[email-masked]>
Date: Mon, 24 Feb 2020 20:55:21 +0000

Thanks claudio

(forwarding this back to the list, seems claudio's mail for some reason
didn't show up in the archives)

On Mon, Feb 24, 2020 at 04:44:49PM +0100, Claudio Jeker wrote:
> On Mon, Feb 24, 2020 at 03:15:32PM +0000, Job Snijders wrote:
> > Hi group,
> > 
> The process at the moment is clear: the embedded X509 certificates
> reference a CRL and so the validator has to check if that certificate was
> revoked via the CRL. This verification needs to follow the procedure of
> X509 certificate validation (which includes timestamp verification).
> If you want a soft landing then please just remove the CRL from the
> system.
> 
> In general it is important that the timestamps in the various RPKI objects
> are checked during verification. Not doing so will put you at risk for
> replay attacks.
> 
> -- 
> :wq Claudio

---

From: Job Snijders <[email-masked]>
Date: Mon, 24 Feb 2020 21:15:31 +0000

To reply myself: 

In case the CRL expired, or is otherwise somehow invalid; and the cache
validator continues to consider the entire validation tree to be valid
for some purpose, the validator is vulnerable to replay attacks because
the use of unencrypted transport such as rsync (which supported in all
validators, by mandate) give no support to trust anything anymore.  I
think ignoring the CRLs state goes against the entire RPKI trust-model
mindset.

A cache validator MUST consider the certificate have been appeared on
the Certificate Revocation List (CRL) issued by the CA represented by
certificate if the CRL is expired.

While one can argue in different contexts of the application of X509
technology different (more forgiving) policies can apply; in the use
case of RPKI I think we cannot tolerate anything less than to assume the
CA has failed when the CRL is inaccessible or expired. And those running
CA's will want to take careful note of this critical operational aspect.

Cache validator implementations which didn't stop parsing RIPE NCC's
tree today, should be aware they are have a security issue and consider
how to upgrade their validation strategy. I think OpenBSD's rpki-client
was the only one to get it right today.

Of course - in making strong statements like this one I can not afford
to assume I am right, so if you disagree - please tell me how I am wrong
(in detail :-) ).

Kind regards,

Job

---

From: Jared Mauch <[email-masked]>
Date: Mon, 24 Feb 2020 17:19:33 -0500

I think you may be right in absolute terms. You are likely wrong that unless you have cached CRL experience saying it's bad you have no reason to distrust. 

Also cryptographically signed data can be sent insecurely and be validated against change or tampering. Perhaps you are worried about privacy?

Sent from my iCar

> On Feb 24, 2020, at 4:15 PM, Job Snijders <[email-masked]> wrote:
>

---

From: Francisco Javier Moreno Arana <[email-masked]>
Date: Mon, 24 Feb 2020 23:29:18 +0000

Hi there,

We had the same discussion (FORT validator dev team) a few weeks ago, since the RFCs don't clarify the subject. Gladly, after reading this thread, we can say that FORT is currently rejecting those CRLs.

We've noticed the RIPE expired CRL and ~70k prefixes were rejected at that moment.

Regards,
Francisco Moreno
________________________________
De: Sidrops <[email-masked]> en nombre de Job Snijders <[email-masked]>
Enviado: lunes, 24 de febrero de 2020 03:15 p. m.
Para: [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
Asunto: Re: [Sidrops] what to do when the CRL is hosed?

To reply myself:

Kind regards,

Job

On Mon, Feb 24, 2020 at 03:15:32PM +0000, Job Snijders wrote:
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNLnetLabs%2Froutinator%2Fissues%2F274&amp;data=02%7C01%7Cfjarana%40nic.mx%7C18b6558503814064281c08d7b96eb91c%7Cc65a3ea60f7c400b89345a6dc1705645%7C0%7C0%7C637181757516231674&amp;sdata=M%2BMTjX2M%2BYYM3bsSr%2F%2BgonBy6IKSAqaGU%2F4J%2BqlnZ%2BU%3D&amp;reserved=0
> (https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ripe.net%2Fsupport%2Fservice-announcements%2Frpki-infrastructure-issues&amp;data=02%7C01%7Cfjarana%40nic.mx%7C18b6558503814064281c08d7b96eb91c%7Cc65a3ea60f7c400b89345a6dc1705645%7C0%7C0%7C637181757516241663&amp;sdata=s3%2F6i%2BhO4bnqncsSj56ZpXDsulvgWUYA13Nm5%2BmnQiM%3D&amp;reserved=0)
>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.nlnetlabs.nl%2Fpipermail%2Frpki%2F2019-December%2F000109.html&amp;data=02%7C01%7Cfjarana%40nic.mx%7C18b6558503814064281c08d7b96eb91c%7Cc65a3ea60f7c400b89345a6dc1705645%7C0%7C0%7C637181757516241663&amp;sdata=8AHf2i8DNdTjlU6TnGkbnkXK%2BLyIP%2FN3bnr0GlSZ%2B3E%3D&amp;reserved=0
>

_______________________________________________
Sidrops mailing list
[email-masked]
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fsidrops&amp;data=02%7C01%7Cfjarana%40nic.mx%7C18b6558503814064281c08d7b96eb91c%7Cc65a3ea60f7c400b89345a6dc1705645%7C0%7C0%7C637181757516241663&amp;sdata=X8IkntXgxnt345fIIKgQEBc%2FpGuLzjtyiwa59Bf2VMM%3D&amp;reserved=0

Este mensaje contiene informaci?n confidencial y se entiende dirigido y para uso exclusivo del destinatario. Si recibes este mensaje y no eres el destinatario por favor elim?nalo, ya que difundir, revelar, copiar o tomar cualquier acci?n basada en el contenido est? estrictamente prohibido. Network Information Center, S.A. de C.V., ubicado en Ave. Eugenio Garza Sada 427 L4-6 Col. Altavista, Monterrey, M?xico, C.P. 64840 recaba tus datos personales necesarios para: la prestaci?n, estudio, an?lisis y mejora del servicio, la realizaci?n de comunicaciones y notificaciones; la transferencia y publicaci?n en los casos aplicables; el cumplimiento de la relaci?n existente; as? como para la prevenci?n o denuncia en la comisi?n de il?citos. Si eres colaborador o candidato a colaborador de NIC M?xico, tus datos ser?n utilizados para: la creaci?n y administraci?n de tu perfil como profesionista; el otorgamiento de herramientas de trabajo; la realizaci?n de estudios; el otorgamiento de programas y beneficios para mejorar tu desarrollo profesional; la gesti?n y administraci?n de servicios de pago y/o n?mina; as? como para contacto y/o notificaciones. Si participas en promociones o en estudios podr?s dejar de participar. Para mayor informaci?n revisa el Aviso de Privacidad<http://www.nic.mx/es/NicMx.AvisosDePrivacidad>.

This message contains confidential information and is intended only for the individual named. If you are not the named addressee please delete it, since the dissemination, distribuition, copy or taking any action in reliance on the contents is strictly prohibited. Network Information Center, S.A. de C.V., located on Av. Eugenio Garza Sada 427 L4-6, Col. Altavista, Monterrey, Mexico, CP 64840 collects your personal data which is necessary to: provide, research, analyze and improve the service; send communications and notices; transfer and publish your personal data when applicable; fulfill the existing relationship; prevent or inform in the commission of unlawful acts or events. If the data is processed in your quality of candidate or collaborator of NIC Mexico, the purpose of treatment is to: create and manage your profile as a professional; provide you with working tools; conduct studies; grant benefits and programs to enhance your professional development; manage and administrate payment services and/or payroll; as well as to contact you. If you participate in promotions or surveys you may stop or quit your participation at any time. For more information read the Privacy Note<http://www.nic.mx/es/NicMx.AvisosDePrivacidad>.

---

From: Ben Maddison <[email-masked]>
Date: Mon, 24 Feb 2020 23:34:53 +0000

Hi all,

On Mon, 2020-02-24 at 17:19 -0500, Jared Mauch wrote:
> I think you may be right in absolute terms. You are likely wrong that
> unless you have cached CRL experience saying it's bad you have no
> reason to distrust. 
> 
> Also cryptographically signed data can be sent insecurely and be
> validated against change or tampering. Perhaps you are worried about
> privacy?
> 
> Sent from my iCar
> 
I think (strong not-a-crypto-guy disclaimer, etc) that Job is right.

Consider the case of a private key compromise that has been discovered
and the corresponding certificate revoked.
An attacker with access to the key could MITM the hosting rsync repo
and serve a stale CRL that pre-dates the revocation, and thereby
continue to create valid-seeming RPKI objects or replay old ones.

I've had a quick scan of rfc5280, and I can't see anything that says
what revocation status should be applied by the validating party either
way. Perhaps I'm looking in the wrong place?

My instinct says that an RPKI cert that has the CRL Distribution Point
extension should be considered revoked if a valid (inc. unexpired) CRL
can't be retrieved from one of the listed distribution points (or
through some OOB means).

Cheers,

Ben

---

From: Job Snijders <[email-masked]>
Date: Mon, 24 Feb 2020 23:43:23 +0000

> unless you have cached CRL experience saying it's bad you have no

CRLs in RPKI != CRLs in TLS/browser context.

> validated against change or tampering. Perhaps you are worried about

I don't think I'm concerned about privacy.

My point was that in the application of X509 in the use case of RPKI: if
the CRL is expired, you have more than enough reason to distrust.

Kind regards,

Job

---

From: Martin Hoffmann <[email-masked]>
Date: Tue, 25 Feb 2020 09:03:38 +0100

Job Snijders wrote:
> to assume I am right, so if you disagree - please tell me how I am
> wrong (in detail :-) ).

Let me counter with a different strong statement: In the context of
RPKI, CRLs offer no additional value and can safely be ignored.

The reason is the manifests. Unlike CRLs which only say "these
certificates are not to be considered valid anymore", manifests say
"this is the complete set of objects currently issued by this CA". They
also refer to concrete objects, identifying them both by URI and a hash
over their content. This is a much more powerful mechanism than CRLs.

What’s more, they offer a soft and a hard deadline for validity. Their
next update field is a soft deadline, much like with CRLs. But there’s
also a hard deadline in that the certificate they’ve been signed with
expires eventually. While the former has the same ambiguity as the next
update field of the CRL, the latter doesn’t. If the manifest is
expired, the CA is basically gone.

Further, the CRL is included in the manifest. So you can’t even replay
an old CRL without also replaying the old manifest.

Essentially, no information conveyed by the CRL isn’t also conveyed by
the manifest. The CRL solely serves to provide additional complexity
for the code generating and validating RPKI objects and, apparently,
creates ambiguity in the interpretation of the specification.

Kind regards,
Martin

---

From: Stephen Kent <[email-masked]>
Date: Tue, 25 Feb 2020 11:55:57 -0500

Martin,
> RPKI, CRLs offer no additional value and can safely be ignored.
I disagree.
> over their content. This is a much more powerful mechanism than CRLs.
As a co-author of the manifest document I can assure you that these 
objects are not intended as a replacement for CRLs. To provide analogous 
functionality one would have to interpret the absence of a cert from a 
Manifest as evidence that it was revoked, or expired. There is a lot of 
experience in the broader PKI community that suggests Subjects are 
sloppy about expiration dates for certs, and some evidence that CAs are 
not much better :-).
> expired, the CA is basically gone.
I don't agree with the reasoning above. It's true that the cert used to 
verify (not sign) a Manifest will expire, but so will a CA cert. Why do 
you think that a CA will be better at maintaining a current Manifest vs. 
a current CRL? F or most RPKI CAs the trigger to generate a new Manifest 
will be the need to issue a new CRL (because many CAs have elected to 
issue CRLs on a daily basis).
> an old CRL without also replaying the old manifest.
The discussion (in 6486) of what an RP should do when a current Manifest 
is not available, or when there are some discrepancies between what  the 
Manifest says and what has been retrieved allows for a lot of local 
policy discretion. As a result, an RP might well accept an old CRL and 
corresponding Manifest even though the Manifest appears to be stale.
> creates ambiguity in the interpretation of the specification.

I disagree.

Steve

---

From: George Michaelson <[email-masked]>
Date: Wed, 26 Feb 2020 10:23:50 +1000

I believe while I may use sloppier language, in broad intent agree
with Steve.  There is information in a CRL which is not expressed in a
manifest. We added manifests to our system, we did not deprecate or
replace CRLs.

We did not design this PKI with intent to make CRL optional or
unnecessary, and the Manifest as I understand my recollections, was to
provide a statement of what MUST be seen, not to exclude any other
things existing, or to repudiate things. it is not "complete" as I
understand it, it is only a statement of things which should not be
missing. There can be other things. it is entirely possible
publication of the manifest is asynchronous to other publication
events, (large flat repositories) and so will not reflect fetch state,
but future manifest state will.

It is a mechanism to show if intermediates are attempting to hide
things. It is not a categorical statement that other things are
invalid or valid, because they can be cryptographically valid, not
expired, but repudiated by a CRL which you are not seeing.

If you cannot see a CRL which you are told should exist, to assume the
CRL doesn't matter feels extremely unwise. I may have mistakenly
published entirely cryptographically valid things which do not now
inform my intent, and I have tried to repudiate, and you cannot see
that repudiation and now interpret states of routing in ways which do
not reflect my true intent.

-George

---

From: Louis Poinsignon <[email-masked]>
Date: Tue, 25 Feb 2020 18:02:05 -0800

Hello all,
We're internally reviewing this with the crypto team which have more
experience maintaining PKIs.

My opinion on an operational standpoint.
I don't think I'd like having our routers suddenly reprocessing the million
routes associated with 78000 ROAs leaving us with an increased surface of
attack and potential reroutings. The outage lasted 8 hours as well.
Even with a replayed certificate, I don't believe the impact would be as
critical. And it would leave traces.
If the impact of a mistake is more punitive than a proper misuse, it's bad.

While I believe Job is right on the discrepancy between validators, I agree
with the point of Martin on the added complexity.

OctoRPKI is tolerant on absent/invalid CRLs. But it does check it against
certificates when valid.
Manifests are more sensitive.

Even though browsers and RPKI are different, I think we can learn from the
experience. Which is to get away from CRLs.
https://blog.mozilla.org/security/2020/01/09/crlite-part-1-all-web-pki-revocations-compressed/

> Certificate Revocation Lists (CRLs) quickly became large, and contained
> mostly irrelevant data, so web browsers didn’t download them;
> The Online Certificate Status Protocol (OCSP) was unreliable, and so web
> browsers had to assume if it didn’t work that the website was still valid.

And lead to alternative solutions:
https://www.theregister.co.uk/2020/02/20/apple_shorter_cert_lifetime/
https://letsencrypt.org/2015/11/09/why-90-days.html

Looking into the data:
rpki.ripe.net/repository/DEFAULT/KpSo3VVK5wEHIJnHC2QHVV3d5mk.crl is the
largest with 1MB and 22027 serials.
Most contain around 100-1000 serials and are a few KB.
The entirety of the 17127 CRLs is 70MB.

RPKI CA are already renewing, regenerating and distributing certificates
often. Why not reducing the maximum validity of certificates to one day?
This would relieve computing and storage as well as reducing the
importance/criticality of CRLs.

On Tue, 25 Feb 2020 at 15:51, Stephen Kent <stkent=
[email-masked]> wrote:

> https://www.ietf.org/mailman/listinfo/sidrops
>

---

From: Job Snijders <[email-masked]>
Date: Wed, 26 Feb 2020 02:25:35 +0000

Dear Louis, group,

On Tue, Feb 25, 2020 at 06:02:05PM -0800, Louis Poinsignon wrote:
> attack and potential reroutings. 

It differs from BGP implementation to BGP implementation whether a
milion routes in their concept of the RIBs need to be reprocessed, or
only the routes covered by the (now invalid) ROAs / VRP removal. I
believe there are high performant BGP/ROV implementations out there.

> The outage lasted 8 hours as well.  Even with a replayed certificate,
> I don't believe the impact would be as critical. And it would leave
> traces.  If the impact of a mistake is more punitive than a proper
> misuse, it's bad.
>

The only punitive effect I observe is towards the CA operators (a small
group, appears not too often to make mistakes), but at the same time
there are continuous benefits going to the relying parties (a large
group, the internet). I see opportunity for a movement to motivate CA
operators to the strictest possible interpretation of how their RPKI CA
& publication process works and how the data distributed will be
validated.

> Even though browsers and RPKI are different, I think we can learn from
> the experience. Which is to get away from CRLs.

I agree this is worthwhile exploring.

Kind regards,

Job

---

From: Christopher Morrow <[email-masked]>
Date: Tue, 25 Feb 2020 22:28:53 -0500

> If you cannot see a CRL which you are told should exist, to assume the

This wasn't missing CRL, but an invalid one, right?
Does that change your mind on how you'd manage this sort of problem from
the (network) operations side of things?

> not reflect my true intent.

So, to bypass the CRL (in all cases) seems bad, sure.
To bypass it (or enable a bypass) in cases like this (clearly some mistake
let us get to this discussion) in order to keep routing packets 'properly' (ha!)
doesn't seem like a total disgrace.

If the CRL mistake is a the RIR publication point/top-of-tree it certainly
seems a bunch more important to fix the problem 'quickly', however, I'm
not sure 'quickly' is much more than the experienced 8hrs this time given
some default numbers in validator code bases, right?

---

From: George Michaelson <[email-masked]>
Date: Wed, 26 Feb 2020 15:29:36 +1000

the questions you pose come back down to two things to me

1) high-in-the-tree burdens exist. They're not more complex than
low-in-the-tree but they (by definition) affect more things, and more
people (RPs) -I have suffered failures in my service as Nathalie has,
and I respect her integrity in doing a public disclosure and
root-cause analysis, and I feel a similar burden. You need me to be
able to say "this happened" and I have to accept you saying "what have
you done, to prevent it happening again" just like Nathalie did (and
accepts I believe)

2) "how do you know what it means" is really hard. A lot of statements
about "..and so this means" are post-hoc interpretations of what the
CA and RP *think* the situation means, but the intent is actually not
overt in the situation, its latent. If you cannot point to a document
which clearly says what a situation means, then I feel its undefined.
That is what spec work is for. Thats why we're here.

The CA did not mean to fail to publish a CRL. The RP is faced with
having only out of date CRL. "what does it mean" is really hard to
infer a priori. Is there a documented intent of what it should mean?
Do we all agree?

I feel the CRL situation is like that. There should have been one.
There are only stale ones, or a badly informed one. It is confusing.
Incoherent. It feels like we didn't say very clearly "what does it
mean" in the spec process, if people feel there are open questions and
are free as RP to interpret this, I feel we didn't adequately wire
this down. If thats not true we shouldn't be discussing it, because it
is documented.

if you like, the question for me is 'how did you get to the word
"clearly" here, because it feels like "clearly some mistake let us get
to this discussion" is not a-priori, but is post-hoc reasoning,
because Nathalie told us it was a mistake'.

How do you know, in advance, the bad CRL situation "clearly" is a
mistake? You have to ask "did you make one" and if the CA says yes,
then it is possibly an RP problem, or an attack along the path to the
data, or some other issue. If the CA say "oh no.. that shouldnt have
happened" you "clearly" know its a mistake, but that depends on
asking, and being told.

Remember we now have RSYNC and RRDP, and they do not actually have to
cohere, they are obviously by intent meant to be eventually
consistent, but we have two ways to say what is the state of a
repository. This means there are at least two ways RPs may synchronise
on the state of a repository.  Persisting state measured in hours
around states of a repository do tend to align in both worlds, but I
believe its possible for the persisting state to be incoherent (I
believe this, because I believe it happened in the APNIC repositories)
which is an added complexity.

I think I would like a world which said "if the thing I see is
malformed, and if I can tell its malformed but I can also tell it was
well signed, I will go with operation error, and consider fallbacks"
-but if the thing is missing, or badly signed, it feels more like an
attack, or risk of attack. Well signed, but badly formed is the
special state of "you did this, but it is wrong" and so I do break it
out. I could believe we want some fallback behaviour. I could also
believe "wrong is wrong" is a better place to be.

-G

---

From: Jared Mauch <[email-masked]>
Date: Wed, 26 Feb 2020 00:53:34 -0500

I get the intent here but 24h is too short. If there is a hardware failure on a holiday weekend or natural disaster it may take too long to recover ones systems. 

In the event of some global pandemic I could even envision it taking longer for small operators. 

Sent from my iCar

> On Feb 25, 2020, at 9:02 PM, Louis Poinsignon <[email-masked]> wrote:
> 
> Why not reducing the maximum validity of certificates to one day?

---

From: Randy Bush <[email-masked]>
Date: Tue, 25 Feb 2020 22:12:36 -0800

> Why not reducing the maximum validity of certificates to one day?

for example, because the CPS of at least one high level CA is, or at
least used to be, only willing to commit to publishing once a day.

as i said elsewhere, this is much ado about a non-damaging ops gl!tch.
exposing our weak PKI clue and inclination to more complexity is not
constructive.

randy

---

From: Di Ma <[email-masked]>
Date: Wed, 26 Feb 2020 15:52:53 +0800

I think CRL is indispensable for the RPKI.

I am reassured of this argument especially by the point that George makes here. 

As far as I observed, CRL is issued in case of some hazard things happening such as key exposures, which we weigh much worse than expiration of CRL.

As the administrator of RP software RPSTIR, our implementation sees certs invalid even if this CRL is expired. We assume that the INRs covered by those invalid certs should be put into newly issue ones if the very INR holder would like to activate them in terms of routing. 

Di

> 2020年2月26日 08:23，George Michaelson <[email-masked]> 写道：
> 
>

---

From: Oleg Muravskiy <[email-masked]>
Date: Wed, 26 Feb 2020 09:28:33 +0100

I tend to agree with Steve and George that manifests could not replace CRLs. And while the rfc5280/section-6.3 only demands to fetch the (time-wise) valid CRL and does not specify what to do if that isn’t possible, my interpretation of this section is that the validator should not accept/ignore a CRL past its validity period.

With regard to manifests, the number of inconsistencies between the content of a manifest and a repository, the absence of rules of how to handle these inconsistencies, and as a result, the complexity of processing manifests during validation is much higher than the complexity of processing the CRL. I would rather get rid of manifests after fully transitioning to RRDP than putting more responsibilities on manifests.

But this is of course me as a software developer speaking. The decision should be made by operators.

Oleg

---

From: Tim Bruijnzeels <[email-masked]>
Date: Wed, 26 Feb 2020 10:25:21 +0100

Hi,

> On 25 Feb 2020, at 17:55, Stephen Kent <stkent=[email-masked]> wrote:
> 
> The discussion (in 6486) of what an RP should do when a current Manifest is not available, or when there are some discrepancies between what  the Manifest says and what has been retrieved allows for a lot of local policy discretion. As a result, an RP might well accept an old CRL and corresponding Manifest even though the Manifest appears to be stale.

To me this is the core of the issue.

I have searched through RFC 5280, and RFC 6487, but I cannot find clear guidance on what to do when there is no valid CRL available at all, or if the only available CRL is stale. RFC 5280 seems to say that one should use the best and latest valid non-stale CRL information to make a list and consider certificates revoked only if they appear on that list. But the language is hard to parse to me. 

We have had discussion about this in past meetings, and my interpretation of the outcome was that stale CRLs should be handled in the spirit of stale Manifest. But there is no written guidance, so it remains pretty vague and as a result I can see any of the following becoming local policy:
- disregard invalid CRL and consider nothing as revoked
- disregard stale CRL and consider nothing as revoked
- disregard invalid CRL and consider everything revoked (because it is now unverifiable whether they are)
- disregard stale CRL and consider everything revoked (because it is now unverifiable whether they are)
- use stale CRL as normal

I have always had issues with the amount of local policy in the manifest RFC (6486). Inevitably this leads to differences between RP implementations and possibly even between installations if software provides configuration options to users.

Again, and again, I find that operators expect that the validation outcome is the same everywhere. This is not the case. I am not talking about local exceptions, routing policy, or temporary differences. They expect the same rules to apply.

And I agree. We would be in a better place if we did have one clearly defined way to deal with all this.

One possible interpretation of the Manifest RFC, one that is used by most RPs, is the following:
- Reject the manifest if it is invalid: its EE cert expired, wrong signature etc.
- Warn if the manifest is valid, but it is stale (here implementations differ, some reject)
- Only consider objects listed on a valid manifest
- Reject any object for which the hash does not match the manifest

I understand that as written today there is no consensus on this. But if we could, finally, agree on a way here then I believe that that would simplify everything greatly - without impacting security.

RFC 6481 (Repository Structure) says that all currently valid objects must be published, and that invalid objects must not be published. So using the manifests as a signed statement of what is current and what must be regarded seems like a very plausible approach to me. The manifest EE is signed by the same key as the CRL - so unless you want enforce security through pedantry - there is no point in checking the CRL for objects that a valid manifest says are current.

There would be a use case for CRLs for anything published outside of the RPKI infrastructure - so I still see a theoretical use case there. I think that it should be applicable to yet to be defined object types, defined by OIDs and all, that are not intended for publication inside the normal RPKI structure. The CRLs would be much smaller, and would not need to be updated whenever a new manifest is published.

With regards to stale vs expired manifests: we discussed this many meetings ago. I like the idea of having things go stale, resulting in warnings that can alert people to fix things, before expiration. This could mean: next update in X hours - after which RPs would see a warning that replay attacks could be a problem, expiration in X days giving CA/repo operators time to fix things, X days being a hard limit on the replay window. But, more importantly, I think we should aim to agree on a common way to deal with this for all RPs.

Tim

---

From: Robert Kisteleki <[email-masked]>
Date: Wed, 26 Feb 2020 10:27:37 +0100

> happened" you "clearly" know its a mistake, but that depends on

And in the meantime the RP software has to make a decision one way or
another, and ultimately to send packets either left or right. It can't
ask the operator to reach out to the CA and wait for the answer, because
the decision depens on what they say.

So ideally there's an ultimate guideline on what to do with stale /
invalid CRLs/manifests or perhaps there should be configuration options
to RP software the operators can tweak with (highly) recommended default
values.

Robert

---

From: Stephen Kent <[email-masked]>
Date: Wed, 26 Feb 2020 12:03:33 -0500

Tim,

Thanks for your thoughtful analysis of the options. Her are some 
additional thoughts.

In general, a stale CRL still represents the best info re revoked certs 
available to an RP, so ignoring it does not seem appropriate, to me.

In the RPKI we also have Manifests, so if a Manifest is current but the 
CRL it references is stale, then that strongly suggests that the CA has 
encountered a problem generating a new CRL, but it's still the best 
revocation status info available, so use it and consider contacting the 
CA (via the GB record) to point out this problem.

A CRL is invalid if it's signature doesn't verify, or if it has a syntax 
error, even in the face of a valid sig. In this case I would ignore the 
CRL contents, contact the CA, and keep using the freshest CRL the RP has.

Because these CRLs MUST contain sequence numbers, receipt of a valid CRL 
that appears to be current, but has an older (or the same) serial number 
relative to a previously validated CRL is suspicious. I would 
definitively contact the CA to see if it just forgot to increment this 
field properly. I probably would stick with the freshest CRL info I had 
for that CA until the matter is resolved.

There are more cases one can analyze, but I think the basic approach is 
to stick with prior, validated CRLs and contact the CA that issued the 
questionable CRL, until the matter can be resolved.

As for the considerable leeway accorded to RPs in the Manifest document, 
I concur that it allows inconsistent local behavior. If the WG can agree 
on more proscriptive language, that would be good. When we wrote 6486 we 
were unable to agree on such, as we tried to balance robustness vs. 
responses to possible active attacks on repositories or communications 
between an RP and a repository.

Steve

---

From: Stephen Kent <[email-masked]>
Date: Wed, 26 Feb 2020 12:03:33 -0500

Tim,

Steve

---

From: Job Snijders <[email-masked]>
Date: Wed, 26 Feb 2020 17:39:35 +0000

On Wed, Feb 26, 2020 at 12:03:33PM -0500, Stephen Kent wrote:
> As for the considerable leeway accorded to RPs in the Manifest document, I
> concur that it allows inconsistent local behavior. If the WG can agree on
> more proscriptive language, that would be good. When we wrote 6486 we were
> unable to agree on such, as we tried to balance robustness vs. responses to
> possible active attacks on repositories or communications between an RP and
> a repository.

Imagine a scenario where a money-in-the-middle (sic) strategically hides
a select few ROAs, example:

MITM shows rsync://rpki.ripe.net/repository/DEFAULT/3e/01d411-d915-4277-8fe2-76b0dda2bf3e/1/r7TSyWn_GbYPjNWvt4r5ewSNAsk.roa
(80.128.0.0/11 AS 0 - expires July 1st, 2021)

but hides rsync://rpki.ripe.net/repository/DEFAULT/3e/01d411-d915-4277-8fe2-76b0dda2bf3e/1/LkKeUPYrfgzjsOIejLjsHGk44cU.roa
(80.128.0.0/11 AS 3320 - expires July 1st, 2021)

If Origin Validating EBGP edge routers ends up honoring only a *subset*
of VRPs, it may result in catastrophic hard-to-troubleshoot outages. In
this example, the victim end up being unable to reach half of Germany.

I think the entire repository should be considered invalid if a single
file is missing but was referenced in the manifest. One can't produce
rules based upon false or incomplete data, and one can't protect against
hijacks using unsigned data.

expired CRL? repository invalid
any file missing that was referenced in manifest? repository invalid
the above also means, is the CRL missing? repository invalid
in addition to any cert being expired? underlaying objects invalid

Any other behaviour is a security problem, unsafe. Leeway does more
damage than good.

I believe the premise for Origin Validation to work on the Internet it
is that in order to get it deployed, BGP has to 'fail open', but in the
RPKI cache validation process one must 'fail close', which depends on
all validators being 'strict'. If the RPKI component doesn't fail
closed, it produces false filters, which goes against our desire for BGP
to be able to 'fail open'.

Kind regards,

Job

---

From: Stephen Kent <[email-masked]>
Date: Wed, 26 Feb 2020 17:35:39 -0500

Job,
> (80.128.0.0/11 AS 3320 - expires July 1st, 2021)
I'm confused by your example. The discussion has been focused on CRL 
issues, not suppression of ROAs.
> hijacks using unsigned data.
Do you mean the pub point for the address space holder, vs. "the 
repository", which suggests a larger set of data, e.g., RIPE operates a 
repository that contains pub points for their members.
> Any other behaviour is a security problem, unsafe. Leeway does more
I'm afraid I have to disagree with your conclusions., in  part because 
of confusing use of terminology.
> to be able to 'fail open'.

A goal RPKI operation is to not make routing worse that it would be in  
the absence of the RPKI. Thus it seems appropriate to tolerate some 
types of operational errors, on a temporary basis, and try to follow up 
with CAs or pub point operators to resolve the errors. I acknowledge 
that different responses may be appropriate depending on  the nature of 
the error - a stales CRL is not the same as a bogus ROA.

Steve

---

From: Stephen Kent <[email-masked]>
Date: Wed, 26 Feb 2020 17:35:39 -0500

Job,

Steve

---

From: Robert Kisteleki <[email-masked]>
Date: Thu, 27 Feb 2020 09:59:43 +0100

> hijacks using unsigned data.

I'm not sure that'd be wise. Knowing you're missing something does not
invalidate the other bits that you can verify. Exactly what you keep
using can depend on what's missing (a CRL, a CA cert, a ROA, ...) though.

Regards,
Robert

---

From: Tim Bruijnzeels <[email-masked]>
Date: Thu, 27 Feb 2020 10:27:06 +0100

Steve,

> On 26 Feb 2020, at 18:03, Stephen Kent <stkent=[email-masked]> wrote:
> 
> Tim,
> 
> Thanks for your thoughtful analysis of the options. Her are some additional thoughts.
> 
> In general, a stale CRL still represents the best info re revoked certs available to an RP, so ignoring it does not seem appropriate, to me.

In general I agree. Which is why I mentioned the example of objects one might find published outside of the context of the RPKI repository, and *not* listed on a manifest. Such objects have yet to be invented but theoretically one can think of signed structures under an existing RPKI CA cert (e.g. using their own embedded EE) - which is shared out-of-band between parties.

> In the RPKI we also have Manifests, so if a Manifest is current but the CRL it references is stale, then that strongly suggests that the CA has encountered a problem generating a new CRL, but it's still the best revocation status info available, so use it and consider contacting the CA (via the GB record) to point out this problem.
> 
> A CRL is invalid if it's signature doesn't verify, or if it has a syntax error, even in the face of a valid sig. In this case I would ignore the CRL contents, contact the CA, and keep using the freshest CRL the RP has.
> 
> Because these CRLs MUST contain sequence numbers, receipt of a valid CRL that appears to be current, but has an older (or the same) serial number relative to a previously validated CRL is suspicious. I would definitively contact the CA to see if it just forgot to increment this field properly. I probably would stick with the freshest CRL info I had for that CA until the matter is resolved.
> 
> There are more cases one can analyze, but I think the basic approach is to stick with prior, validated CRLs and contact the CA that issued the questionable CRL, until the matter can be resolved.
> 
> As for the considerable leeway accorded to RPs in the Manifest document, I concur that it allows inconsistent local behavior. If the WG can agree on more proscriptive language, that would be good. When we wrote 6486 we were unable to agree on such, as we tried to balance robustness vs. responses to possible active attacks on repositories or communications between an RP and a repository.

What I am suggesting is that we *could* update 6486 and make validation more restrictive regarding manifests:
- all objects on a manifest must be present and accounted for (I agree with Job regarding partial withhold attacks)
- all objects on a manifest need to be validated
- objects that are not on a manifest can be considered invalid

This is in-line with the specifications defined in RFC 6481 (A Profile for Resource Certificate Repository Structure), which essentially says that all current objects must be published, and that no invalid objects may be published.

Then, if the manifest is already a signed statement of everything that is current, at least regarding the currently defined object types, as defined in RFC 6481, then what is gained by checking that CA was also capable of generating a CRL - using the same authoritative key and publication method - that confirms that the objects that are current according to the manifest are indeed not revoked?

Requiring the CRLs just feels like unnecessary brittleness to me (again in the context of RFC 6486). It creates multiple loci for bugs in CA implementations, and complicated error conditions that need to be checked by RPs. This is what I meant with the perhaps poorly chosen word 'pedantic'. Maybe I should have used the word 'redundancy'. Redundancy may seem like a good idea in general, but in this case it really only allows for the possibility of two conflicting signed messages. Thus it seems that this does not increase security, but provides more ways for things to break.

Don't get me wrong.. we *are* generating the CRLs in unison with the MFTs. And for this to change one would have to make update to the manifest RFC and restrict the validation rules in a manner like I am suggesting above.

I am just asking that we consider that.

Kind regards,
Tim

---

From: Tim Bruijnzeels <[email-masked]>
Date: Thu, 27 Feb 2020 10:50:19 +0100

Hi,

> On 27 Feb 2020, at 09:59, Robert Kisteleki <[email-masked]> wrote:
> 
> 
> 

I am not saying that I have the complete answer here, but I think it's worthwhile to discuss this further - if the WG is willing to consider updates to the manifest RFC.

It gets complicated.

ROAs at the parent level have the ability to invalidate announcements by children that run delegated CAs. If their CA cert is missing, then this can impact ROV to the point that announcements by children are considered invalid.

In such cases it might be better to mark objects the ROA objects as invalid so that ROV yields not found. It is probably okay to recurse any other CA cert and use ROAs found under it. Probably, because probably they are more specific, and ROAs there would not invalidate covering, presumably, less specific announcements done by the parent (yes, lots of assumptions here). For BGPSec certificates things get more complicated, because at the MFT level they are indistinguishable from delegate CA certificates. Also BGPSec does not have a soft-landing, so invalidating certificates (rather than ROAs) might be quite harmful.

Note, I think this should be applied only to *missing* objects as that indicates either a failure to publish things properly, or possibly a partial withhold attack where the RP receives incomplete information. And even then, there may be ways to recover if an RP still has prior copies of all missing objects (check hash).

If objects are found and accounted for, but they turn out to be invalid, then I believe that other objects should not be impacted.

Regards
Tim