Subject: [Sidrops] trying to limit RP processing variability

From: Stephen Kent <[email-masked]>
Date: Fri, 3 Apr 2020 11:40:10 -0400

*Overview *

A publication point (PP) in an RPKI repository contains the following 
elements:

1.CA certificates (optional, present only for subordinate CAs)

2.EE certificates (optional, present only for routers)

3.CRL (usually one)

4.ROAs

5.Manifest (usually one)

6.Ghostbusters record (one, optional)

*Terminology*

Valid: a signed object (other than a certificate or CRL) is deemed 
*valid* if it is encoded in a fashion consistent with the relevant RPKI 
RFCs (i.e., 6482, 6486, 6487, 6488, 6493) and the signed object’s 
signature can be verified consistent with RFC 6488.

Current: A CRL or Manifest is *current* if the current date & time is 
before the Next Update date and the CRLNumber (or ManifestNumber) is 
greater than any prior, valid CRLs or Manifests processed by the RP.

Stale: A CRL or Manifest is stale if the current date & time is later 
than the Next Update date. A stale CRL or Manifest is not “expired” it’s 
just not as current as it is supposed to be. The case analysis below 
examines what to do when a Manifest, or CRL, or both are stale.

Missing: an object named in a Manifest, but not available for download 
from a PP, is termed *missing*. An RP has no obvious way to acquire 
missing objects, but operators SHOULD be warned about which objects are 
missing.

Corrupted: if an object named in a Manifest, is available for download 
from a PP, but it’s hash does not match the corresponding value in the 
Manifest, the object is termed *corrupted* and the object SHOULD be ignored.

*Case analysis*

If we can agree to ignore any objects at a PP that do not appear on the 
current manifest, the case analysis is easier. (The discussion of 
corrupted objects above addresses the case where an object is named on 
the Manifest, but the hash does not match.)

Below is a proposed outline for the cases. The group needs to decide 
what action is to be taken in each case.

1.Manifest present, valid, current

1.1.CRL present, valid, current

1.2.CRL present, stale

1.3.CRL present, invalid

1.4.CRL missing

2.Manifest present, valid, stale

2.1.CRL present, valid, current

2.2.CRL present, stale

2.3.CRL present, invalid

2.4.CRL missing

3.Manifest present, but invalid

3.1.CRL present, valid, current

3.2.CRL present, stale

3.3.CRL present, invalid

3.4.CRL missing

4.Manifest not present

4.1.CRL present, valid, current

4.2.CRL present, stale

4.3.CRL present, invalid

4.4.CRL missing

---

From: Job Snijders <[email-masked]>
Date: Fri, 3 Apr 2020 16:26:47 +0000

Dear Stephen,

Thank you for spearheading this initiative. It is helpful to see the
various permuations for which a decision tree needs to be constructed
laid out in this way.

On Fri, Apr 03, 2020 at 11:40:10AM -0400, Stephen Kent wrote:
> *valid* if it is encoded in a fashion consistent with the relevant
> RPKI RFCs (i.e., 6482, 6486, 6487, 6488, 6493) and the signed object’s
> than the Next Update date. A stale CRL or Manifest is not “expired”
> it’s just not as current as it is supposed to be. The case analysis
> below examines what to do when a Manifest, or CRL, or both are stale.
> 
> missing objects, but operators SHOULD be warned about which objects
> are missing.
> 

Can you also provide a definition for your use of the word 'invalid'? I
see the phrase used in the case analysis permutations.

Kind regards,

Job

---

From: Stephen Kent <[email-masked]>
Date: Fri, 3 Apr 2020 13:27:14 -0400

Job,
> see the phrase used in the case analysis permutations.

I would call a signed object invalid if it is syntactically broken 
(e.g., bad ASN.1 encoding), or if the signature cannot be validated as 
specified in RFC 6488. This is just another way of saying that it fails 
either of the  criteria cited in definition of *valid*.

Steve

---

From: Stephen Kent <[email-masked]>
Date: Fri, 3 Apr 2020 13:27:14 -0400

Job,

Steve

---

From: Job Snijders <[email-masked]>
Date: Wed, 8 Apr 2020 00:35:21 +0000

Hi,

On Fri, Apr 03, 2020 at 11:40:10AM -0400, Stephen Kent wrote:
> If we can agree to ignore any objects at a PP that do not appear on
> the current manifest, the case analysis is easier. (The discussion of

pass. all lights are green.

> [ snip lots of variants ]

All cases we can come up with (except the 'all green') probably require
some mailinglist back and forth as to why it is that we can safely
ignore one or another validation check. If nobody steps up to advocate a
particular case, perhaps we don't need it.

My suggestion to the group is that we go through Kent's list and start
with a justification for the simplest of cases (as described above), and
then construct justifications for any other variants, if they exist.

I imagine some of the cases have already been discussed in years prior,
but RPKI OV deployment on the Internet at scale is still fairly new,
taking a fresh look at things is exciting :)

Kind regards,

Job

---

From: Robert Kisteleki <[email-masked]>
Date: Thu, 9 Apr 2020 12:25:07 +0200

Hi,

> missing objects, but operators SHOULD be warned about which objects are

IMO an "RP has no obvious way to acquire missing objects" is not
entirely true.

If, at the previous run, the RP fetched the relevant (now missing)
object, then I see no reason to not use it again. Think of the previous
run as an object a cache if you will: if you're looking for an object
mentioned in the manifest, and you have it already (hash / name / etc.
matches) then you can reuse it.

Of course it can be useful to check if it still exists in the PP, but it
seems to me the only benefit is to detect that it is missing from there
and perhaps warn the PP operator. Otherwise the RP has a hard time
arguing "no idea what this is since it's not there!".

For bonus points: an implementation that fetched a PP's manifest,
detected that it's exactly the same as before, and therefore reused the
validation outcome from the previous run would not even have to fetch
*any* other objects from the same PP. The downside is that this process
will not be able to point out the omission. The upside is saves a lot of
resources (bandwidth, CPU and all). A further upside is that as a
side-effect this protects against a malicious attacker (selectively)
hiding objects.

Where this comes back to the current discussion is: would this behaviour
be mandated, recommended, or considered a big no-no?

Robert

---

From: Martin Hoffmann <[email-masked]>
Date: Thu, 9 Apr 2020 14:06:54 +0200

Robert Kisteleki wrote:
> object, then I see no reason to not use it again. Think of the
> previous run as an object a cache if you will: if you're looking for
> an object mentioned in the manifest, and you have it already (hash /
> name / etc. matches) then you can reuse it.

That is theoretical possible, but in practice you treat synchronising
and validation of repository content as two separate steps. I.e, before
you even start looking at a CA’s repository, you synchronize its
content. This is enshrined in the way both rsync and RRDP work: They
don’t update single files but entire directory trees all at once. This
step includes deleting objects that have been deleted on the server.

Since the complete RPKI repository has a hierarchical structure
following the rsync URIs of objects, many RP implementations keep the
objects in the file system only. This is in particular useful for
rsync: Just let rsync update the directory in place. An additional
bonus of this strategy is that you don’t need a fancy database.

You could, of course, concoct a mechanism that marks files for deletion
and only deletes them if they aren’t actually used in the next
validation run. But, considering that this thread is actually
subjected “trying to limit RP processing variability,” I am not sure
this is a good idea. There is a strong likelihood that different
strategies will behave slightly differently. If we really want to
come to a point where every RP implementation produces the same output
from given input, we need to defined simple rules that are easy to
implement in a wide range of circumstances.

Another consequence of doing this is that validation on a newly
deployed RP software differs from one that has been running for a
while. As a consequence, the datasets from two different caches
configured in routers differ. So now you even have difference between
caches running the same software.[0]

Kind regards,
Martin 

[0] Yes, with broken RRDP servers where snapshots differ from a
    sequence of deltas, this can happen too. We should perhaps also
    look into improving the robustness of RRDP.

---

From: Stephen Kent <[email-masked]>
Date: Thu, 9 Apr 2020 15:59:26 -0400

Robert,
> matches) then you can reuse it.
I probably should have said that an RP views an object as "missing" if 
the the object is not present in the RP's cache and cannot be retrieved 
from the relevant PP. Because RPs retrieve objects only if the objects 
have changed (newly added or updated), an RP would not be aware that an 
object is not present at a PP if the object is present in the RP's 
cache. I recall a famous quote about whether an object that has not been 
updated makes any noise when it is deleted from a PP, or something like 
that :-)
> arguing "no idea what this is since it's not there!".
I think I agree about the utility of detecting an object that has gone 
missing from a PP, when the RP has the object locally cached. But, in my 
experience, RP software was not designed to detect this case.
> *any* other objects from the same PP.
If the manifest was not changed, at all, then it would not be fetched, 
right?
> The downside is that this process
If the only things that changed in the manifest were the updates and 
manifest number, then there would be no need to retrieve any additional 
objects, and with rsync I don't believe there would be any other file 
retrievals. I can't be sure, but I think thge BBN RPSTIR software 
behaved that way.  Do we get bonus points?
> be mandated, recommended, or considered a big no-no?

Which behavior? I would say recommended.

Steve

---

From: Di Ma <[email-masked]>
Date: Fri, 10 Apr 2020 13:09:07 +0800

> If the only things that changed in the manifest were the updates and manifest number, then there would be no need to retrieve any additional objects, and with rsync I don't believe there would be any other file retrievals. I can't be sure, but I think thge BBN RPSTIR software behaved that way.  Do we get bonus points?

Yes. 

<On behalf of RPSTIR open source team>

RPSTIR doesn’t retrieve any additional objects by using rsync to keep synchronized with repositories.

However, how to deal with those downloaded objects (use/ignore/warn) depends on how we cope with MFT, CRL and local analysis.

Di

---

From: Robert Kisteleki <[email-masked]>
Date: Tue, 14 Apr 2020 10:54:23 +0200

Hi Steve,

>> If, at the previous run, the RP fetched the relevant (now missing

rsync has a flag (--delete) where it literally syncs, ie. not only
fetches new/updated objects but also removes the local ones that no
longer exist remotely. (Either this, or some other processes have to do
cleanup, otherwise the local copy accumulates objects eternally.)

I believe the NLnetLabs routinator and the NCC's RPKI validator use
--delete with rsync. Maybe this behaviour changes if/when not using
rsync, I'm not sure.

> experience, RP software was not designed to detect this case.

I believe the discussion here is about specifying how RP software should
(or must) behave, so that their behaviour is consistent. IMO in order to
achieve that, the behaviour around missing objects should be part of the
specification.

Cheers,
Robert

---

From: Claudio Jeker <[email-masked]>
Date: Tue, 14 Apr 2020 11:18:18 +0200

> achieve that, the behaviour around missing objects should be part of the

This should not only be about RP software but also the RPKI CA software
and the operation of such software and repositories.
If the CA software publishes only consistent repos the problem would be
solved. Also the operators of rpki repositories should check their repos
before publishing them as part of monitoring.

If the CA software is sloppy then the RP software will always have a hard
time to get consitent behaviour. It is the typical garbage in garbage out
issue.

-- 
:wq Claudio

---

From: Stephen Kent <[email-masked]>
Date: Tue, 14 Apr 2020 13:45:50 -0400

Robert,
> cleanup, otherwise the local copy accumulates objects eternally.)

I forgot about the --delete flag. But, if one chooses to make use of the 
flag, it requires caution. if an attacker deleted files and an RP 
removed instances of those files from the local cache, this is an easy 
DoS attack. One would have to see if there is a current manifest and if 
it lists files that have been deleted from the PP. Only if there is a 
current manifest, and the deleted files are also not present in the 
manifest, would it be safe to delete those files.

If one relies on examination of a current manifest to trigger deletion 
of cached files this problem would be avoided, and thus those files 
would not accumulate forever. maybe that's why I was not thinking about 
the flag.

> achieve that, the behaviour around missing objects should be part of the

Agreed.

Steve

---

From: Robert Kisteleki <[email-masked]>
Date: Wed, 15 Apr 2020 12:37:48 +0200

Hi,

> manifest, would it be safe to delete those files.

I agree. However, that's not the practice today. Which is why I latched
on to your proposal and I'm trying to highlight that we should cover this.

Cheers,
Robert

---

From: Martin Hoffmann <[email-masked]>
Date: Wed, 15 Apr 2020 12:46:10 +0200

Stephen Kent wrote:
> > longer exist remotely. (Either this, or some other processes have
> > to do cleanup, otherwise the local copy accumulates objects
> > eternally.)  
> 
> I forgot about the --delete flag. But, if one chooses to make use of
> the flag, it requires caution. if an attacker deleted files and an RP 
> removed instances of those files from the local cache, this is an
> easy DoS attack. One would have to see if there is a current manifest
> and if it lists files that have been deleted from the PP. Only if
> there is a current manifest, and the deleted files are also not
> present in the manifest, would it be safe to delete those files.
> 
> If one relies on examination of a current manifest to trigger
> deletion of cached files this problem would be avoided, and thus
> those files would not accumulate forever. maybe that's why I was not
> thinking about the flag.

An attacker who can delete files can as easily replace them with
something else with the same result. So I am not sure the added code
complexity is worth it.

Kind regards,
Martin

---

From: "Job Snijders" <[email-masked]>
Date: Wed, 15 Apr 2020 12:50:32 +0200

Agreed. 

fwiw: rpki-client runs as "openrsync -rlt --delete rsync://xxx/repository xxx/repository"

Kind regards,

Job

---

From: Stephen Kent <[email-masked]>
Date: Wed, 15 Apr 2020 07:58:32 -0400

Robert,
> ...

We should ask Di Ma if RPSTIR-2 still follows the design approach I cited.

Steve

---

From: Stephen Kent <[email-masked]>
Date: Wed, 15 Apr 2020 07:58:39 -0400

Martin,
> something else with the same result. So I am not sure the added code

If an attacker deletes files and replaces them with older versions or 
completely different files, and there is a current manifest, then the 
effect is nil, i.e., the older versions or different files will be 
ignored. If there is not a current manifest, then we are precisely in 
the territory that we all agree needs to be explored.

Steve

---

From: Stephen Kent <[email-masked]>
Date: Wed, 15 Apr 2020 07:58:39 -0400

Martin,

Steve

---

From: Martin Hoffmann <[email-masked]>
Date: Wed, 15 Apr 2020 15:01:41 +0200

>                                     *Overview *
[...]
> 
> *Terminology*
>
> Missing: an object named in a Manifest, but not available for
> download from a PP, is termed *missing*. An RP has no obvious way to
> acquire missing objects, but operators SHOULD be warned about which
> objects are missing.

Since "missing" seems to be the opposite of the "present" in the case
overview below, I think we need to more precisely define the meaning
for both manifests and CRLs.

I would define the meaning of "manifest missing": the manifest
referenced in the CA certificate is not available for download from a
PP. That leads to the case where there is a valid manifest at the PP
but it is not the one on the CA certificate.

"CRL missing" is a little more complicated, since there is two places
where CRLs are referenced: in the manifest and in the EE certificates
of signed objects. So we also have the case where a CRL in an EE
certificate isn’t actually mentioned in the manifest.

> *Case analysis*
[...] 
> what action is to be taken in each case.

To get that discussion started, here’s my proposals for these cases (in
line with what I argued earlier).

> 1.Manifest present, valid, current

If there is a present, valid, and current manifest, the CRL can safely
be ignored for all objects.

In order to avoid using partial sets of information, the entire content
of the PP is ignored if any object is missing, corrupted, or invalid.

Presumably this chould only be limited to objects of the same type,
i.e., if there is a least one missing, corrupted, or invalid ROA,
discard all ROAs.

If the aim is to avoid risking accidentally marking routes as invalid,
this should probably also extend to all information published by child
CAs.

> 2.Manifest present, valid, stale

Not sure about this one. Naively I would say "use as above but warn."
There’s also an option to distinguish based on the transport protocol
used for synchronizing the PP: ignore if rsync is used, use but warn if
RRDP is used.

> 3.Manifest present, but invalid

The entire PP is ignored.

> 4.Manifest not present

The entire PP is ignored.

This essentially ignores the CRL for any object published within the
repository and essentially moves its role over to the manifest. The CRL
would still be considered for objects published outside the repository,
such as the proposed Resource Tagged Assertions.

Kind regards,
Martin

---

From: Di Ma <[email-masked]>
Date: Wed, 15 Apr 2020 21:42:20 +0800

> 2020年4月15日 19:58，Stephen Kent <stkent=[email-masked]> 写道：
> 
> Robert,
>> ...

Steve is right.

RPSTIR2 still follows the very design approach.  

In terms of synchronization, RPSTIR2 keeps its incumbent local version exactly the same with global repositories. 

Granted, RPSTIR2 keeps the historic versions for diagnose and analysis. 

I think we just need to keep RP in alignment with PP and then can perform some analysis and diagnose (if necessary) to ascertain errors (RFC 8211) to give warning or make use of SLURM (RFC8416) if the RP operator is confident enough that he catch the wrong data and knows how to remedy locally. 

Di

---

From: Job Snijders <[email-masked]>
Date: Wed, 15 Apr 2020 14:00:12 +0000

Dear Martin,

> be ignored for all objects.

I'd like to ask you for some clarification, Stephen Kent listed 4
different cases, I parapharsed here:

1.1. Manifest present, valid, current AND CRL present, valid, current
1.2. Manifest present, valid, current AND CRL present, stale
1.3. Manifest present, valid, current AND CRL present, invalid
1.4. Manifest present, valid, current AND CRL missing

When you say "can safely be ignored", do you mean, MAY be ignored,
SHOULD be ignored? or MUST be ignored? In all four cases?

> In order to avoid using partial sets of information, the entire
> content of the PP is ignored if any object is missing, corrupted, or
> invalid.

It appears we are inching towards consensus on avoiding to use partial
sets, I think that's good news.

Yes.

Kind regards,

Job

---

From: Martin Hoffmann <[email-masked]>
Date: Wed, 15 Apr 2020 16:14:15 +0200

Job Snijders wrote:
> > If there is a present, valid, and current manifest, the CRL can
> > safely be ignored for all objects.  
> 

In all cases. Since it is ignored, it doesn’t really matter whether it
is good or bad or ugly. 

I suppose I would go with "SHOULD be ignored" here.

> sets, I think that's good news.

Keep in mind that there is still a possibility that partial sets are
created if a resource is delegated to two separate entities and one of
them goes kaputt. I have no idea how likely that is in practice. This
rule should, however, cover the rather more likely case of partial sets
because of partial breakage of a single publication point.

Kind regards,
Martin

---

From: Claudio Jeker <[email-masked]>
Date: Wed, 15 Apr 2020 16:33:21 +0200

> I suppose I would go with "SHOULD be ignored" here.

Which CRL are we talking about? The ones listed in the manifest or the one
referenced by the manifest certificate?
I agree that the CRL for manifests is ignored mainly because of the chicken
and egg situation (the manifest holds the CRL fileAndHash which is uses by
the manifest itself).

For CRL listed in manifests I think the situation is different. If a CRL,
ROA or CERT in a manifest is missing then the manifest must be considered
invalid and everything under it ignored. This is so that missing ROA would
not suddenly result in RPKI invalid routes because only part of a set was
processed.

-- 
:wq Claudio

---

From: Tim Bruijnzeels <[email-masked]>
Date: Wed, 15 Apr 2020 16:56:08 +0200

> On 15 Apr 2020, at 16:14, Martin Hoffmann <[email-masked]> wrote:
> 
> Job Snijders wrote:

A practical approach here could be to add all resources on the issuing CA certificate for this publication point to a SLURM like ignore filter.

This way one can be sure that one does not generate RPKI invalid *announcements* because of missing information.

Kind regards,
Tim

> 
> Kind regards,
> Martin
> 
> _______________________________________________
> Sidrops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidrops

---

From: Tim Bruijnzeels <[email-masked]>
Date: Wed, 15 Apr 2020 16:57:41 +0200

> On 15 Apr 2020, at 16:56, Tim Bruijnzeels <[email-masked]> wrote:
> 
> 
> 

To be clear.. I mean a 'virtual filter' in the context of the validation run. Not a permanent file on disk or anything :)

---

From: Jay Borkenhagen <[email-masked]>
Date: Wed, 15 Apr 2020 11:49:55 -0400

> fwiw: rpki-client runs as "openrsync -rlt --delete rsync://xxx/repository xxx/repository"

I agree with this behavior as well.

Further, it leads me to make a new straw man proposal for our
pre-Interim/Virtual meeting discussion.

Earlier in a related thread I wrote "[...] in the context of the RPKI,
we need to reach a point where all RFC-compliant RPs will generate
identical sets of VRPs when presented with the same input."  That
statement did not generate any pushback (but feel free to disagree now
if you want). 

Now I would like to push to see if we can take it further.  Please
bash this proposed goal:

 Each validation run of each RFC-compliant RP will generate the same
 set of VRPs when presented with identical input, using unexpired
 records from previous retrievals to deal with only complete failure
 to retrieve from a PP.

This implies that the "cache" aspects of an RP serve two purposes: 

 (1) reducing the amount of data to be downloaded on subsequent runs

and:

 (2) addressing the case of a PP that is temporarily completely
 unreachable, for only as long as those retrieved records have not
 expired. 

In other words, RPs should assume an empty cache, except in the case
where a previously-reached PP is now temporarily completely
unreachable. 

Commence bashing. :)

						Jay B.

---

From: Randy Bush <[email-masked]>
Date: Wed, 15 Apr 2020 11:19:32 -0700

> something else with the same result.

not really.

---

From: Robert Kisteleki <[email-masked]>
Date: Thu, 16 Apr 2020 10:50:43 +0200

> be ignored for all objects.

While I see the benefits on reducing code complexity, I believe there
have been arguments before against ignoring the CRL.

> of the PP is ignored if any object is missing, corrupted, or invalid.

Doing so will, in the presence bit flips or non-atomic PP updates or an
attacker hiding/modifying files, invalidate whole subtrees of the
hierarchy. If this happens high enough in the chain, the result is
invalidation a significant portion of the RPKI space.

We can do better.

Robert

---

From: Martin Hoffmann <[email-masked]>
Date: Thu, 16 Apr 2020 11:33:20 +0200

> > something else with the same result.  

I cannot think of a case where an attack can manipulate the rsync
exchange, impersonate the rsync server, or manipulate the file system
of the legitimate rsync server in such a way that they can signal
to an rsync client to delete existing files but not to replace them
partially or completely. 

What am I missing?

Kind regards,
Martin

---

From: Tim Bruijnzeels <[email-masked]>
Date: Thu, 16 Apr 2020 11:40:19 +0200

> On 16 Apr 2020, at 10:50, Robert Kisteleki <[email-masked]> wrote:
> 
> 

With RRDP CAs can publish consistent atomic deltas. This was one of its design goals.
A bit flip on an HTTPS connection seems highly unlikely, if not impossible, to me as well.

Also RPs that have previous copies of files (and know their URI and hash) do not need to be affected by intermittent issues.

While I understand that it's scary to exclude all VRPs from near the apex of the tree, the alternative is that you will need to accept that announcements can be marked as invalid - because you *know* you do not have the full picture.

> We can do better.

How?

---

From: Martin Hoffmann <[email-masked]>
Date: Thu, 16 Apr 2020 11:58:17 +0200

Claudio Jeker wrote:
> > > 1.1. Manifest present, valid, current AND CRL present, valid,
> > > current 1.2. Manifest present, valid, current AND CRL present,
> > > stale 1.3. Manifest present, valid, current AND CRL present,
> > > invalid 1.4. Manifest present, valid, current AND CRL missing
> > > 
> > In all cases. Since it is ignored, it doesn’t really matter whether
> > it is good or bad or ugly. 
> > 
> Which CRL are we talking about? The ones listed in the manifest or
> the one referenced by the manifest certificate?

Isn’t that supposed to be the same? Each CA only issues one CRL for all
its issued certificates and replaces it upon update.

Kind regards,
Martin

---

From: Martin Hoffmann <[email-masked]>
Date: Thu, 16 Apr 2020 12:02:52 +0200

>  records from previous retrievals to deal with only complete failure

I’d like to limit this to: "... using records from the last successful
retrieval ..." meaning that you can deleting unused objects after a
successful retrieval is possible.

I find this an excellent goal and would like the working group to
produce a test data set that can be used to verify that an RP software
complies with the intended interpretation of the validation rules.

Kind regards,
Martin

---

From: Randy Bush <[email-masked]>
Date: Thu, 16 Apr 2020 06:30:41 -0700

> to an rsync client to delete existing files but not to replace them

6486 4.2

   fileList:
      This field is a sequence of FileAndHash objects.  There is one
      FileAndHash entry for each currently valid signed object that has
      been published by the authority (at this publication point).  Each
      FileAndHash is an ordered pair consisting of the name of the file
      in the repository publication point (directory) that contains the
      object in question and a hash of the file's contents.
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

---

From: Stephen Kent <[email-masked]>
Date: Thu, 16 Apr 2020 10:54:07 -0400

Martin
> its issued certificates and replaces it upon update.

I agree.

Steve

---

From: Stephen Kent <[email-masked]>
Date: Thu, 16 Apr 2020 10:54:26 -0400

Martin,
> certificate isn’t actually mentioned in the manifest.

I don't think I follow your reasoning here.

For CA A, there is one PP where all of the signed objects issued by A 
live, ignoring the cases of key rollover or algorithm transition 
(Section 2 of RFC 6481, especially Figure 1). This PP contains the ROAs, 
a manifest, and a CRL, router certs, and any subordinate CA certs. The 
CRL that lives here would contain an entry for a cert associated with a 
manifest if that cert were revoked, and the manifest contains an entry 
for the CRL. If one were to encounter a CRL that contained an entry for 
the EE cert in the manifest, that would indicate an error by the CA.

> be ignored for all objects.
Tim has suggested that, but to do so would make the RPKI not consistent 
with RFC 5280.
> of the PP is ignored if any object is missing, corrupted, or invalid.
This is not consistent with your suggestion that the CRL can safely be 
ignored.
> i.e., if there is a least one missing, corrupted, or invalid ROA,
I'm not sure why the group-oriented approach is useful, expect perhaps 
for ROAs. Why would one elect to ignore some router certs if one were 
missing?
> this should probably also extend to all information published by child
is the concern here that ROAs issued by a subordinate CA might be 
treated differently if there is a missing ROA associated with the parent?
> used for synchronizing the PP: ignore if rsync is used, use but warn if
I'm in favor of warning here, but others, I believe, have suggested, 
that we try to make the outcome independent of the use of rsync vs. RRDP.
> The entire PP is ignored.
So, it's OK to ignore a CRL that is invalid, but not a manifest. 
Personally I believe that if a CA cannot manage to correctly generate 
both a CRL and a manifest, it has a serious operational problem.
> such as the proposed Resource Tagged Assertions.

A missing manifest is a serious concern, but I would consider using 
cached, still valid objects for the PP, and insisting on contacting the 
CA or PP maintainer.

Steve

---

From: Stephen Kent <[email-masked]>
Date: Thu, 16 Apr 2020 10:54:34 -0400

Claudio,
> and egg situation (the manifest holds the CRL fileAndHash which is uses by

See my earlier comment that I believe there really is not a chicken and 
egg problem here re CRLs. I don't see any indication in 6486 that the 
CRLDP in an EE cert for a manifest is different from that of the EE 
certs appearing in ROAs.

> not suddenly result in RPKI invalid routes because only part of a set was

Could you elaborate on this last comment?

Thanks,

Steve

---

From: Stephen Kent <[email-masked]>
Date: Thu, 16 Apr 2020 10:54:34 -0400

Claudio,

Thanks,

Steve

---

From: "Job Snijders" <[email-masked]>
Date: Thu, 16 Apr 2020 17:01:42 +0200

>       object in question and a hash of the file's contents.

I think I see an instance of miscommunication:

I think Martin's perspective is that whether the files are deleted or modified by MITM, the RP outcome is the same: the PP has to be tossed. I don't think Martin suggested that files can trivially be replaced, just that deletion or tampering have the same effect. 

Right?

Kind regards,

Job

---

From: Claudio Jeker <[email-masked]>
Date: Thu, 16 Apr 2020 17:45:09 +0200

On Thu, Apr 16, 2020 at 10:54:34AM -0400, Stephen Kent wrote:
> See my earlier comment that I believe there really is not a chicken and egg
> problem here re CRLs. I don't see any indication in 6486 that the CRLDP in
> an EE cert for a manifest is different from that of the EE certs appearing
> in ROAs.

My point was that the CRL referenced by the MFT is part of the MFT
FileAndHash list and is therefor not yet verified and loaded. So either
one must verify the EE cert of the MFT without checking the CRL or one
must load a not yet verified CRL file to be able to do proper full EE cert
verification. This is the chicken and egg situation I was referring to.
 
> Could you elaborate on this last comment?

I think Job Snijders showed this very well with his examples where removal
of one roa file would result in the insertion of an AS 0 VRP covering a
large part of Germany's IP space unless all of the manifest data is
ignored.

-- 
:wq Claudio

---

From: Martin Hoffmann <[email-masked]>
Date: Thu, 16 Apr 2020 17:53:26 +0200

Stephen Kent wrote:
> >> download from a PP, is termed *missing*. An RP has no obvious way
> >> to acquire missing objects, but operators SHOULD be warned about
> >> which objects are missing.  
> > Since "missing" seems to be the opposite of the "present" in the
> > case overview below, I think we need to more precisely define the
> > meaning for both manifests and CRLs.
> >
> > referenced in the CA certificate is not available for download from
> > a PP. That leads to the case where there is a valid manifest at the
> > PP but it is not the one on the CA certificate.
> >
> > "CRL missing" is a little more complicated, since there is two
> > places where CRLs are referenced: in the manifest and in the EE
> > certificates of signed objects. So we also have the case where a
> > CRL in an EE certificate isn’t actually mentioned in the manifest.  
> 
> (Section 2 of RFC 6481, especially Figure 1). This PP contains the
> ROAs, a manifest, and a CRL, router certs, and any subordinate CA
> certs. The CRL that lives here would contain an entry for a cert
> associated with a manifest if that cert were revoked, and the
> manifest contains an entry for the CRL. If one were to encounter a
> CRL that contained an entry for the EE cert in the manifest, that
> would indicate an error by the CA.

That wasn’t quite the case I was thinking of (although I think this is
a case worth calling out specifically to avoid confusion), but rather
a case where say, a ROA’s EE certificate contains a URI for the CRL
Distribution Point that is referring to a CRL that is not listed in the
issuing CA’s manifest.

So, when validating the EE certificate of a signed object, the CRL
could be missing in the manifest and it could be present in the
manifest but missing in the sense that it can’t be downloaded from the
publication point.

I think this is a case worth clarifying since it seems to currently be
left to local policy.

> Tim has suggested that, but to do so would make the RPKI not
> consistent with RFC 5280.

I’m not sure. In section 6.1.3, RFC 5280 says

|  (3)  At the current time, the certificate is not revoked.  This
|       may be determined by obtaining the appropriate CRL
|       (Section 6.3), by status information, or by out-of-band
|       mechanisms.

Determining certificate revocation via lack of presence on a manifest
could be construed as an "out-of-band" mechanism.

> > content of the PP is ignored if any object is missing, corrupted,
> > or invalid.  
> This is not consistent with your suggestion that the CRL can safely
> be ignored.

Sorry, indeed. Any signed object or certificate.

> I'm not sure why the group-oriented approach is useful, expect
> perhaps for ROAs. Why would one elect to ignore some router certs if
> one were missing?

Hm, true. So perhaps this should be defined per object type? For ROAs
(and in the future ASPAs) partial sets cause hard to debug issues
whereas for router keys and ghostbuster records, having to discard
single objects of a set shouldn’t cause any harm.

> > If the aim is to avoid risking accidentally marking routes as
> > invalid, this should probably also extend to all information
> > published by child CAs.  

I was thinking of the possible case that the parent issues a ROA for a
resource but also delegates it to a child CA which then issues
additional ROAs for the same resource. If the parent ROA is missing,
you cannot rule out that the was indeed the case, so in order to be
safe, you’d have to discard all ROAs for prefixes associated with the
parent. 

> both a CRL and a manifest, it has a serious operational problem.

Indeed. However, bugs exist and unexpected things have a tendency to
happen. I still fail to see how, given a strict adherence to the
manifest, the CRL provides additional value for objects published
within the RPKI repository. So for robustness, it may indeed be
preferable to remove this additional possibility for breakage.

> > repository and essentially moves its role over to the manifest. The
> > CRL would still be considered for objects published outside the
> > repository, such as the proposed Resource Tagged Assertions.  
> 
> cached, still valid objects for the PP, and insisting on contacting
> the CA or PP maintainer.

If the manifest takes over the CRL’s responsibility of expressing
revocation, you’d open a window to replay revoked objects. So this
translates to the question: What would you do today for a stale or
missing CRL?

Kind regards,
Martin

---

From: Martin Hoffmann <[email-masked]>
Date: Thu, 16 Apr 2020 17:55:31 +0200

Job Snijders wrote:
> > > exchange, impersonate the rsync server, or manipulate the file
> > > system of the legitimate rsync server in such a way that they can
> > > signal to an rsync client to delete existing files but not to
> > > replace them partially or completely. 
> > > 
> > > What am I missing?  
> > 
> > 6486 4.2
> > 
> >    fileList:
> >       FileAndHash entry for each currently valid signed object that
> > has been published by the authority (at this publication point).
> > Each FileAndHash is an ordered pair consisting of the name of the
> > file in the repository publication point (directory) that contains
> > the object in question and a hash of the file's contents.
> I think Martin's perspective is that whether the files are deleted or
> modified by MITM, the RP outcome is the same: the PP has to be
> tossed. I don't think Martin suggested that files can trivially be
> replaced, just that deletion or tampering have the same effect. 
> 
> Right?

Correct. This was solely about synchronisation of a publication point,
not yet about validation.

Kind regards,
Martin

---

From: Randy Bush <[email-masked]>
Date: Thu, 16 Apr 2020 09:53:58 -0700

> I think I see an instance of miscommunication:

doh.  sorry.

randy

---

From: Stephen Kent <[email-masked]>
Date: Thu, 16 Apr 2020 13:43:45 -0400

Martin,
> Distribution Point that is referring to a CRL that is not listed in the
This would be an error by the CA. RFC 6480 notes where the CRL is 
supposed to live in the repository system.
> I think this is a case worth clarifying since it seems to currently be
I'm all in favor of reminding folks that there are a lot of details that 
an RP has to check, which is why Di Ma and I co-authored the 
recently-approved Informational RFC on this topic. This  is an example 
of a CA not following the requirements established in one of the many 
applicable RFCs, and I suspect we may have failed to cite this in the 
document in question.
> could be construed as an "out-of-band" mechanism.
This part of 5280 is designed to accommodate OCSP as a revocation status 
distribution mechanism, something that was not part of X.509. The RPKI 
requires CAs to publish CRLs, and it states where the CRLs are to be 
published in the repository system (RFC 6480).
> single objects of a set shouldn’t cause any harm.
That sounds like a good argument.
> safe, you’d have to discard all ROAs for prefixes associated with the
So, the parent issues a ROA binding a range of addresses to an AS that 
the parent holds. But then  the parent delegates the same range to a 
child, which is free to bind the addresses to a different AS (that the 
child holds). Thus a route advertising either AS as the origin is valid. 
If the parent's ROA is missing, the AS bound to it is no longer 
considered valid, but the child's ROA is. So, why does one have to 
discard/ignore all other ROAs associated with the parent?
> preferable to remove this additional possibility for breakage.
As I noted in my reply to Tim a while ago, if we want to view the 
manifest as the only critical determinant of whether a cert is revoked, 
then why bother checking the validity interval for the EE certs? Why not 
just say that if an EE cert is listed in a manifest by a a CA, then the 
CA is declaring it to be valid, period? One can argue that remembering 
to reissue certs with new validity intervals is also a burden for a CA 
and creates another opportunity for the CA to post inconsistent info at 
a PP.
> translates to the question: What would you do today for a stale or

I don't understand your example. If a manifest trumps a CRL, then 
revoked objects would not appear on a current manifest and so replay of 
such objects would be ignored, right?

Steve

---

From: Stephen Kent <[email-masked]>
Date: Thu, 16 Apr 2020 13:43:45 -0400

Martin,

Steve

---

From: Stephen Kent <[email-masked]>
Date: Thu, 16 Apr 2020 13:43:55 -0400

Claudio,
> verification. This is the chicken and egg situation I was referring to.
If the MFT has been updated, but the CRL has not, then an RP has an 
existing CRL against which to check the EE cert in the MFT object. So, 
the case you cite need not always arise. Only when the CRL and MFT 
change at the same time, does the situation you noted arise. If the MFT 
makes use of a single-use cert, and if the MFT is being published at the 
Next Update time, there also is no problem, since the cert expired with 
the MFT and would not be subject to revocation. Thus, it appears that 
the only cases in which the issue you cited arises is if a new MFT is 
published before the Next Update time and/or the MFT EE cert is not 
single-use. Agreed?
> large part of Germany's IP space unless all of the manifest data is
I was puzzled by the opening sentence, i.e., "For a CRL listed in 
manifests ..." since _every_ CRL is listed in a manifest.

Job has been emphasizing cases where one of more ROAs are listed in a 
manifest and are not present at a PP. Your comment said "If a CRL, ROA 
or CWERT in a manifest is missing ..." which is a much broader statement.

We're trying to analyze a set of complex cases; it helps if we can make 
precise statements about the issues of concern.

Steve

---

From: Stephen Kent <[email-masked]>
Date: Thu, 16 Apr 2020 13:43:55 -0400

Claudio,

Steve

---

From: George Michaelson <[email-masked]>
Date: Fri, 17 Apr 2020 08:48:12 +1000

If a ROA object can be removed and a manifest proves it has been
removed, and a CRL confirms it has not been revoked, the unknown
question is:

 * what was the semantic intent of the ROA?

If you have a previously acquired state of the ROA which meets the
manifest checksum/sig and its not in the current CRL *ITS NOT MISSING*
and you know its semantic intent. All is good. This is what
maintenance of local cached state of fetching achieves.

If you do NOT have a previously acquired state of the ROA, you cannot
know its semantic intent. Job showed me that this means a covering
aggregate ROA state which is superceded for a specific prefix can be
invalidly interpreted as applying to the more specific. Any more
specific. The semantic intent of unknown amounts of ROA covered space
defined at this level in the CA hierarchy cannot be stated
categorically because the missing ROA might modify any of them.

Therefore, the ROA states of this level of the hierarchy and children,
cannot be known categorically.

IFF (and the FF is important) you don't have a valid ROA state cached,
and you can detect from valid CRL and valid Manifest some ROA is
missing, you cannot know how it modifies the routing intent in the ROA
otherwise unaffected.

I think therefore, you have to stop processing. But the IFF part is,
if you do not have a valid prior-state fetch of the ROA. Just because
"its missing" is not sufficient. If the one you have before refetch
maches what the Manifest says is a signed state, you aren't missing
anything.

-G

---

From: Tim Bruijnzeels <[email-masked]>
Date: Fri, 17 Apr 2020 10:05:55 +0200

> On 17 Apr 2020, at 00:48, George Michaelson <[email-masked]> wrote:
> 

IFF.. I had to look that one up.. it stands for 'if and only if', right?

> maches what the Manifest says is a signed state, you aren't missing

Well, I think I agree and it's what I have been trying to say. If an object is missing on a specific synchronisation job, but you still have an old copy: you know the URL, you know the hash. So if a current MFT refers to it, you can just use it.

A thought I had on this is that RPs may consider treating withdraws (be it rsync or rrdp) with prejudice. One could argue that an RP should only forget an object seen when it has seen a positive statement that it's no longer relevant:

Object becomes relevant:
- CA cert signs MFT listing object by URI and hash
- The object matching the URI and hash is retrieved (rsync or RRDP, it does not matter how)

Object becomes irrelevant:
- CA cert signs a new MFT where the object URI and hash combination is removed; AND
- The object was indeed validly signed under the CA cert
  (otherwise you open a poisoning window, I could list and delist your objects)

So, do not just remove the object because of rsync '--delete' or a withdraw or update (publish with hash) seen in RRDP. Wait for a signed confirmation.

However, I understand that this will introduce quite some complexity that may not work with existing implementations. There may also be other, better ways. For example you could keep a reference counter of MFTs referring to an object by URI and hash, and delete the object only once:
- it was marked for deletion (RRDP) and
- the reference count is zero.

Well at this point I should probably leave it to the people who write the actual validation software to say something sensible - or shout at me if you will. Just trying to contribute, don't mean to tell you how to code :D

Tim

---

From: Martin Hoffmann <[email-masked]>
Date: Fri, 17 Apr 2020 10:26:06 +0200

> maintenance of local cached state of fetching achieves.

A problem with this strategy is that it makes the resulting set of
validated data depend on the point in time a cache did its last
synchronisation. That is, a cache which has seen the ROA before it went
off the manifest will produce a set of VRPs different from one that has
either never synchronised the repository before or due to some sequence
of updates at the publication point had removed the ROA as legitimately
deleted.

The question is: Are we fine with that? I am speaking from a
perspective of someone who has to maintain and in particular test
software and I would rather we had a validation strategy that depended
only and exclusively on the data present after the last synchronisation.

That also motivates my proposal that an object counts as revoked as
soon as it disappears from the manifest.

Kind regards,
Martin

---

From: Martin Hoffmann <[email-masked]>
Date: Fri, 17 Apr 2020 10:54:08 +0200

Stephen Kent wrote:
> Martin,
> > ...
> >
> > That wasn’t quite the case I was thinking of (although I think this
> > is a case worth calling out specifically to avoid confusion), but
> > rather a case where say, a ROA’s EE certificate contains a URI for
> > the CRL Distribution Point that is referring to a CRL that is not
> > listed in the issuing CA’s manifest.  

I know I am being pedantic (but then implementers are surprisingly
creative), but I can’t find normative text in RFC 6480 either that says
there must only be one CRL per CA.

> > Determining certificate revocation via lack of presence on a
> > manifest could be construed as an "out-of-band" mechanism.  
> This part of 5280 is designed to accommodate OCSP as a revocation
> status distribution mechanism, something that was not part of X.509.

If it intended to only allow OCSP, it should have said so. The way it
is formulated now, it opens up the option to devise any appropriate
mechanism.

> The RPKI requires CAs to publish CRLs, and it states where the CRLs
> are to be published in the repository system (RFC 6480).

The RPKI can be changed based on implementation and operational
experience. This ought to be done in a backwards compatible way, of
course, for instance by still requiring the publication of CRLs but
not considering them in validation of in-repository signed objects. 

> > I was thinking of the possible case that the parent issues a ROA
> > for a resource but also delegates it to a child CA which then issues
> > safe, you’d have to discard all ROAs for prefixes associated with
> > the parent.  
> So, the parent issues a ROA binding a range of addresses to an AS
> that the parent holds. But then  the parent delegates the same range
> to a child, which is free to bind the addresses to a different AS
> (that the child holds). Thus a route advertising either AS as the
> origin is valid. If the parent's ROA is missing, the AS bound to it
> is no longer considered valid, but the child's ROA is. So, why does
> one have to discard/ignore all other ROAs associated with the parent?

Because if the actual route is from the ASN authorised by the child,
the announcement suddenly becomes invalid instead of unknown. Given the
operational split between object creation and object publication, it
doesn’t necessarily have to be the fault of the child that that
happened.

> >> Personally I believe that if a CA cannot manage to correctly
> >> generate both a CRL and a manifest, it has a serious operational
> >> problem.  
> manifest as the only critical determinant of whether a cert is
> revoked, then why bother checking the validity interval for the EE
> certs?

The same reason we check them with CRLs: the possibility of replay
attacks of the revocation source.

Using the manifest for revocation has the exact same issues as using a
CRL. All it does is drop the need to maintain two objects instead of
one.

> revoked objects would not appear on a current manifest and so replay
> of such objects would be ignored, right?

The question was about _today_, as in with the current situation where
a signed object’s certificate remains valid even if the object dropped
off the manifest, what do you do if a current CRL cannot be obtained?

The mechanism from RFC 5280 specifically states that it relies on you
being able to get the current CRL, so it cannot be used at all in this
case. What would your guidance be to me as an implementer of a relying
party software today? Even if I allow users of my software to choose
what to do (aka "local policy"), I need to pick a default that pretty
much everyone will then use.

Kind regards,
Martin

---

From: Randy Bush <[email-masked]>
Date: Fri, 17 Apr 2020 10:07:20 -0700

> If an object is missing on a specific synchronisation job, but you
> still have an old copy: you know the URL, you know the hash. So if a
> current MFT refers to it, you can just use it.

but something is broken

---

From: Stephen Kent <[email-masked]>
Date: Thu, 23 Apr 2020 10:46:04 -0400

Martin,

sorry to be so tardy in replying.
> there must only be one CRL per CA.

In some cases there will be more than one CRL issued by a CA, e.g., 
during key rollover. But I don't think that's the real issue here. The 
text associated with Figure 2 in 6480 (page 13) says "... the SIA 
extension of certificate A points to _the_ repository publication point 
of CA A's subordinate products, which includes certificates B and C, as 
well as _the_ CRL issued by A. The CRL Distribution Points (CRLDP) 
extension in certificates B and C both point to _the_ CRL issued by A." 
That text, and Figure 2 itself argue that, under normal circumstances 
(vs. key rollover or alg transition), there is just one CRL associated 
with a CA and it lives at the same PP as the other subordinate products 
signed by the CA.

> is formulated now, it opens up the option to devise any appropriate
I agree that the wording is very broad; the principle author of 5280 
liked it that way. But, as the PKIX chair my view is that the text was 
intended primarily to accommodate OCSP. There are several places in 5280 
where the principle author bends over backwards to be as accommodating 
as possible.
> not considering them in validation of in-repository signed objects.

I agree that operational experience is an important factor in deciding 
how to evolve the RPKI specs. But, I don't believe that some sloppy 
practices by some CAs should cause us to make major changes. In the Web 
PKI we have had a lot of cases where CAs have failed to behave properly, 
e.g., not issuing CRLs in a timely fashion. For a long time browsers, 
the principle RP software in this context, were content to ignore such 
errors. Fortunately, most browsers not provide warnings and require 
explicit user actions to override these errors. As a result, I think the 
cognizant CAs have become better.

> doesn’t necessarily have to be the fault of the child that that
OK, I see your point.
> CRL. All it does is drop the need to maintain two objects instead of
I don't think your argument above is sound. My point is that the 
argument for ignoring CRLs (or not generating them) is that one wants to 
avoid redundant checks, because a CA might fail to correctly maintain al 
of the signed objects that it generates. That argument could equally 
well be applied to checking the validity interval for the EE cert in a 
ROA. This is not a replay issue; a valid, current manifest can be used 
to verify that each ROA, including the embedded EE cert, is intact and 
current.
> off the manifest, what do you do if a current CRL cannot be obtained?
Some additional context is needed. If a cert has not expired, and if one 
has a current CRL, then the cert is valid irrespective the presence or 
absence of the corresponding signed object in a manifest. However, if 
the corresponding signed object is absent from tghe current manifest, 
then an RP should not rely on that object.
> case.
Ah, you've added the detail that a not stale CRL is unavailable. In that 
case the cert cannot be validated and should be ignored.
> What would your guidance be to me as an implementer of a relying

If an RP does not have access to a not-stale CRL for a CA then the RP is 
unable to validate any certs that would have appeared on the CRL. So, 
all signed products of the CA in question ought be rejected.

Steve

---

From: Stephen Kent <[email-masked]>
Date: Thu, 23 Apr 2020 10:46:04 -0400

Martin,

OK, I see your point.

Steve

---

From: Rob Austein <[email-masked]>
Date: Tue, 28 Apr 2020 00:37:36 -0400

Apologies for coming into this very late, I had not been tracking the
discussion until Randy kicked me earlier today.

...

The DRL validation engine maintains separate caches for:

* Unverified latest stuff fetched from the net;
* Stuff that passed RPKI validation last time; and
* Stuff that passed RPKI validation this time.

rsync's behavior is really only relevant to the first of these caches.
Even back when we kept all objects as separate disk files (DRL
"rcynic" versions 1 and 2) we kept separate caches, we just used a lot
of hard links (in part to avoid putting further strain on filesystems
which made bad assumptions about block/inode allocation ratios...).

rcynic version 3 ended up stuffing everything but the unverified cache
into a database, because adding RRDP support changed the internal
search requirements enough that keeping everything as one disk file
per RPKI object no longer made sense.

In all cases, the basic algorithm remained the same: we walk the tree
from the trust anchors down, looking for URIs from which we need to
fetch. If we can fill everything the manifests tell us to expect from
current data, great, otherwise we check the objects that passed
validation last time before giving up and amputating portions of the
tree.  Yes, this meant that fetch and validation are interleaved and
that validation sometimes has to pause while waiting for fetch.

Overall, this approach seems to work very well, in the sense of
pulling together what looks to the RP like a coherent view of the
world, and recovering automatically from minor synchronization glitches.

RRDP simplifies this a bit, since it tends to reduce the number of
discrete publication events on the CA side and gives the CA something
closer to a transactional publication mechanism.

> implement in a wide range of circumstances.

Your RP's view of the world is never going to be exactly the same as
my RP's view of the world.  This is just life with a distributed
database.  DNS and BGP aren't globally coherent in that sense either,
they're just (usually) close enough.

> caches running the same software.[0]

Correct.  Again, this is just life with caching and a distributed
database.