Subject: Re: [Sidrops] 6486bis: Failed Fetches

From: Jay Borkenhagen <[email-masked]>
Date: Wed, 2 Sep 2020 14:33:53 -0400

Steve,

Stephen Kent writes:
 > The crux of the issue is whether it is preferable to reject all objects 
 > at a pub point if an error is detected (until the error is rectified), 
 > and thus potentially treat many more routes as having an unknown 
 > verification status, or to accept bits and pieces of data from a pub 
 > point with the potential that some routes will be treated as valid, even 
 > if they are invalid. To me, that is a value judgement to be made by 
 > network operators, not by developers of RP software. [...]

No, there is a bit more to it than that.

I want my network's routers and those of all other networks to be fed
with VRPs that represent all the verified and legitimate published
origin-ASN intentions of verified, legitimate IP number resource
holders.

Clearly the challenge there is in determining "verified" and
"legitimate."  In creating the rules (to be documented in
standards-track RFCs) for those determinations, I would like the input
of all SIDROPS participants to be valued, including the input of the
RPKI designers, the CA- and RP-implementers, the RIRs, and the network
operators.

To my point: if RP-implementers feel that the "bits and pieces of
data" (to use your words) should be considered to best make those
determinations, such input should be welcomed; conversely, if
RP-implementers feel that strict publication rules and accompanying
strict handling of retrieved records together create the best way to
make those determinations, we all should listen to what they have to
say.

 > Experience with X.509 cert extensions suggests that the approach I noted 
 > is preferable. Saying that older RPs can ignore objects they don't 
 > understand introduces ambiguity in RP behavior, which is precisely the 
 > concern that motivated the effort to generate 6486bis.

Ambiguity is bad in any technical endeavor where precision is needed.
And we're here discussing this now because there was ambiguity in
rfc6486 (not assigning blame -- I missed it, too).  But I don't see
how it follows that if/when ASPA (for example) is standardized, that
older RPs should be required to throw away all records retrieved from
PPs whose MFTs now cite ASPA records.

Thanks.

						Jay B.

---

From: Tim Bruijnzeels <[email-masked]>
Date: Fri, 4 Sep 2020 22:40:46 +0200

Hi,

> On 2 Sep 2020, at 20:33, Jay Borkenhagen <[email-masked]> wrote:
> 
> Steve,
> 
> Stephen Kent writes:

I don't think it's entirely fair to say that us 'developers' are willing to create security issues for network operators.

This document is still under discussion, and there simply are still cases that are not fully discussed, e.g. how to filter VRPs if a publication point is found to be bad - to which I posted a suggestion only last week. I thought that discussion phases were meant for that. To look at things from different angles. I am quite certain though that consensus can be reached if we have constructive talks, and that implementation can follow quickly when it does.

To the point of "bits and pieces". Rejecting a full publication point is the easiest to reason about, so I can see its appeal. That said, consider the case where a ROA is invalid because it contains a prefix no longer held by the CA. Instead of rejecting the complete PP for a CA certificate, and filtering out all VRPs overlapping the resources on it, one could also reject all the invalid ROAs and only filter out VRPs that have any overlap with those ROAs. I understand this is a bit more elaborate, but not dramatically different. As far as I can tell it is just as secure in that this cannot result in RPKI invalid announcements because of a single invalid ROA object.

Note that I talk about ROA objects that can be fetched, and are known to be current. I.e. the actual manifest content. And provided they can be parsed and validated so you know which resources are affected. If any objects are missing or obsolete then assume the worst - I actually made statements to that effect months ago.

In any case, I can live with the reject-all model for its simplicity. But I would like to hear an operator input on whether this is indeed desirable. And that the consequence of child CAs - even NIR sized ones - blinking out for a few hours every now and then is an acceptable price for this simplicity.

Tim

> _______________________________________________
> Sidrops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidrops

---

From: Tim Bruijnzeels <[email-masked]>
Date: Sat, 5 Sep 2020 01:13:56 +0200

> On 4 Sep 2020, at 22:40, Tim Bruijnzeels <[email-masked]> wrote:
> 
> Hi,
> 
> To the point of "bits and pieces". Rejecting a full publication point is the easiest to reason about, so I can see its appeal. That said, consider the case where a ROA is invalid because it contains a prefix no longer held by the CA. Instead of rejecting the complete PP for a CA certificate, and filtering out all VRPs overlapping the resources on it, one could also reject all the invalid ROAs and only filter out VRPs that have any overlap with those ROAs.

This should have said: overlap with the intersection of resources on those ROAs and the CA certificate.

> I understand this is a bit more elaborate, but not dramatically different. As far as I can tell it is just as secure in that this cannot result in RPKI invalid announcements because of a single invalid ROA object.
>

---

From: Stephen Kent <[email-masked]>
Date: Sat, 5 Sep 2020 14:07:17 -0400

Jay,
> ...
Sorry, but I beg to differ on this point. My judgement of what is 
critical to the "best" operation of an ISP is not as valuable as that of 
an experienced network operator. On the other hand, my judgement on what 
constitutes secure operation in a PKI context, is, IMHO, more valued 
than that of most folks. We each have our strengths and limitations and 
these ought to be taken into account when valuing inputs to a standard 
like this. That's what I expect the WG chairs to do when reviewing 
comments on the list.
> PPs whose MFTs now cite ASPA records.

I quickly reviewed the ASPA ID. I didn't see a description of how the 
RPKI is used to validate these newly-proposed, signed objects.  Until 
that is made clear, it isn't clear why an ASPA even belongs in the RPKI 
repository system. So, your question seems premature.

Steve

---

From: Stephen Kent <[email-masked]>
Date: Sat, 5 Sep 2020 14:08:49 -0400

Tim,
> ...
> In short: Given that I expect that we are never come to a consensus other than the total reject-all-if-one-fails - I can live with this if we must, but don't let it come as a surprise.
Works for me.
>> ...
>> Allowing RPs to ignore object types they don't understand prevents a CA from being able to convey the notion that a new object type is important (to that CA). I don't think this is a good strategy. It means that RP behavior will be ambiguous relative to new object types.
> So far none of the objects have seemed to need this flag.
If the goal is to define a very general scheme, then a flag may be 
necessary to accommodate objects not yet defined, since we cannot yet 
know whether a uniform response for all such objects will be 
appropriate. I agree that this need not be a binary flag, as in cert 
extensions. You suggested a three-value flag below, for example.
> ...
>> If we want to have a consistent and flexible approach to accommodating new objects I suggest the strategy I mentioned earlier. Define an additional SIA URI that points to a pub point (and manifest) where we can introduce the next version of the signed object format, one that includes a critical flag, analogous to X.509v3 extensions. This allows each CA to decide which object types have to be processed  by an RP in order for the whole pub point to be accepted vs. rejected. Note that this will require modifying a lot of RFCs, but it is a flexible, extensible approach to this issue.
> I agree that it's flexible and extensible. I had not thought of this approach.
>
> But it is a lot of work, not just in RFCs, also in code. It also raises questions about how and when old PPs without the new objects can be deprecated. You can give operators more time to upgrade, but at some point plugs will probably be pulled? Maintaining multiple PPs indefinitely seems rather wasteful.
>
> I would like to hear what others have to say.. I have the feeling that ASPA is getting close, and I would really not like to see it delayed because of this.
It will take a while to complete another revision of the manifest doc, 
if we purse additional changes, and even longer before an RFC is 
approved and published. So ASPA will not be accommodated quickly. Also, 
as I noted in my reply to jay, a quick read of the ASPA doc didn't 
indicate how these new objects are validated using the RPKI.
> If we do go down this road then I think that we should also look at the manifest object itself, and let it convey which object (types) are critical (and while we are at it, we can specify types instead of using filename extensions). That way future object types could introduced more easily perhaps - this obviously needs more discussion but it could even allow for semantics like: 1) new object please test, don't use, 2) new objects, use if you can, 3) new objects, critical - fail if you don't understand.

One could combine the new SIA URI and a revised manifest, in which the 
manifest contains the per-object flag, rather than redefining the basic 
object format to accommodate the flag. That would reduce the number of 
RFCs that need to change. Good idea.

Your example bothers me a bit- it seems to argue for CA-directed 
processing flags, perhaps to accommodate experimentation with new object 
types. This sounds like adopting elements of the IRR DB model which 
didn't seem to be so great, IMHO.

Separately, I think we need to make GBR mandatory for all pub points. If 
the intent is to cause RPs to contact a CA/pub point maintainer when 
errors are encountered, then we need to be confident that RPs know who 
to contact and how.

Steve

---

From: Stephen Kent <[email-masked]>
Date: Sat, 5 Sep 2020 14:14:44 -0400

Tim,
> This document is still under discussion, and there simply are still cases that are not fully discussed, e.g. how to filter VRPs if a publication point is found to be bad - to which I posted a sugge…
Sure.
> In any case, I can live with the reject-all model for its simplicity. But I would like to hear an operator input on whether this is indeed desirable. And that the consequence of child CAs - even NI…

In revising 6486 I iterated through several versions, and received 
feedback that the less strict versions were not consistent with the 
perceived) WG consensus. That's how we arrived at the current version of 
the document. Simplicity is certainly a virtue wrt security, so more 
"elaborate" approaches to RP processing of repository data do concern 
me. Also, if a lot of RP software is having a hard time implementing the 
current set of validation procedures correctly, adding more complexity 
seems like a bad idea.

Steve

---

From: Tim Bruijnzeels <[email-masked]>
Date: Mon, 7 Sep 2020 13:17:25 +0200

Hi

> On 5 Sep 2020, at 20:08, Stephen Kent <[email-masked]> wrote:
> 
> Tim,
>> ...
> If the goal is to define a very general scheme, then a flag may be necessary to accommodate objects not yet defined, since we cannot yet know whether a uniform response for all such objects will be appropriate. I agree that this need not be a binary flag, as in cert extensions. You suggested a three-value flag below, for example.
>> ...
> It will take a while to complete another revision of the manifest doc, if we purse additional changes, and even longer before an RFC is approved and published. So ASPA will not be accommodated quickly. Also, as I noted in my reply to jay, a quick read of the ASPA doc didn't indicate how these new objects are validated using the RPKI.

https://tools.ietf.org/html/draft-ietf-sidrops-aspa-profile-02#section-4

The text is still a bit short, and can probably use some feedback, but I believe that the direction is clear. Informally: It's an RPKI signed object, that contains a "customer" ASN which MUST be in the EE certificate. The EE certificate MUST of course be valid.

In my perception ASPA is quite close to testing (no not in the production repositories of course).

> One could combine the new SIA URI and a revised manifest, in which the manifest contains the per-object flag, rather than redefining the basic object format to accommodate the flag. That would reduce the number of RFCs that need to change. Good idea.

Upon reflection I realised that even the introduction of an SIA in the issuing CA certificate will lead to issues. RPs would reject the CA certificate, and as a result the whole PP of the parent CA. This means that the SIA cannot be deployed without leaving a significant number of RP installations behind. E.g. if I run a delegated CA under an RIR which wants to adopt ASPA, and I get a new CA certificate with the additional SIA from my parent, then 1000s (RIPE NCC >12k) other CAs will also be rejected.

So the only option then is to wait for a significant number of updated RP tools to be available, and then at some point take the plunge and let CAs deploy - and tell operators who suddenly find that their VRP count drops to near zero that they should (have) upgrade(d).

This poses a serious problem to future RFCs - I just don't see how they can achieve an incremental deployment that does not result in the issues above.

I understand your desire to keep 6486bis focused - and I sympathise. However, I do believe that it needs to specify some plan. Perhaps an SIA can be reserved? Perhaps it can be published in conjunction with ASPA - timelines permitting - postponing the pain? Perhaps just checking the presence and hashes for unknown object types, but not validating them can be reconsidered? (I heard your objections, just listing it again in this context)

I am quite open to other suggestions. But, if there can be no plan that facilitates future RFC deployments, then I believe that at the very least the document should include a section that discusses this issue (perhaps security considerations, or a separate section altogether).

> Your example bothers me a bit- it seems to argue for CA-directed processing flags, perhaps to accommodate experimentation with new object types. This sounds like adopting elements of the IRR DB model which didn't seem to be so great, IMHO.

Not where I meant to go, but this can be a future discussion.

Tim

> 
> Separately, I think we need to make GBR mandatory for all pub points. If the intent is to cause RPs to contact a CA/pub point maintainer when errors are encountered, then we need to be confident that RPs know who to contact and how.
> 
> Steve

---

From: Tim Bruijnzeels <[email-masked]>
Date: Tue, 29 Sep 2020 16:54:58 +0200

Hi,

> Upon reflection I realised that even the introduction of an SIA in the issuing CA certificate will lead to issues. RPs would reject the CA certificate, and as a result the whole PP of the parent CA…

I have done some testing on this.

Section 4.8.8.1 of RFC 6487 seems to say that additional SIA OIDs for CA certificates can be expected and can be ignored. And indeed it seems that all current RP software will accept (and ignore) additional SIAs.

At least that's the result from adding an extra SIA in a Krill branch and running the end to end tests using the following validator versions:

fortvalidator 1.4.0
OctoRPKI v1.1.4 (2019-08-06T16:51:07-0700)
rcynic version believed to be buildbot-1.0.1544679302
routinator 0.7.1
rpkiclient 6.7p1
rpkivalidator3 3.1-2020.08.06.14.39

I am still not very happy about the overhead that this approach implies:

Additional publication points which need to be maintained, which may be out of sync. What if MFT A has a certain ROA and MFT B doesn't? This can lead to serious operational impact if an announcement would be valid under one, but not the other.

It would also require much more fundamental code changes for producing CAs - one cannot just support a new object type. One has to support an additional publication model. Parent CAs have to be willing to include the new SIAs as well affecting the ability to deploy for delegated CAs.

But, at least it seems that it can be introduced without breaking things immediately for existing deployed validators.

Tim

---

From: Stephen Kent <[email-masked]>
Date: Tue, 29 Sep 2020 20:12:01 -0400

Tim,

Thanks for investigating the new SIA option.

I'm not wedded to an alternate directory solution. A change to the 
Manifest structure to indicate how to process each named entry might be 
a good alternative. But, either approach will require a complex 
transition strategy that will take considerable effort to design and 
document.

The WG needs to decide if the highest priority is to remove ambiguities 
from the current spec, or to pursue a new, substantially different spec. 
Hopefully the Thursday telecon will help clarify the priorities.

Steve

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 30 Sep 2020 02:55:42 -0400

First, thanks! having some tested results on an optional path (for
newly added types) seems cool.
it's at least informed some of the conversation, which seemed to go a
bit outside the original point of order (in my re-re-reading anyway)

I think the salient points in the discussion are really (happy to be
corrected, of course):
  1) the -bis is proposing a simpler handling of publication point collection.
      this simplicity is a bit at odds with some Internet principals
(robustness principal), but
      trades this for very clear behaviors in the RP software.
operations folk (myself, job, mikael though I missed his mail...
others)
      were a bit concerned with the different flavors of 'correct'
that can pop out the RP :( and see that being simpler about
      the flavors/options may make all/MOST RP get the same set of
answers... predictability / consistency are nice.

  2) the CA software and operations have to be very clear now about
what is/not published,
      keeping in mind that mistakes in the CA Publication Point will
have repercussions.
      Perhaps these need to be clearly articulated in the -bis draft?
(more clearly?)
      This will, I think, clearly have impact on both CA / PP software
and on operations of the CA / PP.
      I think this is good, actually as we're (sidrops folks i mean)
attempting to push for:
          o  closer compliance to consistency
          o  more reliance upon the whole system
          o better overall security for the routing system

  3) With the proposed simplicity/strictness comes some potential
problems, at least:
      o problems in a CA / PP mean that CA / PP may not have their
most updated information (routing intent) available
      o continual problems will lead to fallback to 'unknown' in the
routing system
      o adding new object types seems difficult (just my reading so far)
         particularly there's a note about 'people won't upgrade their
RP software'

I think that the simplicity is a goal I would like to see achieved,
specifically because inconsistent results in the RP set is problematic
longer term.
If we can get to more consistency before we roll a bunch more out
we'll be able to course correct better/faster (I think).

I think the idea that people won't upgrade... sure? maybe? Also: "Do
you  like killing kittens? because not managing your business critical
systems a great way to kill kittens." We've been able to push more
people into RPKI and better routing-intent hygiene, I don't see why we
can't keep doing that and get folk to treat part of the routing system
as critical to their business.

We should not shy away from a solution just because we may have to
convince people to upgrade.

I look forward to the discussion tomorrow/thurs.

-chris