Subject: Re: [Sidrops] what to do when the CRL is hosed?

From: Christopher Morrow <[email-masked]>
Date: Wed, 1 Apr 2020 14:32:57 -0400

On Tue, Mar 31, 2020 at 6:21 PM George Michaelson <[email-masked]> wrote:
>
> I think you did capture the spirit.
>
> I would remind people that *all* the products in a repository are
> signed objects. To arrive at a state where a manifest is "wrong"

this was the meat of the first part of my message:
  "We planned from the get-go that transport security wasn't available,
    and instead object security would be required."

This brought along some, perhaps implicit, requirements about the
state of the repository and objects contained therein.

> demands two things: it doesn't reflect some real-world situation
> regarding contents of the repository *and* its signed by keys you can
> validate back to a trust anchor. If you cannot validate the manifest,
> then its not "wrong" its forged, or invalid cryptographically.  To me,

I agree with this,

> a  "wrong" manifest checks out cryptographically but somehow is not
> coherent with the state of the repository. If objects are on the
> manifest but cannot be fetched, you are probably suffering either from
> hiding of objects, or a transitional state of publication. If objects
> can be found which are not on the manifest, you may be seeing
> artifacts which are not defined as critical (must be found) or, the
> manifest has not yet been updated.

This part isn't really what we started talking about, but bears some
discussion I think. The manifest is 'must be there' objects in the repository.
If there are manifest listed objects which are not in the fetch... that
repository is in a bad state and probably operations should decide
what to do about that.
  o Do we have guidance?
  o Does that guidance provide a list of actions/tradeoffs?
  o will resolution come in time (for your particular problem)

I think most of the docs so far are less definitive ;) about this.
This, is one of the things that I think the discussion/wg should provide
in a more clear manner.

As an ops group it seems reasonable for us to do this even,
we're pointing out a difference between theory and practice and
documenting a fix.

> There are potentially transitional states in the production of a large
> repository under eventually consistent production (e.g. multiple
> asynchronous cooperating processes, dual-redundant signing models)
> where what can be fetched and what is on manifest do not align.  If we

Part of the agreement between repository and RP is really, I think:
  "This repository once published is correct"
  (note, not: 'eventually consistent')

This isn't really said in the docs either, and since we let all the flowers
bloom here, and kept saying:
  "the repositories are going to be inconsistent for a bit"

I think we didn't help repository operators pick 'sane' models for
management of their repository content over time.

> wish to demand that the manifest and a given state of the repository
> *always* align, we need to formalise that in a way which makes it
> clear we publish repository state in as close to atomic-update as
> possible. (copying a repo to a new dir, modifying the contents of the
> new dir, and (re)publishing the change through the rename() call in
> UNIX is one such model)

This was, actually, part of the point I made 'years back' about management
of repositories... that lead to the model I (admittedly 'puppet for
dummies' style)
put together for managing the test repository work setup.

I think it'd be worth document (as part of the outcome of the proposed
conversation) what 'contract' we expect the respository operations and
relying parties to have.. 'contract' in a software expectations sense, not
in a legal-beagle sense.

The next bits step into the 'given case X guidance from the WG for
operators says 'do Y'.

> A missing manifest is not a "wrong" manifest -its a sign of data being
> hidden from you either because of transitional effect in publishing
> the repository, or for persisting state a problem in the repository
> (diskfull?) or, production systems, or a MITM attack. Which do you
> think is actually most likely at this current time? This is no
> different to a missing CRL in that regard. Which do you think is the
> most likely reason, because you cannot a-priori know that its a MITM
> attack, or a failure in networks, or storage systems, or production
> systems.
>
> During the early design phases of the system, we determined that since
> the products were all cryptographically signed, there was no strict
> requirement for cryptographic protections on transport. If that
> decision needs to be revised I think it should be done in standards
> work, here, as a discussion. I do not yet see a document which says
> that. I don't see formalisms which go to a normative change in SHOULD
> to MUST on this.
>
> So these comments aside, my plea is that people clearly state when
> they say "wrong" or "bad" if they mean that it is cryptographically

yes, loose lips (phrasing) sinks ships (discussions).
I'm trying to be more clear, if I'm not please say:
  "You are unclear here, more words pls!"

> valid, but does not align with some external reality, or some other
> meaning. (Which btw, is complex because the part which none of this
> can adequately capture is *intentionality* -Just because you think a
> publication of RPKI state is nonsensical does not mean its not
> intentional)
>
> Attack models need to state clearly how they acheve the state. An
> attack on the integrity of the manifest needs to explain coherently
> how it creates a manifest which checks out cryptographically, and yet
> hides or legitemates something which should not exist. They also need
> to explain how the specific attack on the manifest in these situation
> outweighs other considerations: If they have the keys, then surely the
> attack vector is to make validly signed things which cannot be
> detected as an attack?
>
> -George
>
> On Wed, Apr 1, 2020 at 1:42 AM Christopher Morrow
> <[email-masked]> wrote:
> >
> > first, apologies for getting back around to this so late :(
> >
> > On Thu, Mar 26, 2020 at 10:57 AM Stephen Kent
> > <stkent=[email-masked]> wrote:
> >
> > > So, I think discussing MITM attacks is a distraction, unless we have examples of how
> > > such attacks can affect RPs in ways different from attacks on repositories. Maybe re-reading
> > > RFC 8211 would be useful, as it tries to analyze a range of possible "adverse actions"  by CAs or
> > > repository managers in the RPKI context, and discusses how RPKI mechanisms are intended to
> > > detect/counter these actions.
> >
> > In the case of the incident which started this thread we ended up
> > publishing part of the content a
> > repository needs to publish such that the relying parties can verify
> > properly that the content in the
> > repository is correct/valid/usable. The discussion then went along a path like:
> >    1) "well... maybe we shouldn't have belt and suspenders?" (manifest AND crl)
> >    2) "what happens if we don't publish this pesky CRL? and rely only
> > on the manifest?"
> >    3) "what if we don't publish the manifest and only rely on the CRL?"
> >    4) "CRL + Manifest has made 'rp software' hard/buggy"
> >
> > In the world where the protections specified for RPKI exist:
> >   1) self contained content protection (roa / ee-certs / etc are
> > packaged securely)
> >   2) crl signed and available in the repository for revocation actions
> > on objects in the repository
> >   3) manifest signed and listing all objects of interest in the repository
> >
> > Steve (kent!) is right mitm is harder to see as a threat.
> >   "All objects you get are signed by a ca-cert which is signed by the
> > root.. which is in the list of TAL you have. You can't have missing
> > objects and you cant' remove objects without affecting the signed
> > manifest"
> >
> > In a world where we remove one/some of the protections:
> >    A) no more manifest
> >    B) no more crl
> >    C) both
> >
> > I think mitm problems are much harder to detect/deal with :(
> > It sounds like WG folk (RP users and RP/CA software authors) are
> > asking for guidance on handling the problem(s) discussed here.
> > It sounds, to me, like a chat at the upcoming interim meeting would be
> > a great place to start that with some slideware and a proposal to use
> > as kindling.
> >
> > I think the shape of the conversation is roughly:
> >   "What would be the effect (on the routing system) for RelyingParties
> > if we decided to be less strict about CRL existence?"
> >   "What would be the effect (on the routing system) for Relying
> > Parties if we decided to be less strict about repository contents vs
> > Manifest contents?"
> >   "What happens to the routing system if the manifest and crl are
> > either/both 'broken' in a repository?"
> >
> > I don't think it matters much to the routing system where the breakage
> > occurs (my repository or RIPE/ARIN/etc) certainly there's more fallout
> > from ARIN/RIPE/etc, but... you can't get to me either way :)
> > (possibly) until repair and propogation.
> >
> > Thoughts on some slideware and discussion?
> > Did I about capture the meat of the sandwich here?
> >
> > -chris
> > co-chair but asking as a regular chemical engineer at this party.
> >
> > _______________________________________________
> > Sidrops mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/sidrops

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 1 Apr 2020 14:35:03 -0400

On Tue, Mar 31, 2020 at 11:03 PM Di Ma <[email-masked]> wrote:
>
> Yes.
>
> Although the RPKI provides object-oriented security yet the validity of an object relies on a trust chain, any wrong node of which would cause failure.
>
> So-called “wrong” MFT will bring about trouble no matter it is rooted from transport or RPKI provisioning side.
>
> Situations could be complicated as analyzed in section 2.2 of RFC 8211 in terms of adverse actions on MFT but RFC 6488 leaves much to RP to decide.
>
> BTW, in RFC 8211 we authors differentiate if the attacker have got access to the private key of the INR holder, trying to give a threat mode.
>
> So it is right time to reconsider the requirements posed on RP in case of “wrong” MFT.

yes this last sentence is the discussion I'd like to see happen... I
think guiding that with:
  "Here are the ways an manifest could go badly: bad sig, bad content,
missing content, missing manifest..."

and:
  "given the damage from X, Y, Z, the proposed actions for an RP
are... because ..."

> Di
>
> > 2020年4月1日 06:20，George Michaelson <[email-masked]> 写道：
> >

---

From: Robert Kisteleki <[email-masked]>
Date: Thu, 2 Apr 2020 10:02:42 +0200

>   "given the damage from X, Y, Z, the proposed actions for an RP

I was thinking along the same lines while having a related conversation
with Job/Tim here:

> I believe a more nuanced approach is needed, like if there's a problem
> on a particular validation path (a cert is missing or has an error) then
> invalidate that path, if a CRL is missing then warn but use a stale one,
> but leave the otherwise unaffected bits validated.

If repository R hosts objects for CAs A, B, ... Z, A has subtrees
A1...A9 and there's a problem with the manifest of A1, the rest (B..Z)
is perfectly fine an usable. IMO A2..A9 are fine as well. As for A1: the
RP action IMO should really depend on what the exact issue is, for example:
- an object is missing: check if you have it from the previous run and
if so, use it
- a CRL is missing / corrupt / expired: warn but use the previous one

I'd be happy to contribute and I think that if this happens,
contribution from operator(s) would be extremely useful.

Robert

---

From: Robert Kisteleki <[email-masked]>
Date: Thu, 2 Apr 2020 10:07:15 +0200

> new dir, and (re)publishing the change through the rename() call in

Being smart on the repo side surely decreases the probability of a
hickup, but I don't think it can completely avoid point-in-time
inconsistencies. So the RP has to be prepared to deal with this.
Assumptions about a repo always being self-consistent will not hold up,
even without an attacker around.

(I also recall that, at least in theory, there is such a thing as a
"repository service" where one can use a private CA but someone else's
repo to publish. Such a scenario compicates the consistency requirement
even further.)

Robert

---

From: Martin Hoffmann <[email-masked]>
Date: Thu, 2 Apr 2020 11:46:06 +0200

George Michaelson wrote:
> coherent with the state of the repository.

I think we are still having this conceptually backwards: It isn’t the
manifest that is wrong, it is the repository content. Whether it was
originally intended that way or not, having a manifest at all to me only
makes sense if it is considered to contain the truth, the whole truth,
and nothing but the truth.

I honestly believe that the current state of affairs where basically
the guidance for RP software developers is to look at the current
repository content, past repository content, the manifest and the CRL
and then figure stuff out themselves is rather ... unfortunate and
doesn’t exactly improve the confidence in the overall system.

Given the inherent complexities of a distributed repository, it seems
important to me to make the system as straightforward as possible to
implement. Declaring the manifest to be the sole list of currently
published objects is a means to this end.

Kind regards,
Martin

---

From: Claudio Jeker <[email-masked]>
Date: Thu, 2 Apr 2020 12:14:20 +0200

> published objects is a means to this end.

I completely agree with this. As example rpki-client only looks at files
that are referenced via the .mft files. The .mft file tells the truth (it was
validated that it is the truth with the cert chain). Only files in the
MFTs will be used to build the VRP tree (and only if their hash is
correct). There is no concept of previous repository state or some other
file that is per accident around.

-- 
:wq Claudio

---

From: Job Snijders <[email-masked]>
Date: Thu, 2 Apr 2020 10:18:34 +0000

> published objects is a means to this end.

I agree, straightforwardness is crucial here. rpki-client has now
implemented mitigations as following:

When the manifest is valid (valid in the sense that the signatures check
out, the CRL is not expired, the manifest file format is correct), and
it references .roa files which are either missing from the repository,
or where the .roa file's hash does not match with the hash listed in the
manifest, rpki-client will not output any VRPs that could potentially be
extracted from the remaining .roa files at that level of the tree.

At the MITM layer one roa is hidden from the view:

    job@mieli:~/fake-ripe-repo$ rm repository/DEFAULT/3e/01d411-d915-4277-8fe2-76b0dda2bf3e/1/LkKeUPYrfgzjsOIejLjsHGk44cU.roa

After this partial witholding attack, the validator under attack will emit this error:

    rpki-client: rpki.ripe.net/repository/DEFAULT/3e/01d411-d915-4277-8fe2-76b0dda2bf3e/1/1-tcQDnftkRnWbiMhu2cR1-dgmCs.mft: referenced file LkKeUPYrfgzjsOIejLjsHGk44cU.roa: No such file or directory

which results in no VRPs being emitted from any .roa files that
1-tcQDnftkRnWbiMhu2cR1-dgmCs.mft referenced:

    job@anton ~$ grepcidr 80.128.0.0/11 /var/db/rpki-client/openbgpd
    job@anton ~$

The above is the *safe* outcome, because the attacker is now limited to
the ability to flip BGP announcements from 'valid' to 'not-found',
instead of 'invalid' (in case the validator has not been patched).
Scenarios where routes flip to 'invalid' represent hard outages. The
operator's understanding of the RPKI is that when things go wrong in the
RPKI, BGP routes potentially change state to 'not-found'. 

In other words, because LkKeUPYrfgzjsOIejLjsHGk44cU.roa is missing, the
following .roa files are *not* parsed:

    repository/DEFAULT/3e/01d411-d915-4277-8fe2-76b0dda2bf3e/1/IM3xMiMU1QChuWMOGgnbDwYolrU.roa
    repository/DEFAULT/3e/01d411-d915-4277-8fe2-76b0dda2bf3e/1/NpI_Uj3OZb6jvwfUTX0-eOk9QV4.roa
    repository/DEFAULT/3e/01d411-d915-4277-8fe2-76b0dda2bf3e/1/fIeCcC8KpdJQd-olU2APdxNZkyA.roa
    repository/DEFAULT/3e/01d411-d915-4277-8fe2-76b0dda2bf3e/1/r5vtD4isKhTzGXNGulSnZ7XCS04.roa
    repository/DEFAULT/3e/01d411-d915-4277-8fe2-76b0dda2bf3e/1/r7TSyWn_GbYPjNWvt4r5ewSNAsk.roa

Even though they are referenced in 1-tcQDnftkRnWbiMhu2cR1-dgmCs.mft
(valid), and the remaining .roa files are valid themselves.

Proceeding to output VRPs derived from say 7TSyWn_GbYPjNWvt4r5ewSNAsk.roa
while LkKeUPYrfgzjsOIejLjsHGk44cU.roa is missing is akin to "garbage in
garbage out". Manifests are truth. A repository can contain /more/ files
than referenced in the manifest files, but not *less*.

Kind regards,

Job

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 2 Apr 2020 10:18:46 -0400

> - an object is missing: check if you have it from the previous run and

I would like to believe that whether a repository is 1 organization
(one repository only vaslii!)
or an organization that is hosting many repositories, the story about
CRL/MFT is the same,
at the single repository level.

In practical terms this means that RIR who host 'large numbers of
repositories for their customers'
are really just running a slew of little items. Any breakage at the
individual level is indistinguishable
from that which a self-hosted ca/repository may make. At the top of
the tree, yes there are more
possible ways to get broken, and the effect on the locally-hosted (or
self-hosted far away!)
repositories from breakage up-tree is certainly bad... but I'd rather
not get lost in that discussion.

> contribution from operator(s) would be extremely useful.

ok, I think for the upcoming interim let's see if we can have a
strawman to discuss :)

---

From: Jay Borkenhagen <[email-masked]>
Date: Thu, 2 Apr 2020 16:48:03 -0400

Hi,

Re-visiting an older message in preparation for the upcoming SIDROPS
virtual interim meeting.

On 23-March-2020, Stephen Kent writes:
 > I agree that more uniform processing is desirable, but in the routing 
 > system the notion of local policy seems to be ingrained, so ...

When I first saw this, I thought Steve was just making a pithy comment
about local policy and network operators.  But a friend thinks Steve
had not meant it as a joke at all, so let me take Steve's comment at
face value and respond:

As a network operator, yes, I expect to have the flexibility of
configuring my network using local policy.  But in the context of the
RPKI, we need to reach a point where all RFC-compliant RPs will
generate identical sets of VRPs when presented with the same input.
There will be reasons why I may prefer to operate one RP over another,
but the VRPs that a particular RP generates should never be one of
them.

No network operators want to specify any 'local policy' on their RP
instances to control under which conditions an RPKI object would be
accepted.  Some network operators in this WG have voiced strong
opinions about how such acceptance should be decided, but that's not
because they want the ability to exert local policy there.  Such
opinions are all toward the goal we share of all implementations
reliably creating a correct/consistent VRP set.

I encourage feedback from other network operators on this one simple
point.  Hopefully we all see this 'local policy' matter the same way,
and the WG can turn its attention to consistent handling of the corner
cases.

Thanks.
	
						Jay B.

---

From: Randy Bush <[email-masked]>
Date: Thu, 02 Apr 2020 14:32:05 -0700

> reliably creating a correct/consistent VRP set.

you have been testing RP compliance to this for a good number of years,
and having quiet chats with RP implementors.  this has really helped and
is much appreciated.

randy

---

From: Lukas Tribus <[email-masked]>
Date: Fri, 3 Apr 2020 01:04:55 +0200

Hello,

> and the WG can turn its attention to consistent handling of the corner

As a network operator, and without commenting on low-level aspects, I
expect (TL;DR: all the good and none of the bad stuff please):

- all RP's to produce the same VRP sets, given the same input
(otherwise we slow down detection of bugs, misconfigurations and
attacks), I suggest to NOT implement different code paths and local
policy exceptions.
- the end-result of rsync vs RRDP to be the same, in every case, under
every attack scenario
- no MITM attacks possible that lead to incomplete VRP sets (with the
supported retrieval methods, which still includes rsync)

I wanna be able to run different RP's in my network, without my
routers behaving differently based on whether they are currently
considering RP implementation A or RP implementation B.

I also expect some outages of RPKI infrastructure in the coming years,
just as we had outages in the past and just like other ecosystems have
(DNS, DNSSEC, TLS, etc). I consider that acceptable and realistic,
realizing that there are some scenarios where this will not only cause
NOTFOUNDs, but also false-positive INVALIDs. That's really bad, but it
also will be very rare (losing a subset of ROAs at IRR level). But I
would strongly encourage not to weaken or simplify the checks that are
in place, despite being aware that hosting the repository is hard
(atomic updates and all). Maybe we could come up with some best
practices documentation about setup, maintenance of a repository
(BCP?) for repository operators.

I would like for the ecosystem to stay extendable if someday we want
to add something other than ROA to it, and I would still like it to be
cryptographically reliable and consistent in that (and every other)
case.

I know it's not that black and white, I realize I am grossly
oversimplifying. But if I think about TLS or the WebPKI in general,
lax rules, simplifications and fallbacks always turned out to be a bad
idea in the end.

lukas

---

From: Randy Bush <[email-masked]>
Date: Thu, 02 Apr 2020 17:28:31 -0700

>   supported retrieval methods, which still includes rsync)

you are asking for transport security.  we chose not to take that path.
if an evil monkey gets in the middle, it should be detectable.  but the
design does not prevent mitm.

randy

---

From: Martin Hoffmann <[email-masked]>
Date: Fri, 3 Apr 2020 09:22:15 +0200

Randy Bush wrote:
> > - no MITM attacks possible that lead to incomplete VRP sets (with
> > the supported retrieval methods, which still includes rsync)  
> 
> you are asking for transport security.  we chose not to take that
> path. if an evil monkey gets in the middle, it should be detectable.
> but the design does not prevent mitm.

It may also be worthwhile noting that this sort of thing is always a
trade-off. If we too quickly discard suspicious data, it becomes all
that much easier to disable origin validation as step one of a malicious
route hijack.

Kind regards,
Martin

---

From: Lukas Tribus <[email-masked]>
Date: Fri, 3 Apr 2020 09:32:26 +0200

Hello Randy,

> design does not prevent mitm.

What I'm trying to say is something different:

We promised transport agnostic security. I'm asking that we maintain
that promise.

But if we are unable to maintain that promise and start relying on
transport security, then we need to actually fully admit that and drop
rysnc.

This is because I read some arguments on this list akin to "well this
attack will not work against RRDP, so only rsync ...". What I am
saying is that: we should maintain the same guarantees and
consistencies at the end of the day with both retrieval methods.

I understand MITM - as in denying the *entire* service - is an attack
that affects both retrieval methods. That's ok. What would be
unacceptable for me is that a repository retrieved via rsync could be
affected by a partial withhold attack, withholding some VRP's.

The TL;DR here is, as a network operator, I expect both retrieval
methods to provide the same kind of assurances, consistencies and
security guarantees. On the other hand, if rsync is a second class
citizen, then we need big fat warnings about rsync.

Lukas

---

From: Tim Bruijnzeels <[email-masked]>
Date: Fri, 3 Apr 2020 10:12:36 +0200

Hi,

> On 3 Apr 2020, at 09:32, Lukas Tribus <[email-masked]> wrote:
> 
> Hello Randy,
> 

I believe that TLS Verification for RRDP MUST become mandatory. No, it's not perfect, it's not a replacement for the *object* security that exists in RPKI, but it's an additional check. In particular it can help to flag if an RP is misdirected or a MITM might be in play.

With regards to phasing out rsync. I tried to start that discussion at IETF 106, and made an initial write-up:
https://datatracker.ietf.org/doc/draft-sidrops-bruijnzeels-deprecate-rsync/

I have not yet asked for WG adoption - I forgot in the midst of lots of other work etc. That said, I can do so. I am happy to accept co-authors as well. My impression from the room (not wanting to put on any chair hats) is that there is no strong opposition to phasing out rsync.

However, this discussion is somewhat orthogonal to what RPs should do in case they find issues with a MFT, CRL, or know that there are objects that are missing. These problems exist regardless of the transport.

Tim

---

From: Robert Kisteleki <[email-masked]>
Date: Fri, 3 Apr 2020 10:20:36 +0200

> However, this discussion is somewhat orthogonal to what RPs should do
> in case they find issues with a MFT, CRL, or know that there are
> objects that are missing. These problems exist regardless of the
> transport.
I agree. Job pointed out a behavioural example from a tool, and we have
evidence that the RP tools behave differently under different
conditions. If, as Lukas wrote: "[I expect] all RP's to produce the same
VRP sets, given the same input" is a desired goal, then we likely need a
document specifying said behaviour.

Robert

---

From: Martin Hoffmann <[email-masked]>
Date: Fri, 3 Apr 2020 10:32:53 +0200

Robert Kisteleki wrote:
> > However, this discussion is somewhat orthogonal to what RPs should
> > do in case they find issues with a MFT, CRL, or know that there are
> I agree. Job pointed out a behavioural example from a tool, and we
> have evidence that the RP tools behave differently under different
> conditions. If, as Lukas wrote: "[I expect] all RP's to produce the
> same VRP sets, given the same input" is a desired goal, then we
> likely need a document specifying said behaviour.

Ideally, these would be a 6486-bis and 6487-bis with more detailed
instructions on validation. There’s already quite a number of
documents and it isn’t always clear which ones are important. I’d
prefer to have this in the document an implementer will definitely read.

I’ll be happy to help working on these bises.

Kind regards,
Martin

---

From: Stephen Kent <[email-masked]>
Date: Fri, 3 Apr 2020 11:06:30 -0400

OJay,
> about local policy and network operators.
I often strive for pithy, but ...
> But a friend thinks Steve

Thanks for the clarification. If the operator community can agree on a 
consistent way to deal with inconsistencies in downloaded pub point data 
I think that would be ideal.

I'll post some suggestions and an outline of the decision points, shortly.

Steve

---

From: Randy Bush <[email-masked]>
Date: Fri, 03 Apr 2020 09:07:19 -0700

> we should maintain the same guarantees and consistencies at the end of
> the day with both retrieval methods.

i am with you there

randy