Subject: Re: [sidr] WGLC: draft-ietf-sidr-origin-ops

From: Christopher Morrow <[email-masked]>
Date: Sun, 13 Nov 2011 23:03:55 -0500

Checking back on this... I see that Randy had rev'd the document since
this last conversation-set ... Danny has 2 editorial changes and 1
'large' comment... I don't yet see any feedback on those, but the
previous set of comments/requests are taken care of to the original
peoples' satsifaction?

I suspect some feedback to Danny will come soonish, but can we close
out the other set of requests?

-chris

---

From: Randy Bush <[email-masked]>
Date: Mon, 14 Nov 2011 12:30:13 +0800

> out the other set of requests?

one of danny's ed comments confuses me.  NotFound is a keyword.

danny's substantial comment, re marking of stuff originated internally 
within one's own AS, is 
  o in pfx-validate, as it is protocol,
  o has been raised by ops in workshops, and
  o is being madly discussed offline as the router implementors have
    some problems with side-effects of marking them

i have been holding -13 in my edit buffer for some weeks, tweaking as
clues filtered in from mailing list.  i can drop it now, as i presume
the flood gates have opened, or hold it until later in the week.

randy

---

From: Shane Amante <[email-masked]>
Date: Mon, 14 Nov 2011 13:36:41 +0800

Hi Chris, Randy,

> out the other set of requests?

I don't recall seeing a response to the last e-mail I sent, nor do I see anything in the latest -12, related to this doc:
http://www.ietf.org/mail-archive/web/sidr/current/msg03475.html

-shane

---

From: Danny McPherson <[email-masked]>
Date: Mon, 14 Nov 2011 01:07:48 -0500

On Nov 13, 2011, at 11:30 PM, Randy Bush wrote:

> NotFound is a keyword.

I assume it was derived from the normative pfx-validate draft and was 
simply hoping for consistent use:

danny@pork% grep -i found draft-ietf-sidr-pfx-validate-03.txt
   peer will be found to have one of the following "validation states":
   o  Not found: No ROA Covers the Route Prefix.
   //Initialize result to "not found" state
   result = BGP_PFXV_STATE_NOT_FOUND;
   //"not found" applies to this validation operation.
   will result in "not found" state and the prefixes will be advertised
   also result in "not found" state.  The implementation is expected to

"Not found" != "NotFound", that's all...

Makes searching and dependency simpler.

-danny

---

From: Randy Bush <[email-masked]>
Date: Mon, 14 Nov 2011 14:20:05 +0800

>    also result in "not found" state.  The implementation is expected

talk to those authors :)

chris checked roa-validation and now we have "unknown" as well.

oh goodie

randy

---

From: Danny McPherson <[email-masked]>
Date: Mon, 14 Nov 2011 01:24:27 -0500

> out the other set of requests?

Chris, 
I'm not sure I understand the request, can you clarify?  

I.e., until I've had adequate time to review updated I-Ds with changes 
incorporated in context it's hard to determine if we can close on those 
issues.

-danny

---

From: Christopher Morrow <[email-masked]>
Date: Mon, 14 Nov 2011 01:29:02 -0500

> I'm not sure I understand the request, can you clarify?

can try :)

> incorporated in context it's hard to determine if we can close on those

there were a slew of changes (or a slew of comments made) requested, a
document update happened ~13 days ago, did the changes account for the
comments/requests or not?

The outstanding (today) new request I get won't be dealt with yet, but
it seems from the  conversation between you/randy that it's moving in
the right direction, no?

-chris

---

From: "George, Wes" <[email-masked]>
Date: Mon, 14 Nov 2011 01:41:15 -0500

> From: [email-masked]

> comments/requests or not?

[WEG] I diffed 11 and 12 when 12 came out, and no, not really. As I recall, Shane already replied that the proposed text (that made it into -12) was not adequate in explaining rationale around the recommendation of "close" for RPKI cache placement, among other things. (cf. emails on 10/31). Don't know if additional changes are pending for -13 based on those comments, but the mail thread died on the 31st with no further responses.

Wes George

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

---

From: Randy Bush <[email-masked]>
Date: Mon, 14 Nov 2011 14:45:52 +0800

> 1)  From Section 3:
> ---snip---
>    A local valid cache containing all RPKI data may be gathered from the
>    global distributed database using the rsync protocol, [RFC5781], and
>    a validation tool such as rcynic [rcynic].
> ---snip---
> 
> Would it be possible to mention and/or point to how the above process
> is supposed to be bootstrapped?  IOW, is it expected that,
> eventually?, the RIR's are going to publish to their end-users and
> maintain URI's of RPKI publication points?  Since this is an Ops
> guidelines document, some guidance and/or pointers are likely to save
> [lots of] questions down the road.  I'm not expecting this to be a
> tutorial document, but some idea on the theory of how a new SP
> bootstraps their cache(s) would be helpful.

that is software dependent.  relying party software and how it decides
to deal with the global rpki is very software dependent.  e.g. tim's
varies from rob's, and we have some really wild ideas to deal with the
issues of reliable distributed publication.

it has been suggested that you may be asking how the inter-publication
stuff is to be bootstrapped at the global rpki level.  this is the sia
pointer in the cert.  when i tell arin i want my space and will publish
at uri://randy.foo, they put the cert in their pub point with an sia
pointer to uri://randy.foo

> 2)  Given that, to my knowledge, the RPKI is [very] loosely
>     synchronized in a "pull-only" fashion, shouldn't there be some
>     text added below to that effect that: 
>     a)  It may not be best to go more than, say, 2 levels of RPKI
>     caches deep inside a single organization/ASN to avoid RPKI caches
>     from being out of sync with each other?

considering the timings we are seeing in operation, like a few seconds
to fetch from one cache to another, as compared to minutes to load from
the global rpki (see rob's preso at iepg) and my comment above, this may
be 180 degrees out.  inter-cache fetching may be far better.

but code will vary.

>     b)  Operators should look at running more aggressive
> synchronization intervals _internally_ within their organization/ASN,
> from "children" (2nd-level) RPKI caches to the 'parent' (top-level)
> RPKI cache in their organization/ASN, compared to more "relaxed"
> synchronization intervals to RPKI caches external to their
> organization (top-level RPKI caches in their ASN to RIR's)? 

kind of assumed in current text, but i can add something.

> ---snip---
>    Validated caches may also be created and maintained from other
>    validated caches.  Network operators SHOULD take maximum advantage of
>    this feature to minimize load on the global distributed RPKI
>    database.  Of course, the recipient SHOULD re-validate the data.
> ---snip---
> While I'm here, I don't think the text in Section 6, "Notes",
>    addresses the above concerns, at all.  In fact, I find it extremely
>    unhelpful to just dismiss this concern, out of hand, with the text:
>    "There is no 'fix' for this, it is the nature of distributed data
>    with distributed caches".  We know what the answer is here

for some values of "we."

>    you tune the synchronization intervals to strike the appropriate
>    balance between [very] tight synchronization vs. increased load on
>    the systems being synchronized.

could you reference an example, a paper, an algorithm, ...?  we seem to
think differently about how easy this is.

>    I find it hard to believe a simple suggestion such as this is not
>    proposed in the text, even including the phrase "the suggested
>    values for such synchronization are outside the scope of this
>    document, but will likely be subject to further studies to
>    determine optimal values based on field experience".

ahhhh.  so we don't know.  i say that frequently!

    It is hoped that testing and deployment will produce advice on
    relying party cache loading and timing.

> 3)  Granted, the following text is only a "SHOULD", but the text
> offers no reasoning as to why caches should be placed close to
> routers, i.e.: are there latency concerns (for the RPKI <-> cache
> protocol), or is it that a geographically distributed system is one
> way to avoid a single-point-of-failure, or something else entirely?
> As a start, just defining "close" would help, e.g.: same POP, same
> (U.S.) state, same country, same timezone 

i seem to get in trouble every time i do so.  it's an ietf thing.

> but, then a statement as to any latency or resiliency requirement for
> geographic deployment of RPKI caches wold be useful.

so far, actual timings seem not to be very latency sensitive, probably
because they have been done with freebsd on fat wires.

as i said on this thread on nanog, this is why we get the big bucks.  it
is a multi-dimensional design issue.

>     Furthermore, given the [very] loosely synchronized nature of the
> RPKI, should the text point out that the number of RPKI caches
> (internal to the organization) be balanced against the potential need
> of an organization to maintain a more tightly synchronized view,
> across their entire network, of validated routing information?

i have no idea.  you seem to think inter-cache sync dominates timing far
more than i do.  i suspect that publishing party timing dominates.

so send text.

> A concern might be that if routers in Continent A pull information
> from their RPKI caches that tell them that ROA is not "Invalid", but
> other routers in Continent B are still using 'older' information in
> RPKI caches in Continent B that says the same ROA is either "Not
> Found" or "Valid", then the result might be that BGP Path Selection
> swings all traffic from Continent A to Continent B.  At a minimum,
> this could lead to substantially increased latency or, at worst,
> congestion, packet-loss or a unintended DoS.

yep, that is a concern.

how deeply do we want to get into explaining all the issues surrounding
distributed data?  at this point i am tempted to point to a basic cs
text on the subject.

>     In the short-term, the LOCAL_PREF Attribute may be used to carry
> both the validity state of a prefix along with it's Traffic
> Engineering characteristic(s).  It is likely that the SP will have to
> change their BGP policies such that they can encode these two,
> separate characteristics in the same BGP attribute without negatively
> impacting their existing use or leading to accidental privilege
> escalation attacks.  

thanks for text

> 5)  I have three comments on the below:
>     a)  It's not clear, to me, what is meant by "internal metric"
>     below.  Do you mean MED or IGP metric or something else?  I don't
>     see IGP metric as being practical, so I'm assuming you mean
>     additively altering MED (up|down) based on validity state.
>     Regardless, I would recommend you state more precisely which BGP
>     Path Attribute you're referring to below. 
>     b)  Since MED is passed from one ASN to (only) a second, downstream ASN to influence ingress TE policy, is it "OK" from a security PoV that MED is a *trusted* means to convey ROA validity information from one ASN to a second?  Presumably, the answer should be "heck, no", right?  If that's the case, then wouldn't it be wise to state that:
>         i)  MED's, encoded with any ROA validity information, should get reset on egress from an ASN to remove said validity information and only carry TE information, as appropriate; and,
>         ii) MED's should not be trusted on ingress to convey any meaning with respect to validity information?
>     c)  What is meant by the statement, "might choose to let AS-Path rule"?  Is your intent to state that an SP may choose to just use MED, which follows after LOCAL_PREF & AS_PATH in the BGP Path Selection Algorithm, as a means to determining validity of a particular prefix?  If so, then it would be much more clear if you just stated that, e.g.:
> ====
>     If LOCAL_PREF is not used to convey validity information, then MED
>     is likely the next best candidate BGP Attribute that can be used
>     to influence path selection based on the validity of a particular
>     prefix.  As with LOCAL_PREF, care must be taken to avoid changing
>     the MED attribute and creating privilege escalation attacks.

      Other providers may not want the RPKI validation result to be more
      important than AS-path length -- these providers would need to map
      RPKI validation result to some BGP attribute that is evaluated in
      BGP's path selection process after AS-path is evaluated.  Routers
      implementing RPKI-based origin validation MUST provide such
      options to operators.

> 7) Is this document only intended (scoped?) to cover PE's that can
> (or, eventually, will) speak the RPKI-RTR protocol for validation?

yes

randy

---

From: Christopher Morrow <[email-masked]>
Date: Mon, 14 Nov 2011 02:06:46 -0500

> [WEG] I diffed 11 and 12 when 12 came out, and no, not really. As I recall, Shane already replied that the proposed text (that made it into -12) was not adequate in explaining rationale around the…

ok, I was/am looking for status. I think randy replied to shane's note
(or another note form shane) a bit ago, and talks some about 'close'.
I believe that a new version just arrived in the repository as well.

-chris

---

From: Shane Amante <[email-masked]>
Date: Mon, 14 Nov 2011 18:45:09 +0800

Hi Randy,

Thanks for the response.  I think we're getting closer.  See below.

> pointer to uri://randy.foo

Yes, I'm asking about the latter.  More specifically, what I've been attempting to ask here is how one configures, in one's _local_ RPKI cache (that syncs to the outside world), /where/ the RIR's publication points are on Day 1.  Do I contact one RIR (which maintains a list of other RIR's publication points) -or- each RIR individually to ask what is their publication point?  (If you can help provide an answer as to what is the expectation on the operator, I can then potentially help to provide text).

> be 180 degrees out.  inter-cache fetching may be far better.

I'm going to put this comment aside for the moment, since I think the issue below wrt the desire (or, need?) to deploy RPKI servers across the whole network may be a better way to answer the above.

> kind of assumed in current text, but i can add something.

OK, thanks.

>    relying party cache loading and timing.

OK.  How about the following tweak:
---snip---
It is hoped that testing and deployment will produce advice on
RPKI cache loading, intra-domain and inter-domain RPKI cache synchronization
timing, router to RPKI cache synchronization timing, etc.
---snip---

> is a multi-dimensional design issue.

This still hasn't answered the question.  Let me try again.

Money doesn't grow on trees.  Why is it 'advised' (SHOULD) that I deploy a RPKI cache in every single POP throughout my whole network?  Funny thing about business folks that control the budget, they want an answer to this question that is other than "because".  Let me try to now be more constructive.  Is this draft recommending RPKI cache per POP:
1)  For [massive] redundancy/resiliency inside the AS;
2)  To keep CPU (or, network) 'load' down on RPKI caches inside the AS -- even though, above, you've said that this hasn't been quantified, yet?
3)  To keep latency down in order to ensure the fastest possible inter-cache sync time -or- router to RPKI cache fetching time?
4)  To avoid reliance on either IGP -or- BGP within the AS for a router to get access to it's RPKI cache for data?
5)  Other?
6)  All of the above?

Ideally, you can narrow it down to just one answer from above, but if not which of the above are the "most important" and driving this recommendation?

Furthermore, this is not just about the cost of the server HW on top of which one would run RPKI inside the AS.  There is also much more substantial costs associated with [potentially violent] swings of traffic around the network as the "loosely synchronized" (out-of-band) RPKI caches push new RPKI data to routers -or- routers pull new RPKI data from their local RPKI cache that may be different from neighboring RPKI caches inside the same network.  In theory, one way to "solve" this problem is to deploy [much] less RPKI caches inside the AS, thus ensuring that a "large" collection of routers have a consistent view of the same RPKI data.

> more than i do.  i suspect that publishing party timing dominates.

See just below for what is my primary concern.

> distributed data?  at this point i am tempted to point to a basic cs

So, I think there are two issues here:
1)  Inter-RPKI cache synchronization _within_ an AS <-- something an individual operator does have a lot of control over; and,
2)  Inter-RPKI cache synchronization _between_ AS'es (technically, amongst all the authoritative, global RPKI caches in the world) <-- something an individual operator does not have much, if any, control over.

FWIW, my comment above is in regards to #1.  More specifically, with respect to the recommendation to deploy RPKI caches in every POP, globally, there is a [very] large risk (?) that those RPKI caches inside the network will be "loosely synchronized".  Thus, different RPKI caches will be updating BGP policy with their associated routers at much (?) different time intervals.  This results in BGP path selection calculating a new best path and shifting traffic around the network, until _all_ RPKI caches (intra-AS) have updated their associated router, thus getting BGP policy back to an "optimal" state of load-sharing traffic across, say, all connections to a multi-homed customer or peer.

I also think there's substantial operational issues wrt #2, but I don't have a palatable suggestion to offer.

>      implementing RPKI-based origin validation MUST provide such

OK.

>> (or, eventually, will) speak the RPKI-RTR protocol for validation?

OK.

-shane

---

From: Rob Austein <[email-masked]>
Date: Mon, 14 Nov 2011 21:37:04 +0800

At Mon, 14 Nov 2011 18:45:09 +0800, Shane Amante wrote:
> 
> More specifically, what I've been attempting to ask here is how one
> configures, in one's _local_ RPKI cache (that syncs to the outside
> world), /where/ the RIR's publication points are on Day 1.  Do I
> contact one RIR (which maintains a list of other RIR's publication
> points) -or- each RIR individually to ask what is their publication
> point?  (If you can help provide an answer as to what is the
> expectation on the operator, I can then potentially help to provide
> text).

Starting point is most likely one or more Trust Anchor Locator (TAL)
files, see draft-ietf-sidr-ta.  On that glorious day when the RIRs and
IANA have all their ducks in a row, there will be one public TAL for
the root of the promised single tree; in the meantime, you'll likely
have a small collection of TALs.

Where do the TALs come from?  Depends on whose TAL it is.  Some of the
RIRs publish their TALs on their web sites (one RIR, on the other
hand, appears to be hiding the TAL for their pilot system in a locked
filing cabinet in a disused lavatory in a subbasement with a sign
reading "Beware Of Leopard", but that's neither here nor there).
Those of us who write RPKI validation software collect these TALs when
we can find and verify them, and I, at least, include them with my
software.

Ultimately, the problem is the same as distributing DNSSEC TAs, or any
other TA for that matter.  Pretty much by definition, these things
have to be configured outside the automated system, because they're
the bootstrap data.  Inclusion in distributions of software using the
system seems to be the most common way, but one could envision other
methods (T shirts handed out at IETF or *OG meetings, publication in
major newspapers, perhaps as QR codes, invent your own mechanism --
the key point is that grounds for believing the TAL come from outside
the system we're trying to bootstrap).

---

From: Randy Bush <[email-masked]>
Date: Mon, 14 Nov 2011 22:36:17 +0800

> Thanks for the response.  I think we're getting closer.  See below.

i am too stuffed with good food to work tonight.  can you catch me in
the terminal room tomorrow or whenever.  i hang with the rpki interop
testing folk.  we can talk and hack.

randy

---

From: Danny McPherson <[email-masked]>
Date: Mon, 14 Nov 2011 17:57:37 -0500

> the system we're trying to bootstrap).

However, in the interim (until we have a single RPKI root), the origin-ops 
draft should provide some guidance about how an RP should have the 
capability to verify "look-aside" (ugh) what resources an "INR" holds, and 
recommend that they only accept associated RPKI data for those 
resources.  The onus cannot be on the RP to resolve this themselves at 
on a global scale.

The model where each of the TAs in the TAL can assert what it is they're 
authoritative for is even mode broken than the browser/SSL/CA issues 
that we're trying to fix with DANE (the attacker at least has to be on-path 
there, before they consult a compromised CA).

Furthermore, pending the outcome of the discussion in the other thread
I started related to this topic and local TAs, the origin-ops draft should also 
include some discussion about multiple parties involved in LTA-esque 
functions (or extra TALs with "constraints") to preserve inter-domain 
connectivity during putative RPKI override/bypass functions for source, 
destination, and intermediate networks.

-danny