Subject: Re: [Sidrops] [WGLC] draft-ietf-sidrops-roa-considerations-01 - Ends 10/March/2022

From: Randy Bush <[email-masked]>
Date: Sat, 05 Mar 2022 08:59:19 -0800

why i signed on to co-author this document:

fasten seat belts, and take the little yellow pill to think at internet
scale.  the numbers below are fictitious; no prefixes or ASs were harmed
...

i have 420 /24s registered in the repo for my AS 666, folk do seem to
deaggrgate, and we are being taught not to use max-len, eh?

i move one of those /24s to my other pop/continent/whatever, AS 777

we have seen that the CA withdraws the ROA for 420 prefixes on AS 666
and then issues a new ROA for 419 prefixes on AS 666 (and one on AS777)

these two *separate* actions are published in the CA's PP

234 RPs across the internet pick them up

234 RPs across the internet chop the withdrawn ROA into 420 withdraw
VRPs and send the VRPs to their routers

ghod knows what those routers then do to BGP

the 234 RPs chop the new announcement into 419 VRPs and send to their
routers

ghod knows what those routers then do to BGP

and the poor routers are processing all that dren.  and we know the
quality of their implementations has a bit of variance

and i just wanted to move one little /24

and this whole protocol set was designed to minimize impact on routers
so it could run lightly on existing hardware

yes, i am suggesting you publish 420 separate ROAs; they're cheap.  and
then we can act on them individually.

my mommy taught me that it was not polite to 'support' a draft on which
one is a co-author.  but i do suggest others read, comment, and, if they
agree, support.

randy

---

From: Rob Austein <[email-masked]>
Date: Sun, 06 Mar 2022 20:06:32 -0500

I support publication of this draft.

Two comments, for whatever they're worth:

1) In retrospect, the ability to specify multiple prefixes in a single
   ROA seems like premature optimization: it's solving a problem I'm
   not convinced we really have, at the expense of creating a new one.

2) I tend to think about the multiple-prefix ROA problem in terms of
   "fate sharing": unless all the prefixes involved are in some way so
   closely tied to each other that there will never be a change to one
   of them that isn't a change to all, they should be separate ROAs.

---

From: George Michaelson <[email-masked]>
Date: Mon, 7 Mar 2022 11:37:13 +1000

I am not a BGP speaker and I don't forward packets. I no longer have a
role in the Registry RPKI service delivery here at APNIC so in many
respects my comments are superfluous. That noted, I support
publication.

I was asked by one of the authors to comment, otherwise I would have
remained silent given my lack of role in things: On the whole, I think
it is better for the BGP speakers, the RP and CA developers, to be the
voices heard here these days.

I have read this draft, and I think it provides useful information
regarding the distribution of ROA and contents under each TA, as of
last year. It also provides non normative recommendations to single
prefix per ROA. Any quibbles I have to language could be dealt with,
and have no impact on the semantic intent of the document: I would be
amazed if they aren't ironed-out in the publication process anyway.

The analysis shows that many ROA are made by hosted services in two
RIR (RIPE and APNIC) as of 2021 so  we probably need to reflect on
what has to change to stop grouping up the contents, or how to specify
if you want this not to be done as a delegate controlling resources.
This may be OBE anyway, understanding the producer side code
behaviours would be good but doesn't really hinder publication noting
that the document is non normative advice. Thats for other people to
speak to, I no longer occupy the role which has to determine what to
do here.

I think Rob's comment to "premature optimisation" is a good one.
Allowing multiple prefix per ROA was done for good intent, that it
turns out to be counter-productive is a shame, but reality trumps.

I do have some concerns about the impact of unwinding the multiple
prefix per ROA model, but this has to go to scale/time questions not
absolute ones: we can expect the volume of ROA to increase anyway so
this is about changing the rate of update and volume of discrete
objects, and consequence to RRDP delta size, and rsync serial fetch
size/delay which was going to be met anyway in due course.  I think RP
implementations which unpick VRP to then subsequently re-assert
validity possibly need to change to do more of the 'hysteresis'
preserving behaviour in FIB and RIB we see in BGP seeking to limit
change. To me, this goes to the 'all information, not atoms' idea
behind the repository fetch model that we have. It would not be this
document which helped fix or ameliorate that problem so I see no
barrier to publication.

-George

---

From: Tom Harrison <[email-masked]>
Date: Mon, 7 Mar 2022 14:53:27 +1000

On Thu, Feb 24, 2022 at 10:47:16PM +0000, Keyur Patel wrote:
> A working group last call has been issued for
> draft-ietf-sidrops-roa- considerations-01, “Problem Statement and
> Considerations for ROA containing Multiple Prefixes”. Please reply
> to the list with your comments. The WGLC will end on March 10th,
> 2022.
> 
> The draft can be found at:
> https://datatracker.ietf.org/doc/draft-ietf-sidrops-roa-considerations/.

I've read this document and support publication.

-Tom

---

From: Tim Bruijnzeels <[email-masked]>
Date: Mon, 7 Mar 2022 10:48:44 +0100

Hi,

I agree with the general sentiment that one VRP per ROA should be used.

But I think this text is not correct:

>    The potential influence of operations of ROAs containing multiple IP
>    prefixes on BGP routers must be considered.  For the ROA containing
>    multiple prefixes, once increase or delete one <AS, ip_prefix> pair
>    in it, this whole ROA must be withdrawn and reissued.  Through
>    sychronization with repository, Relying Party (RP) fetches a new ROA
>    object and then notify and send all the <AS, ip_prefix> pairs in this
>    ROA to BGP routers.  That is to say, the update of the ROA containing
>    multiple IP address prefixes will lead to redundant transmission
>    between RP and BGP routers.  So frequent update of these ROAs will
>    increase the convergency time of BGP routers and reduce their
>    performance obviously.

This is an issue if, per Randy's example, the entire ROA is withdrawn first,
the RPs fetches and revalidates and sends withdraws for VRPs to the router,
and then the new ROA is published later, validated, etc

Taking the RIPE NCC system, and krill, as examples operators don't get to
do this. They get to specify which *VRPs* they want and the system then
takes care of republishing.. It will publish a new ROA in the same delta
where the previous ROA is either withdrawn or replaced by name. The added
language on failed fetches in the manifest bis further ensures that RPs
will see the complete delta.

RPs will work out what the diff in *VRPs* is. And it's this diff that is
sent to routers. So replacing a ROA with 319 prefixes with one that has
those 319 + 1 new prefix results in a 1 VRP diff.

Similarly if an RP encounter multiple ROAs with the same VRP, then it
will exclude duplicates when talking to the routers.

Another example here.. key rolls.. When a key roll happens all ROAs are
re-issued under a new key. But the VRP set is unchanged.

Re-issuance of a ROA object before it would expire, essentially replacing
it with a new object of the same semantic content: no VRP change.

That said, the following *is* an issue, and in my opinion reason enough:

>    In addition, the validity period of ROA is a long time in default,
>    the prefix ownership change is more possible during this period, this
>    will cause the withdraw or re-issurance of the whole set of prefixes
>    containing within the same ROA

Indeed, resource shrink is troublesome.

(I don't think validity period matters here, as the shrink can happen
at any time)

If VRPS are combined then this could result in the following scenario:
a ROA for 319 is suddenly invalid because of 1 overclaiming VRP - leading
to a drop in 319 VRPs and router churn. Later the child discovers the
shrink and issues a ROA for the remaining 318 VRPs. Resulting in router
churn again.

This is also why krill prefers one VRP per ROA.

So, with single VRP ROAs, a shrink by the parent may cause a child to
temporarily have one, or a couple of single VRP ROAs which are overclaiming,
and then clean them up as soon as it finds out.

Still - this is not the same for the RIPE NCC system. Their hosted system
contains both the parent *and* the child. Context matters. In their case
the child can re-issue ROAs before the parent changes the CA certificate
issued to them - preventing overclaims. They also publish in the same
repository so both changes would be seen together.

Regarding:
>    The extreme case (a single ROA can only contain one IP address
>    prefix) may lead to too many ROA objects globally, which may in turn
>    become a burden for RPs to synchronize and validate all these ROA
>    objects with the fully deployment of RPKI.  So it seems that a
>    tradeoff between the number of ROAs and the number of IP prefixes in
>    a single ROA should be considered.  However, considering the
>    stability and security of RPKI and BGP routing system is the most
>    important target, containing one IP address prefix in a single ROA is
>    recommended if the CA wants to avoids the potential instability and
>    risks.

An example to consider where multi-VRP ROAs may be applicable are the AS0
ROAs for unallocated space that some RIRs publish. There is almost no
risk wrt overclaim - while publishing separate objects would result in
excessive numbers.

Currently krill has a strategy for this trade-off. By default it will
start aggregating when a CA has more than 100 VRPs defined. And it will
start de-aggregating (i.e. start using one ROA per VRP again) when this
number drops below 90. The defaults can be changed in future releases
based on feedback and research.. users can also override them.

If we are to stop using multi-VRP ROAs altogether then this could also
be done, but in my opinion there are contexts where they are not an issue
but a useful optimisation.

To conclude:
- I think the text on router churn needs to be changed.
- I interpret the trade-off discussion to mean that implementers
  can make a decision that in their context multi-VRP ROAs are not
  an issue.
- I am happy to change the default strategy in krill if RP implementers
  are fine with potentially more objects.

Tim

> On 24 Feb 2022, at 23:47, Keyur Patel <[email-masked]> wrote:
> 
> Hi Folks:
>  
> A working group last call has been issued for draft-ietf-sidrops-roa- considerations-01, “Problem Statement and Considerations for ROA containing Multiple Prefixes”. Please reply to the list with your comments. The WGLC will end on March 10th, 2022.
>  
> The draft can be found at: https://datatracker.ietf.org/doc/draft-ietf-sidrops-roa-considerations/.
>  
> Regards,
> Nathalie, Chris & Keyur
>  
>  
> _______________________________________________
> Sidrops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidrops

---

From: Di Ma <[email-masked]>
Date: Tue, 8 Mar 2022 14:26:46 +0800

I think suggestions given by this informational document make real sense with regard to ROA generation.

And it would make no requirement on RP update but bring more ROAs for RPs to fetch, which is a tradeoff.

Granted, it is fair for the community to know about that.

Di

> 2022年2月25日 06:47，Keyur Patel <[email-masked]> 写道：
> 
> Hi Folks:
>

---

From: "chku" <[email-masked]>
Date: Thu, 10 Mar 2022 11:00:06 +0800 (CST)

Keyur,

I have read this draft and think it provides useful information. 
I support publication.

Ching-Heng Ku 

-----Original message-----
From:Keyur Patel<[email-masked]>
To:[email-masked]<[email-masked]>
Date: Fri, 25 Feb 2022 06:47:16
Subject: [Sidrops] [WGLC] draft-ietf-sidrops-roa-considerations-01 - Ends 10/March/2022

Hi Folks:
 
A working group last call has been issued for draft-ietf-sidrops-roa- considerations-01, 𡦖roblem Statement and Considerations for ROA containing Multiple Prefixes_. Please reply to the list with your comments. The WGLC will end on March 10th, 2022.

Regards,
Nathalie, Chris & Keyur

---

From: Randy Bush <[email-masked]>
Date: Wed, 09 Mar 2022 19:53:19 -0800

> and then the new ROA is published later, validated, etc

no need.  CA publishes old an new ROAs.  RP fetches both and processes
sequentially.  oups!

and now we can expect to hear folk suggest that the RP should swallow
and digest the entire ROA set (think of the scale of this a few more
years out), reconcile it all, and only then send the reconciled VRPs
to the router.  if the current state of RP software is any indication,
i would not want to bet my routing on this.

randy

---

From: Ties de Kock <[email-masked]>
Date: Thu, 10 Mar 2022 12:22:24 +0100

> On 10 Mar 2022, at 04:53, Randy Bush <[email-masked]> wrote:
> 

I checked two (StayRTR, routinator) implementations for their behaviour and
they handle this case properly: only the delta in the VRPs after a complete
update is transmitted. If not, imo, that's a bug.

I agree with the information in the draft for de-aggregating VRPs especially for
CAs that do not form one system with their parent CA. In the case where the
parent CA is not under your control, you may not want to have fate-sharing for
all your VRPs.

The risk is mostly there w.r.t. overclaiming, it may be good to spell that out.

-Ties

---

From: Randy Bush <[email-masked]>
Date: Thu, 10 Mar 2022 03:41:15 -0800

maybe i just don't get it.

the CA, in some thought of saving bandwidth or something, packs data
up into AS-based blobs which the RPs are then supposed to unpack and
reconcile.  where's the win that is worth all that work?

and i do not see the RPKI data as having epochs, times when it is
stable.  times when there is a complete snapshot of the data.  data
are being added and removed asynchronously in the distributed tree
cross the net topology.  like bgp, it does not converge to a stable
point; it is constantly in motion.

randy

---

From: Ties de Kock <[email-masked]>
Date: Thu, 10 Mar 2022 12:57:35 +0100

> On 10 Mar 2022, at 12:41, Randy Bush <[email-masked]> wrote:
> 

“less objects, smaller”

It sounds like an attractive optimisation on paper.

I would need benchmark results (with a rir-sized repo with 1-vrp-one-ROA) to see
how much it does in practice.

> point; it is constantly in motion.

My mental model: The RPKI is a directed rooted forest. From a single point of
view, for all CAs, you see a consistent snapshot for a CA (or encounter a failed
fetch).

If there are no split views on the repository content, RPs will eventually see
the same updates.

If a CA shows an intermediate state where part of the VRPs are missing, some RPs
will see that state until their next update.

---

From: Randy Bush <[email-masked]>
Date: Thu, 10 Mar 2022 04:07:26 -0800

> It sounds like an attractive optimisation on paper.

"Premature optimization is the root of all evil (or at least most of it)
in programming.”  -- D Knuth

> I would need benchmark results (with a rir-sized repo with
> 1-vrp-one-ROA) to see how much it does in practice.

i think one would want to measure whole system, from CA repo to the
router

> My mental model: The RPKI is a directed rooted forest. From a single
> point of view, for all CAs, you see a consistent snapshot for a CA (or
> encounter a failed fetch).

[ s/for all/for each/ ? ]

> If there are no split views on the repository content, RPs will
> eventually see the same updates.

except the CA PPs publish asynchronously, on varied schedules, etc.

and the RPs fetch asynchronously.

and the routers ...

randy

---

From: Ties de Kock <[email-masked]>
Date: Thu, 10 Mar 2022 13:33:00 +0100

> On 10 Mar 2022, at 13:07, Randy Bush <[email-masked]> wrote:
> 

Might be even better.

My underlying assumption was that - in a static situation - the resulting set of
VRPs, for a fixed input, is the same independent of VRPs.

The dynamic behaviour is an even lager experiment...

> [ s/for all/for each/ ? ]

Yes - for each.

> and the RPs fetch asynchronously.

are all “observing points in time on a set of timelines”

(given no split views of repo content, etc)

-Ties

---

From: Geoff Huston <[email-masked]>
Date: Thu, 10 Mar 2022 19:53:21 +0000

> On 10 Mar 2022, at 11:07 pm, Randy Bush <[email-masked]> wrote:
> 

How many times do we need to go around and around this space before we realise
that asynchronous demand-pull flooding designs are intrinsically broken?

At least one more time, it seems.

Geoff

---

From: Randy Bush <[email-masked]>
Date: Thu, 10 Mar 2022 12:19:40 -0800

> How many times do we need to go around and around this space before we
> realise that asynchronous demand-pull flooding designs are
> intrinsically broken?

the internet is probably intriniscally broken; but we plod on.

i would be *extremely* interested in your description of how the dns and
rpki would work in push models; and suspect that others would be as
well.  very large scale, wide distribution, and variable connectivity
make difficult bedfellows.

randy

---

From: Geoff Huston <[email-masked]>
Date: Thu, 10 Mar 2022 20:25:01 +0000

> On 11 Mar 2022, at 7:19 am, Randy Bush <[email-masked]> wrote:
> 

agreed, yet we have TLS using just-in-time credential provisioning in the
initial handshake which has very different scaling properties. The models
of DNSSEC provisioning also staple the credentials to the data. RPKI
is one of the few models that attempt to pre-provision the entirety of the
credential sets to all relying parties all of the time, and I find myself
wondering why we ever thought that such an approach would scale!

yes, I agree its just one more aspect of the intrinsic brokenness of the
Internet, and, as you say, we plod on! :-)

Geoff

---

From: Randy Bush <[email-masked]>
Date: Thu, 10 Mar 2022 12:43:02 -0800

note that dns and rpki-rtr try to alleviate this somewhat with NOTIFY.

we both regret that we are old enough to remember when a dns change
might take days to propagate; now we're annoyed if it takes O(10)
minutes.

similarly, the RP cache NOTIFYs (cf 8210) the router "now would be a
good time to pull."

pretty hacky, but improves operation greatly.

randy

---

From: Stephen Kent <[email-masked]>
Date: Thu, 10 Mar 2022 15:46:43 -0500

Geoff,
> Internet, and, as you say, we plod on! :-)

TLS creates a pairwise connection between two entities, so transmitting 
the cert of the server is an obvious, simple approach to solve the 
problem in question. And, because of the artificially shallow PKI 
underlying the issuance of certs to servers (thanks to the browser 
vendors and the cabal of for-profit CAs), it's easy to ensure that a 
client gets the certs (and CRLs) it needs.

The RPKI requires essentially ALL RPs to have access to the certs for 
essentially ALL address space holders. It is much more logical to have a 
bulk distribution mechanism to facilitate this, vs. some sort of 
pairwise distribution mechanism of the sorts you cite. The requirements 
for cert and CRL distribution are completely different for the RPKI vs. 
the Web PKI- apples vs. kumquats (as Randy might say).

Steve

---

From: Tim Bruijnzeels <[email-masked]>
Date: Wed, 23 Mar 2022 12:03:17 +0100

Hi Randy,

As I said I don't oppose the single VRP per ROA recommendation. On the
contrary.

> i would not want to bet my routing on this.

Indeed. Typically what happens in practice is:
1. CAs will publish objects as a set
2. RPs will fetch objects as a set (and reject failed fetches per mft-bis)
3. RPs will do a full analyis of fetched objects to make a set of VRPs
4. RTR servers will send the diff in VRPs to routers and do work to avoid
   duplicates and race conditions

I did not dig up all the normative language in specs, but in my understanding
the above is what is specified. There is a lot of text - some of it is being
discussed but not yet published as RFC - I may have missed something.

In any case if it's bugs or scaling issues that you are worried about,
then I think it would good to spell that out in the draft. I am not
opposing that at all - and indeed single VRPs per ROA may help to
keep things simple. But as written it seems to say that current implementations
do things they are (to my knowledge) not currently doing and I think
that is confusing.

Furthermore.. this draft is informational and avoids using normative
language.. so am I correct in concluding that it does not try to forbid
combining multiple prefixes? Essentially it gives advice to CAs.

For example the APNIC AS0 currently has close to 14k prefixes. They could
of course be split into as many objects but that's a lot of objects..

This is of course an extreme case. But what about a 100 for example?
Would that be enough to decide to combine prefixes in a single object?

It would be useful to me to hear from RP implementers what they believe
a good threshold value might be, or that they prefer to have separate
objects always.. it would help me to set sensible defaults in CA software.
Including possibly: don't aggregate unless specifically instructed by the
operator.

Thanks

Tim

---

From: Tim Bruijnzeels <[email-masked]>
Date: Thu, 24 Mar 2022 08:56:40 +0100

Dear WG

Allow me to share some thoughts outside of the box.. the things below
may not work but I believe they are worth thinking about.

> On 10 Mar 2022, at 21:46, Stephen Kent <stephen.kent=[email-masked]> wrote:
> 
> Geoff,
>> 
> TLS creates a pairwise connection between two entities, so transmitting the cert of the server is an obvious, simple approach to solve the problem in question. And, because of the artificially shallow PKI underlying the issuance of certs to servers (thanks to the browser vendors and the cabal of for-profit CAs), it's easy to ensure that a client gets the certs (and CRLs) it needs.
> 
> The RPKI requires essentially ALL RPs to have access to the certs for essentially ALL address space holders. It is much more logical to have a bulk distribution mechanism to facilitate this, vs. some sort of pairwise distribution mechanism of the sorts you cite. The requirements for cert and CRL distribution are completely different for the RPKI vs. the Web PKI- apples vs. kumquats (as Randy might say).

I believe there may be merit in *combining* current repositories with information
that may be sent in BGP itself.

There are some issues with relying on the current RPKI repositories that may benefit
from this approach.

- Delays

New announcements may be invalid regarding to current ROAs, similarly a new provider
may be invalid under known ASPA providers (when deployed), and a new BGPSec router key
may be unknown.

RPKI objects will have to published, retrieved, validated and the validated content
(VRP, providers, ASN-router-key) have to make their way into routers. Until that time
announcements and routes may be rejected.

The delay can be significant. Optimistically it can be as low as 20 minutes between
a CA publishing a new object and on-path routers getting the validated content, but
it may take much longer as well - more than an hour would not be uncommon.

- Repository unreachable

Consider that an RPKI Repository becomes unreachable because for example a mistake
was made in ROA configuration - and this ROAs is published in said repository..

Obviously this should be avoided, but if it does happen then recovery is painful.

RPs are not going to be able to retreive any updated corrective ROA, until the old
information expires and ROV goes to "not found" rather than "invalid". Slurm may
help a bit but I don't think that expecting all ASNs to start using exceptions would
be realistic.

For BGPSec things may be worse - suppose the old router key is lost and unusable..
The repository could publish a new BGPSec router key, but its route would be invalid,
so RPs in validating networks would not be able to find out.

- Publishing CA (client) unreachable

In this case a delegated CA has invalidated the prefix for their CA software and
is no longer able to reach the repository server.

Again this should be avoided, but if it happens..

If the CA can use an alternative path or Slurm between itself and the repository
server then this may help to fix things. Alternatively it can wait for manifests
and CRLs to expire. Or it can ask the parent to be revoked.

I would say all of these are painful.

- Include objects in BGP

Suppose that in some future we would not do full TLS but allow routers to send
the necessary authorization objects for the context of their announcement/update
along. This could simply be the applicable ROA object, ASPA object of BGPSec
router certificate. Dedicated objects could be invented if needed - but there
may be no need.

Routers that receive such objects for routes lacking a current VRP/ASPA/router
key entry could then dispatch this (rtr++) to RP software which can then validate
these objects directly under the applicable CA certificate that it has. Much
like Job was describing for RSC objects. Routers that already have received
the entries through rtr (or lack this capability of finding the objects in BGP)
can ignore the included objects.

In other words: it would allow capable routers to discover new attestations
sooner, but it would not require any or all routers to do this.

This would of course require that routers have a way to include necessary
object. There could be APIs that allow operators push relevant objects manually,
but one could even consider an RTR like protocol where routers can talk to a
CA to get objects. ROAs/ASPAs/BGPSec certificates are usually long-lived,
so this exchange would be low volume and infrequent.

A slightly more involved approach would be to CMS objects with a full path
of certificates and CRLs. But.. this would require regular updates of all
those - and make the validation process more involved. So, just expecting
objects that are signed directly under a known CA certificate (which btw
could also revoke these objects if needed) would still be complicated, but
much simpler.

I understand that all of this would add complexity to an already complicated
infrastructure. So, not doing any of this may be a valid answer, but.. I
don't see other obvious ways to deal the situations described above. If
anyone has other ideas about ways to handle them I would be very interested.

Tim