Subject: [sidr] comments on the repository analysis I-D

From: Stephen Kent <[email-masked]>
Date: Thu, 14 Mar 2013 17:03:58 -0400

During the first SIDR session I commented that I agree with the need to 
explore new
paradigms for distributing RPKI repository data, but that i was very 
disappointed with
the analysis being used to motivate the exploration.

Attached are my comments on the I-D inn question.  I have used MS Word 
with change control
to associate the comments (and some edits) with the original text. I 
rendered this as a PDF,
so that list members do not need to use MS Word.  I am happy to provide 
the Word doc to the
authors if they wish.

Steve

---

From: Oleg Muravskiy <[email-masked]>
Date: Fri, 15 Mar 2013 14:44:18 -0400

Stephen Kent wrote:
> During the first SIDR session I commented that I agree with the need to explore new
> paradigms for distributing RPKI repository data, but that i was very disappointed with
> Attached are my comments on the I-D inn question.  I have used MS Word with change control
> to associate the comments (and some edits) with the original text. I rendered this as a PDF,
> so that list members do not need to use MS Word.  I am happy to provide the Word doc to the
> authors if they wish.
>
> Steve

Thanks a lot, Steve!
Looks quite constructive.
We'll work on a new version of the document; I already have some questions, not only to you, but also to WG in general. I'll
aggregate them and post to this list.

-- 
Oleg Muravskiy
RIPE NCC

---

From: Arturo Servin <[email-masked]>
Date: Mon, 18 Mar 2013 11:33:24 -0300

Hi,

	Some comments about Steve comments:

Comment 1 (also related with 44): I agree that ISPs may operate caches
in behalf end-users ASNs, but also I think that more than 1 cache may be
operated by a single ISP. Imagine a global ASN operator with routers in
several places. Are they going to have just one master cache? Or are
they have one or two (backup), and just in one location? Considering
this, even the 40k clients may be low as worse case IMHO.

Comment 10: Not sure if I understand the point, but because we do not
trust fully on BIND we also operate DNS with NSD. In fact, I agree with
the authors that we have a problem depending on just one implementation
of rsync.

Comment 28: For LACNIC ~ 2,500 members and ~ 2,200 PI. But as opposite
to RIPE NCC, PI holders are members as well (not sure if this is
relevant to the numbers)

Comment 31: I do not have numbers of prefixes per ROA, but it is a
number that we could get soon.

Comment 44: Does any body has a pointer of that proposal?

Comment 46: This is hard to answer. I have heard some operators asking
for minutes to have fresh new ROAs. Some others do not mind and request
objects every few hours. What is the middle or agreed value? (also
related with 55)

Comment 57: I think the reason is that CDNs today work on http and we
know them more or less well. Although possible it would be more
expensive and complex to have rsync-CDNs. Also this is related to
comment 54, the "single point" is "magically" distributed by the CDN
responding with the "closest" point as it is done today for http content.

	Hope it helps for the discussion and to improve the document.

Regards,
as

On 3/14/13 6:03 PM, Stephen Kent wrote:
> _______________________________________________
> sidr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidr
>

---

From: Christopher Morrow <[email-masked]>
Date: Mon, 18 Mar 2013 11:35:05 -0400

I'm not a fan of word in general.. but this comment numbering rules ;)

> this, even the 40k clients may be low as worse case IMHO.

oops, so... we need to be clear in terminology here there are at least:
  o publication points - places/machines AS Operators would make their
authoritative information available to the world.
  o 'gatherer' machines - machines an AS Operator would use to gather
repository data from all of the publication points.
  o caches - systems inside of and AS which provide the digested data
to the routers of that operator's network.

Note that:
  1) a publication point may share common hardware/network/etc with
another publication point (shared/hosted model)
  2) a AS Operator may operate more than one 'gatherer' perhaps they
partition the publication space ? or some other strategy is in use
(local decisions not really relevant, I think, for overall discussion)
  3) a cache may be only internal facing "all for me, only!" or it may
provide the cache services for external parties (customers,
random-joe-internet-operator, researchers).
  4) it's entirely up to the AS operator to decide how many and in
what configuration the caches are in his/her network.

I think that most of Tim/Oleg's doc is about publication points and
the load placed on the entire system by gatherers... I don't think
they tackle the cache part at all in their doc.

As to the end of the comment in question:
 "So, I don’t think this
parameter is a good estimate (worst case, but
probably not representative). "

I think it's fair to state: "The worst case today is AS's == Gatherers"
which is the intent of tim/oleg's work here... I think :)

I think another part of Stephen's comment is really that there's the
potential for an AS operator to run a gatherer for his/her customer's
to benefit from... That doesn't seem unreasonable, but it's not
knowable how many WILL do that, so I'd err on the side of worst-case
and jump to #AS == #gatherers. (as long as that's stated clearly)

As to the memory limits imposed by platforms chosen for the study,
it's probably fair to not that any system which drives toward
max-memory+ is going to hit a cliff in performance when the system is
forced to swap :( try to avoid swap... it's slow (so I hear).

> Comment 44: Does any body has a pointer of that proposal?

I suspect this is actually a comment about cache's not gatherers?
Either way I'm not sure a 2x factor matters in end here...

> objects every few hours. What is the middle or agreed value? (also

this, I think, is from some of the discussions Danny/Eric have been
driving about time to get CRL updates out everywhere, or times to get
a new ROA published and seen 'everywhere'. I believe we ended up
talking about a time of on the order of 2-5 minutes for: "Have new
prefix, make roa, get published and distributed"

I don't know that that number matters a LOT today, it will certainly
be much more interesting as a target in a few more years (when
deployment is further along).

> expensive and complex to have rsync-CDNs. Also this is related to

'rsync cdn' == 'lots of rsync servers' really, right? since really you
are essentially making new 'cache' (at the cdn) for each client... the
only think you're winning with this approach (and rsync) is moving the
tcp endpoint closer (maybe) to your client/requestor.

Has anyone (not to derail too far) thought about git or another
network-based RCS system for this? It seems that (as much as I was
poking the tiger on the IETF thread about this) the checkpointing and
such is attractive... done right, it'd even give the history data that
some folks want for the changes to objects.

> comment 54, the "single point" is "magically" distributed by the CDN

for comment 54, I think the text is talking about: "with all RP's
talking to all PubPoints, as the time horizon on changes gets smaller
(from 1-3 hrs to 10 mins) each RP will have to talk to the PubPoints
more often, leading to more concurrency at each PubPoint over time"

Making the overall connection time shorter for these conversations
could alleviate the concurrency problem.

---

From: Bryan Weber <[email-masked]>
Date: Mon, 18 Mar 2013 12:22:31 -0400

I'll just address one of your comments here and I'll save the rest until I have time to talk to Tim and Oleg as we update this document based on all the feedback.

I had proposed to the mailing list a DVCS like approach to replace rsync, but after more consideration I joined with Tim and Oleg to work on these documents instead.

Some of the reasons are:

* What do you do in the case of a merge conflict? They require human intervention which means that any solution should prevent merge conflicts. It might be possible to get existing DVCS to always prefer one changeset over another changeset but I imagine that the logic for which should 'win' could get tricky given the number of publishers here. Do you use time to determine which one is newer? Who knows if the other servers have their clocks set correctly. Do you use the manifests? I don't believe that there is a practical, simple solution to this for existing DVCS's, but admittedly I haven't looked into this a great deal so I could be wrong...
* Is there a standard for the DVCS? Would we be moving from one non-standard program, rsync, to another, say Git?
* This one is my opinion. It is only worthwhile to force consistency for a single publication point (not up/down in a delegated world). Otherwise, the system becomes incredibly complex and rife with a new attack vector where a child could "freeze" a parent and prevent it from publishing if the parent and child have to coordinate to guarantee consistency across the two. Of course this only gets worse the deeper the hierarchy is.

I had started writing a Git-like DVCS that would detect file based changes, would not ever have merge conflicts (solving issue #1) and could be proposed as a standard (solving issue #2). After considering #3, I decided that consistency was only practical at a single publication point instead of trying to force it up/down the hierarchy I decided that the extra complexity of a DVCS solution was not better than a simple HTTP/XML with a session id and version id as we proposed in draft-tbruijnzeels-sidr-delta-protocol.  It might also be less CDN friendly depending on the implementation, but that wasn't a real concern to me.

There is another possible technical hurdle to overcome which has to do how Git/Mercurial/etc. implement a custom file system of sorts, which enables super fast checkout, but getting this to be cross platform might be a lot of work. Hint: they aren't doing file copying. :)  Nothing that couldn't be overcome, but this is a lot more work that the immutable data structures side of the DVCS IMHO.

I agree that it does have some nice properties.

* It could be used without the publication protocol which means it could be used by the existing code bases that write to file systems that are in use at the RIR's today. It would simply require the process that writes to the file system to indicate that it would like to 'commit' when it completed writing changes to the repository. This is 100% possible with the delta protocol that we described, however it does not provide any mechanism to do so.
* You could view point in time history of what a particular instance saw. This very well may help with debugging real world issues.

Just one final thought: there is nothing that says you couldn't take a DVCS approach that used the delta protocol we described. The proposed protocol simply describes the format of the messages that are sent over the wire and some very basic information about the state of the publication server (session id and version id). 

Anyway, as someone who has considered this in the past I just wanted to document some of my thoughts regarding the idea.

Just my 2 cents.
-Bryan

---

From: Stephen Kent <[email-masked]>
Date: Mon, 18 Mar 2013 12:47:11 -0400

Arturo,

> this, even the 40k clients may be low as worse case IMHO.
See Chris's comment re the issue of "gather's" vs. caches, for a more
precise description of this issue.
> the authors that we have a problem depending on just one implementation
My point was that there are circumstances in which there is a de facto 
single implementation and
that folks have survived. But, I agree with the principle stated here, 
so, never mind ,...
> to RIPE NCC, PI holders are members as well (not sure if this is
Thanks. I suspected that the dichotomty cited in the RIPE context might
not be representative of all RIRs, as your reply conformed.
> number that we could get soon.
I think it will be useful to have such #'s from several sources.
> Comment 44: Does any body has a pointer of that proposal?
http://www.iepg.org/2012-07-ietf84/120729-randy.pdf
> objects every few hours. What is the middle or agreed value? (also
There are at least two issue here: how quickly a new/changed ROA is 
published
after it is created/modified, and how quickly one should expect all
RPs to have acquired this info. The RPKI propagation model for ROAs was
based on the observation that, typically, the binding between an address 
space holder
and the AS originating routes to that space changed very infrequently, and
not very fast. Conversations with RIR staff supported this notion, based on
IRR DB experience. So, suggesting that this data needs to be propagated in
minutes, vs. 1/2 a day, is quite a change.
> responding with the "closest" point as it is done today for http content.
My comment was independent of the use of CDNs; an RIR could maintain a 
master DB
for RPKI data and push copies to multiple caches for access by RPs, 
using rsync.
But, I agree that it is attractive to explore using CDNs, IF we can make use
of appropriate security mechanisms to ensure that RPs know that the 
current version data
and deltas are authentic.

Steve

---

From: Christopher Morrow <[email-masked]>
Date: Mon, 18 Mar 2013 13:31:04 -0400

On Mon, Mar 18, 2013 at 12:22 PM, Bryan Weber <[email-masked]> wrote:
> Anyway, as someone who has considered this in the past I just wanted to
> document some of my thoughts regarding the idea.

awesome, thanks! I didn't imagine one monolithic repository, but one
per pub-point.. I don't think conflicts exist if you simple sync
always from each repository... well, the conflicts seem simpler to get
out of: "master always wins".

In the end though, I was just imagining :) I'm not wedded to the idea/concept.

---

From: Randy Bush <[email-masked]>
Date: Tue, 19 Mar 2013 11:51:34 +0200

> to the routers of that operator's network.

from the emulation preso series

---

From: Christopher Morrow <[email-masked]>
Date: Tue, 19 Mar 2013 13:14:22 -0400

> from the emulation preso series

your slide confuses me... with the arrows from ARIN -> cache1-asia
(etc) seeming to imply a push from rpki-ARIN (which I assume is the
pub point for arin?) to the cache1-asia system?

---

From: Randy Bush <[email-masked]>
Date: Wed, 20 Mar 2013 07:29:26 +0200

> pub point for arin?) to the cache1-asia system?

push is your imagination.  control flow is not intended, otherwise
interpub would be unnecessarily 'interesing'.  the arrows show direction
of data flow.

randy

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 20 Mar 2013 01:31:03 -0400

> interpub would be unnecessarily 'interesing'.  the arrows show direction

ah-ha! :) gotcha.

---

From: Danny McPherson <[email-masked]>
Date: Wed, 20 Mar 2013 06:25:52 -0600

On 2013-03-18 10:47, Stephen Kent wrote:

> based on the observation that, typically, the binding between an
> address space holder

No, it's not.

The system converges at the speed of routing today, if I have to wait 
12 hours to mitigate an attack for someone then that could be a problem. 
If the RIR's actually conducted any qualitative study to this effect 
I'd appreciate a reference - I've seen none.  However, I can say for 
certain that there are many companies that need "minutes" of 
convergence, just like they have today, else they won't be able to 
perform things like DDoS mitigation for hours or longer, and that's not 
acceptable to many current operations.  We have a business that utilizes 
this today.

We've already had this discussion here.

-danny

---

From: Stephen Kent <[email-masked]>
Date: Wed, 20 Mar 2013 09:44:09 -0400

Danny,
>> minutes, vs. 1/2 a day, is quite a change.
My comment applies to the statement about the frequency with which IRR 
DB entries, some of which are
very analogous to ROAs, change. Your assertion (below) might be that 
folks who do not rely on these
dabatases accept changes in asserted prefix/ASN bindings at the speed of 
routing updates. I
agree with that observaiton, but note that it's also what made the 
Pakistan Telecom vs. YouTube
incident possible.
> 12 hours to mitigate an attack for someone then that could be a 
> problem. If the RIR's actually conducted any qualitative study to this 
> effect I'd appreciate a reference - I've seen none. However, I can say 
> for certain that there are many companies that need "minutes" of 
> perform things like DDoS mitigation for hours or longer, and that's 
> not acceptable to many current operations. We have a business that 
> utilizes this today.
You cite DDoS mitigation as _an_ example, whereas it seems to be more 
_the_ example. Also, it's not clear
that asserting a new origin AS for a target of such an attack is the 
only way to deal with the problem,
just the way that it is performed today, for ASes that do not make 
arrangements in advance to counter
such attacks.

Steve

---

From: Danny McPherson <[email-masked]>
Date: Wed, 20 Mar 2013 10:58:58 -0400

On Mar 20, 2013, at 9:44 AM, Stephen Kent <[email-masked]> 
> You cite DDoS mitigation as an example, whereas it seems to be more the example.

Bah..  How many examples do you need?  

> Also, it's not clear
> that asserting a new origin AS for a target of such an attack is the only way to deal with the problem,
> just the way that it is performed today, for ASes that do not make arrangements in advance to counter
> such attacks.

Hah, right. If you've any magic the industry would be interested in it Dr. Kent.  Until then, please try not to break current capabilities that real operators broadly use..

-danny

---

From: Oleg Muravskiy <[email-masked]>
Date: Thu, 21 Mar 2013 11:09:28 +0100

Hi Christopher,

Christopher Morrow wrote:
> Comment 1 (also related with 44): I agree that ISPs may operate caches in behalf end-users ASNs, but also I think that more than 1
> cache may be operated by a single ISP. Imagine a global ASN operator with routers in several places. Are they going to have just
> one master cache? Or are they have one or two (backup), and just in one location? Considering this, even the 40k clients may be
> low as worse case IMHO.

In our analysis we associate number of CAs in the global RPKI with the number of distinct IP resource holders.

You seem to associate publication points (that directly relate to CAs) with AS Operators.
Since it's a second place where publication points are associated with AS Operators (another is the "RPKI rsync Download Delay
Modeling" presentation), I wonder if I miss something?

-- 
Oleg Muravskiy
RIPE NCC

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 21 Mar 2013 11:27:38 -0400

> In our analysis we associate number of CAs in the global RPKI with the number of distinct IP resource holders.

sure, and as a proxy for that 'AS Operator', it's not a 1:1
correlation to be sure but it should be reasonably close, no?

> Modeling" presentation), I wonder if I miss something?

most likely you are not... I think I jump to 'CA == REPO ==
AS-Operator == ASN allocated' because lacking any direct data
otherwise it seems like a good estimation of numbers. Essentially each
ASN allocated is going to be a repository that needs to be gathered,
right? If there are 10% more repositories due to EndSite allocations
without an ASN also allocated to them I think it's still in the
ballpark to say "number of Repos == ASN allocation number".

I could be wrong.

---

From: Randy Bush <[email-masked]>
Date: Thu, 21 Mar 2013 17:43:49 +0200

>> In our analysis we associate number of CAs in the global RPKI with
>> the number of distinct IP resource holders.

do we have anything other than conjecture on which to base estimations
of the numbers of CAs or repositories?

> I jump to 'CA == REPO == AS-Operator == ASN allocated' because lacking
> any direct data otherwise it seems like a good estimation of numbers.

great example of conjecture.  do you have any fact/measurement-based
idea of how many CAs or repositories there might be five or ten years
from now?  the only data i have is the small number of IRR repositories
and whois servers.  and i suspect/hope that is a poor estimator.

but i very strongly doubt that any significant portion of the stub ASs,
84% of the ASs, will run CAs.

randy

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 21 Mar 2013 11:53:55 -0400

> of the numbers of CAs or repositories?

no, since there aren't any CA's in existence... we have, or I have, a
model that says:
  "If you want to publish a ROA, you need to have a CA and you need to
run a publication point"

To me that means at the least every ASN will have a publication point
(and this a roa and a CA).

> and whois servers.  and i suspect/hope that is a poor estimator.

I really don't know how to estimate ASIDE from saying: "it seems
reasonable that the repo number will track with assigned ASN"

I could be wrong, I could also be corrected if someone else has a
compelling story... I don't think it's harmful to say "tracks to ASN
numbers" though, if it's too large a number we can be surprised by
performance... if it's too small, we can also be surprised :)

> 84% of the ASs, will run CAs.

they may not.
they may use a hosted-model product.
I suspect that very soon after 'hosted model' comes into being in the
large, the operators of these systems will realize that if someone
dislikes one of their customers they can't survive as a business if
all other customers also disappear. They will be forced to run the
system with unique names/ips for each customer (names at a minimum so
they can shift problem children away).

So, in the above case... CA == ASN == REPO

-chris

---

From: Randy Bush <[email-masked]>
Date: Thu, 21 Mar 2013 19:55:00 +0200

> I have, a model that says:
>land this a roa and a CA).
Wherever did you get that?   what is the ratio of hosted LIRs to delegated today?
-- 
phone email, so sucky

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 21 Mar 2013 15:43:14 -0400

On Thu, Mar 21, 2013 at 1:55 PM, Randy Bush <[email-masked]> wrote:
> Wherever did you get that?

I figured in the worst case you'd end up with 1:1... I don't think
it'll be much worse than 1:1, it could be better, of course.

> what is the ratio of hosted LIRs to delegated today?

not sure... I believe that hosted == delegated though in the long run,
from the perspective of what ip/name you have to connect to in order
to access the data stored in the repositories.

---

From: Stephen Kent <[email-masked]>
Date: Thu, 21 Mar 2013 16:11:09 -0400

Chris,

> ...most likely you are not... I think I jump to 'CA == REPO ==
So far the 1,300+ folks who have signed up for managed CA services have also
let the RIRs manage their pub points, which dramatically reduces the
number of repositories. That could change over time, e.g., if these
folks become unhappy with the repository management, but for now I think
it is reasonable to assume a much smaller number of repositories, which
is what Sriram and I did in our model.

---

From: Stephen Kent <[email-masked]>
Date: Thu, 21 Mar 2013 16:15:05 -0400

Chris,

> no, since there aren't any CA's in existence...
well, there are over 1,300 so far, but almost all are managed CAs.
>   we have, or I have, a
still true, even for managed CAs.
> (and this a roa and a CA).
it's a ROA, a manifest, a CRL, maybe a GB record, and a CA cert in the 
parent's pub point.
> reasonable that the repo number will track with assigned ASN"
see my earlier comment re this possible equivalency.
> they may use a hosted-model product.
but even in the "hosted model" each of these fokks is represented by
a CA and a pub point. I think the only issue is whether each is a separate
repository.
> So, in the above case... CA == ASN == REPO
I don't understand that last argument. Can you expand on it?

Thanks,

Steve

---

From: Chris Morrow <[email-masked]>
Date: Thu, 21 Mar 2013 16:29:01 -0400

> number of repositories. That could change over time, e.g., if these

TODAY it reduces the number, yes. 100% agree.
TOMORROW the number of repositories, even those which are 'hosted' will
be split up by name and/or ip-address...

I have a feeling these will be like DNS servers and likely ripe (ha!)
points for attack by bad folks. So sharing fate for all customers just
seems like a bad idea.

> is what Sriram and I did in our model.

yup. but having the ability to increase the number of repositories in
the model means we can say: "today with N repositories and M objects we
see times of Y. Tomorrow when we have X repositories with Y objects we
should see times of Z"

-chris

---

From: Danny McPherson <[email-masked]>
Date: Thu, 21 Mar 2013 14:42:11 -0600

> see times of Y. Tomorrow when we have X repositories with Y objects 

Agreed.

Additionally, this perspective may well change when things like BGPSEC 
router keys need to be published and then ingested by operational 
routers for update signing, as the trust model is fairly different, 
methinks.

Even the "84%'r" morons (i.e., all those stub ASes) might recognize 
this as a threat to their operational autonomy.

fail open,

-danny

---

From: Chris Morrow <[email-masked]>
Date: Thu, 21 Mar 2013 16:46:53 -0400

> it's a ROA, a manifest, a CRL, maybe a GB record, and a CA cert in the

sure, I abbreviated the list.

> but even in the "hosted model" each of these folks is represented by

correct. one contiguous repo is a great target ... once or twice that'll
work, then the repository operator will split all customers into
separate names at the least that she can steer to infrastructure in case
of a problem.

> I don't understand that last argument. Can you expand on it?

Today, if you look at:
  dig NS ly
  ly.cctld.authdns.ripe.net. is one of the servers listed...

  ripe sets up a separate name/ip for each CC they serve. This gives
them the flexibility to move one victim off to a dedicated server(s) in
the case of really bad problems.

I suspect that the hosted repository model will evolve to this over time.
-chris

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 21 Mar 2013 16:59:43 -0400

On Thu, Mar 21, 2013 at 4:42 PM, Danny McPherson <[email-masked]> wrote:
> router keys need to be published and then ingested by operational routers
> for update signing, as the trust model is fairly different, methinks.

so, to me, this is just 'more objects with a tight(er) timeframe on
delivery' right?

meaning: today you have (for sake of the conversation) relatively
static content in the repository, where data changes 1/2/3
objects/day, and it's important to get the data distributed, but
(again, for the conversation) 24hrs is 'fine' as a timeline between:
"publish object" and "rendered onto the router".

but: tomorrow if you followed (for the sake of the conversation)
Sriram's model of 'roll router keys hourly to ensure ttls on routes'
(loosely paraphrased and with times I don't even think he put into his
slides/discussions), you'd be talking about #-of-routers * 24 objects
changing every day, and with a timescale for: "publish" to "render" on
the order of 1hr. (likely less than 1hr, something closer to 5-10
minutes has been discussed previously).

right?

-chris

---

From: Stephen Kent <[email-masked]>
Date: Thu, 21 Mar 2013 17:14:10 -0400

Chris,
> ...
I think the operative term is "might" vs. "will."
> points for attack by bad folks. So sharing fate for all customers just
A single repository need not translate into a single instance of the files
associated with that repository, right?
> see times of Y. Tomorrow when we have X repositories with Y objects we
yes, it is reasonable to incorporate knobs into models to allow
folks to explore their favorite scenarios.

Steve

---

From: Chris Morrow <[email-masked]>
Date: Thu, 21 Mar 2013 17:23:56 -0400

> I think the operative term is "might" vs. "will."

sure, I'm just looking at what's happened in other things that look like
this. I could be wrong.

> associated with that repository, right?

it's a single hostname/ip-address-set right?
  rsync://repository.arin.net/

I think we'll see, for example:
  rsync://customer1.repository.arin.net/

hostnames are free, ip-addresses (for a registry) are free :)
having all your customers down because someone dislikes visa.com (who
chose to use the hosted repository model) is ... unsatisfactory (and
likely expensive)

> folks to explore their favorite scenarios.

cool. thanks!

---

From: Danny McPherson <[email-masked]>
Date: Thu, 21 Mar 2013 15:42:44 -0600

> minutes has been discussed previously).

Yeah, I think so under steady state.

It gets more interesting when the RIR is attacked and I can't reach 
them, or I have a failure of a control card or systems where I cache 
them locally, or something to that effect.

AND I have to know when *everyone* else in the system picks them up and 
pushes them out to their routers in preparation for validation so that I 
know when it's safe to starting signing with them, etc..

-danny

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 21 Mar 2013 21:43:25 -0400

On Thu, Mar 21, 2013 at 5:42 PM, Danny McPherson <[email-masked]> wrote:
> It gets more interesting when the RIR is attacked and I can't reach them, or
> I have a failure of a control card or systems where I cache them locally, or
> something to that effect.
>

true, but the repository conversation stops at: "all gatherers in the
system have the data"

inside each ASN it's really up to the ASN operator to get from
gatherer -> cache -> router in a 'timely fashion'.

---

From: Danny McPherson <[email-masked]>
Date: Thu, 21 Mar 2013 20:30:41 -0600

> gatherer -> cache -> router in a 'timely fashion'.

If you're signing a route with something, and your upstreams are 
signing their secure path elements of that route with something, and my 
routers haven't been provisioned (from data obtained from the RPKI) to 
validate with the corresponding public versions of those somethings (for 
whatever reason - e.g., latency in fetching, DoS, failure, etc..), then 
this *tight coupling* between the public key repositories (i.e., RPKI) 
will result in my router as a relying party not having the information 
it need to validate those signed somethings, and I therefore "downgrade" 
or break for that route.

You not knowing when I have the information to validate something 
because I may not have fetched it from the RPKI yet (or ingested it in 
my routers yet, for whatever reason) means you can't send a route with 
the new signatures because it will fail validation, thereby causing a 
downgrade attack or something to break for the route.

This "looseness" about an expanding array of validation states and 
acceptable downgrade attacks in BGPSEC because the RPKI may not be 
available, or local systems may be operating on stale data, continues to 
give me pause given the overhead we're introducing and the expanding set 
of residual vulnerabilities some are willing to accept.

-danny

---

From: Randy Bush <[email-masked]>
Date: Fri, 22 Mar 2013 06:27:27 +0200

>> So far the 1,300+ folks who have signed up for managed CA services
>> have also let the RIRs manage their pub points, which dramatically
>> reduces the number of repositories. That could change over time,
>> e.g., if these

this is measurement

> TOMORROW the number of repositories, even those which are 'hosted'
> will be split up by name and/or ip-address...

this is pure conjecture.  can we please reduce the ratio of the latter
to the former?

randy

---

From: Chris Morrow <[email-masked]>
Date: Fri, 22 Mar 2013 00:32:37 -0400

> this is pure conjecture.  can we please reduce the ratio of the latter

sure thing.
how do you propose we measure how large the number of CA/Repositories is
going to be in 1yr? 5 yrs? 10 yrs?

-chris

---

From: Randy Bush <[email-masked]>
Date: Fri, 22 Mar 2013 07:50:01 +0200

> going to be in 1yr? 5 yrs? 10 yrs?

we can not measure the future until it gets here.  the best we can do is
estimate it based on the present and past.

i could conject that ten years from now we will have one repository run
by itu-i.  i could conject that ten years from now we have one
repository per announced prefix.  all pure bs.

randy

---

From: Chris Morrow <[email-masked]>
Date: Fri, 22 Mar 2013 09:54:58 -0400

> estimate it based on the present and past.

which was what I was doing, actually.

> repository per announced prefix.  all pure bs.

sure. but again, past experience points to the hosted model eventually
looking like distinct repositories per customer, which looks like
'number of repositories == number of ASN allocated'.

-chris

---

From: Randy Bush <[email-masked]>
Date: Fri, 22 Mar 2013 15:57:36 +0200

> past experience points to the hosted model eventually looking like
> distinct repositories per customer, which looks like 'number of
> repositories == number of ASN allocated'.

actual experience is five large hosted repositories.

randy

---

From: "Osterweil, Eric" <[email-masked]>
Date: Fri, 22 Mar 2013 11:08:04 -0400

> actual experience is five large hosted repositories.

Randy, sound _engineering_ is predicated (in large part) on evaluation and reducing risks.  Using predictive models (especially those based on sound reasonable or observable assumptions) to estimate unknowns is pretty common, among conscientious engineers (and scientists, fwiw).  This is particularly true during the early phases of design work and requirements analysis (the latter of which, I think we are still missing?).  It seems to me that Chris is practicing very advisable prudence.  Conversely, the ready-fire-aim approach is… well, an odd position to be advocating here…

I'd suggest that if you don't like Chris' model, you propose one that you have more faith in.  I (for one) would greatly appreciate seeing your insights in that regard.

Eric

---

From: Shane Amante <[email-masked]>
Date: Fri, 22 Mar 2013 09:58:47 -0600

> I'd suggest that if you don't like Chris' model, you propose one that you have more faith in.  I (for one) would greatly appreciate seeing your insights in that regard.

To add to Eric's excellent points above.  _Capacity_Planning_ of networks, (which we've done since the dawn of time), depends on observations of historical utilization of physical/logical paths across the network.  However, historical utilization does not tell you the whole story, due to (as just one example) [very] large, but *pending* (i.e.: not turned up) customer wins.  In that case, we apply sound engineering to /forecast/ and then input that forecast into __models__ to determine what will be the growth of the network and then plan/execute augments appropriately.  In some circumstances, these exercises are performed over much longer time-scales, particularly when the operator is facing the need to actually build and/or move physical infrastructure to accommodate for the predicted growth, (i.e.: build new facilities, move from old to new facilities, lay new conduit/cable, etc.).  In any of these exercises, sometimes you are "surprised" by the results, (typically costs), and need to explore alternatives ... exploration of those alternatives is part of trying to 'reduce risks', as Eric points out.

So, +1 to "sound engineering" comment, above.

-shane

---

From: Danny McPherson <[email-masked]>
Date: Fri, 22 Mar 2013 12:52:53 -0600

On 2013-03-21 23:50, Randy Bush wrote:

> by itu-i.

Assuming you meant ITU-T, that option isn't looking nearly as bad as it 
did a few years ago, at least I'll know what it takes to get operational 
requirements considered there.

I think the disconnect emerging is the dichotomy between those who have 
to operate, sustain, and rely on this infrastructure, and the "running 
code and run away" school of thought.  I'd have thought the latter would 
have matured at this stage - Fortunately, at least all the RIRs 
recognize the seriousness and implications...

-danny

---

From: Oleg Muravskiy <[email-masked]>
Date: Sat, 23 Mar 2013 01:50:39 +0100

Christopher Morrow wrote:
>>> Comment 1 (also related with 44): I agree that ISPs may operate caches in behalf end-users ASNs, but also I think that more than
>>> 1 cache may be operated by a single ISP. Imagine a global ASN operator with routers in several places. Are they going to have
>>> just one master cache? Or are they have one or two (backup), and just in one location? Considering this, even the 40k clients
>>> may be low as worse case IMHO. oops, so... we need to be clear in terminology here there are at least: o publication points -
>>> places/machines AS Operators would make their authoritative information available to the world.
>>
> sure, and as a proxy for that 'AS Operator', it's not a 1:1 correlation to be sure but it should be reasonably close, no?

Well, I don't see why resource holders should correlate to AS operators. Maybe... But see below.

>> You seem to associate publication points (that directly relate to CAs) with AS Operators. Since it's a second place where
>> publication points are associated with AS Operators (another is the "RPKI rsync Download Delay Modeling" presentation), I wonder
>> if I miss something?
>
> most likely you are not... I think I jump to 'CA == REPO == AS-Operator == ASN allocated' because lacking any direct data
> otherwise it seems like a good estimation of numbers. Essentially each ASN allocated is going to be a repository that needs to be
> gathered, right? If there are 10% more repositories due to EndSite allocations without an ASN also allocated to them I think it's
> still in the ballpark to say "number of Repos == ASN allocation number". I could be wrong. 

Ok, then I'll continue with mine line of thinking.
>From the RIR stats files that RIRs publish daily we could get the numbers of distinct resource holders. They are:

AFRINIC  1310
APNIC    7957
ARIN    35380
LACNIC   4278

For the RIPE NCC you could not get this data from stats files, and the exact number is difficult to get because of our model of
provider-independent end users. But in our registry I could count that it is at least 28912.
That brings the total to77837.

Now, these are only the first level resource holders under RIRs. They all *must* have their own CAs in order to participate in RPKI.
However, many of these first-level resource holders are NIRs or LIRs, who distribute resources further down to their clients. They
could choose to manage their clients' RPKI objects within their single CA, but could also give their clients  own certificates,
creating next level of CA hierarchy. I find it difficult to estimate how many LIRs will do this, and for how many of their clients.
But for RIPE NCC I could see that the number of organisation objects in RIPE DB is 70746, and that should be the upper bound of the
number of CAs in our region. I don't have that number for other regions, and don't know if it's applicable in the same way,
especially where NIRs are present.

Hope that's clear and helps.

-- 
Oleg Muravskiy
RIPE NCC

---

From: Oleg Muravskiy <[email-masked]>
Date: Sat, 23 Mar 2013 02:33:58 +0100

> Wherever did you get that?   what is the ratio of hosted LIRs to delegated today?

Although in a hosted model it's possible to run just one CA and one publication point for all hosted clients, in practice no one
from RIRs is doing that.
In our hosted system we have 1370 CAs, and each CA has it's own publication point, although they all appear to be in the same rsync
tree.

If the number of CAs is used to estimate the size of a global RPKI repository (number of objects), then the distinction between
hosted and delegated model doesn't matter. It matters if you want to estimate the number of different repositories to query. But I
don't know what to do with that number afterwards.

-- 
Oleg Muravskiy
RIPE NCC

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 22 Mar 2013 22:38:06 -0400

(holy giant text batman!)

On Fri, Mar 22, 2013 at 8:50 PM, Oleg Muravskiy <[email-masked]> wrote:
> Comment 1 (also related with 44): I agree that ISPs may operate caches in
> behalf end-users ASNs, but also I think that more than 1 cache may be
> several places. Are they going to have just one master cache? Or are they
> have one or two (backup), and just in one location? Considering this, even
> the 40k clients may be low as worse case IMHO. oops, so... we need to be
> clear in terminology here there are at least: o publication points -
> sure, and as a proxy for that 'AS Operator', it's not a 1:1 correlation to
> be sure but it should be reasonably close, no?
>
>

ah! I made you look :)

>
> You seem to associate publication points (that directly relate to CAs) with
> AS Operators. Since it's a second place where publication points are
> associated with AS Operators (another is the "RPKI rsync Download Delay
> most likely you are not... I think I jump to 'CA == REPO == AS-Operator ==
> ASN allocated' because lacking any direct data otherwise it seems like a
> good estimation of numbers. Essentially each ASN allocated is going to be a
> repository that needs to be gathered, right? If there are 10% more
> repositories due to EndSite allocations without an ASN also allocated to
> them I think it's still in the ballpark to say "number of Repos == ASN
> allocation number". I could be wrong.
>
>
> For the RIPE NCC you could not get this data from stats files, and the exact
> number is difficult to get because of our model of provider-independent end
> users. But in our registry I could count that it is at least 28912.
> That brings the total to 77837.
>
> Now, these are only the first level resource holders under RIRs. They all
> *must* have their own CAs in order to participate in RPKI. However, many of
> these first-level resource holders are NIRs or LIRs, who distribute
> resources further down to their clients. They could choose to manage their
> clients' RPKI objects within their single CA, but could also give their
> clients  own certificates, creating next level of CA hierarchy. I find it
> difficult to estimate how many LIRs will do this, and for how many of their
> clients. But for RIPE NCC I could see that the number of organisation
> objects in RIPE DB is 70746, and that should be the upper bound of the

cool! so already the number of repositories could be north of 100k
(given the other 5 regions)... 'could be'. In the end I think it'd be
nice of the models (some of them at least) had the ability to see
'what if 5 repos? what if 500? 500,000?' which I think is what Capitan
Kent got to in his last mail.... "yes, it's a parameter we can
twiddle".

Some of the more real-world measurement, which Admiral Bush is after,
is indeed harder to do this with (I think), but knowing behaviour of
the current system should help to extrapolate I would think.

> number of CAs in our region. I don't have that number for other regions, and
> don't know if it's applicable in the same way, especially where NIRs are
> present.
>

yes! thanks.

-chris

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 22 Mar 2013 22:40:15 -0400

On Fri, Mar 22, 2013 at 9:33 PM, Oleg Muravskiy <[email-masked]> wrote:
> Although in a hosted model it's possible to run just one CA and one
> publication point for all hosted clients, in practice no one from RIRs is
> doing that.
> In our hosted system we have 1370 CAs, and each CA has it's own publication
> point, although they all appear to be in the same rsync tree.

so, this leads to some confusion, but does make the repository one
monolithic item to kill/sync-from. It's convenient, right up until
it's not :( I'm not sure when that point will arrive, but ...

> If the number of CAs is used to estimate the size of a global RPKI
> repository (number of objects), then the distinction between hosted and
> delegated model doesn't matter. It matters if you want to estimate the
> number of different repositories to query. But I don't know what to do with
> that number afterwards.

I hope that the next step after that estimate is: "real world testing
shows pulling a repository/update-set takes X seconds, as we get
larger in the number of repositories, we need to keep that in mind"

-chris

---

From: Randy Bush <[email-masked]>
Date: Sat, 23 Mar 2013 13:06:21 +0900

> publication point for all hosted clients

yucchhy

> in practice no one from RIRs is doing that.

thank you

> repository (number of objects), then the distinction between hosted
> and delegated model doesn't matter. It matters if you want to estimate
> the number of different repositories to query.

as you know, performance of the current transport, rsync, is most
dependent on the number of repositories, not publication points or
objects.

> But I don't know what to do with that number afterwards.

multiply by a million and post your estimate on the sidr list as a
realistic prediction?  :)

randy

---

From: Arturo Servin <[email-masked]>
Date: Sat, 23 Mar 2013 11:38:21 -0300

> On Fri, Mar 22, 2013 at 9:33 PM, Oleg Muravskiy <[email-masked]> wrote:

<snip> hoping not to cut important parts

> it's not :( I'm not sure when that point will arrive, but ...

	I liked what you proposed in [1]

	ca-1234ab.repository.ur-rir.net
	ca-1235bb.repository.ur-rir.net

	This would help to move a problematic CA out of the monolithic repo.
However, this would make a cache to create multiple rsync sessions to
retrieve objects and lowering its performance (according to Rob's
measurements with RIPE NCC repo). Isn't it?

[1] http://www.ietf.org/mail-archive/web/sidr/current/msg05835.html

/as

---

From: "Osterweil, Eric" <[email-masked]>
Date: Sat, 23 Mar 2013 12:28:30 -0400

On Mar 23, 2013, at 12:06 AM, Randy Bush <[email-masked]> wrote:

<snip>

> dependent on the number of repositories, not publication points or

Actually, our numbers from RPKI Spider and one of the presentations (not ours) at the last sidr wg showed that rsync scaling depends both on the size of the repo and the number of objects...

> multiply by a million and post your estimate on the sidr list as a

Yup, that has been something some people have been advocating:
	http://www.ietf.org/mail-archive/web/sidr/current/msg05351.html
	``i would assume a million''

Eric

---

From: "Osterweil, Eric" <[email-masked]>
Date: Sat, 23 Mar 2013 12:39:56 -0400

On Mar 23, 2013, at 10:38 AM, Arturo Servin <[email-masked]> wrote:

<snip>

> measurements with RIPE NCC repo). Isn't it?

Just to make a general observation: I'm a little worried that this is a case of the tail wagging the dog…  The issue that Chris seems to be raising (at least, to me) is one of scale and resilience… It seems to me that our efforts to optimize _rsync's_ performance at all costs is starting to require us to relax our security and stability concerns.  At a high level, I think that actually points to rsync as a suboptimal component.  Regardless of the transport used, Chris' comments seem sound to me, and I haven't seem too many comments that rebuff his general concerns.  The fact that rsync doesn't fit well into the model of securing repositories while still making them lively enough for caches is worrisome to me.  I would think that it ought to suggest something other than relaxing our security and stability concerns…  Just my 0.02…

Eric

---

From: Randy Bush <[email-masked]>
Date: Sun, 24 Mar 2013 09:26:48 +0900

> During the first SIDR session I commented that I agree with the need
> to explore new paradigms for distributing RPKI repository data, but
> that i was very disappointed with the analysis being used to motivate
> the exploration.

the ripe crew have looked at server load issues.  we've got client
measurements going back to day one, and every cache displays continuous
and historical performance measurements.  

tim, oleg, bryan, rob, i, and others have and continue to explore
protocols other than rsync.

as we have all said before, no one thinks rsync is gonna scale forever.
but it's scaling well enough for the near term (see actual measurements
by all rpki.net caches, emulation reports, etc).  and it looks as if we
have plenty of time to do a next gen effort carefully.

there seems to be a good bunch of actual measurers/designers/implementors.
they seem to be too busy to post a dozen messages a day to the list.  this
is a feature, not a bug.

randy

---

From: Stephen Kent <[email-masked]>
Date: Mon, 25 Mar 2013 10:45:24 -0400

...
> the number of different repositories to query. But I don't know what 
> to do with that number afterwards.
>
> -- 
> Oleg Muravskiy
> RIPE NCC
Oleg,

The time it takes for an RP to acquire all updated/new objects is a 
function of both
the number of objects and the number of repositories, as illustrated by 
the measurements
that Rob, Andrew et al. have posted, and reflected in the models that 
have been developed.
That's why the projected number of repositories (in the future) has 
become a contentious
topic.

Steve

---

From: Stephen Kent <[email-masked]>
Date: Mon, 25 Mar 2013 10:53:26 -0400

...
> From the RIR stats files that RIRs publish daily we could get the 
> numbers of distinct resource holders. They are:
>
> AFRINIC  1310
> APNIC    7957
> ARIN    35380
> LACNIC   4278
What is the definition of a "distinct resource holder?" Does this 
correspond to an account with the RIR, or is there some other definition?
> Now, these are only the first level resource holders under RIRs. They 
> all *must* have their own CAs in order to participate in RPKI. 
> However, many of these first-level resource holders are NIRs 
Many are NIRs? There are no many NIRs in the world, and today the ones 
in APNIC (the region with the most NIRs) act as RAs, not CAs. So it's 
not clear that one should be counting them.
> or LIRs, who distribute resources further down to their clients. They 
> could choose to manage their clients' RPKI objects within their single 
> CA, but could also give their clients  own certificates, creating next 
> level of CA hierarchy.
The distinction you cite here is not quite correct. Even if an LIR 
manages RPKI objects for folks to whom that have sub-allocated 
resources, each of those folks is represented by a CA. The question is 
who runs that CA, and whether the CA's pub point lives in a different 
repository.
> I find it difficult to estimate how many LIRs will do this, and for 
> how many of their clients. But for RIPE NCC I could see that the 
> number of organisation objects in RIPE DB is 70746, and that should be 
> the upper bound of the number of CAs in our region. I don't have that 
> number for other regions, and don't know if it's applicable in the 
> same way, especially where NIRs are present.
>
NIRs are probably not relevant in this counting approach.

Steve

---

From: Oleg Muravskiy <[email-masked]>
Date: Mon, 25 Mar 2013 16:14:29 +0100

Hi Stephen,

Stephen Kent wrote:
> What is the definition of a "distinct resource holder?" Does this correspond to an account with the RIR, or is there some other
> definition?
>> Now, these are only the first level resource holders under RIRs. They all *must* have their own CAs in order to participate in
>> RPKI. However, many of these first-level resource holders are NIRs 
> Many are NIRs?

No. You broke the line in the wrong place. I meant "many (NIRs or LIRs)".

> There are no many NIRs in the world, and today the ones in APNIC (the region with the most NIRs) act as RAs, not CAs. So it's not
> clear that one should be counting them.
>> or LIRs, who distribute resources further down to their clients. They could choose to manage their clients' RPKI objects within
>> their single CA, but could also give their clients  own certificates, creating next level of CA hierarchy.
> The distinction you cite here is not quite correct. Even if an LIR manages RPKI objects for folks to whom that have sub-allocated
> resources, each of those folks is represented by a CA.

Not necessarily. The LIR could create a ROA for client's assignment, using LIR's allocation certificate.
I'm not saying they should do it like this, but they could. And I have a feeling that might become a common case.
But I think LIRs could say for themselves. For example, how many their clients maintain Whois objects themselves, and for how many
LIRs doing it?

> The question is who runs that CA, and whether the CA's pub point lives in a different repository.
>> I find it difficult to estimate how many LIRs will do this, and for how many of their clients. But for RIPE NCC I could see that
>> the number of organisation objects in RIPE DB is 70746, and that should be the upper bound of the number of CAs in our region. I
>> don't have that number for other regions, and don't know if it's applicable in the same way, especially where NIRs are present.
>>

My point was that it's quite difficult to estimate the upper bound of possible CAs. NIR or LIR does not matter, I agree.

-- 
Oleg Muravskiy
RIPE NCC

---

From: Stephen Kent <[email-masked]>
Date: Mon, 25 Mar 2013 14:45:02 -0400

Oleg,
> No. You broke the line in the wrong place. I meant "many (NIRs or LIRs)".
In your text the "many" is distributed over both terms, NIRs and LIRs. 
You should have just omitted NIRs,
since there are not "many" of them .
> Not necessarily. The LIR could create a ROA for client's assignment, 
> using LIR's allocation certificate. I'm not saying they should do it 
> like this, but they could. And I have a feeling that might become a 
> common case. But I think LIRs could say for themselves. For example, 
> how many their clients maintain Whois objects themselves, and for how 
> many LIRs doing it? 
I agree that an LIR could behave the way you indicated, but in so doing 
it needs to track which
other LIRs provide service to the customer in question, in order to 
generate ROAs for each of them. If
it fails to do so, any connections to other LIRs may be ignored, as the 
NLRI in question will be
represented by a valid ROA pointing to another AS#. That might create a 
liability for the LIR. That's why
Section 7.3.2 of RFC 6480 cites this as the least desirable option.

---

From: Oleg Muravskiy <[email-masked]>
Date: Tue, 26 Mar 2013 11:01:57 +0100

> What is the definition of a "distinct resource holder?" Does this correspond to an account with the RIR, or is there some other

I count the number of distinct reg-ids in the RIR statistics files (ftp.<RIR>.net/pub/stats/<RIR>/delegated-<RIR>-extended-latest,
where <RIR> is in {afrinic,apnic,arin,lacnic}). The file format document defines this field as:

    This is an in-series identifier which uniquely identifies a single organisation, an Internet number resource holder. All records
    in the file with the same reg-id are registered to the same resource holder.

-- 
Oleg Muravskiy
RIPE NCC

---

From: Randy Bush <[email-masked]>
Date: Tue, 26 Mar 2013 20:19:12 +0900

> I count the number of distinct reg-ids in the RIR statistics files
> (ftp.<RIR>.net/pub/stats/<RIR>/delegated-<RIR>-extended-latest, where
> <RIR> is in {afrinic,apnic,arin,lacnic}). 

at least for arin, this algorithm produces a result off by a quite large
factor.

randy

---

From: Oleg Muravskiy <[email-masked]>
Date: Tue, 26 Mar 2013 12:55:08 +0100

Hi Steve,

Stephen Kent wrote:
> In your text the "many" is distributed over both terms, NIRs and LIRs. You should have just omitted NIRs,
>> Not necessarily. The LIR could create a ROA for client's assignment, using LIR's allocation certificate. I'm not saying they
>> should do it like this, but they could. And I have a feeling that might become a common case. But I think LIRs could say for
>> themselves. For example, how many their clients maintain Whois objects themselves, and for how many LIRs doing it? 
> I agree that an LIR could behave the way you indicated, but in so doing it needs to track which
> other LIRs provide service to the customer in question, in order to generate ROAs for each of them. If
> it fails to do so, any connections to other LIRs may be ignored, as the NLRI in question will be
> represented by a valid ROA pointing to another AS#. That might create a liability for the LIR. That's why
If I understand correctly, you refer to multi-homed end-users here.
In our region such users normally receive a provider-independent (PI) address space from RIPE NCC directly, so they will have their
own CA and will have to maintain it.

However, there are also many end-users with provider-aggregatable address space that they received from LIRs. And this is where I
find it quite difficult to continue with estimations, because
- these end-users could still be multi-homed
- or they might need to have own CAs for some other reason
- their LIRs might prefer to give them responsibility for their CA
- or prefer to not give them that responsibility

>From my perspective I do not know any source of data to collect/guess the number of end-users who will need their own CA, or the
number of LIRs who would prefer to delegate CAs to their clients. I think LIRs / operators might know better. This is what I said in
my previous email.

-- 
Oleg Muravskiy
RIPE NCC

---

From: Randy Bush <[email-masked]>
Date: Tue, 26 Mar 2013 21:19:46 +0900

> In our region such users normally receive a provider-independent (PI)
> address space from RIPE NCC directly, so they will have their own CA
> and will have to maintain it.

if PI becomes certifyable, while there will be a CA cert for each, and
EE/ROA etc, at least in the near term, it is likely most will be in the
one repository.  similar PA delegated by LIRs.

randy

---

From: Stephen Kent <[email-masked]>
Date: Tue, 26 Mar 2013 10:34:23 -0400

Oleg,

Thanks for clarifying the term.

I defer to others' understanding of the database in question as to 
whether this is a good estimator.

Steve
------
On 3/26/13 6:01 AM, Oleg Muravskiy wrote:
> <RIR> is in {afrinic,apnic,arin,lacnic}). The file format document 
> defines this field as:
>
>     This is an in-series identifier which uniquely identifies a single
>     organisation, an Internet number resource holder. All records in
>     the file with the same reg-id are registered to the same resource
>     holder.
>
>
> -- 
> Oleg Muravskiy
> RIPE NCC

---

From: Stephen Kent <[email-masked]>
Date: Tue, 26 Mar 2013 11:03:42 -0400

Oleg,

> If I understand correctly, you refer to multi-homed end-users here.
yes, but not PI space holders.
> own CA and will have to maintain it.
I don't know if that is the most common practice, vs. a PA-space user 
moving from being single-homed to
multi-homed. This is what the cited text refers to.
> - these end-users could still be multi-homed
if they are not multi-homed, they are invisible to BGP and thus do not 
need RKI credentials. So, let's
assume that the entities in question are multi-homed, with a PA space 
allocation.
> - or they might need to have own CAs for some other reason
The preferred approach, as noted, is for multi-homed subscribers to be 
represented by a CA.
> - their LIRs might prefer to give them responsibility for their CA
agreed.
> - or prefer to not give them that responsibility
also an option.
> number of LIRs who would prefer to delegate CAs to their clients. I think LIRs / operators might know better. This is what I said in
As best I know, so far we have no LIRs who have gotten this far in the 
RPKI space, so I too look forward
to hearing from those who are considering this next step.

But, irrespective of this detail, isn't it reasonable to use the number 
of (live) ASes as the basis for
the number of pub points (CAs)? To first order, any entity that needs to 
be explicitly represented in the
RPKI is associated with an AS#, whether they are an LIR, a PI space 
holder, or a multi-homed holder
of PA space (from  an LIR).

Steve

---

From: Oleg Muravskiy <[email-masked]>
Date: Tue, 26 Mar 2013 17:05:55 +0100

Stephen Kent wrote:
> I don't know if that is the most common practice, vs. a PA-space user moving from being single-homed to
> if they are not multi-homed, they are invisible to BGP and thus do not need RKI credentials. So, let's
> assume that the entities in question are multi-homed, with a PA space allocation.
> The preferred approach, as noted, is for multi-homed subscribers to be represented by a CA.
> As best I know, so far we have no LIRs who have gotten this far in the RPKI space, so I too look forward
> But, irrespective of this detail, isn't it reasonable to use the number of (live) ASes as the basis for
> the number of pub points (CAs)? To first order, any entity that needs to be explicitly represented in the
> RPKI is associated with an AS#, whether they are an LIR, a PI space holder, or a multi-homed holder

I don't know. I see that the 1 AS = 1 CA is often referred as reasonable.
But I do not see why an AS operator, using single AS#, cannot serve multiple organisations that got their address space from
somewhere else.
So all these organisations will have CAs, will create ROAs, but would not operate own ASes.

-- 
Oleg Muravskiy
RIPE NCC

---

From: Oleg Muravskiy <[email-masked]>
Date: Tue, 26 Mar 2013 17:09:39 +0100

Hi Randy,

> at least for arin, this algorithm produces a result off by a quite large

I do

$ cut -f8 -d'|' delegated-arin-extended-latest |sort -u|wc -l
35394

-- 
Oleg Muravskiy
RIPE NCC

---

From: Stephen Kent <[email-masked]>
Date: Tue, 26 Mar 2013 12:25:38 -0400

Oleg,

Glad to see we are converging, a bit.
> But I do not see why an AS operator, using single AS#, cannot serve 
> multiple organisations that got their address space from somewhere else.
In what sense do you mean "serve" here? Do you mean that the 
organizations in question have, for example,
PI space but no AS#'s, and thus their ROAs all point to the AS# of the 
AS operator? Is that common?

Also, I should amend my comment above, slightly. When we model the 
number of pub points (CAs) we do
add in a factor to account for orgs that have allocations from multiple 
sources, but may be represented
by a single AS#.

Steve

---

From: Oleg Muravskiy <[email-masked]>
Date: Tue, 26 Mar 2013 17:51:00 +0100

Stephen Kent wrote:
> In what sense do you mean "serve" here? Do you mean that the organizations in question have, for example,
> PI space but no AS#'s, and thus their ROAs all point to the AS# of the AS operator?
Yes
> Is that common?
This is what I don't know. I think we need operators to tell.

I'm not opposed to the 1AS = 1CA idea. It's just that in my mind RPKI associates with IP space holders,  not AS operators, because
this is how we do RPKI on RIR level. And on this level we already have more distinct IP space holders than the number of active AS.
I don't know much about LIR to end user level, maybe the number of CAs there will be insignificant.
>
> Also, I should amend my comment above, slightly. When we model the number of pub points (CAs) we do
> add in a factor to account for orgs that have allocations from multiple sources, but may be represented
> by a single AS#.
>
> Steve
>

-- 
Oleg Muravskiy
RIPE NCC

---

From: "Carlos M. Martinez" <[email-masked]>
Date: Tue, 26 Mar 2013 14:50:19 -0300

Adding to what Oleg mentions in LACNIC we also equal a user of the RPKI
system with a legitimate resource holder. It doesn't matter for us
whether the user (resource holder) has an AS or not.

In practice, we have quite a few resource holders with no AS, although
no one of them has yet created ROAs.

regards,

~Carlos

On 3/26/13 1:51 PM, Oleg Muravskiy wrote:
>>>> the number of pub points (CAs)? To first order, any entity that
>>>> needs to be explicitly represented in the
>> AS operator?
> I'm not opposed to the 1AS = 1CA idea. It's just that in my mind RPKI
> associates with IP space holders,  not AS operators, because this is how
> we do RPKI on RIR level. And on this level we already have more distinct
> IP space holders than the number of active AS. I don't know much about
> LIR to end user level, maybe the number of CAs there will be insignificant.
>>

---

From: Stephen Kent <[email-masked]>
Date: Tue, 26 Mar 2013 14:10:09 -0400

Oleg,
> associates with IP space holders,  not AS operators, because this is 
> how we do RPKI on RIR level. And on this level we already have more 
> distinct IP space holders than the number of active AS. I don't know 
> much about LIR to end user level, maybe the number of CAs there will 
> be insignificant.
The RPKI is about who holds both sets of resources: addresses and AS#s. 
So, yes, as an RIR issuing certs
for address space, the focus is on address space holders. When we 
discuss using AS#'s to estimate the
number of CAs it is just because that seems like a reasonable estimator, 
not because it is the basis
for the management of address space.

I am curious, though. When RIPE acts as a CA on behalf of 1300+ entities 
with address space, have you
included the AS#s in the CA certs you issued, when the address space 
holders have AS#'s from RIPE?

Steve

---

From: Stephen Kent <[email-masked]>
Date: Tue, 26 Mar 2013 14:12:56 -0400

Carlos,
> whether the user (resource holder) has an AS or not.
If the address space holder has an AS# allocated by LACNIC, then that 
AS# belongs in the
cert you issue to that entity.
> no one of them has yet created ROAs.
Do you mean to say that these address space holders do not have ASes, or 
that
they do not have ASes allocated through LACNIC?

Steve

---

From: "Carlos M. Martinez" <[email-masked]>
Date: Tue, 26 Mar 2013 15:22:41 -0300

Hi Steve,

> cert you issue to that entity.

Yessir. AS numbers are listed in their resource cert(s).

> they do not have ASes allocated through LACNIC?

Good question. What I can affirm is that they don't have ASes allocated
by LACNIC. However, from my personal 'feeling' of the region I'm willing
to bet that most if not all of them have no ASes at all, although there
could be some 'outliers' that have ASes allocated by other RIRs.

We also have a third situation: Members with only AS numbers and no
LACNIC-allocated addressing resources. We also issue resource certs for
them listing only the AS numbers, even though as of today they are a bit
useless.

~Carlos

<funny-sad-bit>
Every now and then I get email with questions about how to multihome
with no AS
</funny-sad-bit>

---

From: Stephen Kent <[email-masked]>
Date: Tue, 26 Mar 2013 15:01:18 -0400

Carols,

I am very happy to hear that LACNIC has been putting AS#s (allocated by 
LACNIC) in the
certs you are issuing!

If you compare the cert you have issued (w/o AS#'s) against RV or RIS 
data, what do you
see:
     - are these prefixed advertised?
     - do they have one ASN associated with the advertisement, or more 
than one?

Steve

---

From: "Carlos M. Martinez" <[email-masked]>
Date: Tue, 26 Mar 2013 16:13:17 -0300

I haven't performed that analysis (how many source ASes per ORG-ID), it
could be an interesting one.... I'll try to find the time to do that
before Berlin.

Glad you like our resource certs :D

Warm regards,

~Carlos

On 3/26/13 4:01 PM, Stephen Kent wrote:
> Carols,
>
> I am very happy to hear that LACNIC has been putting AS#s (allocated
> by LACNIC) in the
> certs you are issuing!
>

---

From: Arturo Servin <[email-masked]>
Date: Tue, 26 Mar 2013 18:35:08 -0300

>     - are these prefixed advertised?

	Last time around ~79% of our allocations for IPv4 were announced and
~28% of IPv6.

>     - do they have one ASN associated with the advertisement, or more

	Mmm, do not know.

Regards,
as

---

From: Randy Bush <[email-masked]>
Date: Wed, 27 Mar 2013 10:24:09 +0900

hi oleg,

>> at least for arin, this algorithm produces a result off by a quite
>> large factor.
> I do

sorry.  it is not you, it is the arin database (personally, all the rir
databases suck, they just suck differently:-).  what the arin database
considers to be an organization does not correspond to what you consider
one, z/s aside.  e.g. check how many 'goog's you find.

randy

---

From: John Curran <[email-masked]>
Date: Wed, 27 Mar 2013 12:22:13 +0000

On Mar 26, 2013, at 9:24 PM, Randy Bush <[email-masked]> wrote:

> one,

Randy is correct - while "organization ids" do map to organizations,
it is often a many-to-one relationship and hence not necessarily useful
as an indicator of number of organizations (in the typical use of the
term) actually involved.  It is a valid upper bound, but the true value
may be less due to multiple network management entities (Foo-wireline, 
Foo-wireless, etc.), multiple contracts with ARIN, or multiple entries 
for hierarchical organizations (e.g. large governments with multiple 
agencies/departments)

FYI,
/John

---

From: Oleg Muravskiy <[email-masked]>
Date: Wed, 27 Mar 2013 14:42:35 +0100

> for hierarchical organizations (e.g. large governments with multiple 
Yes, we also have cases when multiple members belong to a single legal entity.
But I think we are not interested in different organisations as such, but rather the number of CAs.
So the actual question is whether the number of RPKI CAs will differ from the number of "organization ids" in ARIN stats file?

-- 
Oleg Muravskiy
RIPE NCC

---

From: Arturo Servin <[email-masked]>
Date: Wed, 27 Mar 2013 10:49:45 -0300

I think we are around the bushes.

	We want to know an estimate of the number of CAs in the future.

	The current number does not work because of all the reasons that we
already know.

	Then, I found #CA = #AS a good estimate (yes, there are multiple
factors like an RIR or an ISP may provide a hosted solution and reduce
this number in half, a 20% or even keep it very low if only Tier-1 and
large ISPs/enterprises decide to have CAs and the rest is happy with the
RIR hosted model).

Regards,
as

On 3/27/13 10:42 AM, Oleg Muravskiy wrote:
> So the actual question is whether the number of RPKI CAs will differ
> from the number of "organization ids" in ARIN stats file?
> 
> -- 
> Oleg Muravskiy
> RIPE NCC
> 
> 
>

---

From: John Curran <[email-masked]>
Date: Wed, 27 Mar 2013 13:53:01 +0000

> So the actual question is whether the number of RPKI CAs will differ from the number of "organization ids" in ARIN stats file?

Agreed...  very hard to predict.  For those cases where it is distinct network 
management, it could go either way.  For those where it is hierarchical orgs,
likely to remain distinct.   For those were it is contractual in natural, I would
expect modest consolidation over time.

I believe it is fine as an indicator of potential CA entities within +/- 50%,
but even that is simply the best estimate at this time.

/John

---

From: "Murphy, Sandra" <[email-masked]>
Date: Wed, 27 Mar 2013 14:24:30 +0000

>but even that is simply the best estimate at this time.

No smiley?

--Sandy, speaking as regular ol' member

---

From: John Curran <[email-masked]>
Date: Wed, 27 Mar 2013 14:54:35 +0000

>> but even that is simply the best estimate at this time.

Implied by context...  :-)
/John

---

From: Randy Bush <[email-masked]>
Date: Thu, 28 Mar 2013 01:20:24 +0900

> I think we are around the bushes.

excuse me!!

> We want to know an estimate of the number of CAs in the future.

have you looked at sriram's and sk's work.

randy

---

From: Arturo Servin <[email-masked]>
Date: Wed, 27 Mar 2013 13:29:06 -0300

> have you looked at sriram's and sk's work.

	Yes, I assume
http://tools.ietf.org/agenda/86/slides/slides-86-sidr-1.pdf slide 3.

	Which I think is a good estimate.

-as

---

From: Randy Bush <[email-masked]>
Date: Thu, 28 Mar 2013 01:32:12 +0900

> 	Which I think is a good estimate.

of publication points.  != repositories

randy

---

From: Arturo Servin <[email-masked]>
Date: Wed, 27 Mar 2013 13:43:39 -0300

OK, I will use publication points from now own to avoid confusions.

-as

---

From: "Sriram, Kotikalapudi" <[email-masked]>
Date: Wed, 27 Mar 2013 12:48:56 -0400

>of publication points.  != repositories

Arturo,
Please also see slide #15. We vary the # repositories over a wide range,
and the justification is somewhat along the lines of what you also noted in your post: 
http://www.ietf.org/mail-archive/web/sidr/current/msg05888.html 

A lot of CA's/pub points may be there (~order #ASes), but following observations
similar to your rationale, the # repositories (where the pub points are actually published)
can be somewhat smaller (see slide 15). 

Sriram

---

From: Randy Bush <[email-masked]>
Date: Thu, 28 Mar 2013 02:24:18 +0900

> 	Which I think is a good estimate.
 
actually, i think the number of pub points will be closer to the number
of entries in the rir's datamesses, as they will be issuing the certs
based on their datamesses.

randy