Subject: [sidr] A quick note from RPKI in the wild

From: Alex Band <[email-masked]>
Date: Mon, 5 Dec 2011 23:02:31 +0100

The RIPE NCC launched a Resource Certification platform on 1.1.2011, where members can choose to set up a certificate listing their address blocks. They can run RPKI software themselves, or use a hosted platform in our web portal. So far 715 out of our ~7500 members have done this. Out of the top 100 largest LIRs in our region, 28 have a certificate set up. About half of the enabled members have the certificate solely to get validatable proof of holdership of the address space they hold (for now?), the rest use it for BGP origin validation. 

By the latter group, 416 Route Origin Authorization (ROA) objects have been created, covering the equivalent of 230,000 /24 prefixes and 8,600 /32 IPv6 prefixes. MaxLength in ROAs is sorely misunderstood, lots of education is needed there. Most leave the field blank, causing more specific announcements to be invalid.

Lately though, there lots of activity with regards to tooling and testbeds. EuroTransit have set up a testbed with Randy/Rob's tools, as well as the NCC's: 

http://rpki01.fra2.de.euro-transit.net/documentation.html

They also made two public RPKI capable Juniper routers available. You can log into them using telnet with these details:

IPs: 193.34.50.25 and 193.34.50.26
user: rpki
password: testbed

You can run commands such as "show validation database" , "show validation statistics", "show validation session", "show bgp neighbor", "show bgp summary" and lastly "show route protocol bgp validation-state", followed by the state (valid, invalid, unknown or unverified)

I'm curious to hear what you think.

Cheers,

Alex Band
RIPE NCC

P.S. Here you can grab a pre-release of the NCC Validation tool that they run there (requires *NIX w/ Java, rsync):
https://certification.ripe.net/content/public-repo/releases/net/ripe/rpki-validator/rpki-validator-app/1.0.14/rpki-validator-app-1.0.14-bin.zip

---

From: Matthias Waehlisch <[email-masked]>
Date: Mon, 5 Dec 2011 16:39:46 -0600 (Mittelamerikanische Normalzeit)

Hi Alex,

  great! Also the extended RPKI Validator web interface.

  Just for completeness: As mentioned at the last IETF meeting, we also 
provide a public RTR origin server instance. You can connect via 
unprotected TCP or SSH:

  * http://rpki.realmv6.org/wiki/Usage#ChooseanRTR-ServerImplementation

  It currently runs rtr-origin but as soon as NCC Validation tool is 
released we will also set such a cache server up.

  Using both, rpki01.fra2.de.euro-transit.net and rpki.realmv6.org, 
allows to test failover.

  Overall, nice playground!

Cheers
  matthias

--
Matthias Waehlisch
.  Freie Universitaet Berlin, Inst. fuer Informatik, AG CST
.  Takustr. 9, D-14195 Berlin, Germany
.. mailto:[email-masked] .. http://www.inf.fu-berlin.de/~waehl
:. Also: http://inet.cpt.haw-hamburg.de .. http://www.link-lab.net

---

From: "Sriram, Kotikalapudi" <[email-masked]>
Date: Wed, 7 Dec 2011 17:28:46 -0500

>By the latter group, 416 Route Origin Authorization (ROA) objects have been created, covering the equivalent of 230,000 /24 prefixes and 8,600 /32 IPv6 prefixes. 
>MaxLength in ROAs is sorely misunderstood, lots of education is needed there. Most leave the field blank, causing more specific announcements to be invalid.

Alex,

Can you comment if (a) the more specifics are announced from the same AS as seen 
in the ROA, or (b) they are announced from a different AS (e.g., customer AS)?
If it is the latter, then even the ROA creation (for the less specific)
is possibly in violation of: 
"Before issuing a ROA for a super-block, an operator MUST ensure that
   any sub-allocations from that block which are announced by other ASs,
   e.g. customers, have correct ROAs in the RPKI." from the origin-ops doc. 
http://tools.ietf.org/html/draft-ietf-sidr-origin-ops-13#section-3 

As for education, the origin-ops document as well as the use-cases
document (Sections 3, 7) 
http://tools.ietf.org/html/draft-ietf-sidr-usecases-03 
can be referred.

Sriram

---

From: Matthias Waehlisch <[email-masked]>
Date: Wed, 7 Dec 2011 22:36:20 -0600 (Mittelamerikanische Normalzeit)

Hi Sriram,

  I'm not sure if I understand your quote correctly: Do you only 
consider the case where the super-block encloses the sub-allocations 
(e.g., x.y.0.0/16-32 and x.y.z.0/24)? In this case, that would be no 
MaxLength violation, right?.

  If the super-block does not enclose the announced prefix but there is 
an incorrect origin AS, you cannot easily decide about the primary 
reason for the invalidation (i.e., whether the ROA has been created 
without considering sub-allocations or the origin AS has been announced 
by mistake).

  Anyhow, for the measurements we presented last IETF, 80% of the 
invalid announcements are due to invalid origin ASes. The remaining 20% 
are due to obvious MaxLength violation (i.e., same origin AS in BGP 
update and ROA, but longer prefix length in BGP update). Regarding the 
announcements with incorrect origin AS, for approx. 85% of these BGP 
updates there exists at least one ROA with a longer MaxLength value 
compared to the prefix length announced (i.e., the ROA encloses the 
incorrectly announced prefix). ... Maybe it makes sense to investigate 
this case in more detail. For example, looking if there is a direct 
peering between the ROA origin AS and the origin AS within the BGP 
update.

Thanks
  matthias

On Wed, 7 Dec 2011, Sriram, Kotikalapudi wrote:

> >By the latter group, 416 Route Origin Authorization (ROA) objects 
> >have been created, covering the equivalent of 230,000 /24 prefixes 
> >and 8,600 /32 IPv6 prefixes. MaxLength in ROAs is sorely 
> >misunderstood, lots of education is needed there. Most leave the 
> >field blank, causing more specific announcements to be invalid.
> 
> Alex,
> 
> _______________________________________________
> sidr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidr
>

---

From: "Sriram, Kotikalapudi" <[email-masked]>
Date: Thu, 8 Dec 2011 08:26:00 -0500

Hi Matthias:

Thanks. My responses inline.

Sriram
________________________________________
>From: Matthias Waehlisch [[email-masked]]
>Sent: Wednesday, December 07, 2011 11:36 PM

KS: No. I am taking about something different.
If ISP A (with AS A) creates a ROA for its super-block x.y.0.0/16
and its Customer B (with AS B) has a sub-allocation x.y.z.0/24 and
has NOT created a ROA yet, then the customer's announcement will be Invalid.
The same effect happens whether ISP A's ROA is:
{ROA: x.y.0.0/16, AS A} (w/o maxLength; defaulted to maxLength = 16)
or, {ROA: x.y.0.0/16, AS A, maxLength = 24}
or, {ROA: x.y.0.0/16, AS A, maxLength = 18}.
With any of the above ROAs, Cust. B's announcement is Invalid
because it doesn't have a ROA yet for its sub-allocation.

KS: The BCP I quoted from origin-ops requires that ISP A MUST
register its ROA only after the customer sub-allocation ROAs
have been created (or, register all necessary ROAs simultaneously).
In the above example: ISP A can create a ROA for Customer B
along with its own ROA as follows:
{ROA: x.y.0.0/16, AS A} and {ROA: x.y.z.0/24, AS B}
That ensures that Customer B is not in trouble.

This is the bottom up approach for ROA creation.
Question is: Do the ISP's (or super-block owners) in your
measurement data seem to be compliant with said BCP. 

KS: Exactly. That is the info I was requesting. Well, it may not be
just the directly peering ASs that you look for. 
There can be customer's customers, etc.
That is, children, grand children, great grand children kind of sub-allocations
each with its own AS. Please see Slide 8 of this presentation 
http://www.ietf.org/proceedings/82/slides/sidr-2.pdf  
for measurements on customer cone structure
(i.e., # prefixes originated vs. their depth in AS path length) 
for several real ISPs.

Sriram

---

From: Matthias Waehlisch <[email-masked]>
Date: Thu, 8 Dec 2011 11:10:32 -0600 (Mittelamerikanische Normalzeit)

Hi Sriram,

> because it doesn't have a ROA yet for its sub-allocation.
  definitely agree.

> measurement data seem to be compliant with said BCP. 
  My point was that it is not too easy to decide if the ISPs are 
compliant with the BCP. But I think I got your point: If the origin AS 
in the Update has (somehow) a *customer* relationship with the ROA AS it 
is more likely that the ISP's ROA creation is in violation of the BCP.

  In our last IETF presentation, we identified for the incorrect origin 
ASes that there exists a ROA Origin that is 1 hop away (ignoring 
prepending) from the announced origin AS in 90% of the cases. Coming 
back to your question: Yes, most of the announcements are invalid 
probably due to the violation of the BCP but I will try to verify in 
more detail.

Cheers
  matthias

---

From: Carlos Martinez-Cagnazzo <[email-masked]>
Date: Thu, 8 Dec 2011 18:00:53 -0200

A few numbers we gathered with Arturo. After processing yesterday's
RIPE RIS dump from the PTT Metro Sao Paulo sensor, we detected

-  ~5300 invalid prefixes
- ~2800 valid prefixes

Out of the invalid prefixes we have:

- ~ 3100 invalids due to bad maxLen
- ~ 2200 invalids due to wrong origin AS

I fully agree with Alex that maxLen is currently badly misunderstood
by the community at large. A lot of work is badly needed in order to
improve this situation.

However, surprisingly enough, many people do not know which AS should
originate their own prefixes. Out of the collected data it looks like
large companies with operations in different countries are the "worst
offenders" in this case.

We have put online a little tool to play with this data:
http://www.labs.lacnic.net/rpkitools/looking_glass/

The database is refreshed once a day (7 am GMT-2). All feedback is of
course welcome. Beware that it is pretty crude at this point and be
warned that it can blow up in your face without warning :-)

We are working on:

- including more queries
- allowing CSV downloads of result sets
- RESTful-like interface
- WHOIS interfacing for richer display

Warm regards,

Carlos

On Thu, Dec 8, 2011 at 3:10 PM, Matthias Waehlisch <[email-masked]> wrote:
>>  >I'm not sure if I understand your quote correctly: Do you only

-- 
--
=========================
Carlos M. Martinez-Cagnazzo
http://cagnazzo.name
=========================

---

From: Matthias Waehlisch <[email-masked]>
Date: Thu, 8 Dec 2011 15:15:25 -0600 (Mittelamerikanische Normalzeit)

Hi Carlos,

> large companies with operations in different countries are the "worst 
  regarding the first sentence: Have you verified by discussions with 
operators?

Thanks
  matthias

---

From: Randy Bush <[email-masked]>
Date: Fri, 09 Dec 2011 07:30:29 +0900

> compliant with the BCP.

actually, test versions of the GUIs are now telling you if a roa you are
asking to create stomps on anything in the real routing table.

randy

---

From: Arturo Servin <[email-masked]>
Date: Thu, 8 Dec 2011 20:38:07 -0200

Randy,

	How do you know what is the invalid data, the roa or the route?

Regards,
as

---

From: Randy Bush <[email-masked]>
Date: Fri, 09 Dec 2011 07:45:36 +0900

>> asking to create stomps on anything in the real routing table.

did i say "invalid data?"  please do not insert words into my keyboard.

randy

---

From: Arturo Servin <[email-masked]>
Date: Thu, 8 Dec 2011 20:58:55 -0200

Well, I assumed that if a warning showed up it was because something was not correct somewhere and the error/warning messages gave you a hint of why.

	I imagine then that it just warns that the roa does not match the routing table and it is up to roa-creator to figure out what the error is (the roa or the route).

Regards,
as

---

From: Tim Bruijnzeels <[email-masked]>
Date: Fri, 9 Dec 2011 11:08:00 +0100

Hi,

> I imagine then that it just warns that the roa does not match the routing table and it is up to roa-creator to figure out what the error is (the roa or the route).

Speaking for the implementation in the RIPE NCC hosted pilot service (managed CA hosted by us, UI in member portal), and validator....

On the portal page where members can manage their ROAs we also check the announcements that we see(*) for their certified address space, against their ROAs. Yes, this is a warning only. It is up to the member to make the judgement what is right and configure things accordingly. Note that our members can not, at this time, certify their own clients for part of their address space. So if a client of a member uses a prefix then it's up the member to ensure that the ROAs for this prefix are set up correctly.

Members can also subscribe to 'roa-alerts'. If they do they get a daily email about unknown and invalid announcements that we see for them. This is not real-time alerting of course. We're actually also thinking about that, but it would require a very different technical solution and we're focusing on other stuff first... at this stage I think this is mainly useful for members to make sure that their ROAs actually keep matching their announcements. Although if some rogue announcement is happening continuously it will show up. We have also received some feedback where someone mentioned: "oh... I forgot that we were doing that announcement there.. I'd better fix it..". I am not sure if that was based on the UI or the mailing though..

All this was added fairly recently so we actually have quite a lot of ROAs that were created before this functionality was there. We are planning to do a one-time mailing soon where we alert members to announcements invalidated by their ROAs. I am not keen on spamming people, but with thousands of announcements being invalidated, presumably largely because the ROAs are wrong -- not the announcements... I don't see operators trusting this. So, data quality and fixing this is important.

We also compare validated ROAs to the announcements we see in the new version of our validator. And yes, it's showing a *lot* of invalids. This is the RP side of course, so changing those ROAs is not really an option ;) We do provide knobs in the validator though to ignore ROAs for certain prefixes and add your own ASN, prefix, maxLength entries as though ROAs existed for them, but managing this doesn't scale to thousands... 

An official release announcement is scheduled for Tuesday, but it's already available for download, and your feedback is appreciated:

https://certification.ripe.net/content/public-repo/releases/net/ripe/rpki-validator/rpki-validator-app/2.0.0/rpki-validator-app-2.0.0-bin.zip

Regards,
Tim

*: We use the 8-hourly aggregate of routes seen by RIS Route Collectors. Published here: http://www.ris.ripe.net/dumps/