Subject: [Sidrops] Potential negative caching issue in RRDP deployments

From: Ties de Kock <[email-masked]>
Date: Wed, 21 Jul 2021 17:21:18 +0200

Hi all,

We have found and fixed a negative caching issue in RRDP deployments. We have
informed operators of other large RPKI repositories and RRDP endpoint
implementers.

Caching proxies and CDNs cache HTTP responses. The cache TTL for an element is
dependent on the response headers and/or the response code, including caching
requests that did not result in a successful response ("negative caching").
Many CDNs and proxy setups cache 404 responses to reduce the load on the backend
server (e.g. squid caches 404s for 5 minutes by default, defaults for CDNs vary).

For most RRDP repositories the URLs of deltas and snapshots are predictable. An
attacker can repeatedly load notification.xml (for serial initial_serial,
contains links to deltas up to initial_serial, and a reference to the snapshot
for initial_serial) and request the delta/snapshot for initial_serial+1. This
request can result in the proxy caching a 404 response. When notification.xml is
updated for initial_serial+1, clients attempt to retrieve the delta (or snapshot)
for initial_serial+1 which would result in a cached 404. This is then handled
accordingly.

We have implemented two mitigations:

The first is to reduce the negative cache period and writing the delta and
snapshot first, only updating notification.xml after the negative cache period
has passed. This mitigation also prevents sporadic 404s when an RP [not using
keepalive] retrieves files from instances in multiple geographic locations that
are not in sync. This mitigation is transparent to RPs.

The second mitigation is to use unpredictable filenames for deltas and snapshots
(e.g. /<UUID4>/2383/{delta-fdoxzjpt7mtg39fivmdf.xml,snapshot-lvfomf58kph1477mrysx.xml}).
This appears to be widely supported by RPs.

The RIPE NCC has deployed the first mitigation, and we are not aware of the
abuse of this potential issue. I recommend that all repository operators check
their settings and implement either recommendation if they use a caching layer.

Kind regards,
Ties