Subject: [Tools-discuss] Consider requiring login for idnits and rfcdiff

From: Robert Sparks <[email-masked]>
Date: Fri, 27 Aug 2021 16:56:19 -0500

All -

We currently let anonymous users exercise the idnits and rfcdiff 
services. We ask robots not to, using the usual mechanisms, but there 
are robots that don't honor the requests.

These requests take resources that could reasonably be used doing 
something else.

We are considering mechanisms to help shed that load. The first is to 
require login with datatracker credentials. If you know of something 
that would break, please say so.

RjS

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 28 Aug 2021 10:22:43 +1200

> The first is to 
> require login with datatracker credentials.

No problem, as long as you add a login button to them.

Regards
   Brian

On 28-Aug-21 09:56, Robert Sparks wrote:
> ___________________________________________________________
> Tools-discuss mailing list - [email-masked]
> This list is for discussion, not for action requests or bug reports.
> * Report datatracker and mailarchive bugs to: [email-masked]
> * Report tools.ietf.org bugs to: [email-masked]
> * Report all other bugs or issues to: [email-masked]
> List info (including how to Unsubscribe): https://www.ietf.org/mailman/listinfo/tools-discuss
>

---

From: Bob Hinden <[email-masked]>
Date: Fri, 27 Aug 2021 21:04:57 -0700

Robert,

> On Aug 27, 2021, at 2:56 PM, Robert Sparks <[email-masked]> wrote:
> 
> All -
> 
> We currently let anonymous users exercise the idnits and rfcdiff services. We ask robots not to, using the usual mechanisms, but there are robots that don't honor the requests.
> 
> These requests take resources that could reasonably be used doing something else.
> 
> We are considering mechanisms to help shed that load. The first is to require login with datatracker credentials. If you know of something that would break, please say so.

I think requiring datatracker login would be fine.

Bob

---

From: Julian Reschke <[email-masked]>
Date: Sat, 28 Aug 2021 06:10:32 +0200

Am 27.08.2021 um 23:56 schrieb Robert Sparks:
> ...

Hm.

This seems to assume that everybody who ocasionally follows a link to
diffs has and wants a datatracker account. I don't think that is true.

Best regards, Julian

---

From: Jay Daley <[email-masked]>
Date: Sat, 28 Aug 2021 16:58:39 +1200

> diffs has and wants a datatracker account. I don't think that is true.

It sounds as if you’re saying that once a diff has been created there is a use case for retaining that diff and publishing it, for at least a period, by a public URL - is that correct?

Jay

-- 
Jay Daley
IETF Executive Director

---

From: Lars Eggert <[email-masked]>
Date: Sat, 28 Aug 2021 08:56:52 +0300

Can they be fronted by Cloudflare? Or all these all unique requests?

-- 
Sent from a mobile device; please excuse typos.

> On Aug 28, 2021, at 00:56, Robert Sparks <[email-masked]> wrote:
> 
> All -
>

---

From: Julian Reschke <[email-masked]>
Date: Sat, 28 Aug 2021 09:48:53 +0200

Am 28.08.2021 um 06:58 schrieb Jay Daley:
> It sounds as if you’re saying that once a diff has been created there is
> a use case for retaining that diff and publishing it, for at least a
> period, by a public URL - is that correct?
> ...

Absolutely.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 28 Aug 2021 10:12:16 +0200

On 27. Aug 2021, at 23:56, Robert Sparks <[email-masked]> wrote:
> We currently let anonymous users exercise the idnits and rfcdiff services. […]
> 

The kdrfc tool in kramdown-rfc offers remote web invocation of xml2rfc, not yet idnits or rfcdiff, for the case that no local installation of xml2rfc can be found.
Remote invocation of idnits is a logical fallback to add, as this also needs to be “installed” (made available to kdrfc).

With your proposed change, this then would need the addition of a login function to kdrfc.
A login function however also on the roadmap for adding a command line I-D submission feature for kdrfc.

So nothing would break right now, but the next steps for what can be added to kdrfc would change.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Sat, 28 Aug 2021 10:54:19 +0200

> are robots that don't honor the requests.

Out of curiosity: are these robots accessing *existing* diffs (that have
been requested before), or are they creating new diffs?

How do you know that those are robots?

Keep in mind that draft announcements contain links to diffs, so the
first client following these will invoke "rfcdiff", no?

Best regards, Julian

---

From: Robert Sparks <[email-masked]>
Date: Sat, 28 Aug 2021 08:29:25 -0500

> Can they be fronted by Cloudflare? Or all these all unique requests?

They are too unique. Quite often they involve upload (idnits in 
particular) of different files with the same name.

For the ones that don't, they quite often identify a draft without a 
version (rfcdiff in particular) and ask the tool to figure out source to 
use. A constant cache timeout would be a real inconvenience for users 
when they publish a new version, and invalidation, while possible, would 
require some complicated coordination with the datatracker.

---

From: Robert Sparks <[email-masked]>
Date: Sat, 28 Aug 2021 08:33:01 -0500

This would only make sense if the diff were of two things that were 
immutable. That happens sometime, but not all the time. In particular, 
diffs that use a pronoun (a draft name with no version) does not have an 
immutable diff.

We could look to making the links to diffs from the datatracker that are 
of things we know won't change be of the form that uses very explicit 
source urls, and that could become at least locally cacheable, but as I 
noted a moment ago, trying to utilize caching across the entire request 
space will be quite complicate.

On 8/27/21 11:58 PM, Jay Daley wrote:
> It sounds as if you’re saying that once a diff has been created there 
> is a use case for retaining that diff and publishing it, for at least 
> a period, by a public URL - is that correct?
>
> Jay
>
> -- 
> Jay Daley
> IETF Executive Director
>
>> Best regards, Julian
>>
>> List info (including how to Unsubscribe): 
>> https://www.ietf.org/mailman/listinfo/tools-discuss
>>
>

---

From: Robert Sparks <[email-masked]>
Date: Sat, 28 Aug 2021 08:35:01 -0500

> How do you know that those are robots?

Quite clear client id in the requests, and the pattern is not "looking 
at drafts that have recently changed".

---

From: Bob Hinden <[email-masked]>
Date: Sat, 28 Aug 2021 06:40:14 -0700

Jay,

> On Aug 27, 2021, at 9:58 PM, Jay Daley <[email-masked]> wrote:
> ….
>>> 

To be specific, when an Internet Draft is published it includes a links that include one to a diff from  the previous version.  For example:

> 
> The IETF datatracker status page for this draft is:
> https://datatracker.ietf.org/doc/draft-ietf-6man-ipv6-alt-mark/
> 
> There is also an htmlized version available at:
> https://datatracker.ietf.org/doc/html/draft-ietf-6man-ipv6-alt-mark-09
> 
> A diff from the previous version is available at:
> https://www.ietf.org/rfcdiff?url2=draft-ietf-6man-ipv6-alt-mark-09

I think Julian is correct, these links should work without a login.

Bob

---

From: Robert Sparks <[email-masked]>
Date: Sat, 28 Aug 2021 08:57:23 -0500

Perhaps we should move the diffs like those (where we know the result of 
the diff is invariant) to a different service that can be aggressively 
cached.

To be clear - I mean change the form of the URL to somewhere else for 
those, leaving the "diff arbitrary things" service where it is and 
require login for that.

RjS

---

From: Eric Rescorla <[email-masked]>
Date: Sat, 28 Aug 2021 08:13:07 -0700

In assessing this question it would be helpful to have a sense of the CPU
consumption of this service.

For instance, how many requests per second are there? What fraction of the
machine does it take up?

-Ekr

On Sat, Aug 28, 2021 at 6:57 AM Robert Sparks <[email-masked]> wrote:

> those, leaving the "diff arbitrary things" service where it is and require
> login for that.
>
> RjS
> On Aug 27, 2021, at 9:58 PM, Jay Daley <[email-masked]> <[email-masked]> wrote:
> ….
>
> The IETF datatracker status page for this draft is:https://datatracker.ietf.org/doc/draft-ietf-6man-ipv6-alt-mark/
>
> There is also an htmlized version available at:https://datatracker.ietf.org/doc/html/draft-ietf-6man-ipv6-alt-mark-09
>
> A diff from the previous version is available at:https://www.ietf.org/rfcdiff?url2=draft-ietf-6man-ipv6-alt-mark-09
>

---

From: Michael Richardson <[email-masked]>
Date: Sat, 28 Aug 2021 15:13:00 -0400

Robert Sparks <[email-masked]> wrote:
    > We currently let anonymous users exercise the idnits and rfcdiff services. We
    > ask robots not to, using the usual mechanisms, but there are robots that
    > don't honor the requests.

I have no opinion about idnits.
I have and do use rfcdiff URLs as part of postings about what changed in
drafts (as does the DT!), and also in slides.  I'm guessing that it's these
links that robots are following,  not trying to recurse on the service URL
itself.   A cloudflare or other CAPTCHA might work as well here.

As I understand you, following these URLs would require having a DT login.
I feel that this might feel (emphasis about this world) like putting a paywall on IETF content.
Yes, a login is free, but

    > We are considering mechanisms to help shed that load. The first is to require
    > login with datatracker credentials. If you know of something that would
    > break, please say so.

Ideally, it would require a DT login only to create new rfcdiff content, but
not to follow the links.  I don't think that that is workable as things are
setup now.

The content would ideally cache more, although I've often used
a non-versioned .txt file from github as url2, which lets me hit refresh as
things get merged.

In order to make the results public, we'd have to have rfcdiff save it's
output somewhere and return a link to that content.  I recognize that I can
save the results and find a place to put it and use that link.

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Michael Richardson <[email-masked]>
Date: Sun, 29 Aug 2021 17:31:01 -0400

Robert Sparks <[email-masked]> wrote:
    > immutable. That happens sometime, but not all the time. In particular, diffs
    > that use a pronoun (a draft name with no version) does not have an immutable
    > diff.

Exactly.

And this is a useful thing, particularly for design teams having a meeting.
Being able to share a link that updates as the design team merges commits
(generating the .txt file into github or a private web server).

For the design team use, having a DT login is probably fine.  Thanks to
pandemic and meetecho, most contributors have a DT login, but that wasn't
the case three years ago.

    > We could look to making the links to diffs from the datatracker that are of
    > things we know won't change be of the form that uses very explicit source
    > urls, and that could become at least locally cacheable, but as I noted a
    > moment ago, trying to utilize caching across the entire request space will be
    > quite complicate.

I think that this is a good way to go.

The emails that the DT emits on ID publication contain links, and are
archived and indexed by search engines are likely the places where we are
experiencing pain.

I think that it is reasonable to make www.ietf.org/rfcdiff require DT login,
but also, to change the link that the DT emits on ID posting to be something
else which is more easily cached.

You will recall I asked if we can have rfcdiff generate self-referential URLs
at the top.

---

From: Michael Richardson <[email-masked]>
Date: Sun, 29 Aug 2021 17:32:41 -0400

> For instance, how many requests per second are there? What fraction of the

Agreed... is this a problem that can be solved with a bit of cash.

I've suggested at least, https://rfcdiff.ietf.org/ would let us move it
around, and perhaps use the "cloud" to spin CPU up/down for this effort.

---

From: Michael Richardson <[email-masked]>
Date: Sun, 29 Aug 2021 17:35:00 -0400

{again, I have no problem with DT login for idnits.  Can we split the
discussion here?}

Robert Sparks <[email-masked]> wrote:
    > For the ones that don't, they quite often identify a draft without a version
    > (rfcdiff in particular) and ask the tool to figure out source to use. A
    > constant cache timeout would be a real inconvenience for users when they
    > publish a new version, and invalidation, while possible, would require some
    > complicated coordination with the datatracker.

The DT emails say:

} A diff from the previous version is available at:
} https://www.ietf.org/rfcdiff?url2=draft-ietf-anima-constrained-voucher-13

with a version attached, with the implied url1 being -12.
That seems very definite and very cachable, right?

I think we are in agreement that having a different URL for these emails
which went to an aggressively cached thing would be just fine.

---

From: Michael Richardson <[email-masked]>
Date: Sun, 29 Aug 2021 17:37:00 -0400

> Quite clear client id in the requests, and the pattern is not "looking at
    > drafts that have recently changed".

name and shame? :-)

Are some of these robots actually email virus scanning things which are
trying to judge the reputation of the links?

Or are we strictly talking about mis-behaving search engines?

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 29 Aug 2021 23:40:07 +0200

On 2021-08-28, at 15:33, Robert Sparks <[email-masked]> wrote:
> 
> This would only make sense if the diff were of two things that were immutable. That happens sometime, but not all the time. In particular, diffs that use a pronoun (a draft name with no version) does not have an immutable diff.

The varying diffs could redirect to the immutable ones.
(So the processing would be about simply finding the right redirect URL, which in turn could be temporarily cached.)

> We could look to making the links to diffs from the datatracker that are of things we know won't change be of the form that uses very explicit source urls, and that could become at least locally cacheable, but as I noted a moment ago, trying to utilize caching across the entire request space will be quite complicate.

BTW,
is immutable.

Grüße, Carsten

---

From: IETF Executive Director <[email-masked]>
Date: Mon, 30 Aug 2021 10:29:12 +1200

Let me try and put this altogether:

First some definitions:

A.  Category A is 
- a draft that is in DT specified by its name with a version number
- an RFC in DT

B. Category B is
- a draft in DT specified by its name without a version number

C. Category C
- any document that is not within DT including drafts in GitHub

Then the process

1.  Requesting a new diff requires a DT login

2.  Requesting a diff between two Category A documents generates a diff that is cached and is accessible by an unauthenticated URL and which is never re-generated.

3.  Requesting a diff between a Category B document and any other document creates a diff that is cached and is accessible by an unauthenticated URL.  When a new version of the draft is submitted, all such diffs are automatically regenerated.

4.  Requesting a diff between a Category C document and a Category A or Category C document creates a diff that is cached and is accessible by an unauthenticated URL.  There is no automatic regeneration, instead a manual regeneration must be requested, through one of two mechanisms
- by any authenticated user pressing a "regenerate" button
- by an API call using a DT API key

Does that sound about right?

Jay

> On 30/08/2021, at 9:40 AM, Carsten Bormann <[email-masked]> wrote:
> 

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Eric Rescorla <[email-masked]>
Date: Sun, 29 Aug 2021 15:31:08 -0700

On Sun, Aug 29, 2021 at 3:29 PM IETF Executive Director <
[email-masked]> wrote:

> 2.  Requesting a diff between two Category A documents generates a diff
> that is cached and is accessible by an unauthenticated URL and which is
> never re-generated.
>
> 3.  Requesting a diff between a Category B document and any other document
> creates a diff that is cached and is accessible by an unauthenticated URL.
> When a new version of the draft is submitted, all such diffs are
> automatically regenerated.
>
> 4.  Requesting a diff between a Category C document and a Category A or
> Category C document creates a diff that is cached and is accessible by an
> unauthenticated URL.  There is no automatic regeneration, instead a manual
> regeneration must be requested, through one of two mechanisms

Are you summarizing the proposal or are you asking if people agree to this?
If the former, yes, this seems right. If the latter, I'd like to understand
the cost of the current unauthenticated design before having an opinion.

-Ekr

Jay
> > (So the processing would be about simply finding the right redirect URL,
> which in turn could be temporarily cached.)
> >
> >> We could look to making the links to diffs from the datatracker that
> are of things we know won't change be of the form that uses very explicit

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 30 Aug 2021 00:32:42 +0200

> Does that sound about right?

That procedure would invalidate frequently used links such as those on https://github.com/core-wg/senml-versions :

	• Compare Editor's Copy to Working Group Draft

This needs to work without explicit interaction.
(Of course, a github action could be developed for pressing the “regenerate” button, but this all strikes me as fragile and error-prone.)

Grüße, Carsten

---

From: IETF Executive Director <[email-masked]>
Date: Mon, 30 Aug 2021 10:35:23 +1200

> On 30/08/2021, at 10:31 AM, Eric Rescorla <[email-masked]> wrote:
> 
> 
> 
> On Sun, Aug 29, 2021 at 3:29 PM IETF Executive Director <[email-masked] <mailto:[email-masked]>> wrote:
> Are you summarizing the proposal or are you asking if people agree to this? If the former, yes, this seems right. If the latter, I'd like to understand the cost of the current unauthenticated design before having an opinion.

Trying to summarise.  This could be written without mentioning authentication and the generate/cache/regenerate logic would be unchanged.

Jay

> 
> -Ekr
> 
> Jay
> 
> 
> > On 30/08/2021, at 9:40 AM, Carsten Bormann <[email-masked] <mailto:[email-masked]>> wrote:
> > 
> > On 2021-08-28, at 15:33, Robert Sparks <[email-masked] <mailto:[email-masked]>> wrote:
> >> 
> > https://www.ietf.org/rfcdiff?url2=draft-ietf-6man-ipv6-alt-mark-09 <https://www.ietf.org/rfcdiff?url2=draft-ietf-6man-ipv6-alt-mark-09>
> > is immutable.
> > 
> > Grüße, Carsten
> > 
> > Tools-discuss mailing list - [email-masked] <mailto:[email-masked]>
> > * Report datatracker and mailarchive bugs to: [email-masked] <mailto:[email-masked]>
> > * Report tools.ietf.org <http://tools.ietf.org/> bugs to: [email-masked] <mailto:[email-masked]>
> > * Report all other bugs or issues to: [email-masked] <mailto:[email-masked]>
> > List info (including how to Unsubscribe): https://www.ietf.org/mailman/listinfo/tools-discuss <https://www.ietf.org/mailman/listinfo/tools-discuss>
> 
> -- 
> Jay Daley
> IETF Executive Director
> [email-masked] <mailto:[email-masked]>
> 
> 
> 
> Tools-discuss mailing list - [email-masked] <mailto:[email-masked]>
> * Report datatracker and mailarchive bugs to: [email-masked] <mailto:[email-masked]>
> * Report tools.ietf.org <http://tools.ietf.org/> bugs to: [email-masked] <mailto:[email-masked]>
> * Report all other bugs or issues to: [email-masked] <mailto:[email-masked]>
> List info (including how to Unsubscribe): https://www.ietf.org/mailman/listinfo/tools-discuss <https://www.ietf.org/mailman/listinfo/tools-discuss>
-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: IETF Executive Director <[email-masked]>
Date: Mon, 30 Aug 2021 10:37:08 +1200

> On 30/08/2021, at 10:32 AM, Carsten Bormann <[email-masked]> wrote:
> 

That suggests the need for someone to request a diff that is not cached and is regenerated each time it is requested.  Is that correct?

Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 30 Aug 2021 00:42:05 +0200

> That suggests the need for someone to request a diff that is not cached and is regenerated each time it is requested.  Is that correct?

I can’t parse that.

Clicking on the link on the github page indeed requests a diff that cannot be cached (beyond caching the downloaded github version and using a cached result only when data are unchanged or Not Modified etc. is detected).

In my proposal to redirect varying content URIs to immutable URIs, the latter could employ a hash and/or ETags.

Grüße, Carsten

---

From: "Martin Thomson" <[email-masked]>
Date: Mon, 30 Aug 2021 09:13:36 +1000

The usual pattern for dealing with this is to accept the POST request, generate the content, give that content a new URI (either something random or a function of its inputs, it doesn't matter), and then return a response that redirects to the new URI.  That works for any form of input.  It also means that you can use the generated URI without fresh authorization, including caching.

---

From: "Martin Thomson" <[email-masked]>
Date: Mon, 30 Aug 2021 09:17:54 +1000

On Mon, Aug 30, 2021, at 08:42, Carsten Bormann wrote:
> Clicking on the link on the github page indeed requests a diff that 
> cannot be cached (beyond caching the downloaded github version and 
> using a cached result only when data are unchanged or Not Modified etc. 
> is detected).

For the specific case you describe - assuming the load is unacceptable - the link could be gated behind a login.  Following that link is often done by a human using their browser.  Not all of those humans will have a login on datatracker, but many of those that care about that link do, and asking the others to make an account would not be an unreasonable imposition.

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 30 Aug 2021 01:32:58 +0200

On 30. Aug 2021, at 01:17, Martin Thomson <[email-masked]> wrote:
> 
> Not all of those humans will have a login on datatracker, but many of those that care about that link do, and asking the others to make an account would not be an unreasonable imposition.

There is a certain cost (unintended consequence) to that:
The more people we incentivize to get datatracker accounts, the less we can rely on them as something that can serve as a speedbump to limit the set of people who can do something.
But maybe it is necessary to say goodbye to the simplistic idea that datatracker account ~= IETFer.

Grüße, Carsten

---

From: "Martin Thomson" <[email-masked]>
Date: Mon, 30 Aug 2021 09:43:05 +1000

On Mon, Aug 30, 2021, at 09:32, Carsten Bormann wrote:
> The more people we incentivize to get datatracker accounts, the less we 
> can rely on them as something that can serve as a speedbump to limit 
> the set of people who can do something.

True.  But we already have that problem.  Ask how many accounts exist.

(There are other ways to manage this that don't end up relying on this mechanism, but I don't know if those have already been exercised.)

---

From: IETF Executive Director <[email-masked]>
Date: Mon, 30 Aug 2021 11:54:11 +1200

Let me have another go at putting this together, for now leaving aside the authentication part:

A.  Category A is 

B. Category B is

C. Category C

Then the process

1.  Requesting a diff between two Category A documents generates a diff that is cached and is accessible by the same URI that was used to make the request.  Accessing the URI never results in re-generation unless the cache has expired.

2.  Requesting a diff between a Category B document and any other document creates a diff that is cached and returns a new URI pointing to that cached diff.  When the new URI is accessed then the cached diff is returned unless it has expired in which case it is regenerated first.  When the original URI is accessed then that:
- triggers a check to see if any Category B document has changed and if so then a new diff is generated. 
- redirects to the latest diff URI

3.  A diff between a Category C document and a Category A or Category C document can be requested in one of two ways:
a) creates a diff that is cached and is accessible by the same URI that was used to make the request.  There is no automatic regeneration, instead a manual regeneration must be requested, through one of two mechanisms
- by a "regenerate" button
- by an API call to DT
b) creates a diff that is cached and returns a new URI pointing to that cached diff.  This acts exactly as 2 above.

The computational cost is actually in two parts - the generation of a diff and the checking to see if a cached diff needs regeneration - and I imagine that the impact of bots with the processes above is going to be more about forcing checks for regeneration than generating new diffs.  That could mean that DT login is only be used to protect process 2 and process 3b. (Notwithstanding account Ekr’s point that we need data first.)

Jay

> On 30/08/2021, at 11:17 AM, Martin Thomson <[email-masked]> wrote:
> 
> 
> 

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Eric Rescorla <[email-masked]>
Date: Sun, 29 Aug 2021 17:01:03 -0700

On Sun, Aug 29, 2021 at 4:54 PM IETF Executive Director <
[email-masked]> wrote:

> 1.  Requesting a diff between two Category A documents generates a diff
> that is cached and is accessible by the same URI that was used to make the
> request.  Accessing the URI never results in re-generation unless the cache
> has expired.
>
> 2.  Requesting a diff between a Category B document and any other document
> creates a diff that is cached and returns a new URI pointing to that cached
> diff.  When the new URI is accessed then the cached diff is returned unless
> it has expired in which case it is regenerated first.  When the original
> URI is accessed then that:
> 3.  A diff between a Category C document and a Category A or Category C
> document can be requested in one of two ways:
> a) creates a diff that is cached and is accessible by the same URI that
> was used to make the request.  There is no automatic regeneration, instead
> a manual regeneration must be requested, through one of two mechanisms
> b) creates a diff that is cached and returns a new URI pointing to that
> cached diff.  This acts exactly as 2 above.
>
> The computational cost is actually in two parts - the generation of a diff
> and the checking to see if a cached diff needs regeneration - and I imagine
> that the impact of bots with the processes above is going to be more about
> forcing checks for regeneration than generating new diffs.  That could mean
> that DT login is only be used to protect process 2 and process 3b.
> (Notwithstanding account Ekr’s point that we need data first.)
>

Unless things are very inefficient, forcing the check for regeneration is
not going to be that expensive once you have already gone to the trouble of
processing the request at all.

-Ekr

> >>> That suggests the need for someone to request a diff that is not
> cached and is regenerated each time it is requested.  Is that correct?
> >>
> >> I can’t parse that.
> >>
> > For the specific case you describe - assuming the load is unacceptable -
> the link could be gated behind a login.  Following that link is often done
> by a human using their browser.  Not all of those humans will have a login
> on datatracker, but many of those that care about that link do, and asking
> the others to make an account would not be an unreasonable imposition.
> >

---

From: Lars Eggert <[email-masked]>
Date: Mon, 30 Aug 2021 14:31:12 +0300

Hi,

On 2021-8-28, at 18:13, Eric Rescorla <[email-masked]> wrote:
> In assessing this question it would be helpful to have a sense of the CPU consumption of this service.
> 
> For instance, how many requests per second are there? What fraction of the machine does it take up?

+1

I assume the rfcdiff we're talking about is the one with code here: https://tools.ietf.org/tools/rfcdiff/code

That's a bash script around diff, awk and some other standard tools. What parts of it are expensive?

Thanks,
Lars

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Mon, 30 Aug 2021 11:50:11 +0000

> For the specific case you describe - assuming the load is unacceptable - the link
> could be gated behind a login.  Following that link is often done by a human
> using their browser.  Not all of those humans will have a login on datatracker,
> but many of those that care about that link do, and asking the others to make an
> account would not be an unreasonable imposition.

I think "asking others to make an account" *would* be an unreasonable imposition.
For several recent drafts that are of concern to people in my company, I've started pointing these people to the GH draft -- including the diffs to understand how it differs from what's in datatracker. These are people who do not (have not and will not) participate in IETF. I think asking these people to create a datatracker account so they can see diffs between a GH Editor's Copy and the most recent datatracker version is unreasonable. Diffs are of significant use to consumers of IETF drafts and RFCs, and not just IETF participants. And there are a whole lot of "IETF consumers" out there. Hopefully that will continue to be the case. 😊

Barbara

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 30 Aug 2021 13:55:38 +0200

> That's a bash script around diff, awk and some other standard tools. What parts of it are expensive?

The bash part certainly is.

Also, this uses tools URIs to obtain a version from a version-neutral URI, and that historically has been particularly inefficient (as it went through the file system with, like, 250000+ files in a directory).

The first step to fix this (if it still is a problem) would be to create an API for this translation that is blazingly fast because it uses a database index in a datatracker database.

Grüße, Carsten

---

From: Lars Eggert <[email-masked]>
Date: Mon, 30 Aug 2021 14:57:14 +0300

Hi,

> The first step to fix this (if it still is a problem) would be to create an API for this translation that is blazingly fast because it uses a database index in a datatracker database.

if we can retain the current (unauthenticated) functionality with a rewrite, that would certainly be my preference.

Thanks,
Lars