Robots Exclusion Protocol User Agent Purpose Extension
Internet-Draft draft-illyes-rep-purpose-00
Gary Illyes
Google LLC.
garyillyes@google.com
18 October 2024
robotstxt
The Robots Exclusion Protocol defined in [RFC9309] specifies the user-agent rule for targeting automatic clients either by prefix matching their self-defined product token or by a global rule * that matches all clients.
This document extends [RFC9309] by defining a new rule for targeting automatic clients based on the clients' purpose for accessing the service.
About This Document
The latest revision of this draft can be found at https://garyillyes.github.io/ietf-rep-purpose/draft-illyes-rep-purpose.html. Status information for this document may be found at https://datatracker.ietf.org/doc/draft-illyes-rep-purpose/.
Source for this draft and an issue tracker can be found at https://github.com/garyillyes/ietf-rep-purpose.
Introduction
(fill in)
Specification
We define user-agent-purpose as the new rule with a predefined set of values. The values are registered with IANA at ... Below is an Augmented Backus-Naur Form (ABNF) description, as described in [RFC5234].

purpose = *WS "user-agent-purpose" *WS ":" *WS purpose-token NL
purpose-token = "EXAMPLE-PURPOSE-1" /"EXAMPLE-PURPOSE-2" / "EXAMPLE-PURPOSE-3" ; but check IANA for full list
NL = %x0D / %x0A / %x0D.0A
WS = %x20 / %x09

user-agent-purpose
The user-agent-purpose rule is semantically equivalent to the user-agent rule defined in Section 2.2.1. of [RFC9309]. As the user-agent rule, user-agent-purpose acts as a starter of rule groups.
user-agent-purpose tokens
The user-agent-purpose token MUST be a substring of the identification string that the automatic client sends to the service. For example, in the case of HTTP [RFC9110], the purpose token MUST be a substring in the User-Agent header, along with the product token. Here's an example of a User-Agent HTTP request header with the purpose token by the product token:

User-Agent: Mozilla/5.0 (compatible; ExampleBot/0.1; ExamplePurpose; https://www.example.com/bot.html)

The purpose token MUST be one of the tokens registered with IANA. Unrecognized tokens MAY be discarded by parsers. Crawlers MUST use case-insensitive matching to find the group that matches the purpose token and obey the rules of the group. If there's a group that matches the product token of the automatic client, the client SHOULD obey that group. If no matching group exists, crawlers MUST obey the group with a user-agent line with the "*" value, if present. If there is more than one group matching the user-agent-purpose, the matching groups' rules MUST be combined into one group and parsed according to Section X.
Conventions and Definitions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.
Security Considerations
The security considerations are the same as in the parent [RFC9309].
IANA Considerations
The vocabulary used as purpose tokens are registered at IANA-URL.
Examples

# robots.txt with purpose
# FooBot and all bots that are crawling for EXAMPLE-PURPOSE-1 are disallowed.
User-Agent: FooBot
User-Agent-Purpose: EXAMPLE-PURPOSE-1
Disallow: /
# EXAMPLE-PURPOSE-2 crawlers are allowed.
User-Agent-Purpose: EXAMPLE-PURPOSE-2

References
Normative References
[RFC9110]

HTTP Semantics
R. Fielding, Ed.
M. Nottingham, Ed.
J. Reschke, Ed.
June 2022
STD 97
RFC 9110
DOI 10.17487/RFC9110
[RFC9309]

Robots Exclusion Protocol
M. Koster
G. Illyes
H. Zeller
L. Sassman
September 2022
RFC 9309
DOI 10.17487/RFC9309
[RFC2119]

Key words for use in RFCs to Indicate Requirement Levels
S. Bradner
March 1997
BCP 14
RFC 2119
DOI 10.17487/RFC2119
[RFC8174]

Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words
B. Leiba
May 2017
BCP 14
RFC 8174
DOI 10.17487/RFC8174
Informative References
[RFC5234]

Augmented BNF for Syntax Specifications: ABNF
D. Crocker, Ed.
P. Overell
January 2008
STD 68
RFC 5234
DOI 10.17487/RFC5234
Acknowledgments
TODO acknowledge.