Subject: [scim] www-form-url-encoding for query parameters?

From: Matt Randall <[email-masked]>
Date: Fri, 9 Sep 2016 21:36:40 -0500

The SCIM specification mentions RFC7230 (which defines the https:// scheme)
and also mentions that URLs (URIs) must be percent-encoded per RFC3986.
Notably, SCIM queries make use of "query parameters", but does not include
any normative references that provide a first-class definition.  Some
implementations have made the implicit assumption that the presence of
"query parameters" means that the entire query component is structured per
x-www-form-urlencoded, as no definition is provided in any of the normative
specifications for how query parameters are to be lexed.  It's difficult to
know whether that was the intent for SCIM, however; for example, OData (in
which SCIM queries seem to share much in common with) define their own
processing rules for URIs query components [1], which notably does not use
x-www-form-urlencoded conventions.

This has ramifications specifically with regard to handling the "+"
character in queries, and whether the "+" character is considered lexically
equivalent to %20 when parsing the query component.

Should query parameters be treated as x-www-form-urlencoded, or should they
be treated as a more generic interpretation of a "query parameter" (similar
to OData) where only the "=" and "&" are treated as reserved separators,
and the URI is percent-encoded purely based on RFC3986?

[1]:
http://docs.oasis-open.org/odata/odata/v4.0/errata03/os/complete/part2-url-conventions/odata-v4.0-errata03-os-part2-url-conventions-complete.html#_Toc453752335

Thank you for any feedback,

Matt Randall

---

From: Phil Hunt <[email-masked]>
Date: Fri, 9 Sep 2016 21:01:06 -0700

Matt,

Are you reading RFC7644?

See section 1.2 regarding URL encoding.  The examples are not URL encoded for readability reasons as stated in the Notational Conventions.

URL parameters are mostly used in the GET requests (Section 3.4) for things like filters, paging, sorting.  For parameters that can be used in all operations, see section 3.9 and 3.10 on attribute notation.

SCIM’s similarity with ODATA probably arises because it is about RESTful identity data. There was no intended similarity. Though SCIM is being used for Identity Services (a directory) its primary intent in the specification is provisioning. This changes some of the design emphasis.

One other major difference in that the SCIM does not permit nested objects or references (e.g. /Users/<id>/manager).  So where ODATA can have many many ways to reference the same object, SCIM has a relatively flat hierarchy consisting of containers for ResourceTypes and has only one way to reference a resource.  The chief reason is to avoid referential integrity issues when resources reference each other (e.g. such as with groups). 

Hope this helps.

Phil

@independentid
www.independentid.com <http://www.independentid.com/>[email-masked] <mailto:[email-masked]>

> On Sep 9, 2016, at 7:36 PM, Matt Randall <[email-masked]> wrote:
> 
> The SCIM specification mentions RFC7230 (which defines the https:// scheme) and also mentions that URLs (URIs) must be percent-encoded per RFC3986.  Notably, SCIM queries make use of "query parameters", but does not include any normative references that provide a first-class definition.  Some implementations have made the implicit assumption that the presence of "query parameters" means that the entire query component is structured per x-www-form-urlencoded, as no definition is provided in any of the normative specifications for how query parameters are to be lexed.  It's difficult to know whether that was the intent for SCIM, however; for example, OData (in which SCIM queries seem to share much in common with) define their own processing rules for URIs query components [1], which notably does not use x-www-form-urlencoded conventions.
> 
> This has ramifications specifically with regard to handling the "+" character in queries, and whether the "+" character is considered lexically equivalent to %20 when parsing the query component.  
> 
> Should query parameters be treated as x-www-form-urlencoded, or should they be treated as a more generic interpretation of a "query parameter" (similar to OData) where only the "=" and "&" are treated as reserved separators, and the URI is percent-encoded purely based on RFC3986?
> 
> [1]: http://docs.oasis-open.org/odata/odata/v4.0/errata03/os/complete/part2-url-conventions/odata-v4.0-errata03-os-part2-url-conventions-complete.html#_Toc453752335 <http://docs.oasis-open.org/odata/odata/v4.0/errata03/os/complete/part2-url-conventions/odata-v4.0-errata03-os-part2-url-conventions-complete.html#_Toc453752335>
> 
> _______________________________________________
> scim mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/scim

---

From: Phil Hunt <[email-masked]>
Date: Mon, 12 Sep 2016 10:04:23 -0700

No,  x-www-form-urlencoded is not actually required to be supported (as currently specified). They aren’t quite the same. The main difference is encoding of the space character as “%20” or “+”.

SCIM only references Sec 2.1 of RFC3986 which indicates non-permitted characters are to be percent encoded (e.g. space as %20). 

AFAIK supporting “+” is common practice and most implementations probably support *both* %20 and “+”. However officially support for the “+” characters is not actually required.   IMV, this is ok as the spec is silent on x-www-form-urlencoded.

I’ll let others on the list comment as to their support or lack of support for “+” in URL encoding.

Phil

@independentid

> On Sep 12, 2016, at 7:20 AM, Matt Randall <[email-masked]> wrote:
> 
> Thank you for the reply, Phil.
> 
> In response to your question - yes, RFC7644 is the RFC in question.  Can you confirm "URL encoding" with respect to the query component of SCIM URLs specifically means "x-www-form-urlencoded query strings?  The specification mentions "query parameters", but there's not any RFC that I am aware of that defines what a query parameter is (the w3 defines it in HTML forms [1], but that's about it.)
> 
> [1]: https://www.w3.org/TR/html401/interact/forms.html <https://www.w3.org/TR/html401/interact/forms.html>
> 
> On Fri, Sep 9, 2016 at 11:01 PM, Phil Hunt <[email-masked] <mailto:[email-masked]>> wrote:
> Matt,
> 
> Are you reading RFC7644?
> 
>> On Sep 9, 2016, at 7:36 PM, Matt Randall <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> [email-masked] <mailto:[email-masked]>
>> https://www.ietf.org/mailman/listinfo/scim <https://www.ietf.org/mailman/listinfo/scim>
> 
>

---

From: Matt Randall <[email-masked]>
Date: Mon, 12 Sep 2016 15:47:50 -0500

Just for those following along, here's an example of where it is necessary
to know if the incoming request was encoded using x-www-form-urlencoded
rules, or using the basic URI rules allowing for the reserved characters
"&" and "=".  An x-www-form-urlencoded implementation could send:

?filter=meta.lastModified+le+"2011-05-13T04:42:34%2B05:00"

Using x-www-form-urlencoded rules, this value of "filter" would be
interpreted as:

meta.lastModified le "2011-05-13T04:42:34+05:00"

Using only the rules provided in the SCIM spec, the value of "filter" would
be interpreted as:

meta.lastModified+le+"2011-05-13T04:42:34+05:00"

Alternatively, a client could decide not to use x-www-form-urlencoded rules
and might send:

?filter=meta.lastModified%20le%20"2011-05-13T04:42:34+05:00"

Using x-www-form-urlencoded rules, this value of "filter" would be
interpreted as:

meta.lastModified le "2011-05-13T04:42:34 05:00"

Using only the rules provided in the SCIM spec, the value of "filter" would
be interpreted as:

meta.lastModified le "2011-05-13T04:42:34+05:00"

The most straightforward way to prevent the ambiguity is to simply
declare "+" as a reserved character for SCIM query string
implementations, such that consumers are
required to encode plus signs as %2B, and advising SCIM providers to
treat "+" characters as spaces for backwards compatibility.

---

From: Shelley <[email-masked]>
Date: Fri, 23 Sep 2016 15:01:01 -0500

+1 to Matt's proposal:

> The most straightforward way to prevent the ambiguity is to simply declare "+" as a reserved character for SCIM query string implementations, such that
> consumers are required to encode plus signs as %2B, and advising SCIM providers to treat "+" characters as spaces for backwards compatibility.
>
>

On Mon, Sep 12, 2016 at 3:47 PM, Matt Randall <[email-masked]>
wrote:

> Using only the rules provided in the SCIM spec, the value of "filter"
> would be interpreted as:
>
> Alternatively, a client could decide not to use x-www-form-urlencoded
> rules and might send:
>
> Using only the rules provided in the SCIM spec, the value of "filter"
> would be interpreted as:
>
> The most straightforward way to prevent the ambiguity is to simply declare "+" as a reserved character for SCIM query string implementations, such that consumers are
> required to encode plus signs as %2B, and advising SCIM providers to treat "+" characters as spaces for backwards compatibility.
>
>
>