Subject: Re: [weirds] URI templates and WEIRDS

From: Mark Nottingham <[email-masked]>
Date: Thu, 13 Sep 2012 13:53:40 +1000

Hi,

This has been in my inbox for a while; sorry for the delay.

Chipping in from a HTTP / Web perspective --

There are lots of ways to convey links to clients, with different properties based upon the use cases at hand. I'm not sure of the use case here, so I'll speak in generalities. 

The most important things, however, are:
  1) You don't overload the semantics of HTTP (e.g., "when the FooApp gives a 404 status code, it means...")
  2) You don't impinge on the server's control of its URI space (e.g., the /query/foo URI will mean...")
  3) you don't confuse the implementation with the interface exposed (hopefully self-explanatory)

Doing any of these is very likely to violate the Web architecture, and bring down the wrath of various HTTP/Web/URI folk upon you late in your review process, leading to all kinds of mess.

Just from what I read below, it seems like you're at risk of #2 and possibly #3. YMMV.

If you'd like document / idea review, I and I'm sure lots of other folks would be willing to help, as long as it isn't too time-consuming; i.e., you figure out your problem domain, we'll help with how you use the Web.

Overall, the advice that I (and many others) give people designing HTTP-based APIs is to start by defining the formats (identified by media types), and use links (identified by link relations) to tie them together.

Hope this helps,

On 06/08/2012, at 5:56 PM, Byron Ellacott <[email-masked]> wrote:

> Hi Murray-as-participant,
> 
> On 06/08/2012, at 12:40 PM, Murray S. Kucherawy wrote:
> 
>> Please say so if there are other points I've missed on either list.
> 
> d) It's twice the number of HTTP requests for a simple lookup.
> 
> e) Clients might just bypass it anyway; in the numbers space, whatever URI structure ARIN uses has a high likelihood of being hard coded into many clients so they can skip the never changing step of fetching a URI template to do exactly the same transform every time.  This might happen for .com's URI structure, particularly if there's a reference server that a number of common TLDs all operate.
> 
> f) They complicate redirection - if we use an HTTP redirection code, a client would fetch the URI templates list, create the URI for the resource they want, issue a query, and get back (after redirection) another URI templates list.  Or the redirecting server is forced to act as a proxy, introducing a whole new class of error conditions and apparent quality of service based on a third party operator.
> 
> I don't think I raised these at the mic, though.
> 
>> I can say that I implemented a first REPUTE system using URI templates
>> and the implementation wasn't terribly difficult.  The C library I
>> used is now open source, and numerous other implementations also exist
>> in a variety of languages.  Much of that code could easily be recycled
>> into weirds clients, removing much of the implementation complexity.
> 
> It's not hard, by any stretch of the imagination, but it is double the work for the client:
> 
> --- with URI templates, you start here ---
> 1. Fetch the URI template resource
> 2. Parse the URI template resource
> 3. Select the appropriate template, and parse that
> --- without URI templates, you start here ---
> 4. Fetch the data resource
> 5. Parse the data resource
> 6. Do something useful with the data resource
> 
> While the parsing of the template is presumably to be done by a library, this is still a lot of boilerplate to stitch container parsing, template selection, and library call together, given how trivial the rest of the client would be.
> 
> I spoke at the microphone about URI templates solving a problem we don't really have.  By that, I mean we don't have a mapping problem, or a (client-server) configuration problem, and we don't have such a complex discovery problem that we need this solution for it.  I find the arguments in favour of URI templates to be non-compelling:
> 
> 1) If names and numbers diverge, a client is going to need to know how to handle two different responses.  You cannot gloss over differences with a template for the query string.
> 
> 2) A server's architectural change is bound to the same number of moving parts in its URI space as before, to accommodate the standard's description of what variables might be found in a template.  What changes do you think could meaningfully be made to the noise around "example.com" that wouldn't just be deck chair rearrangements?  Isn't best current web software architecture practice to loosely couple URLs to service modules?
> 
> 3) All other behaviour once inside the protocol URI space is dictated by the standard.  I could say "servers own response data formats, not standards" but it would be equally meaningless.  The protocol owns the conversation between clients and servers, and both how to ask and how to respond are part of the protocol.
> 
> 4) There are no existing implementations of this as yet unspecified protocol.  Existing port 80 rest-like registration data directory services do not return results in a compliant format, and may not provide query entry points for the same aggregate roots; the URL structure is the least of the impacts on similar existing services.
> 
> I am not, I admit, a web architecture expert, but to my layperson's eyes the listed benefits to servers seem nebulous and more about technical purity than technical benefit, while the costs to clients are real, and relatively large.
> 
>  Byron
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: "John Levine" <[email-masked]>
Date: 14 Sep 2012 00:30:09 -0000

>  2) You don't impinge on the server's control of its URI space (e.g., the /query/foo URI will mean...")

Wait, stop right there.  We're building a design from scratch here,
with constrained syntax and semantics.  If someone wants to build a
WEIRDS server, he or she can implement the URLs and return values that
WEIRDS defines.  There is no installed base, there are no existing web
services that we have to work around.

What possible advantage is there for WEIRDS to encourage server
operators to invent different syntax and demand that clients kludge
around it with an extra level of lookup and templates?

R's,
John

---

From: Keith Gaughan <[email-masked]>
Date: Fri, 14 Sep 2012 10:24:24 +0100

> Chipping in from a HTTP / Web perspective --

<snip>

> Overall, the advice that I (and many others) give people designing HTTP-based
> APIs is to start by defining the formats (identified by media types), and use
> links (identified by link relations) to tie them together.

Thanks Mark. I tried to make these very same points back last September--almost
a year to this day, in fast--when I attempted to propose the use of URI
templates rather than hardcoded URL patterns in the spec, but was rebuffed. One
of the arguments against it, if I recall, was implementation difficultly.

-- 
Keith Gaughan, Development Lead
PGP/GPG key ID: 82AC3634
Blacknight Internet Solutions Ltd. <http://blacknight.com/>
12A Barrowside Business Park, Carlow, Ireland
Registered in Ireland, Company No.: 370845

---

From: Keith Gaughan <[email-masked]>
Date: Fri, 14 Sep 2012 10:42:02 +0100

On Fri, Sep 14, 2012 at 12:30:09AM -0000, John Levine wrote:

> Wait, stop right there.  We're building a design from scratch here, with
> constrained syntax and semantics.  If someone wants to build a WEIRDS server,
> he or she can implement the URLs and return values that WEIRDS defines.  There
> is no installed base, there are no existing web services that we have to work
> around.

Then how about dropping the 'RESTful' bit from the description of WEIRDS?

The argument *against* using URI templates are frankly weak. The difference
between using them and not using them is a difference of where the information
is given: is a spec, or in the server responses. That's the *only* difference.
Really, it is. The various counterarguments make *very* little sense within the
larger picture. For instance, the request multiplication issue is a non-issue
because... caching! And besides, if this efficiency issue was really all that
much of a big deal, then WEIRDS shouldn't be built on HTTP in the first place as
it's a ludicrously inefficient protocol. The implementation complexity issue is
a non issue because (a) there are plenty of URI template libraries out there and
(b) it's not really all that difficult to implement anyway. The argument that
people will ignore URI templates and just hardwire the URLs is maybe the weakest
of the lot: it could be made with just about any part of the spec, and if people
write broken clients, that's their problem, not ours.

This is the difference between WEIRDS being "of the web" rather than simply "on
the web".

> around it with an extra level of lookup and templates?

What other syntax? We're talking RFC 6570 here.

K.

---

From: Andy Newton <[email-masked]>
Date: Fri, 14 Sep 2012 13:00:43 +0000

>Then how about dropping the 'RESTful' bit from the description of WEIRDS?

What do you think "RESTful" means?

>
>For instance, the request multiplication issue is a non-issue
>because... caching!

A lot of HTTP libraries do not do caching. So that would have to be
another thing clients would need to implement.

>The argument that

Well, we know people hardcode Whois server IP addresses. We've turned down
nodes only to come back a year later to find that IP still getting port 43
queries. Happens with DNS servers too.

And people writing broken clients is a concern server operators have. Just
ask anybody dealing with those DNSSEC amplification issues.

>This is the difference between WEIRDS being "of the web" rather than

I'm not really sure what you mean by this. But it seems to me that if
somebody wants to be "of the web" they could just put a URI template file
on their server while still adhering to a non-URI templated spec. Does "of
the web" offer new functionality? Because that would be interesting.

But the idea that we need URI templates to avoid HTTP URL path collisions
(#2 in the list offered by Mark) is incorrect. So long as the base path is
not hardcoded, path collisions can be avoided.

I think the question to be asked of URI templates is this: what new
feature do we get?

-andy

---

From: Julian Reschke <[email-masked]>
Date: Fri, 14 Sep 2012 15:43:20 +0200

On 2012-09-14 15:00, Andy Newton wrote:
> ...

My take: "adhering to REST principles".

Sure, one can argue whether you need to adhere to *all* of them or not. 
But in this case I really have the impression that the label "restful" 
was used here just because it is en vogue.

Best regards, Julian

---

From: Andy Newton <[email-masked]>
Date: Fri, 14 Sep 2012 14:32:27 +0000

>was used here just because it is en vogue.

So you think what we are describing in the drafts given looks closer to a
SOAP web service or an HTTP API with service specific verbs than like REST?

I believe you first showed us this great tool: http://isitrestful.com  :)

Back to my question about URI templates: what new feature does it bring to
the table? Satisfying a purity argument about REST is not a protocol or
usage feature.

-andy

---

From: "Carlos M. martinez" <[email-masked]>
Date: Fri, 14 Sep 2012 11:58:19 -0300

Hello,

My opinion on the matter has not changed during the past year: URI
templates look like a nice solution to a problem this WG doesn't have.

The efficiency point is still valid and relevant: as Andy noted, most
WEIRDS clients won't be browsers but rather specially-crafted pieces of
software unlikely to implement any form of caching of HTTP objects,
which will be performing hundreds or thousands queries per hour in some
cases.

We (LACNIC) already have one customer of our prototype of this kind (a
CSIRT in Spain). Each RTT is ~300 msec, so the extra query is, IMO, too
large a tax.

As others also put it, if the whole argument revolves around concerns
about the 'correct' (by some definition**) usage of the term RESTful,
I'd rather drop it.

cheers!

~Carlos

**: I'm 'holding in my hand' (metaphorically, as it is a PDF file) a
copy of 'RESTful WebServices Cookbook' from O'Reilly, 2010 edition, and
it doesn't even come close to saying that URI templates are mandatory
for an interface to be qualified as 'RESTful', it even has a recipe
(5.7) named 'When and How to use URI Templates' which is less than 2
pages long over a 314-page book.

Recipe 4.3 even says 'Neither the architectural constraints of REST nor
HTTP require that clients treat URIs as opaque. Doing so reduces
coupling...'.

So again, I'm not convinced we need URI templates in WEIRDS.

---

From: "John Levine" <[email-masked]>
Date: 14 Sep 2012 15:22:18 -0000

>templates look like a nice solution to a problem this WG doesn't have. ...

+1, and to the performance stuff I snipped

>about the 'correct' (by some definition**) usage of the term RESTful,

I would cheerfully change all of the mentions of "RESTful" to "fluffy
bunny" in the charter if it would make this argument go away.

R's,
John

---

From: Francisco Obispo <[email-masked]>
Date: Fri, 14 Sep 2012 09:16:01 -0700

I'm not sure about this.

ICANN wants to have a reference implementation, that includes a Port 43 whois to RDAP proxy, and it would be ideal for it to have some sort of caching in front of it, however, if it can't make distinction between objects it will be limited to caching query/response pairs, which is very inefficient (better than no caching but still inefficient).

I don't like making assumptions about third party software, but it would be nice if we could provide the ability to provide various cacheable objects in the response.

One more reason for having an enclosure format + object definitions in namespace like structures ..

regards

> which will be performing hundreds or thousands queries per hour in some

Francisco Obispo 
email: [email-masked]
Phone: +1 650 423 1374 || INOC-DBA *3557* NOC
PGP KeyID = B38DB1BE

---

From: Andy Newton <[email-masked]>
Date: Fri, 14 Sep 2012 16:39:06 +0000

Having written such a beast, I can tell you that the performance
improvement comes from caching the rendered result and preventing the
requery. Caching the query/response pairs is not as inefficient as you
make it out to be. In fact, MySQL does that to great effect at the
database level. Also, that's exactly what an HTTP proxy or cache would be
doing.

And it is not an enclosure format that enables fine grained cache
semantics, it is an object reference (see 'handle').

-andy

On 9/14/12 12:16 PM, "Francisco Obispo" <[email-masked]> wrote:

>I'm not sure about this.
>
>ICANN wants to have a reference implementation, that includes a Port 43
>whois to RDAP proxy, and it would be ideal for it to have some sort of
>caching in front of it, however, if it can't make distinction between
>objects it will be limited to caching query/response pairs, which is very
>inefficient (better than no caching but still inefficient).
>
>I don't like making assumptions about third party software, but it would
>be nice if we could provide the ability to provide various cacheable
>objects in the response.
>
>_______________________________________________
>weirds mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/weirds

---

From: Francisco Obispo <[email-masked]>
Date: Fri, 14 Sep 2012 11:35:49 -0700

I would say that it depends on the architecture.

I also have an implementation that uses Memcached in the P43 whois proxy, and caches individual objects in memory. It is capable of only fetching in the backend cache-missed objects, thus reducing the amount of queries to the database and improving frontend performance.

We can easily scale the system by adding more front end whois servers, which are cheaper than high end database servers..

But that's an implementation issue..., and there is more than one way to do it.

If we have the need to map another use case, I'm worried that the current model will not make that very easy. In the search for simplicity, we will loose functionality (reusability). It's a balance that the working group will have to leverage.

Francisco

Francisco Obispo 
email: [email-masked]

---

From: "John Levine" <[email-masked]>
Date: 14 Sep 2012 19:39:50 -0000

>If we have the need to map another use case, I'm worried that the current model will not make that
>very easy. In the search for simplicity, we will loose functionality (reusability). It's a balance
>that the working group will have to leverage.

Seems to me that we've had domain and IP WHOIS for several decades,
and the use cases are the same now as they were in the 1990s.  I'm
much more concerned about building something that is stable and
scalable, and that is well enough specified that if there are 2000 or
more registry WEIRDS servers and a thousand registrar servers, my
client software can query them all the same way.

R's,
John

---

From: Francisco Obispo <[email-masked]>
Date: Fri, 14 Sep 2012 12:45:53 -0700

On Sep 14, 2012, at 12:39 PM, "John Levine" <[email-masked]> wrote:

> and the use cases are the same now as they were in the 1990s.  

Yes, but how many other areas are now using the internet as the core of their business (compared to the 1990s)?? I've already seen discussions about registries that can serve more than just domain names, and IP numbers, talking about Trademarks, etc.. 

> I'm

I don't have a problem with querying them all the same way, I have a problem with returning the data in a _consistent_ way and not mixing up data from one object type with another.. Consider the different models and policies that exist today with ccTLDs, and you will notice that one size does not fit all.

Francisco Obispo 
email: [email-masked]

---

From: "John R Levine" <[email-masked]>
Date: 14 Sep 2012 16:17:11 -0400

> Yes, but how many other areas are now using the internet as the core of 
> their business (compared to the 1990s)?? I've already seen discussions 
> about registries that can serve more than just domain names, and IP 
> numbers, talking about Trademarks, etc..

If someone wants to set up a trademark registry, I suppose that's OK 
(actually, it's a dreadful idea, but for now we'll stick to the technical 
issues.)  But the questions are going to be different from the ones about 
IPs or domain names, the answers will be different, and the back end 
database will be different.  All that will really be the same is the http 
framework and the json software, both of which are generic.  It's not our 
job to solve their problems, and even if we thought that it were, we'd 
have little to offer beyond what's already available off the shelf.

> I don't have a problem with querying them all the same way, I have a 
> problem with returning the data in a _consistent_ way and not mixing up 
> data from one object type with another. Consider the different models 
> and policies that exist today with ccTLDs, and you will notice that one 
> size does not fit all.

No kidding.  We're going to have enough trouble solving the problems we do 
have, without adding hypothetical extenstions for problems we don't have. 
Having seen the talk about the field inventory in Vancouver, I think we 
can come up with a design that can describe them all, but it'll take some 
hard work and experiments.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: Francisco Obispo <[email-masked]>
Date: Fri, 14 Sep 2012 13:27:47 -0700

On Sep 14, 2012, at 1:17 PM, "John R Levine" <[email-masked]> wrote:
> If someone wants to set up a trademark registry, I suppose that's OK (actually, it's a dreadful idea, but for now we'll stick to the technical issues.)  But the questions are going to be different from the ones about IPs or domain names, the answers will be different, and the back end database will be different.  All that will really be the same is the http framework and the json software, both of which are generic.  It's not our job to solve their problems, and even if we thought that it were, we'd have little to offer beyond what's already available off the shelf.
> 

Which is why I believe the problem can be solved in two different parts, 1) A standard enclosure format that is capable of transporting _any_ type of object, and 2) the object definitions for both names and numbers.

>>> 
>> 
>> I don't have a problem with querying them all the same way, I have a problem with returning the data in a _consistent_ way and not mixing up data from one object type with another. Consider the different models and policies that exist today with ccTLDs, and you will notice that one size does not fit all.
> 
> No kidding.  We're going to have enough trouble solving the problems we do have, without adding hypothetical extenstions for problems we don't have. Having seen the talk about the field inventory in Vancouver, I think we can come up with a design that can describe them all, but it'll take some hard work and experiments.
> 

That's my point,

We _don't_ need to solve them all, rather provide some common object templates that can be used across _all_, and an proper extension mechanism that allows for the additional use cases.

If someone wants to build a trademark registry (to follow up on the example), they just need to define the new object types and that's it.. guess what, they can do that with EPP already, so why not make it explicit for RDAP ?

I want to make sure we don't lock ourselves in a protocol that will require a similar effort in 10-20 years, because it was not flexible enough to evolve.

Francisco Obispo 
email: [email-masked]

---

From: "John R Levine" <[email-masked]>
Date: 14 Sep 2012 18:00:36 -0400

> We _don't_ need to solve them all, rather provide some common object 
> templates that can be used across _all_, and an proper extension 
> mechanism that allows for the additional use cases.

That would be complete failure.  If a WEIRDS client can't query and parse 
the results mechanically, without having to add extra special extension 
cases for every registry and registrar, we'd be no better off than we are 
now.

> I want to make sure we don't lock ourselves in a protocol that will 
> require a similar effort in 10-20 years,

Since WHOIS hasn't changed materially in 20 years, I see no reason to add 
extra complication for purely hypothetical problems in the distant future, 
particularly if they mean that we won't solve the problems that we're here 
to solve.

R's,
John

---

From: Francisco Obispo <[email-masked]>
Date: Fri, 14 Sep 2012 15:41:25 -0700

On Sep 14, 2012, at 3:00 PM, John R Levine <[email-masked]> wrote:

> Since WHOIS hasn't changed materially in 20 years, I see no reason to add extra complication for purely hypothetical problems in the distant future, particularly if they mean that we won't solve the problems that we're here to solve.

I don't understand where/what the extra complication is, it is merely a design choice. If we are not going to pay attention to what you call hypothetical problems, then we don't need a discussion, merely a braindump of what people are doing now and seek approval for it, ignoring use cases is not going to make the problems go away.

Francisco Obispo 
email: [email-masked]

---

From: "John Levine" <[email-masked]>
Date: 15 Sep 2012 00:05:07 -0000

>> Since WHOIS hasn't changed materially in 20 years, I see no reason to add extra complication for
>purely hypothetical problems in the distant future, particularly if they mean that we won't solve
>the problems that we're here to solve.
>
>I don't understand where/what the extra complication is, it is merely a design choice.

The templates are the extra complication.  More network traffic, more
client code, no useful added function for this application.

> If we are not going to pay attention to what you call hypothetical
> problems, then we don't need a discussion, merely a braindump of what
> people are doing now and seek approval for it, ignoring use cases is
> not going to make the problems go away.

We definitely need to deal with real use cases, notably making sure
that clients can decode and use all the results from domain WEIRDS
servers.  But we do not need to deal with hypothetical cases about
what someone might want to do for a different application in 2022.

This is getting repetitive.  I'm done.

R's,
John

---

From: "John R Levine" <[email-masked]>
Date: 14 Sep 2012 20:25:35 -0400

> HTTP URLs are owned and managed by the server; having standards usurp 
> that name space would fragment the Web.

This seems to be some other argument that the one we were having.

HTTP servers can implement any URLs they want, but if they also purport to 
be WEIRDS servers, it would be a good idea for some of the URLs they 
implement to be the ones that WEIRDS defines.

If you want to look at documents, see the recent archives of this list.

Regards,

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 15 Sep 2012 10:14:01 +1000

> Wait, stop right there.  We're building a design from scratch here,

Not if you're using HTTP.

> services that we have to work around.

You're not creating a new protocol; you're using one with defined semantics. 

HTTP URLs are owned and managed by the server; having standards usurp that name space would fragment the Web.

> around it with an extra level of lookup and templates?

I didn't say that you have to use URI templates; I'm only pointing out the sources of potential objections you'll receive once you request wider review. There are plenty of ways to address different problems within the constraints of the Web.

If you have a document that's suitable for review, I'd be glad to give you more specific feedback.

Regards,

HTTPbis WG Chair
IETF/W3C Liaison

---

From: SM <[email-masked]>
Date: Fri, 14 Sep 2012 23:45:28 -0700

>My take: "adhering to REST principles".

There are usually proposals to re-purpose a well-known 
architecture.  The significance of whether that's a problem or not 
only become apparent after several years or more.  My understanding 
is that if you are going to do it on port 80, you adhere to the principles.

>Sure, one can argue whether you need to adhere to *all* of them or 
>not. But in this case I really have the impression that the label 
>"restful" was used here just because it is en vogue.

:-)

Regards,
-sm

---

From: Chris Wright <[email-masked]>
Date: Sun, 16 Sep 2012 12:09:52 +1000

> My take: "adhering to REST principles".

I think I bought up in the Vancouver meeting that what we are proposing to
do here is not 'RESTful'.

Frankly though I am ok with it, I think we need to do WEIRDS the way that
we are doing it, so I agree with others that have said lets just pull the
references to REST from the docs and move on. We use a term for this
internally, when we are making an RPC style protocol over HTTP that some
might confuse for REST, but it isn't really - we call it a HTTPful
protocol, and I think that¹s probably all that we need for WEIRDS, a well
defined HTTPful protocol (that still needs to consider extensibility).

For those further interested...

If we wanted to I think we could make WEIRDS truly RESTful, its fits the
paradigm well (if done properly), and certainly would be a more flexible
WEIRDS to then be used for other 'objects'. I am just not sure its worth
the effort - maybe it is, and maybe someone who thinks it is could
convince me.

Using URI templates, at least in my understanding, will make the protocol
not RESTful, real REST is supposed to only require an entry point and then
you 'discover' the rest from there. (I suppose if you use the URI
templates to aid in that discoverability that would meet the model, but I
think the discoverability is really supposed to be around following links).

When people get into debates about what is and isn't REST I usually point
them to the following articles, one of them written by Roy Fielding, the
guy who coined the term - the 3rd article down does a good job of making
things clear. 

http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven

http://martinfowler.com/articles/richardsonMaturityModel.html

http://curtis.schlak.com/2012/01/23/hateoas-a-follow-up-to-rest-for-r33lz.h
tml

http://curtis.schlak.com/2012/01/19/fieldings-rest.html

Thanks

c.

---

From: Francisco Obispo <[email-masked]>
Date: Sun, 16 Sep 2012 11:57:34 -0700

> client code, no useful added function for this application.

So the solution to not using templates is to have a chaotic list of attributes smushed together in one single document, and implementors are supposed to figure out how each must be formatted?

Apologies, but I don't think this is a "simple" approach, it seems to me that this is the "shortcut" instead.

> what someone might want to do for a different application in 2022.

So you don't think that this protocol can't be used for any other thing except names and numbers?

Why did we implemented EPP then? we could've just written a plan exchange tab separated values with the fields required to register domain names and that should've been enough to solve the problem.

I know this sounds a bit rude, but I think we are not being coherent by having a provisioning protocol that can be extended, that supports very well any type of object, who is transport agnostic, and then a protocol to query the data that is limited to names and numbers and with a chaotic format.

Francisco Obispo 
Director of Applications and Services - ISC
email: [email-masked]
Phone: +1 650 423 1374 || INOC-DBA *3557* NOC
PGP KeyID = B38DB1BE

---

From: "Carlos M. martinez" <[email-masked]>
Date: Sun, 16 Sep 2012 16:32:45 -0300

Francisco,

>From the work carried by the WG so far I don't see that we have a
problem with the URIs. Most of the hard discussion has been around
responses and not requests, so, really, I still fail to see what benefit
URI templates will bring to the table, and I see many drawbacks.

Redefining WHOIS has failed before, and IMO it failed due to added
unnecessary complexity. I'm sure the added complexity seemed a good idea
at the time of writing the documents, but then harsh reality set in and
the community made a choice for us.

Personally, I believe that the watermark of complexity this particular
user community is willing to accept is that of clients that can be
implemented in bash+wget or in python+urllib2 in less than 20 lines.
They will implement no caching and they will probably hardcode URIs
anyway. After 15-20 years of RIRs, ARIN's WHOIS is still the entry point
for many queries due to a hardcoded string.

They decided that before implementing BEEP they rather stay on a
protocol that looks it was designed when the sabre-toothed tiger roamed
the Earth.

I believe we need to avoid a repetition of that scenario at all costs,
even if it means creating something that is not 'that perfect' (again,
by some definition of 'perfect')

cheers!

~Carlos

---

From: "John R Levine" <[email-masked]>
Date: 16 Sep 2012 15:41:44 -0400

[ snip ]

> So you don't think that this protocol can't be used for any other thing except names and numbers?

Actually, the goal is to design one protocol to ask for information about 
IP addresses, and a probably somewhat different protocol to ask about 
domain names.  If you don't believe me, read the charter.

> I know this sounds a bit rude, but I think we are not being coherent by 
> having a provisioning protocol that can be extended, that supports very 
> well any type of object, who is transport agnostic, and then a protocol 
> to query the data that is limited to names and numbers and with a 
> chaotic format.

That would be a dandy plan for a research group.  But not for a WHOIS 
replacement, unless you want to drive the work into the weeds just like 
the last N attempts to replace WHOIS.

R's,
John

---

From: Francisco Obispo <[email-masked]>
Date: Sun, 16 Sep 2012 14:09:35 -0700

Sent from my iPhone

> URI templates will bring to the table, and I see many drawbacks.

* extensibility
* reusability
* coherence
* simplicity
...

> the community made a choice for us.

Agree but  an enclosure format+object definitions does not bring this to the table.

> for many queries due to a hardcoded string.

I can implement it in less than 20 lines of code, but even more important than that; you re going to need to use a JSON library to parse the results anyway, and having a way to obtain the result objects would be no different Except that you would know how to treat the data and will reduce guess work.. In example, if my app only needs  ip information it doesn't need to understand or deal with domain or contact.. In the current model everything is mixed up in the same data structure.

> by some definition of 'perfect')

What does all cost mean in this context? Designing a protocol that is not going to meet registry needs in the future? Or am I bringing hypothetical cases again? At the minimum the protocol should be able to represent all registration data, not just names and mumbers, and if the charter is excluding everything else, then it's wrong and must be corrected

Regards

---

From: "Smith, Bill" <[email-masked]>
Date: Mon, 17 Sep 2012 01:11:57 +0000

On Sep 16, 2012, at 12:41 PM, John R Levine <[email-masked]> wrote:
> 
> Actually, the goal is to design one protocol to ask for information about IP addresses, and a probably somewhat different protocol to ask about domain names.  If you don't believe me, read the charter.

Sorry to bring up an old subject, but from the charter "but with the explicit additional goals of producing a simple, easy-to-implement protocol". Note the use of the singular.

Of course the charter concludes with "Should the Working Group reach a point where it determines that the problem of producing a grand unified specification for both numbers and names appears to be intractable, it will be permitted to divide the problem into separate tasks and amend its milestones accordingly."

Perhaps I missed the decision on intractability. If so, apologies to all.

---

From: "Smith, Bill" <[email-masked]>
Date: Mon, 17 Sep 2012 01:16:10 +0000

> bunny" in the charter if it would make this argument go away.

I again point to the charter that states "The framework shall be for data to be delivered via a RESTful data service using HTTP (optionally using TLS)".

IMO, that's a strong statement and says this group *will* produce a RESTful service.

Of course we could open up a charter discussion, again, but would that be productive?

---

From: "John Levine" <[email-masked]>
Date: 17 Sep 2012 01:18:42 -0000

>Sorry to bring up an old subject, but from the charter "but with the explicit additional
>goals of producing a simple, easy-to-implement protocol". Note the use of the singular.

Sure, but since both the questions and answers for numbers and names
are different, even under the most optimistic assumptions there won't
be much to share beyond generic json libraries, http redirect codes, and
maybe authorization schemes.

R's,
John

---

From: Byron Ellacott <[email-masked]>
Date: Mon, 17 Sep 2012 11:51:06 +1000

Hi John,

> maybe authorization schemes.

Does this mean you think http://tools.ietf.org/html/draft-newton-weirds-unified-json-response is a dead end?

  Byron

---

From: Byron Ellacott <[email-masked]>
Date: Mon, 17 Sep 2012 11:51:04 +1000

Hi Francisco,

>> URI templates will bring to the table, and I see many drawbacks.

If I understand the basis of your thinking here, it's that the response should look notionally like this:

{
	"object-1": { "some": "data", "reference": [ "object-2", "object-3" ] },
	"object-2": { "some": "data", "reference": [ "object-4" ] },
	"object-3": { "more": "data" },
	"object-4": { "this": "data"}
}

whereas now, it looks notionally like this:

{
	"object-1": {
		"some": "data",
		"object-2": {
			"some": "data",
			"object-4": { "this": "data" }
		},
		"object-3": { "more": "data" }
	}
}

With your justification being that the former can contain more kinds of objects, down the line, more easily than the latter, and that the former is more susceptible to the caching of parts of a given response than the latter?

Does this approach also depend on URI templates, or a similar mechanism, where the first query to any given server looks kind of like:

{
	"object-type-1": "a-uri-template-for-that-object",
	…
}

Do you see that mechanism as primarily required to support the names+numbers work, or to support extensibility for querying entirely separate object types, other than the { ip, domain, entity, autnum, nameserver } set of objects so far identified?

Essentially, I'd like to understand if this is a discussion about the scope of the working group's output, or about the best way to approach the more constrained problem of providing an alternative mechanism for querying WHOIS data, that supports i18n, redirection, consistency, and authen/authz, or something in between.

  Byron

---

From: Andy Newton <[email-masked]>
Date: Mon, 17 Sep 2012 02:23:30 +0000

On 9/16/12 9:16 PM, "Smith, Bill" <[email-masked]> wrote:

>I again point to the charter that states "The framework shall be for data
>to be delivered via a RESTful data service using HTTP (optionally using
>TLS)".
>

It's good that we are having these conversations, otherwise real work
might get accomplished. :)

Though English is a horribly imprecise language, this isn't that hard. The
meaning of the suffix "ful" is "to be characterized by". If somebody is
forgetful, that means they forget a lot of things but it does not mean
they forget everything. If a tool is useful, that means it can be used for
many purposes but does not mean it can be used for every task. If we had
wanted to create a REST service, the charter would say "REST" and not
"RESTful".

The purpose of using the term "RESTful web service" is to differentiate
from other types of web services, notably the SOAP based services. It's a
"useful" distinction in that it gives the reader of this interoperable
specification a starting point in understanding the protocol description.
And after all, an interoperable specification is what we are to produce.
If the greater universe of Internet programmers recognizes what is being
described here as a "RESTful web service", then the term has done its job
regardless of how uncomfortable that might make some protocol purist wonk.

All that being said, if there is some feature or utility that can be
acquired by more closely adhering to a pure REST architecture, I'm willing
to discuss it. So far nothing has been offered.

-andy

---

From: "John R Levine" <[email-masked]>
Date: 16 Sep 2012 22:24:03 -0400

> Does this mean you think http://tools.ietf.org/html/draft-newton-weirds-unified-json-response is a dead end?

With luck it'll let the two applications share some library code, but that 
doesn't that the fact that the questions and answrs are basically 
different.

R's,
John

---

From: Chris Wright <[email-masked]>
Date: Mon, 17 Sep 2012 14:29:51 +1000

Andy et al.

> Though English is a horribly imprecise language, this isn't that hard. The meaning of the suffix "ful" is "to be characterized by". If somebody is forgetful, > that means they forget a lot of things but it does not mean they forget everything. If a tool is useful, that means it can be used for many purposes but > does not mean it can be used for every task. If we had wanted to create a REST service, the charter would say "REST" and not "RESTful".

I like your definition/clarification about the differences between saying being REST compliant as opposed to being 'RESTful', Im not 100% sure I am on board with it in a broader sense, but for the purposes of this working group lets run with it, it makes sense.

> All that being said, if there is some feature or utility that can be acquired by more closely adhering to a pure REST architecture, I'm willing to discuss it. So far nothing has been offered.

A purely REST service for WEIRDS might look something like this (Im using the domain service as an example, but similar could be done for AS numbers, IP Addresses and the like) - and note that this is just one possible service that would qualify as REST... (also note that my examples aren't exactly  syntactically correct)

Say I had a client that was trying to find the information about example.com - the flow in a REST world would go something like this... 

The entry point would be a well know URL let's call it http://weirds.iana.org which would return a document (probably encoded in JSON) that gave you a list of the TLDs in the root zone, and link to a resource that contains more information about the TLDs.. for eg it might return

{
	domain : { name: com, link: { relation: item, url: http://weirds.iana.org/domain?domain=com } }
	domain: { name: net, link:  { relation: item, url: http://weirds.iana.org/domain?domain=new } }
	etc...
}

*note that the relations that are represented by links are actually defined in an IANA registry here (we can add more to this with our protocol if we desire) http://www.iana.org/assignments/link-relations/link-relations.xml Most of this stuff, started from the ATOM protocol I believe, which is a well-known REST protocol - maybe we should ask the ATOM people for assistance.

Anyway moving on, then a client would follow the link to get more information about the com 'item' in the list which might return the following:

{
	domainName: com,
	registryManager: { name: verisign, link: { relation: registry url: http://weirds.versign.com/base?registry=versign } },
	registeredDomains: { link: { relation:collection url: http://weirds.verisign.com/domains }},
	lastUpdated: "1/3/2010 "
	etc
}

Then the client would query the http://weirds.versign.com/domains url (note that that url could have just as easily been ftp://weirds.verisign.com/domain.json.list ) and probably get a response something like the above say:

{
	domain : { name: google.com, link: { relation: item, url: http://weirds.verisign.com/domain?domain=google.com } }
	domain: { name: blah.com, link:  { relation: item, url: http://weirds.verisign.com/newDomain/getDomainInfo?domain=blah.com } }
	domain : { name: example.com, link: { relation: item, url: http://weirds.verisign.com/reservedDomains?domain=example.com } }
	etc...
}

Note the different URLs, this is actually not a problem and exactly what REST is all about, the URLs are 'learned' and then followed, each server can make up their own schemes and so forth.

So the advantages of this are many, we only need define encodings, and link relations, and how we are going to represent those link relations in our objects but thats about it... the actuall transport is irrelevant to the protocol etc (we could lock it down to just HTTP) the protocol specification is smaller, cleaner easier to understand and much more flexible, the disadvantage is (at least in this example of a REST model) it requires a central starting point. We could work around that by having a way in the WEIRDS protocol that you 'infer' a starting point based on what your trying to look up, (the boot strapping we had been talking about ealier), so essentially each registry would still provide a purely REST service, but the single entry point is per registry and looked up external to the REST part of the overall protocol. Another potential problem with this approach is that by very definition I need to be able to get a list of domains to find the URLs to the specific item - some ccTLDs would not like this. Also as the number of domains gets bigger (think verisign with the ~100 million .coms, the lists would be large and would need to be able to be filtered - so in the end, I don’t think that what we are trying to do with WERIDS actually fits nicely into  (this interpretation of) REST. Perhaps there is a way to address these concerns in a pure REST implementation but I have put about 10 mins of thought into the REST example in this email and they weren't immediately obvious.

So I think my vote is still something kinda like REST, but that is not actually REST, and that we stop calling it REST and be done with it, can we get everyone to agree to that? (Although if others felt strongly that they wanted to continue creating a pure REST service then I am fine with that to, but let's make a call one way or the other and then move on).

Thanks

c.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Andy Newton
Sent: Monday, 17 September 2012 12:24 PM
To: Smith, Bill; John Levine
Cc: <[email-masked]>
Subject: Re: [weirds] URI templates and WEIRDS

On 9/16/12 9:16 PM, "Smith, Bill" <[email-masked]> wrote:

>I again point to the charter that states "The framework shall be for 
>data to be delivered via a RESTful data service using HTTP (optionally 
>using TLS)".
>

It's good that we are having these conversations, otherwise real work might get accomplished. :)

Though English is a horribly imprecise language, this isn't that hard. The meaning of the suffix "ful" is "to be characterized by". If somebody is forgetful, that means they forget a lot of things but it does not mean they forget everything. If a tool is useful, that means it can be used for many purposes but does not mean it can be used for every task. If we had wanted to create a REST service, the charter would say "REST" and not "RESTful".

The purpose of using the term "RESTful web service" is to differentiate from other types of web services, notably the SOAP based services. It's a "useful" distinction in that it gives the reader of this interoperable specification a starting point in understanding the protocol description.
And after all, an interoperable specification is what we are to produce.
If the greater universe of Internet programmers recognizes what is being described here as a "RESTful web service", then the term has done its job regardless of how uncomfortable that might make some protocol purist wonk.

All that being said, if there is some feature or utility that can be acquired by more closely adhering to a pure REST architecture, I'm willing to discuss it. So far nothing has been offered.

-andy

---

From: Francisco Obispo <[email-masked]>
Date: Sun, 16 Sep 2012 22:49:19 -0700

here a more complete response/set of thoughts for those interested:

On Sep 16, 2012, at 9:29 PM, Chris Wright <[email-masked]> wrote:

> Also as the number of domains gets bigger (think verisign with the ~100 million .coms, the lists would be large and would need to be able to be filtered - so in the end, I don’t think that what we are trying to do with WERIDS actually fits nicely into  (this interpretation of) REST. Perhaps there is a way to address these concerns in a pure REST implementation but I have put about 10 mins of thought into the REST example in this email and they weren't immediately obvious.
> 

Most people favored JSON as the default transport for the response, I was originally suggesting XML, and one of the advantages, is that I can use an event-driven parser (SAX), and extract only the parts of the XML that I need from a stream (filehandle, network stream, etc.) without having to load all the document in memory.

Most JSON parsers that I've used load the whole document in memory, so you would have to have a lot of memory available to process a COM response. :-S

> So I think my vote is still something kinda like REST, but that is not actually REST, and that we stop calling it REST and be done with it, can we get everyone to agree to that? (Although if others…

I have to confess that I have not reviewed the query document as much as I've wanted to, but here are my thoughts on the whole process: 

We only need an anchor point for the TLD or IP delgation to do a successful discovery. But when I think of RESTful I think of having a table with:

Root: https://weirds.nic.TLD/

Method:				HEAD	GET	PUT	POST	DELETE
URL
/domain/example.com		1	1	0	0	0
/contact/sh8013			1	1	0	0	0
/host/ns1.example.com		1	1	0	0	0
/registrar/foo-registrar	1	1	0	0	0
/search/query-string		1	1	0	0	0	

In this case, since this would be a read-only protocol, we would not implement PUT POST or DELETE, except for creating/ending a session, but that's another issue.

When using the HEAD method, we would check for object existence, the GET method will return the data associated with it.

/search/ would be a special URI to perform whois-equivalent queries where all the supported namespaces would be searched for an exact match, or a partial match based on some rules/permissions.

The response document should contain the object description, and if provided an optional flag, return the information associated to contained objects.

For a domain query:

{
  "version":"1.0",
  "object":{
                "namespace":"domain-1.0",
                "name":"example.com:",
		"registrant":"sh8013",
                "contacts":{
                             "tec":["fo1234","abd4321"],
                             "adm":["foo412","bar543"]
                           },
                "hosts":["ns1.example.com","ns2.example.net"],
                "registrar":"foo-registrar",
                "updatedDate":"2012-04-01T12:00:12.0Z",
                "createdDate":"2010-01-03T14:10:14.0Z"
             },
  "uri":"https://weirds.nic.TLD/domain/example.com",
  "responseDate": "2012-09-16T21:54:12.0Z"
}

For a domain query with the "returnAll" flag (or whatever we want to call it):

{
  "version":"1.0",
  "object":{
                "namespace":"domain-1.0",
                "name":"example.com:",
		"registrant":"sh8013",
                "contacts":{
                             "tec":["fo1234"],
                             "adm":["foo412"]
                           },
                "hosts":["ns1.example.com","ns2.example.net"],
                "registrar":"foo-registrar",
                "updatedDate":"2012-04-01T12:00:12.0Z",
                "createdDate":"2010-01-03T14:10:14.0Z"
             },
  "additional":{
                "contacts":{
                             "sh8013":{
				       "namespace":"contact-1.0",
          			       "id":"sh8013",
                                       "name":"Foo Bar",
                                       "email":"[email-masked]",
                                       "addresses":[ 
						  { 
                                                   "address":["950 Charter St.","Redwood City"],
                                                   "state":"CA",
                                       		   "country:"US"
                                                  }
						],
                                       "createdDate":"2001-04-05T12:00:00.0Z",
                                       },
			     "fo1234":{
				       "namespace":"contact-1.0",
          			       "id":"fo1234",
                                       "name":"Foo Bar",
                                       "email":"[email-masked]",
                                       "addresses":[ 
						  { 
                                                   "address":["950 Charter St.","Redwood City"],
                                                   "state":"CA",
                                       		   "country:"US"
                                                  }
						],
                                       "createdDate":"2001-04-05T12:00:00.0Z",
                                       },
			      "foo412":{
				       "namespace":"contact-1.0",
          			       "id":"foo412",
                                       "name":"Foo Bar",
                                       "email":"[email-masked]",
                                       "addresses":[ 
						  { 
                                                   "address":["950 Charter St.","Redwood City"],
                                                   "state":"CA",
                                       		   "country:"US"
                                                  }
						],
                                       "createdDate":"2001-04-05T12:00:00.0Z",
                                       }
			},
                    "hosts":{
			    "ns1.example.com":{
				       "namespace":"host-1.0",
				       "name":"ns1.example.com",
				       "addresses":["2001:4f8:a:b:c::1","149.20.200.1"],
				       "createdDate":"2010-10-10T12:00:00.0Z",
			    },
                            "ns2.example.net":{
				       "namespace":"host-1.0",
				       "name":"ns2.example.net",
				       "addresses":["2001:4f8:a:b:c::1","149.20.200.1"],
				       "createdDate":"2010-10-10T12:00:00.0Z",
			    },
	            "registrars":{
    				  "namespace":"registrar-1.0",
                                  "name":"foo-registrar",
                                  "url":"http://foo-registrar.com"
			         }
		}

  "uri":"https://weirds.nic.TLD/domain/example.com",
  "responseDate": "2012-09-16T21:54:12.0Z"
}

This is my idea on how we should be representing the responses.. Notice the enclosure format, there is a version, a response section, an additional section, a response date, and a URI that was used to retrieve the query.

There are several advantages of this:

1) If someone stores the file, it can easily identify it;

2) Objects have a "namespace" associated with it, which allows clients to quickly identify the type of object thus be able to process it accordingly (knowing exactly which fields to fetch and how to present them).

3) If the "returnAll" flag is turned "on", (which should be definable by registry policy), the response will also contain an "additional" section with the additional "objects" identifying the query.

4) If I am a system performing thousands of queries, I could cache the individual object responses (with a lifetime), and prevent additional queries to the servers. (whois to RDAP)

5) There is coherence with EPP with the ability to support registry objects in general, so a group of people could work on the various objects for their own types of registries: names, and numbers working independently but agreeing on common object types: contacts, hosts

6) An additional "extension" section could be implemented either at the object level or at the response level to provide additional information not supported natively by the protocol, (AGP, TMCH, DNSSEC, IDN, etc.)

To demonstrate how easy it would be to process this.. If I was a mail administrator trying to contact the technical contact to report a problem, all I needed to do was:

1) query: https://weirds.nic.TLD/domain/example.com?returnAll=1
2) parse the response with a JSON parser and call (perl pseudo code):

	# obtain the first technical contact from the contacts list
	$tech_ref=$response_json->{object}->{contacts}->{tec}->[0];

	# use the contact from the additional section
	$tech_contact=$response_json->{additional}->{contacts}->{$tech_ref};

	&send_email(
		    to=>$tech_contact->{email},
                    name=>$tech_contact->{name}
		   );

If the returnAll option was disabled the main difference would be:

1) query: https://weirds.nic.TLD/domain/example.com?returnAll=0
2) parse the response with a JSON parser and call (perl pseudo code):

	# obtain the first technical contact from the contacts list
	$tech_ref=$response_json->{object}->{contacts}->{tec}->[0];

	# &fetch_contact() would perform a GET request to 
	# https://weirds.nic.TLD/contact/fo1234 and 

	$query2=&fetch_contact($tech_ref);

	# use the returned object as the tech contact
	$tech_contact=$query2->{object};

	# send email
	&send_email(
		    to=>$tech_contact->{email},
                    name=>$tech_contact->{name}
		   );

This should help clear some of the doubts about introducing additional complexity. The idea of separating the response into different sections is very similar, and it uses the same rationale as EPP, with this we now have a more robust protocol that can adapt to more object types and also be extensible for those cases where the standard set of responses don't fit the current schema.

Hopefully there will be some reconsideration going forward, I think I've explained myself enough to get some opinions.

Best regards,

Francisco Obispo

---

From: Byron Ellacott <[email-masked]>
Date: Mon, 17 Sep 2012 17:10:08 +1000

Hi Francisco,

> Most JSON parsers that I've used load the whole document in memory, so you would have to have a lot of memory available to process a COM response. :-S

I would hope that looking  up the entire list of registrations underneath .COM. is NOT a necessary step in looking up the registration data for EXAMPLE.COM. :-)

> For a domain query with the "returnAll" flag (or whatever we want to call it):
[full example cut for brevity]
> This is my idea on how we should be representing the responses.. Notice the enclosure format, there is a version, a response section, an additional section, a response date, and a URI that was used…

How is this objectively superior to:

{
	"version":"1.0",
	"object": {
		"registrant": {
			"addresses":[ { 
			} ],
			"tec": {
				"addresses": [ { 
				} ],
			"adm": {
				"addresses": [ { 
				} ],
               "hosts":[
				"name":"ns2.example.com",
               "registrar": {

> 1) If someone stores the file, it can easily identify it;

Still true;

> 2) Objects have a "namespace" associated with it, which allows clients to quickly identify the type of object thus be able to process it accordingly (knowing exactly which fields to fetch and how t…

Still true; the current unified response draft doesn't include the type of the top level response object, which I think is a weakness we can correct, but the type of subsequent objects can be easily inferred by its position in the response document.  I note your example assumes that something contained in $response_json->{additional}->{contacts} is, in fact, a contact, too. :-)

> 3) If the "returnAll" flag is turned "on", (which should be definable by registry policy), the response will also contain an "additional" section with the additional "objects" identifying the query.

Expand or do not expand the sub-objects; interesting potential discussion over a "returnAll" flag to have, I don't feel strongly either way, though it would be perhaps polite to make the reference a URI instead of an opaque string, if you're not going to include the actual data.

> 4) If I am a system performing thousands of queries, I could cache the individual object responses (with a lifetime), and prevent additional queries to the servers. (whois to RDAP)

Since your front-end is building the final answer each time, it makes no (apparent) difference whether the individual object responses are structurally placed at point A or point B in the output.

> 5) There is coherence with EPP with the ability to support registry objects in general, so a group of people could work on the various objects for their own types of registries: names, and numbers…

Still true;

> 6) An additional "extension" section could be implemented either at the object level or at the response level to provide additional information not supported natively by the protocol, (AGP, TMCH, D…

Still true; you just put the additional information right where you'd otherwise have needed to put a dangling link to the additional information anyway.  (And, just for the record, I would be disappointed with the WG outcome if DNSSEC and IDN data were not supported natively in the drafts we produce in this first chartered set of milestones. :-)

> To demonstrate how easy it would be to process this.. If I was a mail administrator trying to contact the technical contact to report a problem, all I needed to do was:

Agreed, your example is neither harder nor easier to process than my example.  If you're using a SAX parser for an XML document, you'd enter an element with a particular namespace and shoot off to the right handler for that element and its children, regardless of where it appeared in the document.

I think you've answered my original question, which was whether URI templates are a critical part of your preferred representation - I don't think they are, so we are having a discussion around the structure of the container.

I will freely admit that I prefer nested for no particular technical reason.  It "feels" like a flat model exposes an implementation specific abstraction of data, while a nested model "feels" like it talks about the registration information as a cohesive whole as seen from that perspective, but these are clearly not technical justifications, they are aesthetic.

Are there technical justifications either way that I'm missing?

  Byron

---

From: Andy Newton <[email-masked]>
Date: Mon, 17 Sep 2012 14:02:11 +0000

On 9/17/12 12:29 AM, "Chris Wright" <[email-masked]> wrote:

>So I think my vote is still something kinda like REST, but that is not
>actually REST, and that we stop calling it REST and be done with it, can
>we get everyone to agree to that? (Although if others felt strongly that
>they wanted to continue creating a pure REST service then I am fine with
>that to, but let's make a call one way or the other and then move on).

Chris,

Many thanks for the description. I agree, we don't want a pure REST
system. There are serious issue with it in our use cases, and we've even
discussed the gaps in our bootstrapping threads and just didn't know it.

And I agree we shouldn't call our work REST. But "RESTful" is a perfectly
adequate description for the reasons I've given.

And one last thing (not aimed at you Chris :) ): If we want to start
playing these architectural purity games then I suggest the REST purists
pay closer attention to BCP 56.

-andy

---

From: Andy Newton <[email-masked]>
Date: Mon, 17 Sep 2012 14:30:40 +0000

On 9/17/12 3:10 AM, "Byron Ellacott" <[email-masked]> wrote:

>> Most JSON parsers that I've used load the whole document in memory, so
>>you would have to have a lot of memory available to process a COM
>>response. :-S
>
>I would hope that looking  up the entire list of registrations underneath
>.COM. is NOT a necessary step in looking up the registration data for
>EXAMPLE.COM. :-)

Well, there are stream parsers available for JSON. The idea that you can
only do this with XML is bunk. And there is so much more wrong with that
pattern than the need for a stream/event parser.

>> 1) If someone stores the file, it can easily identify it;

Indeed. This argument keeps getting made as if Section 9 of Using-HTTP
didn't exist. 
(http://tools.ietf.org/html/draft-designteam-weirds-using-http-01#section-9
).

>
>> 2) Objects have a "namespace" associated with it, which allows clients
>>to quickly identify the type of object thus be able to process it
>>accordingly (knowing exactly which fields to fetch and how to present
>>them).
>
>Still true; the current unified response draft doesn't include the type
>of the top level response object, which I think is a weakness we can
>correct, but the type of subsequent objects can be easily inferred by its
>position in the response document.  I note your example assumes that
>something contained in $response_json->{additional}->{contacts} is, in
>fact, a contact, too. :-)

Adding an "objectClass" member to the top level object is superfluous.
Certainly if the top level object has the members "startAddress" and
"endAddress", software confusing it for a name server object is poorly
written. But I'm willing to add this if we can finally quit arguing this
topic.

>
>> 3) If the "returnAll" flag is turned "on", (which should be definable
>>by registry policy), the response will also contain an "additional"
>>section with the additional "objects" identifying the query.
>
>Expand or do not expand the sub-objects; interesting potential discussion
>over a "returnAll" flag to have, I don't feel strongly either way, though
>it would be perhaps polite to make the reference a URI instead of an
>opaque string, if you're not going to include the actual data.

It is unnecessary complexity. We do this in the ARIN RESTful Whois service
and people get confused by it and generally hate it. Once they figure it
out, the set returnAll to true for everything.

>
>> 5) There is coherence with EPP with the ability to support registry
>>objects in general, so a group of people could work on the various
>>objects for their own types of registries: names, and numbers working
>>independently but agreeing on common object types: contacts, hosts
>
>Still true;

Agreed, and to expand further on this point: The EPP model is actually
different than the one being described here. In the EPP model, every
difference of a contact, name server, domain, etcŠ is a new object class
(that is if LunarNIC wants to add custom data to the contact class, they
override completely the standard contact class). In the EPP world, making
sure the client and server have a perfect understanding is essential
because money is changing hands, it is a read/write protocol, there is a
contractual relationship between the actors, and set of actors is limited
to a smaller community. None of that is true for WEIRDS. Requiring a
generic WEIRDS client to know a specialized object class for any registry
that simply wants to add one bit of custom data is to high a bar for the
purposes we need.

>I will freely admit that I prefer nested for no particular technical
>reason.  It "feels" like a flat model exposes an implementation specific
>abstraction of data, while a nested model "feels" like it talks about the
>registration information as a cohesive whole as seen from that
>perspective, but these are clearly not technical justifications, they are
>aesthetic.
>

Actually, yes there is a very good technical reason. Not all registries
wish to model all data as first class objects. This is the cause of the
issue of name servers being attributes on domains vs being first class
objects. See section 4 of Unified Response
(http://tools.ietf.org/html/draft-designteam-weirds-using-http-01#section-9
). I also see that we will have this issue with entities.

-andy

---

From: Francisco Obispo <[email-masked]>
Date: Mon, 17 Sep 2012 08:52:07 -0700

Byron:

Thanks for taking the time to go over my concerns,

Here are some comments:

> I would hope that looking  up the entire list of registrations underneath .COM. is NOT a necessary step in looking up the registration data for EXAMPLE.COM. :-)

Yes that was what I meant, not the EXAMPLE.COM data ;-)

> How is this objectively superior to:

Well what worries me is being too repetitive, I'm afraid that if an object is referenced in more than one location, that we would be adding more data than it is actually needed.

> Still true; the current unified response draft doesn't include the type of the top level response object, which I think is a weakness we can correct, but the type of subsequent objects can be easil…

Well we can either assume it, or provide a full URI as you described it, I'm happy with any of the two, but I believe that the URI might not be needed because objects tend to live only in one registry, so it would be redundant to say "https://weirds.nic.TLD/contact/sh8013" than just "sh8013" since to fetch the data the client would have to go to the same URI anyway.

> Expand or do not expand the sub-objects; interesting potential discussion over a "returnAll" flag to have, I don't feel strongly either way, though it would be perhaps polite to make the reference…

I think this is useful, we could set a reasonable default, and let registries define it via policy.

> Since your front-end is building the final answer each time, it makes no (apparent) difference whether the individual object responses are structurally placed at point A or point B in the output.

That is if things are delimited as objects, if that's the case I'm ok with it, but we need some 'namespace' or 'type' attribute to differentiate.
> Still true; you just put the additional information right where you'd otherwise have needed to put a dangling link to the additional information anyway.  (And, just for the record, I would be disap…

I agree as well, but the way we support them can't be by just adding the attributes to existing object, I would like to see a mechanism to do so, preventing the original objects from being polluted with attributes that are not directly related to them, thus we need a framework to extend them.

> I think you've answered my original question, which was whether URI templates are a critical part of your preferred representation - I don't think they are, so we are having a discussion around the…

Good to know, so we should keep moving forward with the minor changes.

> Are there technical justifications either way that I'm missing?

I'm a bit paranoid by not being too redundant, and keeping the payload small as well, so, I do think there is an advantage of only providing references (whatever those may be) in the response, and then figuring out how to list those unique elements (with content or not) in another section of the response.

Thanks again

Francisco Obispo

---

From: Francisco Obispo <[email-masked]>
Date: Mon, 17 Sep 2012 09:04:18 -0700

Hi Andy,

Thanks for taking the time to go over my concerns again,

here are more comments:

> pattern than the need for a stream/event parser.

I have not use such parsers in JSON, I don't doubt they exist, but the ones
that I've used and are common to both Perl, Python and C++ are not event-based.

> (http://tools.ietf.org/html/draft-designteam-weirds-using-http-01#section-9

yes I've seen that, and perhaps it's the fact that the elements are intermixed with the object attributes what generates me the additional noise.

> written. But I'm willing to add this if we can finally quit arguing this

In sync with my previous comment, so please do!.

> out, the set returnAll to true for everything.

I wouldn't call it complexity, it's a feature, users don't have to know about it unless a registry supports it, you can provide some reasonable default for your setup, but let others decide on their experience.

> that simply wants to add one bit of custom data is to high a bar for the

Yes and no,

EPP is not _always_ about exchanging money, secondly, EPP is considered a
standard for Registry-Registrar interaction, so having a protocol to represent
registry data should at least make sure that it is capable of covering the
object types and extensions used by registries with EPP.

I'm not saying that the same information will be made available, that's registry
policy, but at the minimum they should be bundled.

Imaging building a car that can go 100 mph, but only using tires that cannot 
go over 40 mph..

We will start hitting serious limitations as more registries try to adopt it,
and will fall back to P43 whois where they can publish exactly what they need
in free form format.

> ). I also see that we will have this issue with entities.

Registry policy should be outside of this specification, the policy cannot
be whether they want to support an object or not, the policy is whether they
make the information public or not, and if they do, then the way of making
it public is by representing them in objects just like everyone else.

Francisco Obispo

---

From: Andy Newton <[email-masked]>
Date: Mon, 17 Sep 2012 18:09:01 +0000

On 9/17/12 12:04 PM, "Francisco Obispo" <[email-masked]> wrote:

>yes I've seen that, and perhaps it's the fact that the elements are
>intermixed with the object attributes what generates me the additional
>noise.

I'd be happy to clarify the section and add clarifying text if you can
tell us what you suggest isn't clear.

>object types and extensions used by registries with EPP.

Please provide your gap analysis if you have it.

>and will fall back to P43 whois where they can publish exactly what they

Again, if you can provide your actual analysis that would be helpful. Real
examples beat theoretical discussions.

>it public is by representing them in objects just like everyone else.

Did you read the reference above? It's not about policy. It's about object
representation.

-andy

---

From: Edward Lewis <[email-masked]>
Date: Mon, 17 Sep 2012 14:31:03 -0400

>object types and extensions used by registries with EPP.

This is an off-shoot thread, with a different subject.

It's tempting to drag up EPP into the WEIRDS WG discussions, I've 
done it.  But EPP is solving a much different problem and is in use 
by a different set of clients.  Outside of the two protocols dealing 
with registries and the low-level format (addresses and dates), there 
isn't much more in common between them.

One of my working assumptions, because I haven't seen a WG 
requirements document (still weaving my way through what's out there) 
is that the protocol to be developed by the WEIRDS WG is to allow 
inspection of a registry's object->entity relationships.  This isn't 
the same as having access to the registry's database, meaning not all 
data is necessarily accessible.  I.e., I don't expect that all EPP 
carried information is to be exposed in the WEIRDS WG protocol.  I 
don't expect to see the protocol be "capable of covering the object 
types and extensions" used in EPP exchanges.

-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis
NeuStar                    You can leave a voice message at +1-571-434-5468

2012...time to reuse those 1984 calendars!

---

From: Byron Ellacott <[email-masked]>
Date: Tue, 18 Sep 2012 12:08:44 +1000

Hi Andy,

> out, the set returnAll to true for everything.

If someone is willing to demonstrate that their weirds server would require such a flag to meet actual or reasonably likely policy conditions, I'd be OK with including it.  Otherwise, it's just one more thing client software will need to consider, and the protocol will need to include.

I have no such need of a flag, as a server I'd rather have fewer code paths and fewer clients coming back for additional data and just shove everything I know out the door in one go, and as a client I'd rather have fewer code paths and fewer round trips and just get everything I can learn in one go.

> ). I also see that we will have this issue with entities.

(http://tools.ietf.org/html/draft-newton-weirds-unified-json-response-00#section-4)

I was thinking that you could just take the data you'd put inline in that section and move it down to an additional, but you're right in that where there's no handle or other identifier you'd be making up a UUID or something just to link the two items together, and it might be quite misleading to clients to have a handle that cannot be queried for separately.

Regarding duplicated data objects, the unified response draft only has one entity object per entity handle.  There's no difference, that I can see.  It might be fractionally harder to find the particular type of entity you want, but one of the things we quickly see with the domain survey data is that the types of entities recorded is one of the most widely variant facets of registration data - I don't know that it's feasible to specify an enumeration of entity roles, and it might be hard to even specify a minimum set.

  Byron

---

From: Andy Newton <[email-masked]>
Date: Tue, 18 Sep 2012 02:37:31 +0000

On 9/17/12 10:08 PM, "Byron Ellacott" <[email-masked]> wrote:

>If someone is willing to demonstrate that their weirds server would
>require such a flag to meet actual or reasonably likely policy
>conditions, I'd be OK with including it.  Otherwise, it's just one more
>thing client software will need to consider, and the protocol will need
>to include.

That's an interesting take. I'll posit that even if such a flag were
needed for some policy reason, another policy will pop up needing another
flag, etcŠ But like you said, if somebody can demonstrate the need...

>(http://tools.ietf.org/html/draft-newton-weirds-unified-json-response-00#s

DOH!

>I was thinking that you could just take the data you'd put inline in that
>section and move it down to an additional, but you're right in that where
>there's no handle or other identifier you'd be making up a UUID or
>something just to link the two items together, and it might be quite
>misleading to clients to have a handle that cannot be queried for
>separately.

Exactly!

>Regarding duplicated data objects, the unified response draft only has
>one entity object per entity handle.  There's no difference, that I can
>see.  It might be fractionally harder to find the particular type of
>entity you want, but one of the things we quickly see with the domain
>survey data is that the types of entities recorded is one of the most
>widely variant facets of registration data - I don't know that it's
>feasible to specify an enumeration of entity roles, and it might be hard
>to even specify a minimum set.

Agreed.

-andy

---

From: Francisco Obispo <[email-masked]>
Date: Mon, 17 Sep 2012 20:31:34 -0700

> flag, etcŠ But like you said, if somebody can demonstrate the need...

I think Byron was very clear with his use case, which is exactly the same
that I have.

If you provide an answer that does not have all the components, the client might
generate additional requests asking for the missing pieces.

But I also don't want to make that behavior mandatory, in some other occasions, 
I want exactly the opposite, to prevent hitting the backend with unnecessary
requests if the client does not want them.

>> misleading to clients to have a handle that cannot be queried for

Can't speak for number registries, but it is very common for name registries
to have unique identifiers for the internet objects they manage: contacts (contact_id),
hosts (host_name), and domains (domain_names).

The ID doesn't need to be universal, it just needs to be unique to that repository.

In my opinion the handle should be used to retrieve the specific object via another
restful request, why not do it? or am I missing something

>> to even specify a minimum set.

We can start with the general set that EPP has, and the rest will need 
to be done using an extension mechanism, which is nothing more than
placing the additional information in 'extension' field that has a namespace
identifier associated to it.

Francisco Obispo