Subject: [weirds] comments on draft-kucherawy-weirds-requirements-00

From: Andy Newton <[email-masked]>
Date: Mon, 13 Jun 2011 12:49:40 -0400

My comments on draft-kucherawy-weirds-requirements-00

> 4.1
>    1.  A client SHOULD be able to handle replies that contain data that
>        are not exclusively 7-bit clean.

Would we want to go further and lock this down to UTF-8 as mandatory to support... others optional?

>    2.  A client MAY be able to handle UDP replies, and if the server
>        provides service with UDP, the client SHOULD attempt a UDP query
>        prior to a TCP query.
> 
>    With respect to a UDP service, a subset of the service could be
>    implemented via the DNS, using [EXPAND-DNS].
> 
I'm not really sure what is envisioned here, but I would assume to make this useful in any sort of way there would have to be mandatory server-side support. I really don't think we should be doing this. It complicates the model, and people who write clients will now have to find DNS libraries. And while to some of us that might sound like a small barrier, to the people we are aiming this at such a thing is not easy to understand or come by. As I've been discussing Whois-RWS and demoing over the past year, one of the big benefits is that I could demonstrate the concepts using a Web-browser. I've always wanted as a goal for it to be possible to write a client in bash with not much more than curl or xsltproc.

> 4.2.  Servers
> 
>    1.  A server MUST accept and process all queries except when in a
>        defensive posture against a denial-of-service or other security
>        attack.  Limiting the number of queries per time interval from a
>        particular source MUST NOT be done.
> 
If this is meant to say that we are developing a protocol for the open publication of records, then I think new wording is needed. But the current wording infringes upon an operators view of what constitutes an abusive user and necessary corrective measures. Its all very subjective, and to place MUST language around it only invites disagreements between people doing things incorrectly and those having to deal with many peoples doing things incorrectly.

>    2.  A server MUST reply in a univerally standard format; free-form
>        replies MUST NOT be used, although the standard format may have
>        provisions for some fields that are free-form within it.  In
>        particular:
> 
>        *  All date and/or time fields MUST be formatted as per
>           [DATETIME].
> 
I think the date/time format should be left to the schema language used in the standard. Which leads into this...

>    3.  NOTE: The standard format is expected to be a significant portion
>        of the work on the way to describing a new overall WHOIS
>        specification.  In any case, machine-parsability of replies is
>        crucial to the success of this work.
>  
With the risk of getting into a data serialization religious debate, I think we should lock down the standard format to XML. And my reason for this is that XML can be made to work seamlessly with a web browser using XSLT (well, modern browsers). Again, the use of demoing Whois-RWS with a web browser has been very beneficial with getting people to understand this area of work.

-andy

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 13 Jun 2011 10:04:50 -0700

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Andy Newton
> Sent: Monday, June 13, 2011 9:50 AM
> To: [email-masked]
> Subject: [weirds] comments on draft-kucherawy-weirds-requirements-00
> 
> > 4.1

Works for me.  This (and all the other requirements I listed) are just starting points for discussion.

> I'm not really sure what is envisioned here, but I would assume to make
> this useful in any sort of way there would have to be mandatory server-
> side support. I really don't think we should be doing this. It
> complicates the model, and people who write clients will now have to
> find DNS libraries. And while to some of us that might sound like a
> small barrier, to the people we are aiming this at such a thing is not
> easy to understand or come by. As I've been discussing Whois-RWS and
> demoing over the past year, one of the big benefits is that I could
> demonstrate the concepts using a Web-browser. I've always wanted as a
> goal for it to be possible to write a client in bash with not much more
> than curl or xsltproc.

The UDP idea comes from the rate at which some commercial evaluation services hit WHOIS servers with high query rates and seeks to reduce the network load.  We don't have to do that if it's a lot harder than the problem it solves.

The DNS idea comes from some hallway track chatter at the last ICANN meeting.  One registrar I talked to would be fine providing information that is helpful to security needs without revealing privacy data and thought adding them to the DNS would be easiest.  The intent would be to provide that limited set of information via the DNS and also provide full WHOIS service via more conventional means like TCP.  The DNS wouldn't be rate-limited for privacy reasons, but the full service could be.  The limited information that would be made available is just stuff like the registration/change date, the registered nameservers, the ID of the registrar, etc., but not any of the personal information of the registrant.  Those clients that want that specific subset of information will probably have no problem adapting to a DNS-based (or really, any UDP-based) service, but there's no expectation that everybody will change to DNS for the full service.

> If this is meant to say that we are developing a protocol for the open
> publication of records, then I think new wording is needed. But the
> current wording infringes upon an operators view of what constitutes an
> abusive user and necessary corrective measures. Its all very
> subjective, and to place MUST language around it only invites
> disagreements between people doing things incorrectly and those having
> to deal with many peoples doing things incorrectly.

I'm happy to rework the requirement.  What I'm hoping to discourage is the practice of disallowing queries from sites on the automatic presumption that just about any client making the query is trolling data for the purpose of poaching customers.  This interferes with legitimate security operations, and it is quite frequent.  It might be enough to have informational provisions for making some kind of arrangements out-of-band between security-based services that want to consume this data and the WHOIS providers.

> With the risk of getting into a data serialization religious debate, I
> think we should lock down the standard format to XML. And my reason for
> this is that XML can be made to work seamlessly with a web browser
> using XSLT (well, modern browsers). Again, the use of demoing Whois-RWS
> with a web browser has been very beneficial with getting people to
> understand this area of work.

XML works for me.  I'm not as familiar with it as I'd like, so if XML has some standard date/time format, then I'm fine with using that instead of RFC3339.

-MSK

---

From: Andy Newton <[email-masked]>
Date: Mon, 13 Jun 2011 14:49:53 -0400

> The DNS idea comes from some hallway track chatter at the last ICANN meeting.  One registrar I talked to would be fine providing information that is helpful to security needs without revealing priv…

Well, there's a lot I could say about this if backed into a corner. But for now, I'll leave it with "We don't have to do that if it's a lot harder than the problem it solves." At the very least, this should be pushed off until our core work is finished.

> I'm happy to rework the requirement.  What I'm hoping to discourage is the practice of disallowing queries from sites on the automatic presumption that just about any client making the query is tro…

I hear ya. And I support what you are getting at. I'll see about formulating some text to help here.

Perhaps we could note that some service operators also have bulk whois access, and therefore have legitimate reasons for targetting abuses attempting to get around bulk whois AUPs.

> XML works for me.  I'm not as familiar with it as I'd like, so if XML has some standard date/time format, then I'm fine with using that instead of RFC3339.

XML Schema uses a profile of ISO 8601. RelaxNG defers to XML Schema. RFC3339 is also a profile of ISO 8601. The gulf is not that wide here, but I worry about tools that assume a subtle difference in format.

Thanks.

-andy

---

From: Benedetto Fiorelli <[email-masked]>
Date: Tue, 14 Jun 2011 13:06:58 +0200

> Works for me.  This (and all the other requirements I listed) are just starting points for discussion.
This is something that could be added also to the RIPE draft, actually
the RIPE RESTful API adopt UTF-8 encoding.
> I'm happy to rework the requirement.  What I'm hoping to discourage is the practice of disallowing queries from sites on the automatic presumption that just about any client making the query is tro…
Imho it is important to adopt a framework for "terms and conditions"
defined also in terms of explicit "acceptable use policies" that the
WEIRDS systems could publish and enforce in order to avoid abuse/misuse
to happen at all.
Stating that systems must accept and process any number of request and
than keep the door open to generic anti-DoS and anti-abuse mechanism may
translate into:
-behaviours that are difficult for clients to predict and test,
-different implementations of blocking (anti-abuse) policies in
different Registries,
-something not always true because there may be services that must have
limited access even though being public, see for example the actual
blocking policies being enforced on queries that return personal data.

In other words from a client point of view it may be better to have a
contract that defines explicit terms on usage than just knowing that in
theory any use is unlimited but in practice it may be blocked whenever
considered abusive.
So I wouldn't even exclude for some services to define rate limits in
terms simple enough to be predictable and testable by clients.

Another reason for having different usage policies on different services
is simply that different interfaces and protocols have different
qualities, for example REST and in general web services are perfect for
operations that are 'online', suitable within HTTP limits of timeout,
reliability, etc.
Other interfaces or frameworks can be more adequate and convenient for
processes that are 'batch', 'bulk', 'data mining' etc.

-Benedetto

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 14 Jun 2011 10:32:27 -0400

On Mon, Jun 13, 2011 at 02:49:53PM -0400, Andy Newton wrote:
> 
> Well, there's a lot I could say about this if backed into a
> corner. But for now, I'll leave it with "We don't have to do that if
> it's a lot harder than the problem it solves." At the very least,
> this should be pushed off until our core work is finished.

I'm not sure I completely agree.  That is, I think it's something that
needs to be considered carefully from the outset and included in all
of the basic thinking that amounts to the core work.  If we don't keep
this in mind from the outset, we'll find ourselves using a big hammer
later to address it.

It might well be that the "a lot harder" stipulation is correct, but
I'd rather address that possibility early than leave it late and
discover we're in a deep hole.

> XML Schema uses a profile of ISO 8601. RelaxNG defers to XML Schema. RFC3339 is also a profile of ISO 8601. The gulf is not that wide here, but I worry about tools that assume a subtle difference i…

Given the tendency of RESTful things to be built atop JSON instead of
XML, why aren't we considering JSON for this? 

A

-- 
Andrew Sullivan
[email-masked]

---

From: Gavin Brown <[email-masked]>
Date: Tue, 14 Jun 2011 15:46:27 +0100

> XML, why aren't we considering JSON for this? 

Here's one reason why I'd prefer XML: schemas. JSON has them but they
are not mature yet (there is a draft, draft-zyp-json-schema, see
http://json-schema.org/).

Using XML and having a schema means that both client and server can
validate output, usually fairly simply.

G.

-- 
Gavin Brown
Chief Technology Officer
CentralNic Ltd
Innovative, Reliable and Flexible Registry Services
for ccTLD, gTLD and private domain name registries
https://www.centralnic.com/

CentralNic Ltd is a company registered in England and Wales with company
number 4985780. Registered Offices: 35-39 Moorgate, London, EC2R 6AR.

---

From: Andy Newton <[email-masked]>
Date: Tue, 14 Jun 2011 11:03:15 -0400

> discover we're in a deep hole.

Honestly, I'm not following you here.

But let me speak to one of the issues I perceive hinders IRIS. It's got too much neato stuff in it that is mandatory. None of the single technologies found in IRIS are that cumbersome or hard to implement on their own, but taken together in their mandatory-to-implement form they constitute an unacceptable hurdle. I'd prefer not to hinder this effort with non-HTTP, non-REST mechanisms. Also see my statement below about the usefulness of web browsers.

> XML, why aren't we considering JSON for this? 

As I stated previously, XML can be styled with XSLT in any modern web browser. That's what we do at ARIN. Our web query form redirects to a pure RESTful call, which renders XML in a browser. That, to the best of my knowledge, cannot be done with JSON without a lot of custom Javascript code. That ability to demonstrate Whois-RWS using a web-browser has been very beneficial for us. It demonstrates the power of a RESTful web service to non-programmers.

Not that I don't support JSON. At ARIN we do XML and JSON and plain text and XHMTL with our RESTful web service. I'm just suggesting the XML serialization be the mandatory-to-implement version. I'm not against a JSON version at all. 

We may want to additionally address the mechanism to request a specific format. One would think that the Accept header would satisfy, but there are complications with that which is why a lot of RESTful web services have resorted to using file extension mechanisms, which also have their warts.

-andy

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 14 Jun 2011 11:55:08 -0400

> Honestly, I'm not following you here.

What I'm saying is that the _first_ part of our basic work is not to
deliver answers in a web page -- we've already determined that people
can do that -- but to decide the basic set of functionality.  At least
for high-volume cases, something lighter than http may be necessary,
and I'm anxious that we not prejudge that requirement from the
beginning as "nice to have".  It might be an effective requirement for
the protocol work, because even if it doesn't turn out to be mandatory
to implement (I'm fairly sure it need not be), it might constrain what
we can depend on in the rest of our specification.

> But let me speak to one of the issues I perceive hinders IRIS. It's
> got too much neato stuff in it that is mandatory. None of the single
> technologies found in IRIS are that cumbersome or hard to implement
> on their own, but taken together in their mandatory-to-implement
> form they constitute an unacceptable hurdle.

I get that objection, and I'm not saying, "We need|don't need X."  I'm
instead asking for arguments in each direction so that we address the
real use cases that drive this work.  A significant number of people
want real-time access to the data this service is supposed to offer,
and I suspect that the operators of data sources are going to get
cranky about overhead if they don't have another option.  How might
that other option affect what we can do?

> be done with JSON without a lot of custom Javascript code. That
> ability to demonstrate Whois-RWS using a web-browser has been very
> beneficial for us. It demonstrates the power of a RESTful web
> service to non-programmers.  

I fully appreciate the value of the ability to demonstrate something
in a web browser.  But if we're going to go to the trouble of
standardizing something, I think we need something stronger than "Easy
demonstration" as a premise.  (I'd probably buy the
JSON-sucks-in-browsers argument, but only if we also conclude that
browsers are the primary target for many of these cases.  Are they?
The requirements document doesn't say so yet, and I can't tell.)

I'm not trying to be religious about anything (note that I haven't
developed even one of the examples).  I'm just trying to make sure we
specify things in line with all the needs of the client community
we're planning to serve, so that we don't later have to fill the hole.

A
-- 
Andrew Sullivan
[email-masked]

---

From: Andy Newton <[email-masked]>
Date: Tue, 14 Jun 2011 12:34:21 -0400

> we can depend on in the rest of our specification.

We run a Whois infrastructure that at times is the busiest on the Internet. HTTP and TCP are not our performance limiters. Anybody saying they can't do it this way is gonna have to convince me they aren't doing something wrong.

Also, on the domain side, I would question the need for this requirement by those who have already not deployed IRIS DCHK, which is the only part of IRIS that has actually seen deployment. Additionally, since IRIS DCHK is used, I don't think we should attempt to redo it.

> that other option affect what we can do?

It seems odd to me that DNS is being considered a solution to a real-time access protocol.

> The requirements document doesn't say so yet, and I can't tell.)

On the contrary, I think easy-to-demonstrate is a goal that nearly all technologies should have and a more worthy justification than you provided for JSON. And a modern web browser should be considered as a simple client in this space.

-andy

---

From: Arturo Servin <[email-masked]>
Date: Tue, 14 Jun 2011 22:25:21 -0300

> The DNS idea comes from some hallway track chatter at the last ICANN meeting.  One registrar I talked to would be fine providing information that is helpful to security needs without revealing priv…

	I can live with UDP, still not sure why DNS is involved here. I would prefer keeping DNS out of whois.

> I'm happy to rework the requirement.  What I'm hoping to discourage is the practice of disallowing queries from sites on the automatic presumption that just about any client making the query is tro…

	Rate-limiting the number of queries per time unit currently protects whois servers from DoS/DDoS and abusers (or at least we think it does). How are we going to "accept and process all queries except when in a defensive posture against a denial-of-service or other security attack" if "limiting the number of queries per time interval from a particular source MUST NOT be done", I feel that we are in a kind of of a deadlock here.

Regards,
.as

---

From: "John Levine" <[email-masked]>
Date: 15 Jun 2011 03:13:39 -0000

>The UDP idea comes from the rate at which some commercial evaluation
>services hit WHOIS servers with high query rates and seeks to reduce
>the network load.  We don't have to do that if it's a lot harder than
>the problem it solves.

I have some small scale experience here with abuse.net, the popular
service that publishes abuse contact info for domains.  You can look
stuff up using the abuse.net web site (ipv4 or ipv6), a WHOIS server,
or the DNS, and it works quite well.  The data that abuse.net provides
about a domain is simple enough that you can get it all with two DNS
queries.

For anti-abuse purposes, I think it would be useful to have a
lightweight query that returned a subset of the whois information such
as when it was registered, when it expires, what country the
registrant is in, and whether the contact info is available or
proxied.  That's little enough that I expect people would be OK not
rate limiting it.

If you're going to have a lightweight UDP query, DNS is the only
reasonable way to do it since the client firewalls already have the
requisite holes, and the caches work.  People who like shell scripts
are welcome to use "host" and similar commands.  I can tell you from
experience that serving this kind of stuff over the DNS is not hard.

R's,
John