Subject: [ire] Extended verification process of the escrow deposit files

From: Gustavo Lozano <[email-masked]>
Date: Thu, 13 Dec 2012 16:28:20 -0800

Colleagues,

Specification 2, Part A, 8. of the new gTLD applicant guidebook specifies:

" (5) If [1] includes a verification process, that will be applied at this step."

We are in the process of defining the extended verification process that the data escrow agents must perform daily with the escrow deposit file. This process will be included in new versions of the data escrow draft.

The purpose of this continuous verification process is to mitigate the risk of  an inconsistent data escrow deposit file.

In case of differentials deposits, the escrow agent shall use the last full and all the differentials deposits to perform the extended verification process.

I am interested in your feedback regarding:

 1. Other tests that you consider should be present in the draft.

 2. The proposed tests itself.

Proposed tests (this is not a final list of tests, we want to create the final list of tests with your input):

 1. Validate the escrow deposit file using the schema defined for the TLD including extensions as defined in Specification 2, Part A, 3.2 of the new gTLD base agreement.

 2. All contacts linked to domain names in the escrow deposit files are present.

 3. All hosts linked to domain names in the escrow deposit files are present.

 4. All the required glue records are present.

 5. The IPv4 addresses of the glue records are valid global address, or in other words not present in the special use IPv4 address blocks defined in: RFC5735 and RFC6598.

 6. The IPv6 addresses of the glue records are part of 2000::/3 with the exception of sub allocations mentioned in RFC5156.

 7. The creation and last update date of objects is not in the future.

 8. Email addresses are syntactically valid.

Note: heuristics tests like the abrupt increase or decrease of domain names escrowed will be performed by ICANN using the data escrow report sent by the data escrow agents to ICANN.

Regards,
Gustavo Lozano

---

From: James Mitchell <[email-masked]>
Date: Sat, 15 Dec 2012 01:45:08 +1100

Comments below.

James

From: Gustavo Lozano <[email-masked]<mailto:[email-masked]>>
Date: Friday, 14 December 2012 11:28 AM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: [ire] Extended verification process of the escrow deposit files

Colleagues,

JM: My biggest gripe with this is that valid is subjective to the party that has to restore the registry. I say this because different registries have different business rules, or have different interpretations of specifications etc. Verisign's comments on the provreg list about the crID field mirrors exactly that. We (ARI) would expect the crID field to be a registrar id, matching the same definition of client as used in the definition of clID (sponsoring client) and would consider invalid any deposit where the crID is not a registrar. We store the creating user information separate however this is not available through the EPP interface.

I am fairly concerned that introducing this validation seems a somewhat roundabout way at mandating business rules on the registry's provisioning systems. Consider four possible scenarios:

 *   The EBERO compares contact identifiers in a case insensitive manner (lets assume that they store them uppercase), however they receive a deposit from a registry where contact identifiers "abc" and "ABC" match different ROIDs.
 *   The EBERO can store domain passwords up to 16 characters, however they receive a deposit with a password that is 17 characters.
 *   The EBERO requires contact country codes are on the ISO 3166-2 list, however they receive a deposit with a contact having 'xx' or '12' as their country.
 *   The EBERO uses a schema validating parser, however receives a deposit where the contact identifiers "abc def" and "abc  def" (two spaces) match different ROIDs.

I could go on for quite awhile but I hope by now you get my point. If you want to make sure that data is valid then I would much prefer the discussion is about provisioning system requirements and it is made clear that every registry would have to implement these requirements (to the theme of consensus policy or equivalent). As a side note, if you want to go so far as to state all these validations then I would expect to see these requirements somewhere in the EBERO contract.

JM: Fine. I am dubious to what value this provides other than syntactical validation of certain fields (date, phone etc). I could theoretically put "the quick brown fox jumps over the lazy dog" as a domain name and apparently you'd never think this invalid.

JM: Agree. I see no reason for referenced contact objects to be missing.

JM: Disagree. Hosts are required to be objects only for glue records. There is no  reason for the host "ns1.example.com" to be an object in my registry because 1) it cannot have addresses, 2) we do not allow change of host name. A registrar could create the aforementioned host, however they will never be able to do anything with it.

Have you considered that registries may use host attributes instead of host objects? Perhaps those yet unknown SRSU registries may use attributes for simplicity since they have to implement EPP even though there will be only one registrar...

JM: What is a required glue record? If a registrar omits glue address records then the name may not resolve. This is indifferent to the registrar providing the wrong name servers, or failing to supply name servers for delegation when the child zone is "live". Registrars will fix the issue if it is a problem. I disagree with validation of glue records.

JM: I see no real problem with the above two. I am wary of having too many restrictions that will require changes if and when new allocations are made.

JM: This test seems to provide no value, especially given that it should be almost impossible for a registry to have data that would invalidate this test. Testing that the creation date is not later than the last update date makes some sense as it may identify producers that have selected data into the wrong field/element.

JM: Please define the syntax that registries will be held accountable to for validation of email addresses. Note that I do not see anywhere that the registry must validate email addresses (we do not send emails to registrants). Actually, if email data is incorrect then ICANN should be looking at the registrars who provided "invalid" data to the registry.

JM: fine as long as your heuristics are not considered immediate validation failures.

Regards,
Gustavo Lozano

---

From: Christopher Browne <[email-masked]>
Date: Fri, 14 Dec 2012 11:07:40 -0500

On Thu, Dec 13, 2012 at 7:28 PM, Gustavo Lozano
<[email-masked]> wrote:
> In case of differentials deposits, the escrow agent shall use the last full
> and all the differentials deposits to perform the extended verification
> process.
>
>

In keeping with James Mitchell's comments about "the quick brown fox
jumps over the lazy dog", which could be set as a domain name, which
may be "structurally valid," but structurally nonsense, I would
suggest that mere comparison of escrow deposits is nowhere near
sufficient.

We could have a near-infinite number of policies, for each bit of the
escrow format, and someone with a Markov Chain generator could make up
a conforming escrow file out of random, but still policy-conforming,
junk.  Verifying that it's not just junk requires looking to other
sources such as WHOIS or DNS.

I daresay I have seen all the cases he mentions as problems, and more.
 Most typical and painful to resolve is to receive phone numbers not
complying with the ITU E164 standard. My "favourite" was receiving
contact <contact:id> values that did not conform to RFC 5733, so that
I had to remap them to new values in multiple places before
proceeding.

A more legitimate test would be more "end-to-end"; it would compare
escrow with other notionally authoritative sources.

The approach that comes to mind is to select data from 4 sources and
make sure that they match:

The data sources:

1.  Escrow is the one that is obviously at hand;

2.  Zone files tend to be not difficult to obtain, and they indicate
the state of the resolving portions of the 'portfolio' of domains in a
registry;

3.  WHOIS reports information on the state of the registry, albeit on
a one-by-one per-request basis;

4.  Submitting DNS requests to the authoritative DNS resolver for the
registry returns authoritative information on resolution information
for active domains.

#1 and #2 (perhaps just #1) are available "en masse", for the entire registry.

A test that escrow matches against the registry would be to pick a
sample of domain names from sources 1 (escrow) and 2 (zone file), and
agree them against WHOIS (#3) and the DNS resolver (#4).

It would not be reasonable to try to do this with all names in the
registry; a reasonably small statistically random sample should, if
collected regularly, provide confidence that samples taken from escrow
match against the "real data" in the registry.  Using stratification
would be appropriate to make sure that different sorts of cases get
examined (or perhaps avoided - it's desirable not to have many "false
mismatches" as would happen when objects legitimately change between
the time escrow is dumped and when validation tests are done.)

Over time, a validation involving doing a domain data reconciliation
of even as small a sample as a few dozen names per day should provide
useful assurance that the sources are not vastly out of sync,
particularly if discrepancies get escalated into larger targeted
samples.  Note that this is the standard sort of thing you'll find in
literature on auditing procedures.

---

From: "Gould, James" <[email-masked]>
Date: Mon, 17 Dec 2012 18:57:18 +0000

Gustavo,

Has the concept of an "extended verification process" for the data escrow agents been previously discussed or documented?   It looks like some of the data integrity checks (linked contacts and hosts) along with business rules like whether or not there are glue records and the creation and last update dates in the future are better handled by an EBERO provider based on the deposits made.  The Data Escrow Provider should be able to validate the completeness of the deposit by verifying the signatures and verify the deposit against the format definition.  Items like email address validation should be covered by the XSD format definition.  Applying differential deposits to the full deposit also sounds like the job of the EBERO provider and not the Data Escrow Provider.    In summary, I believe that only #1 below should be required of the Data Escrow Provider and the remainder should be discussed as a requirement for an EBERO provider.

--

JG

James Gould
Principal Software Engineer
[email-masked]

703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

From: Gustavo Lozano <[email-masked]<mailto:[email-masked]>>
Date: Thursday, December 13, 2012 7:28 PM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: [ire] Extended verification process of the escrow deposit files

Colleagues,

Regards,
Gustavo Lozano

---

From: Gustavo Lozano <[email-masked]>
Date: Mon, 17 Dec 2012 12:44:22 -0800

James, James, and Christopher,

Thank you for your comments.

The objective of the extended verification process is to reduce the risk of finding that the escrow deposit is inconsistent when it is too late (e.g. the EBERO is needed and the current registry operator is uncooperative or unable to provide the requisite data).

The objective is not to create provisioning requirements nor create an audit process.

ICANN access to the escrow deposit is sporadic; therefore the data escrow agent is in the best position to execute a verification procedure of the data escrow deposit file.

The standard verification procedure performed by the data escrow agent, specification 2 of the new gTLD AGB part A, 8, 1-4, basically require the data escrow agent verify the signature and validate using the XML schema of the dnrd draft. The XML schema of the dnrd draft is too generic and it does not include extensions.

Step 5 of the verification procedure mentioned in the AGB (specification 2 of the new gTLD AGB, part A, 5) is there to define extra tests that must be performed by the escrow agent. We envisioned use step 5 to at least define that escrow agents must validate the escrow deposit file using the customized XML schema for the new gTLD and the data escrow extensions.

We want to add more tests to this verification procedure (step 5) in order to reduce the risk mentioned above. These tests must be universally applicable regarding of the business model of the registry operator. We believe that the expertise present in this list could help to refine this extended verification process to narrow the filter of what appears to be a consistent data escrow deposit.

The extended verification tests could be separated in two categories: applicable to all TLDs (e.g. ccTLDs) and applicable to the new gTLD program.

It is worth mentioned that the extended verification process is not the only mechanism envisioned by ICANN to mitigate the risk of an inconsistent escrow deposit file.

Thank you.

Regards,
Gustavo

From: <Gould>, James <[email-masked]<mailto:[email-masked]>>
Date: Monday, December 17, 2012 10:57 AM
To: Gustavo Lozano <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [ire] Extended verification process of the escrow deposit files

Gustavo,

--

JG

James Gould
Principal Software Engineer
[email-masked]<mailto:[email-masked]>

703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

Colleagues,

Regards,
Gustavo Lozano

---

From: Francisco Obispo <[email-masked]>
Date: Mon, 17 Dec 2012 13:00:06 -0800

Hi *,

On Dec 17, 2012, at 10:57 AM, "Gould, James" <[email-masked]> wrote:
...
> The Data Escrow Provider should be able to validate the completeness of the deposit by verifying the signatures and verify the deposit against the format definition.  Items like email address validation should be covered by the XSD format definition. 

As long as we use the XML, if the CSV is used, the validation will have to proceed by type-checking the values in each file.

>  Applying differential deposits to the full deposit also sounds like the job of the EBERO provider and not the Data Escrow Provider.    

I agree,

What I propose is to ICANN use its contracted EBEROs to validate and confirm that they can rebuild a registry with the information in the deposits. I would say at least two (selected at random).

> In summary, I believe that only #1 below should be required of the Data Escrow Provider and the remainder should be discussed as a requirement for an EBERO provider.  

+1

Francisco Obispo 
Director of Applications and Services - ISC
email: [email-masked]
Phone: +1 650 423 1374 || INOC-DBA *3557* NOC
PGP KeyID = B38DB1BE

---

From: "Gould, James" <[email-masked]>
Date: Mon, 17 Dec 2012 21:13:05 +0000

Francisco,

Just to clarify the comment on CSV and XML.  The CSV format is defined in
XML and via XSD(s), so a validator (custom) would be able to verify the
CSV field formats of the CSV files in a similar fashion as using an XML
parser to validate an individual XML file against the XSD(s).  The email
address field value would be validated against the XSD simple element type
in either event.  
  
-- 

JG

James Gould
 
703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

On 12/17/12 4:00 PM, "Francisco Obispo" <[email-masked]> wrote:

>> The Data Escrow Provider should be able to validate the completeness of
>>the deposit by verifying the signatures and verify the deposit against
>>the format definition.  Items like email address validation should be
>>covered by the XSD format definition.
>
>
>As long as we use the XML, if the CSV is used, the validation will have
>to proceed by type-checking the values in each file.
>
>>  Applying differential deposits to the full deposit also sounds like
>>the job of the EBERO provider and not the Data Escrow Provider.
>
>I agree,
>
>What I propose is to ICANN use its contracted EBEROs to validate and
>confirm that they can rebuild a registry with the information in the
>deposits. I would say at least two (selected at random).
>
>
>> In summary, I believe that only #1 below should be required of the Data
>>Escrow Provider and the remainder should be discussed as a requirement
>>for an EBERO provider.
>
>+1
>
>
>
>Francisco Obispo

---

From: Gustavo Lozano <[email-masked]>
Date: Mon, 17 Dec 2012 13:39:30 -0800

Christopher,

Comments below.

>sources such as WHOIS or DNS.

GL. The objective of this extended verification procedure is to verify
that the escrow deposit file appears to be consistent.

>the time escrow is dumped and when validation tests are done.)

GL. As you mentioned this steps are appropriate for an audit procedure. I
will send your suggestions to the persons involved in the audit process
for new gTLD.

---

From: Gustavo Lozano <[email-masked]>
Date: Mon, 17 Dec 2012 14:02:03 -0800

James,

Comments below.

Gustavo,

Has the concept of an "extended verification process" for the data escrow agents been previously discussed or documented?   It looks like some of the data integrity checks (linked contacts and hosts) along with business rules like whether or not there are glue records and the creation and last update dates in the future are better handled by an EBERO provider based on the deposits made.  The Data Escrow Provider should be able to validate the completeness of the deposit by verifying the signatures and verify the deposit against the format definition.

GL. The problem perceived with this approach is that the format definition on the draft is too generic. An extension mechanism is being incorporated in the new version of the draft-arias-noguchi-dnrd-objects-mapping draft and we require that data escrow agents validate the data related to extensions. I understand that you define some extensions guidelines in the draft-gould-thippeswamy-dnrd-csv-mapping draft as well.

 Items like email address validation should be covered by the XSD format definition.  Applying differential deposits to the full deposit also sounds like the job of the EBERO provider and not the Data Escrow Provider.    In summary, I believe that only #1 below should be required of the Data Escrow Provider and the remainder should be discussed as a requirement for an EBERO provider.

GL. Requiring the EBEROs perform verification of the escrow deposit files frequently is an unlikely scenario. The data escrow agent is the party other than the registry with frequent (daily) access to escrow files, therefore the data escrow agent is in best position to perform a verification procedure. Depending on the day of the week, the full deposit and differentials are require to have a snapshot of the SRS database at that point in time, and it is expected that the extended verification procedure will require access to a snapshot of the SRS database.

--

JG

James Gould

703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

Colleagues,

---

From: "Gould, James" <[email-masked]>
Date: Mon, 17 Dec 2012 22:20:31 +0000

Gustavo,

My feedback is below.

--

JG

James Gould

703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

From: Gustavo Lozano <[email-masked]<mailto:[email-masked]>>
Date: Monday, December 17, 2012 5:02 PM
To: James Gould <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [ire] Extended verification process of the escrow deposit files

James,

Comments below.

Gustavo,

JG - I don't believe that the extensions changes anything related to the responsibility of the data escrow agent.  Both draft-arias-noguchi-dnrd-objects-mapping and draft-gould-thippeswamy-dnrd-csv-mapping provide extensibility of attributes and objects by the registry defining their own extension XSD, which I'm assuming would need to be passed to the data escrow agent out-of-band or with each deposit.  The data escrow agent would be able to verify the completeness of the deposit by verifying the signatures and verifying the deposit (standard and extensions) against the format definition.

JG – I am still not sure where the requirement for an extended verification procedure has been defined.  It sounds like you are adding new requirements for the data escrow providers that has contractual and policy impacts.  The data escrow providers simply take the data, do completeness and format checks, archive the data, and make the data available based on contractual conditions.  I have never heard of a deep data scrub that is being referred to as "extended verification" for the data escrow provider.  This kind of deep data scrub would be within the expertise of the EBERO provider to perform if they had access to the data.

--

JG

James Gould

703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

Colleagues,