Subject: Re: [ire] DNRD CSV Draft

From: Francisco Obispo <[email-masked]>
Date: Wed, 5 Dec 2012 17:59:52 -0800

So I'm now trying to go in more detail on the CSV implementation, to try to understand its benefits.

So far I've found the following cons of this approach:

1) It does not eliminate the need of having to code, whoever implements
   this, is going to have to write some scripts to generate the enclosing
   XML file, and generate all the CSVs in the right format. In fact, I believe
   there is going to be more work associated with this approach, see #2.

2) There's the issue of Validation of the data. The fields are defined
   using an XML Schema (i.e.: <rdeCsv:fName>  Name field with type="eppcom:labelType")
   which will create the need of writing a program to implement all of the
   types used in the schema for validation.

3) We now have more files to process/store/sign, instead of just one.

I'm not done with my tests yet, but I've been able to compress an XML file
with BZIP2 to 1/15th of its original size. I'm not envisioning that this is
going to be a problem for most registries.

I'll provide more feedback later,

Francisco Obispo 
Director of Applications and Services - ISC
email: [email-masked]
Phone: +1 650 423 1374 || INOC-DBA *3557* NOC
PGP KeyID = B38DB1BE

On Oct 25, 2012, at 1:43 PM, "Gould, James" <[email-masked]> wrote:

> All,
> 
> We have created a draft of the Domain Name Registration Data (DNRD) Comma-Separated Values (CSV) Objects Mapping that is attached for review and feedback.  We intend to post it to the IETF as an I-D once the submission page is available on November 5th.  This draft fully supports the Registry Data Escrow Specification (draft-arias-noguchi-registry-data-escrow-04).  It defines the CSV files and the order and format of the CSV fields for the data escrow of domain name, host, contact, registrar, and IDN language objects.  If there is interest in this model we can consider merging it with the Domain Name Registration Data (DNRD) Objects Mapping (draft-arias-noguchi-dnrd-objects-mapping-01).  The basis of using CSV files for DNRD objects includes:
> 
> 	• CSV is a natural format for exporting and importing data from and to a database.  This could greatly simplify the generation of the data escrow files as well as the consumption of the files by an EBERO provider.  
> 	• XML is a highly verbose format that will adversely affect the processing of large data sets .  With the draft, XML can be used for definition and CSV can be used for data, so the duplication of the descriptive information does not have to be used for every record.  
> 		• If you take the domain object (<rdeDomain:domain>) example from draft-arias-noguchi-dnrd-objects-mapping-01 and convert it to CSV files (domain, dnssec, and domainTransfer) you save around 75% uncompressed and 82% compressed using CSV.  
> 		• Extrapolating out the uncompressed size of a <rdeDomain:domain> records (1718 bytes versus 416 bytes uncompressed per record) for XML and CSV,  you get to 1.7 GB with XML and 443 MB for CSV with 1 million records and 170 GB with XML and 44.3 GB for CSV with 100 million records.
> 		• The deposits are generated uncompressed, validated, compressed and transferred by registry, and uncompressed, validated and stored by the data escrow provider.  Both the size difference and the processing resources required for both the registry and the data escrow provider should be considered when comparing the two models.  
> 		• EBERO providers must transfer, uncompress, validate, and import the data into their database from the data escrow deposits, where the larger the files and the processing resources required, the longer it will take to recover the TLD.  
> 		• The full deposit is done weekly, so it is a weekly hit for all registries, where the larger the registry the bigger the hit.  
> Please review the attached draft and provide any feedback for consideration.  
> 
> Thanks,
> 
> --
>   
> JG
>  
> <86BF0728-DD04-4F90-8380-5AA8A9AB5D0B[81].png>
>  
> James Gould
> Principal Software Engineer
> [email-masked]
>  
> 703-948-3271 (Office)
> 12061 Bluemont Way
> Reston, VA 20190
> VerisignInc.com
> 
> 
> <draft-gould-thippeswamy-dnrd-csv-mapping.txt>_______________________________________________
> ire mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ire

---

From: "Gould, James" <[email-masked]>
Date: Thu, 6 Dec 2012 14:32:09 +0000

Francisco,

Thanks for the reply, my feedback is embedded below.

-- 

JG

James Gould
 
703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

>   there is going to be more work associated with this approach, see #2.

I disagree with this.  First the XML definition that follows
draft-arias-noguchi-registry-data-escrow-04 would be primarily static
content, with the dynamic content being the file names and optionally the
file checksums.  You could use a templating language (e.g. Apache
Velocity) or simple string replacement to replace the file names and
checksums in the static definition XML file.  This is very straight
forward and might take less time then the time that I'm taking creating
this e-mail.  You define the fields that are applicable to your registry
in the XML definition and then you're free to use any tool (off the shelf
or custom) to generate the CSV files. The CSV files map one-to-one with a
relational schema, so there is very little transformation required.  The
all XML approach of draft-arias-noguchi-dnrd-objects-mapping-01 requires a
relational to object conversion, that is far more complex and is far more
work then dumping fields from a relational schema to a set of CSV files.

>   types used in the schema for validation.

I do agree that there is the need to create a custom validation program
that validates XSD field format definitions to the CSV data fields.  I
believe that the community can create this tool, and we would certainly
like to participate in that effort.  The fact that you can validate XML
using an XML parser doesn't mean that we should make the generation and
consumption of the data escrows more complex to forgo having to build a
new validation program.  The field types defined in
draft-gould-thippeswamy-dnrd-csv-mapping-00 utilize XSD types, so it's a
matter of reusing parts of the XML Parsers to validate individual field
elements.     

>3) We now have more files to process/store/sign, instead of just one.

The files could be placed and compressed into a ZIP file per the AGB, so
the number of contained files is not relevant for transfer, signing, and
storage.  

>going to be a problem for most registries.

You can also compress the CSV files down to 1/15th of their original size.
 You can't ignore the uncompressed size from the processing though.  The
compression can be done to reduce the transfer time and the storage costs.
 You should compare the uncompressed and compressed deposits using CSV and
XML with incrementally larger randomized data sets.  I believe you will
find a large advantage with the use of CSV from a size perspective.  I
fundamentally believe that the draft should address all registries
independent of its size.  Saying that some of the registries will have an
problem with the use of XML to me is a showstopper for XML deposits.

>> We have created a draft of the Domain Name Registration Data (DNRD)
>>Comma-Separated Values (CSV) Objects Mapping that is attached for review
>>and feedback.  We intend to post it to the IETF as an I-D once the
>>submission page is available on November 5th.  This draft fully supports
>>the Registry Data Escrow Specification
>>(draft-arias-noguchi-registry-data-escrow-04).  It defines the CSV files
>>and the order and format of the CSV fields for the data escrow of domain
>>name, host, contact, registrar, and IDN language objects.  If there is
>>interest in this model we can consider merging it with the Domain Name
>>Registration Data (DNRD) Objects Mapping
>>(draft-arias-noguchi-dnrd-objects-mapping-01).  The basis of using CSV
>>files for DNRD objects includes:
>> 
>> 	€ CSV is a natural format for exporting and importing data from and to
>>a database.  This could greatly simplify the generation of the data
>>escrow files as well as the consumption of the files by an EBERO
>>provider.  
>> 	€ XML is a highly verbose format that will adversely affect the
>>processing of large data sets .  With the draft, XML can be used for
>>definition and CSV can be used for data, so the duplication of the
>>descriptive information does not have to be used for every record.
>> 		€ If you take the domain object (<rdeDomain:domain>) example from
>>draft-arias-noguchi-dnrd-objects-mapping-01 and convert it to CSV files
>>(domain, dnssec, and domainTransfer) you save around 75% uncompressed
>>and 82% compressed using CSV.
>> 		€ Extrapolating out the uncompressed size of a <rdeDomain:domain>
>>records (1718 bytes versus 416 bytes uncompressed per record) for XML
>>and CSV,  you get to 1.7 GB with XML and 443 MB for CSV with 1 million
>>records and 170 GB with XML and 44.3 GB for CSV with 100 million records.
>> 		€ The deposits are generated uncompressed, validated, compressed and
>>transferred by registry, and uncompressed, validated and stored by the
>>data escrow provider.  Both the size difference and the processing
>>resources required for both the registry and the data escrow provider
>>should be considered when comparing the two models.
>> 		€ EBERO providers must transfer, uncompress, validate, and import the
>>data into their database from the data escrow deposits, where the larger
>>the files and the processing resources required, the longer it will take
>>to recover the TLD.
>> 		€ The full deposit is done weekly, so it is a weekly hit for all
>>registries, where the larger the registry the bigger the hit.

---

From: Francisco Obispo <[email-masked]>
Date: Thu, 6 Dec 2012 10:57:55 -0800

Hi James,

Thanks for replying.

Here's another set of comments:

On Dec 6, 2012, at 6:32 AM, "Gould, James" <[email-masked]> wrote:

> file checksums.  

And the watermark, and options (resend, etc.)

> You could use a templating language (e.g. Apache
> checksums in the static definition XML file.

I've found this to be very dangerous in some situations, so I prefer to generate XML files the proper way to assure that they are well formed.

Also we're dealing with Name registries that are already using or will be using EPP, which is XML based, so they should have expertise in house to deal with that.

>  This is very straight
> or custom) to generate the CSV files.

I've written both specs in two days, I'm also planning to build a tool that can load the XML into a database without the need of having to do any conversions. Using a FDW (Foreign Data Wrapper). And if I end up with free time, I'll write a XSLT to convert the XML to CSV, which should be very straightforward. :-)

> The CSV files map one-to-one with a

This is not necessarily true, I've found myself doing some massaging to make our schema fit some of the specs, so it might be the case for some registries that might be able to do just one SELECT or COPY and export the data, but not all.

Also, ICANN might want to enforce a minimum set of fields to make sure that the recovered registry is useful, this is something that is up for discussion of course.

And BTW, we're not planning to use neither Incremental or Differential files, Data Escrow providers will not keep track of the state of the registry, just merely validate and store files in case ICANN decides to failover a registry. So we decided that the safest method was to use full dumps to keep the data consistent at all times.

One last thing:

It seems at least from having implemented both approaches, that both solutions will have its pro and its cons, its kind of like trying to keep a piece of cardboard flat on the ground , you step on it on one side, and the other side comes up. 

If we make it really easy for registries, we might end up making it harder for escrow providers and EBEROs, and if we make it really easy for escrow and EBEROs we might end up adding more work for registries. 

I personally think that the XML is much easier to recover, it's self contained, and with some of the tools mentioned above (FDW, XLTs), anyone wanting to consume the data can do it without much effort (at least for a registry without complicated extensions). The CSV approach might be simpler or not, but will require more effort to recover since the consumer will have to implement a recovery procedure based on the XML spec and massage data to make it fit to their schemas, and since we're allowing registries to specify custom fields, it's not going to be trivial.

Best regards,

Francisco Obispo

---

From: "Gould, James" <[email-masked]>
Date: Thu, 6 Dec 2012 21:47:01 +0000

Francisco, 

My replies are again embedded below.

-- 

JG

James Gould
 
703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

>And the watermark, and options (resend, etc.)

JG - Yes, the watermark and options can be variables in the mostly static
XML definition file.

>I've found this to be very dangerous in some situations, so I prefer to
>generate XML files the proper way to assure that they are well formed.
>
>Also we're dealing with Name registries that are already using or will be
>using EPP, which is XML based, so they should have expertise in house to
>deal with that.

JG - I'm not saying that the registries can't produce the XML definition
file without the use of templates, I'm just saying that since the XML file
itself is mostly static, simple techniques can be used to fill the dynamic
content.  In either event it should be very straight forward to generate
the XML definition file that includes references to a set of external CSV
files.  

>I've written both specs in two days, I'm also planning to build a tool
>that can load the XML into a database without the need of having to do
>any conversions. Using a FDW (Foreign Data Wrapper). And if I end up with
>free time, I'll write a XSLT to convert the XML to CSV, which should be
>very straightforward. :-)

JG - The good news it that you don't need to do much work at all to
support CSV data files since the databases should already support them :-)

>This is not necessarily true, I've found myself doing some massaging to
>make our schema fit some of the specs, so it might be the case for some
>registries that might be able to do just one SELECT or COPY and export
>the data, but not all.

JG - Correct, one size doesn't fit all.  In general, a registry that is
using a relational database can define the CSV fields in the XML
definition file that matches their system and then generate the CSV files
using standard tools.  The only massaging required is to format some of
the fields in a specific way, like formatting date and times in an XSD
dataTime format.  

>
>Also, ICANN might want to enforce a minimum set of fields to make sure
>that the recovered registry is useful, this is something that is up for
>discussion of course.

JG - I would like to know what the minimum set of fields would be.  I do
agree that if a registry is supporting EPP that there should be a minimum
set of fields stored in the database and subsequently in the data escrow.

>
>And BTW, we're not planning to use neither Incremental or Differential
>files, Data Escrow providers will not keep track of the state of the
>registry, just merely validate and store files in case ICANN decides to
>failover a registry. So we decided that the safest method was to use full
>dumps to keep the data consistent at all times.

JG - Does this mean that incremental or differential deposits is not a
requirement but an option?

>
>One last thing:
>
>It seems at least from having implemented both approaches, that both
>solutions will have its pro and its cons, its kind of like trying to keep
>a piece of cardboard flat on the ground , you step on it on one side, and
>the other side comes up.
>
>If we make it really easy for registries, we might end up making it
>harder for escrow providers and EBEROs, and if we make it really easy for
>escrow and EBEROs we might end up adding more work for registries.
>
>I personally think that the XML is much easier to recover, it's self
>contained, and with some of the tools mentioned above (FDW, XLTs), anyone
>wanting to consume the data can do it without much effort (at least for a
>registry without complicated extensions). The CSV approach might be
>simpler or not, but will require more effort to recover since the
>consumer will have to implement a recovery procedure based on the XML
>spec and massage data to make it fit to their schemas, and since we're
>allowing registries to specify custom fields, it's not going to be
>trivial.

JG - I believe that CSV will be easier and faster for the EBERO provider
to recover the data.  The EBERO provider will have some target database
schema to import into and they will be given a deposit that includes a
definition file of the records and fields that that can easily map to
their schema.  The extension of attributes is a reality independent of the
deposit format, where an EBERO provider might store additional attributes
in general fields to keep the data together, but might not actually do
anything with them.  In both cases (XML and CSV), extensions are defined
in separate XSD files.  The only difference is that with XML the EBERO
provider would need to decode the XML that includes the attribute
extensions and with CSV they would identify attribute extensions in the
field definition and import them into the appropriate generic location.

---

From: Francisco Obispo <[email-masked]>
Date: Thu, 6 Dec 2012 13:52:01 -0800

Hi James,

Well, it's ok to disagree on some implementation issues, but I'm glad to see that we are mostly in the same page with regards to the rest of the implementation.

> requirement but an option?

It seems like that is the case, the feedback that I got was that as long as what we provide is better than what is required, we should be good to go.

Francisco Obispo

---

From: Francisco Obispo <[email-masked]>
Date: Fri, 7 Dec 2012 13:13:04 -0800

What is the next step on this?

Should we arrange a call between:

* ICANN
* Ebero applicants
* Registry Backend Operators
* Escrow providers
* Anyone else interested

To try to understand the various point of views and try to derive a conclusion?

I'm a bit worried that time keeps passing and there are still several pieces that have not come to clear definition.

Francisco Obispo 

On Dec 6, 2012, at 1:52 PM, Francisco Obispo <[email-masked]> wrote:

> _______________________________________________
> ire mailing list
> [email-masked]

---

From: "Gould, James" <[email-masked]>
Date: Fri, 7 Dec 2012 22:17:53 +0000

Francisco.

A call is a good idea.  Can you set that up?

JG

James F. Gould
Principal Engineer
Verisign

[email-masked]