Subject: draft dnsind interop test report

From: William King <[email-masked]>
Date: Thu, 09 Dec 1999 10:00:08 -0600

46th IETF DNSIND interoperability testing report

ABSTRACT

This report describes the testing configuration and methodology used
for testing dynamic update, notify and ixfr capabilities and summarizes
the results obtained.

Testing took place Monday, November 8, 1999 in the Capitol Room, Omni
Shoreham hotel, Washington D.C. Checkpoint, CISCO, IEI, ISC, Lucent,
Microsoft and Namesurfer attended.

TEST CONFIGURATION

Each vendor configured as primary for its own update.<vendor>.ind.test.
zone and as secondary for each of the remaining update.<vendor>.ind.test.
zones.

TEST METHODOLOGY

A perl test script, acting as a reference client and available at
http://www.isc.org/interop/update_test.pl, tested the 2136 functional
requirements described in the section 'Test references' below against
each primary server in turn, and the behavior of the primary and each
secondary server was observed. The results are provided in the following
table. Numeric references in vendor columns refer to test result notes
provided in the section 'Notes on results' following the 'Test references'
section. A 'x' denotes a pass result. A '-' denotes an unreported result.
Further information about test conditions can be obtained from
review of the reference client and configuration files available at
http://www.isc.org/interop/. Vendor references in the table below are
intentionally obfuscated.

December 9, 1999				      page 1

	   46th IETF DNSIND interoperability testing report

      Dynamic update test results with server acting as primary.
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | Test  |  V1   |  V2   |  V3   |  V4   |  V5   |  V6   |  V7   |
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | 1	   |  1    |  x    |  x    |  13   |  15   |  x    |  20   |
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | 2	   |  x    |  x    |  x    |  x    |  -    |  x    |  x    |
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | 3	   |  x    |  x    |  x    |  x    |  -    |  x    |  x    |
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | 4	   |  x    |  x    |  10   |  x    |  -    |  x    |  x    |
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | 5	   |  x    |  3    |  x    |  x    |  -    |  x    |  x    |
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | 6	   |  x    |  4    |  x    |  x    |  -    |  x    |  x    |
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | 7	   |  x    |  x    |  x    |  x    |  -    |  x    |  x    |
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | 8	   |  x    |  x    |  x    |  x    |  -    |  x    |  x    |
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | 9	   |  x    |  5    |  11   |  x    |  -    |  x    |  x    |
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | 10    |  x    |  6    |  x    |  x    |  -    |  16   |  x    |
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | 11    |  x    |  x    |  x    |  x    |  -    |  x    |  x    |
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | 12    |  x    |  x    |  x    |  x    |  -    |  x    |  x    |
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | 13    |  x    |  7    |  x    |  x    |  -    |  17   |  x    |
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | 14    |  x    |  x    |  12   |  14   |  -    |  x    |  21   |
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | 15    |  2    |  8    |  x    |  x    |  -    |  18   |  x    |
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | 16    |  x    |  x    |  x    |  x    |  -    |  x    |  x    |
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | 17    |  x    |  x    |  x    |  x    |  -    |  x    |  x    |
   +-------+-------+-------+-------+-------+-------+-------+-------+
   | 18    |  x    |  9    |  x    |  x    |  -    |  19   |  x    |
   +-------+-------+-------+-------+-------+-------+-------+-------+

December 9, 1999				      page 2

	   46th IETF DNSIND interoperability testing report

TEST REFERENCES

Section 1: test simple prerequisites in the absence of data.

1) RCODE of NXDOMAIN when prereq is yxdomain and name doesn't exist
2) RCODE of NXRRSET when prereq is yxrrset and name,type doesn't exist
3) RCODE of NOERROR when prereq is nxdomain and name doesn't exist
4) RCODE of NOERROR when prereq is nxrrset and name,type doesn't exist
5) RCODE of NXRRSET when prereq is yxrrset and RR doesn't exist

Section 2: test simple creation of data

1) RCODE of NOERROR when update add of RR and RR does not exist

Section 3: test simple prerequisites in the presence of data. Tests
performed using a configuration with one A record for name.

1) RCODE of NOERROR when prereq is yxdomain and name with A record
exists
2) RCODE of NOERROR when prereq is yxrrset and name,type exists
3) RCODE of YXDOMAIN when prereq is nxdomain and name exists
4) RCODE of YXRRSET when prereq is nxrrset and name,type exists
5) RCODE of NOERROR when prereq is yxrrset and RR exists

Section 4: test RRset exists prerequisites

1) RCODE of NOERRROR when prereq of yxrrset for both names, no data.
2) RCODE of NOERRROR when prereq of yxrrset for both names, different
order to case 1, no data.
3) RCODE of NXRRSET when prereq of yxrrset for first existing RR
4) RCODE of NXRRSET when prereq of yxrrset for second existing RR
5) RCODE of NXRRSET when prereq of yxrrset for 3 RRs, only two of which
exist

Section 5: test more complex RRset existence prerequisites.

1) RCODE of NOERROR when prereq of yxrrset with 4 existing A records
for name2, 2 existing TXT records for name2 and 1 existing A record for
name3.

Section 6: test simple deletion. Tested with multiple A records for name.

1) RCODE of NOERROR when deleting existing name,ttl,type,rdata
2) RCODE of NOERROR when prereq is yxrrset and name,ttl,type,rdata exists
3) RCODE of NOERROR when deleting existing name,ttl,type,rdata
4) RCODE of NOERROR when prereq is nxrrset and name,ttl,type doesn't exist
5) RCODE of NOERROR when prereq is nxdomain and name doesn't exist

December 9, 1999				      page 3

	   46th IETF DNSIND interoperability testing report

Section 7: test other forms of deletion

1) RCODE of NOERROR when prereq is nxrrset and name,ttl,type does not
exist in zone containing multiple RRs for name
2) RCODE of NOERROR when deleting name and multiple RRs for name exist.
Verify with:
3) RCODE of NOERROR when prereq is nxdomain for name

Section 8: case insensitivity

1) RCODE of NOERROR when prereq is yxrrset for existing
name,ttl,type,rDaTa

Section 9: special CNAME rules

1) RCODE of NOERROR and operation not successful when attempting to add
an A RR when CNAME RR for name exists
2) RCODE of NOERROR and operation not successful when attempting to add
a CNAME RR when an A RR for name exists

Section 10: special NS rules

1) RCODE of NOERROR and operation successful when attempting to delete the
next to last NS RR for name at zone apex. Tested by adding two NS records
for name and performing an update delete operation using
name,ttl,type,rdata
2) RCODE of NOERROR and operation not successful when attempting to delete
the last NS RR for name at zone apex. Tested by performing an update
delete operation using name,ttl,type,rdata.
3) RCODE of NOERROR and operation successful when attempting to delete the
last NS RR for name not at zone apex. Tested by performing an update
delete operation using name,ttl,type.
4) RCODE of NOERROR and operation successful when attempting to delete the
last NS RR for name not at zone apex. Tested by performing an update
delete operation using name.
5) RCODE of NOERROR and operation not successful when attempting to delete
the last NS RR for name at zone apex. Tested by performing an update
delete operation using name,type.
6) RCODE of NOERROR and operation not successful when attempting to delete
the last NS RR for name at zone apex. Tested by performing an update
delete operation using name.
7) Ensure that attempts to delete a NS RR do not delete the SOA RR as a
side effect. Tested using yxrrset SOA prereq.

Section 11: idempotency tests

1) RCODE of NOERROR and deletion operation successful when testing update
consisting of an add of name,ttl,type,rdata and a delete of name,type.
2) RCODE of NOERROR and addition operation successful when testing update
consisting of a delete of name,ttl,type and an add of name,ttl,type,rdata

December 9, 1999				      page 4

	   46th IETF DNSIND interoperability testing report

Section 12: out-of-zone prerequisites tests

1) RCODE of NOTZONE when prereq is yxrrset and name not in zone
2) RCODE of NOTZONE when adding a name not in the zone

Section 13: glue RR tests

1) RCODE of NOERROR and operation successful when adding a glue record for
existing NS record for name in zone.

Section 14: wildcard character tests

1) Ensure that a wildcard character is treated literally in both an update
and a prereq Section. Tested by:
a) RCODE of NOERROR on update with RR with wildcard character in name
b) RCODE of NOERROR on yxrrset prereq of same name
c) RCODE of NXRRSET on yxrrset prereq of same name with another letter
substituted for wildcard character
d) RCODE of NOERROR on nxrrset prereq of same name with another letter
substituted for wildcard character
e) RCODE of NOERROR on nxdomain prereq of same name with another letter
substituted for wildcard character

Section 15: SOA handling tests

1) Test that the SOA can be replaced with a new SOA with a
serial > oldserial. Tested by updating an existing SOA, and performing
a query for the new SOA and checking its (changed) mname.
2) RCODE of NOERROR and operation not successful when attempting to update
an existing SOA with a new SOA whose serial < existing serial.
3) RCODE of NOERROR and oepration not successful when attempting to update
an existing SOA with a new SOA whose serial == existing serial.

Section 16: Large rdata tests

1) known problems in the Perl package used for testing prevent exercise
of this test.

Section 17: TTL updates

1) Test that an RR can be replaced with a second RR differing only in TTL
value using a single delete/add update operation.

Section 18: Obscuring existing data by zone cuts

1) Test that existing data can be successfully obscured by a zone cut.
Tested by adding two A records and one TXT record below name using update
add and then adding a NS record for name using update add and expected

December 9, 1999				      page 5

	   46th IETF DNSIND interoperability testing report

RCODE of NOERROR.

NOTES ON RESULTS

General)

Policy issues noted below refer to test problems with
server implementations which intentionally disallow SOA and NS
RR dynamic updates as a matter of policy.

When a test section contains more than one test, the result noted
in the result table is an aggregate result for the section. Future
versions of the test used in this report will record results for
each test rather than each section.

1) Issued NOTIFY to secondary when no data changed

2) Server crashed

3) Wrong RCODE

4) Wrong RCODE

5) Got YXRRSET back in both cases

6) Policy issue

7) Policy issue

8) Policy issue

9) Policy issue

10) Got NXDOMAIN

11) Allowed A RR to be created at same name as existing CNAME

12) Expected NXRRSET, got NOERROR

13) Server delays notification and has known problems with IXFR,
which was disabled during testing.

14) Implementation problem

15) Server crashed due to intended exception caused by memory leak in
error handling code entered as a result of an inappropriate FORMERR
response from another server. Problem was fixed during the interop but
the vendor did not complete primary testing.

16) Policy issue

December 9, 1999				      page 6

	   46th IETF DNSIND interoperability testing report

17) Policy issue

18) Policy issue

19) Policy issue

20) Server delays notification and has known problems with IXFR.

21) Implementation problem

SPECIFICATION ISSUES IDENTIFIED DURING TESTING

1) a clarification is needed on behavior wrt update of a SOA RR when the
new SOA RR has a serial which is the same as the existing SOA RR

2) can you specify in a prerequisite a record below a zone cut ?

CONCLUSIONS

Although update of secondaries with small zone changes using IXFR
mechanisms was observed to work between several servers, IXFR was not
thoroughly tested. In general, the group concensus was that 1995 needed
further testing before it could be advanced to DS.

Most servers were observed to correctly issue NOTIFY queries and
responses. One server issued delayed NOTIFY queries as a matter of policy.
Another server issued inappropriate NOTIFY queries. Another server had not
implemented NOTIFY queries or responses. The group concensus was that
1996 was clear, at least two implementations interoperated, and that it
should be advanced to DS.

There was no concensus that 2136 should be advanced to DS at this time.

The group suggests that the specification issues identified during testing
be raised before DNSIND for discussion and resolution.

ACKNOWLEDGEMENTS

Andreas Gustafsson of IEI for providing the perl dynamic update test
script update_test.pl

Microsoft for providing lunch and dinner.

NAI Labs for providing networking equipment and cables.

ISC for making it possible for the coordinator to attend.

7 vendors for finding the time and having the patience to participate.

December 9, 1999				      page 7