Subject: Transitioning IETF DNS services

From: Ray Pelletier <[email-masked]>
Date: Wed, 12 Dec 2007 16:32:30 -0500

It is time to begin the process of transitioning IETF IT services
from Neustar Secretariat Services (NSS) to Association Management
Solutions (AMS).  The complete process will take place over
approximately 7 weeks, beginning Friday, 14 December 2007, and
continuing through Thursday, 31 January 2008.

The first step in the process is to move DNS services.  The actual
transition will take place on Friday, 14 December 2007, at 12pm
Eastern (1700 UTC), approximately 44 hours from now.  At that time
the NS records for the following domains:

    iab.org
    iesg.org
    ietf.org
    irtf.org

Will all be changed to the following:

    NS4.AMSL.COM    64.170.98.30
    NS5.AMSL.COM    64.170.98.31
    NS6.AMSL.COM    2001:5c0:9758::1:1

Please note that IPv6 DNS services will initially be provided by an
IPv6 tunnel.  AMS is adding an additional OC-1 fiberoptic circuit to
serve the IETF, which will include native IPv6 services.  When that
installation completes, which is expected in January 2008, all IPv6
services will be available natively from AMS.

While we wait for AMS' dedicated circuit, only DNS services will be
made available through this IPv6 tunnel.  This may affect email and
mailing list services, which are expected to transition in early
January 2008.  If the circuit installation is not complete before we
transition email and mailing lists then those services may
temporarily be unavailable on IPv6.  We will keep you informed.

We understand that this is an inconvenience for part of the user
community but, unfortunately, it is unavoidable.  AMS will make
native IPv6 available as soon as they can.  Note that the DNS will
continue to return IPv6 addresses when queried for those services
that have not yet transitioned (for example, the web and ftp which
transition last).

On the upside, all IT services will have IPv6 addresses when the
transition is complete.  Until then, those services that currently
have IPv6 addresses will lose them as they transition.

Finally, some of you will notice that those two IPv4 nameservers are
both on the same subnet.  This is a transitional situation.  AMS is
building a mirrored architecture in their primary facility.  When
the transition from NSS completes on 31 January 2008, they will
begin the process of moving the mirrors to their secondary facility
in Orlando, FL.  This transition is expected to be transparent to
the community and will take place over approximately 30 days.  There
will be more to say about this when that transition begins.

Ray Pelletier
IETF Administrative Director

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: "Mr. James W. Laferriere" <[email-masked]>
Date: Wed, 12 Dec 2007 19:53:25 -0900 (AKST)

Hello Ray ,

>   NS6.AMSL.COM    2001:5c0:9758::1:1
 	Fmi ,  Are the preceding name servers physically diverse ?
 		Tia ,  Jiml

...snip...
-- 
+------------------------------------------------------------------+
| James   W.   Laferriere | System    Techniques | Give me VMS     |
| Network&System Engineer | 2133    McCullam Ave |  Give me Linux  |
| [email-masked] | Fairbanks, AK. 99701 |   only  on  AXP |
+------------------------------------------------------------------+

---

From: Mark Andrews <[email-masked]>
Date: Thu, 13 Dec 2007 16:08:11 +1100

>  	Fmi ,  Are the preceding name servers physically diverse ?

	There has to be common failure points with NS4.AMSL.COM
	and NS5.AMSL.COM.  Think route announcements.

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Brian Dickson <[email-masked]>
Date: Thu, 13 Dec 2007 00:50:35 -0500

> 	and NS5.AMSL.COM.  Think route announcements.
Well, yes, of course.

The question that is more critical is, are there any elements which 
aren't common failure points?

If the two IPs are on one physical server, there's very little that 
could fail without taking down both instances.
(E.g. single disk failure, memory failure, NIC, crash/reboot, etc.)

If the two IPs were on different subnets, there would be an expectation 
of physical diversity.
However, the same subnet makes this less certain, thus the OP's question.

Brian

---

From: Mark Andrews <[email-masked]>
Date: Thu, 13 Dec 2007 17:11:47 +1100

> aren't common failure points?

	For them to meet physical diversity requirement there would
	been to be /32's in the IRP.

	I suspect you will find that they are just two boxes on the
	same switch.

> However, the same subnet makes this less certain, thus the OP's question.

	You need both physical (power, hardware, location) and
	operational (different global prefixes, preferably different
	AS's) diversity for reliable DNS.

	It's clear from the infomation above that the later is not
	being met.

	Mark

---

From: Dave Crocker <[email-masked]>
Date: Wed, 12 Dec 2007 22:22:16 -0800

> aren't common failure points?

There are two views that make sense:

1. The servers should be on different tectonic plates (and power sources) and 
probably different continents; or

2. If the machines the DNS entries point to are, themselves, a single point of 
failure, it's not clear how much difference it really makes to have the DNS 
servers distributed.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Bill Manning <[email-masked]>
Date: Wed, 12 Dec 2007 22:28:57 -0800

> 	It's clear from the infomation above that the later is not

	it -could- be that these are two XEN images on the ssmae
	box.   HOWEVER. the point remains taht Ray explicitly
	told us that this was a transitional attribute that
	would be corrected  before the transition was complete.
	
	so take a breath, go out and cut the grass (or burn it off)
	and try not to micro-manage the transition... this is
	one of those cases where operational expedience and a 
	desire to let folks know what is happening seems to have caused
	an over reaction.

	as usual, YMMV

-- 
--bill

Opinions expressed may not even be mine by the time you read them, and
certainly don't reflect those of any other entity (legal or otherwise).

---

From: Mark Andrews <[email-masked]>
Date: Thu, 13 Dec 2007 18:15:17 +1100

> 	desire to let folks know what is happening seems to have caused

	I hope they correctly manage the transition.  By that I
	mean that all nameservers (both old and new) serve the
	*same* zone content during the transition.  Otherwise
	there will be servers that will stay locked on the
	old nameservers until they cease to serve the zone.

	Mark

---

From: Russ Housley <[email-masked]>
Date: Thu, 13 Dec 2007 10:48:08 -0500

> 	AS's) diversity for reliable DNS.

We knew about this problem, but choose to make the announcement 
anyway.  We are in the process of working out a secondary.  I got a 
verbal agreement last night, and I expect that it will all be worked 
out by Friday.  We just did not want to hold up the announcement 
while this was sorted out.

Russ

---

From: "Daniel Brown" <[email-masked]>
Date: Thu, 13 Dec 2007 10:54:30 -0500

> while this was sorted out.

    I'm guilty of posting without reading everything right now (since
I deleted the thread up to now), but if it's a simple matter of
physical DNS diversity, I'm certainly willing to offer the aid of
several of my servers (in various global locations, which just sounds
cool to say) to answer DNS requests during the transition.  If I
missed out on something bigger, feel free to ignore and/or scold me
for speaking out of turn.

-- 
Daniel P. Brown
[office] (570-) 587-7080 Ext. 272
[mobile] (570-) 766-8107

If at first you don't succeed, stick to what you know best so that you
can make enough money to pay someone else to do it for you.

---

From: John C Klensin <[email-masked]>
Date: Thu, 13 Dec 2007 11:25:27 -0500

I've changed the subject line because, if this turns into a
discussion, it will be a different one...

--On Wednesday, 12 December, 2007 22:22 -0800 Dave Crocker
<[email-masked]> wrote:

>...
> 2. If the machines the DNS entries point to are, themselves, a
> single point of failure, it's not clear how much difference it
> really makes to have the DNS servers distributed.

AFAIK, this is part of a long-standing discussion in the IETF
and the broader community.  One view is that one I think you are
stating above, i.e., if the services cannot be reached and used,
then it doesn't make any difference whether the DNS records are
accessible or not.

The other is that knowing that the hosts exist (i.e., that one
did not commit a typing or memory error) is useful information
even if the services cannot be reached is useful information and
that the original DNS recommendations about no fate sharing
remain reasonable and valid.  If anything, that recommendation
becomes more important in these days in which various entities
feel free to snag failed DNS queries and divert the responses to
point to places of their choosing. 

IMO, it would be really helpful if relevant WGs or other groups
concerned with DNS operations and configurations would take the
question up again, review it, and make some sort of definitive
contemporary statement (even if that were only "it depends" with
an explanation of the tradeoffs).

    john

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 13 Dec 2007 17:35:17 +0100

On Thu, Dec 13, 2007 at 11:25:27AM -0500,
 John C Klensin <[email-masked]> wrote 
 a message of 40 lines which said:

> contemporary statement (even if that were only "it depends" with an
> explanation of the tradeoffs).

An important reason why we SHOULD have DNS reliability (including
geographical variety) even if service X is down is because there are
several services, not just the Web. OK, if the Web site is down, it is
not *too* important (but it is important, for the reasons you mention)
if I cannot resolve www.ietf.org but email, jabber, etc, continue to
live and need the DNS.

So, I believe that the future DNS service of the IETF, announced on
ietf-announce is badly designed. As we say in french "The shoemaker
always has the worst shoes".

---

From: John C Klensin <[email-masked]>
Date: Thu, 13 Dec 2007 13:44:02 -0500

--On Thursday, 13 December, 2007 17:35 +0100 Stephane Bortzmeyer
<[email-masked]> wrote:

>> IMO, it would be really helpful if relevant WGs or other
>> groups concerned with DNS operations and configurations would
>> take the question up again, review it, and make some sort of
>> definitive contemporary statement (even if that were only "it
>> depends" with an explanation of the tradeoffs).
> 
> An important reason why we SHOULD have DNS reliability
> (including geographical variety) even if service X is down is
> because there are several services, not just the Web. OK, if
> the Web site is down, it is not *too* important (but it is
> important, for the reasons you mention) if I cannot resolve
> www.ietf.org but email, jabber, etc, continue to live and need
> the DNS.

This is consistent with my instincts (and has been for many
years), but Dave is not the first person to take the other
position.

> So, I believe that the future DNS service of the IETF,
> announced on ietf-announce is badly designed. As we say in
> french "The shoemaker always has the worst shoes".

Let's drop this.  Russ has indicated that an arrangement for one
or more secondaries that don't fate-share is in the process of
being finalized and I've seen off-list correspondence from Ray
that indicates the same thing.  I'm also aware of other
well-distributed parties who have volunteered if additional
distribution is needed (and so, presumably, is Ray by now).  My
understanding is that there will be no period in which all of
the IETF's DNS servers fate-share.   I hope, as others have
suggested, that the transition will be handled carefully and in
the right order of events, but I don't have any reason to
believe that it will not be (and assume that Ray and others will
ask for help from experts if they need it).  That moves the
"future DNS service..." issue into "transitory learning pains of
the new secretariat" rather than a [dead] horse we need to kick
further.

    john

---

From: Dave Crocker <[email-masked]>
Date: Thu, 13 Dec 2007 12:11:29 -0800

John C Klensin wrote:
>> because there are several services, not just the Web. 
...
> years), but Dave is not the first person to take the other

Well, I was less trying to take a position than to list what I thought were 
reasonable choices.  Unfortunately I had a nagging feeling that I was missing 
an issue and, in fact, it was the one Stephane raised, which translates into: 
  DNS entries need higher reliability than the services they point to.

As such, I would say that the goal of broad-spectrum -- essentially all of the 
operational parameters -- independence for the two servers is universal.

And yeah, this is a topic unto itself; it's relevance to IETF operations is OBE.

d/
-- 

   Dave Crocker

---

From: Mark Andrews <[email-masked]>
Date: Fri, 14 Dec 2007 11:54:02 +1100

> years), but Dave is not the first person to take the other

And it is a essentially very bad position to take.

The DNS was designed to handle many more "connections" per second
than any TCP based server can handle but to achieve that it needs
to get it needs to be able to find a working server for every zone
it is querying.  If it doesn't the list of outstand recursive queries
quickly becomes a resource issue on the caching server as the amount
of time connection state needs to be remembered goes from sub second
to multi second.

If everyone took the attitude that if the services pointed to by
the DNS are offline then the DNS can be offline then the DNS just
would not work.  Luckily most don't or we would have a "tragesty
of the commons" here.

DNS servers are also often a shared shared resource for thousands
if not hundreds of thousands of clients machines.  This means that
lookup failures don't just affect the user making the lookup.  They
are cumulative and affect all the users of the shared resource.

We see these effects with spam runs which use host names which point
to lame / non-existant servers so this is not just idle speculation.

If the DNS lookup succeeds then the impact of the failure moves
away from the shared resource and back towards individual resources
(browers) or to shared resources which don't have the volume of
lookups (MTAs) and have different failure recovery strategies (store
to disk rather than in memory of a caching server).

Mark

---

From: Ray Pelletier <[email-masked]>
Date: Thu, 13 Dec 2007 21:36:24 -0500

Afilias has agreed to provide DNS services to the IETF.  What this
means is all DNS servers will be dual-homed on both IPv4 and IPv6,
and servers will be available in Asia, North America, and Europe as
follows:

  ns1.ams1.afilias-nst.info  (Amsterdam)
    199.19.48.151
    199.19.48.171
    2001:500:6:300::151
    2001:500:6:300::171

  ns1.yyz1.afilias-nst.info  (Toronto)
    199.19.49.151
    199.19.49.171
    2001:500:7:300::151
    2001:500:7:300::171

  ns1.sea1.afilias-nst.info  (Seattle)
    199.19.50.151
    199.19.50.171
    2001:500:8:300::151
    2001:500:8:300::171

  ns1.hkg1.afilias-nst.info  (Hong Kong)
    199.19.51.151
    199.19.51.171
    2001:500:9:300::151
    2001:500:9:300::171

  ns1.mia1.afilias-nst.info  (Miami)
    199.19.52.151
    199.19.52.171
    2001:500:a:300::151
    2001:500:a:300::171

Thank you Afilias!

Everything else in the previous announcement is still valid and
included here (with some editorial changes) for completeness.

The first step in the process is to move DNS services.  The actual
transition will take place on Friday, 14 December 2007, at 12pm
Eastern (1700 UTC), approximately 48 hours from now.  At that time
the NS records for the following domains:

    iab.org
    iesg.org
    ietf.org
    irtf.org

Will all be changed to the servers indicated above.

Please note that while the transition is in progress, some services
(not DNS) may lose their IPv6 connectivity temporarily.  It will
return when the transition is complete.

AMS is adding an additional OC-1 fiberoptic circuit to serve the
IETF, which will include native IPv6 services.  When that
installation completes, which is expected in January 2008, all IPv6
services will be available natively from AMS.

While we wait for AMS' dedicated circuit, services available on IPv6
now may not be available on IPv6 after they are transitioned.  This
may affect email and mailing list services, which are expected to
transition in early January 2008.  If the circuit installation is
not complete before we transition email and mailing lists then those
services may temporarily be unavailable on IPv6.  We will keep you
informed.

We understand that this is an inconvenience for part of the user
community but, unfortunately, it is unavoidable.  AMS will make
native IPv6 available as soon as they can.  Note that until a
service is transitioned (for example, the web and ftp which
transition last), it will continue to be available on IPv6 at NSS.

---

From: Ray Pelletier <[email-masked]>
Date: Fri, 14 Dec 2007 07:27:04 -0500

Well, the good news is that Afilias has agreed to provide DNS services 
to the IETF.  The bad news is I gave out incorrect addresses.  Here are 
the correct addresses for the nameservers:

    199.19.48.79
    2001:500:6::79

    199.19.49.79
    2001:500:7::79

    199.19.49.79
    2001:500:8::79

    199.19.51.79
    2001:500:9::79

    199.19.52.79
    2001:500:a::79

These addresses can also be found in the DNS

Once again, thank you Afilias!

Ray

Ray Pelletier wrote:

> <snip>

---

From: Ray Pelletier <[email-masked]>
Date: Fri, 14 Dec 2007 08:14:35 -0500

<sigh>  Typo

Make it 199.19.50.79, instead of 199.19.49.79, above

Ray

Ray Pelletier wrote:

> to the IETF.  The bad news is I gave out incorrect addresses.  Here 
> are the correct addresses for the nameservers:
>