Subject: [DNSOP] Re: [secdir] secdir review of draft-ietf-dnsop-reflectors-are-evil-04.txt (fwd)

From: Dean Anderson <[email-masked]>
Date: Tue, 2 Oct 2007 15:43:36 -0400 (EDT)

I think this may be of interest. It was offlist, so I won't identify the 
author I am responding to.

> [off-list]
> 
> On Monday, September 24, 2007 06:25:49 PM -0400 Dean Anderson 
> <[email-masked]> wrote:
> 
> 
> 
> > I.   Harm only possible for ENDSO; Update RFC 2671 Instead
> >
> > The maximum non-EDNS amplification factor is 8
> 
> 8x can be significant.

Yes. But they can get more than that from a couple hundred+ root
servers.

> > II.   Authority Servers not Addressed, but Present More Harm
> > V.   Mitigation Options Limited in Authority Case
> 
> The document I read does cover these issues, though lightly.  It is much 
> more difficult to construct an effective attack using an authoritative 
> server, because one must first find an existing record large enough to be 
> worthwhile.  

Your argument was addressed on the DNSOP list. To conduct the attack
with anything approaching anonymity, one __must__ find authority servers
with legitimately large records.   But once you have this list, you may 
as well use those instead of searching again for reflectors.

It is far easier to search for authority servers than it is to find 
reflectors.

> Furthermore, the real danger here is the ability to mount a 
> _distributed_ attack, in which large numbers of servers send bogus 
> responses at a rate far beyond that which the original authoritative server 
> could manage.  That requires caching servers; a handful of authoritative 
> servers mostly on the same network won't cut it.

Caching servers are not a requirement for a distributed attack. To
conduct the attack with any servers (caching or authority), one still
needs a botnet to send the spoofed source packets.  This botnet is
amplified by the same factor, whatever type of server is used.  Thus, 
the type of server is irrelevant to the damage caused.

> As you note, attacks involving authoritative servers can be considerably 
> more difficult to mitigate, since restricting access to them is a cure 
> worse than the problem.  The document does discuss this issue.

Well, obviously, you didn't get even the gist of the issues _already_
discussed on list for the document.  No one else will get it, either,
given the current document.

> > III.  Motivating Attacks Seem Contrived
> > There have been no further similar attacks.
> 
> There have been many DDOS attacks involving using recursive nameservers as 
> reflectors.  We seem to see them on a fairly regular basis (no, I don't 
> have any data -- I'm not responsible for tracking this stuff; I just notice 
> the load spikes on the nameservers once in a while).

I don't see that data. I don't see any reports outside the original
"urgency--must do this now"  reported on Nanog (2005 I think).  People
have to cooperate to mitigate these attacks. There would be records.  
There would be more complaints about recursors.

> > IV.  Ordinary DDOS Mitigation Appropriate for Reflector Case
> 
> > Because the mitigation options are the same as with any other spoofed
> > DDOS attack, this attack does not merit special attention.
> 
> A large part of defending against various attacks is going after so-called 
> "low-hanging fruit"; that is, taking actions which obtain a large result at 
> a small cost.  This document suggests such an action, which prevents the 
> use of caching nameservers in attacks of the type that has been observed. 
> This is a proactive step; as far as I know, all "ordinary DDOS mitigation" 
> methods are reactive in nature.

This isn't low hanging fruit; no one is seriously abusing this, even
over 20+ years. Interestingly, this proposal comes from the Anycast
crowd.  BTW, a number of the 'attacks' they have previously reported
(excessive TCP syns, TCP unmatched to syns, incomplete UDP fragments)  
are symptomatic of anycast instabilities, and aren't attacks, either.

> It's worth noting that while many distributed attacks can be mitigated by 
> applying broad filters at a point close to the victim, that is particularly 
> difficult in this case.  If you apply a filter that blocks "all incoming 
> DNS responses", then you have denied DNS service to the victim, doing the 
> attacker's job for him.

This mitigation is no different from authority servers. Certainly, more
sophisticated methods are required to scrub packets, such as matching
queries sent to responses.  I think UUnet, for example, does this kind
of thing in DDoS attacks and has some sophisticated tools to scrub
traffic during an attack.  But this is just what I meant that 'ordinary
DDoS mitigation' is appropriate to these attacks.

> > VI.  Using "Evil" in a RFC Title is Unprofessional
> 
> Uh, the title of the document is "Preventing Use of Recursive Nameservers 
> in Reflector Attacks"; I don't see the word "Evil" anywhere there.  It does 
> appear in the draft filename, but that is just the filename of a working 
> document of the IETF, and by convention can be whatever its author wants.

Ok.

> > VII. Reflectors are Useful for Scientic Measurement
> 
> True, and also for various forms of debugging.
> 
> 
> > Because the NSID draft (RFC 5001) makes it impossible to independently
> > identify Anycast Root DNS server instances (because the returned nonce
> > is encrypted and only decryptable by the root operators) and therefore
> > impossible to measure the reliability of Anycast Root DNS services; the
> > open recursor method is the only economical method to measure anycast
> > services.
> 
> Actually, RFC5001 leaves the content of the NSID option entirely up to the 
> server operator; it does not mandate a form which is useless to anyone but 
> the server operator.

Ah, yes. Amazingly trusting.  Suppose SEC rules made it 'optional' for
the Public Company to encrypt its SEC filings so that only the Company
could decrypt them.  Do you suppose the Ebbers and Skillings would allow
you to see that data?

Likewise RFC5001 __enables__ operators to hide their dubious operations
in ways that can't easily be independently tested.  It is of no
consequence that they could 'choose to be honest'. The IETF role is to
ensure measurable proper conduct of, particularly, root and TLD DNS
operators.  

The only reason to hide this information is to obscure data on the
unreliability of DNS Anycast.  Further, hiding DNS information is
contrary to established architectural policy written to clarify DNSSEC
goals.

This ability to hide Anycast operation data advances no legitimate
public interest. It was put in at the last minute by root and TLD 
anycast operators, despite objections.

> Further, I dispute your claim that your inability to distinguish
> servers makes it impossible to measure the reliability of the service.  
> Speaking as one who operates many redundant services, the reliability
> of a service is about the behavior its clients see, not whether
> individual servers providing it are functioning correctly.

This is a specific kind of reliability measure. It is not the only kind.

> For example, if an anycast nameserver address is backed by three
> servers and one of them is down, but the routing is such that only 10%
> of requests go to the down server, then on average, clients will see
> 10% of their requests dropped, not one third.

This is overly simplistic. It is not true for stateful DNS packets,
because packets can always be routed to multiple anycast instances.  
Earlier, I had thought that this could only happen with PPLB, and in
theory (RFC 1812 could happen at anytime. I've since tested this
experimentally. Theory is was right. Even routers that route using flow
caches expire those cache entries every 60 seconds.  Send 2 packets more
than 60 seconds apart, and they can go to different servers. I've
detected anycast open recursors this way.

But to tell how bad the anycast problem is on an authority server (such
as the root or TLD servers), one needs to identify uniquely (with NSID),
the instance each query goes to, and measure how often one gets a
different server.

Alternately, one can indirectly use TCP, or indirectly use reflectors to
make measurements. These measurements can't be as accurate as NSID
measurements would be.

> Now, the ability to distinguish between servers is useful if you are trying 
> to determine the cause of the failures, but is completely unnecessary for 
> determining that the service has 90% reliability.

Incorrect, as shown above.  Besides, we expect high availability from
from root servers.  Anycast appears to give no better than about 97%
over TCP under ideal conditions (from a paper on Nanog by Anycast HTTP
advocate).  It might not be that good, even. A figure of 90% would be
abysmally bad. 3% packet loss is usually unservicable.

		--Dean

-- 
Av8 Internet   Prepared to pay a premium for better service?
www.av8.net         faster, more reliable, better service
617 344 9000   

_______________________________________________
DNSOP mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/dnsop

---

From: Brian Dickson <[email-masked]>
Date: Tue, 02 Oct 2007 17:20:01 -0400

> author I am responding to.
[Did you think to perhaps ask the author first? He/she may have been 
willing to be identified...]
> Yes. But they can get more than that from a couple hundred+ root
You miss the point. Your original comment was 100% wrong. You even admit 
so, but don't seem to realize it.
"Not possible" you wrote.
"Can be" he/she replied.
"Yes, but [...]" you wrote.

The fact that something else is a *bigger* risk, doesn't have any 
bearing on whether the first thing is a risk.

Consider:
Banks have *way* more money than convenience stores. (That fact is even 
common knowledge.)
However, it doesn't seem to have much impact on their respective robbery 
rates.
> the type of server is irrelevant to the damage caused.
In any DDOS attack, the multiplier effect is not unconstrained.
If the devices being used *as* the multipliers by any of the 
participants, are shared among participants,
then the aggregate result can, and likely will, be limited by those devices.
A botnet of 100,000 using a set of 20 servers each on a 1Gbps link, can 
generate at most 20 * 1Gbps of traffic,
no matter how massive the aggregate rate of the botnet is.

At some point, there is even a crossover, where the botnet could do more 
damage than the so-called multiplier.
If the botnet of 100,000 could each send 33 kbps, i.e dial-up speeds, 
that would be 33 * 100,000 kbps, or 33 Gbps.

More than that set of 20 servers could generate at line rate.
> detected anycast open recursors this way.
The vast majority of equipment used on high-capacity links, is neither 
PPLB, nor cache flow switched.
It is pre-computed FIB in hardware (ASIC) switched, via TCAM. Those 
entries do not expire.
They *do* periodically change when route selection to destinations 
change, something that happens
often enough on the aggregate - but not often on individual prefixes.

The PPLB and cache-aging occurs near the edge, on smaller networks. 
There are a very limited number
of locations where this is even theoretically visible, and in most 
cases, solution to this is under operator
control (e.g. deploy software that does CEF/DCEF instead of cached 
switching, and possibly upgrade
obsolete hardware that is unable to do CEF which is now end-of-life.)
> the instance each query goes to, and measure how often one gets a
No, one does not need to identify specific servers. It is sufficient to 
*disambiguate* servers, to separate out
loci in the vernacular. Once disambiguated, what their identity is, is 
not of primary importance.

And, it is not difficult to disambiguate anycast instances, given a 
sufficient number of places from which to observe.
Traceroutes, ttl values on dns responses, round trip times, all can be 
used to help isolate instances from each other.

It would be nearly impossible to distinguish servers *within* an anycast 
instance, e.g. behind any kind of load balancer.

Without anycast, load balancers would in theory also contribute to the 
occasional problem of stale cache removal affecting
stateful DNS traffic. Not just in theory, but in practice. Try doing DNS 
queries from very close by a given authority server.
Do so over tcp, with a connection that lasts more than, for example, 300 
seconds. See if what you observe looks the same
as your alleged "anycast" bogeyman.

[ I invite you to put your reputation on the line. Build a tool that 
does the above, and sends mail as you, to dnsop.
  Run it blindly, i.e. without advance knowledge of the results. View 
the results, the same as us, on the list.
  I double dog dare you.]

And what exactly *is* "the anycast problem"?

You refer to "the anycast problem" here, as if it is common knowledge.
I am not familiar with the actual existence, let along details of, an 
anycast problem.
Yet you refer to it as *the* anycast problem.

Could you elaborate on what it is that you are referring to here?

I think the dnsop folks share interest in fully exploring problems, if 
they exist, and it is the right venue.

If you are quite certain that there is a problem, and you do seem to be, 
can you do a better job of
identifying it than just using the term, "the anycast problem"?

What is the scope?
What is the nature?
How does one detect it?
What are the problems observed?
Where are the boundaries of the problem space?
What is excluded from the problem space?

Don't just answer these off-hand, one at a time. Take the time. Write 
down your ideas in a cohesive,
structured manner. Present them to the list. Provide data backing up 
your anecdotes. Statistics are good.
Especially when the methodology for how they were gathered is included, 
as well as locality, timestamps,
etc., so the data points can be verified and possibly cross-referenced 
against routing activity at the time.

Any trend in behavior of name resolution, which has a relationship to 
routing events, is something that
the folks on dnsop have more than a passing interest in.
> abysmally bad. 3% packet loss is usually unservicable
The 90% in the original authors example was meant to illustrate the 
measurable impact on reliability,
of a specific failure rate of a component, to show that there isn't 
always a linear relationship.

The author never ascribe a specific rate to availability of root 
servers, and certainly not 90%.
Red herring. Let this "90%" sub-thread die, please. Or maybe you didn't 
realize what the OA meant?

Brian

---

From: Brian Dickson <[email-masked]>
Date: Tue, 02 Oct 2007 17:20:01 -0400

Consider:

What is the scope?
What is the nature?
How does one detect it?

Brian

---

From: Dean Anderson <[email-masked]>
Date: Tue, 2 Oct 2007 21:59:33 -0400 (EDT)

On Tue, 2 Oct 2007, Brian Dickson wrote:

> Dean Anderson wrote:
> > I think this may be of interest. It was offlist, so I won't identify
> > the author I am responding to.
> >   

The author is not on DNSOP.

> >>> I.  Harm only possible for ENDSO; Update RFC 2671 Instead The
> >>> maximum non-EDNS amplification factor is 8
> >>>       
> >> 8x can be significant.
> >>     
> You miss the point. Your original comment was 100% wrong. You even
> admit so, but don't seem to realize it. "Not possible" you wrote.

I didn't say 'not possible'. I said 8x.

> "Can be" he/she replied.

They didn't say 'can be' they said 8x can be significant.

> "Yes, but [...]" you wrote.

I said they can easily get more from root servers. The low hanging fruit 
is in the root servers. (authority servers)

> bearing on whether the first thing is a risk.

Yes, it really does.  Especially if the bad guy doesn't have to even 
change his source code to get more bang for his botnet, and doesn't 
incur any more risk.  In fact, using authority servers is _less_ risk to 
the abuser, because to compose the reflector attacks, s/he has to crack 
into a server, craft a record, and search 3.7 million IP addresses for 
a list of reflectors. All of these things leave a forensic trail. Any 
one of which might lead back to the bad guy. 

By contrast, searching authority servers for, say, large SPF records
isn't suspicious. There is no trail besides the botnet, which is
necessary for either case. Authority servers give more bang and less
risk, and same code and payload size. One must presume the attacker a
complete idiot to make the open reflectors attack look attractive.

While it may be the case that some attackers are indeed complete
technical idiots, such idiots only seem to know 'ping -f and other
rootkit naughties.

> Consider: Banks have *way* more money than convenience stores. (That
> fact is even common knowledge.) However, it doesn't seem to have much
> impact on their respective robbery rates.

Banks have correspondingly higher security, and robbing a bank is a
federal crime, and engages the FBI and the Secret Service. Robbing a
convenience store is a matter for the local cops. You example doesn't
fly.  

> More than that set of 20 servers could generate at line rate.

For some inexplicable reason, you assume there are only 20 authority 
servers in the internet.

At great effort, a DNS researcher has compiled a list of about 20000
open recursors by brute force search of 3.7 billion IP addressses.  
Overnight, CAIDA collects about a million or so authority servers by
merely walking in-addr.arpa for NS records.

> PPLB, nor cache flow switched. It is pre-computed FIB in hardware
> (ASIC) switched, via TCAM. Those entries do not expire. They *do*
> periodically change when route selection to destinations change,
> something that happens often enough on the aggregate - but not often
> on individual prefixes.

I'm not sure of this. The last time an operator told me something about
Cisco 12000 architecture and implementation, I later found out it was
wrong. But it appears the behavior of the core is of no consequence.
Closer to the edges, where there are consequences, routers use flow
caches. I think that's why I can detect anycast open recursors.

But the important fact is: I CAN detect anycast open recursors by
sending packets at 60+ second intervals, and don't as easily detect them
at 30 second intervals.

> There are a very limited number of locations where this is even
> theoretically visible, and in most cases, solution to this is under
> operator control (e.g. deploy software that does CEF/DCEF instead of
> cached switching, and possibly upgrade obsolete hardware that is
> unable to do CEF which is now end-of-life.)

This doesn't matter. If the PPLB or flow cache is injecting packets into
different parts of the core with different exits, it doesn't matter
whether the core does load balancing or not. The key feature is the fact
that packets could exit though more than one router.  Unless you are
single homed, you have this 'problem'.

Its not a 'problem' of the router. All routers follow RFC1812 in theory
and in implementation and IPv4 does not have special Anycast host
routes.  That is a feature of IPv6.

> No, one does not need to identify specific servers. It is sufficient
> to *disambiguate* servers, to separate out loci in the vernacular.
> Once disambiguated, what their identity is, is not of primary
> importance.

Well, you can't do that either with NSID encryption. The nonce varies
over the same server from request to request.

(I actually agreed on DNSEXT during NSID discussion to allow hiding of
version or whatnot, so long as the nonce was fixed over a long period of
time---the root/tld folks still wouldn't agree, and pushed their scheme
through)

> sufficient number of places from which to observe. Traceroutes, ttl
> values on dns responses, round trip times, all can be used to help
> isolate instances from each other.

That is indirect information.  Direct information is much better. 

> It would be nearly impossible to distinguish servers *within* an
> anycast instance, e.g. behind any kind of load balancer.

One wonders if that is actually 'anycast'.  The load balancer is really
just a special kind of stateful NAT.

> occasional problem of stale cache removal affecting stateful DNS
> traffic. Not just in theory, but in practice. Try doing DNS queries
> from very close by a given authority server. Do so over tcp, with a
> connection that lasts more than, for example, 300 seconds. See if what
> you observe looks the same as your alleged "anycast" bogeyman.

Stateful NATs and load balancers keep TCP state for an hour+. Otherwise
your SSH sessions would drop.

I haven't argued against using load balancers, or firewalls.

> the results, the same as us, on the list.

I have built a tool. I have run it. And I have detected anycast open 
recursors.

> anycast problem. Yet you refer to it as *the* anycast problem.

I do have to update. It hasn't been updated in a while.
http://www.av8.net/IETF-watch/DNSRootAnycast/History.html

For most people on this list, this is common knowledge.

> > Incorrect, as shown above.  Besides, we expect high availability
> > from from root servers.  Anycast appears to give no better than
> > about 97% over TCP under ideal conditions (from a paper on Nanog by
> > Anycast HTTP advocate).  It might not be that good, even. A figure
> > of 90% would be abysmally bad. 3% packet loss is usually
> > unservicable

> measurable impact on reliability, of a specific failure rate of a
> component, to show that there isn't always a linear relationship.

Yes, I know that 90% was a example. But the 97% was a statistic from a
real (optimistic) paper on HTTP anycast presented by a proponent on
Nanog. 3% loss is unacceptable performance for root and tld nameservers.

		--Dean

---

From: Dean Anderson <[email-masked]>
Date: Tue, 2 Oct 2007 21:59:33 -0400 (EDT)

--Dean

---

From: John Kristoff <[email-masked]>
Date: Tue, 2 Oct 2007 22:08:52 -0500

On Tue, 2 Oct 2007 21:59:33 -0400 (EDT)
Dean Anderson <[email-masked]> wrote:

> In fact, using authority servers is _less_ risk to the abuser, because
> to compose the reflector attacks, s/he has to crack into a server,
> craft a record,

One can create a large record anwhere in the namespace.  There are many
free DNS services available.  If for some reason that won't work,
miscreants can and routinely do, use fraudulent financial credentials
to purchase DNS, hosting, or whatever they need elsewhere.  If that
won't work, were you aware that there are numerous providers who for
one reason or other either cater to miscreants or will tolerate it
to the point that their only response is to simply terminate the
harmful service after a litany of complaints and the damage has been
done?

> and search 3.7 million IP addresses for a list of reflectors.

That is less than a /8.  Piece of cake.  It can be done with hardly
any effort and in almost no time at all.  No?

> All of these things leave a forensic trail.

Not in the real world.  As I've told you before, in practice this
just isn't an issue for a miscreant.  Hardly anyone is logging or
noticing valid, even repeated queries, TXT or otherwise, that land
in their address space.  Do you have a forensic trail of the
queries I sent to your address space and servers?  I can confirm
the timestamps and source addresses offlist if you'd like.

> Any one of which might lead back to the bad guy.

Probably not.  You think the bad guy is running probes from his home
computer?

> open recursors by brute force search of 3.7 billion IP addressses.

That does sound pretty sad.  My experience doesn't echo that at all.
What effort and what researcher are you referring to?

> I have built a tool. I have run it. And I have detected anycast open 

Is the tool, the data, a presentation or paper in a peer-reviewed
journal available?

> Nanog. 3% loss is unacceptable performance for root and tld nameservers.

As far as I know, that was a presentation by a some folks who were
sharing their operational experience with TCP and anycast.  Not a
formal refereed paper.  Do you have a pointer to a paper?

John

---

From: John Kristoff <[email-masked]>
Date: Tue, 2 Oct 2007 22:08:52 -0500

John

---

From: Brian Dickson <[email-masked]>
Date: Wed, 03 Oct 2007 01:07:00 -0400

Dean Anderson wrote:
> The load balancer is really just a special kind of stateful NAT.
>   
No.

Load balancers can load balance, without any translation being done at all.

And a load balancer is by definition doing *anycast*.
The same address is used as a destination, and the packets are delivered 
to multiple hosts.
That is anycast.

Anycast can be either, or both of, global load balancer (using network 
announcements for accomplishing anycast),
or local load balancer (using L2/L4 to serve multiple servers from a 
single apparent host address.)

It's sort of like how what we call a layer-2 "switch", is just an n-port 
"bridge".
> your SSH sessions would drop.
I beg to differ. NATs may keep multi-hour sessions up, however...

The load balancers I am familiar with, and those would be the ones 
relevant to anycast, have default timeout on tcp of 5 minutes.
> I haven't argued against using load balancers, or firewalls.
Well, I guess that makes you an *inconsistent* crackpot.

If you continue to argue against anycast, you should also argue against 
load balancers, application layer gateways, network address translators, 
and stateful firewalls. And perhaps seat belts.
>> And what exactly *is* "the anycast problem"
(BTW - many of the links from  your kitchen sink, return 404 errors. As 
does the "admin" link for av8.net's home page.)
> For most people on this list, this is common knowledge.
After having read your sad story, I can only say: my condolences to them.

Of course, the "history" doesn't seem to do anything other than say, 
repeatedly, "There is a problem with anycast."
(If you read my original request, I was asking what the problem is, not 
what the history is, or what any of the meta-arguments have been about.)

The one minor thing you seem to have raised in, what, several *years* 
(?), is your
RFC 1812 overture.

Which is basically, that unless a root server answers every query it 
gets, it is in violation of the RFC.

Well, guess what - if PPLB occurs, and the properly formed query
never *gets* to the root server, that's the *network's* problem, not a 
DNS problem or RFC violation. QED.

Oh, and it's only a real problem if it happens consistently, for 
requests to *all* authority servers for a given zone.
As long as there's a *single* non-anycast instance, or as long as the 
anycast instances don't *all* have the *same*
 load balancing cost, it's not a big deal.

I'd now like to address the rest of the technical content of the site:

Okay. That was easy.

BTW, your "history" as such, ends with you being banned from *this* list.
How appropriate.

(Apologies to everyone else. Didn't realize I was feeding a troll.)

Brian

---

From: Brian Dickson <[email-masked]>
Date: Wed, 03 Oct 2007 01:07:00 -0400

No.

Okay. That was easy.

Brian

---

From: Sam Hartman <[email-masked]>
Date: Tue, 02 Oct 2007 23:31:43 -0400

>>>>> "Dean" == Dean Anderson <[email-masked]> writes:

    >> The fact that something else is a *bigger* risk, doesn't have
    >> any bearing on whether the first thing is a risk.

    Dean> Yes, it really does.  Especially if the bad guy doesn't have
    Dean> to even change his source code to get more bang for his
    Dean> botnet, and doesn't incur any more risk.  In fact, using
    Dean> authority servers is _less_ risk to the abuser, because to
    Dean> compose the reflector attacks, s/he has to crack into a
    Dean> server, craft a record, and search 3.7 million IP addresses
    Dean> for a list of reflectors. All of these things leave a
    Dean> forensic trail. Any one of which might lead back to the bad
    Dean> guy.

    Dean> By contrast, searching authority servers for, say, large SPF
    Dean> records isn't suspicious. There is no trail besides the
    Dean> botnet, which is necessary for either case. Authority
    Dean> servers give more bang and less risk, and same code and
    Dean> payload size. One must presume the attacker a complete idiot
    Dean> to make the open reflectors attack look attractive.

I have to agree with Dean on the specific point that analyzing related
risks to determine whether one problem is worth fixing is actually
important in doing security work.  Fixing a problem has a cost--both
in implementation and inconvenience.  If other related risks reduce
the value of that fix,then the cost may not be justified.

So, to the extent that Dean is trying to encourage that sort of
analysis here, I think it is very good.

At this time, I make no comment on the rest of his message or on
whether his technical analysis of this issue is in fact correct.

Sam Hartman
Security Area Director

---

From: Sam Hartman <[email-masked]>
Date: Tue, 02 Oct 2007 23:31:43 -0400

Sam Hartman
Security Area Director

---

From: Sam Hartman <[email-masked]>
Date: Tue, 02 Oct 2007 23:31:43 -0400

Sam Hartman
Security Area Director

---

From: Dean Anderson <[email-masked]>
Date: Wed, 3 Oct 2007 12:33:09 -0400 (EDT)

On Wed, 3 Oct 2007, Brian Dickson wrote:

> And a load balancer is by definition doing *anycast*. The same address
> is used as a destination, and the packets are delivered to multiple
> hosts. That is anycast.

No, that isn't anycast. A loadbalancer is actually a stateful NAT with
several different hosts behind the load balancing NAT. Those
loadbalancer devices you buy from cisco and other companies are
specialized NAT boxes.  The servers behind the load balancer have 
all have different ip addresses.  The loadbalancer makes it appear they 
servers share the same address through stateful address translation 
techniques.

By contrast, anycast is a load distributing technique that doesn't
involve any load balancing hardware. There is no box, there is not state
maintenance. One just gives several computers the same IP address and
drops them on the network. Obviously, TCP doesn't work, but stateless
services can work.

> announcements for accomplishing anycast), or local load balancer
> (using L2/L4 to serve multiple servers from a single apparent host
> address.)

You quite obviously don't understand the difference between anycast and
ordinary load balancing devices.

> BTW, your "history" as such, ends with you being banned from *this* list.

I wasn't banned from this list. I was told to shut up about the subject,
and banned from a different list---the list where you complain about
misconduct on this list.  What part of "hardball" don't you get?

Sam Hartman has noted they have been using "hardball" tactics. I guess I
thought you were more aware of this than you are. Let me see if I can
boil it down to a paragraph:

Kessens, then IESG member, said that root server operation was not a
topic of DNSOP.  [See item 1 of the DNSOP charter.]
Kessens threatened to silence me if I pointed out Anycast flaws.
So, I complained to IETF about the improper Kessens threat.
Kessens/Crocker/Carpenter filed PR action, citing my complaint as a 
'continuing a discussion declared off-topic'
PR-action consensus favored no-PR-Action
Carpenter misled the IAB on the consensus

Carpenter, then IETF chair, was responsible for a false report to the
IAB that there was a consensus approving the PR-action. In fact, the
consensus was against the PR-action: 15 out of 17 people who responded
indicated they were against the PR-action.
http://www.av8.net/IETF-watch/IESG/IESG-PR-discussion.html

The IAB was apparently told that there was less than 10 messages on the
IETF list.  Though, even that is still not a consensus! [See section 5.4
of http://www.iab.org/appeals/2006-07-13-anderson-response.html]

It also turns out the PR-action is contrary to the law governing
corporations (the IETF is an activity of the ISOC, a US corporation):  
under the law, without a bylaw on suspension or expulsion, a corporation
cannot suspend or expel a member without a vote of the membership.  The
ISOC has no such bylaw, and no vote of the membership was held. There
are some other conditions that also weren't met. Carpenter, while still
IETF chair, was informed of this, but he apparently has no respect for
the law.  
http://www.av8.net/IETF-watch/Suspension_and_Expulsion_Requirements.pdf
(it wouldn't matter if the IETF weren't part of the ISOC. Essentialy the
same rules apply to unincorporated associations and clubs)

BTW, the ISOC isn't just a regular corporation, its a special kind of
non-profit corporation, that is supposed to benefit the public interest.
So its managers aren't allowed to have conflicts of interests. Its 
kinda-sorta like a government agency, but different.

There were loads of other problems, not the least of which was the fact
that none of the messages that Carpenter/Kessens/Crocker complained
about were in the least way abusive.  As you might guess, money was the
motive behind the false charges, and silencing criticism was a way to
continue to make money with the IETF.

Then GROW considers an Anycast Draft, by your company.  Appeals are
filed with the IESG and IAB. I had our lawyer review these appeals. As
my lawyer commented, 'No one could read this and not get the impression
that something really smells'

Hartman cites "Hardball tactics"
https://datatracker.ietf.org/idtracker/draft-ietf-grow-anycast/comment/57703/?

The Nanog issue was briefly noted. I was silenced there, too, for 
disputing BGP anycast stability, more hardball.

		--Dean

---

From: Dean Anderson <[email-masked]>
Date: Wed, 3 Oct 2007 12:33:09 -0400 (EDT)

--Dean

---

From: Dean Anderson <[email-masked]>
Date: Wed, 3 Oct 2007 13:14:29 -0400 (EDT)

> harmful service after a litany of complaints and the damage has been

These all leave forensic trails.  

> Not in the real world.  

Yes. In the real world. You are merely failing to distinguish the mere
miscreant that doesn't merit investigation with the genuine criminal
that does. You assume that because your miscreants aren't caught, that
it is because it isn't possible to find them.  In fact, it is __your__
powers that are limited, not the powers of the government to find real 
criminals.

> As I've told you before, in practice this just isn't an issue for a
> miscreant.  Hardly anyone is logging or noticing valid, even repeated
> queries, TXT or otherwise, that land in their address space.

Yes, actually people are making such logs. That they don't use those 
logs to track your mere annoyance doesn't mean those logs aren't there.

> Do you have a forensic trail of the queries I sent to your address
> space and servers?  I can confirm the timestamps and source addresses
> offlist if you'd like.

Could be in logs. I don't have any inclination to look. But I have
noticed strange activity from ultradns before.

> Probably not.  You think the bad guy is running probes from his home

Doesn't matter. Perhaps you recall the 'great northeast power failure' a
few years ago. It coincided with a virus release, and briefly, it was
thought the virus was responsible. The virus wasn't responsible, but the
suspicion caused the various LEAs to get the 14 year responsible for the
virus. It took about 3 days to get the kid, which includes the time to
become suspicious of the virus, and then begin to find it.

> What effort and what researcher are you referring to?

And give you a list of 20000 open recursors? I think not.

> Is the tool, the data, a presentation or paper in a peer-reviewed

Not yet. The tool and the data will be published soon.

> > Yes, I know that 90% was a example. But the 97% was a statistic from
> > a real (optimistic) paper on HTTP anycast presented by a proponent
> > on Nanog. 3% loss is unacceptable performance for root and tld
> > nameservers.
> 

A paper presented to a professional organization, as Nanog claims to be,
is indeed a professional paper, subject to professional standards.  
Peer-review is just an assurance that those standards were actually met.  
I don't know for sure whether Nanog does peer-review, though, it seems
to have a committee that does that.

http://www.nanog.org/mtg-0606/pdf/matt.levine.pdf

---

From: Dean Anderson <[email-masked]>
Date: Wed, 3 Oct 2007 13:14:29 -0400 (EDT)

---

From: Joe Abley <[email-masked]>
Date: Wed, 3 Oct 2007 13:36:13 -0400

This is highly tangential to dnsop in general and this draft in  
particular, but perhaps some clarity on the specific load balancing  
point is useful.

>> Load balancers can load balance, without any translation being  

Note "can" above.

> No, that isn't

(always)

> anycast. A loadbalancer is actually

(sometimes)

> a stateful NAT with

(sometimes)

> specialized NAT boxes

(or can be configured that way)

> .  The servers behind the load balancer have
> all have different ip addresses

(but it's often possible to bind a single address to multiple origin  
servers, e.g. as a loopback, and have the host process inbound  
traffic which has not been subject to address translation)

As a data point, f5's BigIP supports load balancing of singly- 
addressed services amongst many origin servers without address  
translation, along the lines parenthesised above. Indeed, in many  
cases this approach is recommended in order to maximise performance.  
I have no recent hands-on experience of other vendors' load  
balancers, but I would be surprised if f5 is unique in this respect.

Just one more tiny, procedural point: when you said

> Then GROW considers an Anycast Draft, by your company.

that's not accurate. That document was co-authored by me when I was  
variously an employee of ISC, a self-employed consultant to various  
people, and an employee of Afilias Canada. The affiliation shown on  
the document, as with all RFCs, is for the purposes of identification  
only. I have never shared an employer with the other co-author.

Also, s/considers/considered/. The document was published some time  
ago as RFC 4786/BCP 126. This is all old news; perhaps it's time to  
move on.

Joe

---

From: Joe Abley <[email-masked]>
Date: Wed, 3 Oct 2007 13:36:13 -0400

Note "can" above.

(always)

(sometimes)

(sometimes)

Joe

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 3 Oct 2007 13:44:59 -0400

> Then GROW considers an Anycast Draft, by your company.  

Just as a point of information, Afilias (in any of its guises --
Afilias Canada, Afilias USA, &c. &c.) has never written any Internet
Draft.  Afilias does employ people who are active in the IETF.  This
is a subtle but, at least to me, important difference.

A

-- 
Andrew Sullivan                         204-4141 Yonge Street
Afilias Canada                        Toronto, Ontario Canada
<[email-masked]>                              M2P 2A8
jabber: [email-masked]                 +1 416 646 3304 x4110

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 3 Oct 2007 13:44:59 -0400

A

---

From: Brian Dickson <[email-masked]>
Date: Wed, 03 Oct 2007 13:47:42 -0400

Dean Anderson wrote:
> specialized NAT boxes.
A load balancer is a device that balances load.
A load balancer *may* be stateful.
A load balancer *may* do L3 address rewriting. It is possible too that
it *may not* rewrite L3 headers.

The address rewriting is orthogonal to the stateful-ness of the device.
And both of *those* are orthogonal to the load balancing function of the
device.

It is possible that a device do stateless load balancing, with L3 rewrite.
It is possible that a device do stateful load balancing with L3 rewrite.
It is possible that a device do stateless load balancing without L3 rewrite.
It is possible that a device do stateful load balancing without L3 rewrite.
> The servers behind the load balancer have 
> all have different ip addresses.
This is not a strict requirement.

There are other methods and kinds of devices that work with servers with
the same IP addresses.
> The loadbalancer makes it appear they 
Address translation is only one of the techniques available.

Others involve activity only on L2 frames.

Those make the choice of which L2 destination to use, from a set of
devices with *identical* L3 config (i.e. which all have the *same* IP
address).

The company that practically *owns* the load-balancing market for
high-end service load balancing, is F5.
F5 make devices that, among other things, can be configured to do
stateful, L4 "fast" switching, which does *not* rewrite L3 headers at all.
> drops them on the network. Obviously, TCP doesn't work, but stateless
Load balancers that do not keep state information, and forward "flows"
(same source IP+port) to consistent destinations, are known to be
harmful to TCP.

It does not matter whether the stateless load balancer has the label
"load balancer", or "router" on the front.
If it makes decisions about where to send packets/frames, where the
choices of destination link for the same L3 destination vary over a
short duration, it is a load balancer.
PPLB *is* a stateless load balancer.
Cache-flow based routers doing load balancing, where there is
significant churn in the cache and premature age-out of entries, is
functionally identical to a load balancer.
May so-called load balancers act in a stateless manner, with or without
NAT on the back end.

If it is load balancing, either it is stateful and TCP-friendly, or it
is stateless and TCP-unfriendly.

Network operators doing PPLB in ways that result in paths being chosen
that are not topologically congruent, are not TCP-friendly.

(Detailed explanation for those who are still listening follows. Feel
free to use it to refute anyone who disagrees with the above assertions,
if you encounter someone who doesn't get it.)

Examples of topologically-congruent paths would be where loads are
balanced across the set of links between the same pair of routers on
both ends. *Only* topologies where this kind of parallel link is the
only source of multiple paths, can divergence *across* the topology be
avoided. Divergence within the topology itself is a local phenomenon,
and expected.

Consider a pair of routers, with three parallel links between them.
A and B are routers and L1, L2, L3 are links.
The paths from A to B are  A-L1-B, A-L2-B, and A-L3-B.
There is a risk of packet reordering, but no risk of divergence in the
tail end of paths to the same destination, in the absence of route churn.
Every path that includes A and any of the parallel links, also includes
B, *after* the parallel link.
If we presume these are the only parallel links in use, then at B, the
per-hop behavior is fully deterministic, even if the behavior between A
and B is only probabilistic.
By induction, then, if we only add links of this parallel nature, then
unless the destination is in some other way non-deterministic, then any
path including the topology with lots
of parallel links, is still deterministic.

When load is balanced across a more complex topology, the cycle parity
(of nodes) is critical. A ring of routers with an even number of nodes
*may* be free of divergence. A ring of routers with an odd number of
nodes, is *very likely* to result in divergence.

Some day I may decide to do a PhD thesis on this subject, but for now,
it is merely a reasonably open "trade secret" - known by not enough
people, practised by fewer, and not something that vendor software will
handle on its own. It's why backbone is an *engineering* discipline.
> ordinary load balancing devices.
 I beg to differ. See the detailed explanation above, and then I suggest
you retract your assertion. I know about as much as anyone on both subjects.

Brian

---

From: Brian Dickson <[email-masked]>
Date: Wed, 03 Oct 2007 13:47:42 -0400

Brian

---

From: bill fumerola <[email-masked]>
Date: Wed, 3 Oct 2007 14:30:54 -0700

> servers share the same address through stateful address translation 

not all load balancers work the same.

direct server return aka one-arm load balancing does no translation or
rewrite of any headers (l3 or l4). all it does is make a switching
decision based on health check and other weighting criteria.

i'll leave the annals of IETF archives as sufficient documentation about
how wrong you are about anycast. as for load balancers, just because
some of them perform NAT in some configurations does not mean that load
balancer = NAT.

-- bill

---

From: bill fumerola <[email-masked]>
Date: Wed, 3 Oct 2007 14:30:54 -0700

-- bill

---

From: Dean Anderson <[email-masked]>
Date: Wed, 3 Oct 2007 20:10:03 -0400 (EDT)

> not all load balancers work the same.

I agree. However, there aren't any load balancer devices that are
implemented with Anycast.  A make and model number, if you know of one.

> decision based on health check and other weighting criteria.

Header rewriting? as in http header rewriting? That isn't anycast 
either.

> how wrong you are about anycast. 

Hey, we agree!  I'm good with that.  They never have to play hardball
with the whistleblowers who were in fact, wrong.

> as for load balancers, just because some of them perform NAT in some
> configurations does not mean that load balancer = NAT.

I agree with that, too. Especially if you broaden the notion to http
load balancing schemes. However, I've never heard of any load balancer
device that uses Anycast, as Brian Hickson asserted.

But none of this is relevant to the claims that Hickson made.

And Hickson's dispute isn't relevant to anything about reflector 
attacks.  You apparently digress.

		--Dean

---

From: Dean Anderson <[email-masked]>
Date: Wed, 3 Oct 2007 20:10:03 -0400 (EDT)

--Dean

---

From: Brian Dickson <[email-masked]>
Date: Thu, 04 Oct 2007 00:10:54 -0400

> decision based on health check and other weighting criteria.
Just to clarify, for those who aren't familiar with the basic idea:

By leaving the IP headers unmodified, the individual servers all expect 
to receive packets
that look like they came directly from the internet (and in fact, did) 
unmodified.

The return packets are thus suitable for being sent straight out without 
needing any rewritee, and
thus without touching the LB.

The F5 BigIP LTM models I've looked at that do that are the 6400 and 
6800 series, running 9.* level code.
There's nothing secret about it - it's a generic, vanilla function they 
ship with. The documentation is on-line.
Google for "l4 fast bigip". (I have no connection with F5 other than 
being employed by a satisfied customer.)

It means that the servers are configured identically, are reachable 
without NAT, and are,
in effect, anycast. The Load Balancer is making a stateful decision 
about which individual server
to send each stream to, in the case of TCP, and stateless in the case of 
UDP.

It operates in exactly the same way, as if there were two equal cost 
routes to two or more routers, each
advertising the existence of one of these servers, on the other side of 
a PPLB router - except that it has
the ability to handle the state issue for TCP.

Anyone who operates a network with PPLB towards *external* routes, via 
BGP multipath, would
have to be an idiot or a fool, and would certainly have trouble 
retaining customers with clue.

Brian

P.S. I do not respond to trolls.
P.P.S. I will not respond the troll.

---

From: Brian Dickson <[email-masked]>
Date: Thu, 04 Oct 2007 00:10:54 -0400

Brian

---

From: Dean Anderson <[email-masked]>
Date: Thu, 4 Oct 2007 15:27:30 -0400 (EDT)

> Just to clarify, for those who aren't familiar with the basic idea:

Perhaps you should leave the clarification until _you_ are more familiar 
with the basic idea.

> The return packets are thus suitable for being sent straight out
> without needing any rewritee, and thus without touching the LB.
> 
> 6800 series, running 9.* level code. There's nothing secret about it -
> it's a generic, vanilla function they ship with. The documentation is
> on-line. Google for "l4 fast bigip". (I have no connection with F5
> other than being employed by a satisfied customer.)

http://www.f5.com/products/big-ip/product-modules/local-traffic-manager.html

> without NAT, and are, in effect, anycast. The Load Balancer is making
> a stateful decision about which individual server to send each stream
> to, in the case of TCP, and stateless in the case of UDP.

Neither of these models use Anycast in their implementation.

Anycast isn't a stateful technique. See RFC 1546.  A load balancer that 
keeps TCP state will work correctly with TCP. But Anycast doesn't keep 
TCP state, nor state for UDP fragmentation. This is why it doesn't work 
for stateful protocols (TCP and UDP)

> routes to two or more routers, each advertising the existence of one
> of these servers, on the other side of a PPLB router - except that it
> has the ability to handle the state issue for TCP.

The F5 LTM box doesn't operate the way you describe. The LTM doesn't
advertise two routes to routers.  There is a nifty flash demo on the F5
website, for those who are still interested.

The LTM isn't anycast. It isn't 'effectively anycast'.  You clearly
don't know what anycast is.

'operates the same way' doesn't mean that it is implemented the same
way.  You suffer from an unfortunately common affliction of junior
operators and administrators: the inability to distinguish a high level
operational effect from actual implementation. This defect is only a
serious problem when the operator assumes (incorrectly) that because
some things have similar high level objectives and uses, that they are
the actually the same.

> BGP multipath, would have to be an idiot or a fool, and would
> certainly have trouble retaining customers with clue.

Really? What do you suppose BGP multipath is meant to do?  Let me give 
you a hint: It installs equal cost routes into the router for 
loadbalancing across peers.

http://www.cisco.com/univercd/cc/td/doc/product/software/ios122/122newft/122t/122t11/ft11bmpl.htm

"The BGP Multipath Load Sharing for eBGP and iBGP feature allows you to 
configure multipath load balancing with both external BGP (eBGP) and 
internal BGP (iBGP) paths in Border Gateway Protocol (BGP) networks that 
are configured to use Multiprotocol Label Switching (MPLS) Virtual 
Private Networks (VPNs). This feature provides improved load balancing 
deployment and service offering capabilities and is useful for 
multi-homed autonomous systems and Provider Edge (PE) routers that 
import both eBGP and iBGP paths from multihomed and stub networks."

Once again, you seem to be uniquely mistaken in your definition of what
would constitute an idiot or a fool. Often, that is the sign of an idiot
or a fool.

> P.P.S. I will not respond the troll.

I will try not to respond to junior system administrators who are without
a clue about the differences between certain network equipment and
certain networking techniques, but are still adamant about their views
despite evidence to the contrary and despite their lack of experience.

---

From: Dean Anderson <[email-masked]>
Date: Thu, 4 Oct 2007 15:27:30 -0400 (EDT)

---

From: bill fumerola <[email-masked]>
Date: Thu, 4 Oct 2007 16:57:11 -0700

> Header rewriting? as in http header rewriting? That isn't anycast

you said a load balancers is "actually a stateful NAT". it's not.

even us 'junior sysadmins' know that "no translation or rewrite of headers
(l3 or l4)" is a reference to changes (or lack thereof) to the IP and/or
TCP,UDP headers.  even the 'junior sysadmins' that we all are know that
if i meant http header rewriting i would have said "L7" not "L3 or L4".

which of course has nothing to do with anycast. it has everything to do
with your incorrect assertion you made that "a load balancer is actually
a stateful NAT".

> > several different hosts behind the load balancing NAT. 

my post (and many others) pointed out, rightfully, that a load balancer
is not a stateful NAT unless configured in such a mode that requires it.
sometimes no translation occurs. it keeps some sort of state, but not
always a NAT table. you're 50% right (or 50% wrong depending on your
perspective).

others have explained that some use NAT in some configuration modes, but
many configurations do nothing resembling NAT.  they've done so more
than adequately. a duck is a bird, but a bird isn't always a duck, and
so on.

as far as "no load balancers use anycast". many do route health injection
which when you add more than one load balancer injecting the same route
then you have.... that thing you claim is a complete failure and can
never work but somehow plenty of installations use it with success when
configured properly within their environment.

> attacks.  You apparently digress.

i just must be a fraud and liar, not to mention a "junior sysadmin".

if that makes you feel superior, feel free to think it.

-- bill

---

From: bill fumerola <[email-masked]>
Date: Thu, 4 Oct 2007 16:57:11 -0700

-- bill

---

From: Dean Anderson <[email-masked]>
Date: Thu, 4 Oct 2007 23:20:59 -0400 (EDT)

> you said a load balancers is "actually a stateful NAT". it's not.

And indeed, I agreed with you.  You are correct: only some load
balancers are stateful NATs. There are indeed other kinds of load
balancers, as you pointed out.  But none of them are implemented with
Anycast.

But by contrast, Dickson said:

   "And a load balancer is by definition doing *anycast*."

which is still incorrect.  By contrast, I admitted to correction in my
incomplete taxonomy of load balancers. Yet Dickson _still_ claims that
certain devices use Anycast in spite of evidence to the contrary.

It seems that you cannot admit to mistakes.

> then you have.... 

BTW, (HSRP) Hot Swap Redundancy Protocol, and (VRRP) Virtual Router
Redundancy Protocol, aren't Anycast either.

But apparently you are defining stateful load balancers as "Anycast" and
then claiming success. That isn't success for Anycast, since it isn't
Anycasting.  But F root, K root, etc are in fact doing Anycasting. They
are not using load balancers like the LTM to provide services.

> that thing you claim is a complete failure and can never work but
> somehow plenty of installations use it with success when configured
> properly within their environment.

I never asserted the term "Complete failure" for Anycast.  I asserted
Anycast was __unreliable__ for stateful services. We don't have to show
"complete failure" to criticize the Root and TLD operations.  We just
have to show their services are unreliable. Indeed, Anycast DNS services
are unreliable: I've shown in theory why Anycast is unreliable for
stateful services, and I've collected data that shows that the real
world matches the theory.  Root and TLD operators and their crony's have
used hardball techniques to try to suppress that information. Next, I'd
like to collect NSID data on root and TLD anycast servers to have direct
data on root and tld operations.

> i just must be a fraud and liar, not to mention a "junior sysadmin".

I try not to mistake simple ignorance for fraud and lies.  But both
indeed deserve correction.  A false statement becomes a lie after you
can no longer claim ignorance that the statement isn't true.

Scientific fraud comes in two types: fabricated data and fabricated
conclusions. If the data wasn't collected as reported, then the reported
data is fabricated. If the collected data doesn't imply the conclusion,
then the conclusion is fabricated.

Fraud, such as civil and criminal sort, has even more elements that are 
necessary for a well-founded accusation. I won't go into those.

I don't expect you to meet scientific standards. However, I do expect
that you can reason and discover simple facts, such as whether the LTM
uses Anycast.  I also expect you to be able to admit mistakes.  I don't 
think that is too much.

		--Dean

---

From: Dean Anderson <[email-masked]>
Date: Thu, 4 Oct 2007 23:20:59 -0400 (EDT)

--Dean

---

From: Dean Anderson <[email-masked]>
Date: Fri, 5 Oct 2007 01:20:01 -0400 (EDT)

> i just must be a fraud and liar, not to mention a "junior sysadmin".

There's nothing wrong with being a junior admin. I was one once, too. I
was a programmer before I was an admin, and I sort of became an admin
because I screwed up. Well, this wasn't my _first_ experience with IT,
but its a good junior IT story:

In 1989 I went to work for the Open Software Foundation in the systems
engineering department. Basically, this meant I fixed bugs in Motif,
then OSF/1, then DCE (It was a very small department, and I introduced
each new technology).  Maintenance programming is usually kind of yuck,
and at the time I had a "impressive" job as an IBM mainframe 'online
programmer' doing TSO user interface programming in PL/1 with ISPF and
DB2; doing design and implementation work. This was a "top of the line"  
dream job for what I trained to do in the mid 80's.  Most mainframe
programming was batch-oriented druggery.  But the OSF job was unique---I
got access to the Internet with a 56K (a big pipe in those
days---through another 56K to Dec, we were an internet backbone backup
for Nearnet). Anyway what really made it worthwhile was that I got
access to Unix SysVr2,r3,r4, AIX, BSD etc source code, which was a
special thing in those days. So skipping foward, I became friends with
IT folks, and I fixed bugs, wrote tests, and assisted OSF member
companies like Dec to get OSF/1 and Motif updates and patches and other
miscellaneous things. Occasionally, I had to make tapes to send to
vendors.

So, one night, I was making a tape for something. It was 9 track tape,
which was made on our Vax. We had no access control to the computer room
in those days.  People made their own tapes and stuff at OSF in those
days. We had operators, but they just did backups. There was a big sign
on the console that said don't type ctrl-e (if I recall), because that
started the monitor, like L1-A on a sparc.  So, while the tape was
running, I was bored, and leaned up against one of the Vax disk drives.
These were the size of refrigerator, about 5 feet tall by 3 feet wide.  
There was a long row of them. Each drive cabinet had 3 buttons, about
1in square each. I accidentally leaned on the power button, and
immediately heard the declining whir of the big platters spinning down,
and my stomach quickly matched the sinking pitch. "Oops". 

I went right to the console, which was unresponsive. "Crap" doesn't
quite do it.  Kids today probably can't appreciate the feeling of total
work stoppage that a multiuser computer caused in those days. There
would be people in Japan and California using that computer. So, I
called a friend of mine, an IT guy, at home (pre-cellphone) and said
"Mark, I did a bad thing", and explained what happened. He said "Dean,
tonight you get to learn how to boot a vax 8550".  After that, I was on
the root password update list, and gradually became sucked into IT and
networking.

The moral is, Its not what you do wrong, its what you do after you screw
up that really makes the difference for IT.

I also have a good IT story about Jeff Schiller and X Window 
from about 1987 or 1988.  

		--Dean

---

From: Dean Anderson <[email-masked]>
Date: Fri, 5 Oct 2007 01:20:01 -0400 (EDT)

--Dean

---

From: Brian Dickson <[email-masked]>
Date: Sat, 06 Oct 2007 11:28:22 -0400

Brian Dickson wrote:
> advertising the existence of one of these servers, on the other side 
> of a PPLB router - except that it has
I should clarify this even further: the above is *completely 
independent* of the "anycast" issue itself.

Consider the following set-up:

A single prefix is announced by a single ASN, for each of which there is 
only one instance. (I.e. non-anycast.)

The prefix is used solely for offering services that are front-ended by 
a stateful load-balancer pair.
There are two LB's for redundancy reasons. The LB's participate in the 
IGP of the ASN, also for redundancy.

The ASN (call it X) has two upstream ASNs, call them A and B.
Each ISP is connected to a separate router, for redundancy.

Inbound traffic from upstream ASN A hits router X-A, and inbound traffic 
from upstream B hits router X-B.

Router X-A prefers LB-A, and router X-B prefers LB-B.

LB-A and LB-B are state-independent. The LB support for stateful traffic 
(e.g. TCP) works only if all incoming packets
for a particular TCP session hit the same LB.

This is *not* anycast to the world. Some *may* consider it to be 
IGP-anycast.

It is an extremely common set-up, perhaps *the* most common 
configuration for load balancers.

Anyone using PPLB between the two AS paths, the one containing A and the 
one contain B, *will*
absolutely have problems using TCP, as in, it won't work except in 
unusual circumstances (one of
the two paths is withdrawn, for instance.)

The real issue is, when any load-balancer is used to affect access to an 
advertised service:
- is the LB breaking that access?
- is the LB operated by the operator of the service?
- does the service have customers or potential customers on the other 
side of the LB?
- does the operator of the LB offer a competing service?

If the answers to all but the second question are "yes", then the LB 
operator may in fact be interfering with someone
else's business. And if they are aware of that interference, it may in 
fact be considered tortuous interference.
Especially if that LB set-up is not considered standard in the industry, 
and even more so if it contracts any
widely-used "best common practice" document for the industry.

Services that use set-ups like "X" above, would include DNS hosting 
companies, web hosting companies, email
hosting companies, content distribution networks, any number of "B2B" 
services, etc.
A lot of those companies are big players, with big legal departments.

I know *I* wouldn't want to interfere with their business...

And if I was doing PPLB, I certainly wouldn't be telling everyone 
operating services that I could potentially
be interfering with about it, on the mailing list that they all 
participate on.

Even if I had been doing PPLB, and *had* told everyone, once I became 
aware of this interference issue,
I would very quickly turn off PPLB, and tell everyone I had done so, and 
that I was wrong, so as not to get
sued into bankruptcy.

IMHO.

Brian

---

From: Brian Dickson <[email-masked]>
Date: Sat, 06 Oct 2007 11:28:22 -0400

IMHO.

Brian

---

From: Dean Anderson <[email-masked]>
Date: Sun, 7 Oct 2007 21:34:03 -0400 (EDT)

On Sat, 6 Oct 2007, Brian Dickson wrote:

> A single prefix is announced by a single ASN, for each of which there
> is only one instance. (I.e. non-anycast.)
> 
> The prefix is used solely for offering services that are front-ended
> by a stateful load-balancer pair. There are two LB's for redundancy
> reasons. The LB's participate in the IGP of the ASN, also for
> redundancy.
> 
> The ASN (call it X) has two upstream ASNs, call them A and B. Each ISP
> is connected to a separate router, for redundancy.

    A   B
     \ /
      X

Ok. 

> Inbound traffic from upstream ASN A hits router X-A, and inbound
> traffic from upstream B hits router X-B.

    Ax     Bx
     \    /
     Xa--Xb

Still with you.

> Router X-A prefers LB-A, and router X-B prefers LB-B.

Not sure what you mean by 'LB' Assuming you mean prefer 'Link B to
destination in A', I'm with you.

That just means that you prefer the link to B. That's pretty common,
too. No problem with that.

> LB-A and LB-B are state-independent. The LB support for stateful
> traffic (e.g. TCP) works only if all incoming packets for a particular
> TCP session hit the same LB.
> 

Since you haven't reused the same IP addresses anywhere in this example,
you don't have bgp anycast. IGP anycast isn't known. There may or may
not be anycast within an AS. You example doesn't show that.

> configuration for load balancers.

I'll grant that one ISP with 2 upstreams is very common. 

However, each AS has its own load balancer with a unique IP address for
each load balancer.  When loadbalancers like the LTM redirect to other
sites, (as the LTM example shows) it uses http redirection. It does not
use Anycast.

> Anyone using PPLB between the two AS paths, the one containing A
> andthe one contain B, *will* absolutely have problems using TCP, as
> in, it won't work except in unusual circumstances (one of the two
> paths is withdrawn, for instance.)

Absolutely wrong.  In _every_ non-anycast case, _all_ packets are
delivered to _unique_ IP addresses, which identify individual, _unique_
hosts.  There is not requirement that all packets take the same path.

The _only_ way that subsequent packets can arrive at the wrong host is
to use Anycast. If you haven't used Anycast, there is no problem no
matter how many different paths are used.

PPLB won't harmed anything in the non-Anycast case. RFC 1812 indicates
that one can load balance packets across multiple paths.  PPLB and the
LTM will work just fine.

RFC 1546 notes that Anycast can only work for stateless protocols.   

But it is apparent that you still don't understand the difference
between Anycast and the methods of load balancing implemented by load
balancers such as the LTM.

		--Dean

---

From: Dean Anderson <[email-masked]>
Date: Sun, 7 Oct 2007 21:34:03 -0400 (EDT)

A   B
     \ /
      X

Ok. 

    Ax     Bx
     \    /
     Xa--Xb

Still with you.

		--Dean

---

From: [email-masked]
Date: Sun, 7 Oct 2007 22:36:52 -0400 (EDT)

> destination in A', I'm with you.

LB means Load Balancer. (You assume wrong.)
LB-A is the LB more closely associated with Xa and ISP A.
LB-B is the LB more closely associated with Xb and ISP B.

Both LB-A and LB-B front-end a set of servers providing service on the
prefix, which for clarity, we will call S.

It does not matter much whether the servers front-ended by LB-A and LB-B
are the same servers, or two distinct sets. The operational behaviour
will be identical.

The diagram looks like:

Ax   Bx
 |    |
Xa---Xb
 |    |
LBa--LBb
  \  /
  B{1..n}  (backend) servers 1 through N

On Xa, the preferred path for S is -> LBa.
On Xb, the preferred path for S is -> LBb.

> not be anycast within an AS. You example doesn't show that.

The prefix S is advertised by both LBa and LBb. The resulting preferred
paths are as described, i.e. each is equal-cost, and the shortest path
to either is used. The closest of LBa or LBb is used.

>> configuration for load balancers.

This statement applies specifically to the detailed description and
diagram above, certainly from the perspective of Xa and Xb, and everything
below it in the diagram. The specifics of A, B, and additional ISPs and
peering above, are variations on the theme, but don't affect any aspect
of the primary functional description of what happens on any two AS paths
that ultimately connect on different Xn routers.

> sites, (as the LTM example shows) it uses http redirection. It does not

In the diagram, there is exactly one ASN with load balancers. That is X.
X includes the AS border routers Xa and Xb. All of LBa, LBb, and B1 to Bn
are inside the AS X.

The load balancers do not have unique IP addresses. They have the same
IP address, call it S1. No other IP addresses from S are in use.

AS A and AS B have the same IP for both load balancers, and will only send
packets to S1.

All packets from A sent to the single IP address S1, get sent to LBa.

All packets from B sent to the single IP address S1, get sent to LBb.

The AS paths are unique. Every instance of S is originated from X, and
there is only one instance of X on the Internet. This is not BGP anycast.

(BTW - the LTM supports lots of other kinds of load balancing than http.
The profile examples for TCP and UDP, as well as L4 Fast, do not do http
redirect, for instance.)

>> and the one contain B, *will* absolutely have problems using TCP, as

Nope. This is an instance where it is not anycast (as you seem to define
it at least). The IP address identifies a service.

There is no requirement that all packets take the same specific hop-by-hop
path.

However, if all packets from a tcp stream do not take the same *as*-path,
then all bets are off.

And in this instance, if the packets are distributed between the two
AS paths ending in "_A_X" and "_B_X", tcp flows will break, absolutely.

> matter how many different paths are used.

This example contradicts this assertion, and does not do (bgp) anycast.

> that one can load balance packets across multiple paths.  PPLB and the

You should read the above. The non-anycast example above shows that
PPLB will not work in this case, regardless of make/model of LB, and
regardless of the mechanism used by the LB (be it an LTM or other).

BTW, RFC 1812 says only that "load-splitting" may work in some instances,
and may not be suitable for others. It predates BGP anycast, and lists
"load-splitting" in the section of future development that should be
watched. The only subsection of route selection that further references
load-splitting is "IGP" route selection, where equal cost paths can
be kept for load splitting.

Extrapolating this to mean that RFC 1812 explicitly supports PPLB,
is reading way to much into the document.

It makes the general observation that it might work. Your experience,
and the above example, serve as useful counter-examples.

Here is another example, in case you are interested in example cases
where PPLB via the same ASN example (A and B upstreams for X), can break
TCP:

If the upstream routers are both connected by Ethernet, and the server
pool used for providing load-balanced services is directly attached to
a single switch, which hashes the choice of server based on MAC addresses
plus IP addresses and ports, then the PPLB will break TCP.

Path via A will have, at layer 2, MAC_Xa + IP foo + port bar
Path via B will have, at layer 2, MAC_Xb + IP foo + port bar

These will, as a general rule, hash differently, and be sent to different
servers, if there is more than one server.

No anycast is needed at all.

AS-path PPLB is just an utterly unreliable way for delivering TCP-based
services.

(The irony is, that TCP becomes unreliable, but UDP is reliable, when
PPLB across AS paths is used - considering tcp is meant to be reliable,
and udp basically unreliable.)

Lots of other PPLB uses are still perfectly valid, and will be compatible
with the above examples - just as they would with bgp anycast.

Brian

---

From: [email-masked]
Date: Sun, 7 Oct 2007 22:36:52 -0400 (EDT)

The diagram looks like:

Ax   Bx
 |    |
Xa---Xb
 |    |
LBa--LBb
  \  /

Brian

---

From: Shane Kerr <[email-masked]>
Date: Mon, 08 Oct 2007 10:43:02 +0200

Brian,

[email-masked] wrote:

[ 194 lines of discussion about routing and anycasting, without a single
  reference to DNS reflectors, or indeed DNS at all! ]

This is all very fascinating, but it seems to have drifted quite far from the
draft in the subject line. In fact, the e-mail I'm replying to is itself more
than half the length of the draft.

Anyway, just to add some hopefully on-topic content, let me say:

    I support the draft as is.

- --
Shane

---

From: Shane Kerr <[email-masked]>
Date: Mon, 08 Oct 2007 10:43:02 +0200

Brian,

- --
Shane

---

From: Dean Anderson <[email-masked]>
Date: Mon, 8 Oct 2007 11:33:43 -0400 (EDT)

> IP address, call it S1. No other IP addresses from S are in use.

The load balancers in this picture are not Anycast. I think you
misunderstand both Anycast and HSRP/VRRP. The second one takes over IP
address S1 when the primary fails.

> All packets from B sent to the single IP address S1, get sent to LBb.

Sorry, this isn't what happens.  LBa (the primary) acts and exchanges
HSRP or VRRP with LBb (the failover). If LBa fails, LBb takes over.  
There is no Anycast.  There is no 'effective' Anycast.  You just
misunderstand the concepts.

> it at least). The IP address identifies a service.

I define it as RFC 1546 defines it.  Indeed, your description above is
not Anycast. But it remains true that:

In _every_ non-anycast case, _all_ packets are delivered to _unique_ IP
addresses, which identify individual, _unique_hosts.  There is not
requirement that all packets take the same path.

> However, if all packets from a tcp stream do not take the same *as*-path,

There is no requirement that they take the same AS path, either.  BGP is
just a protocol to exchange routes. BGP does not do the routing. So
where the routing is done, there is no knowledge of AS paths. You can 
figure this out from RFC1812.

So, all 'bets' are still on. And they remain on for _Stateless_ Anycast.  
They are only broken by stateFUL anycast, as RFC1546 actually notes
explicitly.

		--Dean

---

From: Dean Anderson <[email-masked]>
Date: Mon, 8 Oct 2007 11:33:43 -0400 (EDT)

--Dean

---

From: [email-masked]
Date: Mon, 8 Oct 2007 12:24:33 -0400 (EDT)

> address S1 when the primary fails.

The diagram above was created by me. I said both LBa and LBb have the
same IP address, and that the two LB's are independent.

This means that they operate *without knowledge* of each other's state,
and only inter-operate at the routing level - both announce S into their
IGP (e.g. OSPF). BGP announces the prefix, once only, and only from the
ASN in question (X).

If you want to change the diagram, and *then* say the diagram is wrong,
that's classic "straw man" crap. *Your* diagram is wrong, but that's
irrelevant - it has no relationship to my diagram.

Whatever you have to say about any diagram where the LB's are aware of
each other, and are configured in primary/standby, is totally irrelevant.

If you want to say anything about the actual scenario as described,

>> All packets from B sent to the single IP address S1, get sent to LBb.

It's my scenario, my diagram. You might not believe that it is possible
for two devices to be configured independently, and still function.
It doesn't mean that, *given* the configuration described, that the
rest follows from that.
If you want to discuss the above, you must first do so within the context
of presuming the preconditions (the diagram as described) are true.
Doing so does not necessarily require you to believe it is true, or that
it is possible - but the two things (the precondition and its consequences)
are independent, and should only be discussed independently.

The purpose of creating a particular scenario is to discuss the merits
and behavior of the system as described.

Changing the scenario invalidates everything you say about the scenario.

> misunderstand the concepts.

While it is true that *one* possible way to configure LB's is HSRP/VRRP.
It isn't the *only* way, and it isn't how they are configured in the
scenario above.

> requirement that all packets take the same path.

IP addresses do not always identify unique hosts.
Counter-examples include hosts whose IP address is assigned in a non-static
method, e.g. DHCP, or via ephemeral services such as NAT. IPv4 to IPv6
gateways, where there is 1:1 NAT, allow an IPv6 host to be reached, but
do not, as such, identify the IPv6 host per se.

RFC 1546 is informational only, experimental, and 14 years old.
It describes hypothetical methods of handling the state requirements
of TCP.

In the intervening time, vendors have provided state capturing mechanisms
to provide this support, and the Load Balancers in question are in fact
both doing anycast, *and* preserving state so TCP flows go to the same
servers behind the Load Balancers.

That a second, third-party-ISP-centric load balancer, doing PPLB across
BGP paths, breaks the TCP access to such a BGP-unicast prefix, is
the only consideration of relevance, and is under the control of the ISP.

Caveat emptor.

Brian

---

From: [email-masked]
Date: Mon, 8 Oct 2007 12:24:33 -0400 (EDT)

Caveat emptor.

Brian

---

From: Dean Anderson <[email-masked]>
Date: Mon, 8 Oct 2007 17:12:44 -0400 (EDT)

> same IP address, and that the two LB's are independent.

Well, that's probably where you are wrong. The LBs probably aren't
independent.  At least, they aren't _normally_ independent. They use a
health monitoring protocol to make sure that only one is doing the work
and the other is a hot standby. When the health monitoring indicates a
failure, the standby takes over the primary IP address. The load
balancers would normally share a lan on the router side, and a lan on
the backend server side.

> IGP (e.g. OSPF). BGP announces the prefix, once only, and only from the

I suppose it is possible to turn off the HSRP/VRRP/etc health monitoring
protocols and actually Anycast Load Balancers, but that would be quite
unusual. It would suffer the usual problems with Anycast:

An ISP Z that used BGP multipath with A and B would send subsequent tcp
packets to LBa and LBb. Only one of the LBs received the syn packet, and
the other one would send a reset back, closing the connection.

    Z
   / \
 Ax   Bx
  |    |
 Xa---Xb
  |    |
 LBa--LBb
   \  /

I'm not aware of any vendors that would recommend your scheme.  
Certainly F5 doesn't.

While I've seen stranger things, my hunch is that you didn't set it up
and that it isn't actually Anycast, but rather instead that you just
don't know the difference between pictures that are similar.

BTW, It is _possible_ for a vendor make a state maintanence protocol
that keeps the state of each tcp connection in sync on both systems. In
fact, linux clustering technology does this. However, this isn't Anycast
either, but is a single cluster computer assembled out of generic
computers with a special operating system that makes them work as a
single computer system.  I don't know of any loadbalancer vendor that
does this. The F5 LTM doesn't do this.

		--Dean

		--Dean

---

From: Dean Anderson <[email-masked]>
Date: Mon, 8 Oct 2007 17:12:44 -0400 (EDT)

Z
   / \
 Ax   Bx
  |    |
 Xa---Xb
  |    |
 LBa--LBb
   \  /

		--Dean

		--Dean