Subject: [Rserpool] Requirements draft

From: Tuexen Michael <[email-masked]>
Date: Wed, 4 Apr 2001 17:31:14 +0200

Dear all,

we have now a new version of a requirements draft which
only contains requirements.

I would like to focus the discussion on this list to that
draft to proceed to WG last call as soon as possible.

So please send comments regarding that draft to the list so
that we can discuss them.

Hope to hear soon from you

Michael

Michael Tuexen    Tel.:   +49 89 722 47210
Siemens AG        Fax:    +49 89 722 48212
ICN WN CS SE 51   E-mail: [email-masked]

_______________________________________________
rserpool mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/rserpool

---

From: Eliot Lear <[email-masked]>
Date: Wed, 04 Apr 2001 18:59:24 -0700

Hi,

The requirements document must clearly state what RSERPOOL wants that DNS
(or other existing mechanisms) don't offer.  An example here would be
exceedingly useful.  In section 1.1 you mention two requirements which
quite possibly could indicate a need for another mechanism: support for
real time applications and high availability.  However, you don't go far
enough for people understand the requirement.

For instance (and you can use these words or derivatives if you find them
useful):

We require a name space that reflects pool membership changes to the
client application as rapidly as they occur, rather than waiting for the
client application to reconnect.

Here's another way of looking at it:

What if I were to deploy a DNS zone with TTL 1 (second) such that pool
members would add themselves through a backend process that updates the
zone and causes a NOTIFY/IXFR?  Why is that not acceptable?

Also, in Section 2.5, the first paragraph scopes the problem, but perhaps
beyond the point where you can really answer my above question.

Thanks,
--
Eliot Lear
[email-masked]

---

From: "Bernard Aboba" <[email-masked]>
Date: Thu, 5 Apr 2001 07:27:38 -0700

I agree with Elliot. The case against DNS is not made clearly
and convincingly in the current document. 

To my mind, the issue is NOT registration/de-registration. That
is done today with NOTIFY/IXFR. Fundamentally, high updates and
replication are incompatible -- but this is not a DNS problem
per se. 

Vague appeals against "overloading saddle bags on an old horse"
don't cut it either -- DNS SRV is a proposed standard, and thus
the burden of proof is on those who think we should use something
else to prove why DNS SRV isn't good enough.

The fundamental issue to my mind is the architecture of host
resolvers. These are stub resolvers -- which means that they
require their local DNS servers to do recursion for them. 

In turn, this implies that setting TTL low or 0 will dramatically
increase the load not only on the authoritative DNS servers -- 
but also on these third party servers. 

A secondary effect of this is that the authoritative DNS will
not know the IP address of the DNS client -- only the IP
address of the local DNS. This affects the ability to do
global load balancing correctly. 

There is no way to get around these issues unless you are ready to
put full resolvers on *all* hosts. Just putting them on newer
hosts wouldn't be enough because then the issues would still
exist for all the legacy systems, which will form the bulk
of the host population for years to come. 

To my mind, using ENRP solves this problem because clients
contact ENRP servers directly once they have located them via
DNS, not via a third party server. The ENRP server itself need
not set TTL low in its SRV record. So there is no third party
dependence there.

Since the client contacts the ENRP server directly, the ENRP server
knows the real IP address of the client. Since there is no third
party involved, the caching TTL can be set as low as desired (even
zero). That will increase load on the ENRP server, but not on
anybody else.

There are of course added benefits to this -- you can design a 
protocol uniquely fitted to the problem, yada yada. But I think
we first need to make a convincing argument to get over the DNS
hurdle, which is very substantial -- after all, we are effectively
saying that we won't be able to provide these capabilities to
legacy hosts, which *dramatically* reduces the applicability of
this technique. 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of Eliot Lear
Sent: Wednesday, April 04, 2001 6:59 PM
To: Tuexen Michael
Cc: [email-masked]
Subject: Re: [Rserpool] Requirements draft

Hi,

Thanks,
--
Eliot Lear
[email-masked]

---

From: Neil Deason <[email-masked]>
Date: Thu, 05 Apr 2001 17:11:46 +0100

Comments below...

> legacy hosts, which *dramatically* reduces the applicability of

RFC 2782 itself points out some of the limitations of using 
DNS SRV for load balancing between servers ... 

"Weight is only intended for static, not dynamic, server 
selection. Using SRV weight for dynamic server selection 
would require assigning unreasonably short TTLs to the 
SRV RRs, which would limit the usefulness of the DNS
caching mechanism, thus increasing overall network load 
and decreasing overall reliability."

With this in mind DNS can only really support stochastic 
load balancing, redirecting clients to servers randomly 
as various caches in various resolvers expire at random 
(although small) intervals. DNS offers excellent network 
scalability but poor control over load balance. It also
has a problem with detecting failure of servers within a 
pool. I would like to see it a requirement that any protocol 
for reliable server pools offers dynamic server selection
and better fault tolerance.

Cheers
Neil.
-- 
Ubiquity Software Corporation, UK        http://www.ubiquity.net

---

From: Eliot Lear <[email-masked]>
Date: Thu, 05 Apr 2001 09:44:49 -0700

Just to be clear I believe that while mentioning how RSERPOOL relates to 
DNS it is important that we not make excessive demands between the 
interaction of the two, as the purpose of one is quite different from 
the other.  This is another area in which I think we need to work on the 
draft.

Also, while DNS doesn't offer real load sharing via the SRV record, if 
we were only offering load sharing there are enough mechanisms out there 
to do that today via DNS A records, and we don't need another.  The 
distributed load balancing policy is very interesting.  Also, the only 
thing DNS has to go on for load sharing is a client IP address.

Finally, the low TTL case does indeed increase the load on a DNS server, 
but would that load be any different on an ENRP server?  Let's not 
confuse the use with the mechanism.  I think one can legitimately review 
two aspects of DNS in this area.  The first is in the dynamic update 
mechanism, and the second is in the ability to do a reasonable job of 
distributing load across secondaries.

---

From: Qiaobing Xie <[email-masked]>
Date: Thu, 05 Apr 2001 13:35:37 -0500

Eliot Lear wrote:
...snip...
> I think one can legitimately review

I also tend to think ENRP server provides a "directory service" rather
than a "namespace" as the DNS does. Here, I use the term "namespace" to
refer to a mapping relationship between a host name and a list of IP
address(es) assigned to a machine, while a "directory service" is more
general in nature and in the special case of ENRP provides a mapping
between a logical pool handle and a list of access address(es) to a
group of software server instances. Of cause, one can argue that the DNS
mechanism can be extended to offer similar "directory services", but my
impression is that this kind of extensions is what the DNS world
deliberately wants to avoid.

Besides, to make the Rserpool "directory service" effective, there are
two more things to cover: 1. robustness (i.e., no single point of
failure) for the "directory service" itself, and 2. operation status
monitoring of the entries currently registered in the directory. DNS
does not seem to have these functions available.

-Qiaobing

---

From: "Bernard Aboba" <[email-masked]>
Date: Thu, 5 Apr 2001 23:22:36 -0700

>and better fault tolerance.

A portion of the issues discussed in RFC 2782 are generic,
and so will apply to ENRP as well as to DNS. 

The portion that applies solely to DNS relates to the
effects of assigning very low TTLs. As noted in an 
earlier message, in ENRP one can set low TTLs
with consequences that only affect your server, not anyone
else's. With DNS you will also be loading third party local
DNS servers. So doing this is bad with DNS from the point 
of view of Internet etiquette. 

However, regardless of the TTL, it is nevertheless a bad
idea to attempt to update load metrics on too frequent
a basis. The fundamental issue here is that load balancing 
based on unsmoothed load metrics is typically unstable. This
is why, for example, we typically do not use load-based routing
metrics. 

Achieving stability with load metrics is somewhat of an
art, and typically requires smoothing and setting of 
hysterisis, as described in the OSPF OMP work by
C. Villamizar. 

So all you are getting with ENRP is the ability to set
lower TTLs. However, beyond a certain point,
lowering TTL won't get you much in the way of 
ability to respond to load changes.

---

From: "Bernard Aboba" <[email-masked]>
Date: Thu, 5 Apr 2001 23:28:35 -0700

>but would that load be any different on an ENRP server?  

Indeed, there is no difference whatsoever in the effect of TTL on the
DNS or ENRP servers. The difference lies in the effect of lowering
DNS TTL on *third party servers*. Whereas ENRP is a non-recursive
protocol in which lowering TTL affects only the endpoint ENRP server
load, with DNS, third parties do recursive queries for the client
and thus their load is affected by lowering TTL on the authoritative
DNS server.

---

From: "Bernard Aboba" <[email-masked]>
Date: Thu, 5 Apr 2001 23:33:11 -0700

>does not seem to have these functions available.

Neither of these two elements are relevant. If the DNS had a single
point of failure, the Internet would not be nearly as usable as it
is today. The entire DNS design is based on the principle of 
primary/secondary servers that make DNS largely immune against
failures of individual servers. Thinking that ENRP will somehow
improve upon this is wishful thinking. For a replicated, distributed
and largely fault tolerant distributed system, DNS is extremely
well designed.

As for status monitoring, this has nothing whatsoever to do with
the DNS protocol. There are DNS server products that include very
sophisticated monitoring functionality without any need to modify
the DNS protocol. In fact, I would argue that including such
monitoring functionality in a naming protocol is a mistake
because the results of such "monitoring" will not reflect very
accurately on the availability of the service in question. 

An example: How can the "monitoring" functionality detect that
my web server is serving HTTP 404 errors? Or that it is serving
up the wrong page? Or that the SQL server backend has died?
To understand this you not only have to probe the service in 
question at the application layer, but you also have to 
parse the response. Judging availability via some
kind of "heartbeat" functionality only detects
catastrophic errors. With today's more reliable hardware
and operating systems, the bulk of failiures are typically
application related.

---

From: Neil Deason <[email-masked]>
Date: Fri, 06 Apr 2001 09:46:06 +0100

> ability to respond to load changes.

I agree with what you are saying here but shouldn't
we be trying to get:
- a document for the requirements of reliable server pools
- documents saying why DNS, CORBA, blah do not fully meet
these requirements 
which do not even consider ENRP/ASAP.

My impression from the IETF was that the authors
saw ENRP/ASAP evolving further. To aid this we shouldn't
limit rserpool requirements to what is already possible.
Therefore should the ability to perform dynamic load 
balancing within a reliable server pool be discussed as a 
requirement.

Thanks,
Neil.

---

From: Qiaobing Xie <[email-masked]>
Date: Fri, 06 Apr 2001 12:02:07 -0500

> and largely fault tolerant distributed system, DNS is extremely

I must disagree with you on this. I have a pool of ENRP servers running
in my call processing lab that support a totally distributed pool
handler directory. There had never been _a_single_second_ of service
interruption to the call control (people continue to make calls) in the
lab when some engineer accidentally shutdown several machines which were
hosting some of the ENRP servers; they fell over to each other and
automatically re-synced their directory data in almost real-time. The
directory service in my lab is so robust that at one point I had to
provide a special utility to my lab engineers to conveniently shutdown
the entire ENRP server pool - they found it very hard to manually
shutdown the entire ENRP service because the servers took over for each
other and if they missed to kill one of them, the next time the system
re-start all the data entries from the previous life would magically
re-appear in the distributed directory. And this level of robustness IS
needed when you are building a mission critical system such as a
carrier-grade call processing center.

In a large scale, yes, the DNS design is fairly reliable since the big
"I" Internet is working so far. But in a small scale, there are
countless occasions of service interruption due to DNS outage. Even at
the past Minn IETF, on the first day (Sunday) of the meeting I was
unable to get DNS services from the terminal room at one point. I just
gave up and tried again a few hours later and found it working. "Just
call the ITS people about the DNS problem and have them to fix it, no
big deal". This is the life familiar to anyone who is using Internet or
intranet. But this is not nearly good enough to any mission critical
real time applications.
 
> and operating systems, the bulk of failiures are typically

There is huge a difference between service availability and endpoint
reachability. The Rserpool is _below_ the service application and hence
has no interest in monitoring whether the application above is providing
the right services. The monitoring function for ENRP is to make sure the
access address(es) it is book-keeping in its directory is still valid
and _reachable_. This is important for real time applications since
otherwise the users of the directory service will end up to find out any
unreachability in the hard way, i.e., by time-outs and data transfer
failures. 

regards,
-Qiaobing

---

From: "Bernard Aboba" <[email-masked]>
Date: Sat, 7 Apr 2001 12:07:39 -0700

>these requirements which do not even consider ENRP/ASAP.

Sure. All I'm saying is that we need to be very clear what
the problems are with DNS. In this particular case, the
issue is *not* that DNS is incapable of dynamic load
balancing -- or that the DNS NOTIFY/XFER or Dynamic Update 
mechanisms are somehow inadequate. 

The issue is that doing DNS-based dynamic load balancing
on short time scales will have impacts on third parties.
This is due to presence of "stub resolvers" on hosts. 
If hosts were to use full resolvers, this problem would 
not exist.

>balancing within a reliable server pool be discussed as a 

What I am pointing out is that some of the issues are
not DNS-specific, and therefore should not be cited
as reasons why DNS is inadequate.

For example, updating load on a very short time scale
will cause a high volume of replication between name
servers as the updates are propagated. This is not
specific to DNS -- the requirements "high write
volume" and "globally replicated" are intrinsically
at odds and these problems exist in other protocols 
such as LDAP. Also, such short time scale updates would
be likely to cause instability.

---

From: "Bernard Aboba" <[email-masked]>
Date: Sat, 7 Apr 2001 12:18:05 -0700

>intranet. But this is not nearly good enough to any mission critical

User experiences in the IETF terminal room are not sufficient 
justification for creation of new protocols. If they were, we'd
have overhauled the Internet Architecture from top to bottom a
long time ago ;)
 
There have been a number of studies of DNS reliability. If you would
like to make this an issue in the requirements document, then I
think we need to cite specific design flaws as well as to 
review the literature and make a convincing case. 

My understanding of the results is that many of the reliability 
problems are due to packet loss within high loaded DNS servers, 
such as the roots, as well as due to mis-configuration of DNS 
servers by inexperienced admins. For example, my understanding 
is that a substantial fraction of all DNS servers are mis-configured 
-- lame delegations, etc.

>unreachability in the hard way, i.e., by time-outs and data transfer

That's fine, but there are many ways to offer "heartbeat" style
reachability monitoring. Saying that DNS is deficient because it
doesn't offer such functionality isn't very convincing. Your 
argument would be better if you were to review existing 
server-server "heartbeat" protocols (e.g. VRRP) and describe 
why these are inadequate.

---

From: Lyndon Ong <[email-masked]>
Date: Mon, 09 Apr 2001 22:45:09 -0700

Folks,

Are we reaching any conclusions regarding the requirements document and any 
modifications to it?  So far, my impression is that most of the discussion 
has been around DNS, which applies more towards the second document with 
the comparison with DNS, CORBA, etc.

Cheers,

Lyndon

---

From: Randall Stewart <[email-masked]>
Date: Tue, 10 Apr 2001 10:26:44 -0500

> and largely fault tolerant distributed system, DNS is extremely

I would not dispute this Bernard... However when you look at ENRP you
get not only primary/secondary.. you get N occurances of the namespace
all working together.. so in that sense it does provide a small
amount more replication than DNS .. that is as long as N > 2 :)

> and operating systems, the bulk of failiures are typically

Everyone thinks in terms of HTTP. I don't get it :0 

In general if ASAP is running inside the user space of an
application (which is what we are designed to have) then it will
detect when the user stops reading messages. Now granted you could
have something start spitting out garbage responses.. but this is
well beyond the design of what ENRP was attempting.. The idea here
is that you want to get knowledge if the process receiving your
request is around and valid. Yes it can be insane but it is far
more likely (in my experience) that it takes a core dump and dies.

Continuing to try to contact a dead application when a peer exists
somewhere else that can service the request is the goal behind ENRP...
This is a big benefit for using ENRP and ASAP. I don't see how
DNS can server the role here either..

Typically in a ASAP appliation you do

AsapSend("pool-name",message, sizeof(message));

This then does the queury for pool-name and gets a list of applications
and ports.. NOT IP addresses... which is more along the lines of what
DNS returns. And the neat thing about it is I don't depend on a static
port number here.. I get a variaity of ports on possibly the same
machine/ip address.

R

-- 
Randall R. Stewart
Systems & Solutions Engineering
Cisco Systems Inc.
[email-masked] 815-342-5222 or 815-477-2127

---

From: Randall Stewart <[email-masked]>
Date: Tue, 10 Apr 2001 10:33:20 -0500

> the comparison with DNS, CORBA, etc.
Lyndon:

I agree with you on this point.. it seems all the chatter is centered on
the second "as-yet" unwritten document that John will provide. I am sure
he is taking copious notes.. thanks John!

What about the requirements themselves folks? What think you of these...
fogetting
the DNS issues and saving this debate for the actual document... 

Maybe we should create a seperate thread if we still wish to discuss...
one
on DNS and the other on the requirments..

R

---

From: Bernard Aboba <[email-masked]>
Date: Tue, 10 Apr 2001 14:18:47 -0700 (PDT)

> amount more replication than DNS .. that is as long as N > 2 :)

There is no limit to the number of DNS secondaries that can exist; you can
have as many as necessary. I think you are saying that ENRP supports
multi-master replication, as opposed to master-slave. This does have an
advantage over DNS in that in DNS if the primary goes down then typically
you cannot continue to do dynamic updates, although the existing RRs will
continue to be served. This means that while there will be no name service
interruption, you will not be able to change the load balance until the
primary comes back up. In particular, if a server were to go down while
the primary DNS was out, then you might not be able to take it out of the
cluster, which would impact cluster availability. 

> Everyone thinks in terms of HTTP. I don't get it :0 

The same issues exist with other protocols. The FTP service can be up but
the required files can be missing. The Telnet service can be up, but login
can be denied. The SQL database can be up but queries don't return the
expected result. So the notion of "availability" also includes
application-layer issues, including "content availability."

> detect when the user stops reading messages. 

I have seen situations where one particular thread of an
application goes down, but not the rest of it. The SNMP agent continues to
function, churning counters; the "looks alive" and "is
alive" functionality continues to function -- but the application is not
producing the expected results. As operating systems have incorporated
heartbeats and process restart capabilities these kind of scenarios are
becoming more common on a relative basis.

> Now granted you could
> well beyond the design of what ENRP was attempting. The idea here

Actually, the "core dump" scenario is becoming less common than the
partially broken scenario. After all, it's pretty easy to write a
script that will restart a process if it dies. Some operating systems do
this automatically now.  Think of all the HTTP 404  error messages you
see in the course of a day.

> DNS can server the role here either..

DNS can't solve that problem as well as ENRP because of caching related
issues. In other words, with ENRP you can set a very low or even zero TTL,
whereas in DNS, doing this would make you a "bad Internet citizen." That
means that ENRP will be much less likely to put you  in contact with a bad
server because the ENRP server will give you its take on what is available
NOW, as opposed to DNS, which might have cached the result of a query done
several hours ago. 

Howevever, once you've gotten a bad server, failover is driven by the
failover functionality of the transport, which is orthogonal to ENRP &
ASAP. This is where the SCTP failover functionality and the  "TCP failover
socket option" would come in. 

> DNS returns. 

Let's not confuse APIs with protocols. The above APIs can be implemented
using any name service. In fact, this is more or less the semantic that
you get with existing APIs like WININET. The question is whether the
functionality of ENRP/ASAP can do the overall job better than DNS. I think
that the answer is yes because of the caching issue, and potentially the
ability to know the actual client IP address (if you want to do failover
between clusters). However, the abstraction layer you are presenting,
though attractive, really isn't ENRP/ASAP specific. Nor is the ability to
failover between cluster members. That's a property of the transport. 

> And the neat thing about it is I don't depend on a static

Well, that might be useful if you felt it was likely that one instance
would die while another might remain living. However, there's nothing to
say that you can't have multiple SRV records in DNS, pointing to different
ports. So that isn't really an advantage of ENRP/ASAP either.

---

From: Lyndon Ong <[email-masked]>
Date: Tue, 10 Apr 2001 21:13:03 -0700

>application-layer issues, including "content availability."

Bernard,

Can I fish a little further on this?  Ideally "availability" includes 
application-layer
issues, but what avenues do we have in a transport-layer protocol to verify
this level of availability of the application?

Cheers,

Lyndon

---

From: Bernard Aboba <[email-masked]>
Date: Wed, 11 Apr 2001 05:45:46 -0700 (PDT)

> application-layer issues, but what avenues do we have in a
> transport-layer protocol to verify this level of availability of the
> application?

One of the points that was being made was that because DNS has no
equivalent of a "heartbeat" that it is not appropriate for your
purposes. What I'm saying is that such a heartbeat, if integrated within
the naming service, tells you a lot about the availability of the naming
component within the host, as well as the layers below it (transport,
drivers, network interface, etc.)  There are reasons why you'd want a
naming component to be available and should monitor for this -- for
example, to guarantee that hosts are de-registered in a timely way and
that their metrics remain accurate. 

Specifically, the ENRP heartbeat enables you to monitor availability of
the naming component on a finer time scale than would be possible with
DNS. There are DNS implementations that do "garbage collection" of 
RRs whose DHCP leases have expired, for example. However, this is only
useful for detecting failures on the order of the DHCP lease time -- which
is typically on the order of minutes if not hours or days. 

There are good reasons why DNS does not incorporate an application layer
heartbeat and will never be modified to do so. This would dramatically
boost traffic levels, and given the unavoidable third party dependencies
of DNS, the resulting loading would be unacceptable. ENRP, as a separate
naming protocol, lacks the third party dependency problem and so is able
to address this. 

So this is the logic behind this particular requirement, and the reason
why ENRP is more appropriate than DNS. 

However, it should be understood that there are many circumstances in
which the service can be malfunctioning that will not be discovered by a
naming service heartbeat. As a result, you will need to be able to also
take various "third party feedback" -- some of which will be from trusted
parties (application layer monitoring stations), and some of which will
not (e.g. client feedback). DNS load balancing implementations have
frequently been able to take trusted feedback into account. So the
ability to handle monitoring in general is not an advantage of ENRP over
DNS.

---

From: Neil Deason <[email-masked]>
Date: Fri, 20 Apr 2001 17:22:04 +0100

I brought up a suggestion for a minor addition of text to the 
requirements on Server Selection policies but it received no 
direct comment at the time.

Presently the draft says:
--
2.6.  Server Selection

The RSerPool mechanisms must be able to support different server
selection mechanisms. These are called server pool policies.

Examples of server pool policies are:

     -    Round Robin

     -    Least used

     -    Most used

The set of supported policies must be extensible in the sense
that new
policies can be added as required.
--

At this point could we add text to the effect off:

"Pool policies that allow for server selection based on 
non stochastic load balancing should be supported".

Thanks,
Neil.

---

From: Qiaobing Xie <[email-masked]>
Date: Fri, 20 Apr 2001 12:09:51 -0500

> Presently the draft says:

It is never intended to exclude the non-stochastic policies. In fact,
the first sentence "...able to support different server selection
mechanisms" already makes it clear enough that nothing is excluded. I
think your additional text is redundant.

And the reason we use term "policy" is to ensure the scope is broad
enough for any selection mechanism. Term "load balancing" is too narrow,
people may want to have selection policies which have nothing to do with
balancing the load.

-Qiaobing

---

From: Neil Deason <[email-masked]>
Date: Mon, 23 Apr 2001 12:40:13 +0100

> think your additional text is redundant.

I understand the intention but I don't think being too 
explicit when stating requirements causes problems. Thus 
I would still like the text included as all the example 
policies given (Round Robin, Least used, Most used) could be 
supported by stochastic policies. I would like it to be 
clearer that the requirements intend for more than that.
I also think this will be beneficial in the analysis of
why something new is needed rather than existing mechanisms.

Thanks,
Neil.