Subject: RE: [AAA-WG]: Question on Diameter behavior in Failure cases (VER SION 1.2)

From: [email-masked]
Date: Wed, 12 Sep 2001 14:39:06 +0300

Hi Bernard,

my comments are embedded in the text below.

Br, 
John

> -----Original Message-----
> From: ext Bernard Aboba [mailto:[email-masked]]
>
> > One question I have is what happens if a Diameter client goes 
> > down while sending messages, before getting a response to a 
> > pending message, when there is no secondary peer?
> 
> If the NAS goes down, you will have a AAA failure whether there
> are secondary servers or not. The way to handle that kind of failure
> is via clustering technology (some NASes already support this today)
> so that the connection can be migrated to another NAS. 
> Diameter can't help
> with this. 

Yes, clustering is an option if costs are not really relevant.
But the Diameter could be defined so that it is reliable
and robust. On the other hand, it would be a quite expesive
requirement if every Diameter node would have to be clustered
(if the Diameter could not itself be made reliable).

> > Does that imply that a Diameter node may not ALWAYS have a
> > secondary or additional connections available.
> 
> Failover protection is not mandatory. But if you want high
> availability it is a good idea.

Ok. As you stated (and I understood from the draft),
Diameter node is thus intended to work in both cases:

a) with additional connections available
b) as a node permanently without additional connections.

Additionally there may be the case
c) the additional connection(s) is/are temporarily out of order.

> > In that case, if the Diameter client stops temporarily getting 
> > answer messages (and does not have an alternate agent to send 
> > the payload packet), the primary peer may recover (after a boot,
> > for example). Then the following problem arises:
> > 
> 
> I'm confused. You were initially talking about a *client* 
> failure. Now you
> are talking about *server* failures. If the primary server 
> fails and there
> is no alternate, then availability is compromised. Again, there isn't
> much that Diameter can do about that. 

Sorry if my wording was not clear. I meant that 
this email thread discusses _several_ failure cases,
as indicated in the email Subject.

The words were there just as intended: "The Diameter client
stops temporarily _getting_ messages" meant that the
primary peer (Diameter Agent) is down (or the link to it).

So I was (as intended) talking in the referred text about a
Diameter agent failure instead of Diameter client failure.

To the second part of the above comment it could be
answered that Diameter _could_ really do something
about the situation, even if there are no Diameter
agents temporarily available.

The Diameter client certainly has at least some RAM
memory available and it may have non-volatile memory too.

(It is already defined in the chapter 8.8 and
9.4 that there may be non-volatile memory available
and that the Diameter node can keep count of the
restarts and even communicate the number outwards
if needed, using the Session-Id AVP.)

These memory resources allow (as long as they
last) the Diameter client to put all the pending
messages in buffers, waiting for a peer node/link to come
up.

=> The Diameter client that gets the communication
working again to the awakening peer node, may indeed
be in position to overcome a temporary state of
isolation. It may be able to deliver all the locally
(volatile/non-volatile memory) buffered pending
messages. So, the Diameter client could
well try for more than abandon all its packets
pending to be sent, if temporarily there is no
working link to a peer node. It could resend the
before link failure not yet unacknowledged
messages (minimum amount: 0, max amount: the
protocol window size) again. But to "warn" the
Diameter server, the 'D' bit would be set for the
one (or very few) yet unacknowledged messages. This
would tell the Diameter server that it should check
them (with the E2E Identifier pairs & the Origin-Host
AVP pairs), as they (only) need the detail level check.

> 
> > Should the Diameter client re-send the pending payload
> > message to its original (& now recovered) primary (& only) peer 
> > node, as the client does not know if the packet had been
> > properly received went sent before the failure.
> > 
> 
> This points out that failure scenarios can result in 
> duplicates, not only
> to different servers, but also to the same server. That is what the
> e2e session identifier was created to handle.

But, comparisons of the the End-to-End Identifier pairs
together with the Origin-Host AVP pairs for every packet
(including the non-duplicated majority) would be far less efficient
if the possible duplicates would not be marked by the Diameter
client node that is the only node knowing when it sends possible
duplicates. Since the ratio of duplicate messages is less than
one to a billion normal messages, and since the checking of
a single bit is faster than searching and decoding AVP pairs
and checking e.g. 70 octets one by one, the checking of just
one bit would be a most useful method.

This does not mean that the E2E Identifier pairs and Origin-Hot
AVP pairs would not be used, but that they would be mandatorily 
used only for the (marked) possible duplicates.

> In the case of accounting,
> this should be sufficient to deal with a duplicate accounting record. 

In the case of accounting, a very similar big difference in
the efficiency appears: Either to robotically search every
single message for the Session-Id AVP and for the
Accounting-Record-Number AVP and after that check their contents,
octet by octet. It would mean some decades more uniqueness
checking work at the Diameter server if we would not have the
'D' bit in a constant offset place in the Diameter header
(allowing the Diameter server to limit the (mandatory)
uniqueness checks only to those very few potential
messages for which the detailed processing (= E2E Identifier &
Origin-Host AVP checking) is useful.

> 
> In the case of authentication/authorization, I'm not sure 
> that there is a problem. You'll just get a new response.

In a commercial, country wide operator network a Diameter
server could be serving a huge number of Diameter clients
and message uniqueness checking efficiency at the Diameter
server would look as a very useful thing. If only the
'D' bit existence is checked most of the time (and the
E2E Identifier pairs and the Origin-Host AVPs pairs compared
only in less cases than e.g. one in a billion), then
Diameter would work better, even _if_ the "application
purpose" (e.g. authentication) would allow occasional
failures in message handlings.

Anyway, it would look a safe way if unpredictable
and abnormal situations are avoided as carefully as
possible, not relying on e.g. that it does not matter
usually or so. And since there may be yet unseen
Diameter extensions in the future, it would be
far most simplest to have the already the Diameter
Base protocol as a reliable message handler.

> The only problem that could have
> arisen is if the server wrote something about the session to stable
> storage before responding, and *then* the connection failed (such as
> incrementing the simultaneous session count). In that case, when the
> server comes up again, it will have incorrect information in 
> its database. 
> 
> The solution to this is for the server to *not* write session info to
> stable storage until it receives the equivalent of an 
> application-layer
> ACK from the NAS. There is no app-layer ACK of an 
> authentication response
> in Diameter, because the Accounting Request largely serves 
> this function. 

(May I first correct the last line of my paragraph which
was intended to be: "> > properly received before the failure.")

Yes, session info is not looking so useful to be written
to stable storage before the ACK.

> > It seems that this case a solution could be offered by the 
> > by marking the the packet 'possibly duplicated' (see my
> > proposal sent yesterday for usage of a 'D' bit in the Diameter 
> > header) since the receiving primary (and only) peer node could 
> 
> I don't think that this is necessary, if the advice above is 
> followed. Can
> you come up with a scenario in which a problem would be 
> caused by receipt of a duplicate authentication packet?

The main benefit is an quite big (e.g. 2 decades)
efficiency boost in the message uniqueness checkings
in the Diameter server.

I have not yet investigated all the possible authentication
messaging scenarios and what problems the allowing of
duplicated messages could cause but it would be easiest
just to mark the potential duplicates. The need for the
(slower) detail uniqueness checking arises only very seldom
and the sending node can mark those cases.
(Less than one message in a billion would typically
be a duplicated message, typically caused to an
travelling message by a broken link).

Summary:

  It would be a simple solution to just prevent
the duplicates to appear in an uncontrolled manner.
This would be achieved by the most simple way
that is used in these cases, by setting one ('D') bit by
any Diameter message sending node (not the Siameter server).

By marking the possible duplicates and checking
thorougly at the Diameter server only the marked
messages. The marked possibly duplicates messages
form an extremely small minority of the whole
message traffic of a network. Therefore the
uniqueness checking processor load at the Diameter
server would decrease to almost zero for the total
traffic.

Therefore I would recommend that approach in the
first place, especially as we do not yet even
know what future applications and extensions
could be requiring from the Diameter Base
protocol reliability. The bottom line is therefore
that it would be risky to allow duplicates, but
beneficial to do that with almost zero processor
effort per packet in average at the Diameter Server.
The currently existing Diameter message uniqueness
checking could be used, but mandatorily only for
those few packets, which possibly actually could
be duplicates.

John

---

From: Bernard Aboba <[email-masked]>
Date: Wed, 12 Sep 2001 07:12:47 -0700 (PDT)

> memory available and it may have non-volatile memory too.

Yes. Diameter clients can be designed to store requests in NVRAM, and
retry indefinitely, either to the same agent or a failover agent. 

> messages in buffers, waiting for a peer node/link to come

Actually, as described in RFC 2975, it is often desirable to utilize
non-volatile storage so that requests can survive a client
failure. So there is no need to lose data just because memory resources
have run out. 

> isolation. 

Yes, this is how the protocol should behave. 

> It may be able to deliver all the locally
> working link to a peer node. 

In general, a Diameter node MUST NOT abandon "all packets" waiting to be
sent, just because a connection is closed. If NVRAM exists, then
unacknowledged requests can be stored there to protect against a client
failure. They should only be deleted there when an app-layer ack is
received (accounting) or when the client connection drops
(authentication). 

If there is no NVRAM, but memory resources are available, then the packets
should be stored awaiting delivery to an alternate peer, or recovery of
the primary peer. 

Since loss of data is one of the major flaws of RADIUS, I hope that the
Diameter specification is clear about the expected behavior. If the
behavior you describe is not clearly indicated in the spec, then it
should be. If not, please file an issue on this. 

> This would tell the Diameter server that it should check

The problem is that in these failover situations, there are no ordering
guarantees between packets sent on different connections. If the packets
are sent within the "time wait" interval of each other, then it is
conceivable that the request with the 'D' bit set could arrive *before*
the initial request. So it's not clear to me that having a 'D' bit 
removes the need to check for dupes on packets without the 'D' bit set. In
that case, what good does the 'D' bit do? 

Also, if we have a 'D' bit, we need to be clear about the 
obligations of a Diameter server receiving a request with a 'D' bit set. 

In failover scenarios, duplicate checking may not necessarily be feasible
at the Diameter Server. For example, in accounting, the record may be
stored with the E2E identifier, and then the billing engine may remove the
duplicates later. 

In authentication, if the request is idempotent, there is no need for the
server to take any action based on the 'D' bit -- the server can just send
another response. However, if simultaneous usage is being tracked, then it
would make sense to check for duplicates. However, it's important to
understand that ordering is not guaranteed -- so the request with the 'D'
bit could arrive *before* the other request that does *not* have the 'D'
bit set. So it seems to me that where we have non-idempotent requests, 
duplicate detection *always* needs to be carried out.  

> duplicates. 

The problem is that the 'D' bit provides a false sense of security. I
think that you can still have duplicates in packets without the 'D' bit
set. 

I presume that the 'D' bit is being set for all requests that were pending
an application layer ack when the connection went down, whether they are
resent on a connection to the same peer or an alternate peer. 

> octet by octet. 

In practice, I don't think you'll do this -- won't you just write the
accounting record to stable storage, and do the dupe checking down the
line, unless simultaneous usage is being monitored?