Subject: [Dime] New version of DOIC rate draft

From: Steve Donovan <[email-masked]>
Date: Mon, 10 Sep 2018 15:44:08 -0500

I've posted a new version of the rate draft.

I've attached the diff file.

Regards,

Steve

---

From: Ben Campbell <[email-masked]>
Date: Sat, 15 Sep 2018 18:56:46 -0500

Hi, thanks for posting the update; I think it is making progress. However,  I still have some comments I would like to address before IETF LC:

Thanks!

Ben.

Substantive Comments:

- General: I still think more discussion is needed about allocating rate to multiple input sources. I get that the actual allocation is a matter of local policy, but there’s still implications that need discussion. I’m not sure I got my concern across in previous discussion, so here’s another attempt:

The issue I think needs elaboration on is how the offered load varies with the number of sources times the (average) rate per each source. That is, if the number of sources changes, the reacting node may need to change the rate limits assigned to each existing source.

As a hypothetical example, lets assume a reporting node wants to limit its entire offered load to 1000 tps. Further assume it has 10 active reacting nodes (all supporting the rate algorithm). Local policy is to allocate the rate limit equally across sources. So it sends an OLR to each of those clients to give it a rate limit of 100 tps. Now, if another 10 reacting nodes become active, it needs to reallocate the load across all 20, giving each a limit of 50 tps. Now, if some of those reacting nodes go off-line, or simply reduce their activity beneath the limit for an extended period of time, the reporting node may need to increase the allocation to the remaining nodes.

This is a fairly fundamental difference between rate and load; rate uses absolute numbers while load uses percentages.

§5.1:

- The text is still not clear when a reporting node must send the first OLR. I understand that the choice of when to set a rate limit is local policy, but some of the text in this section suggests to me that you expect a reporting node to send an OLR immediately when it selects the rate algorithm for a specific reacting node.

For example, paragraph 6 says:

"A reporting node that has selected the rate overload abatement algorithm MUST indicate the rate requested to be applied by DOIC reacting nodes in the OC-Maximum-Rate AVP included in the OC-OLR AVP."

This needs to talk about _when_ it must do that. Without some comment, it seems like it means “when it selects the algorithm” (i.e. “when you indicate support” ). That seems to conflict with the idea of this being local policy.

- new 5th paragraph: Why SHOULD instead of MUST? Is there a situation where you have an OCS but no allocated rate? (e.g. you’ve selected the rate algorithm for the reacting node, but have not send an OLR?)

§5.4, paragraph 1:

Discussion indicated that the intent of this paragraph was the reacting node keeps OCS for each server than indicated support for the rate algorithm. But I don’t see text that says when the reacting note needs to actually create the state entry. I think the answer is “immediately when a reporting node indicates support”, right?

§5.6, first two paragraphs:

I still think the text talking about using different algorithms needs to say something normative about the characteristics of those algorithms. Janet’s comments indicated the normative text is in §7.3.1. But that’s part of the algorithm that they MAY use, so it would not be constraining against other algorithm choices. Perhaps the 2nd paragraph should be normative?

§7.2: “ But the resulting request rate presented to the overloaded reporting node will converge towards the target Diameter request rate.”

Wasn’t there discussion to change this to “... the target Diameter request rate or a lower rate”?

§7.3: "In situations where reacting nodes are configured with some knowledge
   about the reporting node (e.g., operator pre-provisioning), it can be
   beneficial to choose a value of TAU based on how many reacting nodes
   will be sending requests to the reporting node.”

I previously commented that this requires knowledge of other traffic sources, not just the reporting node. I did not see a response.

Editorial Comments:

[I note a number of editorial comments that fell into the “made changes unless indicated otherwise” category did not seem to get changed. I included those here again.]

- There are still things reported by IDNits that need checking. Some are obviously noise, but some appear to be real. (Line length, references in abstract, and references to RFC 5226)

- Was there a reason not to use the RFC 8174 boilerplate in the “Requirements” section? (I thought you had intended to do so.)

§1:
- first paragraph: There were some editorial fixes that I thought were agreed that did not appear in the new version:
s/“protect the stability”/“ensure the stability”
s/“subjected with”/“subjected to”
(new comment): In the new last sentence, is there a reason for the all-caps? That’s normally reserved for normative keywords.

§4:
- first paragraph: Please consider active voice in the last sentence,

§5.1:
- New 5th paragraph:
s/entery/entry

§7.1, 2nd paragraph: “ signal one another support for rate-based overload
  control”: This seems awkward; are there missing words? Perhaps there should be something like “their” or “that they” between “another” and “support”?

§7.2, last two paragraphs: The MUSTs do not seem necessary. 2119 keywords should be used when there is some sort of choice or room for error. You don’t need them to define the basic operation of the protocol.

§7.3.1: I found the text hard to follow. It would help to declare all the identifiers and initialization up front, and to present things in more of a stepwise fashion.

- T is effectively a time interval, right? It would help to say that, especially later when you subtract a different time interval from it.

- paragraph 9: Should “admit” be “emit”?

- the example code has several mentions of SIP requests.

§7.3.2:

- “ Request candidates for reduction, requests not subject to reduction (except under extenuating circumstances when there aren’t any messages in the first category that can be reduced).”:

That seems like an awkward way to say that the second category is the set of requests that is only subject to reduction if there are no messages left in the first category.

- “ This can be generalized to n priorities using n thresholds for n>2 in the obvious way.”: I suggest you refrain from calling it “obvious”.

§7.3.3: Paragraph starting with “ Then (only) if the arrival is admitted, increase the bucket by an amount…”: I think you increase the bucket _count_, right?

> On Sep 10, 2018, at 3:44 PM, Steve Donovan <[email-masked]> wrote:
> 
> <Diff  draft-ietf-dime-doic-rate-control-08.txt - draft-ietf-dime-doic-rate-control-09.txt.html>_______________________________________________
> DiME mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dime

---

From: "Gunn, Janet P (NONUS)" <[email-masked]>
Date: Sun, 16 Sep 2018 01:31:21 +0000

Just addressing the first point- deleted the rest

-----Original Message-----
From: DiME <[email-masked]> On Behalf Of Ben Campbell
Sent: Saturday, September 15, 2018 7:57 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [Dime] New version of DOIC rate draft

Substantive Comments:

<JPG> I am not sure if it is as fundamental a difference  as you think. (in both cases assuming each node has a relatively stable  load.)
If there are 10 active reacting nodes, it asks each node to reduce its traffic to  (say) 50%.
 If 10 more nodes become active, it will need to ask each node to reduce its traffic to 25%
If some of those nodes go offline, or reduce their traffic, it will increase the (percentage) allocation to each remaining node.

The "fundamental" difference is when each node has a fluctuating, unstable, load.
Under "load" a node which currently has a small load needs to cut its traffic by the SAME percentage as the node which currently has a large load.
Under "rate" a node which currently has a small load does NOT need to cut its load while the node which currently has a large load DOES need to cut its load.</JPG>

This electronic message transmission contains information from CSRA that may be attorney-client privileged, proprietary or confidential. The information in this message is intended only for use by the individual(s) to whom it is addressed. If you believe you have received this message in error, please contact me immediately and be aware that any use, disclosure, copying or distribution of the contents of this message is strictly prohibited. NOTE: Regardless of content, this email shall not operate to bind CSRA to any order or other contract unless pursuant to explicit written agreement or government initiative expressly permitting the use of email for such purpose.

---

From: Ben Campbell <[email-masked]>
Date: Sun, 16 Sep 2018 00:44:25 -0500

> On Sep 15, 2018, at 8:31 PM, Gunn, Janet P (NONUS) <[email-masked]> wrote:
> 

Hi,

There can be more than one fundamental difference :-)

I wasn’t talking about how clients behave, I was talking about how the server expresses it’s intention. If a server wants to reduce the current load by half using the loss algorithm, it doesn’t need to know how many clients there are. Of course, it may have to keep adjusting the percentage if the offered load is not stable. OTOH, If it wants to reduce the load to a specific rate using the rate algorithm, it does need to know many clients there are.

 I’m not saying this is a flaw in any way. I agree that in many cases an absolute TPS value may be more useful than a relative value. I’m just saying the draft should talk about it.

Thanks,

Ben.

---

From: Steve Donovan <[email-masked]>
Date: Fri, 21 Sep 2018 11:02:31 -0500

Thanks for the continued review.  I'll deal with the first issue in this
email.  I'll handle the remaining issues in (a) separate email(s).

For the first issue, I propose the following changes:

I propose adding the following wording between the 6th and 7th paragraph
of the introduction section:

"It should be noted that one of the implications of the rate based
algorithm is that the reporting node needs to determine how it wants to
distribute it's load over the set of reacting nodes from which it is
receiving traffic.  For instance, if the reporting node is receiving
Diameter traffic from 10 reacting nodes and has a capacity of 100
transactions per second then the reporting node could choose to set the
rate for each of the reacting nodes to 10 transactions per second. 
This, of course, is assuming that each of the reacting nodes has equal
performance characteristics.  The reporting node could also choose to
have a high capacity reacting node send 55 transactions per second and
the remaining 9 low capacity reacting nodes send 5 transactions per
second.  The ability of the reporting node to specify the amount of
traffic on a per reacting node basis implies that the reporting node
must maintain state for each of the reacting nodes.  This state includes
the current allocation of Diameter traffic to that reacting node.  If
the number of reacting node changes, either because new nodes are added,
nodes are removed from service or nodes fail, then the reporting node
will need to redistribute the maximum Diameter transactions over the new
set of reacting nodes."

I also propose the following wording added at the end of section 5.3.
Reporting Node Maintenance of Overload Control State

" A reporting node that receives a capability announcement from a new
reacting node, meaning a reacting node for which it does not have an OCS
entry, and the reporting node chooses the rate algorithm for that
reacting node SHOULD recalculate the rate to be allocated to all
reacting nodes.  Any changed rate values will be communicated in the
next OLR sent to each reacting node."

Steve

---

From: Ben Campbell <[email-masked]>
Date: Fri, 21 Sep 2018 11:06:57 -0500

> On Sep 21, 2018, at 11:02 AM, Steve Donovan <[email-masked]> wrote:
> 
> Thanks for the continued review.  I'll deal with the first issue in this email.  I'll handle the remaining issues in (a) separate email(s).
> 
> I propose adding the following wording between the 6th and 7th paragraph of the introduction section:
> 
> "It should be noted that one of the implications of the rate based algorithm is that the reporting node needs to determine how it wants to distribute it's load over the set of reacting nodes from which it is receiving traffic.  For instance, if the reporting node is receiving Diameter traffic from 10 reacting nodes and has a capacity of 100 transactions per second then the reporting node could choose to set the rate for each of the reacting nodes to 10 transactions per second.  This, of course, is assuming that each of the reacting nodes has equal performance characteristics.  The reporting node could also choose to have a high capacity reacting node send 55 transactions per second and the remaining 9 low capacity reacting nodes send 5 transactions per second.  The ability of the reporting node to specify the amount of traffic on a per reacting node basis implies that the reporting node must maintain state for each of the reacting nodes.  This state includes the current allocation of Diameter traffic to that reacting node.  If the number of reacting node changes, either because new nodes are added, nodes are removed from service or nodes fail, then the reporting node will need to redistribute the maximum Diameter transactions over the new set of reacting nodes.”

Thats exactly what I had in mind, thanks!

> 
> I also propose the following wording added at the end of section 5.3. Reporting Node Maintenance of Overload Control State
> 
> " A reporting node that receives a capability announcement from a new reacting node, meaning a reacting node for which it does not have an OCS entry, and the reporting node chooses the rate algorithm for that reacting node SHOULD recalculate the rate to be allocated to all reacting nodes.  Any changed rate values will be communicated in the next OLR sent to each reacting node.”

I will leave it to you to decide, but I don’t know that the SHOULD is necessary. We could treat this as a statement of fact of which people need to be aware. (“may need to” or “needs to”).

>>> On Sep 15, 2018, at 8:31 PM, Gunn, Janet P (NONUS) <[email-masked]> <mailto:[email-masked]> wrote:
>>> 
>>> From: DiME <[email-masked]> <mailto:[email-masked]> On Behalf Of Ben Campbell
>>> To: [email-masked] <mailto:[email-masked]>
>>> Cc: [email-masked] <mailto:[email-masked]>

---

From: Steve Donovan <[email-masked]>
Date: Fri, 21 Sep 2018 11:58:21 -0500

I've dealt with the majority of the remaining issues in this email.

There are a few questions that deal with section 7 that I have left for
Eric to address.

Steve

> This is a fairly fundamental difference between rate and load; rate uses absolute numbers while load uses percentages.
SRD> See previous email.
> This needs to talk about _when_ it must do that. Without some comment, it seems like it means “when it selects the algorithm” (i.e. “when you indicate support” ). That seems to conflict with the id…
SRD> Well, it must indicate the rate anytime an OLR is sent.  The when
is specified in RFC7683, Section 5.2.3.  Here's part of that section
(paragraph 1) included to hopefully illustrate that no additional
wording is needed in the rate draft:

   If there is an active OCS entry, then a reporting node SHOULD include
   the OC-OLR AVP in all answers to requests that contain the
   OC-Supported-Features AVP and that match the active OCS entry.
> - new 5th paragraph: Why SHOULD instead of MUST? Is there a situation where you have an OCS but no allocated rate? (e.g. you’ve selected the rate algorithm for the reacting node, but have not send…
SRD> There is a possible implementation that assumes the rate for each
reacting node is calculated by dividing the number of active nodes by
the maximum rate.  With this implementation the OCS would not require
storing the rate.  I did see an error in that paragraph, and as such,
I've changed that paragraph to the following:

The rate OCS entery SHOULD include the rate allocated to the reacting note.

Old text:

   The rate OCS entery SHOULD include the rate allocated to each
reacting note.

New text:

   The rate OCS entery SHOULD include the rate allocated to the reacting
note.

> Discussion indicated that the intent of this paragraph was the reacting node keeps OCS for each server than indicated support for the rate algorithm. But I don’t see text that says when the reactin…
SRD> This is covered by the following paragraph from RFC7683:

   If the received OLR is for a new overload condition, then a reacting
   node MUST generate a new OCS entry for the overload condition.
> I still think the text talking about using different algorithms needs to say something normative about the characteristics of those algorithms. Janet’s comments indicated the normative text is in §…
SRD> How about the following change in paragraph 2:

Old text:

      Note: Other algorithms for controlling the rate can be implemented
      by the reacting node as long as they result in the correct rate of
      traffic being sent to the reporting node.

New text:

   Other algorithms for controlling the rate MAY be implemented
   by the reacting node.  Any algorithm implemented MUST result
   in the correct rate of traffic being sent to the reporting node.
> Wasn’t there discussion to change this to “... the target Diameter request rate or a lower rate”?
SRD> Agreed, change made.
> I previously commented that this requires knowledge of other traffic sources, not just the reporting node. I did not see a response.
SRD> I don't understand the issue.  I'm hoping Eric can comment or you
can clarify.
> [I note a number of editorial comments that fell into the “made changes unless indicated otherwise” category did not seem to get changed. I included those here again.]
SRD> My apologies if I missed these in the last refresh.
> - There are still things reported by IDNits that need checking. Some are obviously noise, but some appear to be real. (Line length, references in abstract, and references to RFC 5226)
SRD> There were no issues reported by IDNits when I submitted the
document.  Is there a different IDnits check than the one used when
submitting drafts?
> - Was there a reason not to use the RFC 8174 boilerplate in the “Requirements” section? (I thought you had intended to do so.)
SRD> Yes, this slipped through the cracks.  Change made.
> s/“protect the stability”/“ensure the stability”
SRD> Change made
> s/“subjected with”/“subjected to”
SRD> Change made
> (new comment): In the new last sentence, is there a reason for the all-caps? That’s normally reserved for normative keywords.
SRD> I don't remember.  I've changed it to all lower.
> - first paragraph: Please consider active voice in the last sentence,
SRD> Old text:

   This document defines the rate abatement algorithm (referred to as
   rate in this document) feature.  Support for the rate feature by a
   DOIC node will be indicated by a new value of the OC-Feature-Vector
   AVP, as described in Section 6.1.1, per the rules defined in
   [RFC7683].

New Text:

   This document defines the rate abatement algorithm (referred to as
   rate in this document) feature.  A DOIC node indicates support for the
   rate feature by indicating a new value of the OC-Feature-Vector
   AVP, as described in Section 6.1.1, per the rules defined in
   [RFC7683].
> s/entery/entry
SRD> Change made.
>   control”: This seems awkward; are there missing words? Perhaps there should be something like “their” or “that they” between “another” and “support”?
SRD> Changed from:

   Following the procedures defined in [RFC7683], the reacting node and
   reporting node signal one another support for rate-based overload
   control.

To:

   Following the procedures defined in [RFC7683], the reacting node and
   reporting node signal their support for rate-based overload
   control.
> §7.2, last two paragraphs: The MUSTs do not seem necessary. 2119 keywords should be used when there is some sort of choice or room for error. You don’t need them to define the basic operation of th…
SRD> I'm okay with removing the MUSTs.  Here the proposed change:

Old:

   Upon detection of overload, and the determination to invoke overload
   controls, the reporting node MUST follow the specifications in
   [RFC7683] to notify its clients of the allocated target maximum
   Diameter request rate and to notify them that the rate overload
   abatement is in effect.

   The reporting node MUST use the OC-Maximum-Rate AVP defined in this
   specification to communicate a target maximum Diameter request rate
   to each of its clients.

New:

   Upon detection of overload, and the determination to invoke overload
   controls, the reporting node follows the specifications in
   [RFC7683] to notify its clients of the allocated target maximum
   Diameter request rate and to notify them that the rate overload
   abatement is in effect.

   The reporting node uses the OC-Maximum-Rate AVP defined in this
   specification to communicate a target maximum Diameter request rate
   to each of its clients.
> - T is effectively a time interval, right? It would help to say that, especially later when you subtract a different time interval from it.
SRD> I'll leave this for Eric to handle.
> - the example code has several mentions of SIP requests.
SRD> These have been changed to Diameter requests
> §7.3.3: Paragraph starting with “ Then (only) if the arrival is admitted, increase the bucket by an amount…”: I think you increase the bucket _count_, right?
SRD> I'll leave these for Eric to handle.
> _______________________________________________
> DiME mailing list
> [email-masked]

---

From: Steve Donovan <[email-masked]>
Date: Fri, 21 Sep 2018 12:02:20 -0500

On 9/21/18 11:06 AM, Ben Campbell wrote:
>
>
>> On Sep 21, 2018, at 11:02 AM, Steve Donovan <[email-masked]
>> <mailto:[email-masked]>> wrote:
>>
>> Thanks for the continued review.  I'll deal with the first issue in
>> this email.  I'll handle the remaining issues in (a) separate email(s).
>>
>> algorithm is that the reporting node needs to determine how it wants
>> to distribute it's load over the set of reacting nodes from which it
>> is receiving traffic.  For instance, if the reporting node is
>> receiving Diameter traffic from 10 reacting nodes and has a capacity
>> of 100 transactions per second then the reporting node could choose
>> to set the rate for each of the reacting nodes to 10 transactions per
>> second.  This, of course, is assuming that each of the reacting nodes
>> has equal performance characteristics.  The reporting node could also
>> choose to have a high capacity reacting node send 55 transactions per
>> second and the remaining 9 low capacity reacting nodes send 5
>> transactions per second.  The ability of the reporting node to
>> specify the amount of traffic on a per reacting node basis implies
>> that the reporting node must maintain state for each of the reacting
>> nodes.  This state includes the current allocation of Diameter
>> traffic to that reacting node.  If the number of reacting node
>> changes, either because new nodes are added, nodes are removed from
>> service or nodes fail, then the reporting node will need to
>> redistribute the maximum Diameter transactions over the new set of
>> reacting nodes.”
>
>> reacting node, meaning a reacting node for which it does not have an
>> OCS entry, and the reporting node chooses the rate algorithm for that
>> next OLR sent to each reacting node.”
>
> I will leave it to you to decide, but I don’t know that the SHOULD is
> necessary. We could treat this as a statement of fact of which people
> need to be aware. (“may need to” or “needs to”).
I like this suggestion.  I'll change it to may need to.