Subject: RE: draft-asveren-dispatch-http-overload

From: "Asveren, Tolga" <[email-masked]>
Date: Tue, 1 May 2018 09:41:36 +0000

I certainly will include these explanation, and more about a few other issues, in the next version of the draft. My initial goal was to get some initial idea/feedback from other folks.

Regarding interpretation of 503: I think it is quite clear, at least IMHO, based on RFC2616 that it indicates the status of the server in general not just for a single request:
10.5.4 503 Service Unavailable
   The server is currently unable to handle the request due to a
   temporary overloading or maintenance of the server. The implication
   is that this is a temporary condition which will be alleviated after
   some delay. If known, the length of the delay MAY be indicated in a
   Retry-After header. If no Retry-After is given, the client SHOULD
   handle the response as it would for a 500 response.

I also never saw it being interpreted/used on “per request” basis in practice.

Thanks,
Tolga

From: Martin Thomson <[email-masked]>
Sent: Monday, April 30, 2018 7:45 PM
To: Asveren, Tolga <[email-masked]>
Cc: [email-masked]
Subject: Re: draft-asveren-dispatch-http-overload

________________________________
NOTICE: This email was received from an EXTERNAL sender
________________________________

Thanks for the responses. It would be really helpful if the draft
explained this.

FWIW, I didn't interpret Retry-After in the way that 5390 does. I
understood it to apply to the request, not the entire server. But you
will observe that this off/on problem only causes the bursty traffic
pattern described there if there are relatively few clients. Larger
numbers of clients will result in better distribution of retries.
That happens less often in HTTP than in SIP.

For the working group: Is the assumption that 503+Retry-After means
that the client should withhold all requests to the same server (I
assume that server ~=> origin in this case), the identified resource,
or is it just a single request? Worth an http-core issue?

On Tue, May 1, 2018 at 1:25 AM, Asveren, Tolga <[email-masked]<mailto:[email-masked]>> wrote:
> Martin,
>
>
>
> Thanks for the feedback/questions.
>
>
>
> i- Why Retry-After is inadequate
>
> Not that I am lazy but I think https://tools.ietf.org/html/rfc5390 (5.3.
> The Off/On Retry-After Problem is probably the most relevant part for your
> question) already explains this better than I could do. It is for SIP but
> the same arguments would apply for any real-time application where response
> times matter.
>
>
>
> BTW, the issues listed in RFC5390 and the mechanism defined in RFC7339 to
> deal with it are tested/observed/verified by a group consisting of multiple
> vendors while preparing these specifications.
>
>
>
> ii- The goal of this mechanism is not to provide a solution against
> malicious attacks. It is meant to be used between well-behaving entities.
> Nonetheless I don’t think it makes DoS more likely nor does it leak
> information which significantly can increase efficiency of DoS. It is true
> that it conveys information about load status of a server but I think this
> can be deducted by an attacker anyhow (at least to some reasonable extent)
> by sending requests and checking the response times. So, a server should do
> whatever it does today when it suspects DoS activity. No changes are
> defined/assumed on that front.
>
>
>
> iii- “The table” is specific for the application type. It has multiple
> values as most -if not all- real-time applications have the notion of
> regular/priority/emergency request types. The drop percentage for each could
> be different depending on server and current overload conditions.
>
>
>
> Thanks,
>
> Tolga
>
>
>
> From: Martin Thomson <[email-masked]<mailto:[email-masked]>>
> Sent: Monday, April 30, 2018 1:55 AM
> To: Asveren, Tolga <[email-masked]<mailto:[email-masked]>>
> Cc: [email-masked]<mailto:[email-masked]>
> Subject: Re: draft-asveren-dispatch-http-overload
>
>
>
> ________________________________
>
> NOTICE: This email was received from an EXTERNAL sender
>
> ________________________________
>
>
> This is the right working group, for sure. However, I don't find the
> introduction convincing. Maybe I'm missing something. How is
> Retry-After inadequate?
>
> As for the mechanism described, how does it not expose information
> about the server configuration such that DOS could be more precisely
> targetted? What is the scope of the table? What is the point of
> multiple values?
>
>
> On Mon, Apr 30, 2018 at 3:50 AM, Asveren, Tolga <[email-masked]<mailto:[email-masked]>> wrote:
>> I submitted draft-asveren-dispatch-http-overload-control to DISPATCH WG a
>> while ago. It aims to specify a generic overload control mechanism for
>> HTTP/HTTPS applications.
>>
>>
>>
>>
>> https://www.ietf.org/id/draft-asveren-dispatch-http-overload-control-00.txt
>>
>>
>>
>>
>>
>> I thought it could be a good idea to herald it here as well as some folks
>> may not be following DISPATCH WG.
>>
>>
>>
>> I would appreciate any feedback about overall idea/need/alternatives/in
>> general.
>>
>>
>>
>> Thanks,
>>
>> Tolga
>>
>>

---

From: Julian Reschke <[email-masked]>
Date: Tue, 1 May 2018 12:18:49 +0200

On 2018-05-01 11:41, Asveren, Tolga wrote:
> I certainly will include these explanation, and more about a few other 
> issues, in the next version of the draft. My initial goal was to get 
> some initial idea/feedback from other folks.
> 
> Regarding interpretation of 503: I think it is quite clear, at least 
> IMHO, based on RFC2616 that it indicates the status of the server in 
> general not just for a single request:
> 
> *10.5.4 503 Service Unavailable*
> 
> ...

RFC 2616 is irrelevant. RFC 7231 says:

> The 503 (Service Unavailable) status code indicates that the server is currently unable to handle the request due to a temporary overload or scheduled maintenance, which will likely be alleviated after some delay. The server MAY send a Retry-After header field (Section 7.1.3) to suggest an appropriate amount of time for the client to wait before retrying the request.
> 
> Note: The existence of the 503 status code does not imply that a server has to use it when becoming overloaded. Some servers might simply refuse the connection.

That IMHO leaves the nature of the overload open; it could be the server 
itself, a specific service, or just one resource.

Best regards, Julian

---

From: "Asveren, Tolga" <[email-masked]>
Date: Tue, 1 May 2018 11:00:30 +0000

Agreed that it could be server or the application or the resource (at least theoretically and maybe also in practice). This draft is about server/application overload cases. For real-time application use, these need to be addresses in a satisfactory way for enabling better response times once they happen IMHO.

So, maybe a new parameter indicating the nature of the overload could be useful. Any thoughts on that?

And my apologies for not being clear about the terminology causing confusion.

Thanks,
Tolga

From: Julian Reschke <[email-masked]>
Sent: Tuesday, May 1, 2018 6:19 AM
To: Asveren, Tolga <[email-masked]>; Martin Thomson <[email-masked]>
Cc: [email-masked]
Subject: Re: draft-asveren-dispatch-http-overload

Best regards, Julian

---

From: "Asveren, Tolga" <[email-masked]>
Date: Sun, 6 May 2018 19:46:15 +0000

To regroup:

i- Is there any existing mechanism (503 with Retry-After does not provide a satisfactory solution) which would address the server/application overload problem? AFAICT the answer is “No” (but I am ready to be corrected)

ii- (Assuming that “No” holds good for i-) I am planning to update the draft with explanations about:

  *   Why 503 with Retry-After is inadequate
  *   That the intent is not to prevent attacks from malicious users
  *   The scope is server/application overload

Any other suggestions/feedback are welcome.

Thanks,
Tolga

From: Asveren, Tolga
Sent: Tuesday, May 1, 2018 7:00 AM
To: 'Julian Reschke' <[email-masked]>; Martin Thomson <[email-masked]>
Cc: [email-masked]
Subject: RE: draft-asveren-dispatch-http-overload

Thanks,
Tolga

From: Julian Reschke <[email-masked]<mailto:[email-masked]>>
Sent: Tuesday, May 1, 2018 6:19 AM
To: Asveren, Tolga <[email-masked]<mailto:[email-masked]>>; Martin Thomson <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: draft-asveren-dispatch-http-overload

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 14 May 2018 17:36:21 +1000

Hi Tolga,

My .02 - 

> On 7 May 2018, at 5:46 am, Asveren, Tolga <[email-masked]> wrote:
> 
> To regroup:
>  

I'm not sure that that 503+Retry-After is inadequate. The text that you reference in 5390 seems to suggest that in SIP, Retry-After "allows a server to tell an upstream element to stop sending traffic for a period of time," thereby causing the problems you refer to. In HTTP, this is not the case; the scope of applicability for Retry-After is only the request that it's associated with. From <https://httpwg.org/specs/rfc7231.html#status.503>:

"The server MAY send a Retry-After header field (Section 7.1.3) to suggest an appropriate amount of time for the client to wait before retrying the request."

So, the server can be selective about the requests it defers, giving a more fine-grained control mechanism. It could be that the semantics of Retry-After might need some refinement (e.g., in an applicability statement for a particular use case, or a protocol extension that adds more information), but it seems like its definition in SIP and HTTP diverge enough to justify caution in making assumptions.

All that said -- what would really help would be a set of use cases that explain how you expect this to be used in HTTP; it's difficult to design or evaluate an extension without them.

Cheers,

> 	• Why 503 with Retry-After is inadequate
> 	• That the intent is not to prevent attacks from malicious users
> 	• The scope is server/application overload
>  
>  

--
Mark Nottingham   https://www.mnot.net/

---

From: "Asveren, Tolga" <[email-masked]>
Date: Mon, 14 May 2018 09:51:44 +0000

Hi Mark,

Thanks for the reply.

Quoting the full text from RFC7231 Section 6.6.3:

   The 503 (Service Unavailable) status code indicates that the server
   is currently unable to handle the request due to a temporary overload
   or scheduled maintenance, which will likely be alleviated after some
   delay.  The server MAY send a Retry-After header field
   (Section 7.1.3<https://tools.ietf.org/html/rfc7231#section-7.1.3>) to suggest an appropriate amount of time for the
   client to wait before retrying the request.

      Note: The existence of the 503 status code does not imply that a
      server has to use it when becoming overloaded.  Some servers might
      simply refuse the connection.

I agree that 503 with Retry-After may pertain to a specific request, to the server overload or to the application overload condition (actually the last two are probably the same as far as their impact/interpretation on the client is concerned IMHO). The problem I have is for server/application overload case. This is an issue mainly for environments where the client is not an end-device, e.g. browser, but a network entity. The relationship between a Session Processing Entity and Signature Server is an example of that (and there are many more in practice and it just keeps growing) :

                            +----------------+
                            |                |
                            |   Signature    |
                            |   Server       |
                            |                |
                            |                |
                            +----------------+
                             ^           |
                             |           |
                           POST         200
                         (JSON body   (JSON body
                        with claims)   with signature
X                            |         generated based on claims
                             |         and private key)
X                            |           |
                             |           |
X  Many end-devices          |           V
   generating many        +----------------+
X  new session requests   |                |
                          |   Session      |
X          -new session-->|   Processing   |--new session-->
            request       |   Entity       |request with signature
X                         |                |
                          +----------------+

Why does the HTTP client being a network entity matter from overload perspective? The answer is the volume of traffic exchanged between client and server. In such a case, the need is not for a “per request” indicator but for one which would pertain to all subsequent requests to be sent to the server.

I think what you state as “ a protocol extension that adds more information” is what I am trying to achieve with this draft.

Thanks,
Tolga

From: Mark Nottingham <[email-masked]>
Sent: Monday, May 14, 2018 3:36 AM
To: Asveren, Tolga <[email-masked]>
Cc: Julian F. Reschke <[email-masked]>; Martin Thomson <[email-masked]>; [email-masked]
Subject: Re: draft-asveren-dispatch-http-overload

Hi Tolga,

My .02 -

> On 7 May 2018, at 5:46 am, Asveren, Tolga <[email-masked]<mailto:[email-masked]>> wrote:
>
> To regroup:
>

Cheers,

> To: 'Julian Reschke' <[email-masked]<mailto:[email-masked]>>; Martin Thomson <[email-masked]<mailto:[email-masked]>>

---

From: "Asveren, Tolga" <[email-masked]>
Date: Thu, 24 May 2018 19:58:24 +0000

After a brief hiatus I would like to repeat my questions:

i- Is there any existing mechanism (503 with Retry-After does not provide a satisfactory solution) which would address the server/application overload problem? The answer, still, seems to be “No” IMHO.

ii- Should the WG spend cycles to address this problem (I, obviously, think it should and am willing to continue editing the draft)?

Replies are highly appreciated.

Thanks,
Tolga

From: Asveren, Tolga
Sent: Monday, May 14, 2018 5:52 AM
To: 'Mark Nottingham' <[email-masked]>
Cc: Julian F. Reschke <[email-masked]>; Martin Thomson <[email-masked]>; [email-masked]
Subject: RE: draft-asveren-dispatch-http-overload

Hi Mark,

Thanks for the reply.

                            +----------------+
                            |                |
                            |   Signature    |
                            |   Server       |
                            |                |
                            |                |
                            +----------------+
                             ^           |
                             |           |
                           POST         200
                         (JSON body   (JSON body

Thanks,
Tolga

From: Mark Nottingham <[email-masked]<mailto:[email-masked]>>
Sent: Monday, May 14, 2018 3:36 AM
To: Asveren, Tolga <[email-masked]<mailto:[email-masked]>>
Cc: Julian F. Reschke <[email-masked]<mailto:[email-masked]>>; Martin Thomson <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: draft-asveren-dispatch-http-overload

Hi Tolga,

My .02 -

Cheers,