Subject: Re: [Simple] Suggested next actions with draft-ietf-simple-interdomain-scaling-analysis

From: "Bernard Aboba" <[email-masked]>
Date: Tue, 1 Dec 2009 15:16:11 -0800

The problems of this document go beyond the issue of "better numbers".  The
fundamental goal of scaling analysis is to elucidate the factors that
influence performance.   The document as currently written does not do this.
As Cullen noted in his review:

In summary, you can see I don't think the document actually shows most the

things it claims in the conclusion. I believe that it would be very harmful
to

publish the document as is. The numbers will be used by folks to justify
work

that is not needed and claim that other protocol are 10x better than SIMPLE.
I

don't mind claims like that if they are true, but working off these numbers
I

would not feel they are true.

My suggestion is that the document be re-organized along the lines of Peter
St. Andre's XMPP scaling document:

http://tools.ietf.org/html/draft-saintandre-xmpp-presence-analysis

Peter's document (while containing plenty of calculations)  first lays out
the formulas and assumptions that go into the numbers.  This enables the
reader to understand and if necessary, modify those assumptions to fit their
needs. 

By re-organizing the document along these lines (possibly moving
calculations of specific scenarios to an Appendix),  more emphasis would be
placed on the underlying performance model and how this is influenced by
various protocol extensions and capabilities, rather than on the assumptions
being made in the calculation of specific scenarios.    

While it's ok to attempt to improve the realism of the assumptions as they
stand, this won't address the more fundamental problems of logical structure
and readability which are the heart of the concerns that have been expressed
so far.  

Avshalom said:

"This did not get much attention when it was sent before the IETF. 

We really need some input from actual deployments of SIMPLE regarding the 
following in order to be able to address the IESG comment in a new 
revision of this draft: 

- Message sizes.

- Different refresh (re-subscribe) rates used and when they are used.

- Presence document data sizes

- Geolocation presence document data sizes.

- Size of data per subscription that the presence server needs to 
  allocate 

- Any other data that we can get from the field.

Any other comments are also welcome. 

Thanks
--Avshalom

__________________ 

The IESG had substantial comments on the draft: 
 
<https://datatracker.ietf.org/idtracker/draft-ietf-simple-interdomain-scalin
g-analysis/comment/102863/>
https://datatracker.ietf.org/idtracker/draft-ietf-simple-interdomain-scaling
-analysis/comment/102863/ 

The following is a plan for revising the draft to address these comments. 
The plan is based on discussions with Ben, Cullen and Robert. 

One very important thing that will help us a lot in this work is getting 
real data from actual deployments of SIMPLE regarding the following: 

- Message sizes.

- Different refresh (re-subscribe) rates used and when they are used.

- Presence document data sizes

- Geolocation presence document data sizes.

- Size of data per subscription that the presence server needs to 
  allocate 

- Any other data that we can get from the field.

**** Any numbers here will be greatly appreciated and will help a lot 
in creating the new revision of the document **** 

Note that the above numbers are more SIP specific and we need to get  
them from actual SIP deployments. They are different from the  
general constants of buddy list size, rate of state changes per day etc. 
which are more usage numbers.

General Changes:
----------------

* Remove subjective language as much as possible

* The document should not be written as a protocol comparison document 
but as a document that provides input that will help deployment  
considerations for SIMPLE and directions for optimizations.

* Clarify the assumptions behind the chosen input values.

* Discuss elasticity of inputs and which inputs have more effect.

* Make it clear that we are talking only about inter-domain  
traffic.

* The document will have to go through the working group last call  
again.

Changes in Phase I
-------------------

* Separate the scenarios into enterprise and consumer related  
scenarios. Number as the time that a user will be logged in which is 
reasonable to be 8 in an enterprise will be different for a consumer 
network. The rate of status changes will be different also.

* Since we can not mention explicit numbers from actual deployments,  
we should say e.g. "hypothetical consumer service"...

* Add per active (logged in) user calculations

* Section 2.2: The 2000/4000 difference in login/logouts is probably  
a typo mistake that needs to be clarified. It does NOT affect the  
calculations.

* Add a section describing changes with different constants. Provide  
several points on the possible reasonable values.

* Add geolocation information as part of the model. 

* Provide different calculations for the refresh time. The 3600  
seconds from the RFC is not the number that is usually used. 
Provide both extremes.

* Message sizes: Need to do sanity check of messages. Talk to people  
that may have these numbers.

* Section 2.10 - Change to 48 changes per day from 24. It was done  
in order to show that the optimized protocol can be better even with 
doubling the number notifications. This is probably confusing and need to 
be fixed.

* Sizes for state calculations. Need to revisit the numbers that are  
used. Get actual numbers from the same sources that we will get message 
sizes.

* Talk about the limits that are imposed by privacy constraints on  
the way that SIMPLE works and possible optimizations.

* Provide numbers for when SIGCOMP compression is used.

Phase II
--------

* Revisit Processing Complexities, Current Optimizations, Summary and
Conclusions sections after doing the changes in the main part of the  
document.

Thanks
--Avshalom"

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 01 Dec 2009 22:01:42 -0700

On 12/1/09 4:16 PM, Bernard Aboba wrote:

> My suggestion is that the document be re-organized along the lines of
> Peter St. Andre’s XMPP scaling document:
> 
> Peter’s document (while containing plenty of calculations)  first lays
> out the formulas and assumptions that go into the numbers.  This enables
> the reader to understand and if necessary, modify those assumptions to
> fit their needs.
> 
> calculations of specific scenarios to an Appendix),  more emphasis would
> be placed on the underlying performance model and how this is influenced
> by various protocol extensions and capabilities, rather than on the
> assumptions being made in the calculation of specific scenarios.    
> 
> While it’s ok to attempt to improve the realism of the assumptions as
> they stand, this won’t address the more fundamental problems of logical
> structure and readability which are the heart of the concerns that have
> been expressed so far. 

I reiterate my interest in syncing the two analyses and providing
realistic profiles of presence usage, as previously expressed on this
list. [1][2][2][4][5] Although it's been almost two years since I've
updated draft-saintandre-xmpp-presence-analysis, I would be happy to
make further changes for the sake of consistency with
draft-ietf-simple-interdomain-scaling-analysis.

Peter

[1] http://www.ietf.org/mail-archive/web/simple/current/msg07226.html
[2] http://www.ietf.org/mail-archive/web/simple/current/msg07489.html
[3] http://www.ietf.org/mail-archive/web/simple/current/msg07650.html
[4] http://www.ietf.org/mail-archive/web/simple/current/msg07685.html
[5] http://www.ietf.org/mail-archive/web/simple/current/msg07697.html

---

From: Avshalom Houri <[email-masked]>
Date: Wed, 2 Dec 2009 09:09:49 +0200

Section 2.2 and 2.3  contain plenty of explanations regarding the 
assumptions behind the document, but I will try to make them less "dry" 
and more explanatory.

Peter's document actually took this document as a basis.

Maybe you mean that it should not be assumed that the readers are familiar 
with the way that SIMPLE works?

Note that the IESG comments specifically said that actual field numbers 
are necessary and we should try to get them as much as possible. I have 
submitted a request to the list for these kind of numbers and got almost 
no response. Hope that someone will be able to provide some field numbers.

--Avshalom

From:
"Bernard Aboba" <[email-masked]>
To:
<[email-masked]>
Date:
02/12/2009 01:16 AM
Subject:
Re: [Simple] Suggested next actions with 
draft-ietf-simple-interdomain-scaling-analysis
Sent by:
[email-masked]

The problems of this document go beyond the issue of ?better numbers?. The 
fundamental goal of scaling analysis is to elucidate the factors that 
influence performance.   The document as currently written does not do 
this.  As Cullen noted in his review:
 
In summary, you can see I don't think the document actually shows most the
things it claims in the conclusion. I believe that it would be very 
harmful to
publish the document as is. The numbers will be used by folks to justify 
work
that is not needed and claim that other protocol are 10x better than 
SIMPLE. I
don't mind claims like that if they are true, but working off these 
numbers I
would not feel they are true.
 
My suggestion is that the document be re-organized along the lines of 
Peter St. Andre?s XMPP scaling document:
http://tools.ietf.org/html/draft-saintandre-xmpp-presence-analysis
 
Peter?s document (while containing plenty of calculations)  first lays out 
the formulas and assumptions that go into the numbers.  This enables the 
reader to understand and if necessary, modify those assumptions to fit 
their needs. 

While it?s ok to attempt to improve the realism of the assumptions as they 
stand, this won?t address the more fundamental problems of logical 
structure and readability which are the heart of the concerns that have 
been expressed so far. 

Avshalom said:
 
?This did not get much attention when it was sent before the IETF. 

- Message sizes.

Thanks
--Avshalom

__________________ 

The IESG had substantial comments on the draft: 
https://datatracker.ietf.org/idtracker/draft-ietf-simple-interdomain-scaling-analysis/comment/102863/ 

- Message sizes.

General Changes:
----------------

Changes in Phase I
-------------------

Phase II
--------

Thanks
--Avshalom?_______________________________________________
Simple mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/simple

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 08 Dec 2009 10:17:29 -0700

Avshalom and I just had a chat about this I-D. He might post in more
detail, but my main conclusions are as follows:

1. Analyzing the network impact of presence technologies is the
responsible thing for the IETF to do.

2. The analyses in draft-ietf-simple-interdomain-scaling-analysis and
draft-saintandre-xmpp-presence-analysis are not intended to force work
on optimizations, instead they are intended to frame a broader
discussion about the network impact of presence technologies.

3. It's unproductive to set up a competition in this regard between SIP
and XMPP.

4. It would be more productive to move up a level of abstraction by
discussing presence technologies at a higher level (not specific to SIP
or XMPP).

5. A more general treatment might help protocol designers, application
developers, and service operators think more clearly about protocol
optimizations, implementation improvements, and deployment sizing.
However, detailed discussion of those issues belongs somewhere else, not
in the general analysis document.

If people on this list think it would be helpful, Avshalom and I are
open to working together on a more general analysis.

Peter

On 12/2/09 12:09 AM, Avshalom Houri wrote:
> Maybe you mean that it should not be assumed that the readers are
> familiar with the way that SIMPLE works?
> 
> From: 	"Bernard Aboba" <[email-masked]>
> To: 	<[email-masked]>
> Date: 	02/12/2009 01:16 AM
> Subject: 	Re: [Simple] Suggested next actions with      
> Sent by: 	[email-masked]
> 
> 
> ------------------------------------------------------------------------
> 
> 
> 
> The problems of this document go beyond the issue of “better numbers”.
>  The fundamental goal of scaling analysis is to elucidate the factors
> that influence performance.   The document as currently written does not
> do this.  As Cullen noted in his review:
>  
> *In summary, you can see I don't think the document actually shows most
> the*
> *things it claims in the conclusion. I believe that it would be very
> harmful to*
> *publish the document as is. The numbers will be used by folks to
> justify work*
> *that is not needed and claim that other protocol are 10x better than
> SIMPLE. I*
> *don't mind claims like that if they are true, but working off these
> numbers I*
> *would not feel they are true.*
>  
> _http://tools.ietf.org/html/draft-saintandre-xmpp-presence-analysis_
>  
> “This did not get much attention when it was sent before the IETF.
> 
> The IESG had substantial comments on the draft: _
> __https://datatracker.ietf.org/idtracker/draft-ietf-simple-interdomain-scaling-analysis/comment/102863/_
> 
> 
> --Avshalom”_______________________________________________
> Simple mailing list
> [email-masked]
> ------------------------------------------------------------------------

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 08 Dec 2009 12:40:00 -0500

Peter,

I think the general analysis would be a useful place to go next.
That should indeed lead to the discussion of issues. It seems very 
likely that there will need to be some architectural changes to arrive 
at a system that can ultimately support a planet's worth of people who 
each want the status of a bunch of buddies.

	Thanks,
	Paul

Peter Saint-Andre wrote:
> _______________________________________________
> Simple mailing list
> [email-masked]