Subject: Re: [Atoca] Requirement D2: "Large Audience"

From: "DALY, BRIAN K (ATTCINW)" <[email-masked]>
Date: Sun, 16 Jan 2011 22:35:51 -0800

That would have to be dependent on the regional requirement and delivery mechanism. For example in the US it is not a requirement for EAS or CMAS nor can it be provided.
Brian K. Daly
   Sent to you by AT&T... 
   America's Fastest Mobile Broadband
       Network.  Rethink Possible.
   +1.425.580.6873
-------
Sent from my Blackberry

----- Original Message -----
From: [email-masked] <[email-masked]>
To: Art Botterell <[email-masked]>; Hannes Tschofenig <[email-masked]>
Cc: [email-masked] <[email-masked]>
Sent: Sun Jan 16 21:12:05 2011
Subject: Re: [Atoca] Requirement D2: "Large Audience"

Interesting tidbit from the Australian warning system discussions: the originating systems (the first relay) provides a report to the emergency authority on the number of deliveries.  That suggests that feedback occurs at some level.

I don't know whether this sort of reporting is something that would be part of a requirement, but I'll try to find out.

--Martin

On 2011-01-17 at 14:20:37, Art Botterell wrote:
> Mainly I was just suggesting one way of looking at the "how large is
> 'large'" question.  Although it does highlight some of Martin's
> questions about how to adapt a session initiation protocol to a one-to-
> very-many application, if that's what we have in mind.  As he rightly
> points out, responses don't only happen at the application level.
> 
> Thanks!
> 
> - Art
> 
> On Jan 16, 2011, at 11:22 AM, Hannes Tschofenig
> <[email-masked]> wrote:
> 
> > Hi Art,
> >
> > that's an interesting comment particularly since we do not have a
> requirement for responses by receivers (or Recipients) of alert
> messages. I am not sure to whom they would sent their response to. To
> the originator, to some gateway, etc?
> >
> > Ciao
> > Hannes
> >
> > On Jan 15, 2011, at 11:58 PM, Art Botterell wrote:
> >
> >> One difference in "broadcast" delivery is that it's frequntly
> impractical and/or undesirable to try to track individual deliveries or
> process individual responses.
> >>
> >> Thanks!
> >>
> >> - Art
> >>
> >> On Jan 15, 2011, at 11:30 AM, Hannes Tschofenig
> <[email-masked]> wrote:
> >>
> >>> Hi all,
> >>>
> >>> during the ATOCA meeting at IETF#79 Igor raised a question
> regarding requirement D2:
> >>> "
> >>> Req-D2:
> >>>
> >>>    The protocol solution MUST allow delivery of messages
> >>>    simultaneously to a large audience.
> >>> "
> >>>
> >>> He was asking whether there is a need to put a number for large
> audience there.
> >>>
> >>> Maybe Igor has a specific number in mind that would matter from a
> protocol design perspective but I do not have such a number that would
> provide a meaningful distinction.
> >>>
> >>> There is, however, the question what types of problems we expect
> with very large number of alert message receivers and how we document
> this aspect in the requirements document.
> >>>
> >>> Suggestions? Thoughts?
> >>>
> >>> Ciao
> >>> Hannes
> >>>
> >>> _______________________________________________
> >>> earlywarning mailing list
> >>> [email-masked]
> >>> https://www.ietf.org/mailman/listinfo/earlywarning
> >
> _______________________________________________
> earlywarning mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/earlywarning

_______________________________________________
earlywarning mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/earlywarning

---

From: <[email-masked]>
Date: Mon, 17 Jan 2011 11:03:21 -0000

Hello chaps,

Yes, it's only possible to have any kind of delivery reports regarding
specific terminals on a one-to-one kind of unicast delivery. The Australians
were not considering Cell Broadcast or any other kind of multicast
technology as they have selected a SMS based delivery system, so this is
probably what they had in mind. I noticed that atoca language about this is
reserved mostly for the 'subscription push' kind of distribution.  We need
to take care not to make it look like a mandated requirement for
broadcast/multicast bearers. 

IMHO, the other special problems with multicast,  mean that it needs
different and special requirements and cannot be generalized with other
'push' or 'pull'  technologies quite so easily, tempting though that is. I
think this problem happens because most IT professionals are familiar with
TCP and UDP but rarely use multicast so are unfamiliar with its very
different attributes.

Many vendors offer systems which do have a positive indication of who got
what,  and this is a good thing for applications of up to about 10,000
receivers. Netherlands studies showed that by the time you scale up to
millions,  the mass scale of all this becomes a problem, so large scale
systems will have to reluctantly abandon the notion of  individual
confirmation.

Besides, the police authority of a large city don't have the time, during an
emergency, to check out millions of replies nor take any action if someone
does not reply (they are too busy). On the other hand a smaller community
may value such a facility, so it seems that they authority will need to
choose whatever is appropriate for the circumstances at their discretion.
There is no 'one size fits all' solution so we will have to blend many
bearers in any real situation.  For example the Norwegians take the opposite
view.

However, it is possible to confirm that the multicast was transmitted, as a
network of feedback receivers is envisaged in conjunction with an outer loop
reporter system, which would report of the message was transmitted on air.
If it did not, then an alarm would be generated by the originating
aggregator for the log file. So the positive indication of the fact of the
transmission over the air is satisfied even though individual delivery is
not. 

In summary, IMHO,  Broadcast/multicast bearers will need special treatment
as they don't neatly conform to the classic notions of 'push' or 'pull'. 

Warm regards, Mark  Wood.

---

From: Brian Rosen <[email-masked]>
Date: Mon, 17 Jan 2011 09:44:02 -0500

It may be, but I'd like to explore this a bit anyway.

Millions of messages (acknowledgements) is a scale we can deal with today.  Hundreds of millions is probably beyond what we can deal with in a response to a very large alert.

Most systems consist of several smaller subsystems.  The purpose of an acknowledgement is to make sure everyone got the message.  If the subsystem can determine that every one of its clients got it, it can report that up the line.  It can save missed acks for later analysis, or if there are few enough of them, report them up.

This means messages national scale which have small effectivity times can't reasonably ask for message acknowledgement.  Anything smaller than that probably can.

Since most alerts really don't involve hundreds of millions of notifications, most alerts probably can ask for them.

If your delivery mechanism is multicast, the multicast mechanism itself doesn't track who gets the alert in any way we can use.  That implies something else is tracking who gets the alert, a complication that could loom large.  Some systems do know who gets the alert (sometimes because it knows who it is connected to, and all of them get the alert).  Certainly, anything with a subscription has the characteristic that the sender knows who all the recipients are.  

It's VERY valuable to know that every entity that should get the alert got it.  The only other mechanism we have is some repeating of the sending in the hopes that everyone got it.  In some cases you may have more than one "path" to the same recipient.  That might be multiple devices, multiple services, or multiple logical or physical connections.  You may try one first, and if that doesn't get an ack, try another.  Although we often think of this mechanism as needing no more than seconds to deploy, in fact many alerts would be fine with a few minutes, and trying some things sequentially may make sense.

So, yes, probably a Tsunami alert to all of East Asia can't ask for acknowledgements.  An "Amber Alert" (possible abducted child) to a county might very well.  Certainly, a snow emergency closing to the parents of an elementary school could.

Brian

---

From: "Richard L. Barnes" <[email-masked]>
Date: Mon, 17 Jan 2011 10:41:25 -0500

Just one brief comment: Mark's statement that "it's only possible to have any kind of delivery reports regarding specific terminals on a one-to-one kind of unicast delivery" is not necessarily true.  The availability of delivery reports depends entirely on the capabilities of the terminals, and not the intervening access network.  In particular, terminals can send back delivery reports regardless of whether they are sending or receiving over broadcast or unicast.   

This pattern could actually be quite efficient in certain layer-2 networks that have broadcast downstream, unicast upstream (e.g., PON, DSL, maybe Cable IIRC).  A notification to an entire subnet takes one message, and all acknowledgements can be received in parallel.

--Richard

---

From: "Peter Sanders" <[email-masked]>
Date: Mon, 17 Jan 2011 17:04:28 +0100

Hi Brian and Mark,

Brian, you state in your email the following:
"It's VERY valuable to know that every entity that should get the alert got
it"

The authorities may not know who in particular needs to receive the alert
message. They wish that everyone in a specific area receives it. However,
the authorities may have no idea who is in the target area at that moment,
so they can't ask for confirmation; they can, but they don't know if they
miss a few acks.

Finding out who is in a specific area is technically possible, but may
generate far more network traffic than the acks you expect from a few
million people, and it takes time too. 

I just wanted to add these remarks to the discussion.

Best regards,
Peter

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
On Behalf Of Brian Rosen
Sent: maandag 17 januari 2011 15:44
To: [email-masked]@engineer.com
Cc: [email-masked]
Subject: Re: [Atoca] Requirement D2: "Large Audience"

Millions of messages (acknowledgements) is a scale we can deal with today.
Hundreds of millions is probably beyond what we can deal with in a response
to a very large alert.

Most systems consist of several smaller subsystems.  The purpose of an
acknowledgement is to make sure everyone got the message.  If the subsystem
can determine that every one of its clients got it, it can report that up
the line.  It can save missed acks for later analysis, or if there are few
enough of them, report them up.

This means messages national scale which have small effectivity times can't
reasonably ask for message acknowledgement.  Anything smaller than that
probably can.

Since most alerts really don't involve hundreds of millions of
notifications, most alerts probably can ask for them.

If your delivery mechanism is multicast, the multicast mechanism itself
doesn't track who gets the alert in any way we can use.  That implies
something else is tracking who gets the alert, a complication that could
loom large.  Some systems do know who gets the alert (sometimes because it
knows who it is connected to, and all of them get the alert).  Certainly,
anything with a subscription has the characteristic that the sender knows
who all the recipients are.  

It's VERY valuable to know that every entity that should get the alert got
it.  The only other mechanism we have is some repeating of the sending in
the hopes that everyone got it.  In some cases you may have more than one
"path" to the same recipient.  That might be multiple devices, multiple
services, or multiple logical or physical connections.  You may try one
first, and if that doesn't get an ack, try another.  Although we often think
of this mechanism as needing no more than seconds to deploy, in fact many
alerts would be fine with a few minutes, and trying some things sequentially
may make sense.

So, yes, probably a Tsunami alert to all of East Asia can't ask for
acknowledgements.  An "Amber Alert" (possible abducted child) to a county
might very well.  Certainly, a snow emergency closing to the parents of an
elementary school could.

Brian

---

From: <[email-masked]>
Date: Mon, 17 Jan 2011 17:30:25 -0000

Yes, I stand corrected, Thanks Richard.

-----Original Message-----
From: Richard L. Barnes [mailto:[email-masked]]
Sent: Monday, January 17, 2011 3:41 PM
To: Brian Rosen
Cc: <[email-masked]>; [email-masked]
Subject: Re: [Atoca] Requirement D2: "Large Audience"

Just one brief comment: Mark's statement that "it's only possible to have
any kind of delivery reports regarding specific terminals on a one-to-one
kind of unicast delivery" is not necessarily true.  The availability of
delivery reports depends entirely on the capabilities of the terminals, and
not the intervening access network.  In particular, terminals can send back
delivery reports regardless of whether they are sending or receiving over
broadcast or unicast.   

This pattern could actually be quite efficient in certain layer-2 networks
that have broadcast downstream, unicast upstream (e.g., PON, DSL, maybe
Cable IIRC).  A notification to an entire subnet takes one message, and all
acknowledgements can be received in parallel.

--Richard

---

From: "mark.wood" <[email-masked]>
Date: Mon, 17 Jan 2011 16:13:13 -0000

-----Original Message-----
From: Richard L. Barnes [mailto:[email-masked]] 
Sent: Monday, January 17, 2011 3:41 PM
To: Brian Rosen
Cc: <[email-masked]>; [email-masked]
Subject: Re: [Atoca] Requirement D2: "Large Audience"

--Richard

> 
> This means messages national scale which have small effectivity times
can't reasonably ask for message acknowledgement.  Anything smaller than
that probably can.
<[email-masked]> wrote:
> 
>> Hello chaps,
>> 
>> Yes, it's only possible to have any kind of delivery reports 
>> regarding specific terminals on a one-to-one kind of unicast 
>> delivery. The Australians were not considering Cell Broadcast or any 
>> other kind of multicast technology as they have selected a SMS based 
>> delivery system, so this is probably what they had in mind. I noticed 
>> that atoca language about this is reserved mostly for the 
>> 'subscription push' kind of distribution.  We need to take care not 
>> to make it look like a mandated requirement for broadcast/multicast
bearers.
>> different and special requirements and cannot be generalized with 
>> other 'push' or 'pull'  technologies quite so easily, tempting though 
>> that is. I think this problem happens because most IT professionals 
>> are familiar with TCP and UDP but rarely use multicast so are 
>> unfamiliar with its very different attributes.
>> 
>> Many vendors offer systems which do have a positive indication of who 
>> got what,  and this is a good thing for applications of up to about 
>> 10,000 receivers. Netherlands studies showed that by the time you 
>> scale up to millions,  the mass scale of all this becomes a problem, 
>> so large scale systems will have to reluctantly abandon the notion of  
>> individual confirmation.
>> 
>> Besides, the police authority of a large city don't have the time, 
>> during an emergency, to check out millions of replies nor take any 
>> action if someone does not reply (they are too busy). On the other 
>> hand a smaller community may value such a facility, so it seems that 
>> they authority will need to choose whatever is appropriate for the
circumstances at their discretion.
>> There is no 'one size fits all' solution so we will have to blend 
>> many bearers in any real situation.  For example the Norwegians take 
>> the opposite view.
>> 
>> However, it is possible to confirm that the multicast was 
>> transmitted, as a network of feedback receivers is envisaged in 
>> conjunction with an outer loop reporter system, which would report of the
message was transmitted on air.
>> aggregator for the log file. So the positive indication of the fact 
>> of the transmission over the air is satisfied even though individual 
>> delivery is not.
>> 
>> In summary, IMHO,  Broadcast/multicast bearers will need special 
>> treatment as they don't neatly conform to the classic notions of 'push'
or 'pull'.

---

From: "Thomson, Martin" <[email-masked]>
Date: Tue, 18 Jan 2011 07:38:21 +0800

On 2011-01-18 at 01:44:02, Brian Rosen wrote:
> acknowledgement is to make sure everyone got the message.  If the
> subsystem can determine that every one of its clients got it, it can
> report that up the line.  It can save missed acks for later analysis,
> or if there are few enough of them, report them up.

I suspect that this sort of delegation system is one that can scale.  Having the originator collect all the replies probably wont.  Of course, this all depends on the exact circumstances, as you observe.

--Martin

---

From: <[email-masked]>
Date: Tue, 18 Jan 2011 11:49:39 -0000

Thanks Brian,

Brian's point is well made and correct, thanks Brian, but there are some
special issues that need to be borne in mind with some bearers.

I first came into this project with the brief from the UN  to "protect the
(Mobile) mobile networks from catastrophic overload situations during
disasters". 
When I "did the numbers" I discovered that the real problem is not what many
think. The bottleneck is in the mobility management system
(HLR/VLR/Paging/Access grant). EU sponsored Studies by Prof Sophocles
Kiriazakos  for the Greek government after the Athens earthquake and
subsequent crash of the Greek networks, confirmed this.  This is what lead
me to work on cell broadcast (which does not use the mobility management
system at all).

I agree that it is reasonable to allow that the acknowledgments may indeed
take much longer to 'traffic' than the outbound multicast message. Obviously
the scale is the same both ways, but while the latency is critical going
forward, the reverse path is not in the least bit time critical, so
relatively slower 'best effort'  bearers would be fine. The server which is
wishing to know its subscribers got a message may send the message both by
unicast and multicast (as mine do), but inevitably the acknowledgement will
have to be unicast. There is no specific problem in allowing the
acknowledgement of a multicast message by unicast means as long as we
understand that the latency is indeterminate. (However since it's not clear
when the ack may come, I send the message by both means simultaneously
without waiting for acks.) 

My concern is really for the Mobile Network, at layer 2. 
For example if a large number of terminals all receive a multicast at the
same time, then they will all want to acknowledge at the same time. This
will result is a tsunami wave of random access bursts to the cells uplink
timeslot, MSC call set up load, 'channel allocation algorithm' threads and
SDCCH allocation attempts. Then there will be huge load on the SMS gateways.
Mobiles that don't get an access grant message will obviously try again but
for a while the whole mobility management system will be significantly
loaded. This affects circuit switched voice just as much because the
mobility management system is common for voice and SMS, (but maybe not
GPRS?). Recall that in cellular network design, erlang calculations are done
such that it's the assumption that only a small fraction of terminals will
make random access burst attempts at any one time, so the mobility
management system is designed for this load only.

In other words, consider that a public warning message (such as a USA  CMAS
presidential message) will reach 100% of terminals simultaneously, rather
than the small percentage that the signaling system can cope with. This is
why both the CMAS and ETSI standards intentionally disallow embedded numbers
or URLs for large scale (Public) warnings.

So in fact the "scale" of the problem may not be as significant as the
impact on the local infrastructure (such as a cell). Maybe "scale" is a less
important factor than, let's say, penetration?  

On the other hand a smaller scale (of penetration)  message would not have
such a profound impact. So in some cases it may be reasonable to expect
acknowledgements in 'best effort'  time. Norway, for example,  likes this
approach.

I am unclear as to if IP systems have such problems because there is not a
'stateful'  mobility management system in the core and though acks are on a
large scale, they represent very small packets of less than 1K each. Maybe
the problem will go away in the future? Any comments on that?

Warm regards Mark Wood DRCF.

-----Original Message-----
From: Brian Rosen [mailto:[email-masked]] 
Sent: Monday, January 17, 2011 2:44 PM
To: <[email-masked]>
Cc: [email-masked]
Subject: Re: [Atoca] Requirement D2: "Large Audience"

Brian

---

From: Brian Rosen <[email-masked]>
Date: Tue, 18 Jan 2011 09:02:45 -0500

I understand your concern.  As I stated, in really, really large events, it may not be practical to get confirmation.  On the other hand, when you have burst events that cause overload, we have developed techniques for dealing with them.  This specific problem is probably best addressed by a random backoff delay.  The devices, upon receiving a broadcast wait a random time before attempting a reply, and if they encounter busy conditions, increase the range of the delay and delay another random time.

Brian

---

From: "James M. Polk" <[email-masked]>
Date: Tue, 18 Jan 2011 16:33:26 -0600

Peter

I believe that is true in push based alerting, but not in 
subscription based alerting. If SIP is used, and each end-system that 
wants to receive a certain alert SUBSCRIBEs to the alerting service, 
then the alerting service will NOTIFY the end-system that wanted to 
receive the alert, with a 200 OK as the return acknowledgement that 
the NOTIFY with the CAP message/container (or whatever is ultimately 
chosen to tell the receiver what's happening) would be the confirmation.

This is all predicated on the use of SIP for this discussion.

As Brian effectively points out, if there are many alerting servers 
alerting a subset of the subscribed end-systems, this scales far 
better, and I bet it doesn't need to be coalesced in real time for 
any reports. Further, the 200 OKs are far smaller in size than the 
alert message itself, so the traffic is far less in the upstream direction.

James

---

From: "James M. Polk" <[email-masked]>
Date: Tue, 18 Jan 2011 16:51:01 -0600

Brian

Are you taking into account the difference in the packet size of the 
NOTIFY vs. the 200 OK? I believe any burst is the worry of the 
downstream capacity, and not the 200 OKs in the upstream.

That said, what events are we taking about accounting for?

- Local tornado/flooding alert? (affecting a creek or the eastern 
half of Australia?)
- metro amber alert? (backwoods sheriff's department or US Tri-state area?)
- hurricane alert? (Canary Islands or Katrina sized?)
- earthquake alert? (California dessert or Japan?)
- limited scale tsunami alert? (only Sumatra or the Indian Ocean?)
- ocean scale tsunami alert?

I think scoping - with examples - could serve some use in these 
discussions, and not just refer to them in the abstract.

Much of these will have vastly different scaling issues and design 
considerations - or will they?

We could go down the path of scaling regardless of the type of event 
that caused the alert. In other words, for a

    #1 - alerting <500 end-systems, do X

    #2 - alerting <10,000 end-systems, do Y

    #3 - alerting <500,000 end-systems, do Z

...all the way up to at least the ~140M that are in Japan because 
they have had events that would have alerted that many within the 
last 30 years.

Obviously, as the number of end-systems subscribe for a certain 
notification, the scale of the delivery system becomes more 
distributed to handle the load of the alerts and confirmations.  Our 
documents really ought to recommend ways to aggregate large scale 
confirmations too.

just a couple of thoughts

James

At 08:02 AM 1/18/2011, Brian Rosen wrote:
>I understand your concern.  As I stated, in really, really large 
>events, it may not be practical to get confirmation.  On the other 
>hand, when you have burst events that cause overload, we have 
>developed techniques for dealing with them.  This specific problem 
>is probably best addressed by a random backoff delay.  The devices, 
>upon receiving a broadcast wait a random time before attempting a 
>reply, and if they encounter busy conditions, increase the range of 
>the delay and delay another random time.
>
>Brian
>