Subject: [PCN] rate measurement capabilities of today's routers

From: Michael Menth <[email-masked]>
Date: Tue, 12 Jun 2007 16:27:28 +0200

Hi all,

some proposals for a PCN architecture require the measurement of the 
rate of marked packets per ingeress-egress-aggregates at the egress 
nodes. Is that currently feasible in today's routers? If so, what is the 
minimum measurement interval (before EWMA smoothing is performed)?

Kind regards,

    Michael

-- 
Dr. Michael Menth, Assistant Professor
University of Wuerzburg, Institute of Computer Science
Am Hubland, D-97074 Wuerzburg, Germany, room B206
phone: (+49)-931/888-6644, fax: (+49)-931/888-6632
mailto:[email-masked]
http://www3.informatik.uni-wuerzburg.de/research/ngn

_______________________________________________
PCN mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pcn

---

From: Fred Baker <[email-masked]>
Date: Tue, 12 Jun 2007 08:27:35 -0700

On Jun 12, 2007, at 7:27 AM, Michael Menth wrote:

> some proposals for a PCN architecture require the measurement of  
> the rate of marked packets per ingeress-egress-aggregates at the  
> egress nodes. Is that currently feasible in today's routers? If so,  
> what is the minimum measurement interval (before EWMA smoothing is  
> performed)?

We have been doing it for a number of years. I'd suggest you look on  
www.cisco.com for our product specifications.

---

From: Delei Yu <[email-masked]>
Date: Thu, 14 Jun 2007 15:13:57 +0800

I have a question.
At the Egress node of PCN domain, ingress-egress-aggregates traffic rate
shall be measured. So an Egress node has to know which part of the traffic
to the Egress is from a particular Ingress node. I mean, there is a mapping
relation at the Egress node:
Ingress		Traffic
A			Traffic 1
			Traffic 2
			Traffic 3
B			Traffic 4
			Traffic 5
How to get this mapping relation? Configured by NMS? Using RSVP?

> sender: Fred Baker [mailto:[email-masked]]
> Re: [PCN] rate measurement capabilities of today's routers
> 
>

---

From: Fred Baker <[email-masked]>
Date: Thu, 14 Jun 2007 07:41:59 -0700

I believe that PCN is intended to not use a signaling protocol.

I don't see an obvious was to identify that actual ingress at the  
egress apart from the use of some form of tunnel (MPLS etc) from the  
ingress, which in the case would be "the other end of the tunnel". We  
don't mark traffic at ISP ingress (although we probably should,  
IMHO), and in a transit situation the traffic may have entered the  
network in a variety of places.

---

From: Delei Yu <[email-masked]>
Date: Fri, 15 Jun 2007 09:51:29 +0800

> I believe that PCN is intended to not use a signaling protocol.
[Delei] I'm not sure about it.
> network in a variety of places.
[Delei] Tunnel is good solution for this problem. But what about in a
non-tunnel scenario? In a route domain, How does egress node know a flow is
come from ingress node A or B?

Original information:
> I have a question.
> At the Egress node of PCN domain, ingress-egress-aggregates traffic 
> rate shall be measured. So an Egress node has to know which part of 
> the traffic to the Egress is from a particular Ingress node. I mean, 
> there is a mapping relation at the Egress node:
> Ingress		Traffic
> A			Traffic 1
> 			Traffic 2
> 			Traffic 3
> B			Traffic 4
> 			Traffic 5

---

From: Fred Baker <[email-masked]>
Date: Fri, 15 Jun 2007 04:44:47 -0700

As I said, in a network that doesn't connect ingress to egress, that  
can be hard to know.

---

From: Tom Taylor <[email-masked]>
Date: Fri, 15 Jun 2007 08:00:11 -0400

There may be some wiggle room in the charter -- the WG is allowed to 
provide requirements to other WGs.

Fred Baker wrote:
> I don't see an obvious was to identify that actual ingress at the egress 
> apart from the use of some form of tunnel (MPLS etc) from the ingress, 
> which in the case would be "the other end of the tunnel". We don't mark 
> traffic at ISP ingress (although we probably should, IMHO), and in a 
> transit situation the traffic may have entered the network in a variety 
> of places.
> 
...

---

From: "Jozef Babiarz" <[email-masked]>
Date: Sun, 17 Jun 2007 23:49:35 -0400

I'm not sure if I understand the question, but here goes.
There are several different scenarios for PCN. Currently the charter
focus on defining the metering and marking approach and its use in
single domain, network edge to network edge deployment. PCN requires
that the marking information observed at the egress network node somehow
gets back to the correct ingress network node. For non-tunnel flows, one
approach is to use signalling between ingress and egress nodes to
establish and exchange flow information. My understanding is that it is
in scope of the PCN charter for the WG to define signalling requirements
for PCN. As well there are deployment models where ingress edge network
node is the IP source and egress is the destination. After
re-chartering, hopefully the WG will work on solution for end-to-end
admission control using PCN.

Regards, Joe
email:[email-masked]
Telephone:613-763-6098

-----Original Message-----
From: Delei Yu [mailto:[email-masked]] 
Sent: June 14, 2007 9:51 PM
To: 'Fred Baker'
Cc: [email-masked]
Subject: RE: [PCN] rate measurement capabilities of today's routers

> don't mark traffic at ISP ingress (although we probably should, IMHO),

> and in a transit situation the traffic may have entered the network in

> a variety of places.
[Delei] Tunnel is good solution for this problem. But what about in a
non-tunnel scenario? In a route domain, How does egress node know a flow
is come from ingress node A or B?

---

From: Steven Blake <[email-masked]>
Date: Tue, 19 Jun 2007 17:10:12 -0400

> I believe that PCN is intended to not use a signaling protocol.

The PCN charter is (intentionally?) vague here:

=  Description of Working Group:
=
=  The Congestion and Pre-Congestion Notification (PCN) working group
=  develops mechanisms to protect the quality-of-service of established
=  inelastic flows within a DiffServ domain when congestion is imminent
=  or existing. These mechanisms operate at the domain boundary, based
=  on aggregated congestion and pre-congestion information from within
=  the domain.

I note that the "S" word is never used in the charter, except to
identify the following milestone:

=  Mar 2008 Submit "Requirements for Signaling of (Pre-)Congestion
=  Information from Egress to Ingress in a DiffServ Domain"
=  to the IESG for consideration as a Informational RFC

> network in a variety of places.

Agreed.  Absent signaled microflows, or some form of ingress->egress
tunneling, then it seems that ACLs on the egress would have to be used.
Not fun.

Regards,

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Steven Blake                <[email-masked]>
Ericsson/Redback Networks               +1 919-472-9913

---

From: András Császár (IJ/ETH)<[email-masked]>
Date: Wed, 20 Jun 2007 08:40:31 +0200

> tunneling, then it seems that ACLs on the egress would have to be

We seem to have the following "options" at the moment:

1) Egress infers ingress from routing and packet header: generally cannot be done. ;-)

2) Some form of signalling. In this case, care must be taken with re-routing situations affecting the egress router itself. In some cases, the egress router itself may change, in which case the ingress router should have some very quick(*) means to update the states in the egress routers! (Preferably before the egress finished measuring the marked packet rate...) Otherwise, the non-re-routed flows will be notified as the egress has no ingress ID for the re-routed flows.

3) Some form of ingress-egress tunnelling. This elliminates the previous problem. Has the disadvantage of tunnell interface/addressing configuration potentially between each pair of edge routers. Unless, some automatic tunnelling mechanism is proposed.

4) ACLs. Has the disadvantage of a lot of configuration. Also, preparing for a change of egress node due to re-routing seems unfeasible.

Some automatic tunnelling essentially piggybacks the ingress ID onto each packet. Couldn't we do it with less overhead? With some IP header option, whatever?

(*) How quick congestion notification/handling is PCN targeting? What is the timescale? I'm asking because for some applications, users are stopping their sessions after a few seconds of bad quality. The routing convergence time should also be taken into account.

BTW, if we are at routing, some fast re-route proposals are able to / need to identify re-routed packets. Couldn't PCN use such extra knowledge as well?

Kind regards,
András

---

From: "Geib, Ruediger" <[email-masked]>
Date: Wed, 20 Jun 2007 09:07:20 +0200

Andras,

|1) Egress infers ingress from routing and packet header: 
|generally cannot be done. ;-)

Without being a routing expert I must say that Intra Domain 
Routing results in every router to store the complete 
topology of  its own domain. Some IP reroute optimisations 
discuss application of this topology awareness. May be a 
closer look is useful?

|2) Some form of signalling. In this case, care must be taken 
|with re-routing situations affecting the egress router itself. 
|In some cases, the egress router itself may change, in which 
|case the ingress router should have some very quick(*) means 
|to update the states in the egress routers! (Preferably before 
|the egress finished measuring the marked packet rate...) 
|Otherwise, the non-re-routed flows will be notified as the 
|egress has no ingress ID for the re-routed flows.

Admission Control should work in the case of rerouting. 
Preemption may not work properly as indeed the ingress can 
only preempt the per flow reservations whose egress nodes 
it knows.

|3) Some form of ingress-egress tunnelling. This elliminates 
|the previous problem. Has the disadvantage of tunnell 
|interface/addressing configuration potentially between each 
|pair of edge routers. Unless, some automatic tunnelling 
|mechanism is proposed.

Tunnels certainly work, but I hope that PCN finds a way to 
get around them.

|4) ACLs. Has the disadvantage of a lot of configuration. Also, 
|preparing for a change of egress node due to re-routing seems 
|unfeasible.
|
|Some automatic tunnelling essentially piggybacks the ingress 
|ID onto each packet. Couldn't we do it with less overhead? 
|With some IP header option, whatever?
|
|(*) How quick congestion notification/handling is PCN 
|targeting? What is the timescale? I'm asking because for some 
|applications, users are stopping their sessions after a few 
|seconds of bad quality. The routing convergence time should 
|also be taken into account.
|
|BTW, if we are at routing, some fast re-route proposals are 
|able to / need to identify re-routed packets. Couldn't PCN use 
|such extra knowledge as well?
|
|Kind regards,
|András
|
|
|_______________________________________________
|PCN mailing list
[email-masked]
|https://www1.ietf.org/mailman/listinfo/pcn
|

---

From: András Császár (IJ/ETH)<[email-masked]>
Date: Wed, 20 Jun 2007 09:30:28 +0200

Hi!

> Without being a routing expert I must say that Intra Domain Routing
> results in every router to store the complete topology of  its own
> domain. Some IP reroute optimisations discuss application of this
> topology awareness. May be a closer look is useful?

In those cases, where the packet is not coming from an other AS and you know the whole topology, this could work.

I'm not so sure whether you can always infer the ingress if within the domain there are (e.g. OSPF) areas.

And if you receive the packet from another network, you cannot always infer the incoming ingress. You can do it if you advertise the destination prefix with BGP through only one incoming router :-)

>> 2) Some form of signalling. In this case, care must be taken with
>> re-routing situations affecting the egress router itself.
>> In some cases, the egress router itself may change, in which case the
>> ingress router should have some very quick(*) means to update the
>> states in the egress routers! (Preferably before the egress finished
>> measuring the marked packet rate...) Otherwise, the non-re-routed
>> flows will be notified as the egress has no ingress ID for the
>> re-routed flows.
> 
> Preemption may not work properly as indeed the ingress can only
> preempt the per flow reservations whose egress nodes it knows.

Without per-flow states in the interior, you cannot rely on admission control immediately after re-routing because the router cannot (normally) differentiate re-routed flows from original flows. In my understanding, specifically after re-routing you have to use pre-emption.

For instance, the problem statement draft writes:
"Pre-Congestion Notification (PCN) investigates the use of per-flow admission control to provide the required service guarantees for the admitted traffic.  While admission control will protect the QoS under normal operating conditions, an additional flow pre-emption mechanism is necessary in the times of heavy congestion (e.g. caused by route changes due to link or node failure)."

Or do I miss something? Can you elaborate on how you think admission control can be used in case of re-routing?

>> 3) Some form of ingress-egress tunnelling. This elliminates the
>> previous problem. Has the disadvantage of tunnell
>> interface/addressing configuration potentially between each pair of
>> edge routers. Unless, some automatic tunnelling mechanism is
>> proposed. 
> 
> Tunnels certainly work, but I hope that PCN finds a way to get around
> them. 

Me, too :-)

---

From: Michael Menth <[email-masked]>
Date: Wed, 20 Jun 2007 09:43:03 +0200

Hi,

> Or do I miss something? Can you elaborate on how you think admission control can be used in case of re-routing?
The gap between the admissible rate (where admission stops) and the 
supportable rate (where termination starts) can be used as backup 
capacity such that no admitted traffic is lost in case of re-routing 
provided that the admissible and supporable rate thresholds of the links 
are properly configured. See:
http://www3.informatik.uni-wuerzburg.de/~menth/Publications/Menth07i.pdf
We covered the fact that traffic is rerouted within the network, but not 
to another egress router. However, this can also be respected.

Regards,

    Michael

---

From: "Geib, Ruediger" <[email-masked]>
Date: Wed, 20 Jun 2007 09:49:29 +0200

Andras,

|>> 1) Egress infers ingress from routing and packet header:
|>> generally cannot be done. ;-)
|> 
|> Without being a routing expert I must say that Intra Domain Routing
|> results in every router to store the complete topology of  its own
|> domain. Some IP reroute optimisations discuss application of this
|> topology awareness. May be a closer look is useful?
|
|In those cases, where the packet is not coming from an other 
|AS and you know the whole topology, this could work.
|
|I'm not so sure whether you can always infer the ingress if 
|within the domain there are (e.g. OSPF) areas.
|
|And if you receive the packet from another network, you cannot 
|always infer the incoming ingress. You can do it if you 
|advertise the destination prefix with BGP through only one 
|incoming router :-)

Thanks, that's a more differentiated statement than your first one.

|>> 2) Some form of signalling. In this case, care must be taken with
|>> re-routing situations affecting the egress router itself.
|>> In some cases, the egress router itself may change, in 
|which case the
|>> ingress router should have some very quick(*) means to update the
|>> states in the egress routers! (Preferably before the egress finished
|>> measuring the marked packet rate...) Otherwise, the non-re-routed
|>> flows will be notified as the egress has no ingress ID for the
|>> re-routed flows.
|> 
|> Admission Control should work in the case of rerouting.
|> Preemption may not work properly as indeed the ingress can only
|> preempt the per flow reservations whose egress nodes it knows.
|
|Without per-flow states in the interior, you cannot rely on 
|admission control immediately after re-routing because the 
|router cannot (normally) differentiate re-routed flows from 
|original flows. In my understanding, specifically after 
|re-routing you have to use pre-emption.

Sorry, I don't understand.

|For instance, the problem statement draft writes:
|"Pre-Congestion Notification (PCN) investigates the use of 
|per-flow admission control to provide the required service 
|guarantees for the admitted traffic.  While admission control 
|will protect the QoS under normal operating conditions, an 
|additional flow pre-emption mechanism is necessary in the 
|times of heavy congestion (e.g. caused by route changes due to 
|link or node failure)."
|
|Or do I miss something? Can you elaborate on how you think 
|admission control can be used in case of re-routing?

If an egress changes, the ingress gets a correct pre-congestion 
indication for the new egress. If RSVP or NSIS is applied, any 
flow asking for admission will be routed to the new egress as 
soon as routing reconverged and the pre-congestion indication 
is available. It will take a few seconds at least until that 
happens and during this time nothing works properly.

It is possible to engineer link loads so 
that the maximum loads caused by re-routing after single 
link failures can be estimated fairly accurate in 
advance. If the admission-rate is set to "normal maximum rate"
and the preemption rate to "maximum estimated load after re-routing",
the QoS of admitted calls doesn't decrease while 
new calls aren't admitted after a single link loss. After a 
catastrophic failure, all bets are off. But let's consider 
standard operation first.

|>> 3) Some form of ingress-egress tunnelling. This elliminates the
|>> previous problem. Has the disadvantage of tunnell
|>> interface/addressing configuration potentially between each pair of
|>> edge routers. Unless, some automatic tunnelling mechanism is
|>> proposed. 
|> 
|> Tunnels certainly work, but I hope that PCN finds a way to get around
|> them. 
|
|Me, too :-)
|

---

From: Fred Baker <[email-masked]>
Date: Wed, 20 Jun 2007 09:59:58 -0700

> 1) Egress infers ingress from routing and packet header: generally  

Yes. You can infer where you would expect it to come in, but not  
where it actually came in.

The canonical example is asymmetric routing in the backbone. Each AS  
accepts some number of BGP advertisements from preceding AS's and  
selects one or more routes to some remote prefix, and then advertises  
the prefix on to other AS's. The AS can infer that a datagram from  
that prefix entered its network from one of the neighboring AS's that  
advertised it. It cannot, at its egress, say *which* of those, only  
that those are the obvious candidates.

But in the case of a spoofed source address (this is what BCP 38 is  
all about), it might have entered from some neighboring system that  
didn't advertise it. On sleepless nights I wonder whether there is an  
AS in the world that originates datagrams from some prefix and sends  
them via an AS that it doesn't advertise the prefix to. I don't know,  
but I hope not.

> re-routing situations affecting the egress router itself. In some  
> cases, the egress router itself may change, in which case the  
> states in the egress routers! (Preferably before the egress  
> finished measuring the marked packet rate...) Otherwise, the non-re- 
> routed flows will be notified as the egress has no ingress ID for  
> the re-routed flows.

see (1). Routing develops a spanning tree toward a prefix from all  
possible ingresses, and it is pretty common for data from any given  
remote prefix to be able to use several of those paths. We call that  
"multipath routing" or "asymmetric routing" depending on whether it  
was intentional or a by-product of differing routing policies. First  
mention in the RFC series is in RFC 46, April 1970.

> previous problem. Has the disadvantage of tunnell interface/ 
> addressing configuration potentially between each pair of edge  
> routers. Unless, some automatic tunnelling mechanism is proposed.

however the tunnel is set up, a packet that comes out of it can be  
inferred to have entered it at the tunnel ingress.

> 4) ACLs. Has the disadvantage of a lot of configuration. Also,  
> preparing for a change of egress node due to re-routing seems  
> unfeasible.

See comment on (1).

> Some automatic tunnelling essentially piggybacks the ingress ID  
> onto each packet. Couldn't we do it with less overhead? With some  
> IP header option, whatever?

I have in the past proposed a trailer - in a network that is 9K  
clean, one could have the ingress router affix one if its IP  
addresses at the tail of each packet that it handles, perhaps the  
address of its ingress interface. It would be ignored by a receiving  
host as it would appear to be in the noise at the end of the packet.  
There are good reasons to not have routers affixing trailers, and  
perhaps better reasons to not have them inserting options or optional  
headers in datagrams that haven't left places for them to do so. But  
this is a possibility that addresses the multipath and spoofing  
problems.

> (*) How quick congestion notification/handling is PCN targeting?  
> What is the timescale? I'm asking because for some applications,  
> users are stopping their sessions after a few seconds of bad  
> quality. The routing convergence time should also be taken into  
> account.

I don't think this is about routing convergence, although routing  
events are possible. In the general case, PCN is about sessions using  
existing routes and adding more traffic to them than some threshold.  
The simulations have been about "so what happens when I'm running  
close to the line and I add a new session" or "...and a session  
changes its characteristics in some way".

On a re-route, one has to believe that one has two disjoint paths  
with M and N sessions on them respectively, re-routing puts N+M>T  
sessions on some portion of one of their paths, and one would really  
like N+M-T of the traffic (either data within the sessions or  
sessions themselves) to go away. All of the sessions will receive the  
same signal (ECN marks or whatever). Presumably all will apply a  
similar algorithm but with varying configuration parameters. What we  
will probably see is a random subset of the N+M sessions go away,  
never less that N+M-T but very possibly more. The question is what  
the statistical distribution is and what mean it is distributed around.

Time scale... I would have to believe that expected timescales are in  
small multiples of RTTs, which on an end-to-end basis would generally  
be in the tens to hundreds of milliseconds. It would take at least  
one RTT to pick up a mark and report it to the sender, and probably  
another RTT for the sender to determine whether it was a one-of event  
or a stable state. If we're talking aobut the sender then reporting  
something to its SIP Proxy etc, the control system executing some  
algorithm, and then potentially replying "take down the call",  
following which there is at least one more RTT for a SIP shutdown  
exchange, we're looking at the equivalent of four RTTs at least.