Subject: [tsvwg] Using intelligent dropping to improve video quality under congestion (draft-lai-tsvwg-normalizer-00.txt)

From: "Toerless Eckert (eckert)" <[email-masked]>
Date: Wed, 10 Jul 2013 11:01:47 -0700

When dealing with congestion for real-time datagram (UDP/RTP) based video flows, one idea
that we think has shown to provide good value is to intelligently drop in the network
preferentially the least important packets within a video flow, for example non-referenced
P-frames, because their loss creates the lowest impact to video quality.

One of the main concerns raised against such schemes was one of fairness. If flows 
would start to indicate different priorities for packets in them, how would we
be able to avoid that flows would indicate all their packets to be of high priority
to shift packet drops to other flows. Or even more importantly: how could we motivate
flows to indicate that some of their packets are low priority without them having to
fear that they would experience more loss than than flows that don't do this.

So, there is one draft that we would like to present (presumably in TSVWG)  explaining
how we think this problem can be solved: draft-lai-tsvwg-normalizer-00.txt

(Actually, this is how we did solve the problem in our implementatin of intelligent dropping,
 so this is not theoretical).

Now granted, the problem of fairness is but one element of building a complete ssytem
around the idea of intelligent dropping, so if folks here on the lists would like to
see for example 

  a) how well can it work in routers to shift packet drops to low-prio packets
  b) whats the evidence that it's good for video quality
  c) How should we do the marking best going forward

Then please let us know. We'd be very interested to provide insight into the parts of
those pieces that we have also worked out or discuss the ones we have not (primarily c ;-)

Thanks!
    Toerless

In-Reply-To: <[email-masked]>

On Sat, Feb 09, 2013 at 03:00:48AM +0000, Cheng-Jia Lai (chelai) wrote:
> Hi all,
> 
> We have submitted a new Internet draft to describe a Normalization Marker (NM) for DiffServ AF PHB groups, based on our work at Cisco. The goal of NM deployment is to create a new incentive for video encoders to generate more discardable packets when using AF PHB in DIffServ. We are looking forward to your review comments.
> 
> Best regards,
> CJ
> 
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] 
> Sent: Friday, February 08, 2013 6:25 PM
> To: Cheng-Jia Lai (chelai)
> Cc: Wenyi Wang (wenywang); Stan Yang (stanyang); Toerless Eckert (eckert); Fred Yip (fyip)
> Subject: New Version Notification for draft-lai-tsvwg-normalizer-00.txt
> 
> 
> A new version of I-D, draft-lai-tsvwg-normalizer-00.txt
> has been successfully submitted by Cheng-Jia Lai and posted to the
> IETF repository.
> 
> Filename:	 draft-lai-tsvwg-normalizer
> Revision:	 00
> Title:		 Normalization Marker for AF PHB Group in DiffServ
> Creation date:	 2013-02-08
> WG ID:		 Individual Submission
> Number of pages: 15
> URL:             http://www.ietf.org/internet-drafts/draft-lai-tsvwg-normalizer-00.txt
> Status:          http://datatracker.ietf.org/doc/draft-lai-tsvwg-normalizer
> Htmlized:        http://tools.ietf.org/html/draft-lai-tsvwg-normalizer-00
> 
> 
> Abstract:
>    This memo describes a Normalization Marker (NM) at DiffServ (DS)
>    nodes to normalize the distribution of DS codepoint (DSCP) markings
>    for an Assured Forwarding (AF) Per-Hop Behavior (PHB) group.  NM is
>    useful for traffic conditioning with Active Queue Management (AQM)
>    when the AF PHB group is deployed for multimedia service classes that
>    carry video packets with advanced codec technologies.  Using NM can
>    offer an incentive that is however missing in today's ecosystem of
>    practical deployment, i.e., when congestion occurs in the AF PHB
>    group, the network should allow a video codec to benefit from its
>    effort of generating finer layers of intra-flow precedence (IFP) with
>    discardable packets in a more balanced distribution.
> 
>                                                                                   
> 
> 
> The IETF Secretariat
> 

-- 
---
Toerless Eckert, [email-masked]
Cisco NSSTG Systems & Technology Architecture
SDN: Let me play with the network, mommy!

---

From: Dave Taht <[email-masked]>
Date: Wed, 10 Jul 2013 12:02:26 -0700

> P-frames, because their loss creates the lowest impact to video quality.

I've begun to look at webrtc of late in the context of fq_codel.

One thing that might work would be ECN marking important frames and
not ECN marking
others. Problem overall is that ecn can be gamed and is largely not
enabled, and so on...

> fear that they would experience more loss than than flows that don't do this.

same gaming problem...

>  so this is not theoretical).

In the case of the fq_codel algorithm I can see a "delayed drop"
mechanism happening,
where upon deciding to drop, it could defer a drop up to a few packets
based on the classification
of the packet, say 3 packets at most. Since flows are already
identified, this is kind of easy
by adding another state variable...

At the moment what I have deployed and tested most is a simple 3 tier
fq_codel based shaper,
which has a priority, best effort, and background queue. The priority
queue is generally only DNS traffic,
the background is only CS1, and fq_codel does such a good job with
sparse streams in
the general case that for VOIP, etc on the networks I fiddle with,
there is no point in having
much more than the BE queue. And presently all other markings are ignored.

video is another story. (or could be, I don't know enough about
webrtc's behaviors to
"see" what will happen)

So I can see trying to utilize the AF markings as you describe in your
draft in codel's
drop decision. While this is also subject to being gamed, the
disincentives are slight.

This is a little different from using "drop probability" as the
decider, however, but I
would hope that this mapping of the old AF style random drop ideas
into the new style delay
ideas could work...

And what codel depends on is a definition of "TCP friendly". That if a
stream is not
going to behave in a tcp friendly manner, then an entirely different
strategy for such
flows seems necessary, and glomming on some additional state isn't going to
be the right thing.

>   a) how well can it work in routers to shift packet drops to low-prio packets

I guess where I fall apart here is how to shift the encoders to send thinner
streams at  any level of packet drop.

> those pieces that we have also worked out or discuss the ones we have not (primarily c ;-)

I'd be interested in specific codecs, example video streams, and test
scripts to try
and fiddle with this, as well as viable metrics.

-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

---

From: "Cheng-Jia Lai (chelai)" <[email-masked]>
Date: Wed, 10 Jul 2013 23:42:52 +0000

Yes, NM could also work with ECN (and it should be made so). In fact, we did implementation using WRED on routers that support ECN markings. If RTP endpoints support ECN markings, WRED will mark ECN instead of dropping the packets as avg Q size is between min & max threshold. But it still suffers from unfairness at congestion, where the avg Q size starts to exceed the max threshold for the lowest priority packet class...

We did experience a hard time trying to properly configure WRED parameters for preferential dropping of video packets with our NM implementation... It is a good insight you said here: maybe it is worth exploring how our work can perhaps use other AQM algorithms like CoDel, PIE, etc.

In fact, NM doesn't need specific AQM to drop by probabilities... it will be interesting to see how it can work with delay-based dropping. However, NM does require some "weighted" version of those AQMs. Hopefully a weighted version (if any) won't introduce new parameters (or too many) to configure the AQM.

Regards,
CJ

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Dave Taht
Sent: Wednesday, July 10, 2013 12:02 PM
To: Toerless Eckert (eckert)
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [tsvwg] Using intelligent dropping to improve video quality under congestion (draft-lai-tsvwg-normalizer-00.txt)

same gaming problem...

-- 
Dave Täht

---

From: ken carlberg <[email-masked]>
Date: Thu, 11 Jul 2013 12:04:58 -0400

Toerless,

some food for thought...

about 10 years ago, I put together a draft that added an optional TLV prioritization header to RTP to accomplish something similar in terms of having the app decide which was the better RTP packet to drop during times of congestion.  My aim was to have these decisions made at forwarding application-layer gateways that did not act as a RTP/RTCP endpoint.  The strength of the approach (in my opinion :-) was that it could span pockets of non-diff-serv regions and still carry significance downstream.  The weakness was the need for DPI if one wanted to make a more informed decision along the path about dropping a packet.

I wonder if such a dual-layer design might be helpful in your efforts.  In any case, I have an interest in the normalizer draft.

-ken

---

From: "Cheng-Jia Lai (chelai)" <[email-masked]>
Date: Thu, 11 Jul 2013 22:12:46 +0000

Ken,

Thanks very much for the comment and showing an interest in our draft.

Yes, I agree it is beneficial if the intra-flow priorities info can be carried in some bit field other than DSCP markings. I guess we need to better clarify that NM is not necessarily specific to AF PHB but potentially applicable to other service classes. (In fact, our implementation did try DPI in RTP and H.264 NAL unit header for that, with some limited overheads in packet processing.)

We know there can be a lot of intelligent ways to carrying priorities in packets (e.g. your TLV in RTP hdr) which are likely beyond our knowledge and expertise, so we just introduced the CB mode in Section 3.1, quoted as follows:

"When NM operates in "color-blind" (CB) mode, which is optionally
   supported, it reads certain bits field(s) other than the AF-markings
   in the packets to determine the actual drop precedence of that
   packet.  This implies that NM may need DPI in the packets, e.g.
   parsing into H.264 AVC header in each RTP packets, or alternatively
   use some method where the drop precedence is carried from the encoder
   in a customized bits field other than the AF-marking in each packet."

CB might not be the best name... sorry if confusing. Anyway, we hope this allows NM to collaborate better with video endpoints.

Regards,
CJ

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of ken carlberg
Sent: Thursday, July 11, 2013 9:05 AM
To: Toerless Eckert (eckert)
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [tsvwg] Using intelligent dropping to improve video quality under congestion (draft-lai-tsvwg-normalizer-00.txt)

Toerless,

some food for thought...

-ken

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 11 Jul 2013 17:30:02 -0700

Thanks, Ken

Indeed yes, that's exactly the line of thought that we would like to
see proceed as well. Any URL still existing for that draft ?

"application layer gateway" could specifically mean switching MCU using
intelligent dropping in conjunction with outbound congestion control.

I definitely would like to see RTP headers as the ultimatele marking
option even for routers to do intelligent dropping (or other packet
processing "beyond DSCP)). Intra-flow (per-packet) DSCP is just a big pain 
(aka: impossible) for applications. Many OSs don't have any APIs to support
this. And DSCP management in networks is already difficult enough as it
stands.

Would certainly like to see if there is interest to write up r present
more pieces of this puzzle, especially an RTP extension. We just felt that the
fairness in intelligent dropping was the biggest concern our video
application folks had, that's why we wanted to explain how that piece
could work first. 

Cheers
    Toerless

---

From: "Toerless Eckert (eckert)" <[email-masked]>
Date: Thu, 11 Jul 2013 17:59:30 -0700

On Wed, Jul 10, 2013 at 12:02:26PM -0700, Dave Taht wrote:
> not ECN marking others.

True. But i think there are at least 3 layers of priority that would
really be great to have: really important packet (long term reference
frames), "normal" packets, and low-proirity-packets (eg: discardable P).
And ECN could only do two.

Most of the time when you would want intelligent dropping, your drop
rate should be fairly low, eg< 20% instantaneous within the queue. If
you can do discardable P-frames it's of course best to drop those first.
Only when you instantaneously have really big bursts would you want to
drop "normal" packets and refrain to protection really important packets.

This does work really well with different queue-length (or WRED profiles).
If you only had two priorities (via ECN) it's hard to say how you
would best assign these two priorities to at least those three priority
levels.

> Problem overall is that ecn can be gamed and is largely not
> enabled, and so on...

Well. The gaming could be inhibited exactly by this draft because it
would try to guarantee the same amount of drops on a per-flow or "class"
basis, so if you're trying to game the system, you would be put into
your own class and would still see the same amount of drops as other flows.

Well, the main issue is really to redesignate ECN for a different
semantic than what it has today. And indicating packet priority is just
not quite the same as congestion feedback.

> by adding another state variable...

Yes, this is all quite interesting. We also played around with droppping
during enqueuing vs. during dequeuing, so there are a bunch of things
that could optimize intelligent dropping if you can build a more
intelligent queue.

[...]

> disincentives are slight.

Lets separate the marking from the mechanism to inhibit gaming. This
draft really is about the latter, and the marking described with AF is
really just an example. Yes, it would be lovely to agree on markings
too.

> stream is not going to behave in a tcp friendly manner, then an entirely
> different  strategy for such flows seems necessary, and glomming on some
> additional state isn't going to be the right thing.

Who stood up at RMCAT some time back and claimed to have worked for
> 10 years on TCP and declared TCP not self friendly to itself, so RMCAT
shouldn't bother ? ;-))

Kidding aside: Intelligent dropping IMHO makes perfect sense when being
applied to RT traffic sharing a queue friendly with TCP. And intelligent
dropping can then still improve the performance.

I am pretty positive though that RMCAT traffic in general will behave
better and get lower delay and more video-appropriate loss if it can have
its own queue. Primarily because it will just take decates to root out
all the bad TCP traffic. 

I don't think that the strategies if you do or if you don't share the queue
with TCP would be fundamentally be different. 

> streams at  any level of packet drop.

Video codec coders are known to be able to do crazy cool things.
There are a bunch of known mechanisms and probably more proprietary ones.

> scripts to try and fiddle with this, as well as viable metrics.

Sure. Best done in person. If we're going to be at the WG for this, we can
talk on the side how we've been doing this. 

Cheers
    Toerless

---

From: ken carlberg <[email-masked]>
Date: Mon, 15 Jul 2013 09:41:21 -0400

Toerless,

sorry for the tardy reply.  I'll dig through my archives and send you a copy.  my draft proposed adding a "classifier" field, but I only did one rev (which was missing some better rationale after a long talk I had with Dave Oran) and i suspect that it would need a fair amount of scrubbing before re-issuing another version.  the first order of business, though, will be to gauge the feedback/comments to your draft.

-ken

ps, CJ thanks for your clarification in your previous response

---

From: <[email-masked]>
Date: Wed, 31 Jul 2013 10:11:05 +0200

Toerless,

discussion seems to conclude with:

- fairness must be enforced if senders don't set DSCPs according to the global
  fairness scheme.
- video applications have a hard time setting DSCPs anyway due to a missing API.
- ECN is not meant to indicate drop priorities and offers an insufficient
  number of states.

It might be easier to use a single PHB/DSCP and add PCN like ECN. In an e2e mode with receivers giving PCN congestion feedback to senders. My definition of PCN would be "rate control on a congested link" (rather than admission control and traffic termination as applied by the WG - I've been involved in the WGs work).

This should result in:
- avoidance of sudden overload in that class (no burst drops)
- rate adaption of senders if congestion is pending
- a simpler design as compared to what you and Ken discussed

I'm going to drop another message soon.

Regards,

Ruediger

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Toerless Eckert (eckert)
Sent: Friday, July 12, 2013 3:00 AM
To: Dave Taht
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [tsvwg] Using intelligent dropping to improve video quality under congestion (draft-lai-tsvwg-normalizer-00.txt)

[...]

shouldn't bother ? ;-))

Cheers
    Toerless

---

From: <[email-masked]>
Date: Wed, 31 Jul 2013 10:26:48 +0200

Toerless,

prior to setting up a new call, some kind of available bandwidth measurement or indication might be added. This isn't part of your draft. But an additional video stream may cause congestion on a bottleneck.

Measurement based admission control has been controversial so far. If used, the main point is to use a measurement class sharing the same queue as the traffic to be admitted, but having a higher drop probability.

- PCN is an option, but requires a new marking behaviour
 (threshold marker). End to end PCN as mentioned in another email.
- Using AF41 for admitted traffic and AF42 to perform measurements
  is an option to me, both should support ECN marking (AF42 seeing
  the higher marking and drop probability).

As marked packets indicating an issue can be interpreted by routers along a path, while packets dropped earlier elsewhere can't be, I prefer marking to dropping. I also think, it is easier to detect packet marks than delay variation measurements in routers, if it is about detecting congestion.

Take the above as an example. I used a separate thread as this isn't part of your draft an I accept if this shouldn't be discussed in the context of this draft. It fit's well to the RTCWeb QoS discussion though, I think.

Regards,

Ruediger