Subject: [rmcat] Following up on my ECN feedback question at the meeting today

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 17 Nov 2016 10:20:56 +0000

I have some follow-up comments on things relating to the transport use 
of ECNthat may relate to draft-dt-rmcat-feedback-message-01.

(1) My meeting comment on jabber was (I may have missed the point 
earlier in the talk) that I wanted to check if a new method could 
potentially hide the presence of a CE-Mark (specifically receiving a 
CE-marked packet and not reporting this seems like something we may wish 
to worry about and I could not work out is the proposal ever omitted ECN 
information).

(2) Now looking at the design team draft, this statement is of course OK 
with me:

"If ECN
[RFC3168] is used, it is necessary to report on the 2-bit ECN mark
in received packets, indicating for each packet whether it is
marked not-ECT, ECT(0), ECT(1), or ECN-CE."

And following this it seems to me that there are two very basic pieces 
of information that I’d love to be able to know as a sender when using 
an ECN-capable network:

(2a) It *can* be important to know that an ECT(?) codepoint is being 
received at the IP layer. A sender may well know that it is sending 
either ECT(0) or ECT(1), but doesn’t definitely know these are being 
received unchanged (i.e., the path has not remarked these).
- Reporting all ECN marks would clearly satisfy this - read no further 
if you plan to always do this.
- A count of how many ECT marks were received is really helpful to know 
that the current ECN forwarding path is broken, it could even allow a 
sender to check that any chosen ECT(x) encoding is working, by 
independently setting x=0 or x=1, if it wishes.
- Knowing which specific ECT(?) is received is potentially useful 
information. This could perhaps be used in a method to determine which 
flavour of ECN-marking is supported by a path, and more info about 
whether ECN is being handled correctly by a network (knowing middleboxes 
don’t do unwanted things can always utilise extra information). (As new 
info, the TSVWG is currently considering work proposing to eliminate the 
use of NONCE encoding where an apps sends ECT(0) and ECT(1) randomly. 
This could simplify the need to encode which ECT(?) codepoints were 
received - see draft-black-tsvwg-ecn-experimentation.)

(2b) A sender must *at least* know if there were any CE marks in a 
reporting interval.
- If you report each CE received than this is sufficient. But I have a 
word of caution. *If* an app looses a report with a CE-mark report it 
may never know, and does not impact user experience in terms of loss, 
but the CC still needs to respond to behave correctly for AQM, and may 
impact delay :-).
- It can be very valuable for the sender to know if ANY CE-marked 
packets were received, even if sending not-ECN capable packets at this 
moment (where it is an error that indicates ECT(?) is unreliable).
- Knowing how many CE-marks (or CE-marked bytes) per interval is 
probably really valuable for a CC. I’d think a sender could easily 
utilise more CE per packet information, and may need to do (e.g., to 
utilise L4S should that be wanted in future).
- Knowing each CE-mark is something that any CC can utilise and could be 
needed for L4S. It could also help to know about whether bursts are 
being marked, rather than average rate.
- Some AQMs may use shallow marking and that is something that COULD be 
nice to detect in future from per-packet CE-mark reports. (At the 
moment, I really think we don’t completely know yet how to do this).

So to me, this suggests at least some counters per reporting interval 
are needed - and per-packet info will be much better.

(3) In relation to "Feedback Frequency and Overhead", section 4. I have 
one less useful comment:

Amortising overhead by reporting less often could perhaps result in a 
performance tradeoff. There may well be quite stringent timing 
implications if an RMCAT app wants to use "low latency" ECN - that’s 
something to be watched as the transport as the L4S work in TSVWG 
progresses, but it would be nice to know that the report interval does 
not constrain our options. I’m arguing that the WG may need to note this 
(or consider this), but I for one, don’t yet know how this will evolve 
in future.

In case anyone needs it, draft-ietf-aqm-ecn-benefits-08 may have some 
more info.

Best wishes,

Gorry

---

From: "Michael Ramalho (mramalho)" <[email-masked]>
Date: Thu, 17 Nov 2016 14:59:14 +0000

Hi Gorry,

Speaking for myself (and not necessarily all on the RMCAT Design Team), every proposal so far has the ability to report both ECN bits for ALL the packets the receiver received. Thus the sender, knowing what was sent and what was received, could determine any changes in these bits.

The proposals since the "01" draft have put the ECN information in different places - to attain the goal of trivial compression. Note that we have not come to closure if compression is needed (that was what Colin's presentation addressed).

In the proposal shown at the meeting - if all the ECN bits received were identical, one instance of those two bits appeared in the fixed body of the report (two bits stolen from where the end_seq was previously) along with a flag (another bit stolen from the same field) that stated that all received ECN bits were identical. If all received ECN bits were not identical, then ALL of the individual ECN bits for all packets received were reported elsewhere (in the "C bits" after all the arrival time offset fields).

There was discussion in one of the DT meetings that one might want disable the reporting of the individual ECN bits - perhaps dynamically via signaling - and I briefly mentioned that during my presentation. Such discussion has not been reflected in any consensus nor in any text. I think the option was discussed in the context of a low bandwidth stream (but I am not certain on this point).

However, if such an option to disable ECN reporting was desired and enabled - the format in the proposal shown had two ECN bit locations above as "a representative ECN". Again, this was not defined or discussed - but such could be the most prevalent ECN ... or the "worst-case" ECN (i.e., if there was congestion flagged in some of them) ... or [who knows].

In closing, we are not certain that we need compression. But we do know how to compress loss indications and individual ECN bits efficiently via run length encoding (perhaps using what is already defined in RTCP XR). My vote would be to compress the ECN (rather than disabling ECN reporting) if we ever decided that the reporting all ECN bits individually was too burdensome.

Again, those are my opinions and not necessarily all on the Design Team. I'll let Colin and others respond to the frequency and overhead portions of your comments.

Best,

Michael Ramalho

-----Original Message-----
From: rmcat [mailto:[email-masked]] On Behalf Of Gorry Fairhurst
Sent: Thursday, November 17, 2016 5:21 AM
To: [email-masked]
Subject: [rmcat] Following up on my ECN feedback question at the meeting today

I have some follow-up comments on things relating to the transport use of ECNthat may relate to draft-dt-rmcat-feedback-message-01.

(1) My meeting comment on jabber was (I may have missed the point earlier in the talk) that I wanted to check if a new method could potentially hide the presence of a CE-Mark (specifically receiving a CE-marked packet and not reporting this seems like something we may wish to worry about and I could not work out is the proposal ever omitted ECN information).

(2) Now looking at the design team draft, this statement is of course OK with me:

"If ECN
[RFC3168] is used, it is necessary to report on the 2-bit ECN mark in received packets, indicating for each packet whether it is marked not-ECT, ECT(0), ECT(1), or ECN-CE."

And following this it seems to me that there are two very basic pieces of information that I'd love to be able to know as a sender when using an ECN-capable network:

(2a) It *can* be important to know that an ECT(?) codepoint is being received at the IP layer. A sender may well know that it is sending either ECT(0) or ECT(1), but doesn't definitely know these are being received unchanged (i.e., the path has not remarked these).
- Reporting all ECN marks would clearly satisfy this - read no further if you plan to always do this.
- A count of how many ECT marks were received is really helpful to know that the current ECN forwarding path is broken, it could even allow a sender to check that any chosen ECT(x) encoding is working, by independently setting x=0 or x=1, if it wishes.
- Knowing which specific ECT(?) is received is potentially useful information. This could perhaps be used in a method to determine which flavour of ECN-marking is supported by a path, and more info about whether ECN is being handled correctly by a network (knowing middleboxes don't do unwanted things can always utilise extra information). (As new info, the TSVWG is currently considering work proposing to eliminate the use of NONCE encoding where an apps sends ECT(0) and ECT(1) randomly. 
This could simplify the need to encode which ECT(?) codepoints were received - see draft-black-tsvwg-ecn-experimentation.)

(2b) A sender must *at least* know if there were any CE marks in a reporting interval.
- If you report each CE received than this is sufficient. But I have a word of caution. *If* an app looses a report with a CE-mark report it may never know, and does not impact user experience in terms of loss, but the CC still needs to respond to behave correctly for AQM, and may impact delay :-).
- It can be very valuable for the sender to know if ANY CE-marked packets were received, even if sending not-ECN capable packets at this moment (where it is an error that indicates ECT(?) is unreliable).
- Knowing how many CE-marks (or CE-marked bytes) per interval is probably really valuable for a CC. I'd think a sender could easily utilise more CE per packet information, and may need to do (e.g., to utilise L4S should that be wanted in future).
- Knowing each CE-mark is something that any CC can utilise and could be needed for L4S. It could also help to know about whether bursts are being marked, rather than average rate.
- Some AQMs may use shallow marking and that is something that COULD be nice to detect in future from per-packet CE-mark reports. (At the moment, I really think we don't completely know yet how to do this).

So to me, this suggests at least some counters per reporting interval are needed - and per-packet info will be much better.

(3) In relation to "Feedback Frequency and Overhead", section 4. I have one less useful comment:

Amortising overhead by reporting less often could perhaps result in a performance tradeoff. There may well be quite stringent timing implications if an RMCAT app wants to use "low latency" ECN - that's something to be watched as the transport as the L4S work in TSVWG progresses, but it would be nice to know that the report interval does not constrain our options. I'm arguing that the WG may need to note this (or consider this), but I for one, don't yet know how this will evolve in future.

In case anyone needs it, draft-ietf-aqm-ecn-benefits-08 may have some more info.

Best wishes,

Gorry

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 17 Nov 2016 15:12:10 +0000

Sounds fine, thanks - I'll let others also get back if they wish, 
otherwise keep designing!

Gorry

---

From: "Bob Briscoe" <[email-masked]>
Date: Thu, 17 Nov 2016 18:00:49 -0000

Michael, Gorry,

On Thu, November 17, 2016 3:12 pm, Gorry Fairhurst wrote:
>> Speaking for myself (and not necessarily all on the RMCAT Design Team),
>> every proposal so far has the ability to report both ECN bits for ALL
>> the packets the receiver received. Thus the sender, knowing what was
>> sent and what was received, could determine any changes in these bits.
>>
>> The proposals since the "01" draft have put the ECN information in
>> different places - to attain the goal of trivial compression. Note that
>> we have not come to closure if compression is needed (that was what
>> Colin's presentation addressed).
>>
>>
>> In the proposal shown at the meeting - if all the ECN bits received
>> were identical, one instance of those two bits appeared in the fixed
>> body of the report (two bits stolen from where the end_seq was
>> previously) along with a flag (another bit stolen from the same field)
>> that stated that all received ECN bits were identical. If all received
>> ECN bits were not identical, then ALL of the individual ECN bits for
>> all packets received were reported elsewhere (in the "C bits" after all
>> the arrival time offset fields).
>>
>> There was discussion in one of the DT meetings that one might want
>> disable the reporting of the individual ECN bits - perhaps dynamically
>> via signaling - and I briefly mentioned that during my presentation.
>> Such discussion has not been reflected in any consensus nor in any
>> text. I think the option was discussed in the context of a low
>> bandwidth stream (but I am not certain on this point).
>>
>> However, if such an option to disable ECN reporting was desired and
>> enabled - the format in the proposal shown had two ECN bit locations
>> above as "a representative ECN". Again, this was not defined or
>> discussed - but such could be the most prevalent ECN ... or the
>> "worst-case" ECN (i.e., if there was congestion flagged in some of
>> them) ... or [who knows].
>>
>> In closing, we are not certain that we need compression. But we do know
>> how to compress loss indications and individual ECN bits efficiently
>> via run length encoding (perhaps using what is already defined in RTCP
>> XR). My vote would be to compress the ECN (rather than disabling ECN
>> reporting) if we ever decided that the reporting all ECN bits
>> individually was too burdensome.

[BB]: In any compression scheme, please try to avoid allowing the receiver
to opt out of giving an accurate count of CE received...

One of the reasons for (potentially) deciding we do not need the ECN nonce
in the IP header any more (see draft-black-tsvwg-ecn-experimentation) is
that an alternative is possible without consuming any IP header
codepoints. It has not been written up, but it would involve the sender
randomly setting CE itself once or twice, and checking that the receiver
reports these faithfully. I believe it would be sufficient for the
receiver to just report the sum of CE marks (so the sender's injected CE
marks would set a lower bound to how much marking a receiver could
suppress).

Like the ECN nonce, this is only applicable if the sender is willing to
police ECN on behalf of the network. But despite its limitations, it is
worth ensuring that a receiver cannot downgrade feedback precision
sufficiently to enable its own attack.

I believe it will be sufficient for CE feedback to always be accurate. ECT
0/1 do not need to be accurate.

Also, please ensure the feedback always gives bytes marked not just
packets marked (or allows the sender to infer bytes marked, e.g. if all
packets are the same size). Not just to aid testing for receiver cheating,
but also as motivated in RFC7141.

Until someone writes up a scheme for testing receiver ECN feedback
compliance, I'm afraid I cannot give more details. But you can find the
equivalent scheme for drop here:
draft-moncaster-tcpm-rcv-cheat-03 (Expired)

Also I suggest a read of rfc7560. Despite its title, "Problem Statement
and Requirements for Increased Accuracy in ECN Feedback", it is solely
about TCP, but still a useful checklist for rmcat.

Cheers

Bob

>>
>> Again, those are my opinions and not necessarily all on the Design
>> Team. I'll let Colin and others respond to the frequency and overhead
>> portions of your comments.
>>
>> Best,
>>
>>
>> Michael Ramalho
>>
>>
>> CE-marked packet and not reporting this seems like something we may
>> wish to worry about and I could not work out is the proposal ever
>> omitted ECN information).
>>
>> [RFC3168] is used, it is necessary to report on the 2-bit ECN mark in
>> received packets, indicating for each packet whether it is marked
>> not-ECT, ECT(0), ECT(1), or ECN-CE."
>>
>> of information that I'd love to be able to know as a sender when using
>> an ECN-capable network:
>>
>> either ECT(0) or ECT(1), but doesn't definitely know these are being
>> received unchanged (i.e., the path has not remarked these). - Reporting
>> all ECN marks would clearly satisfy this - read no further if you plan
>> to always do this. - A count of how many ECT marks were received is
>> really helpful to know that the current ECN forwarding path is broken,
>> it could even allow a sender to check that any chosen ECT(x) encoding
>> is working, by independently setting x=0 or x=1, if it wishes. - Knowing
>> which specific ECT(?) is received is potentially useful information.
>> This could perhaps be used in a method to determine which flavour of
>> ECN-marking is supported by a path, and more info about whether ECN is
>> being handled correctly by a network (knowing middleboxes don't do
>> unwanted things can always utilise extra information). (As new info,
>> the TSVWG is currently considering work proposing to eliminate the use
>> of NONCE encoding where an apps sends ECT(0) and ECT(1) randomly. This
>> could simplify the need to encode which ECT(?) codepoints were received
>> - see draft-black-tsvwg-ecn-experimentation.)
>>
>>
>> reporting interval. - If you report each CE received than this is
>> sufficient. But I have a word of caution. *If* an app looses a report
>> with a CE-mark report it may never know, and does not impact user
>> experience in terms of loss, but the CC still needs to respond to
>> behave correctly for AQM, and may impact delay :-). - It can be very
>> valuable for the sender to know if ANY CE-marked packets were received,
>> even if sending not-ECN capable packets at this moment (where it is an
>> error that indicates ECT(?) is unreliable). - Knowing how many CE-marks
>> (or CE-marked bytes) per interval is probably really valuable for a CC.
>> I'd think a sender could easily utilise more CE per packet information,
>> and may need to do (e.g., to utilise L4S should that be wanted in
>> future). - Knowing each CE-mark is something that any CC can utilise and
>> could be needed for L4S. It could also help to know about whether
>> bursts are being marked, rather than average rate. - Some AQMs may use
>> shallow marking and that is something that COULD be nice to detect in
>> future from per-packet CE-mark reports. (At the moment, I really think
>> we don't completely know yet how to do this).
>>
>>
>> implications if an RMCAT app wants to use "low latency" ECN - that's
>> not constrain our options. I'm arguing that the WG may need to note
>> this (or consider this), but I for one, don't yet know how this will
>> evolve in future.
>>

---

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 21 Nov 2016 13:14:13 +0000

Hi Bob + others

For what it's worth, in the proposed SCReAM feedback there is only an ECN-CE counter. I have however not pursued this as I left to the feedback design team to come up with a generic feedback format that works for all RMCAT candidates.

There may be cases where one would like to know exactly which packets that are ECN-CE marked, what I can come up with is :
1) ConEx : Achieve a perfect indication of bytes congested in the re-inserted feedback
2) Scalable congestion control (L4S) : Compute the exact amount of "congested" bytes for proper computation of the congestion window backoff.
The reason why I think a counter is sufficient is that the feedback for each media is individual, so if you have two media streams with vastly different bitrates you will still be able to get (hopefully) decent approximations for 1) and 2) based of the average packet size for each stream. For scalable congestion control I don't believe that an average congestion bytes that is a little off plays that much of a role, it will temporarly lead to a congestion window decrease that is a little off but that would correct itself in the form of a few more or less ECN-CE marked packets in the subsequent round trips, it is most likely an error that will drown in the other congestion noise.

Reporting of ECT(0) and ECT(1) is guess mainly for e.g. black hole detection. It is however possible to use RFC6679 for this purpose.

/Ingemar

> From: Bob Briscoe [mailto:[email-masked]]
> Sent: den 17 november 2016 19:01
> To: [email-masked]; Michael Ramalho (mramalho)
> <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [rmcat] Following up on my ECN feedback question at the
> meeting today
> 
> Michael, Gorry,
> 
> >> Speaking for myself (and not necessarily all on the RMCAT Design
> >> Team), every proposal so far has the ability to report both ECN bits
> >> for ALL the packets the receiver received. Thus the sender, knowing
> >> what was sent and what was received, could determine any changes in
> these bits.
> >>
> >> different places - to attain the goal of trivial compression. Note
> >> that we have not come to closure if compression is needed (that was
> >> what Colin's presentation addressed).
> >>
> >>
> >> previously) along with a flag (another bit stolen from the same
> >> field) that stated that all received ECN bits were identical. If all
> >> received ECN bits were not identical, then ALL of the individual ECN
> >> bits for all packets received were reported elsewhere (in the "C
> >> bits" after all the arrival time offset fields).
> >>
> >> disable the reporting of the individual ECN bits - perhaps
> >> dynamically via signaling - and I briefly mentioned that during my
> presentation.
> >> In closing, we are not certain that we need compression. But we do
> >> know how to compress loss indications and individual ECN bits
> >> efficiently via run length encoding (perhaps using what is already
> >> defined in RTCP XR). My vote would be to compress the ECN (rather
> >> than disabling ECN
> [BB]: In any compression scheme, please try to avoid allowing the receiver to
> opt out of giving an accurate count of CE received...
> 
> that an alternative is possible without consuming any IP header codepoints.
> It has not been written up, but it would involve the sender randomly setting
> CE itself once or twice, and checking that the receiver reports these
> faithfully. I believe it would be sufficient for the receiver to just report the
> sum of CE marks (so the sender's injected CE marks would set a lower bound
> to how much marking a receiver could suppress).
> 
> Like the ECN nonce, this is only applicable if the sender is willing to police ECN
> on behalf of the network. But despite its limitations, it is worth ensuring that
> a receiver cannot downgrade feedback precision sufficiently to enable its
> own attack.
> 
> Also, please ensure the feedback always gives bytes marked not just packets
> marked (or allows the sender to infer bytes marked, e.g. if all packets are the
> same size). Not just to aid testing for receiver cheating, but also as motivated
> in RFC7141.
> 
> Also I suggest a read of rfc7560. Despite its title, "Problem Statement and
> Requirements for Increased Accuracy in ECN Feedback", it is solely about
> TCP, but still a useful checklist for rmcat.
> 
> 
> Cheers
> 
> 
> Bob
> 
> >>
> >> I have some follow-up comments on things relating to the transport
> >> use of ECNthat may relate to draft-dt-rmcat-feedback-message-01.
> >>
> >> And following this it seems to me that there are two very basic
> >> pieces of information that I'd love to be able to know as a sender
> >> when using an ECN-capable network:
> >>
> >> received unchanged (i.e., the path has not remarked these). -
> >> Reporting all ECN marks would clearly satisfy this - read no further
> >> if you plan to always do this. - A count of how many ECT marks were
> >> received is really helpful to know that the current ECN forwarding
> >> path is broken, it could even allow a sender to check that any chosen
> >> ECT(x) encoding is working, by independently setting x=0 or x=1, if
> >> it wishes. - Knowing which specific ECT(?) is received is potentially useful
> information.
> >> ECN-marking is supported by a path, and more info about whether ECN
> >> is being handled correctly by a network (knowing middleboxes don't do
> >> the TSVWG is currently considering work proposing to eliminate the
> >> received
> >> valuable for the sender to know if ANY CE-marked packets were
> >> received, even if sending not-ECN capable packets at this moment
> >> (where it is an error that indicates ECT(?) is unreliable). - Knowing
> >> how many CE-marks (or CE-marked bytes) per interval is probably really
> valuable for a CC.
> >> I'd think a sender could easily utilise more CE per packet
> >> information, and may need to do (e.g., to utilise L4S should that be
> >> wanted in future). - Knowing each CE-mark is something that any CC
> >> can utilise and could be needed for L4S. It could also help to know
> >> about whether bursts are being marked, rather than average rate. -
> >> Some AQMs may use shallow marking and that is something that COULD
> be
> >> moment, I really think we don't completely know yet how to do this).
> >>
> >>
> >> progresses, but it would be nice to know that the report interval
> >> does not constrain our options. I'm arguing that the WG may need to
> >> note this (or consider this), but I for one, don't yet know how this
> >> will evolve in future.
> >>