Subject: Re: [forces] AD review of draft-ietf-forces-interfelfb-01

From: Jamal Hadi Salim <[email-masked]>
Date: Sun, 2 Aug 2015 14:40:46 -0400

Hi Alia,

On Fri, Jul 31, 2015 at 6:08 PM, Alia Atlas <[email-masked]> wrote:

> Hi Damascane and Jamal,
>
> First, thank you for your work on this draft.
>
> As is customary, I have done my AD review of this draft before requesting
> IETF Last Call.
>
>
Thanks for making the time. Your effort is very much appreciated.

> As part of this, I see that Evangelos, acting as Document Shepherd, found
> a number of issues that are written up in his report - and that still
> haven't been addressed two months later.  Picking an IANA value that isn't
> available is one of them.  I know these are editorial, but still needed.
>
>

Sorry.
Based on the last discussions we had it didnt seem that it was  worth
changing
until we got AD input (for some reason i thought it was you we discussed
this with - but likely it was Adrian; i just got back so still jetlagged).
We will make an update taking into consideration the comments from
 Evangelos and yourself.

At this time, I really don't think this draft is ready to advance.
> There are some things that are editorial or clarifications, as I described
> below.
>
> Please take some serious time and focus to improve this document if you
> want it to progress.
>
> There are two primary major issues.
>
> First, you haven't in any way justified why the inter-FE communication
> needs to be encapsulated in an Ethernet frame.  Since you are asking for
> the IETF to get an EtherType allocated, that's kind of a bare minimum.
>

We have this working over ethernet today and our scope is on ethernet
L2 networks in a private setting (because that was the charter work item).
We need the ethertype  for dis-ambiguation of the frames.
We do recognize it is possible to run data across different underlying
protocols - but it seems even talking about that in the doc was unnecessary
editorializing.

Second, the draft bounces around a lot around whether this is an
> encapsulation that is just inside Ethernet or a general encapsulation that
> can be inside IP and other things.
>

Ok, we will review and remove any uncertainty
and just focus on ethernet.

> Moreover, it doesn't actually describe the reasoning or handling of the
> src/dest information and the packets.
>
>

We will try to clarify.

> You really need to better describe precisely the assumptions, behaviors,
> and error handling.
>
>
Section 6 tries to cover the above. We will go over it again and fix
any gaps we can find.

Issues:
>
> 1) In Sec 5.1.1:  "The frame may be dropped if there is congestion on the
> receiving FE side.  One approach to mitigate this issue is to make sure
> that inter-FE LFB frames receive the highest priority treatment
> when scheduled on the wire."  This suggestion just destroyed the
> possibility of respecting the requested CoS of the encapsulated packet and
> probably any actual high priority or EF traffic.   This recommendation must
> be fixed.
>
>

Would wording around this form help:
"The frame may be dropped if there is congestion on the receiving FE side.
To mitigate this issue is the sending FE side should treat inter-FE LFB
frames
with high priority. When 802.1p is present, then it is recommended a value
of XXX should be used. In the absence of 802.1p, implementation specific
treatments at the source FE may be used".
To your earlier point i think we need to say something like this around
section
6 as well.

2) In Sec 6.1.1: One step is " create the outer ethernet header which is a
> duplicate of the
>       incoming frame's ethernet header.  The outer ethernet header may
>       have an optional 802.1q header (if one was included in the
>       original frame)."  First, what if the incoming packet wasn't in an
> ethernet frame?  POS does still exist.
>

Our intent (and deployment)  is on ethernet. Clearly we need to emphasize
that view.
Do you see POS as a necessary ingredient?

> Second, why would the 802.1q vlan tag be relevant/valid on the outgoing
> port in between FEs?   This doesn't really make sense to me.  I do see the
> next bullet talks about changing the vlan tag value but nothing happens if
> the NEID is 0 or absent.
>
>
The context of this was taken directly from code;-> So something was
lost in translation. We will fix it.

As to the relevance of the vlan tags - the idea of NEID is one of tenancy
differentiation.

> 3) How can the DSTFE be optional???  Why does sending the Ethernet Frame
> to the original destination MAC address work?
>
>
You put some doubt in my mind when I read this;-> So I went and looked
at testcases. And in the majority of the policies, we do specify the DSTFE.
But there are some cases where _we dont_ and reuse the originally
target DSTFE MAC address on the outer header.
Does that explain? If not, why do you think it wont work?

4) In Sec 6.2, what happens if none of the optional elements of an IFEInfo
> are specified?

Thanks for catching this.
There would be no such table entry allowed (and so no processed packets
will encounter this scenario). We will update the doc to indicate that when
the
CE inserts policies in the policy table - the FE MUST validate that the non
optional fields are all present (the implementation already does that).

> How can this work if there's no required way to describe the destination
> FE?
>
>
Am I misunderstanding or are you are implying we need some default
DSTFE where a lookup failure happens?
Last para in 6.1.1 describes what happens on lookup failure.

5) From p. 22, it looks like the IFETable is defined assuming only
> transport over Ethernet.
Yes - that is the intent

> 6) Sec 10:  I think you'll find that there are security implications of
> taking packets and sending them in an Ethernet packet.  Now, I realize that
> the different FEs are probably inside the same device on a private network
> - but you need to clarify why that's the case.
>
>
Yes, the scope is within the same network. We will add clarification.

> Minor Quibble:
>
> a) There are places where this draft talks about other documents being
> written.  For instance, on the top of p.13 "For any mapping towards these
> definitions a different   document to describe the mapping, one per
> transport, is expected to be defined."  It's fine to have a process in mind
> - but this is completely unclear, given that the WG is closed.  Are these
> documents with a well-known place on-line?  Are these ISE drafts?  Do you
> think these would be AD-sponsored or in rtgwg?
>
>

We can get rid of such text altogether to avoid any confusion we are
talking about ethernet only.

> b) On the bottom of p. 13, it says "In this version of the specification,
> we only focus on data and metadata.  Therefore we are not going to describe
> how to carry the ExceptionID information (future versions may). "   PLEASE
> go through the document and get rid of these bits of tentative and future
> language.  The document is doing and defining what it is.  OWN that and fix
> the language everywhere.  (e.g. bottom of p.14 "would save us")
>
>
Will do.

>
> Nits:
>
> 1) On p. 6, "This information may include any source and
> destination information (MAC address to use, if ethernet;)"  Sadly, I think
> the ;) needs to go - what about "e.g. MAC address to use, if Ethernet)".
>
>
will do.

> 2) In Sec. 3.1.1, the last paragraph is "The purpose of the inter-FE LFB
> is to define standard mechanisms for interconnecting FEs and for that
> reason we are not going to touch
>    anymore on proprietary chip-chip interconnects other than state the
>    fact they exist and that it is feasible to have translation to and
>    from proprietary approaches."  is a bit informal.  Could you please
> clean it up a bit?
>
For instance:  "This document defines the inter-FE LFB, a standard
> mechanism for interconnecting FEs.  Existing proprietary approaches, such
> as chip-chip interconnects, may be able to translate to and from the
> standard inter-FE LFB."
>
>
Will do.

> 3) Sec 4 starts with "We address the inter-FE connectivity requirements by
> proposing the inter-FE LFB class."  Remember this will be an RFC - please
> replace "proposing" with "defining".  Similarly, the title of "Figure 6:
> Packet format suggestion" - is this what the document is standardizing or
> not??
>
>
Will do.

Thanks for the thorough review.

cheers,
jamal

> Thanks,
> Alia

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sun, 2 Aug 2015 15:35:56 -0400

A couple of points where I see mis-communication, in line below:

On 8/2/15 2:40 PM, Jamal Hadi Salim wrote:
> Hi Alia,
>
>
> On Fri, Jul 31, 2015 at 6:08 PM, Alia Atlas <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Hi Damascane and Jamal,
>
>     As is customary, I have done my AD review of this draft before
>     requesting IETF Last Call.
>
>
...
>
>
>     Issues:
>
>     1) In Sec 5.1.1: "The frame may be dropped if there is congestion on
>     the receiving FE side.  One approach to mitigate this issue is to
>     make sure that inter-FE LFB frames receive the highest priority
>     treatment when scheduled on the wire."  This suggestion just
>     destroyed the possibility of respecting the requested CoS of the
>     encapsulated packet and probably any actual high priority or EF
>     traffic.   This recommendation must be fixed.
>
>
>

This answer seems to miss Alia's point.  Suppose that we are processing 
a mix of packets from multiple end-applications (this seems very 
likely).  Then some of those packets will be more important than others. 
  If there is congestion, and if packet drops are needed to keep things 
working, then you want to drop the inter-FE packets whose underlying IP 
packet is lower priority, while retaining the ones whose underlying IP 
priority (DSCP inferred drop precedence, ...) is higher.  So setting 
everything as high between the two FEs is not a good idea.

>
>     2) In Sec 6.1.1: One step is " create the outer ethernet header
>     which is a duplicate of the
>            original frame)."  First, what if the incoming packet wasn't
>     in an ethernet frame?  POS does still exist.
>
>
>

I think that there are actually two issues here.
1) There is no reason for the outer MAC header to be (from a 
specification point of view) a copy of the inner MAC header.  It needs 
to have the right contents for sending the packet to the next FE, which 
has nothing to do with the underlying Ethernet MAC header.  It may have 
to do with the underlying priority, but specifying it as a copy of the 
header is not a good model.
2) The mechanism should be specified in such a way that it works even if 
we are not carrying a subscriber MAC header.  It should work with just 
an IP header.  It should work with media other than Ethernet.  This 
mechanism is not specific to external Ethernet support.  For example, if 
you are using this for processing isnide a service environment, the 
subscribers MAC header is likely long gone and irrelevant.

>
>     Second, why would the 802.1q vlan tag be relevant/valid on the
>     outgoing port in between FEs?   This doesn't really make sense to
>     me.  I do see the next bullet talks about changing the vlan tag
>     value but nothing happens if the NEID is 0 or absent.
>
>

It is fine to carry packets from multiple tenants across this mechanism. 
  However, ForCES does not rely on vlan tags for differentiation. 
Rather, it uses metadata.  So I don't see why you are worrying about the 
incoming vlan tag (which may not even be there.)

>
>     3) How can the DSTFE be optional???  Why does sending the Ethernet
>     Frame to the original destination MAC address work?
>
>

That does not make sense.  The InterFE LFB needs to know where to send 
the packet, and needs to send it there.  You can take some 
implementation shortcuts (although I would bet they will cause you 
trouble later), but that is not how you specify the protocol behavior.
...

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 3 Aug 2015 07:05:41 -0400

> This answer seems to miss Alia's point.  Suppose that we are processing a
>> mix of packets from multiple end-applications (this seems very likely).
>> Then some of those packets will be more important than others.  If there is
>> congestion, and if packet drops are needed to keep things working, then you
>> want to drop the inter-FE packets whose underlying IP packet is lower
>> priority, while retaining the ones whose underlying IP priority (DSCP
>> inferred drop precedence, ...) is higher.  So setting everything as high
>> between the two FEs is not a good idea.
>
>

How about we totally remove that text? upstream LFBs will set priority
fields (which pass through the Inter-FE LFB) and down
stream LFBs (like schedulers) will pay attention to it.

> 1) There is no reason for the outer MAC header to be (from a specification
> point of view) a copy of the inner MAC header.  It needs to have the right
> contents for sending the packet to the next FE, which has nothing to do
> with the underlying Ethernet MAC header.  It may have to do with the
> underlying priority, but specifying it as a copy of the header is not a
> good model.

2) The mechanism should be specified in such a way that it works even if we
> are not carrying a subscriber MAC header.  It should work with just an IP
> header.  It should work with media other than Ethernet.  This mechanism is
> not specific to external Ethernet support.  For example, if you are using
> this for processing isnide a service environment, the subscribers MAC
> header is likely long gone and irrelevant.
>
>

Ok, got it ;->
The implementation ended dealing with Ethernet as the "raw
packet" input into IN1. As i was saying that text was direct translation
from the code. So we need to fix the text.
And yes, I agree that it is possible there will be no ethernet frame
arriving
at IN1.
We would like however want to keep the ethernet encapsulation
and not have to deal with POS for example - would that be fine?

> However, ForCES does not rely on vlan tags for differentiation. Rather, it
> uses metadata.  So I don't see why you are worrying about the incoming vlan
> tag (which may not even be there.)
>
>
Agreed.

cheers,
jamal

---

From: Alia Atlas <[email-masked]>
Date: Mon, 3 Aug 2015 10:31:23 -0400

Hi Jamal,

> stream LFBs (like schedulers) will pay attention to it.

Congestion control and prioritization need to be addressed.  What I think
you are missing is that it is important to clearly specify how QoS or CoS
is handled so that doing an inter-FE mechanism doesn't break CoS.

>> specification point of view) a copy of the inner MAC header.  It needs to
>> have the right contents for sending the packet to the next FE, which has
>> nothing to do with the underlying Ethernet MAC header.  It may have to do
>> with the underlying priority, but specifying it as a copy of the header is
>> not a good model.
>
>> we are not carrying a subscriber MAC header.  It should work with just an
>> IP header.  It should work with media other than Ethernet.  This mechanism
>> is not specific to external Ethernet support.  For example, if you are
>> using this for processing isnide a service environment, the subscribers MAC

Are you describing a generic packet format that can then be encapsulated in
an Ethernet frame or IP or so on, or something which is just for
Ethernet?   You need to decide what you are documenting.

Regards,
Alia

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 3 Aug 2015 10:51:19 -0400

While we need to explain the choice of Ethernet, assuming such 
explanation then using that for the outer encapsulation seems fine.  It 
was the assumption that the inner encapsulation was Ethernet that seemed 
problematic.

It seems to me that if we are careful in the wording, we probably can 
also allow for other carriage means to be defined later.

Yours,
Joel

On 8/3/15 7:05 AM, Jamal Hadi Salim wrote:
>
> On Sun, Aug 2, 2015 at 3:35 PM, Joel M. Halpern <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>         This answer seems to miss Alia's point.  Suppose that we are
>         processing a mix of packets from multiple end-applications (this
>         seems very likely).  Then some of those packets will be more
>         important than others.  If there is congestion, and if packet
>         drops are needed to keep things working, then you want to drop
>         the inter-FE packets whose underlying IP packet is lower
>         priority, while retaining the ones whose underlying IP priority
>         (DSCP inferred drop precedence, ...) is higher.  So setting
>     specification point of view) a copy of the inner MAC header.  It
>     needs to have the right contents for sending the packet to the next
>     FE, which has nothing to do with the underlying Ethernet MAC
>     header.  It may have to do with the underlying priority, but
>     specifying it as a copy of the header is not a good model.
>
>     2) The mechanism should be specified in such a way that it works
>     even if we are not carrying a subscriber MAC header.  It should work
>     with just an IP header.  It should work with media other than
>     Ethernet.  This mechanism is not specific to external Ethernet
>     support.  For example, if you are using this for processing isnide a
>     service environment, the subscribers MAC header is likely long gone
>     and irrelevant.
>
>
>
> Ok, got it ;->
>     It is fine to carry packets from multiple tenants across this
>     mechanism.  However, ForCES does not rely on vlan tags for
>     differentiation. Rather, it uses metadata.  So I don't see why you
>     are worrying about the incoming vlan tag (which may not even be there.)
>
>
> Agreed.
>
> cheers,
> jamal

---

From: Alia Atlas <[email-masked]>
Date: Mon, 3 Aug 2015 10:52:07 -0400

Hi Jamal,

> We will make an update taking into consideration the comments from

As you may understand, for an AD-sponsored draft, I do expect that the
authors are extremely pro-active.
Claiming that the draft is ready to progress when it is very far from that
is just a way of wasting all our time - particularly
when I pick up the same issues and see whether or not they've been
addressed.

As this draft is written, it is taking a single implementation and doesn't
think through implications or the different aspects.

>> There are some things that are editorial or clarifications, as I
>> described below.
>>

A personal choice to implement it using Ethernet is not a technical
justification as to why this needs to be a standard.
You are asking for a limited resource - and, as Pat's email indicated, the
IEEE are asking that new EtherTypes have formats that
include a subtype so that reuse is possible.

Are there others interested in deploying this over Ethernet?   Why is it
necessary to use Ethernet and not IP?

>> the receiving FE side.  One approach to mitigate this issue is to make sure

You are describing what is presented as an inter-FE communication and then
your text ASSUMES that the
traffic is always Ethernet frames where the original Ethernet frame is
still available at that point in the pipeline.
This is just sloppy and not generic for no reason that I can currently
understand.

> As to the relevance of the vlan tags - the idea of NEID is one of tenancy

But how is that connected to the inter-FE communication?

> Does that explain? If not, why do you think it wont work?

Why would a random destination MAC address have anything to do with the
next FE in the FE graph?

Presumably, the outer encapsulation is to go from one FE to another FE in
the graph.  If so, then what
information the packet being processed has is irrelevant.  If not, then
you've totally failed to explain your
assumptions and purpose.

> will encounter this scenario). We will update the doc to indicate that
> when the

But you made them all optional!

> 5) From p. 22, it looks like the IFETable is defined assuming only

Why do you have text describing a more generic solution and then actually
define an extremely limited one?

Regards,
Alia

>> written.  For instance, on the top of p.13 "For any mapping towards
>> these definitions a different   document to describe the mapping, one
>> per transport, is expected to be defined."  It's fine to have a process in
>> mind - but this is completely unclear, given that the WG is closed.  Are
>> these documents with a well-known place on-line?  Are these ISE drafts?  Do
>> you think these would be AD-sponsored or in rtgwg?
>>
>>
>
>> 3) Sec 4 starts with "We address the inter-FE connectivity requirements
>> by proposing the inter-FE LFB class."  Remember this will be an RFC -
>> please replace "proposing" with "defining".  Similarly, the title of
>> "Figure 6: Packet format suggestion" - is this what the document is
>> standardizing or not??
>>
>>
> Will do.
>

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 3 Aug 2015 18:14:03 -0400

Alia,

You raise legit issues. We will work on putting out a new version
in the next day or two which will hopefully address all the issues you
brought up by being more focussed. Thanks for your patience.

cheers,
jamal

On Mon, Aug 3, 2015 at 10:52 AM, Alia Atlas <[email-masked]> wrote:

>>> As part of this, I see that Evangelos, acting as Document Shepherd,
>>> found a number of issues that are written up in his report - and that still
>> 2) In Sec 6.1.1: One step is " create the outer ethernet header which is
>>> a duplicate of the
>> 4) In Sec 6.2, what happens if none of the optional elements of an
>>> IFEInfo are specified?
>>>
>>
>>> b) On the bottom of p. 13, it says "In this version of the
>>> specification, we only focus on data and metadata.  Therefore we are not
>>> going to describe how to carry the ExceptionID information (future versions
>>> may). "   PLEASE go through the document and get rid of these bits of
>>> tentative and future language.  The document is doing and defining what it
>>> is.  OWN that and fix the language everywhere.  (e.g. bottom of p.14 "would
>>> save us")
>>>
>>>
>> Will do.
>>
>>
>>>
>>> Nits:
>>>

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 3 Aug 2015 18:24:04 -0400

On Mon, Aug 3, 2015 at 10:51 AM, Joel M. Halpern <[email-masked]>
wrote:

> While we need to explain the choice of Ethernet, assuming such explanation
> then using that for the outer encapsulation seems fine.  It was the
> assumption that the inner encapsulation was Ethernet that seemed
> problematic.
>
>

We are going to get rid of that detail. It was confusing.

> It seems to me that if we are careful in the wording, we probably can also
> allow for other carriage means to be defined later.
>
>
Ok, we will publish a new version in the next 1-2 days. We feel that
a focus will help to get rid of the confusion. If there is tweaking that
can be made to allow other transports that would be great.

cheers,
jamal

---

From: Alia Atlas <[email-masked]>
Date: Mon, 3 Aug 2015 19:04:57 -0400

Jamal,

Thanks - I've asked Joel to help and do a routing directorate review.
The timeline is up to you, but I'm not as backlogged anymore.

Regards,
Alia

On Mon, Aug 3, 2015 at 6:14 PM, Jamal Hadi Salim <[email-masked]> wrote:

>> Claiming that the draft is ready to progress when it is very far from
>> that is just a way of wasting all our time - particularly
>> As this draft is written, it is taking a single implementation and
>> doesn't think through implications or the different aspects.
>>
>> You are asking for a limited resource - and, as Pat's email indicated,
>> the IEEE are asking that new EtherTypes have formats that
>>>> You really need to better describe precisely the assumptions,
>>>> behaviors, and error handling.
>>>>
>>>>
>>>>       original frame)."  First, what if the incoming packet wasn't in
>>>> an ethernet frame?  POS does still exist.
>>>>
>>>
>>>
>> You are describing what is presented as an inter-FE communication and
>> then your text ASSUMES that the
>>>> 2) In Sec. 3.1.1, the last paragraph is "The purpose of the inter-FE
>>>> LFB is to define standard mechanisms for interconnecting FEs and for that