Subject: Tsvart early review of draft-ietf-trill-over-ip-10

From: Magnus Westerlund <[email-masked]>
Date: Thu, 15 Jun 2017 10:32:35 -0700

Reviewer: Magnus Westerlund
Review result: Not Ready

Early review of draft-ietf-trill-over-ip-10
Reviewer: Magnus Westerlund
Review result: Not Ready

TSV-ART review comments:

I have set this to not ready as there are several issues, some significant that
could affect the protocol realization significantly. Some may be me missing
things in TRILL, I was not that familiar with it before this review and I have
only tried looking up things, not reading the whole earlier specifications. So
don't hesitate to push back and provide pointers to things that can resolve
issues. The authors and the WG clearly have thought about a lot of issues and
dealt with much already.

Diffserv usage
--------------

Section 4.3:

   TRILL over IP implementations MUST support setting the DSCP value in
   the outer IP Header of TRILL packets they send by mapping the TRILL
   priority and DEI to the DSCP. They MAY support, for a TRILL Data
   packet where the native frame payload is an IP packet, mapping the
   DSCP in this inner IP packet to the outer IP Header with the default
   for that mapping being to copy the DSCP without change.

I think it is fine to require that implementations are capable  of setting
DSCP values on the outer IP header. However, I fail to see any discussion of
the potential issues with actually setting the DSCP values. It is one thing to
do this in an IP back bone use case where one can know and have control over
the PHB that the DSCP values maps to. But otherwise, over general internet the
behavior is not that predictable. One can easily be subject to policers or
remapping. Also as the actual DSCP code point usage is domain specific this is
difficult. Priority reversal is likely the least of the problems that this can
run into over general Internet.

Section 4.3:

   The default TRILL priority and DEI to DSCP mapping, which may be
   configured per TRILL over IP port, is an follows. Note that the DEI
   value does not affect the default mapping and, to provide a
   potentially lower priority service than the default priority 0,
   priority 1 is considered lower priority than 0. So the priority
   sequence from lower to higher priority is 1, 0, 2, 3, 4, 5, 6, 7.

      TRILL Priority  DEI  DSCP Field (Binary/decimal)
      --------------  ---  -----------------------------
                  0   0/1  001000 / 8
                  1   0/1  000000 / 0
                  2   0/1  010000 / 16
                  3   0/1  011000 / 24
                  4   0/1  100000 / 32
                  5   0/1  101000 / 40
                  6   0/1  110000 / 48
                  7   0/1  111000 / 56

This appear to be an problematic mapping. At least for prio 0 and 1. As
priority 1 appears to be intended to be higher than priority 0, it is
interesting that it is mapped to CS1, which to quote
https://datatracker.ietf.org/doc/rfc7657/:

CS1 ('001000') was subsequently designated as the recommended
      codepoint for the Lower Effort (LE) PHB [RFC3662].

So what is proposed can in a network using default mapping, result in that you
get priority 0 to be lower priority than 1. Plus that in some networks this can
also results in strange remapping that results in a different PHB for CS1 than.

MTU and Fragmentation
---------------------

I think there are two main issue here. The first one is MTUD discovery
of the actual IP path MTU between the ports. That will be needed to prevent
a lot of traffic going into MTU black holes. Especially as TRILL requries
1470 byte support which is likey above a lot of paths.

Section 8.4:

   Path MTU discovery [RFC4821] should be useful
   in determining the IP MTU between a pair of RBridge ports with IP
   connectivity.

The issue with RFC4821 is that it has requirements on the packetization layer.
Trill appears to have several components that are useful. However, it will
require a specification of the procedure to result in a useful tool.

Section 8.4:

   TRILL IS-IS MTU PDUs, as specified in Section 5 of [RFC6325] and in
   [RFC7177], can be used to obtain added assurance of the MTU of a
   link.

Yes, that can confirm working MTUs that are at 1470 or above, but appears
prevented from working below 1470?

Thus, it appears that there is a lack of mechanism here to actually get a valid
and functional MTU from TRILL in the cases where the Path MTU is below 1470. If
I am wrong good, but I think this is an important piece for how to handle the
next main issue.

UDP encapsulation and IP fragments.
I see it as a big issue that UDP encapsulation is the native one, and that
relies on IP fragmentation despite the need for reliable fragmentation. With
the setup of having to support 1470 MTU on TRILL level some packets will be
fragmented in many environments. That will lead to a lot of losses, and as
discussed below a very big problem with middleboxes. The main problem here is
that if one tries to rely on IP fragments one will have issues with packets
ending up in black holes. And different problems depending on IPv4 or IPv6.
IPv6 is lilkely the lesser problem assuming that one have working PMTUD.

There are several ways out of this.

1. Detect issues and use TCP encapsulation with correctly set MSS to not get IP
fragements 2. Determine MTU and implement an fragmentation mechanism on top of
UDP.

Zero Checksum:
--------------

Section 5.4:

UDP Checksum - as specified in [RFC0768]

Considering the fast path encapsulation desire, I am surprised to not see any
mentioning of use of zero checksum here. Raising the zero checksum and forward
reference would be good I think.

And then Section 8.5:

   The requirements for the usage of the zero UDP Checksum in a UDP
   tunnel protocol are detailed in [RFC6936]. These requirements apply
   to the UDP based TRILL over IP encapsulations specified herein
   (native and VXLAN), which are applications of UDP tunnel.

If you actually intended to allow zero checksum, then you actually should
document that Trill fulfills the requirements that the applicability statement
raises. I have not analyzed how well it meets these requirements.

Please review Section 6.2 of RFC 8086 for example how that can be done.

TCP Encapsulation issue
-----------------------

Section 5.6:

The TCP encapsulation appear to be missing an delimiter format allowing each
individual TRILL packet/payload to be read out of the TCP's byte stream. In
other words, a normal implementation has no way of ensuring that the TCP
payload starts with the start of a new TRILL payload. Multiple small TRILL
payloads may be included in the same TCP payload, and also only parts as TCP is
one way of dealing with TRILL packets that are larger than the IP+Encapsulation
MTU that actually will work.

This comment is based on that there appear to be no length fields included in
the TRILL header. The most straight forward delimiter is a 2-byte length field
for the TRILL payload to be encapsulated.

Section 5.6:

TCP endpoint requirements. I do wonder if an application like TRILL actual
would need to discuss performance impacting implementation choices or
limitations. For example use of NAGLE, the requirements on buffer sizes in
relation to Bandwidth delay products, as buffer memory in a RBridge will impact
performance.

Congestion Control
------------------
First thanks for the effort here.

8.1.2 In Other Environments

   Where UDP based encapsulation headers are used in TRILL over IP in
   environments other than those discussed in Section 8.1.1, specific
   congestion control mechanisms are commonly needed.  However, if the
   traffic being carried by the TRILL over IP link is already congestion
   controlled and the size and volatility of the TRILL IS-IS link state
   database is limited, then specific congestion control may not be
   needed. See [RFC8085] Section 3.1.11 for further guidance.

This is correct, however my question is if the RBridges have anyway of knowing
which traffic is actually congestion controlled, considering that TRILL provides
an layer 2 abstraction. I wonder if there should be any type of white list of
the types of layer 2 payloads that can be assumed to be congestion controlled,
and thus okay to forward over IP paths? I am worried that without any
recommendation to prevent traffic that is not controlled to be forwarded, can
lead to congestion issues.

The other issue I think may exist is the issue serial unicast emulation of
broadcast/multicast creates. As this amplifies the outgoing packet rate with
a factor of how many addresses are configured for serial unicast this can
be significant traffic expansion. Thus, I think additional considerations are
needed here, and maybe rate limiting of the amount of traffic to be multicasted.

Flow and ECMP
-------------

Section 8.3:

For example, for TRILL
   Data, this entropy field could be based on some hash of the
   Inner.MacDA, Inner.MacSA, and Inner.VLAN or Inner.FGL.

I would appreciate clearer references to what these fields are.

If I understand this correctly, the idea here is to look into the inner
layer 2 frames, and use the flow equivalents that exists on that level and
hash that into value that maps the flows onto the source port range.

I think this text should include a summary of the principle and ensure to
note the important requirement that what is considered flows in the inner
must not result in being striped over multiple source ports as this may lead to
reordering issues due to packets taking different paths.

NAT and TRILL over IP:
Section 8.5:

If one like to use TRILL over IP through a NAT, then there are some very
important considerations that are missing. First the need for static binding
configurations or the need for determining ones external address(es) and be
able to communicate that to the peer RBridges, and in addition ensure that one
has keep-alives to that the NAT binding never times out.

Next is the issue that there is almost zero chance of getting a IP/UDP
encapsulation TRILL payload through the NAT if it results in IP fragmentation,
as NATs don't do defragment and refragmented on the internal side, and an IP
fragment lacks UDP port and thus can't be matched to binding.

Also if you like to run IP/ESP through a NAT, then you most likely need the
IP/UDP/ESP encapsulation (https://tools.ietf.org/html/rfc3948). Note that this
will restrict the MTU even further and thus ensure that the 1470 requirement
cannot be fulfilled even without additional tunnels over an 1500 bytes MTU
Ethernet infrastructure.

I would note that also firewalls likely have issues with IP fragments for the
same reason, they require significant amount of state to be verified if they
should be let through.

In general I think you should create a configuration that has chance to work
through most middleboxes, but I think you should require static bindings. I
think that configuration is, and don't laugh now, but IP/UDP/ESP/TCP/TRILL,
otherwise you will not be able to have both security and reliable fragmentation
of TRILL packets.

Cheers

Magnus Westerlund

---

From: "Susan Hares" <[email-masked]>
Date: Thu, 15 Jun 2017 21:39:10 -0400

Magnus: 

Thank you for the very careful review.  Please let me chat with the authors regarding this document.  It make take some time, but each of your points will be responded to by the authors. 

Sue Hares 

-----Original Message-----
From: Magnus Westerlund [mailto:[email-masked]] 
Sent: Thursday, June 15, 2017 1:33 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Tsvart early review of draft-ietf-trill-over-ip-10

TSV-ART review comments:

I have set this to not ready as there are several issues, some significant that could affect the protocol realization significantly. Some may be me missing things in TRILL, I was not that familiar with it before this review and I have only tried looking up things, not reading the whole earlier specifications. So don't hesitate to push back and provide pointers to things that can resolve issues. The authors and the WG clearly have thought about a lot of issues and dealt with much already.

Diffserv usage
--------------

Section 4.3:

I think it is fine to require that implementations are capable  of setting DSCP values on the outer IP header. However, I fail to see any discussion of the potential issues with actually setting the DSCP values. It is one thing to do this in an IP back bone use case where one can know and have control over the PHB that the DSCP values maps to. But otherwise, over general internet the behavior is not that predictable. One can easily be subject to policers or remapping. Also as the actual DSCP code point usage is domain specific this is difficult. Priority reversal is likely the least of the problems that this can run into over general Internet.

Section 4.3:

This appear to be an problematic mapping. At least for prio 0 and 1. As priority 1 appears to be intended to be higher than priority 0, it is interesting that it is mapped to CS1, which to quote
https://datatracker.ietf.org/doc/rfc7657/:

So what is proposed can in a network using default mapping, result in that you get priority 0 to be lower priority than 1. Plus that in some networks this can also results in strange remapping that results in a different PHB for CS1 than.

MTU and Fragmentation
---------------------

I think there are two main issue here. The first one is MTUD discovery of the actual IP path MTU between the ports. That will be needed to prevent a lot of traffic going into MTU black holes. Especially as TRILL requries
1470 byte support which is likey above a lot of paths.

Section 8.4:

The issue with RFC4821 is that it has requirements on the packetization layer.
Trill appears to have several components that are useful. However, it will require a specification of the procedure to result in a useful tool.

Section 8.4:

Yes, that can confirm working MTUs that are at 1470 or above, but appears prevented from working below 1470?

Thus, it appears that there is a lack of mechanism here to actually get a valid and functional MTU from TRILL in the cases where the Path MTU is below 1470. If I am wrong good, but I think this is an important piece for how to handle the next main issue.

UDP encapsulation and IP fragments.
I see it as a big issue that UDP encapsulation is the native one, and that relies on IP fragmentation despite the need for reliable fragmentation. With the setup of having to support 1470 MTU on TRILL level some packets will be fragmented in many environments. That will lead to a lot of losses, and as discussed below a very big problem with middleboxes. The main problem here is that if one tries to rely on IP fragments one will have issues with packets ending up in black holes. And different problems depending on IPv4 or IPv6.
IPv6 is lilkely the lesser problem assuming that one have working PMTUD.

1. Detect issues and use TCP encapsulation with correctly set MSS to not get IP fragements 2. Determine MTU and implement an fragmentation mechanism on top of UDP.

Zero Checksum:
--------------

Section 5.4:

Considering the fast path encapsulation desire, I am surprised to not see any mentioning of use of zero checksum here. Raising the zero checksum and forward reference would be good I think.

And then Section 8.5:

If you actually intended to allow zero checksum, then you actually should document that Trill fulfills the requirements that the applicability statement raises. I have not analyzed how well it meets these requirements.

TCP Encapsulation issue
-----------------------

Section 5.6:

The TCP encapsulation appear to be missing an delimiter format allowing each individual TRILL packet/payload to be read out of the TCP's byte stream. In other words, a normal implementation has no way of ensuring that the TCP payload starts with the start of a new TRILL payload. Multiple small TRILL payloads may be included in the same TCP payload, and also only parts as TCP is one way of dealing with TRILL packets that are larger than the IP+Encapsulation MTU that actually will work.

This comment is based on that there appear to be no length fields included in the TRILL header. The most straight forward delimiter is a 2-byte length field for the TRILL payload to be encapsulated.

Section 5.6:

TCP endpoint requirements. I do wonder if an application like TRILL actual would need to discuss performance impacting implementation choices or limitations. For example use of NAGLE, the requirements on buffer sizes in relation to Bandwidth delay products, as buffer memory in a RBridge will impact performance.

Congestion Control
------------------

This is correct, however my question is if the RBridges have anyway of knowing which traffic is actually congestion controlled, considering that TRILL provides an layer 2 abstraction. I wonder if there should be any type of white list of the types of layer 2 payloads that can be assumed to be congestion controlled, and thus okay to forward over IP paths? I am worried that without any recommendation to prevent traffic that is not controlled to be forwarded, can lead to congestion issues.

The other issue I think may exist is the issue serial unicast emulation of broadcast/multicast creates. As this amplifies the outgoing packet rate with a factor of how many addresses are configured for serial unicast this can be significant traffic expansion. Thus, I think additional considerations are needed here, and maybe rate limiting of the amount of traffic to be multicasted.

Flow and ECMP
-------------

Section 8.3:

For example, for TRILL

If I understand this correctly, the idea here is to look into the inner layer 2 frames, and use the flow equivalents that exists on that level and hash that into value that maps the flows onto the source port range.

I think this text should include a summary of the principle and ensure to note the important requirement that what is considered flows in the inner must not result in being striped over multiple source ports as this may lead to reordering issues due to packets taking different paths.

NAT and TRILL over IP:
Section 8.5:

If one like to use TRILL over IP through a NAT, then there are some very important considerations that are missing. First the need for static binding configurations or the need for determining ones external address(es) and be able to communicate that to the peer RBridges, and in addition ensure that one has keep-alives to that the NAT binding never times out.

Next is the issue that there is almost zero chance of getting a IP/UDP encapsulation TRILL payload through the NAT if it results in IP fragmentation, as NATs don't do defragment and refragmented on the internal side, and an IP fragment lacks UDP port and thus can't be matched to binding.

Also if you like to run IP/ESP through a NAT, then you most likely need the IP/UDP/ESP encapsulation (https://tools.ietf.org/html/rfc3948). Note that this will restrict the MTU even further and thus ensure that the 1470 requirement cannot be fulfilled even without additional tunnels over an 1500 bytes MTU Ethernet infrastructure.

I would note that also firewalls likely have issues with IP fragments for the same reason, they require significant amount of state to be verified if they should be let through.

In general I think you should create a configuration that has chance to work through most middleboxes, but I think you should require static bindings. I think that configuration is, and don't laugh now, but IP/UDP/ESP/TCP/TRILL, otherwise you will not be able to have both security and reliable fragmentation of TRILL packets.

Cheers

Magnus Westerlund

---

From: Donald Eastlake <[email-masked]>
Date: Sun, 25 Jun 2017 20:07:13 -0400

Hi Magnus,

Thanks for the extensive review. See my responses below.

> issues. The authors and the WG clearly have thought about a lot of issues and

OK. Hopefully we can resolve these one way or the other.

> Diffserv usage

It sounds like appropriate discussion and warnings about these issues
would resolve the above comment.

>       TRILL Priority  DEI  DSCP Field (Binary/decimal)

The intent in the draft is to reflect the default relative priority of
the different priority code points in IEEE Std 802.1Q where priority 1
is lower than priority 0. At a quick look, it appears to me that RFC
2474 requires that 0x001000 be handled as being of a priority not
lower than the priority with which 0x000000 is handled. Yet RFC 3662,
which you point to, seems to suggest using 0x001000 as a lower
priority code point than 0x000000. Given that 3662 not only does not
update 2474 but is only Informational while 2474 is Standards Track, I
would say that 2474 dominates and that this draft makes the best
assumptions it can about default behavior...

> MTU and Fragmentation

Seems like it would depend on the environments where TRILL was used.
For example, I do not think 1470 would be a problem in most Data
Center or Internet Exchange point uses, for example. Data Centers
sometimes support 9K jumbo frames and the like.

In fact, it is probably bad to focus too much on 1470 -- that is a
required minimum to be sure that reasonable size link state PDUs can
be successfully flooded through the TRILL campus so that routing will
work. However, it would commonly be the case that, for the TRILL
campus to be useful in a particular case, links need to be able to
carry the expected size TRILL Data packets. For example, if there were
two parts of a TRILL campus connected by one or a few TRILL over IP
links and the end stations in each part were assuming they could use
1500 byte Ethernet packets, then the TRILL over IP links would need to
support an MTU based on 1500 + TRILL Header + IP and TRILL over IP
encapsulation. And more if security was being used or there were any
other reasons for additional headers/encapsulation...

> require a specification of the procedure to result in a useful tool.

See below.

> prevented from working below 1470?

While there is a minimum size for TRILL IS-IS MTU PDUs, determined by
header size, it is well below 1470, probably (depending on whether
secuirty is in use, etc.) below 150 bytes.

> I am wrong good, but I think this is an important piece for how to handle the

How about referencing Section 3 of
https://tools.ietf.org/html/draft-ietf-trill-mtu-negotiation-05
which is currently in IETF Last Call? (The wording of that section is
probably going to be improved based on an OPS review by Brian
Carpenter.)

> UDP encapsulation and IP fragments.
  ----------------------------------
> fragements 2. Determine MTU and implement an fragmentation mechanism on top of

So, I don't see that much problem with UDP being the general default
consistent with the TRILL philosophy of defaulting to need zero or
minimal configuration. The default should be to use multicast Hellos
for discovery of neighbors which sure points at UDP to me. Having to
traverse a NAT should be a rare case. Since, in the NAT case, you have
to configure things related to the static binding and the IP
address(es) of peer(s) anyway you can also configure to use a
different encapsulation than UDP, such as TCP, at the same time. I
don't see it as much of a problem if, by default, TRILL won't operate
through a NAT. If you are using UDP and it fragments and fragments are
dropped at a NAT, probably you can't exchange Hellos so you will not
form an adjacency and anything on the other side of the NAT will not
be visible.

> Zero Checksum:

OK. We'll look into it.

> TCP Encapsulation issue

Right. It might also be useful to include some sort of check field, as
is done in BGP, to detect if you are out of sync in parsing the TCP
stream.

Another point is that, while with UDP it seems fine to send packets
with assorted QoS, you don't want to encourage re-ordering of TCP
packets in a stream. So if TCP encapsulation is being used, you want
to use the same DSCP value for the packets in a particular TCP stream.
So, generally, you need to have a TCP connection per priority handling
category. Mapping the 8 priority levels into a smaller number of
handling categories is a normal thing to do so you certainly don't
necessarily need 8 TCP connections. Adding material on this should not
be too hard.

> relation to Bandwidth delay products, as buffer memory in a RBridge will impact

Well, I'm not sure how deeply this document should get into such
performance issues. What about just saying something about
consideration being given to tuning TCP for performance and pointing
to one or a few other RFCs that talk about this?

> Congestion Control

You're welcome.

> This is correct, however my question is if the RBridges have any way of knowing

OK. We can think about those issues.

> Flow and ECMP

In a TRILL Data packet, the payload after the TRILL Header looks like
an Ethernet frame except that there is always either a VLAN tag or,
alternatively, where the VLAN tag would be, a Fine Grained Label
[RFC7172]. (The preceding is the view in the TRILL RFCs, but there is
an equivalent and equally valid view in which all the fields through
and including the VLAN or FGL tag are part of the TRILL Header.) The
TRILL base protocol specification focuses on Ethernet as a link
technology between TRILL switches, in which case there will be a link
header including an Outer.MacDA and Outer.MacSA fields and possibly an
Outer.VLAN, all before the TRILL Header. See Figure 1 and Figure 2 in
RFC 7172.

Some of the above could be added to the draft for clarity.

> hash that into value that maps the flows onto the source port range.

Yes.

> reordering issues due to packets taking different paths.

Well, we can add some text. But when would the relative ordering
matter for two TRILL Data packets where the two inner native payloads
have different values for any one or more of these three fields
(Inner.MacDA, Inner.MacSA, and inner VLAN/FGL tag) ? If any of those
fields are different, you are talking about different streams.

> has keep-alives to that the NAT binding never times out.

I think those are good points. There is an additional problem that
TRILL Hellos detect neighbors with which they have 2-way connectivity
by indicating, inside the Hellos that are sent, from what neighbors
Hellos have been received on that port. If a NAT is involved, these
neighbor addresses inside Hellos need to be mapped.

> fragment lacks UDP port and thus can't be matched to binding.

So perhaps the recommendation should be to configure the port to use
TCP if there will be fragmentation.

> otherwise you will not be able to have both security and reliable fragmentation

OK. Thanks again for this review. It has pointed out a number of
problems and in thinking about those, I believe a couple of further
problems have come to mind that I mentioned above. We'll work on a
revised draft.

Thanks,
Donald
===============================
 Donald E. Eastlake 3rd   +1-508-333-2270 (cell)
 155 Beaver Street, Milford, MA 01757 USA
 [email-masked]

---

From: Joe Touch <[email-masked]>
Date: Mon, 26 Jun 2017 10:15:15 -0700

Hi, Donald,

On 6/25/2017 5:07 PM, Donald Eastlake wrote:
> ...

There is nothing in BGP that ever assumes that TCP write boundaries are
preserved. BGP uses markers and length fields to create message
boundaries in TCP's bytestream. The same is needed here.

Note that BGP also never claims to craft TCP packets by 'encapsulating'
a BGP message in a TCP segment. That part of this document needs to be
removed - it not how TCP is ever used.

> packets in a stream. So if TCP encapsulation is being used,
Again - please, NO. NEVER use this term.

> you want
Again, this is nonsensical. TCP would set a DSCP for the connection,
never in different ways for individual segments of a connection.

> necessarily need 8 TCP connections. Adding material on this should not
Perhaps, but please - again, please - omit any mention or implication
that this occurs via encapsulation.

If you want to use TCP, please use it properly.

> to one or a few other RFCs that talk about this?
Because your use of TCP (even if changed to describe it correctly) isn't
listed in those TCP RFCs.

And it's not so simple - NAGLE helps performance for interactive systems
that use single-byte messages (e.g., telnet) and reduces the number of
outstanding "less than full" segments. When used for encapsulation,
turning NAGLE off is the right thing for multibyte messages (e.g., TRILL
messages) and can avoid the "gathering" delay (200 ms stalls when there
isn't enough source data - i.e., incoming TRILL packets - to keep up
with the outgoing segments), but could also generate a large number of
small segments (which can interfere with segment-based congestion
control, vs. ABC).

Unless you want a very poorly performing result, *THIS* is what you need
to drill down into.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 26 Jun 2017 11:42:16 -0700

PS - the idea that TCP segments within a single connection should ever
have different DSCPs is a good example of why it's a bad idea to even
'think' of TRILL over TCP as direct encapsulation. I.e., that concept is
inherently hazardous and should be avoided.

Joe

On 6/26/2017 10:15 AM, Joe Touch wrote:
> _______________________________________________
> trill mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/trill

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 27 Jun 2017 17:13:09 +0000

Hi Donald,

After having read your response I think there is an important question about the applicability of this document that affects several of the issues below and what solution you need. That is the question of what type of paths one expect to get Trill over IP working over. Because if the target is general Internet and also through middleboxes such as NATs and Firewall (Not intending to block) then there are a lot more work to ensure this. If you for example changes the applicability to require any on path middleboxes to fulfill certain requirements things can be more easily addressed.

Den 2017-06-26 kl. 02:07, skrev Donald Eastlake:

Hi Magnus,

On Thu, Jun 15, 2017 at 1:32 PM, Magnus Westerlund
<[email-masked]><mailto:[email-masked]> wrote:

Diffserv usage
--------------

Section 4.3:

I would note that the choice of encapsulation here do becomes important. Your's and Joe Touch's observation that for TCP, you can only have a single DSCP marking per TCP connection for example. For others, see the discussion in Section 5.1 of https://datatracker.ietf.org/doc/rfc7657/ on this issue.

David Black also raised an important question if one should treat this as a tunnel with a single predictable behavior or let the inner networks marking show through. Establishing a tunnel with a single PHB has less risk of running into issues than multiple different markings.

Section 4.3:

David Black provide a good answer on this.

MTU and Fragmentation
---------------------

Yes, and over general Internet you should be happy if you get 1500 bytes of IP MTU, it may easily be lower with a couple of additional tunnel headers. Thus, what you say is the goal is not feasible without a solution that supports fragmentation and reassembly, enabling one TRILL packet to be sent in multiple IP packets. The re-assembly do requires buffering and not something to easily perform on a router fast path. And attempting to use IP fragmentation is likely doomed if you have any type of NAT or Firewall in the way.

This points to a dedicated solution or using a transport protocol that supports carrying arbitrary data sizes, like TCP or SCTP. And you need to use the byte-stream API of TCP to achieve this.

Section 8.4:

See below.

Section 8.4:

Okay, if you say so, it was not obvious from the spec that is was allowed to probe for paths with lesser MTUs than 1470.

I looked at this, and it appears to have the same issue, that it can't probe for MTU values below 1470.

   2) RB1 tries to send an MTU-probe padded to the size 1470.

      a) If RB1 fails to receive an MTU-ack from RB2 after k tries, RB1
         sets the "failed minimum MTU test" flag for RB2 in RB1's Hello
         and stop.

But, the algorithm clearly performs a binary search for the MTU. If
one look at RFC 4821 one will notice that there are some additional considerations
there how to make the probing better and robuster. But, cleary Trill has some other
criterias for what is a success. Verification that Sz works appears sufficient,
and there are no need to probe further upwards.

Yes, but this is the issue of applicability and documenting that applicability. I don't know what goals and requirements that exist for Trill. If the WG are fine with some restrictions, then document them and focus on solving the issues that must be solved.

You can clearly choose to require TCP for cases where the IP MTU is insufficient for carrying the Sz sized trill packets between the RBs using UDP.

Zero Checksum:
--------------

Section 5.4:

And then Section 8.5:

OK. We'll look into it.

TCP Encapsulation issue
-----------------------

Section 5.6:

As you need to actually perform re-assembly, the solution is to use the byte stream semantics the TCP API provides and have a framing for each packet.

Yes, agreed it is a possibility and points into possible considerations that David raised.

Section 5.6:

As Joe said, these are important considerations. If your intention is to enable this to run at substantial fractions of line rates of the interfaces. Then this do require considerations.

Congestion Control
------------------

You're welcome.

Flow and ECMP
-------------

Section 8.3:

For example, for TRILL

Yes.

Okay, then this is very straightforward.

NAT and TRILL over IP:
Section 8.5:

Yes, and the question is how that can be handled, by the receiver of the packet, or if the sender needs to determine what address it uses and provide that in the HELLOs. If the first is possible that can simplify a lot.

Yes, I think that are likely the simplest solution for you.

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Media Technologies, Ericsson Research
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Torshamnsgatan 23           | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]<mailto:[email-masked]>
----------------------------------------------------------------------

---

From: Donald Eastlake <[email-masked]>
Date: Thu, 29 Jun 2017 22:17:11 -0400

HI Magnus,

On Tue, Jun 27, 2017 at 1:13 PM, Magnus Westerlund <
[email-masked]> wrote:

> Hi Donald,
>
> After having read your response I think there is an important question
> about the applicability of this document that affects several of the issues
> below and what solution you need. That is the question of what type of
> paths one expect to get Trill over IP working over. Because if the target
> is general Internet and also through middleboxes such as NATs and Firewall
> (Not intending to block) then there are a lot more work to ensure this. If
> you for example changes the applicability to require any on path
> middleboxes to fulfill certain requirements things can be more easily
> addressed.
>

The use cases in the document support communication over the general
Internet in the brach office case but that does not necessarily imply
NATs/Firewalls.

Den 2017-06-26 kl. 02:07, skrev Donald Eastlake:
> On Thu, Jun 15, 2017 at 1:32 PM, Magnus Westerlund<[email-masked]> <[email-masked]> wrote:
>
>
> Diffserv usage
> --------------
>
> Section 4.3:
>
> I would note that the choice of encapsulation here do becomes important.
> Your's and Joe Touch's observation that for TCP, you can only have a single
> DSCP marking per TCP connection for example. For others, see the discussion
> in Section 5.1 of https://datatracker.ietf.org/doc/rfc7657/ on this
> issue.
>

Well, if a TRILL over IP implementation using TCP transport wants to have
more than one priority category for traffic where there might be one or
more intervening IP routers, which would be the normal case, it would just
need a TCP connection per priority category. Mapping the 8 priority levels
into a smaller number of categories is a routine thing to do. Note that the
base TRILL protocol specification (RFC 6325) says:

   RBridges are not required to implement any
   particular number of distinct priority levels but may treat one or

   more adjacent priority levels in the same fashion.

> David Black also raised an important question if one should treat this as
> a tunnel with a single predictable behavior or let the inner networks
> marking show through. Establishing a tunnel with a single PHB has less risk
> of running into issues than multiple different markings.
>

It is an implementation choice whether to have a single PHB or eight, one
per priority level, or something in between.

> interesting that it is mapped to CS1, which to quotehttps://datatracker.ietf.org/doc/rfc7657/:
>

I'll reply to him.

> Yes, and over general Internet you should be happy if you get 1500 bytes
> of IP MTU, it may easily be lower with a couple of additional tunnel
> headers. Thus, what you say is the goal is not feasible without a solution
> that supports fragmentation and reassembly, enabling one TRILL packet to be
> sent in multiple IP packets. The re-assembly do requires buffering and not
> something to easily perform on a router fast path. And attempting to use IP
> fragmentation is likely doomed if you have any type of NAT or Firewall in
> the way.
>
> This points to a dedicated solution or using a transport protocol that
> supports carrying arbitrary data sizes, like TCP or SCTP. And you need to
> use the byte-stream API of TCP to achieve this.
>

OK.

> Okay, if you say so, it was not obvious from the spec that is was allowed
> to probe for paths with lesser MTUs than 1470.
>
> How about referencing Section 3 ofhttps://tools.ietf.org/html/draft-ietf-trill-mtu-negotiation-05

I think the thing was that, before TRILL over IP, it would not have been
useful to determine an MTU below 1470. But there is no particular problem
in constructing a smaller MTU-probe PDUs and an RBridge receiving such a
PDU is generally required to respond with an equal length MTU-ack.

> Yes, but this is the issue of applicability and documenting that
> applicability. I don't know what goals and requirements that exist for
> Trill. If the WG are fine with some restrictions, then document them and
> focus on solving the issues that must be solved.
>
> You can clearly choose to require TCP for cases where the IP MTU is
> insufficient for carrying the Sz sized trill packets between the RBs using
> UDP.
>

OK.

> As you need to actually perform re-assembly, the solution is to use the
> byte stream semantics the TCP API provides and have a framing for each
> packet.
>

Of course.

My point was that the framing might usefully have some sort of flag field,
like the BGP framing has, so that there was a good chance of detecting if
the parsing of the byte stream into frames has gotten out of synch.

> As Joe said, these are important considerations. If your intention is to
> enable this to run at substantial fractions of line rates of the
> interfaces. Then this do require considerations.
>

I see.

> Yes, and the question is how that can be handled, by the receiver of the
> packet, or if the sender needs to determine what address it uses and
> provide that in the HELLOs. If the first is possible that can simplify a
> lot.
>

I'm not sure. this would require a little detailed design work.

> Yes, I think that are likely the simplest solution for you.

OK

Thanks,
Donald
=============================
 Donald E. Eastlake 3rd   +1-508-333-2270 (cell)
 155 Beaver Street, Milford, MA 01757 USA
 [email-masked]

> Ericsson AB                 | Phone  +46 10 7148287 <+46%2010%20714%2082%2087>
> Torshamnsgatan 23           | Mobile +46 73 0949079 <+46%2073%20094%2090%2079>
> SE-164 80 Stockholm, Sweden | mailto: [email-masked]