Subject: [armd] review of

From: Anoop Ghanwani <[email-masked]>
Date: Thu, 3 May 2012 19:21:54 -0700

As one of the people who had volunteered to review
draft-ietf-armd-problem-statement-02, here are my
comments on the document.  They are mostly editorial
but there are a few minor issues with the content.
Otherwise, the document accurately captures the gist
of the problem.

Anoop

=====================

All sections
==========
For consistency
Data Center -> data center
Access Layer, Access layer -> access layer
Aggregation Layer, Aggregation layer -> aggregation layer
Layer 2, layer 2 -> L2
Layer 3, layer 3 -> L3
Use ARP Request or ARP request consistently (prefer ARP Request)

Section 1
=========
datacenters -> data centers

Section 2
=========
In the definition for VM:
Replace "bare" with "physical, non-virtualized".

In the definition for EoR:
Delete extraneous "network".

Section 3
=========
Change
"poor implementations of loop detection and prevention"
to
"poor implementations of loop detection and prevention or
misconfiguration errors".

Change
"With virtualization, a single physical server can host 10 (or more)
VMs, each having its own IP (and MAC) addresses"
to
"With virtualization, a single physical server can host many VMs,
each having its own network
interfaces with IP and MAC addresses."
[The number is captured in a later sentence.]

"services" is used all over the place, but there is no definition
for services.  Äpplication, however, is defined.  I think what is
meant here is application.

Last paragraph:
These number -> This number.

Below Figure 1:
Delete "Figure 1".

Section 4.1
==========

Change
"The access switches might be placed either on
  top-of-rack (ToR) or at end-of-row (EoR) physical configuration."
to
"The access layer may be implemented by wiring the
servers within a rack to a  top-of-rack (ToR) switch or,
less commonly, the servers could be wired directly to an
end-of-row (EoR) switch."

Section 4.4.1
============

For consistency with the following 2 sections
change title from Layer 3 to L3.

"This topology is ideal for scenarios where servers
  attached to a particular access switch generally run applications
  that are are confined to using a single subnet."
I'm not sure I agree with this.  There are many issues
surround this including the capabilities of the devices
in the network, the use of the multicast, and the preferences
of the network administrator.

Section 4.4.2
============
"This topology allows for a great deal
   of flexibility as servers attached to one access switch can be re-
   loaded with applications with different IP prefix and VMs can now
   migrate between racks without IP address changes. "
to
"This topology allows a greater level of flexibility as
servers attached to any access switch can be reloaded
with applications and provisioned with IP addresses
from multiple prefixes as needed.  Further, in such
an environment, VMs can migrate between racks without
IP address changes.

Change
"layer 2 traffic are still partitioned by VLANs"
to
"layer 2 traffic is still partitioned using VLANs"

"Even though
   layer 2 traffic are still partitioned by VLANs, the fact that all
   VLANs are enabled on all ports can lead to broadcast traffic on all
   VLANs to traverse all links and ports, which is same effect as one
   big Layer 2 domain. "
I disagree with this because all VLANs would only
need to be provisioned on the aggregation-facing ports.
The disadvantage here is that a lot more broadcast traffic
hits the aggregation layer, and when we need to cross
VLAN boundaries, the traffic must go all way to the
aggregation switch even though the source and destination
may be on the same access switch, and the requirement
for larger ARP tables at the aggregation switches.

Section 4.4.4
=============
"   There are several approaches regarding how overlay networks can make
   very large layer 2 network scale and enable mobility. "
to
"There are several approaches where overlay networks
can be used to build very large L2 networks to enable VM mobility"

"This can help the data
   center designer to control the size of the L2 domain.  "
It should be clarified that this only applies when L3 overlays
are in use.

"However, the
   Overlay Edge switches/routers which perform the network address
   encapsulation/decapsulation must ultimately perform a L2 address
   resolution and could still potentially face scaling issues at that
   point."
It's not the overlay edge switches that have the scaling
problem, its the volume of broadcasts that need to be
sent across the core and that is not helped simply by
using an L3 overlay.

Change
"physical addresses (MAC)"
to
"physical (MAC) addresses"

For consistency, change
"Layer 2 / Layer 3 boundary"
to
L2/L3 boundary

Section 4.5
===========

Change
"appropriately sized Access, Aggregation and Core networks"
to
"appropriately sized access, aggregation and core layers"

Change
"Broadly speaking it is desirable"
to
"Broadly speaking, it is desirable"

Section 5
=========
ARP response -> ARP Reply

rerun ARP -> reissue an ARP Request

Section 6
==========

"Thus, whereas all
   nodes must process every ARP query, ND queries are processed only by
   the nodes to which they are intended."
When virtualization is in use, the NIC is often operated
in promiscuous mode, which means that the packet would
be delivered to the hypervisor/vswitch and the filtering
would have to be done there (usually implemented in software),
making the problem almost as bad as with ARP.

Section 7.1
=========
ARP query -> ARP Request

Change
   "One common router implementation architecture has ARP processing
   handled"
to
"ARP processing in routers is commonly handled..."

revalidate timer -> aging timer

their ASIC fast paths -> their forwarding ASICs

target's Subnet -> target's subnet

Section 7.2
===========

nodes to which they are intended -> nodes for which they are intended

Section 7.3
===========
ten (or  more) VMs -> many VMs
(in the 10's today, but growing rapidly as the number of cores per CPU
increases)

---

From: Thomas Narten <[email-masked]>
Date: Fri, 25 May 2012 15:43:40 -0400

H Anoop.

Thanks for your review very detailed review comments. I've adopted
most of them directly. Some questions below.

> of the network administrator.

I agree that "is ideal" is too strong. How about if I say instead:

    This topology has benefits in scenarios ...

Would that address your concerns?

> for larger ARP tables at the aggregation switches.

I struggled with this for a long while. Is this text any better?:

     <t> When the L3 domain only extends to aggregation switches,
         hosts in any of the IP subnets configured on the aggregation
         switches can be reachable via L2 through any access switches
         if access switches enable all the VLANs.  This topology
         allows a greater level of flexibility as servers attached to
         any access switch can be reloaded with applications that have
         been provisioned with IP addresses from multiple prefixes as
         needed.  Further, in such an environment, VMs can migrate
         between racks without IP address changes.  The drawback of
         this design however is that multiple VLANs have to be enabled
         on all access switches and all access-facing ports on
         aggregation switches. Even though L2 traffic is still
         partitioned by VLANs, the fact that all VLANs are enabled on
         all ports can lead to broadcast traffic on all VLANs to
         traverse all links and ports, which is same effect as one big
         L2 domain on the access-facing side of the aggregation
         switch.  In addition, internal traffic itself might have to
         cross different L2 boundaries resulting in significant ARP/ND
         load at the aggregation switches.  This design provides a
         good tradeoff between flexibility and L2 domain size.  A
         moderate sized data center might utilize this approach to
         provide high availability services at a single location.
         </t>

> sent across the core and that is not helped simply by

I also struggled quite a bit with this comment. Is the following an
improvement?:

    <t> A potential problem that arises in a large data center is when
        a large number of hosts communicate with their peers in
        different subnets, all these hosts send (and receive) data
        packets to their respective L2/L3 boundary nodes as the
        traffic flows are generally bi-directional.  This has the
        potential to further highlight any scaling problems.  These
        L2/L3 boundary nodes have to process ARP/ND requests sent from
        originating subnets and resolve physical (MAC) addresses in
        the target subnets for what are generally bi-directional
        flows.  Therefore, for maximum flexibility in managing the
        data center workload, it is often desirable to use overlays to
        place related groups of hosts in the same topological subnet
        to avoid the L2/L3 boundary translation.  The use of overlays
        in the data center network can be a useful design mechanism to
        help manage a potential bottleneck at the L2 / L3 boundary by
        redefining where that boundary exists.  </t>

> making the problem almost as bad as with ARP.

Revised text:

    Thus, whereas all nodes must process every ARP query, ND queries
    are processed only by the nodes to which they are intended. In
    cases where multicast filtering can't effectively be implemented
    in the NIC (e.g., as on hypervisors supporting virualization),
    filtering would need to be done in software (e.g., in the
    hypervisor's vSwitch).

Thomas