Subject: Power consumption reduction in edge and core routers in the internet

From: balaji venkat Venkataswami <[email-masked]>
Date: Sat, 08 Sep 2012 14:19:58 +0530

Dear all,

After due consultation with the chairs and IRTF folks we have been
instructed to ask for discussion on the following drafts that deal with
power consumption reduction in edge and core devices in the internet.

We deal with the problem in a hierarchical approach. One within an
Autonomous System and the other between Autonomous Systems and hence for the
whole internet.

We duly submit to you this work that we have done for your kind review and
consideration.

We deal with solutions for unicast and multicast at the AS level and the
intra-AS level. Multicast solutions mostly in the intra-AS level.

We submit this work and request you to comment on it.

Multicast intra-AS level
* mjsraman-pce-power-replic
<http://tools.ietf.org/html/draft-mjsraman-pce-power-replic>  (timeline)
<http://www.arkko.com/tools/lifecycle/draft-mjsraman-pce-power-replic-timing
.html> 
* mjsraman-pim-ecmp-redirect-power-replic-cap
<http://tools.ietf.org/html/draft-mjsraman-pim-ecmp-redirect-power-replic-ca
p>  (timeline) 
<http://www.arkko.com/tools/lifecycle/draft-mjsraman-pim-ecmp-redirect-power
-replic-cap-timing.html>
Unicast AS level
* mjsraman-rtgwg-bgp-power-path
<http://tools.ietf.org/html/draft-mjsraman-rtgwg-bgp-power-path>  (timeline)
<http://www.arkko.com/tools/lifecycle/draft-mjsraman-rtgwg-bgp-power-path-ti
ming.html> 
* mjsraman-rtgwg-inter-as-psp
<http://tools.ietf.org/html/draft-mjsraman-rtgwg-inter-as-psp>  (timeline)
<http://www.arkko.com/tools/lifecycle/draft-mjsraman-rtgwg-inter-as-psp-timi
ng.html> 
* mjsraman-rtgwg-inter-as-psp
<http://tools.ietf.org/html/draft-mjsraman-rtgwg-inter-as-psp>  (timeline)
<http://www.arkko.com/tools/lifecycle/draft-mjsraman-rtgwg-inter-as-psp-timi
ng.html> 
* mjsraman-rtgwg-inter-as-psp-protect
<http://tools.ietf.org/html/draft-mjsraman-rtgwg-inter-as-psp-protect>
(timeline) 
<http://www.arkko.com/tools/lifecycle/draft-mjsraman-rtgwg-inter-as-psp-prot
ect-timing.html> 
Unicast intra-AS level
* mjsraman-rtgwg-intra-as-psp-te-leak
<http://tools.ietf.org/html/draft-mjsraman-rtgwg-intra-as-psp-te-leak>
(timeline) 
<http://www.arkko.com/tools/lifecycle/draft-mjsraman-rtgwg-intra-as-psp-te-l
eak-timing.html> 
* mjsraman-rtgwg-ospf-power-topo
<http://tools.ietf.org/html/draft-mjsraman-rtgwg-ospf-power-topo>
(timeline) 
<http://www.arkko.com/tools/lifecycle/draft-mjsraman-rtgwg-ospf-power-topo-t
iming.html> 
Multicast intra-AS level
* mjsraman-rtgwg-pim-power
<http://tools.ietf.org/html/draft-mjsraman-rtgwg-pim-power>  (timeline)
<http://www.arkko.com/tools/lifecycle/draft-mjsraman-rtgwg-pim-power-timing.
html> 

Thanks again to the WG and IETF chairs for letting us share our work with
this group.

Thanks and regards
Shankar and team

---

From: Curtis Villamizar <[email-masked]>
Date: Sat, 08 Sep 2012 16:02:34 -0400

In message <[email-masked]>
 
> Thanks again to the WG and IETF chairs for letting us share our work with

My first reaction was that this was just a really bad idea from some
students who didn't understand very much about how provider networks
were built.

But looking at some of the citations I'm wondering if this whole thing
is just a joke.

  draft-mjsraman-rtgwg-ospf-power-topo-01

   [EVILBIT]  Bellovin, S., "The Security Flag in the IPv4 Header",
              RFC 3514, April 1 2003.

   [RFC5513]  Farrel, A., "IANA Considerations for Three Letter
              Acronyms", RFC 5513, April 1 2009.

   [RFC5514] Vyncke, E., "IPv6 over Social Networks", RFC 5514, April
             1 2009.

I'm really hoping the whole thing is a joke, but unfortunately I do
think that you are seriously proposing this.  Just confirming first.

Curtis

---

From: Balaji Venkat <[email-masked]>
Date: Sun, 9 Sep 2012 02:15:27 +0530

Dear Curtis,

We used nroffedit for preparing these drafts and forgot to delete the default references that got stuck in there. 

We intended these to be serious material.

If you could comment on the specifics of why you disagree with our proposals it would be most beneficial.

Regards
Balaji

Sent from my iPad

---

From: Curtis Villamizar <[email-masked]>
Date: Sat, 08 Sep 2012 18:57:00 -0400

In message <[email-masked]>
Balaji Venkat writes:
 
> Dear Curtis,
>  
> We used nroffedit for preparing these drafts and forgot to delete the
> default references that got stuck in there.
>  

Without too lengthy a response I have to say that I don't think your
approach is going to yield much if any benefit.

First reducing power in core and edge are very different problems.

There are a few things in the core.  For example:

  1.  Constraining the number of forwarding entries to get the
      existing external TCAM back into the forwarding chip and doing
      the forwarding with a less power greedy method than TCAM.

  2.  If possible eliminating the very large buffer on one side of a
      switching fabric, making it small enough to use on chip RAM and
      putting the bulk of the buffering on one side.  Some think
      putting enough overspeed in the fabric and putting large buffer
      on the output side may be a solution.  Others favor a VOQ
      approach.  The extreme memory bandwidth needed to go to external
      DRAM is the culprit.  As on chip SRAM sizes increase over time
      this may be possible.  Putting external DRAM on both sides of a
      fabric is a big power drain.

The first above is supported by using MPLS with a BGP free core (no IP
forwarding except for the small number of prefixes needed for control
traffic and then just a set of MPLS labels to traverse the topology.
Comments on LDP vs RSVP-TE to /dev/null please (not important here).
Comments on IPFRR are also not relevant as IPFRR doesn't address this.

For edge, forwarding speeds are much lower (about 3 decimal orders of
magnitude give or take one) and buffering requirements are smaller.
On chip buffering of one side is clearly feasible but needs to be done
right.  At the very low end, on chip buffering on both ends might be
feasible.  Access is further out than edge and on chip buffer should
be sufficient.  With IP, if global routing is needed, then an external
TCAM is generally used, though some chips will use external DRAM
(realizing that multiple accesses would be needed for a trie
approach).  I'm far less familiar with where state-of-the-art is for
edge and access.

For large networks the parallel links provide a further potential for
power reduction.  Generally some traffic must be run for monitoring
even if the link is unused.  However in principle on-chip power
islands could be used to eliminate the leakage current that unused
power forwarding engines would draw, while keeping the minimal
circuitry needed for an OAM or link layer equivalent active.  I don't
know any chip that does this.  Generally that would be taking
component links out of a parallel set of links (for example a wide LAG
of 100Gb/s links).

Powering down the laser is very problematic in long haul (LH).  The LH
optical systems monitor and balance power levels on the optical fiber.
Lighting up additional wavelengths on a WDM can take minutes before
power levels are adjusted right and transmit and receive sides and all
the optical amplifiers along the way are sufficiently in sync to bring
bit error rates down.  LH optics and modulation is a black art,
particularly ultra LH (ULH) and subsea (no OEO for up to 6000km) and
right now shutting lasers down can be disruptive.  The optical
amplifier pump lasers have to be powered at about the same levels
regardless of how many optical channels are carried.

This does not apply to edge and access.  Between edge and access, the
physical topology is often a set of logical rings over WDM with ROADM
add/drop.  Taking down one side of each ring would not be viable.  If
there is parallel links or other redundancy, then shutting down lasers
in short haul (less than 40 or less than 100 km) is feasible.

Moving "high touch" functions such as deep packet inspection and
filtering to edge helps.  For core where a provider peering might
require filtering, keeping the rule set very small can keep it on
chip.  Adding still more TCAM or DRAM for filtering drive power up.
If a card must be all things, then it would help to allow those
components to be powered down if not needed and the forwarding chip
SERDES facing them to be powered down if not needed (internal to core
for example, rather than peering).

So I really think that you haven't started by stating a valid problem
other than recognizing that reducing power would be a good thing.

Maybe you should start over with one draft stating the problem.
Identify where the power goes in a provider core network or content
provider data center (or other extremely large data center) or in a
service provider edge and access.  Is it forwarding logic?  Is it chip
externals such as forwarding TCAM or DRAM, buffer SERDES/DRAM, filter
TCAM or DRAM, switching fabric, etc.  What is the contribution of
optical components driving the line?  (hint: don't look here).

Then once you know where the power goes, try to fix it.  IMHO you'll
end up with a very different approach.

Well - I suppose that wasn't as concise as it could be.

Curtis

---

From: balaji venkat Venkataswami <[email-masked]>
Date: Sun, 09 Sep 2012 17:34:30 +0530

Dear Curtis,

Thank you for your detailed reply on this matter.
Agreed. We should definitely try optimising at the device level.
What you have suggested all of them fall at the device level.

Once you have reduced power to the extent possible at the device level
there needs to be a protocol based approach to ascertain and discover
which devices consume the least power in the topology (in a system of
devices and interconnecting links). After all the system level of looking
at a collection of such devices and links matters too.

Our approach in these documents refers to the method of discovering such
low power consuming devices in a topology of such devices and routing the
traffic through them. We leave it to others to concentrate on the device
level while we apply algorithms and protocols at the intra-AS and inter-AS
level. 

Every one of our ideas in these drafts concentrates at a system level
where such a system consists of more than ONE device and not just a device
in isolation. After all there is bound to be differences in power
consumption at the device level between two or more devices in a topology
of nodes and links. This difference we exploit to direct traffic through
those nodes and links that are efficient in power consumption and hence we
reward the devices in an intra-AS scenario and Ases in an inter-AS
scenario with more traffic. The greater the load they can take on without
consequent linear or more than linear power consumption the better the
internet is with regard to system / Internet level power consumption.

Hope this helps.

Thanks and regards,
Shankar and team

---

From: Mick Seaman <[email-masked]>
Date: Sun, 09 Sep 2012 10:58:01 -0700

Balaji,

You say below "Once ... there needs to be a *protocol based* approach to 
ascertain and discover which devices ..."

Frankly I doubt it.

If the network administrators/operators are truly interested (which I 
don't doubt) they will know the power/load characteristics of their 
devices and can feed that data into the necessary computation. Having a 
protocol run around and auto discover such input information seems to be 
just one additional source of uncertainty/instability. The possibility 
of such a protocol mediating between two connected 
administrators/operators as to who should bear the power load is far 
fetched.

Moreover, in the absence of a prior study as to what the power load 
characteristics of the devices of interest actually are, this seems too 
theoretical for words. The usable characteristics are likely to be (so 
far as they are really predictable) device/interface on (draws power) or 
off (does not) with some interfaces capable of operating in a lower 
power/much reduced data rate mode. An operator can assess the network 
load using some offline tool and then decide what devices to switch 
on/off/into low power mode - with a clear and obvious feed through into 
link availability/routing metric. This is of course already being done 
today.

Mick

On 9/9/2012 5:04 AM, balaji venkat Venkataswami wrote:
> _______________________________________________
> rtgwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtgwg
>

---

From: Curtis Villamizar <[email-masked]>
Date: Sun, 09 Sep 2012 17:48:49 -0400

Banji,

Sorry if I have to get a bit harsh in my wording, but your persistance
forces it.

In message <[email-masked]>
 
> What you have suggested all of them fall at the device level.

You have no basis for the statement "What you have suggested all of
them fall at the device level".

I just spent a good part of my time for the last few year working at
an transport equipment vendor and working with a small number of chip
vendors who make high end packet forwarding chips and have a good
understanding of where the power goes in some of today's high end
routers and have a good relationship with number of service providers
who would like to reduce their power draw.  Prior to that I was at a
high end router vendor.  Prior to that in Internet operations and
routing software work for the NSFNET.

I suspect that you've never even seen a high end router such as a
CRS-1 or CRS-3 or T-1600 and have no idea what the power feeds look
like, let along where in those big 10+ KW chassis the power goes.  If
you did have any idea where the power goes, you would not make a
statement like "What you have suggested all of them fall at the device
level".

> at a collection of such devices and links matters too.

You have no basis for the assertion "there needs to be a protocol
based approach to ascertain and discover which devices consume the
least power in the topology".  If you talk to someone at a service
provider you won't hear this.  Mick just expressed his doubt.

> internet is with regard to system / Internet level power consumption.

I'm amused by "where such a system consists of more than ONE device
and not just a device in isolation".  Did you think anyone here
thought a core router had only one chip in it?  Do you have any clue
what the "fabric" I mentioned was?

I think it is time for you to stop making uniformed assertions and go
do your homework and find out what service provider topologies look
like, what type of equipment they deploy, where the power goes within
those types of equipment and therefore the topology as a whole, and
then think about how to reduce it.

> Shankar and team

The question for the WG is whether this work is of any interest to the
IETF.  My opinion is a *NO*, there is no interest in this work.

At this point we need to hear other opinions, but so far I hear no
support.

Curtis

---

From: Tony Li <[email-masked]>
Date: Sun, 9 Sep 2012 15:29:08 -0700

> At this point we need to hear other opinions, but so far I hear no

I have an interest in this area, but like Curtis, would like to suggest that we need to understand the degrees of freedom that are possible and practical.  We should try to focus our directions on the pragmatic.

Like Curtis, I'm highly skeptical of the ability or opportunity for a provider to shut down an entire system.  If that were truly practical, it would imply that the system would be wholly unnecessary for primary or protect paths.  While there is a substantial optimization possible because of diurnal traffic patterns, I suspect that the granularity of the power savings is at the link level, not at the system level.

Unlike Curtis, I'm not convinced that manipulating DWDM optics on the fly is an insurmountable challenge.  I agree that it is non-trivial and certainly not something supportable immediately, but as our control systems mature, it is not inconceivable that we will someday get to the point where this can be automated.

Thus, the net is that the real power savings that are possible today are at the NPU (and associated fabric) level.  Having a protocol to support this, as well as the TE facilities to automate the process seem like a worthwhile direction.  And, as always, solving problems at the intra-AS level is far, far easier than addressing them at the inter-AS level.

Regards,
Tony