Subject: [Nfvrg] Requesting review of and comments on draft-mlk-nfvrg-nfv-reliability-using-cots

From: "B. Khasnabish" <[email-masked]>
Date: Wed, 7 Oct 2015 11:11:13 -0400

Dear All,

We have recently published a draft
(*datatracker.ietf.org/doc/draft-mlk-nfvrg-nfv-reliability-using-cots/*) on
"NFV Reliability using COTS Hardware" discussing the results of a recent
study on the feasibility of using Commercial Off-The-Shelf (COTS) hardware
for virtualized network functions in telecom equipment.

This draft is based on the presentation
(https://www.ietf.org/proceedings/92/slides/slides-92-nfvrg-9.pdf)
during IETF/IRTF-92 mtg. in Dallas, TX, USA on
the 26th of March 2015. This draft attempts to address the work item
no. 4 in the charter (http://datatracker.ietf.org/rg/nfvrg/charter/)

-------------------------------------------------------------------------------
4. Service Verification with Regards to Security and Resiliency

Reliability and security issues and relevant solutions related to the
nature of VNFs are the objectives of this work item.

----------------------------------------------------------------------------------
We are requesting your review of this draft and comments on it
so that we can update the draft ASAP.

Many thanks in advance for your kind attention.

Best.
Bhumip
=====================================================
I-D Action: draft-mlk-nfvrg-nfv-reliability-using-cots-00.txt
------------------------------

   - *To*: <i-d-announce at ietf.org <[email-masked]>>
   - *Subject*: I-D Action: draft-mlk-nfvrg-nfv-reliability-using-cots-00.
   txt
   - *From*: internet-drafts at ietf.org <[email-masked]>
   - *Date*: Thu, 01 Oct 2015 08:33:39 -0700
   - *Archived-at*:
<http://mailarchive.ietf.org/arch/msg/i-d-announce/nHNFdrK5mNXR4eN0nScn6BAsFCo>

   - *Auto-submitted*: auto-generated
   - *Delivered-to*: i-d-announce at ietfa.amsl.com
   <[email-masked]>
   - *List-archive*: <https://mailarchive.ietf.org/arch/browse/i-d-announce/>

   - *List-help*: <mailto:[email-masked]?subject=help
   <[email-masked]?subject=help>>
   - *List-id*: Internet Draft Announcements only <i-d-announce.ietf.org>
   - *List-post*: <mailto:[email-masked] <[email-masked]>>
   - *List-subscribe*: <https://www.ietf.org/mailman/listinfo/i-d-announce>,
   <mailto:[email-masked]?subject=subscribe
   <[email-masked]?subject=subscribe>>
   - *List-unsubscribe*: <https://www.ietf.org/mailman/options/i-d-announce>,
   <mailto:[email-masked]?subject=unsubscribe
   <[email-masked]?subject=unsubscribe>>
   - *Reply-to*: internet-drafts at ietf.org <[email-masked]>

------------------------------

A New Internet-Draft is available from the on-line Internet-Drafts directories.

        Title           : NFV Reliability using COTS Hardware
        Authors         : Li Mo
                          Bhumip Khasnabish
	Filename        : draft-mlk-nfvrg-nfv-reliability-using-cots-00.txt
	Pages           : 22
	Date            : 2015-10-01

Abstract:
   This draft discusses the results of a recent study on the feasibility
   of using Commercial Off-The-Shelf (COTS) hardware for virtualized
   network functions in telecom equipment.  In particular, it explores
   the conditions under which the COTS hardware can be used in the NFV
   (Network Function Virtualization) environment.  The concept of silent
   error probability is introduced in order to take software error or
   undetectable hardware failures into account.  The silent error
   probability is included in both the theoretical work and the
   simulation work.  It is difficult to theoretically analyze the impact
   of site maintenance and site failure events.  Therefore, simulation
   is used for evaluating the impact of these site management related
   events which constitute the undesirable feature of using COTS
   hardware in telecom environment.

The IETF datatracker status page for this draft
is:https://datatracker.ietf.org/doc/draft-mlk-nfvrg-nfv-reliability-using-cots/

There's also a htmlized version available
at:https://tools.ietf.org/html/draft-mlk-nfvrg-nfv-reliability-using-cots-00

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP
at:ftp://ftp.ietf.org/internet-drafts/

=====================================================

​

---

From: Rick Casarez <[email-masked]>
Date: Tue, 20 Oct 2015 08:00:32 -0400

Hello,

Some comments from me:

Section 1:

"Comparing the MTTR of COTS hardware and the typical telecom grade
hardware, the COTS time to repair is not a random variable and actually is
fixed. Hence the COTS MTTR is the time required to bring up a server and
ready to serve. In the traditional telecom hardware, the time to repair is
a random variable and MTTR is the mean of this random variable.Because
manual intervention is normally required in the telecom environment, the
NFV COTS MTTR is normally assumed to be less than the traditional telecom
equipment MTTR."

For your statement about COTS to be true there would need to be spare or
unused servers on site. Those servers must be available (fully functioning)
and ready to take on the role from whichever controller you are using. If
sufficient sparing is not available then the COTS MTTR becomes the same as
the telecom MTTR. Then of course if you are including all variables you
must consider whether or not Telecom equipment sparing is used. While
slower than a controller moving the function automatically to an open
server it is still considerably faster than the normal RMA process.

Another variable of course is the function being scrutinized. A fabric or
spine deployment is built to withstand multiple nodes failing no matter the
equipment used. A single ToR would have more impact due to the lack of
similar redundancy.

"While telecom equipment takes pains to minimize the impact of maintenance
on system availability, the COTS hardware normally is maintained in a
cowboy fashion (e.g. reset first and ask questions later)."

This statement is not true for a lot of companies. Depending on the role
server maintenances fall under the same maintenance times, notifications,
etc as network gear. Having said that if you have a good NFV controller
then this statement is moot. This is because the functions currently on the
COTS can be moved to a different underlying hardware. This allows work to
be done on the server needing maintenance (or troubleshooting) without
impacting anything. I know many companies prefer this method (Shooting it
in the head option) for any node not acting right rather than keeping
functions on the node and accepting degraded performance.

Section 3:

"In another words, if FIVE 9s are required for system availability, both
the server part and the network part of the availability need to be better
than FIVE 9s so their products can be more than FIVE 9s."

Having worked at ISPs I can tell you those "Five 9s" claims are gamed
pretty well. If you do not know the actual formula a company uses it will
be difficult to know if you have achieved it.

Section 6:

"While traditional telecom grade equipment putting a lot of emphasis and
engineering complexity to ensure smooth migration, smooth software upgrade,
and smooth patching procedures, the COTS hardware and its related software
are notorious in lacking such capabilities. This is the primary reason for
operators to be hesitate on utilizing COTS hardware, even though COTS
hardware in the NFV environment does having the improved MTTR as compared
to the traditional telecom hardware."

Actually, the main hesitation for using COTS for network functions is
mundane things like forwarding rates, amount of ports, etc. In fact with a
good controller, and architecture, you can lose several racks of gear at
once and your customer may notice a slight blip in their performance (If
they notice at all). On top of that "telecom gear" (I assume you mean 1RU
switches) prices have dropped considerably. So the ROI for going to COTS
for network functions like 1RU switching is hard to justify. If by "telecom
gear" you refer to carrier grade routers then a whole other multitude of
arguments against COTS raise their head.

Frankly, even telecom gear upgrades are not always smooth. I know I have
had some nice failures before. One of the reasons most companies have
procedures for upgrades, and isolate the box, is to protect against the
upgrade not going right.

-------------------
Cheers, Rick

Experiences not things.

On Wed, Oct 7, 2015 at 11:11 AM, B. Khasnabish <[email-masked]> wrote:

>    - *Archived-at*: <http://mailarchive.ietf.org/arch/msg/i-d-announce/nHNFdrK5mNXR4eN0nScn6BAsFCo>
>
>    - *List-unsubscribe*: <https://www.ietf
>    .org/mailman/options/i-d-announce>, <mailto:i-d-announce-request@ietf
>    .org?subject=unsubscribe
> The IETF datatracker status page for this draft is:https://datatracker.ietf.org/doc/draft-mlk-nfvrg-nfv-reliability-using-cots/
>
> There's also a htmlized version available at:https://tools.ietf.org/html/draft-mlk-nfvrg-nfv-reliability-using-cots-00
>
>
> Internet-Drafts are also available by anonymous FTP at:ftp://ftp.ietf.org/internet-drafts/
>
>
> _______________________________________________
> Nfvrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/nfvrg
>
>

---

From: "B. Khasnabish" <[email-masked]>
Date: Tue, 20 Oct 2015 13:00:36 -0400

​Dear Rick,

Thanks for your kind review and comments/suggestions.

We'll address these in the next version of the draft and send you the
updated draft before publishing.

Once again, many thanks in advance.

Best.

Bhumip

​

On Tue, Oct 20, 2015 at 8:00 AM, Rick Casarez <[email-masked]>
wrote:

>>    - *Subject*: I-D Action: draft-mlk-nfvrg-nfv
>>    -reliability-using-cots-00.txt
>>    - *List-subscribe*: <https://www.ietf.org/mailman/listinfo
>>    /i-d-announce>, <mailto:i-d-announce-request@ietf
>>    .org?subject=subscribe