Subject: [Nfvrg] 转发: New Version Notification for draft-fu-nfvrg-hierarchical-orchestraotr-00.txt

From: Fu Qiao <[email-masked]>
Date: Mon, 19 Oct 2015 17:02:40 +0800

Hi, all. I have just uploaded a draft about the hierarchical framework of orchestrator deployment in NFV. This document mainly discusses about the deployment of Orchestrator in the Operators' network, in which multi layers of Orchestrator should be defined. This documents also describes the different responsibilities of the Orchestrators and their interfaces. The multi-layer deployment of Orchestrator is quite practical, and should be considered in the future protocol and deployments.
Your comments are more than welcome! Thank you!

-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]] 
发送时间: 2015年10月19日 16:54
收件人: Qiao Fu; Qiao Fu
主题: New Version Notification for draft-fu-nfvrg-hierarchical-orchestraotr-00.txt

A new version of I-D, draft-fu-nfvrg-hierarchical-orchestraotr-00.txt
has been successfully submitted by Qiao Fu and posted to the
IETF repository.

Name:		draft-fu-nfvrg-hierarchical-orchestraotr
Revision:	00
Title:		Framework of Hierarchical Orchestrator in NFV Deployment
Document date:	2015-10-19
Group:		Individual Submission
Pages:		7
URL:            https://www.ietf.org/internet-drafts/draft-fu-nfvrg-hierarchical-orchestraotr-00.txt
Status:         https://datatracker.ietf.org/doc/draft-fu-nfvrg-hierarchical-orchestraotr/
Htmlized:       https://tools.ietf.org/html/draft-fu-nfvrg-hierarchical-orchestraotr-00

Abstract:
   This document discusses about the framework of Orchestrator in the
   NFV deployment.  A hierarchical framework of Orchestrator is then
   proposed.  Such framework is very important for large scale NFV
   deployment in Operator networks, since multi-site of NFV deployment
   and management should be considered.  This draft will further discuss
   the responsibilities of the multi-layers of the Orchestrator and the
   interfaces between them.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

---

From: Gino Carrozzo <[email-masked]>
Date: Mon, 19 Oct 2015 11:28:19 +0200

Dear Fu

please have a look at
https://tools.ietf.org/html/draft-felix-nfvrg-recursive-orchestration-00#page-5
where a similar concept of recursiveness is described.

br
Gino

On 19 October 2015 at 11:02, Fu Qiao <[email-masked]> wrote:

> Hi, all. I have just uploaded a draft about the hierarchical framework of
> orchestrator deployment in NFV. This document mainly discusses about the
> deployment of Orchestrator in the Operators' network, in which multi layers
> of Orchestrator should be defined. This documents also describes the
> different responsibilities of the Orchestrators and their interfaces. The
> multi-layer deployment of Orchestrator is quite practical, and should be
> considered in the future protocol and deployments.
> 主题: New Version Notification for
> draft-fu-nfvrg-hierarchical-orchestraotr-00.txt
>
>
> URL:
> https://www.ietf.org/internet-drafts/draft-fu-nfvrg-hierarchical-orchestraotr-00.txt
> Status:
> https://datatracker.ietf.org/doc/draft-fu-nfvrg-hierarchical-orchestraotr/
> Htmlized:
> https://tools.ietf.org/html/draft-fu-nfvrg-hierarchical-orchestraotr-00
>
>
> Abstract:
> _______________________________________________
> Nfvrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/nfvrg
>

-- 
==============================================================
  Dr. Gino Carrozzo
  Deputy Head of R&D

  mobile    : +39 348 7610 081
  landline  : +39 050 3871 679
  fax         : +39 050 3871 601
  e-mail    : [email-masked]
  web       : www.nextworks.it
  address   : Nextworks s.r.l.
                  via Livornese, 1027
                  56122 S.Piero a Grado, Pisa, ITALY
==============================================================

---

From: PEDRO ANDRES ARANDA GUTIERREZ <[email-masked]>
Date: Mon, 19 Oct 2015 10:27:45 +0000

+1
De: Nfvrg <[email-masked]<mailto:[email-masked]>> en nombre de Gino Carrozzo <[email-masked]<mailto:[email-masked]>>
Fecha: lunes, 19 de octubre de 2015, 11:28
Para: Fu Qiao <[email-masked]<mailto:[email-masked]>>
CC: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Asunto: Re: [Nfvrg] 转发: New Version Notification for draft-fu-nfvrg-hierarchical-orchestraotr-00.txt

Dear Fu

please have a look at https://tools.ietf.org/html/draft-felix-nfvrg-recursive-orchestration-00#page-5 where a similar concept of recursiveness is described.

br
Gino

On 19 October 2015 at 11:02, Fu Qiao <[email-masked]<mailto:[email-masked]>> wrote:
Hi, all. I have just uploaded a draft about the hierarchical framework of orchestrator deployment in NFV. This document mainly discusses about the deployment of Orchestrator in the Operators' network, in which multi layers of Orchestrator should be defined. This documents also describes the different responsibilities of the Orchestrators and their interfaces. The multi-layer deployment of Orchestrator is quite practical, and should be considered in the future protocol and deployments.
Your comments are more than welcome! Thank you!

-----邮件原件-----
发件人: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>]
发送时间: 2015年10月19日 16:54
收件人: Qiao Fu; Qiao Fu
主题: New Version Notification for draft-fu-nfvrg-hierarchical-orchestraotr-00.txt

Abstract:

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org>.

The IETF Secretariat

_______________________________________________
Nfvrg mailing list
[email-masked]<mailto:[email-masked]>
https://www.irtf.org/mailman/listinfo/nfvrg

  mobile    : +39 348 7610 081
  landline  : +39 050 3871 679
  fax         : +39 050 3871 601
  e-mail    : [email-masked]<mailto:[email-masked]>
  web       : www.nextworks.it<http://www.nextworks.it>
  address   : Nextworks s.r.l.
                  via Livornese, 1027
                  56122 S.Piero a Grado, Pisa, ITALY
==============================================================

---
Dr. Pedro A. Aranda Gutiérrez

Technology Exploration -
Network Innovation & Virtualisation
email: pedroa d0t aranda At telefonica d0t com
Telefónica, Investigación y Desarrollo
C/ Zurbarán,12
28010 Madrid, Spain

Fragen sind nicht da, um beantwortet zu werden.
Fragen sind da, um gestellt zu werden.
Georg Kreisler

________________________________

Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.

The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.

Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição

---

From: <[email-masked]>
Date: Tue, 20 Oct 2015 12:23:35 +0000

A couple of comments/questions

Section 3

- 1st paragraph: I don't think the concept of access vs. core network still makes sense with NFV. The criteria for deploying a VNF in a local PoP or a centralized DC are typically related to latency requirements, e2e reliability, organizational matters, etc...). A VNF that corresponds to a NF (or part of) that used to be deployed in access networks or even customer premises may well be deployed in a centralized DC (e.g. vCPE's DHCP servers) and vice-versa. Different instances of the same VNF may even be deployed in different locations...

- 2nd paragraph: Saying that multisite distribution of VNFs REQUIRES a hierarchical framework of orchestrator is a very strong statement and would thus deserve further explanations. What prevents supporting multi-side distribution of VNFs with an NFVO connected to multiple VIMs/WIMs? If some form of hierarchy is required, what are the benefits of an NFVO hierarchy vs. a VIM hierarchy?

Section 4:

- It is unclear how are handled Network Services involving VNFs to be deployed in different DCs. It looks like the S-Orchestrator will have to break the NS into smaller NSs, each containing VNFs to be deployed in a single DC. If yes, is not the S-Orchestrator simply an end-to-end orchestrator in the OSS, sitting above the NFVO?

- How do you handle the case where two instances of the same VNF have to be deployed in different DCs (e.g., for reliability purposes)?

Bruno

-----Message d'origine-----
De : Nfvrg [mailto:[email-masked]] De la part de Fu Qiao
Envoyé : lundi 19 octobre 2015 11:03
À : [email-masked]
Objet : [Nfvrg] 转发: New Version Notification for draft-fu-nfvrg-hierarchical-orchestraotr-00.txt

-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]] 
发送时间: 2015年10月19日 16:54
收件人: Qiao Fu; Qiao Fu
主题: New Version Notification for draft-fu-nfvrg-hierarchical-orchestraotr-00.txt

Abstract:

The IETF Secretariat

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Fu Qiao <[email-masked]>
Date: Wed, 21 Oct 2015 17:55:02 +0800

Hi, Thank you for Gino and Pedroa for notifying this. And I am quite agree the hierarchical RO mentioned in the draft below is quite similar with what I want to explain in this draft. 

If I understand correctly, the RO draft basically focuses on resource orchestrator, and discusses about how we could orchestrate and manage the different resource pool with the parent and child RO. As in this draft, I want to focus not only on resource orchestrator, but also including service orchestrator. As you may have noticed, I discuss a lot about NS deployment, topology monitoring, fault management and etc.. I consider the two draft as describing from two aspects, one from the DC resource perspective, and one from the telecom operation perspective. And specific requirements are proposed then from the two perspectives. However, I think in the real deployment, these two things might be co-located, or be two modules in one components.

I do feel quite exited that there are other drafts that also agree with such hierarchical framework of orchestration of NFV. I think future discussions can be proposed so that we can work out a general framework and orchestration system for NFV.

发件人: Nfvrg [mailto:[email-masked]] 代表 PEDRO ANDRES ARANDA GUTIERREZ
发送时间: 2015年10月19日 18:28
收件人: Gino Carrozzo; Fu Qiao
抄送: [email-masked]
主题: Re: [Nfvrg] 转发: New Version Notification for draft-fu-nfvrg-hierarchical-orchestraotr-00.txt

+1

De: Nfvrg <[email-masked]> en nombre de Gino Carrozzo <[email-masked]>
Fecha: lunes, 19 de octubre de 2015, 11:28
Para: Fu Qiao <[email-masked]>
CC: "[email-masked]" <[email-masked]>
Asunto: Re: [Nfvrg] 转发: New Version Notification for draft-fu-nfvrg-hierarchical-orchestraotr-00.txt

Dear Fu 

br
Gino

-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]]
发送时间: 2015年10月19日 16:54
收件人: Qiao Fu; Qiao Fu
主题: New Version Notification for draft-fu-nfvrg-hierarchical-orchestraotr-00.txt

Abstract:

The IETF Secretariat

 

-- 

  Dr. Gino Carrozzo

  Deputy Head of R&D

  fax         : +39 050 3871 601                  

  web       : www.nextworks.it               

                  via Livornese, 1027                        

 

---

Technology Exploration -

C/ Zurbarán,12

28010 Madrid, Spain

Georg Kreisler

  _____

---

From: Fu Qiao <[email-masked]>
Date: Wed, 21 Oct 2015 17:55:17 +0800

Hi, Bruno. Thank you for the comments. I put some of my understanding as follows. Please see if they make sense to you.

-----邮件原件-----
发件人: Nfvrg [mailto:[email-masked]] 代表 [email-masked]
发送时间: 2015年10月20日 20:24
收件人: Fu Qiao; [email-masked]
主题: Re: [Nfvrg] 转发: New Version Notification for draft-fu-nfvrg-hierarchical-orchestraotr-00.txt

Section 3

[fq] Quite agree with this actually. I mentioned about the core and access network just to make it easier for the readers to understand why we should have different regional DC. I am totally agree that the criteria for deploying a VNF in a local PoP or a centralized DC are typically related to latency requirements, e2e reliability, organizational matters, etc.... But one thing we should consider is, the operators can not just build a completely new NFV network with all the VNFs deployed based on the metrics that you mentioned above. The transformation from the legacy network to the NFV network is very complicated and important to the operators. Therefore, in this sense the concept for core and access network may still make sense for a while, since the Operators may consider deploying a small scale NFV based on these legacy access/core network DC. Start with these, the Operators will gradually transform such legacy DC into the NFV DC which may include VNFs that are not originally included in the DC. Therefore, I think the concept of access/core network can still exist for some time for NFV.

- 2nd paragraph: Saying that multisite distribution of VNFs REQUIRES a hierarchical framework of orchestrator is a very strong statement and would thus deserve further explanations. What prevents supporting multi-side distribution of VNFs with an NFVO connected to multiple VIMs/WIMs?

[fq] what you mentioned could be one of the solutions. I am not saying that there must always be one orchestrator for each VIM and we will not accept one NFVO for multiple VIM. But we should also consider the distance between the centrally deployed NFVO with the multiple VIM and the reliability of the link between them. These are all practical issues. And also management system should also be considered. As far as I know, the operator may have its local team at each DC. An orchestrator will always need for them to monitor its own VNFs in the DC. If this is the case, why don't we utilize these Orchestrator and have them as the R-orchestrator in our system.
I do agree what you mentioned can also be one of the deployment. For example, in one province, we can only have one orchestrator for all the DCs in this province. However we will still need a S-Orchestrator for the national wide orchestration. This hierarchical framework can be used in this level. 

 If some form of hierarchy is required, what are the benefits of an NFVO hierarchy vs. a VIM hierarchy?
[fq] Quite a good point. And as I said above, can be a solution in some scenarios as well, even though currently both open source openstack and commercial ones do not well support such hierarchical feature. But as I mentioned above, we may still need a hierarchical orchestrator framework if we consider large scale deployment. As you may have already captured in the above words, what I put here is not a mandatory structure for NFVO. For NFV deployment, we can have multiple solutions, including what you suggested above. But hierarchical Orchestrator can be and should be one option for the NFV deployment. This is something we should consider, I think.

Section 4:

[fq] There could be one of the case. The S-O can be like a plugin for the OSS, to do the translation of NS, monitor the e2e service topology, fault management and etc.. And download to the R-O to do the actual deployment. Another deployment could be, the S-O is also the R-O for one of the DC, we can call it as a central DC. 

- How do you handle the case where two instances of the same VNF have to be deployed in different DCs (e.g., for reliability purposes)?
[fq] for reliability purpose, such deployment is possible. In such case, we should consider whether the VNF is stateful or stateless. For stateful VNF, the state should be stored in a shared storage, or the storage should be copied in a timely manner between the active and the standby VNF. For the stateless VNF, such thing do not need to be considered. Another issue is how to do the failover between these two VNFs on different site. Currently most of the active/standby or active/active VNFs have an HA manager to do the failover. In the multi site case, we will have the HA manager to control two VMs on two different sites. As far as I know, I haven't seen any POC about this now. So I don't know if technically, openstack support such scenario. But there certainly are discussions about multisites in openstack, and reliability of VNF on different site and GR is considered as one of the usecases.

Bruno

-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]] 
发送时间: 2015年10月19日 16:54
收件人: Qiao Fu; Qiao Fu
主题: New Version Notification for draft-fu-nfvrg-hierarchical-orchestraotr-00.txt

Abstract:

The IETF Secretariat

---

From: <[email-masked]>
Date: Sun, 25 Oct 2015 21:39:15 +0000

Hi Qiao,

About my comments on the 2nd paragraph of section 3, I would suggest replacing "requires" with "may require" or "could require" or "can benefit from". 

About my last point on Section 4 regarding two instances of the same VNF being deployed in different DCs, I think it deserves further explanations in the I-D on how this is taken into account when splitting an NS into smaller NSs to be distributed to the relevant regional Orchestrators. 

Best Regards,
Bruno

-----Message d'origine-----
De : Fu Qiao [mailto:[email-masked]] 
Envoyé : mercredi 21 octobre 2015 11:55
À : CHATRAS Bruno IMT/OLN; [email-masked]
Objet : 答复: [Nfvrg] 转发: New Version Notification for draft-fu-nfvrg-hierarchical-orchestraotr-00.txt

-----邮件原件-----
发件人: Nfvrg [mailto:[email-masked]] 代表 [email-masked]
发送时间: 2015年10月20日 20:24
收件人: Fu Qiao; [email-masked]
主题: Re: [Nfvrg] 转发: New Version Notification for draft-fu-nfvrg-hierarchical-orchestraotr-00.txt

Section 3

Section 4:

Bruno

-----Message d'origine-----
De : Nfvrg [mailto:[email-masked]] De la part de Fu Qiao Envoyé : lundi 19 octobre 2015 11:03 À : [email-masked] Objet : [Nfvrg] 转发: New Version Notification for draft-fu-nfvrg-hierarchical-orchestraotr-00.txt

-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]]
发送时间: 2015年10月19日 16:54
收件人: Qiao Fu; Qiao Fu
主题: New Version Notification for draft-fu-nfvrg-hierarchical-orchestraotr-00.txt

Abstract:

The IETF Secretariat

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorization.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange shall not be liable if this message was modified, changed or falsified.
Thank you.

---

From: Fu Qiao <[email-masked]>
Date: Mon, 26 Oct 2015 10:12:25 +0800

Hi, Bruno. Thank you for the comments.
I will update the draft to include the comments you give. 

-----邮件原件-----
发件人: Nfvrg [mailto:[email-masked]] 代表 [email-masked]
发送时间: 2015年10月26日 5:39
收件人: Fu Qiao; [email-masked]
主题: Re: [Nfvrg] 转发: New Version Notification for draft-fu-nfvrg-hierarchical-orchestraotr-00.txt

Hi Qiao,

Best Regards,
Bruno

-----邮件原件-----
发件人: Nfvrg [mailto:[email-masked]] 代表 [email-masked]
发送时间: 2015年10月20日 20:24
收件人: Fu Qiao; [email-masked]
主题: Re: [Nfvrg] 转发: New Version Notification for draft-fu-nfvrg-hierarchical-orchestraotr-00.txt

Section 3

Section 4:

Bruno

-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]]
发送时间: 2015年10月19日 16:54
收件人: Qiao Fu; Qiao Fu
主题: New Version Notification for draft-fu-nfvrg-hierarchical-orchestraotr-00.txt

Abstract:

The IETF Secretariat