Subject: [alto] New ALTO I-D: draft-xiang-alto-exascale-network-optimization

From: Qiao Xiang <[email-masked]>
Date: Fri, 8 Jul 2016 19:53:04 -0400

Dear All,

We have submitted a new I-D for consideration. It describes a dataset
transfer orchestrator framework for exascale science networks. It collects
network information from multiple ALTO services utilizing proposed topology
extensions and
leverages emerging SDN control capabilities to orchestrate the scheduling
of multiple large dataset transfers, leading to improved data transfer
latency and reliability as well as more efficient utilization of limited
network resources.

It can be found at
https://datatracker.ietf.org/doc/draft-xiang-alto-exascale-network-optimization/
 . and your comments are welcomed.

Thanks.

Best
Qiao
-- 
Qiao Xiang
Postdoctoral Fellow,
Department of Computer Science,
Yale University

---

From: May （Haoran） <[email-masked]>
Date: Sat, 16 Jul 2016 22:09:43 +0800 (CST)

Dear Qiao and all,

This draft is a great example to provide a scenario for using ALTO information. However, I have several questions and want to make it clear:

First of all, if I understand right, EDTO has three information sources: the DTR, the network information from ALTO, and the information of the datasets. When you mentioned that ALTO server will provide link bandwidth in line 3 in section 5.4, I’m not very clear why and how ALTO server provides this kind of information.

Besides, if EDTO will do the scheduling work, supposedly it will firstly get the information of link capacity. But in the architecture of EDTO now, I’m not quite sure how does the scheduler get the information of link capacity.

One small suggestion, the diagram in section 5.1 which shows the architecture of EDTO platform, will be more precise if it can display the direction of dataflow. For example, the data from users’ requests will flow to the DTR Collector, and the data from DTR Collector, Dataset Transfer Manager and ALTO Client will flow to DTR Scheduler, etc. Using arrows to replace the lines when there is dataflow will be more clear and understandable.

Plus, three trivial problems. First is that the page links of 5.7.3 and 6.4 doesn’t work. Second, in Section 1, Paragraph 2, line 4, ANd -> And. Third, in Page 4, first line, not sure if you want to say flow steering by saying flow sterring( a typo?)

Please correct me if I misunderstand anything. Thanks!

May 

At 2016-07-09 07:53:04, "Qiao Xiang" <[email-masked]> wrote:
Dear All,

We have submitted a new I-D for consideration. It describes a dataset transfer orchestrator framework for exascale science networks. It collects network information from multiple ALTO services utilizing proposed topology extensions and
leverages emerging SDN control capabilities to orchestrate the scheduling of multiple large dataset transfers, leading to improved data transfer latency and reliability as well as more efficient utilization of limited network resources.

It can be found at https://datatracker.ietf.org/doc/draft-xiang-alto-exascale-network-optimization/   . and your comments are welcomed.

Thanks.

Best
Qiao
--

Qiao Xiang
Postdoctoral Fellow,

---

From: Qiao Xiang <[email-masked]>
Date: Thu, 21 Jul 2016 23:45:17 -0400

Hi All,

First, thank May for helping present the draft on yesterday. I really
appreciate it!

Secondly, I believe most of May's questions raised in her email have been
addressed during our offline discussion. Essentially, network information
such as link bandwidth can be collected through cost map or ECS, but to
provide sufficient information for the scheduler to make optimal scheduling
decisions, topology extension services such as path vector, RSA, FCS, etc.,
are in great need. We plan to deploy RSA on the CMS testbed developed by
CalTech for demonstration in SuperComputing 2016. And we will update the
details and results on our progress to the working group as soon as
possible.

Thirdly, to further answer Vijay, Jan and Richard's question, other than
multi-domain, another important feature making it different from other
network applications is that the data transfer scheduling in science
networks need to consider the relation between different flows/datasets.
For instance, different datasets may have raw->processed relation, for
instance, a processed dataset A may need to wait till other raw datasets
transferred to a certain site before A can be transferred. In current
development,  we have not designed scheduling policies for supporting
general dataset transferrs relations. We plan to explore an efficient
scheduling algorithm for this scenario in the short term future.

Thanks.

Best
Qiao

On Sat, Jul 16, 2016 at 10:09 AM, May （Haoran） <[email-masked]
<javascript:_e(%7B%7D,'cvml',[email-masked]');>> wrote:

> Dear Qiao and all,
>
> This draft is a great example to provide a scenario for using ALTO
> information. However, I have several questions and want to make it clear:
>
> First of all, if I understand right, EDTO has three information sources:
> the DTR, the network information from ALTO, and the information of the
> datasets. When you mentioned that ALTO server will provide link bandwidth
> in line 3 in section 5.4, I’m not very clear why and how ALTO server
> provides this kind of information.
>
> Besides, if EDTO will do the scheduling work, supposedly it will firstly
> get the information of link capacity. But in the architecture of EDTO now,
> I’m not quite sure how does the scheduler get the information of link
> capacity.
>
> One small suggestion, the diagram in section 5.1 which shows the
> architecture of EDTO platform, will be more precise if it can display the
> direction of dataflow. For example, the data from users’ requests will flow
> to the DTR Collector, and the data from DTR Collector, Dataset Transfer
> Manager and ALTO Client will flow to DTR Scheduler, etc. Using arrows to
> replace the lines when there is dataflow will be more clear and
> understandable.
>
> Plus, three trivial problems. First is that the page links of 5.7.3 and
> 6.4 doesn’t work. Second, in Section 1, Paragraph 2, line 4, ANd -> And.
> Third, in Page 4, first line, not sure if you want to say flow steering by
> saying flow sterring( a typo?)
>
> At 2016-07-09 07:53:04, "Qiao Xiang" <[email-masked]
> <javascript:_e(%7B%7D,'cvml',[email-masked]');>> wrote:
>
> Dear All,
>
> network information from multiple ALTO services utilizing proposed
> topology extensions and