Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

From: "George, Wes" <[email-masked]>
Date: Thu, 3 Oct 2013 15:31:15 -0400

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Linda Dunbar
>
> There are a lot of issues with L3 NAT too, but L3 NAT is one of the key
> components that make today's internet scale.
>
[WEG] One might also argue that MS Outlook is a key component in many corporate environments, but it's false equivalence to then assume that this is because it's the best solution, rather than the truth, which is that it's the "least bad" alternative. At least in NAT's case, we have a better solution (IPv6). ;-)

Wes George

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

---

From: Linda Dunbar <[email-masked]>
Date: Thu, 3 Oct 2013 21:21:41 +0000

Wes, 

> -----Original Message-----
> >
> [WEG] One might also argue that MS Outlook is a key component in many
> corporate environments, but it's false equivalence to then assume that
> this is because it's the best solution, rather than the truth, which is
> that it's the "least bad" alternative. At least in NAT's case, we have
> a better solution (IPv6). ;-)

[Linda] Can you show how IPv6 can solve the problem? The problem is the amount of addresses that DC Gateway has to handle when hosts in one subnet being spread across different Data Centers. We would like the all hosts in remote DCs to be represented by their corresponding Gateway nodes. 

If you have better suggestions, we'd love to hear from you.

Linda

---

From: Ted Lemon <[email-masked]>
Date: Thu, 3 Oct 2013 22:00:22 +0000

> If you have better suggestions, we'd love to hear from you.

This is actually not the right question to ask.   Suppose IPv6 doesn't solve the problem.   Does that mean that the solution you are proposing is now the right solution?   No, it's unrelated.

What matters is that an observation has been made that a L2 VPN could have severe negative behavior characteristics in plausible circumstances.  This is a really strong reason not to use an L2 VPN.   It doesn't matter whether or not there is a better solution; what matters is that this solution is not (it has been asserted) a good solution.

So your response should be to explain why the problem that has been described can't happen.   If you can't do that, then I think it's a fairly compelling reason not to pursue this solution.

---

From: Linda Dunbar <[email-masked]>
Date: Thu, 3 Oct 2013 22:42:04 +0000

Ted, 

> What matters is that an observation has been made that a L2 VPN could
> have severe negative behavior characteristics in plausible
> circumstances.  This is a really strong reason not to use an L2 VPN.

[Linda] I understand that many people are still not convinced of the need for hosts within one subnet to be placed in different server racks.

For a typical Data Center, there may be many instances of WebServer in one subnet (typical one VLAN), many instances of AppServer in another subnet, and many instances of DBServer in yet another subnet.

Being in one subnet doesn't mean that those hosts (or applications) have higher traffic among themselves. One of the main reasons for partitioning hosts to multiple Subnets is for policy enforcement and load balance purposes. The amount of traffic between "WebServer 
Instances" <-> "AppServer instances" <-> "DB server instances" is magnitude higher than the heartbeat traffic among all instances of one function.

Placing hosts based on their Subnets actually creates more across rack traffic  than placing instances of different functions (or subnets) based on their traffic patterns. For some DC operators who want to optimize their traffic across racks, placing instances from different subnets in one rack can eliminate a lot of traffic traversing among racks.

Linda

---

From: David Allan I <[email-masked]>
Date: Thu, 3 Oct 2013 23:05:48 +0000

HI Linda:

I have no problem with the concept that a subnet may span multiple server racks, I expect that to be a common deployment scenario. 

In fact the more fragmented the space is the more likely that the cardinality of VM to ARP proxy will approach 1:1 (in which case SARP will offer no state reduction) and the more likely it is that you will need a lot more than 4K VLANs in which case a flat S-tagged bridging domain will not cut it and you would need PBB or some similar technology that enlarged the VLAN space, and while it was at it, summarized C-MACs, which largely solves the state inflation problem SARP purports to solve. 

So if you fragment to the level you suggest such that SARP supposedly is needed, SARP offers little benefit, and if you do not, you would not need it given current technology as not EVERY MAC in the system would appear in EVERY switch. So I do not see the merit in something that takes away existing L2 capability, and offers a questionable scaling benefit.

That's the view from here
Dave

-----Original Message-----
From: Linda Dunbar [mailto:[email-masked]] 
Sent: Thursday, October 03, 2013 3:42 PM
To: Ted Lemon
Cc: George, Wes; David Allan I; [email-masked]
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Ted, 

Being in one subnet doesn't mean that those hosts (or applications) have higher traffic among themselves. One of the main reasons for partitioning hosts to multiple Subnets is for policy enforcement and load balance purposes. The amount of traffic between "WebServer Instances" <-> "AppServer instances" <-> "DB server instances" is magnitude higher than the heartbeat traffic among all instances of one function.

Linda

---

From: Ted Lemon <[email-masked]>
Date: Fri, 4 Oct 2013 01:20:21 +0000

> [Linda] I understand that many people are still not convinced of the need for hosts within one subnet to be placed in different server racks.

Again, Linda, I'm really sorry to keep harping on this point, but the problem is not that people do not believe that there is a need for this functionality.   The problem is that they have identified serious problems with the solution you are proposing.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 4 Oct 2013 07:30:30 +0200 (CEST)

On Thu, 3 Oct 2013, Linda Dunbar wrote:

> [Linda] I understand that many people are still not convinced of the 
> need for hosts within one subnet to be placed in different server racks.

"need" here is relative. With current popular ways of doing things, there 
is "need". If the architecture changed, the need would go away.

For instance, iSCSI can be routed. AoE (ATA over Ethernet) cannot. One 
architecture means you have to create big L2 domains, the other one 
doesn't. iSCSI spec is 500 pages, AoE is 30 pages. I can see why people 
might choose AoE for simplicity, but scaling AoE in current incarnation 
means more complexity on the network side instead. Btw, the proposed 
solution (SARP) wouldn't work with AoE anyhow, because it requires the 
payload to be IP.

Datacenter architects should look more for L3 solutions since these scale 
better. There is a reason the Internet today isn't a huge L2 domain, and 
scaling the L2 domain by means of requiring IP payload but not routing the 
IP packets just doesn't make much sense to me.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "George, Wes" <[email-masked]>
Date: Fri, 4 Oct 2013 10:00:06 -0400

> [Linda] Can you show how IPv6 can solve the problem?
[WEG] Not necessarily, but I do know that IP can solve the problem.

Partially I was being a bit puckish by pointing out that using L3 NAT's prevalence for justification of... well... anything is probably not the right approach to garner support, especially in this crowd, and my reference to a better solution was noting that IPv6 is a better solution than L3 NAT to the problems L3 NAT is typically trying to solve, not specifically your problem statement. ;-)

With regards to your problem, I'm of the mind that the solution to switches with MAC tables that are too large is to reduce the size of the L2 MAC domain (bring layer 3 further down), and thus I'm skeptical of any solution that involves replicating the same subnets in multiple places (especially physically different locations). Solving an L2 scaling problem this way is roughly equivalent to saying "I have 10 tons of gravel to move, and I can only make one trip, but the trunk of my car only holds 1/100th of that, so I need to be able to drive 100 cars simultaneously" instead of simply renting a big truck. Yes, it's a real problem, but it's one that you created by trying to scale beyond reasonable design limitations.

You're right that affinity should be taken into account when locating applications so that you reduce the area across which traffic between them must be carried. However, if that conflicts with one's choice of subnet hierarchy (e.g. by application), such that the subnet must be replicated in multiple locations (or worse, you're dealing with an application that assumes L2 connectivity and isn't smart enough to deal with different subnets), then one probably should rethink their choice of subnet allocation.
This problem is only going to get worse with IPv6, because the subnets have so many addresses that we don't even get the benefit of IPv4's bit-induced limit on number of hosts per subnet to make things sane when it comes to the size of an L2 domain. With the right L2VPN design, I could number every application of a certain type across a worldwide set of datacenters out of a single /64 and still have tens of thousands of addresses to spare. That doesn't mean that I should.

Wes George

---

From: Zhenghewen <[email-masked]>
Date: Tue, 8 Oct 2013 20:08:18 +0800

Hi Thomas,

> 1) As DaveA points out, FDB tables are large on chips these days.

[Hewen]In real world,  FDB tables seem small, especially for large-size data centre in cloud era. 
	Engineers at least two choices to implement FDB table, TCAM or DRAM, but TCAM is very expensive and costs more power (thus high temperature), those limit feasible TCAM capacity on a board, then TCAM is not a good choice for very large table. DRAM is more popular and cheap, but DRAM chip has the limitation of capacity and speed. Usually, the single DRAM chip in the market has the maximum capacity of 4Gb (for most available DRAM chip in the market).
	Now, let's assume the packet lookup chipset would use 128 bits search/hash item, one MAC lookup means at least 3 items, then a 4Gb DRAM chipset can support up to 10M MAC entries.
                In real world, some operator has 80 data centres in the scope with the diameter of 1000KM, each data centre has average 1,500 racks, each rack has average 20 servers, each servers has 20 VMs. That means that there are up to 48millions MAC entries. Even only 20% VMs generate traffic outside from local data centres, there are about 10M MAC entries.
	It seems that single DRAM chip is enough, but there are some other obvious limitations:
	First, is 20% enough? Are all capacity of all DRAM chipsets used for MAC entries (nothing for L3 FIB, nothing for ACL)?
	Second, a DRAM at least means a [email-masked] link with packet lookup chipset, now if we want 125Gbps line-rate, we must place 10 copies of DRAM chipsets for 10M MAC entries; if we want 500Gbps line-rate, it means 40 DRAM chipsets for 10M MAC entries. It would be not feasible for engineers, especially considering the available board space, power consumption and SerDes links number.
	Now, as we see, current FDB tables seem small, especially for cloud data centre. Usually high-speed switch or router only has limited table size, for example 128K or 512K MAC entries.

	Maybe DRAM chipset capacity or speed's grow will solve this issue, but according to RFC4984, "Historically, DRAM capacity grows at about 4x every 3.3 years.  This translates to 2.4x every 2 years, so DRAM capacity actually grows faster than Moore's law would suggest.  DRAM speed, however, only grows about 10% per year, or 1.2x every 2 years ". It seems that we need some years to support 50millions MAC entries with single high-speed DRAM chipset.
	Some solutions to solve the issue will be valuable before the day with high-speed & high-capacity DRAM chip come.

Best regards,
Hewen

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Saturday, September 28, 2013 5:28 AM
To: Suresh Krishnan
Cc: Internet Area
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Given the flurry of mail, I went back and reviewed my notes on this
document.

I believe I understand the problem (reduce size of FDB), but am
skeptical that solving it via the approach in this document is
worthwhile.

1) As DaveA points out, FDB tables are large on chips these
days. However, at least initially, SARP would be implemented in the
service processor (consuming CPU cycles). It would be years (if ever)
before silicon would implement this. But in parallel, silicon will
just get better and have bigger FDB tables... Moreover, the SARP
motivation section specifically cites a shortage of processor cycles
as a problem ... but doing SARP in software will increase the demands
on the CPU...  its unclear to me that the increased CPU burdens SARP
implies would be offset by the reduced amount of ARP processing that
the solution is hoping will result... I.e., at least in the short
term, it's unclear that SARP would actually help in practice.

2) Doing L2 NAT at line rates (an absolute requirement for edge
devices) will only happen if this is done in silicon. I don't see that
happening unless there is strong support from vendors/operators/chip
makers... Software based solutions simply will not have sufficient
performance. I think the IEEE would be a better place to have the
discussion about what L2 chip makers are willing to implement...

3) Only works for IP. Doesn't work for non-IP L2. Doesn't that only
solve part of the problem?

4) High availability is complex (but presumably also necessary). It
smells a lot like multi-chassis LAG (with the need to synchronize
state between tightly-coupled peers). Although IEEE is working in this
general area now, there are tons of vendor-specific solutions for
doing this at L2. Do we really want to tackle standardizing this in
the IETF?  Isn't the relevant expertise for this general area in IEEE?

5) This solution touchs on both L2 (NATing L2 addresses) and L3
(ARPing for IP addresses). We absolutely would need coordinate with
IEEE before deciding to take on this work.

6) ARP caching will be a tradeoff. You want to cache responses for
better performance, but long cache timeouts will result in black holes
after VMs move. There is no great answer here. I expect long timeouts
to be unacceptable operationally, which means that the benefits of
caching will be limited (and these benefits are the key value
proposition of this approach). It is really hard to estimate whether
the benefits will be sufficient in practice. Gratuitous ARPs can help,
but they are not reliable, so they will help, but have limitations...

7) I haven't fully worked this out, but I wonder if loops can form
between proxies. There is the notion that when a VM moves, proxies
will need to update their tables. But careful analysis will be needed
to be sure that one can't ever have loops where proxies end up
pointing to each other. And since the packets are L2 (with no TTL),
such loops would be disasterous. (Note point 6: we are using
heuristics (like gratuitous ARP) to get tables to converge after a
change. Heuristics tend to have transient inconsistencies.. i.e.,
possibly leading to loops.)

Again, overall, I understand the generic problem and that it would be
nice to have a solution. However, I don't see a simple solution
here. I see a fair amount of complexity, and I'm skeptical that it's
worth it (e.g., when the next gen of silicon will just have a larger
FDB).

What I'd really like to see (before having the IETF commit to this
work) is:

1) operators who are feeling the pain described in the document
stepping forward and saying they think the solution being proposed is
something they would be willing to deploy and is better than other
approaches they have in their toolkit.

2) Vendors (including silicon) saying they see the need for this and
think the would implement it.

Thomas

_______________________________________________
Int-area mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/int-area

---

From: David Allan I <[email-masked]>
Date: Tue, 8 Oct 2013 14:11:36 +0000

I'm confused here, is the problem you are trying to solve one big subnet of 10M MACs? Your message would seem to suggest this.

IMO the problem is a large number of VLANs with a much much smaller number of MACs per VLAN, which makes the problem divisible and tractable with current technology. I consider that a solved problem.

Dave

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Zhenghewen
Sent: Tuesday, October 08, 2013 5:08 AM
To: 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Hi Thomas,

Best regards,
Hewen

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Saturday, September 28, 2013 5:28 AM
To: Suresh Krishnan
Cc: Internet Area
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Given the flurry of mail, I went back and reviewed my notes on this document.

I believe I understand the problem (reduce size of FDB), but am skeptical that solving it via the approach in this document is worthwhile.

1) As DaveA points out, FDB tables are large on chips these days. However, at least initially, SARP would be implemented in the service processor (consuming CPU cycles). It would be years (if ever) before silicon would implement this. But in parallel, silicon will just get better and have bigger FDB tables... Moreover, the SARP motivation section specifically cites a shortage of processor cycles as a problem ... but doing SARP in software will increase the demands on the CPU...  its unclear to me that the increased CPU burdens SARP implies would be offset by the reduced amount of ARP processing that the solution is hoping will result... I.e., at least in the short term, it's unclear that SARP would actually help in practice.

2) Doing L2 NAT at line rates (an absolute requirement for edge
devices) will only happen if this is done in silicon. I don't see that happening unless there is strong support from vendors/operators/chip makers... Software based solutions simply will not have sufficient performance. I think the IEEE would be a better place to have the discussion about what L2 chip makers are willing to implement...

3) Only works for IP. Doesn't work for non-IP L2. Doesn't that only solve part of the problem?

4) High availability is complex (but presumably also necessary). It smells a lot like multi-chassis LAG (with the need to synchronize state between tightly-coupled peers). Although IEEE is working in this general area now, there are tons of vendor-specific solutions for doing this at L2. Do we really want to tackle standardizing this in the IETF?  Isn't the relevant expertise for this general area in IEEE?

5) This solution touchs on both L2 (NATing L2 addresses) and L3 (ARPing for IP addresses). We absolutely would need coordinate with IEEE before deciding to take on this work.

6) ARP caching will be a tradeoff. You want to cache responses for better performance, but long cache timeouts will result in black holes after VMs move. There is no great answer here. I expect long timeouts to be unacceptable operationally, which means that the benefits of caching will be limited (and these benefits are the key value proposition of this approach). It is really hard to estimate whether the benefits will be sufficient in practice. Gratuitous ARPs can help, but they are not reliable, so they will help, but have limitations...

7) I haven't fully worked this out, but I wonder if loops can form between proxies. There is the notion that when a VM moves, proxies will need to update their tables. But careful analysis will be needed to be sure that one can't ever have loops where proxies end up pointing to each other. And since the packets are L2 (with no TTL), such loops would be disasterous. (Note point 6: we are using heuristics (like gratuitous ARP) to get tables to converge after a change. Heuristics tend to have transient inconsistencies.. i.e., possibly leading to loops.)

Again, overall, I understand the generic problem and that it would be nice to have a solution. However, I don't see a simple solution here. I see a fair amount of complexity, and I'm skeptical that it's worth it (e.g., when the next gen of silicon will just have a larger FDB).

1) operators who are feeling the pain described in the document stepping forward and saying they think the solution being proposed is something they would be willing to deploy and is better than other approaches they have in their toolkit.

2) Vendors (including silicon) saying they see the need for this and think the would implement it.

Thomas

---

From: "George, Wes" <[email-masked]>
Date: Tue, 8 Oct 2013 10:20:01 -0400

> Behalf Of Zhenghewen
>                 In real world, some operator has 80 data centres in the
> scope with the diameter of 1000KM, each data centre has average 1,500
> racks, each rack has average 20 servers, each servers has 20 VMs. That
> means that there are up to 48millions MAC entries. Even only 20% VMs
> generate traffic outside from local data centres, there are about 10M
> MAC entries.

[snip]
>       Now, as we see, current FDB tables seem small, especially for cloud
> data centre. Usually high-speed switch or router only has limited table
> size, for example 128K or 512K MAC entries.
>
[snip]
>       Some solutions to solve the issue will be valuable

[WEG] This is all very interesting theoretical math, but all it says to me is what we've known for years - L2 MAC domains don't scale infinitely - whatever the hardware capabilities, someone will come up with a theoretical case that exceeds it by many orders of magnitude. Several of us have said that the already extant solution is to shrink the L2 domain so that the number of MACs necessary to track is much smaller. Yes, RFC 6820 discusses some problems with that when it comes to VM mobility, but I'd argue that there are other solutions to those problems that start looking increasingly attractive at this sort of scale, including heavier use of DNS that is updated when a VM IP changes, etc. You need to explain why your proposed solution is less complicated than simply designing your VM infrastructure to be flexible enough that subnet changes don't break things.

Wes George

---

From: Zhenghewen <[email-masked]>
Date: Wed, 9 Oct 2013 09:10:37 +0800

Hi, Dave,
	
	I do not catch your meanings, and I think that maybe you do not understand my meanings.
	My description shows the fact that current FDB tables seem small, especially for cloud data centre, it has nothing with VLAN. It is not one big subnet ("subnet" in my understanding means IP subnet) of 10M MACs, it is only about larger layer2 domain, for example, one operator provide layer2 service across data centres for tenants and those tenants can own separate and overlapped IP address space.
	I do not see any other solution for this issue, would you provide some info? thanks.

Hewen

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of David Allan I
Sent: Tuesday, October 08, 2013 10:12 PM
To: Zhenghewen; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Dave

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Zhenghewen
Sent: Tuesday, October 08, 2013 5:08 AM
To: 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Hi Thomas,

Best regards,
Hewen

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Saturday, September 28, 2013 5:28 AM
To: Suresh Krishnan
Cc: Internet Area
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Thomas

---

From: Zhenghewen <[email-masked]>
Date: Wed, 9 Oct 2013 09:43:18 +0800

Hi,
	I just want to discuss a solution to solve MAC address scalability issues in cloud data centres.
	
	Just as I discussed with Mikael Abrahamsson, MAC address scalability issues is not serious in internet data centres (IDC), but it is serious in cloud data centres who providing (virtual) private data centres (vPDC) for different tenants.
	Due to limitations such as legal regulation, cultural barriers or skilled experts distribution, one company (i.e. a tenant of vPDC service provisioned by cloud service provider) have to distribute its staffs into many geographically dispersed sites, those staffs get services from a data centre near their office site. The cloud service provider provide vPDC service to tenant, each tenant has its separate private IP network across sites(data centres) and the IP network may overlap with another tenant's IP network, the cloud service provider isolate those tenants through underlay, layer 2 technologies can be used as underlay, Layer3 NAT is another feasible but costly and more complicated option.
	It is very skilled and complicated for a tenant to design IP subnet per vPDC site. The underlay will have to touch millions host addresses (IP addresses or MAC addresses), but current and mid-term future hardware cannot support that.

	We need focus the issue, I support to adopt SARP as WG draft, at least SARP is one try to solve that issue.

Best regards,
Hewen

-----Original Message-----
From: George, Wes [mailto:[email-masked]] 
Sent: Tuesday, October 08, 2013 10:20 PM
To: Zhenghewen; 'Thomas Narten'; 'Suresh Krishnan'
Cc: 'Internet Area'
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

[snip]
[snip]

Wes George

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 9 Oct 2013 06:40:55 +0200 (CEST)

On Wed, 9 Oct 2013, Zhenghewen wrote:

> 	Just as I discussed with Mikael Abrahamsson, MAC address 
> scalability issues is not serious in internet data centres (IDC), but it 
> is serious in cloud data centres who providing (virtual) private data 
> centres (vPDC) for different tenants.

I am still not convinced.

I think the better engineering solution is to encapsulate in additional 
header (provider bridging, MPLS or similar) to solve the scalability 
problem you're trying to solve.

The fact that broadcast, multicast and non-IP isn't addressed either (and 
I don't see a good way forward that they would be), is a showstopper to 
me.

I do not support adoption of this document in current state.

---

From: Melinda Shore <[email-masked]>
Date: Tue, 08 Oct 2013 20:53:39 -0800

On 10/8/13 8:40 PM, Mikael Abrahamsson wrote:
>> scalability issues is not serious in internet data centres (IDC), but
>> it is serious in cloud data centres who providing (virtual) private
>> data centres (vPDC) for different tenants.

> The fact that broadcast, multicast and non-IP isn't addressed either
> (and I don't see a good way forward that they would be), is a
> showstopper to me.
> 

I've been finding this a really interesting discussion and some
of the arguments against adoption have been compelling.  This one
surprised me a bit - it reads as if you're skeptical about using
this mechanism to address scaling issues, but it's the lack of
text addressing broadcast/multicast/layer 2 that you find the
larger problem.  So I have to ask: if they provided text addressing
broadcast/multicast/layer 2 would that address your objection, or
is it actually to the basic proposal?

Thanks,

Melinda

---

From: Zhenghewen <[email-masked]>
Date: Wed, 9 Oct 2013 14:07:38 +0800

Hi,

	A new header into current protocol is a option, but I think that the draft suggests a more general and feasible solution than defining new encapsulation into many protocols (e.g., MPLS, PW, VXLAN, NVGRE, OTV, STT, etc).
	The broadcast, multicast and non-IP is not the main factor who consumes FDB table, it is not a issue at all. We cannot image that VMs generate many non-IP traffic.
	I am very happy that we finally recognize the issue.

Hewen

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 9 Oct 2013 08:11:36 +0200 (CEST)

On Tue, 8 Oct 2013, Melinda Shore wrote:

> I've been finding this a really interesting discussion and some of the 
> arguments against adoption have been compelling.  This one surprised me 
> a bit - it reads as if you're skeptical about using this mechanism to 
> address scaling issues, but it's the lack of text addressing 
> broadcast/multicast/layer 2 that you find the larger problem.  So I have 
> to ask: if they provided text addressing broadcast/multicast/layer 2 
> would that address your objection, or is it actually to the basic 
> proposal?

I don't see how they would address broadcast/multicast/L2 and make it 
work, but if they did, I'd take another look at it and re-evaluate. As 
written currently, I'm against.

Also, putting an additional header/label/whatever on the packet is a much 
cleaner solution than to try to address per-protocol, looking into L3 and 
trying to do the right thing, especially since this "look into L3" is 
different for each L3 protocol.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 9 Oct 2013 08:25:51 +0200 (CEST)

On Wed, 9 Oct 2013, Zhenghewen wrote:

> 	A new header into current protocol is a option, but I think that 
> the draft suggests a more general and feasible solution than defining 
> new encapsulation into many protocols (e.g., MPLS, PW, VXLAN, NVGRE, 
> OTV, STT, etc).

I don't see why you need to create new encapsulation in these protocols.

I thought they solved the same problem that you're trying to solve. Please 
elaborate what you're solving that they're not solving currently.

> 	The broadcast, multicast and non-IP is not the main factor who 
> consumes FDB table, it is not a issue at all. We cannot image that VMs 
> generate many non-IP traffic.

Well, I do. LLDP and different kinds of heartbeat protcols just naming a 
few. You need to remember that just a single packet hosted from a device 
will create an FDB entry. So basically your solution needs to implement 
ethertype based filtering within the core so you only allow packets with 
ethertypes you recognise, otherwise you'll kill the whole solution by FDB 
explosion if devices all of a sudden start emitting frames that you're not 
doing SARP for. So the SARP devices need to just forward frames it 
understands and drop the rest, right?

Also, what will happen in case you're sending an IPv4 packet with an IPv6 
ethertype. Are you going to look at the IP version field on L3 and decide 
what to do, are you going to do checking and drop if ethertype and IP 
version doesn't match? The document currently doesn't say anything about 
this.

---

From: Zhenghewen <[email-masked]>
Date: Wed, 9 Oct 2013 15:34:14 +0800

Please see inline.

-----Original Message-----
From: Mikael Abrahamsson [mailto:[email-masked]] 
Sent: Wednesday, October 09, 2013 2:26 PM
To: Zhenghewen
Cc: 'Internet Area'
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

[Hewen]I think that a new encapsulation will be desired as you said ("I think the better engineering solution is to encapsulate in additional header (provider bridging, MPLS or similar) to solve the scalability problem you're trying to solve.") I do not see available existed solution to solve the issue, would you provide some info? Thanks in advance.

[Hewen]In my understanding, one proposed solution in the draft acts as an ARP-aware Ethernet switch and forwarding decision is made via destination MAC address, it do not have to recognize frame type (Ethertype). In the world of Ethernet switching, FDB has nothing with Ethertype, it only care destination MAC address (excluding broadcast, sometimes excluding multicast when switch chipset dispose multicast as broadcast). As you point out, if use the solution in the draft, the device will need to recognize ARP, but do not need to recognize other frame or modify Ethertype according to my understanding. I am waiting for the text of section 3.4 and 3.5, it is same as you, but different from you, I support the draft because it provides a basic solution to solve MAC address scalability issue of cloud data centre.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 9 Oct 2013 10:09:03 +0200 (CEST)

On Wed, 9 Oct 2013, Zhenghewen wrote:

> [Hewen]I think that a new encapsulation will be desired as you said ("I 
> think the better engineering solution is to encapsulate in additional 
> problem you're trying to solve.") I do not see available existed 
> solution to solve the issue, would you provide some info? Thanks in 
> advance.

MPLS fits my example, so does provider bridging. You take a packet, smack 
on a label or additional header, send it elsewhere (inbetween boxes 
doesn't need to know what is in the packet, thus solving scaling), 
destination box decapsulates, and it's sent locally there as well.

> [Hewen]In my understanding, one proposed solution in the draft acts as 
> an ARP-aware Ethernet switch and forwarding decision is made via 
> destination MAC address, it do not have to recognize frame type 
> (Ethertype). In the world of Ethernet switching, FDB has nothing with 
> Ethertype, it only care destination MAC address (excluding broadcast, 
> sometimes excluding multicast when switch chipset dispose multicast as 
> broadcast). As you point out, if use the solution in the draft, the 
> device will need to recognize ARP, but do not need to recognize other 
> frame or modify Ethertype according to my understanding. I am waiting 
> for the text of section 3.4 and 3.5, it is same as you, but different 
> from you, I support the draft because it provides a basic solution to 
> solve MAC address scalability issue of cloud data centre.

I don't understand how your proposed solution would work without 
understanding contents of the packets it needs to SARP. All packets. And 
it needs to drop the packets it doesn't understand, or at least not 
forward them to the "wide area network" side of things? Right?

---

From: Zhenghewen <[email-masked]>
Date: Wed, 9 Oct 2013 16:58:01 +0800

Please see inline.

-----Original Message-----
From: Mikael Abrahamsson [mailto:[email-masked]] 
Sent: Wednesday, October 09, 2013 4:09 PM
To: Zhenghewen
Cc: 'Internet Area'
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

[Hewen]I know your meaning. MPLS or PBB or Trill can use shim to make intermediate nodes not being aware the address of inner payload, it is right. It can reduce the FDB table burden of intermediate nodes, but it does not impact the edge nodes. Those solutions do not solve MAC address scalability issue of edge nodes.

[Hewen]Right, if the device does not recognize a frame, or does not know where the frame should go, the device will drop the frame. I wanted to say in previous email was that Ethernet switch does not have to distinguish IPv4 or IPv6 when it only runs as Ethernet frame switching. Traditional Ethernet switch needs to recognize 802.1Q (ethertype: 0x8100), Q-in-Q(ethertype: 0x9100), but does not need to recognize IPv4(ethertype: 0x0800) or IPv6 (ethertype: 0x86DD). In my understanding, from your viewpoint, it is some kind of Ethertype filter. 
Actually, traditional Ethernet switch does not need to recognize ARP/ND, but according to SARP, at least ARP/ND-aware is necessary to SARP.
Another thing, some kind of heartbeat messages use raw Ethernet frame such as Sun Enterprise Cluster system's DLPI, DLPI uses explicit MAC address, it cannot be routed but it can be switched. I think that SARP could let it be because the amount of cluster's members is far less than the number of tenants' VMs, I think that SARP should focus on Layer3 packets (I do not know whether the authors think so).

---

From: Ted Lemon <[email-masked]>
Date: Wed, 9 Oct 2013 08:55:19 -0400

On Oct 9, 2013, at 4:09 AM, Mikael Abrahamsson <[email-masked]> wrote:
> MPLS fits my example, so does provider bridging. You take a packet, smack on a label or additional header, send it elsewhere (inbetween boxes doesn't need to know what is in the packet, thus solving scaling), destination box decapsulates, and it's sent locally there as well.

This doesn't actually solve the stated problem.   In the solution you propose, you still need a table on the origin switch that maps either L2 or L3 addresses to destination switches.

I am not convinced that the proposal is the right way to solve this problem, or even that the problem needs solving, but let's make sure that we are all talking about the same thing, or we'll never reach consensus!   :)

---

From: David Allan I <[email-masked]>
Date: Wed, 9 Oct 2013 14:00:54 +0000

My point was I would not have a design whereby every MAC for every vNIC in a cloud had to be in every FDB, which is what I believe you are describing.

I do not think I've seen mention of how SARP plays into an Ethernet active topology or VLANs. But if  you ran a single instance of spanning tree and a single VLAN and each VM had a sufficiently diverse set of peers that MAC table entries never aged out, you would end up with the scenario you described.  I do not think that is a realistic scenario.

Once it is partitioned into multiple VLANs and uses MSTP or SPBV the number of MACs in any individual FDB starts to divide, if you add MACinMAC/SPBM in divides significantly further, orders of magnitude.  The set of MACs a TOR would see would be a fraction of the whole network even in extreme scenarios. It is not a simple linear sum of endpoints.

Dave

-----Original Message-----
From: Zhenghewen [mailto:[email-masked]] 
Sent: Tuesday, October 08, 2013 6:11 PM
To: David Allan I; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Hi, Dave,

Hewen

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of David Allan I
Sent: Tuesday, October 08, 2013 10:12 PM
To: Zhenghewen; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Dave

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Zhenghewen
Sent: Tuesday, October 08, 2013 5:08 AM
To: 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Hi Thomas,

Best regards,
Hewen

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Saturday, September 28, 2013 5:28 AM
To: Suresh Krishnan
Cc: Internet Area
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Thomas

---

From: Zhenghewen <[email-masked]>
Date: Thu, 10 Oct 2013 10:43:27 +0800

Hi, Dave,

	MPLS or PBB or Trill can use shim to make intermediate nodes not being aware the address of inner payload, it is right. It can reduce the FDB table burden of intermediate nodes, but it does not impact the edge nodes. Those solutions do not solve MAC address scalability issue of edge nodes. Those edge nodes include TOR switch and DC gateways.
	A host must consume at least a MAC entry of FDB table of edge nodes, VLAN only divides those MAC addresses into many groups but not reduce the consumption.
	Some solutions in L2VPN, such as PBB or VPLS, solve MAC address scalability issue of core network, but MAC address scalability issue of edge nodes is out of the scope, for example, we do not see a solution to solve C-MAC scalability issue on CE or DC Gateway.
	Network virtualization breaks subnet into many pieces in Cloud era, edge nodes (e.g., DC Gateway or TOR) will have to be aware other MAC address from other data centres, part or all of the other hosts in other data centres, it is a fact.

Hewen

-----Original Message-----
From: David Allan I [mailto:[email-masked]] 
Sent: Wednesday, October 09, 2013 10:01 PM
To: Zhenghewen; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Dave

-----Original Message-----
From: Zhenghewen [mailto:[email-masked]] 
Sent: Tuesday, October 08, 2013 6:11 PM
To: David Allan I; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Hi, Dave,

Hewen

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of David Allan I
Sent: Tuesday, October 08, 2013 10:12 PM
To: Zhenghewen; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Dave

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Zhenghewen
Sent: Tuesday, October 08, 2013 5:08 AM
To: 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Hi Thomas,

Best regards,
Hewen

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Saturday, September 28, 2013 5:28 AM
To: Suresh Krishnan
Cc: Internet Area
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Thomas

---

From: David Allan I <[email-masked]>
Date: Thu, 10 Oct 2013 20:10:11 +0000

Hi Hewen:

My point would be that something like MACinMAC (or other tunneling mechanism that implements VLANs), not only hides state from the core, but the state at the edge a function of the sum of end points in the set of locally attached VLANs, not the all MACs in all  VLANs supported by the whole network. So I am not seeing a scenario whereby any edge device sees all MACs in a network, just a fraction of them...even when the edge device grows rather large...

If you have a use case whereby this is not true (and not just a consequence of extrapolating a bad design) I would be interested.

I hope that is clearer
Dave

-----Original Message-----
From: Zhenghewen [mailto:[email-masked]] 
Sent: Wednesday, October 09, 2013 7:43 PM
To: David Allan I; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Hi, Dave,

Hewen

-----Original Message-----
From: David Allan I [mailto:[email-masked]] 
Sent: Wednesday, October 09, 2013 10:01 PM
To: Zhenghewen; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Dave

-----Original Message-----
From: Zhenghewen [mailto:[email-masked]] 
Sent: Tuesday, October 08, 2013 6:11 PM
To: David Allan I; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Hi, Dave,

Hewen

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of David Allan I
Sent: Tuesday, October 08, 2013 10:12 PM
To: Zhenghewen; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Dave

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Zhenghewen
Sent: Tuesday, October 08, 2013 5:08 AM
To: 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Hi Thomas,

Best regards,
Hewen

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Saturday, September 28, 2013 5:28 AM
To: Suresh Krishnan
Cc: Internet Area
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Thomas

---

From: Zhenghewen <[email-masked]>
Date: Fri, 11 Oct 2013 10:11:28 +0800

Hi, Dave,

	Let me repeat my viewpoint, the edge nodes will have to touch more addresses (host IP address or MAC address) from outer data centres under cloud data centre mode than under traditional internet data centre mode, because larger layer 2 domain introduces more addresses to the edge nodes. How large? Maybe all addresses of hosts inside the whole layer 2 domain, maybe part. More data centres join the cloud, More addresses the edge nodes touch.
	
	I use the Figure 1 ("Interconnecting TRILL Networks with TRILL-EVPN") in "TRILL-EVPN" (draft-ietf-l2vpn-trill-evpn-01) as the basic topology to explain the scenario, I add some notes on the figure as below.
                (In the first try to response your mail, I have to insert a picture as the attachment because my Outlook cannot correctly display the figure, but the response is blocked by the ietf mail system because " Message body is too big: 73900 bytes with a limit of 40 KB", so I have to align those characters to form a visible and readable figure in Outlook, hope that your mail client can read it, you can refer to figure 1 of draft-ietf-l2vpn-trill-evpn-01 if your mail client not work;
	If ietf mail system let the first response go, please ignore it, sorry the duplicated mail)
                            +---------+          +---------------------------------+                +-------+
   +--------------+  |            |           |                                           |                  |         |  +---------------+
   | Trill Edge |  |            |           |        MPLS                         |                  |         |  | Trill Edge |
   |    RB#1     |--|            |           |                                           |                  |         |--|    RB#3     |
   |    (TOR)    |   |            |    +--------------------+      +--------------------+   |          |  |    (TOR)    |
   +--------------+  |            |    | MPLS Edge     |      | MPLS Edge      |    |         |  +--------------+
                             |  Trill  |---|        PE#1          |      |         PE#2         |---| Trill |
   +--------------+   |           |    |(DC Gateway) |      |(DC Gateway) |   |          |  +--------------+
   | Trill Edge |  |            |    +--------------------+      +--------------------+   |          |  | Trill Edge |
   |    RB#2     |--|            |          |                                            |                  |         |--|    RB#4     |
   |    (TOR)    |   |            |          |                                            |                  |         |  |    (TOR)     |
   +--------------+  |            |          |       Backbone                   |                  |         |  +--------------+
                            +---------+          +---------------------------------+                 +-------+

   |<-----Data Centre #1------->|<-------------Core---------->|<----------Data Centre #2-------->|
   
   |<------------- IS-IS ------------->|<------------BGP------------>|<---------------- IS-IS ----------------->|  CP
   
   |<-------------------------------------------------  TRILL ------------------------------------------------------->|  DP
                                                      |<-------------MPLS---------->|

	There are two data centres (Data Centre #1 and Data Centre #2) interconnected via MPLS-based L2VPN to form a cloud data centre. We can see a larger layer2 domain across two data centres than just in any single data centre, and the edge nodes will have chance to touch more hosts' addresses, it is must.
	"TRILL-EVPN" (draft-ietf-l2vpn-trill-evpn-01) uses C-MAC Address Transparency to reduce FDB table burden of MPLS edge nodes, the way to do that should be that MPLS edges do not act as Trill Edge.
	But we do not see any solution to consider FDB table burden of Trill edge nodes (e.g., TOR).

Hewen

-----Original Message-----
From: David Allan I [mailto:[email-masked]] 
Sent: Friday, October 11, 2013 4:10 AM
To: Zhenghewen; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Hi Hewen:

I hope that is clearer
Dave

-----Original Message-----
From: Zhenghewen [mailto:[email-masked]] 
Sent: Wednesday, October 09, 2013 7:43 PM
To: David Allan I; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Hi, Dave,

Hewen

-----Original Message-----
From: David Allan I [mailto:[email-masked]] 
Sent: Wednesday, October 09, 2013 10:01 PM
To: Zhenghewen; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Dave

-----Original Message-----
From: Zhenghewen [mailto:[email-masked]] 
Sent: Tuesday, October 08, 2013 6:11 PM
To: David Allan I; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Hi, Dave,

Hewen

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of David Allan I
Sent: Tuesday, October 08, 2013 10:12 PM
To: Zhenghewen; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Dave

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Zhenghewen
Sent: Tuesday, October 08, 2013 5:08 AM
To: 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Hi Thomas,

Best regards,
Hewen

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Saturday, September 28, 2013 5:28 AM
To: Suresh Krishnan
Cc: Internet Area
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Thomas

---

From: Zhenghewen <[email-masked]>
Date: Fri, 11 Oct 2013 09:56:14 +0800

Hi, Dave,

	I use the Figure 1 ("Interconnecting TRILL Networks with TRILL-EVPN") in "TRILL-EVPN" (draft-ietf-l2vpn-trill-evpn-01) as the basic topology to explain the scenario, I add some notes on the figure as below.
               (I have to insert a picture as the attachment because my Outlook cannot correctly display the figure).

Hewen
-----Original Message-----
From: David Allan I [mailto:[email-masked]] 
Sent: Friday, October 11, 2013 4:10 AM
To: Zhenghewen; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Hi Hewen:

I hope that is clearer
Dave

-----Original Message-----
From: Zhenghewen [mailto:[email-masked]] 
Sent: Wednesday, October 09, 2013 7:43 PM
To: David Allan I; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Hi, Dave,

Hewen

-----Original Message-----
From: David Allan I [mailto:[email-masked]] 
Sent: Wednesday, October 09, 2013 10:01 PM
To: Zhenghewen; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Dave

-----Original Message-----
From: Zhenghewen [mailto:[email-masked]] 
Sent: Tuesday, October 08, 2013 6:11 PM
To: David Allan I; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: RE: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Hi, Dave,

Hewen

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of David Allan I
Sent: Tuesday, October 08, 2013 10:12 PM
To: Zhenghewen; 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Dave

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Zhenghewen
Sent: Tuesday, October 08, 2013 5:08 AM
To: 'Thomas Narten'; Suresh Krishnan
Cc: 'Internet Area'
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Hi Thomas,

Best regards,
Hewen

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Narten
Sent: Saturday, September 28, 2013 5:28 AM
To: Suresh Krishnan
Cc: Internet Area
Subject: Re: [Int-area] Call for adoption of draft-nachum-sarp-06.txt

Thomas