Subject: Re: [pcp] Comments on draft-cheshire-pcp-recovery-02

From: RJ Atkinson <[email-masked]>
Date: Tue, 1 Nov 2011 10:04:59 -0400

Earlier, Reinaldo Penno wrote, in part:
> In the worst (but simpler) case firewall/NAT router would send 
> packet on all (possibly thousands of interfaces).  Still works 
> but it will trigger an avalanche of renewal that the PCP Server 
> will need to handle. 

Yes.  However, that is entirely a "quality of implementation" issue.
That is NOT a reason to change the PCP Rapid Recovery specification.

A firewall/NAT router needs to be able to handle large volumes
of packets on all interfaces ANYWAY, independently of PCP.

If it can't handle such volumes, then the device either needs 
a more capable control plane or fewer interfaces.  Neither of
those fixes have anything to do with PCP Rapid Recovery.

> Apart from that, text is okay.

Then the Rapid Recovery multicast text should remain a MUST, 
because the IETF ordinarily does NOT water down specifications 
for "quality of implementation" issues.

Yours,

Ran

---

From: Reinaldo Penno <[email-masked]>
Date: Tue, 1 Nov 2011 15:28:36 -0400

> That is NOT a reason to change the PCP Rapid Recovery specification.

When a TCP segment is lost you do not say, "retransmit everything from start
because I have quality". When a link fails BGP does not tell all peers to
withdraw all prefixes.

You do not tell another device that you might have lost state when in fact
you didn't because you do not bother to keep track. There is another word
for this and it is certainly not related to quality.

Thanks,

Reinaldo

> _______________________________________________
> pcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/pcp

---

From: RJ Atkinson <[email-masked]>
Date: Tue, 1 Nov 2011 15:49:09 -0400

On 01  Nov 2011, at 15:28 , Reinaldo Penno wrote:
> When a link fails BGP does not tell all peers to
> withdraw all prefixes.

There are numerous deployed BGP implementations
that are bad quality and don't behave well in
the field. 

I can't imagine any large site or service provider 
using those implementations, yet they are still out 
there in the deployed Internet.  I'm told by various
folks that those implementations cause real operational 
problems too.  

Quite separately, all implementations have some bugs.
(Of course, most implementers fix their bugs as they 
find them, but bug-free hardware/software doesn't exist.)

The IETF learned a long while back that it cannot
force implementers to have good quality -- that is
something that only the marketplace can sort out.

This is partly why Postel's Law is so popular here.

> You do not tell another device that you might have
> lost state when in fact you didn't because you do not
> bother to keep track. There is another word for this
> and it is certainly not related to quality.

The word "quality" defines a continuum -- from good 
quality to bad quality -- not a particular point 
on that continuum.

Also, and particularly applicable to PCP, there are
a range of operational environments.  Many small end 
sites are quite happy to sacrifice some implementation 
quality to gain lower capital cost.  Meanwhile, the 
PCP WG Charter covers the whole broad spectrum of 
environments.

Cheers,

Ran

---

From: james woodyatt <[email-masked]>
Date: Tue, 01 Nov 2011 13:12:23 -0700

Likewise, some implementers occasionally find they can sacrifice some additional NRE costs in exchange for software quality assurances. (At least, I've heard people tell stories to that effect. And they were sober at the time, so I believe them.)

--jhw (sent from my phone)

On Nov 1, 2011, at 12:49, RJ Atkinson <[email-masked]> wrote:

> Many small end 
> quality to gain lower capital cost.

---

From: Reinaldo Penno <[email-masked]>
Date: Tue, 1 Nov 2011 16:13:59 -0400

No, but this is protocol design issue. Saying a PCP Server 'must' send
multicast to all interfaces is not the appropriate design because it leaves
no room for implementations that want to do the right thing.

If the design itself does not reward good implementations then we are
already starting with the wrong foot.

Here is the proposal

- "SHOULD ..." with 'MAY send recovery packet only on those interfaces that
have PCP Clients bound to the PCP Server that may have lost state'.

Let me know if it addresses your concerns.

---

From: RJ Atkinson <[email-masked]>
Date: Tue, 1 Nov 2011 16:46:05 -0400

On 01  Nov 2011, at 16:13 , Reinaldo Penno wrote:
> No, but this is protocol design issue.

It is not.  

Any implementation can have issues; more below.

> Saying a PCP Server 'must' send multicast to all interfaces
> is not the appropriate design because it leaves no room
> for implementations that want to do the right thing.

The Juniper SRX in my garage loses NAT/Firewall state 
regularly.  By your own statements, this is not good
operationally, and yet you claim "to do the right thing".

Please don't try talking about your superior implementation 
approach to me.  I've got experimental evidence otherwise.

> If the design itself does not reward good implementations
> then we are already starting with the wrong foot.

Well, the SRX in my basement is an existence proof 
that you don't have the "good implementation" that 
you claim to have, OR -- alternately put -- if your 
implementation is so great, then your implementation 
is an existence proof that this particular capability 
is absolutely essential rather than optional.

> Here is the proposal
> 
> - "SHOULD ..." with 'MAY send recovery packet only on those
> interfaces that have PCP Clients bound to the PCP Server
> that may have lost state'.
> 

You make impressively arrogant claims, but they
are not a solid technical argument for your preference.

It is important that PCP servers MUST send the multicast 
packets on all interfaces where PCP Clients might exist.
because I've seen your implementation (and several other
genetically different NAT/Firewall implementations) 
lose NAT/Firewall state many times.  

Losing state is inevitable, unavoidable, in the real 
world of deployments.  We need this mechanism to provide 
resilience in the face of such incidents.

Yours,

Ran

---

From: Reinaldo Penno <[email-masked]>
Date: Tue, 1 Nov 2011 17:41:01 -0400

> operationally, and yet you claim "to do the right thing".

Never said, nope. I said that when state is lost, we should not tell clients
_that did not loose state_ that they might have.

The use case in question is the following:

- multiple subscribers/PCP clients connected to an edge router. These PCP
clients are served by more than one PCP Server (embedded or not). If you one
PCP Server reboots, should a multicast message be sent on all interfaces?

> lose NAT/Firewall state many times.

Again, it seems there is a disconnect. Certainly state might be lost but a
PCP Server should multicast packets on all interfaces where _its_ PCP
clients might exist.

Another example, A PCP Server/NAT should not send multicasting recovery
packets on all interfaces of a VRF different than the one the PCP Server
(and clients) are located.

---

From: Stuart Cheshire <[email-masked]>
Date: Tue, 01 Nov 2011 14:43:41 -0700

> no room for implementations that want to do the right thing.

If the PCP Server sends an announcement with the right Epoch time  
value, then the clients will shrug it off and it's a no-op.

If the PCP Server sends its packets with the *wrong* Epoch time value  
then this is pretty good evidence that the PCP Server is broken and  
clients are right to doubt that it is competent to maintain their  
state reliably.

Either way, a PCP Server multicast announcement is the right thing to  
do.

When no state has been lost it's benign, and when state *has* been  
lost it's very beneficial.

Stuart Cheshire <[email-masked]>
* Wizard Without Portfolio, Apple Inc.
* www.stuartcheshire.org

---

From: Reinaldo Penno <[email-masked]>
Date: Tue, 1 Nov 2011 17:52:10 -0400

Hello Stuart,

I'm not opposing multicast recovery at all. Hopefully that was clear based
on our last interaction. I'm fine with it.

I have a few uses cases that concern me. One is multiple PCP Servers (with
same IP address) in different VRFs. If one of them reboots and a multicast
message is sent one all interfaces of all VRFs then:

- The source IP address would pass sanity check
- the epoch would be most certainly trigger a recovery

All PCP Clients would try recovering. Depending on the client/server
implementation that might cause a disruption since client might/will assume
the mapping is gone.

Thanks,

Reinaldo

---

From: Stuart Cheshire <[email-masked]>
Date: Tue, 01 Nov 2011 15:10:17 -0700

> - the epoch would be most certainly trigger a recovery

If you want to make the PCP server be different hardware from the NAT  
gateway you certainly face additional challenges. That's like making  
your DHCP server be different hardware from the device that actually  
assigns the IP addresses. The PCP server will need to communicate  
with the NAT gateway using some proprietary protocol of your own.  
That proprietary protocol needs to be adequate to represent PCP  
semantics, or you will have a hard time making it work.

The Epoch time is *not* the time since your (hypothetically separate)  
PCP server crashed. The Epoch time is the time since the inception of  
the current collection of NAT mapping state.

The Epoch time is a property of the NAT gateway state, not the PCP  
server.

The granularity of what you define to be "NAT gateway state" is up to  
you. For a home NAT gateway it is probably the state of the entire  
box, and resets when it is power cycled. For NAT configurations where  
the NAT system is able to lose part of its state but not all, the  
granularity is the unit of potential state loss, whatever that may  
be. If the NAT system is able to lose state for some clients but not  
others, then it makes sense for each client to be told its own Epoch  
time value which pertains to it's own little bit of the overall NAT  
state. If that client's bit of NAT state is lost, then that client's  
Epoch time value resets to zero, but the Epoch time for non-lost NAT  
state is unaffected.

I think this is unnecessarily complex, but you could easily design  
your system that way if you expect this kind of partial loss to be  
common.

---

From: Reinaldo Penno <[email-masked]>
Date: Tue, 1 Nov 2011 18:54:49 -0400

> semantics, or you will have a hard time making it work.

I was actually separating subscriber interfaces from (PCP Server, NAT
Gateway). It goes back to my original email that you need more than PCP
Server and NAT gateway on the same device - you need subscriber interfaces
to be part of this device as well.

If they are, you can multicast packets only on the subscriber facing
interfaces associated with the 'NAT gateway' that went down. Otherwise on a
trigger from the NAT gateway the subscriber router will need to multicast on
all interfaces of all VRFs.

                                ,------.
                               /        `--.
                             ,'             `.
                            (    Internet     )  +------------+
                             `.              /   | NAT A      |
                               `------------'  .-' PCP Server |
                                             .'  +------------+
                                          .-'
                     +------------------.'+
                     | +--------+    .-'  |      +------------+
   Subscribers A.......|        |  .'     |      | NAT B      |
                     | |  VRFA  .-'       |   _.-| PCP Server |
                     | +--------+       _..-''   +------------+
                     | |        | Edge''  |
   Subscribers B'''''''|  VRFB  ''Router  |
                     | +--------+         |
                     | |        |         |
   Subscribers C-------|  VRFC  +...___   |      +------------+
                     | +--------+      ``-+-...__| NAT C      |
                     +--------------------+      ` PCP Server |
                                                 +------------+

> the current collection of NAT mapping state.

Correct, but my use case was a bit different. Hopefully the picture
clarifies. The topology above is very common on both wireless and wireline.

> The Epoch time is a property of the NAT gateway state, not the PCP

Yes, if PCP Server/NAT 'A' above goes down, and we want to use multicast
recovery:

- edge router should send multicast only on those interfaces and VRFs that
are associated with the given NAT Gateway
- edge router must send a multicast recovery on all VRF/interfaces? This is
what the current text implies (or does not fully specify).

Notice that from a PCP Server/NAT perspective it has only a single
interface. Therefore the current text does not translate well to these
deployments.

I understand that on, say, an airport extreme, this is all collapsed and
makes things easier.

Bottom line, I would like multicast recovery to work (as much as possible)
on distributed environments such as ISP edge/core as well. For that, it
seems we need some more wording/scenarios on the draft. I can provide text
is necessary.

Thanks,

Reinaldo

---

From: RJ Atkinson <[email-masked]>
Date: Wed, 2 Nov 2011 08:39:23 -0400

>                               `------------'  .-' PCP Server |

The picture and description are of some help, but of limited value, 
because PCP does not have the concept of a VRF.  The PCP-related
I-Ds being discussed here (base specification, Rapid Recovery)
both are silent about the behaviour of VRFs.

(ASIDE:  I'll note that the acronym "VRF" is used in a small 
number of IETF RFCs.  It does NOT have the same meaning in 
all of those RFCs.  For example, the "VRF" definition in 
RFC-4026 is not identical to the "VRF" definition in RFC-4364.)

Any questions about VRFs necessarily will have answers
that depend upon the details of one's VRF implementation,
since PCP does not have a standard definition for a VRF.
For help with how one's VRF implementation relates to 
one's PCP implementation, one ought to look internally, 
not to the IETF.

>> The Epoch time is a property of the NAT gateway state,
>> not the PCP server.
> 
> Yes, if PCP Server/NAT 'A' above goes down,
> and we want to use multicast recovery:
> 
> - edge router should send multicast only on those interfaces
>   and VRFs that are associated with the given NAT Gateway
> - edge router must send a multicast recovery on all VRF/interfaces?
>   This is what the current text implies (or does not fully specify).

I think the current text does NOT say what you write above.
I'll quote what I think is the applicable text from 
the I-D (draft-cheshire-pcp-recovery-02, Section 3, Page 2):

	When a PCP server device [PCP] that implements 
	PCP Rapid Recovery reboots, restarts its NAT engine, 
	or otherwise enters a state where it may have lost 
	some or all of its previous mapping state (or enters
	a state where it doesn't even know whether it may 
	have had prior state that it lost) it MUST inform 
	PCP clients of this fact by multicasting the UDP packet 
	shown below on all multicast-capable interfaces 
	on which it accepts PCP requests.

If you think the current text is confusing, it might be
more productive if you proposed text that you find more 
clear.  Often, sadly not always, that helps focus the 
conversation.  It is at least worth trying in this case.

Kindly note that any proposed text ought NOT talk about VRFs
at all, because PCP does not have the concept of a VRF.

Also, minimal edits are usually best, as minimal editing 
simplifies attaining consensus.

> Notice that from a PCP Server/NAT perspective it has only a
> single interface.

This is NOT obvious either from your description or from your 
diagram.

Again, note that the PCP specification is silent about VRFs,
and does not talk about them in ANY way.  VRFs are ENTIRELY 
a proprietary implementation detail of your system, and are
NOT part of the PCP specification set.

> seems we need some more wording/scenarios on the draft. I can provide text

It is probably helpful if you provide some minimally different
proposed text, and clearly indicate which existing text you 
propose to replace.

To repeat the comment above, please do NOT talk about VRFs 
in your proposed new text, as the PCP specification set 
does not contain the concept of a VRF.  Trying to add the
concept of a VRF to the PCP specification at this late date
is very likely to slow progress of the PCP base specification 
further.

Yours,

Ran

---

From: Reinaldo Penno <[email-masked]>
Date: Thu, 3 Nov 2011 00:45:48 -0400

Hello,

Separation of subscriber management interfaces from NAT/PCP and use VRFs are
common practice in CGN deployments.

Such deployments and strategy are discussed in multiple draft related to
IPv4/IPv6 co-existence and transition. A few references.

http://tools.ietf.org/html/draft-kuarsingh-lsn-deployment-05
http://tools.ietf.org/html/draft-ietf-softwire-gateway-init-ds-lite-05
http://tools.ietf.org/html/rfc6342
http://tools.ietf.org/html/draft-ietf-v6ops-3gpp-eps-08

The charter talks about CGNs but given the recent thread a clarification is
needed if PCP protocol should be appropriately specified so it can work on
wireless/wireline service provider networks as described above.

Can the chairs please clarify the scope of PCP?

If such network architectures are in scope will this be part of the base
spec or can we have a new deliverable on the charter?

Thanks,

Reinaldo

---

From: Dave Thaler <[email-masked]>
Date: Thu, 3 Nov 2011 20:00:58 +0000

Reinaldo Penno writes:
> Separation of subscriber management interfaces from NAT/PCP and use VRFs
> are common practice in CGN deployments.
> 
> If such network architectures are in scope will this be part of the base spec or
> can we have a new deliverable on the charter?

The charter says:
"The PCP working group is chartered to standardize a client/server Port
 Control Protocol (PCP) to enable an explicit dialog with a middlebox
 such as a NAT or a firewall to open up and/or forward TCP or UDP port,
 regardless of the location of that middlebox."

In my view the "regardless of the location of that middlebox" means all topologies
are in scope for discussion for the WG.   That doesn't mean the WG has to
support all possible topologies, if consensus is to constrain them in some way.
It also doesn't imply that the base PCP spec itself has to support all topologies 
itself, that's a WG decision.   If the WG chooses to put extensions to support 
certain topologies in a separate document, the WG could do so.   We already have
WG consensus to do that for DS-lite topologies.

-Dave

---

From: Reinaldo Penno <[email-masked]>
Date: Thu, 3 Nov 2011 16:39:35 -0400

Thanks for the answers Dave.

In order to move forward with the base spec I will work on a separate draft
to document or specify PCP for Service Provider networks.

If there are any other people on the list interested on working on such
document, please send me a message.

Thanks,

Reinaldo

---

From: Alain Durand <[email-masked]>
Date: Thu, 3 Nov 2011 13:46:55 -0700

On Nov 3, 2011, at 4:00 PM, Dave Thaler wrote:

> R

Agreed. The base spec has to remain simple. Extensions can be defined in separate documents.
An example of that is the Zone ID proposal.

  - Alain.