Subject: Re: [Pppext] Review of draft-bberry-pppoe-credit (Informational)

From: Mark Lanzo <[email-masked]>
Date: Wed, 27 Jul 2005 15:38:03 -0400 (EDT)

On 27 Jul, Vernon Schryver wrote:

>> From: Mark Lanzo <[email-masked]>
>> To: [email-masked]
>> cc: [email-masked], [email-masked], [email-masked],
>>         [email-masked], [email-masked]
> 
> 
>> PPPoE is really just being used as a means to multiplex multiple
>> PPP sessions over the ethernet link to the radio, and as a carrier
>> for signalling info between radio and router.
> 
> Which raises my question of what PPPoE has to do with the solution
> and my point that PPP(oE) is widely misunderstood as other than a
> link layer protocol.  Why use PPPoE to link networks instead of IP?

The proposal *is* using PPPoE as the link layer.  If anything, more 
than PPPoE is normally used as such.

The whole point is that you have a router and an external radio.  
The radio has an ethernet port on it, by which it attaches to the 
router.  The radio itself detects other radios in its vicinity,
and creates a semblance of a serial connection to each of those
adjacent radios.  The customer wants the router to run PPP on these 
serial connections.

The radio doesn't provide a slew of serial ports to attach to the
router, one for each of the remote radio nodes.  All it provides is an
ethernet port.  So the radio is basically a smarter-than-average serial 
port card with lots of ports, which plugs into the router via an 
ethernet rather than being inserted into a slot on the router's backplane.  
The serial links have to be simulated at the router end. Bottom line 
being that the ethernet is really just a funky "backplane" here, and
the goal is to couple the router as close as is reasonably possible to
its "serial" ports.

All the different serial sessions have to somehow be multiplexed
over the ethernet port which acts as the bus to the radio.  So how do 
you do that?  Obviously you need some sort of frame format which can carry 
a "serial" packet to/from the radio, and which also includes some sort 
of identifier so that both router and radio can distinguish which particular 
remote serial user this packet is for.  It is also desirable to have some 
sort of method for conveying signalling information related to the virtual
serial links - a new link has appeared, an old one is gone, a link
is congested, etc.  

You could invent a new protocol for this, get a new ethertype assigned
to you, etc etc.  Or you could recognize that the PPPoE frame format,
and even the PPPoE protocol (with some relatively minor additions)
fits the bill.

I understand why you think of PPPoE as a kludge, and can understand
reluctance to propogate use of a kludge.  Still, it seems to me that
if anything that the intended use here is a better use - less of a 
kludge - than PPPoE as it is normally deployed.  

It occurs to me that the biggest faux pas here is mentioning PPP in the 
name.  This scheme really does relegate PPPoE to the link layer, and 
for that matter practically divorces PPP from PPPoE altogether.  In
truth, it's just a method for transferring serial packets over the 
"backplane" to the radio.  The fact that the packets contain PPP frames 
is barely relevant.  Unfortunately, describing the proposal as an 
extension to "PPPoE" seems to be having a very negative effect, because 
it brings with it all the negative baggage associated with PPPoE.

>> A completely new protocol could have been devised for all of this,
>> but since the nature of the info traded by that protocol would have
>> been very much like the nature of the info traded by PPPoE, I think
>> the authors wanted to use PPPoE precisely so they wouldn't be
>> completely reinventing the same old wheel.  The purpose of the document
>> isn't to make a general-purpose extension to PPPoE, it's to create
>> a special purpose extension for a very specific sort of environment.
> 
> So why not use the official IETF standard way to link networks instead
> of piling kludges on top of the kludge that is PPPoE? 

What standard official IETF way of multiplexing virtual serial 
connections over an ethernet (or any media) "backplane" are you 
referring to?
 
> The IETF standard way would be to give the routers IP addresses and
> use a real routing protocol (or even static routes) instead of what
> looks sounds like a retread of IGRP.  You might use PPPoE (or not)
> between the access concentrators and their neighboring stationary
> radios.  You might define a PPP over RLP encapsulation that would have
> credits, battery condition, and so forth among the radios.

> Wouldn't that produce products with a wider potential market 

Just the opposite, actually (hopefully).  Routers already exist which 
support PPPoE.  Using PPPoE in this rather unusual fashion means the 
ability to hook these specialized radios into existing router equipment.
Significantly improved performance is gained for those routers which 
implement the proposed extensions.  It seems more likely that manufacturers 
would augment already developed PPPoE implementations than implement a 
whole new protocol.

> ... as well
> as fewer nastinesses such as an MTU that is even smaller than the bad
> old PPPoE MTU of 1492?

That is admittedly still an issue.  As long as the "backplane" to the 
"radio card" is an ethernet, it doesn't seem easy to avoid this (short
of resorting to some form of link layer fragmentation).  It is also
a bit off-topic since this is a technical limitation not arising from
anything within the draft itself, well understood, and which the draft 
makes no claim of trying to overcome.

> Why egregiously violate layering by trying to convert IEEE 802.3
> into an inter-network protocol?

Hopefully it is clear by now that the above statement is inaccurate,
and arises from misperceptions about the proposed mechanisms.

As I indicated above, it seems to me that there is a lot of reaction 
to this draft, but the reaction is due (at least in part) to the bias 
against traditional PPPoE and not due to the technical merits (or lack 
thereof) of this proposal or its intended goals.  My thought is that the 
draft may not be wholly clear about the environment and application, 
and this is leading to confusion.  This suggests to me that the draft 
could bear a fair amount of expansion and augmentation, before it
is ready for RFC status.  I.e. there may be legitimate reason to not
accept the draft as it currently stands, but those reasons have more
to do with clarity and completeness of the document than the technical 
merits of the proposal.

At any rate, I'm going to try to stay out of the discussion after
this.  Despite all the verbiage I've tossed out, it is not my intent
to be a spokesman for the proposed mechanism -- the authors can do 
that.  I don't have a personal stake in this, but I've had the benefit 
of in-person discussions with the group behind this, without which,
I suspect the draft wouldn't have made a lot of sense to me either.

_______________________________________________
Pppext mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pppext

---

From: Vernon Schryver <[email-masked]>
Date: Mon, 1 Aug 2005 10:33:29 -0600 (MDT)

> than PPPoE is normally used as such.

No, as far as I can tell PPPoE is being used as a network layer.

> adjacent radios.  

The radio is also a router.  That it is routing PPPoE frames instead
of IP packets does not make it other than a router.  There were routers
before IP was invented.  There are still routers that have nothing to
do with IP.

>                   The customer wants the router to run PPP on these 
> serial connections.

Why does the customer care whether PPP runs on the radio links?
What kind of PPP does the customer want?  
Are you sure the customer will be happy to have the crippled PPPoE MTU
further crippled?

> router, one for each of the remote radio nodes.

That seems irrelevant.  Many routers for many inter-network protocols
do not involve banks of serial ports.

>                                                  All it provides is an
> ethernet port.  

Ok, but that is like many other routers.  Consider an 802.3 or FDDI to
ATM or FR router with one physical fiber or RJ-11 connection for the
LAN and one physcial connection to the telephone company ATM or FR wire.

>                 So the radio is basically a smarter-than-average serial 

That does not change the fact that it is a router.  It routes or
gateways packets between one link layer network that happens to be
running eithernet and another link layer network that happens to involve
the real ether.

I have not figured out whether the frames on the real ether are PPPoE
or some other flavor of PPP or no PPP at all.  I've seen words about
that seem to conflict.

> the goal is to couple the router as close as is reasonably possible to

By that reasoning, there are no routers in NPXs, where FDDI, 802.3,
and other link layers are just being used as funky backplanes.

> and even the PPPoE protocol (with some relatively minor additions)

Why do that?  Why couldn't one do as I said and assign IP addresses
to the radios and use IP packets for all of this multiplexing
and demultiplexing?

> I understand why you think of PPPoE as a kludge, 

Without intending any offense, I really think you do not know why I
think PPPoE is wrong and evil and are even less clear about what I
do not like about this proposal.

> it brings with it all the negative baggage associated with PPPoE.

No the biggest issue (I won't say faux pas) is that the confusion
between the natures of link layers and network layers.  

As far as I can tell, this proposal is a classic example of
re-inventing the idea of an inter-network protocol within a link
layer.  It is certainly not the only example of such re-inventions
since TCP/IP beat the ISO OSI protocol suite.  Other examples
including the obvious case from Cisco may be justified on performance
grounds...or perhaps *were* justified before people started making
silicon that could classify IP headers at very high speed.

> connections over an ethernet (or any media) "backplane" are you 

What's that about "virtual serial connections"?  Exactly what do you
mean?  Many and perhaps most serial connections are "virtual" today.

The IETF standard way for multiplexing virtual serial over an Ethernet
is TCP/IP.
Note that I am *NOT* talking about PPP/telnet/TCP/IP, PPP/rsh/TCP/IP,
or certain other kludges.  I'm expressing my confusion about "virtual
serial connections."

I suspect the idea is to run PPPoE over the radios, and instead of
terminating those PPPoE links in the base station radio card, idea to
be cheap and nasty and let some *other* router do the work of an access
concentrator router.

> would augment already developed PPPoE implementations than implement a 

What is that about a "whole new protocol"?  

I do not understand what is running on the radio links.  What do is
used for access control/collision detection/etc., something in the
ALOHA tradition?  Or some kind of slotting or (virtual) token?  How do
you do framing?

What about authentication?  I think you or someone said it was plain
old PPP.   If that's right, is it PPPoE?  But I think other words
said no PPPoE on the real ether.

> and arises from misperceptions about the proposed mechanisms.

Again, as far as I can tell, the misperceptions are not mine but
fundamental to the proposal to use PPPoE as an inter-network protocol.
The glaring clue is it uses PPPoE for (de)multiplexing of 802.3 packets.

> thereof) of this proposal or its intended goals.

That is mistaken.  My reaction is not against using PPPoE, but against
using PPPoE as a replacement for IP.

>                                                   My thought is that the 
> is ready for RFC status. 

I agree with that.

Vernon Schryver    [email-masked]

---

From: James Carlson <[email-masked]>
Date: Mon, 1 Aug 2005 12:45:39 -0400

Fred Baker writes:
> I'll remind you of a statement earlier in the thread. The objective  
> was specifically *not* to impose reliability on the link in the sense  
> that LAPB or TCP are reliable. The objective was to provide a way  
> that a system-engineered interface could advise a higher layer entity  
> in a different system implementation (connected via Ethernet) of a  
> rate at which it was permitted to send.

In other words, flow control of Ethernet traffic.

Here are the issues I see:

  - What exactly do we do if there's more than *one* L2 link in the
    path between these two nodes that requires flow control?

  - Isn't this a more general problem with bridged Ethernets using
    different-speed links?  Why is a point solution acceptable in this
    one case where the real problem is that there's no end-to-end flow
    control mechanism in Ethernet networks?

Apart from the questions of endpoint parameter assignment (which I
think are more than adequately dealt with using existing protocols,
such as DHCP, rather than pressing PPP into service in non-point-to-
point applications) and authentication (802.1X), the purpose here
seems to be to use PPPoE just because it could be extended to provide
a flow-control mechanism.

In short, it seems to be a local "optimization" that would have little
general usability, would likely complicate other's implementations for
no good reason, and would even inhibit or conflict with real solutions
to the problem.  (If bridging over these radios is an important
solution space, and if the lack of buffer space and variance in
transmit access times are such that per-peer flow control is a
requirement, why isn't this dealt with either in the radio itself or
in the way bridging is run in these special cases, or even as a
general 802 problem?)

It seems to me like PPPoE is being used as a willing victim.

The reason *I* mentioned RFC 1663's LAPB was that I was a bit
surprised that the document contained no actual references to other
solutions in the same area.  At least I think it would help to be able
to say "we considered LAPB framing as part of the mechanism rather
than inventing our own credit-based scheme and header changes and
network monitoring gear to go with it, but LAPB cannot be extended to
omit retransmission and do flow control alone because of <X>."

I can't quite fill in that <X>, though.

> than PPPoE is normally used as such.

That doesn't answer the "why" question.  IP runs fine over Ethernet.

> adjacent radios.  The customer wants the router to run PPP on these 

It's that "semblance" that I think is a problem here.

> the goal is to couple the router as close as is reasonably possible to

I think that's an oversimplification in at least one area: if the
radio can speak Ethernet to each of the end nodes, why do you care
about having any mesh of "serial ports" overlaid on this topology?

IP already works on Ethernet, and just sending IP/Ethernet packets to
those remote destinations ought to work fine over any useful bridge.

What seems to be missing is that this device, although it tries to be
a bridge, just isn't a very good Ethernet bridge; it lacks the burst
handling and flow control characteristics that would make it good
enough to use.  I'm not sure that papering over that problem by
creating a complex web of additional layered protocols (plus a
smattering of *new* protocols that need to be implemented and tested)
is the right way to solve that problem.

> kludge - than PPPoE as it is normally deployed.  

That's the point where we disagree.  The way I read it, you've already
assumed that a mesh of point-to-point connections is the right
representation of this topology, and that PPPoE follows naturally from
that.  I agree that if that topology is part of the requirements
somehow, then this answer is _perhaps_ less awful than others, but I
don't see why it's actually required.

> it brings with it all the negative baggage associated with PPPoE.

No, I don't think that's the problem.

Plus some of the issues (such as the MTU problem) occur *BECAUSE* this
is the chosen solution, not because of any negative connotation of
PPPoE.

> connections over an ethernet (or any media) "backplane" are you 

I don't think that's the right question at all.  The official IETF
standard way of running IP over Ethernet, though, is documented in RFC
894.

> ability to hook these specialized radios into existing router equipment.

None of the existing PPPoE implementations support this credit scheme.

What to do about them?

> would augment already developed PPPoE implementations than implement a 

This *is* a whole new protocol.

I think you're arguing that they're already familiar with PPPoE and
are thus comfortable with hacking on that.  I'm not convinced that
this is really a good way to make architectural decisions about
networking protocols.

It's all too easy to run into the hammer problem: when all you have is
a hammer, every problem begins to look like a nail.

> makes no claim of trying to overcome.

I don't think it's off-topic at all.  It's right on topic for the
question of whether the proposed solution is solving the right
problem.

> and arises from misperceptions about the proposed mechanisms.

I don't think it's really all that inaccurate.  There are flow control
problems here directly due to the implementation choices.

> thereof) of this proposal or its intended goals.  My thought is that the 

While I'm no fan of PPPoE (having implemented it in Solaris), I don't
think this is true.  I just see wider issues here that aren't being
properly addressed.

-- 
James Carlson, KISS Network                    <[email-masked]>
Sun Microsystems / 1 Network Drive         71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

---

From: Karl Fox <[email-masked]>
Date: Mon, 1 Aug 2005 13:01:21 -0400

> It's that "semblance" that I think is a problem here.

Jim, I think you just hit the nail on the head--why does the customer  
want to simulate a serial connection?  And as you and Vernon have  
both asked, why interpose this troublesome serial emulation (which is  
ITSELF the source of the need for flow control) in the middle of a  
packet networking stack when the real service to the end user is  
packet delivery and the radios are themselves packet delivery devices?

Let me speculate: Is it because the customers are ISPs who started  
out with serial modems carrying PPP, and who now are so ossified by  
their legacy serial-centric back office systems that they want every  
new technology to look exactly like a serial modem?  Or is there a  
real technical need that I haven't grasped yet?

Karl

---

From: James Carlson <[email-masked]>
Date: Mon, 1 Aug 2005 13:15:13 -0400

> real technical need that I haven't grasped yet?

Even if the reason is the former, that's not a very good argument for
adding the flow control bits here.  (Why not add credits to the IP
header?  Or add special radio signal and battery life extensions to
TCP?)

The more benign answer (expressed by others so far) is that PPPoE was
an available solution that could be modified to work in this
environment.

The aspect that I don't like about it is the idea that this somehow
makes it "cheaper" to modify PPPoE to solve the problem.  I don't
think it does.  What does happen instead is that every vendor who has
begrudgingly added support for PPPoE ends up being told by customers
that this new "performance extension" documented in RFC XXXX is
'required' for deployment.  In other words, those who are completely
uninvolved in the local world of radio-based Ethernet bridges end up
tithing towards supporting that niche.

Stupid question time: why are these inherent radio-caused flow control
issues _not_ a problem for 802.11 and related technologies?  Why is it
that this one radio technology requires special flow control
considerations in *other* equipment outside of that domain, but 802.11
does seemingly the same task and doesn't require special support?

Or are these issues an unresolved (and until-now unrecognized) problem
with 802.11 that simply must be handled at higher levels?

---

From: Vernon Schryver <[email-masked]>
Date: Mon, 1 Aug 2005 11:50:57 -0600 (MDT)

> From: James Carlson <[email-masked]>
> To: Karl Fox <[email-masked]>
> Subject: Re: [Pppext] Review of draft-bberry-pppoe-credit (Informational)
> Cc: [email-masked], [email-masked], Fred Baker <[email-masked]>,
>         Vernon Schryver <[email-masked]>, [email-masked],
>         [email-masked], [email-masked], Mark Lanzo <[email-masked]>

> with 802.11 that simply must be handled at higher levels?

I don't think that's a stupid question, because I have one that really
is.  Exactly what bits are going over the air?  

  Is this about 802.11?  As far as I can tell, the string "802" does
     not appear in draft-bberry-pppoe-credit.

  If not, what other headers, trailers, whatever are involved?

  Are there PPP headers and trailers in the air, and if so, what flavors?

I also have a counterproposal.  Assume that traffic shaping of the
sort needed is not available with other IETF protocols.  (I wonder if
Fred Baker was right when he gave his "no" answer, because old style
IP TOS and less old diffserv don't sound that different.)  What is
special about PPPoE that makes credits and so forth need there but not
in other PPP flavors?  I think there are other palces where it could
be more valuable.  What about ABR ATM?--yes you could get a committed
bit rate but that might cost more and blah blah blah.

So why do as was done with the multi-link protocol and add another new
PPP sub-layer?  Or modify MP itself to have some control packets
that would do the crediting?  Or extend LCP with some new types?

That would be a more general solution to the particular problem.
It would also not burn more of the poor, withered PPPoE MTU.

---

From: Bo Berry <[email-masked]>
Date: Mon, 01 Aug 2005 14:03:33 -0400

in-line
-Bo

>      not appear in draft-bberry-pppoe-credit.
Correct, 802 is not in the draft.  In fact the extenstions are
independent of the radio technology.

The radio terminates the PPPoE session and forwards the PPP frames
over the airwaves.  The frames are transported by the radio
independently of the PPPoE.  The radio requirement is that it
can associate both ends of the radio channel, by whatever
means/technology is required in the radio.   The overall
requirement is that PPP flow be maintained, end to end.

---

From: James Carlson <[email-masked]>
Date: Mon, 1 Aug 2005 14:17:57 -0400

> >>with 802.11 that simply must be handled at higher levels?
[...]
> independent of the radio technology.

So, then, at least my stupid questions are still valid.  Why does this
draft provide a solution for something that's not generally recognized
as a problem with 802.11?  What has made it necessary in this
instance?  I'm missing something because it seems like the issues
ought to be more alike than not.

> requirement is that PPP flow be maintained, end to end.

That makes the requirements even less clear to me.  If this device is
essentially doing routing with a mesh of point-to-point links, why
doesn't it actually behave as a router?  Why is it instead a PPP
bridge that exposes internal implementation details (the operation of
the radio link) to other Ethernet devices?  And why is it dependent on
the use of PPPoE outside of this domain?

If it were used with something other than IP+PPP+PPPoE (i.e., bridging
arbitrary LANs together, perhaps running some 'foreign' protocol such
as MPLS), what happens?  Does every Ethernet-based protocol need an
upgrade to handle credit-based flow control so that this radio link
will work correctly?

---

From: Vernon Schryver <[email-masked]>
Date: Mon, 1 Aug 2005 12:22:21 -0600 (MDT)

> From: Bo Berry <[email-masked]>
> To: Vernon Schryver <[email-masked]>
> Cc: [email-masked], [email-masked], [email-masked], [email-masked],
>         [email-masked], [email-masked], [email-masked],
>         [email-masked]

(I've tried to trim that Cc: list to reduce duplicate copies.  It's probably
still too long)

> requirement is that PPP flow be maintained, end to end.

Ok.  So how does this requirement differ from the requirements intended
to be handled by by IP TOS or diffserv?  How does it differ from
   host  <- 802.3 ->  router  <-  ATM or FR PVC ->  host
or
   host  <- router (lots of other routers) router ->  host

Then there are the streaming protocols above IP.

> >   Are there PPP headers and trailers in the air, and if so, what flavors?

What is in the air, PPPoE, some other flavor of PPP, 802.11, and/or
something else entirely?  Maybe it doesn't really matter, but I have
the nagging suspicion that it does.

---

From: Bo Berry <[email-masked]>
Date: Mon, 01 Aug 2005 14:33:28 -0400

> upgrade to handle credit-based flow control so that this radio link

These extensions are optional and are not required by existing
PPPoE implementations.  Systems that require flow control
through (certain) radios could employ this solution.  We do
not want the router to overrun the radio buffers, nor do we
want the radio to overrun the router, hence the need for
flow control (via credits).

---

From: James Carlson <[email-masked]>
Date: Mon, 1 Aug 2005 14:50:51 -0400

> flow control (via credits).

I don't think that addresses the questions that have been raised.

First of all, "not required" doesn't match reality.  Either this is an
important niche for someone, and thus needs a solution like that
proposed in the draft, and thus will also force *many* vendors to
support it in their offerings (or suffer the consequences of inducing
very poor performance), _OR_ it's unimportant and doesn't need
solution.

Merely saying it's an "option" doesn't help if what's being specified
is something that (effectively) everyone is forced to implement.  What
I don't see is a rationale for why this feature would actually be
useful for others outside of this domain (some sort of redeeming path
forward) or something that constrains it to just a subset of equipment
vendors who all support this one market and never talk to anyone else.

Secondly, the question I asked above wasn't whether there was a need
for flow control, but rather what happens to protocols that are *not*
based on PPPoE.  In other words, this sounds like a very general
problem with Ethernet bridging in this particular device, but the
solution is an extension to PPPoE and not to any other Ethernet-based
protocol.  Does this mean that PPPoE becomes the only feasible
solution here, and is that really the right path forward for something
that otherwise appears to the rest of the network to be a bridge?

What stops some other Ethernet-based protocol from overrunning the
"router?"

Finally, why does this device just bridge PPP rather than route
packets?  What's wrong with routing?

---

From: Bo Berry <[email-masked]>
Date: Mon, 01 Aug 2005 15:11:27 -0400

> very poor performance), _OR_ it's unimportant and doesn't need

Sorry if I gave the impression that it is not important.
It is important, that's why it was submitted.

> vendors who all support this one market and never talk to anyone else.

We're not requiring that this be required by those
outside the domain.

> that otherwise appears to the rest of the network to be a bridge?

The radios we are currently working with require
flow control.  They are not radios that forward
any protocol.

> packets?  What's wrong with routing?

Routing is envolved, but outside of the PPPoE session. For
example, if the radio link is poor, the router may choose
another route.

---

From: Karl Fox <[email-masked]>
Date: Mon, 1 Aug 2005 16:45:49 -0400

> It is important, that's why it was submitted.

I would like to see an answer to the question that has been  
repeatedly asked but never answered:  Why is "serial emulation"  
necessary?

This group is a problem solving group; we are not interested in  
approving someone else's solution to a problem for which we have not  
been given all the relevant information.  We are tasked with coming  
up with the best solutions we can so that publicly promoted protocols  
with the IETF moniker attached reflect the good name of our  
organization and do not bear the aroma that so often accompanies  
industry alliance protocols.

Karl