Subject: Re: [Int-area] I-D Action: draft-shen-traceroute-ping-ext-03.txt

From: Thomas Narten <[email-masked]>
Date: Tue, 20 Dec 2011 14:01:32 -0500

I've been looking at this document recently.

Apologies for the length of this, but this is an important document
that I would like to see the community understand and weigh in on.

Let me see if I understand the basics of what is  being proposed.

The idea is to extend ICMP in such a way that it can provide more
specific or better info in response to certain error conditions. E.g.,
like when traceroute and ping are used.

Currently, the ICMP multipart extensions defined in RFC 4884 (Extended
ICMP to Support Multi-Part Messages) and RFC 5837 (Extending ICMP for
Interface and Next-Hop Identification) are limited to the following
ICMP response messages:

     o Time Expiry
     o Destination Unreachable
     o Parameter Problem

E.g., RFC 5837 allows the ICMP resonse message to include information
about the interfaces involved when the packet error was encountered.

draft-shen-traceroute-ping-ext goes much further, allowing the
inclusion of additional information in the message that *causes* an
ICMP response to be generated. That additional information is used to
signal (or 'request') what additional information to include in the
ICMP response.

The traceroute utility allows the sending of UDP or TCP messages with
increasing TTLs, so that traceroute works for arbitrary packets (as
opposed to working only for say, an ICMP echo request). Let's call
such messages "probe" messages, as they probe a network path and
result in some sort of ICMP error being returned.

In order to include additional information in an ICMP TTL exceeded for
such a UDP message, there has to be some way for the probe packet to
signal support for such extensions (but of course, there is no way to
do so in a random UDP or TCP messages, so there is no obvious way to
provide for such a signal).

This document proposes the following:

1) begin the packet with what looks like a normal TCP/UDP header

2) within the packet (past the TCP/UDP header), include a "magic
cookie" that signals this is a special packet that contains additional
ICMP extensions.

3) when generating ICMP errors, have the responder check for the magic
cookie, and if present, parse and process the included fields.

Section 4.1.1 shows the header containing the "magic cookie". There is
a 64-bit "trace-ping" common header that goes where the TCP/UDP data
would normally begin. That header can be followed by additional
information. For example, the document defines an "Information-Request
Object" that includes a Info Request bit-vector, defined as follows:

              Bit Number Information Item
                   0     MPLS label related attributes
                   1     Interface related attributes
                   2     IP/IPv6 address related attributes
                   3     Routing Instance related attributes
                   4     Nexthop(s) related attributes
                   5     Device role related attributes

Question: what does these bit numbers mean? I would assume that (say)
bit 3 means there are Routing Instance attributes include. Where are
these defined? How would they be used?

Second, the document defines a Trace-Ping Extension field. It gives
the offset within the paket where the extension data structures begin.

Question: why is this field needed at all? Why not just assume it
follows the fixed length header that holds the magic cookie?

Further, why is the field actually overlaid on the Source Port of the
TCP/UDP header? What is the motivation for that?

Thomas

---

From: Naiming Shen <[email-masked]>
Date: Tue, 20 Dec 2011 12:22:33 -0800

Hi Thomas,

Thanks for this detailed review, your understanding of the document is
excellent. Basically this document is to extend the capability for application
to include their own information in the traceroute "probe" packets and "ping"
packets for all types, including ICMP, TCP and UDP types. For example,
UDP traceroute is most popular one, but there is no generic mechanism
to include user data inside this UDP probe packet.

I'll move several of your questions below in the front, so folks don't have to

> these defined? How would they be used?

This document tries to pre-define some of the attributes we think the
network management might be interested, some of the bits are already
RFCs, such as MPLS and Interface related, the bit 3 is with this in mind:

  http://tools.ietf.org/html/draft-shen-icmp-routing-inst-00

others may not have been defined yet, such as device role. I would
think it's useful to return the icmp response like: "i'm the VM for location
tracking service with SQL database, CPU 80% busy"

> follows the fixed length header that holds the magic cookie?

I think there is two things here for this purpose. For the sender (probe) it
can put '0xf' in this "Off-Ext" field to indicate I have no extension in my
packet, so the receiver does not need to waste cpu cycle for every
TTL expired TCP/UDP packet, or ICMP request/reply packets.

Second, most of the current traceroute or ping applications already has
some "private" data at the beginning of the "data field", such as
time-stamp or other information. We are saying we don't have to change
those, we just skip those "private" data, and have a well-defined
data after that.

> TCP/UDP header? What is the motivation for that?

This is only because for the traceroute and ping applications. The source
port for those applications are a random number for tracking the
process-IDs, in case there is more than one user issuing the "traceroute"
application on the same host or router. So it's not really used for anything
important, and we here carve out 4-bits for this Extension-Offset field.
Again this is first for '0xf' in case we have no user data, and second, we
will skip the 'private' data to define where exactly the extension data is.

thanks.
- Naiming

---

From: Thomas Narten <[email-masked]>
Date: Tue, 20 Dec 2011 16:38:16 -0500

Hi Naiming.

> tracking service with SQL database, CPU 80% busy"

I would suggest that you reissue the draft with references to all the
defined bits.

Where there is no document, don't reserve the bits, as that is
premature.

Its hard for me to evaluate  this document without some specific
examples of how the proposed mechanism would actually be used.

Also, do you expect a packet to set more than one bit at the same
time? I'm a bit surprised to see a bit vector. Why not use an encoding
where the individual requested parameters are listed individualy?

Having a single 32-bit vector seems limiting - you can't have more
than 32 different types without defining some sort of extension
mechanism.

> TTL expired TCP/UDP packet, or ICMP request/reply packets.

But, isn't this information already implied by the Length field of the
Common Header?

> those, we just skip those "private" data, and have a well-defined

The document says that. Can you provide some detail/examples on what
current implementations already do? I.e., is this ID documenting what
(some) existing implementations already do? (If so, that is useful
information. I.e., what have they done, how well does it work, etc.)

> will skip the 'private' data to define where exactly the extension

If the source port of ping/traceroute applications already indicates
process IDs (in some applications) it seems like this spec should not
be saying "you can't do that anymore". I think PIDs are 16 bit fields
in some systems, so if you reserve some of the port bits for this
document, you are doing something that is not backwards compatable, it
seems to me.

Thomas

---

From: "George, Wes" <[email-masked]>
Date: Tue, 20 Dec 2011 16:52:51 -0500

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Naiming Shen
> Sent: Tuesday, December 20, 2011 3:23 PM
>
> I would

[WEG] No, absolutely not. We have multiple protocols to handle this information already, with significantly better granularity of information as well as access control permissions (who is allowed to query/receive this information), most notably SNMP. There is *zero* reason to try to bolt this sort of information into an ICMP datagram.
Please stick to information that is not available via alternate means if you are trying to justify its existence.

Thanks
Wes George

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

---

From: Naiming Shen <[email-masked]>
Date: Tue, 20 Dec 2011 14:38:36 -0800

Hi Thomas,

> Where there is no document, don't reserve the bits, as that is

Yes, we can do that.

> where the individual requested parameters are listed individualy?

I think the original intention is to have a combination of the things within
a single query packet.

> than 32 different types without defining some sort of extension

True, but this bitfield currently is quite broad, such as one bit to cover all
the mpls info. If this is really used up, then a new object can be further
defined.

> But, isn't this information already implied by the Length field of the

If there is a Common Header in the packet. I'm saying if in the "Ext-Off"
field we have the '0xf', this will indicate the probe packet does NOT include
this "trace-ping-extension" structure or data, no need to look and compute
for that. Otherwise, the receiver needs to do for every packet to see if 
the magic signature is there, and base on the common header length
and checksum to verify if this is really the extension data.

> information. I.e., what have they done, how well does it work, etc.)

Sure. But there are so many applications on various routers, OSes doing
their ways. Most often people put 8 bytes timestamp information there.
We can mention what the latest linux is doing though.

> document, you are doing something that is not backwards compatable, it

Not really.
Certainly the application (the sender) supports this need to change the
scheme. Since the sender modifies this will receive the same source-port
number back for the reply packet, there is no backwards compatable issue
here. this is not an inter-operability issue, it's a software implementation
issue of how to do their modification. No one else besides the sender
needs to know this information. Now instead of just match the pid, they
can do a partial pid (12-bits), plus the destination port (changing normally
for every packet); or even write their 16-bit pid in the "private" data area
at the beginning of the data field. Those are just implementation tricks.

thanks.
- Naiming

---

From: Naiming Shen <[email-masked]>
Date: Tue, 20 Dec 2011 14:41:33 -0800

Hi Wes,

Sure you can certainly use tools such as SNMP to query on those info,
the same thing can be said using SNMP to query on MPLS and interface
related information also. Although this is not only for one host, this is generally
used by Traceroute application when traverse a list of hosts/routers.

thanks.
- Naiming

---

From: "Tissa Senevirathne (tsenevir)" <[email-masked]>
Date: Tue, 20 Dec 2011 14:56:19 -0800

Some of the ICMP extensions in question e.g. "my link VM is x" are not
to replace SNMP or such, but provide an integrated framework to retrieve
critical debug information when troubleshooting a network problem. In
other words to reduce the need for the person troubleshooting the
network to jump back and forth between CLI, SNMP and other applications.

I think this draft provide some creative forward looking extensions and
it would be great to see more traction at the WG level.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of George, Wes
Sent: Tuesday, December 20, 2011 1:53 PM
To: Naiming Shen (naiming); Thomas Narten
Cc: Carlos Pignataro (cpignata); Enke Chen (enkechen);
[email-masked]; [email-masked]
Subject: Re: [Int-area] I-D Action:
draft-shen-traceroute-ping-ext-03.txt

> tracking service with SQL database, CPU 80% busy"

[WEG] No, absolutely not. We have multiple protocols to handle this
information already, with significantly better granularity of
information as well as access control permissions (who is allowed to
query/receive this information), most notably SNMP. There is *zero*
reason to try to bolt this sort of information into an ICMP datagram.
Please stick to information that is not available via alternate means if
you are trying to justify its existence.

Thanks
Wes George

This E-mail and any of its attachments may contain Time Warner Cable
proprietary information, which is privileged, confidential, or subject
to copyright belonging to Time Warner Cable. This E-mail is intended
solely for the use of the individual or entity to which it is addressed.
If you are not the intended recipient of this E-mail, you are hereby
notified that any dissemination, distribution, copying, or action taken
in relation to the contents of and attachments to this E-mail is
strictly prohibited and may be unlawful. If you have received this
E-mail in error, please notify the sender immediately and permanently
delete the original and any copy of this E-mail and any printout.
_______________________________________________
Int-area mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/int-area

---

From: Naiming Shen <[email-masked]>
Date: Wed, 21 Dec 2011 14:34:42 -0800

BTW, this issue of pid to source-port mapping for traceroute and ping hack
is not just raised by this draft, in the current Linux implementation the maximum
pid is a 32bit value, one can use:

 "sysctl kernel.pid_max = xxxxxxx"

to set to any value of an integer. thus the application using those pid hacks needs
to be modified if absolute deterministic mapping is needed regardless.

thanks.
- Naiming

---

From: Jared Mauch <[email-masked]>
Date: Wed, 21 Dec 2011 17:57:01 -0500

The idea that network operators will allow someone outside their administrative control have this level of visibility is interesting but unworkable. 

Those who wish to reveal this data can do so via other means such as public looking glasses and graphs. The major networks will not reveal this. 

Jared Mauch

---

From: Naiming Shen <[email-masked]>
Date: Wed, 21 Dec 2011 15:02:27 -0800

Hi Jared,

I completely agree. That is one of the reasons this draft has the authentication object also defined.
The source of the query has to be a trusted host in order to reveal network/enterprise private
information from the responder.

thanks.
- Naiming

---

From: Thomas Narten <[email-masked]>
Date: Wed, 21 Dec 2011 18:12:27 -0500

> True, but this bitfield currently is quite broad, such as one bit to
> cover all the mpls info. If this is really used up, then a new
> object can be further defined.

Is it really as simple as saying "all mpls info"? I'd think that for
this kind of mechanism to be useful, it would need to be fine
grained. Be specific in what information is requested, and not provide
"all mpls info" (whatever that means). The kind of information that
would be interesting presumably varies depending on what is being
debugged...

> and checksum to verify if this is really the extension data.

Having 2 ways to check for the same things does not seem like the best
way to do things. And checking for the magic cookie (if in a fixed
place, easily located) seems easy to me.

> We can mention what the latest linux is doing though.

I'd really like to have an explanation of some examples of what is
being done today. Not necessarily in the draft. Right now, I have no
idea of what is done today and whether and why. Without that, I can't
evaluate whether allowing for such private information  is useful.

Pick 2 or 3 examples of what vendors have done.

Thanks!

Thomas

---

From: Thomas Narten <[email-masked]>
Date: Wed, 21 Dec 2011 18:17:04 -0500

Naiming Shen <[email-masked]> writes:

> I completely agree. That is one of the reasons this draft has the
> authentication object also defined.  The source of the query has to
> be a trusted host in order to reveal network/enterprise private

I'm pretty skeptical that the authentication object is a viable
approach.

Yes, technically it can work. But the idea that before a router (or
device) processes the packet it needs to do some sort of
authentication check smacks of making it much more expensive to
process such packets. And for debugging/OAM, you can't have the
per-packet processing cost be high.

We're talking about stuff like ping and traceroute here...

Thomas

---

From: Alia Atlas <[email-masked]>
Date: Wed, 21 Dec 2011 18:36:24 -0500

The router can process the authentication object before deciding to put the
sensitive information in there.  If there is too much load to do that,
then the router
can return the non-sensitive information and an error object, if we
need to do that.

Alia

---

From: Naiming Shen <[email-masked]>
Date: Wed, 21 Dec 2011 15:41:49 -0800

And this is up to the responder, if it wants to give out sensitive information or not.
It can choose to or being configured to just give out vanilla traceroute and ping
response, or it can be configured to respond to any request because it's in a close
environment without the authentication; or it can be configured to only care about
authentication if the source is from a network management subnet. In the rfc of
icmp interface related document, it has the interface name, someone might think
this is sensitive info and only give out this info if it's a trusted request. This authentication
is just one option, have an acl on checking the source network is another of course.

thanks.
- Naiming

---

From: Jared Mauch <[email-masked]>
Date: Wed, 21 Dec 2011 19:47:35 -0500

Problem is many devices can't handle any significant number of packets that go to the control plane. I am seeing only risk to anyone who deployed code supporting this capability. Typically the expense of checks end up being more CPU intensive than processing the packet. 

Jared Mauch

---

From: Jared Mauch <[email-masked]>
Date: Wed, 21 Dec 2011 19:49:17 -0500

The data is available via alternate means. Not sure why encoding it in an exception packet is necessary. 

Jared Mauch

---

From: Naiming Shen <[email-masked]>
Date: Wed, 21 Dec 2011 16:51:50 -0800

Hi Thomas,

> would be interesting presumably varies depending on what is being

In today's traceroute reply, it dumps everything, so those bits at least
define some categories to ask a subset. If later we want to have very
detailed description, those bits are useless I agree; then people want
to do similar thing as TRILL guys doing, define their own information
using their own objects.

> place, easily located) seems easy to me.

Think I didn't express my point clearly. I'm saying if the sender does not
have anything special to request, in other words, the sender just want to
use the traceroute/ping without this extension, then it only set a flag in the
ICMP header for this '0xf', it does not need to put the Common Header
in the ICMP data field, then have the length is equal to the size of the
Common Header only (without any object included). I would assume most
of the simple ping and traceroute, they only need to have vanilla/default
information, this way we save the responder some cpu cycles.

> Pick 2 or 3 examples of what vendors have done.

Sure, I just goggled the bsd and linux implementation for traceroute.c

in bsd, it defines a union for this 'private' data:

union
  {
    struct
    {
      u_int32_t its_otime;
      u_int32_t its_rtime;
      u_int32_t its_ttime;
    } id_ts;
    struct
    {
      struct ip idi_ip;
      /* options and then 64 bits of data */
    } id_ip;
    struct icmp_ra_addr id_radv;
    u_int32_t   id_mask;
    u_int8_t    id_data[1];
  } icmp_dun;

only timestamp is stuffed in the traceroute packet:

        gettimeofday((struct timeval *)icmp->icmp_data, NULL);

while in Linux, it's simpler, only 3 items defined:

typedef struct outdata {
        unsigned char seq;             /* sequence number of this packet */
        unsigned char ttl;             /* ttl packet left with */
// UNUSED. Retaining to have the same packet size.
        struct timeval tv_UNUSED PACKED; /* time packet left */
} outdata_t;

only seq and ttl is saved in this 'private' data space, but timestamp is skipped
and commented out:

        /* Payload */
        outdata->seq = seq;
        outdata->ttl = ttl;
// UNUSED: was storing gettimeofday's result there, but never ever checked it
        /*memcpy(&outdata->tv, tp, sizeof(outdata->tv));*/

thanks.
- Naiming

---

From: Naiming Shen <[email-masked]>
Date: Wed, 21 Dec 2011 17:14:23 -0800

I think due to traceroute and ping are so commonly used, and take an example of the mpls label stack info,
we could have first to fire a normal traceroute, have a perl script to access each router on the path to use
xml/snmp/telnet and get the label information from the link of the ip address identifies, and return the
result on each hop. But to allow the mpls info to be piggy-backed from the exception packet helps
a lot during the troubleshoot also.

thanks.
- Naiming

---

From: "Tissa Senevirathne (tsenevir)" <[email-masked]>
Date: Wed, 21 Dec 2011 18:45:47 -0800

I agree with Alia. 

Also, do not forget, the authentication object specified here is similar to what is used in OSPF. If routers can do that for OSPF, they certainly can do this for Ping. If they cannot we can fall back to default information and return additional error code to say authentication failure.

In addition to the authentication, this draft has lots of other good things such as ability to add additional extensions to ICMP messages that are non compliant to RFC 4884. Which is the number rone strength of this draft.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Alia Atlas
Sent: Wednesday, December 21, 2011 3:36 PM
To: Thomas Narten
Cc: Carlos Pignataro (cpignata); Enke Chen (enkechen); [email-masked]; [email-masked]
Subject: Re: [Int-area] I-D Action: draft-shen-traceroute-ping-ext-03.txt

Alia

---

From: "Tissa Senevirathne (tsenevir)" <[email-masked]>
Date: Wed, 21 Dec 2011 18:49:25 -0800

Yes. But the problem is you have to switch back and forth between
different applications (SNMP browsers, Network Management applications
etc..) to get the information. With integrated ICMP Ping, you should be
able to get interested data as part of the ping output.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Jared Mauch
Sent: Wednesday, December 21, 2011 4:49 PM
To: Naiming Shen (naiming)
Cc: Thomas Narten; Carlos Pignataro (cpignata); [email-masked];
[email-masked]; Enke Chen (enkechen)
Subject: Re: [Int-area] I-D Action:
draft-shen-traceroute-ping-ext-03.txt

The data is available via alternate means. Not sure why encoding it in
an exception packet is necessary. 

Jared Mauch

> I completely agree. That is one of the reasons this draft has the
authentication object also defined.
>> The idea that network operators will allow someone outside their
administrative control have this level of visibility is interesting but
unworkable. 
>> 
>> Those who wish to reveal this data can do so via other means such as
public looking glasses and graphs. The major networks will not reveal
this. 
>>>>> think it's useful to return the icmp response like: "i'm the VM
for
>>>>> location
>>>> Please stick to information that is not available via alternate
means if you are trying to justify its existence.
>>>> 
>>>> Thanks
>>>> Wes George
>>>> 
>>>> This E-mail and any of its attachments may contain Time Warner
Cable proprietary information, which is privileged, confidential, or
subject to copyright belonging to Time Warner Cable. This E-mail is
intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient of this E-mail, you are
hereby notified that any dissemination, distribution, copying, or action
taken in relation to the contents of and attachments to this E-mail is
strictly prohibited and may be unlawful. If you have received this
E-mail in error, please notify the sender immediately and permanently
delete the original and any copy of this E-mail and any printout.

---

From: Naiming Shen <[email-masked]>
Date: Wed, 21 Dec 2011 21:31:20 -0800

The packets punting to the control plane have to be rate limited for normal operation or under
DOS attack, some devices already have error exception packet handling on the forwarding
plane instead of punting to upstairs, many forwarding plane has general CPUs and we'll see
more applications moving downstairs actually. But yes, agree with Alia that if it hits the throttle
point, it should send a vanilla reply back with optionally an error object attached.

thanks.
- Naiming

---

From: Jared Mauch <[email-masked]>
Date: Thu, 22 Dec 2011 06:33:10 -0500

I think you are overestimating the ability of the routers to respond.

Cisco and Juniper have so far failed in many ways that have created pain for operators over the years when it comes to processing packets directed to the device.  Developers see these as vulnerabilities and have moved to rate-limit packets directed at the control-plane, either by processing on the FPC, DFC, Linecard CPU, or the centralized processor.

The experiences I've had demonstrate that platforms are underpowered for processing these packets and leave the system vulnerable to resource exhaustion attacks, even with CPP.  If you want to take the discussion off-line I can provide you a number of examples where platforms have gotten this wrong despite the experiences already learned by other groups inside and outside your respective employers.

Choosing to ignore these hard learned lessons leads us down a dim path for the future.

Additionally, nothing is stopping someone from developing proper management software that processes the TTL expiry packets and mapping that with the various enterprise mibs for the device.  Think of a version of MTR that sends a series of SNMP queries based on these messages and can be configured to understand the scope of ones network, e.g.:

snmp_poll {
  address-space 3ffe:2001::/48 {
    snmp-community public;
  }
}

This combined with existing shipping tools/software would keep excessive traffic from degrading the device.

- Jared

* MTR == http://www.bitwizard.nl/mtr/

---

From: Jared Mauch <[email-masked]>
Date: Thu, 22 Dec 2011 06:46:28 -0500

> able to get interested data as part of the ping output.

I don't think that you actually meant to say SHOULD there.

this is proposing taking the SNMP browsers/Network Management applications
and pushing them into an ICMP reply based on an arbitrary set of authentication
criteria that makes responding in hardware (e.g.: Juniper does this on the FPC)
harder or impossible, requiring additional IPC within a distributed platform
to populate the data.

If I didn't experience crashes from vendors having a hard time processing 20+
year old regular IPv4 options this year, I may consider giving you access to
add complexity to my distributed platforms, but adding complexity for the sake
of complexity here seems to be the wrong path forward.  Improving the network
management software and tools would make operators life better vs adding
complexity.  (And this is a SIRT issue we experienced in Aug 2011 that is
not yet public, responding to a TTL expiry).

Call me biased against this complexity, but when your routers die because
of a well-formed packet processing defect, adding more unused capabilities
to the code is unwanted by network operators.

- Jared

---

From: Jared Mauch <[email-masked]>
Date: Thu, 22 Dec 2011 07:00:10 -0500

Naiming,

When routers are under attack in the real world, it is better to just not reply vs fail-back to a vanilla process.

A colleague at another operator observed that when devices were under attack the NOC lost monitoring of the devices due to the policers involved (since they looked primarily at ICMP replies) yet the device would continue to forward packets.

Have you talked to large network security teams about their operational experiences in this area?

- Jared

---

From: "Tissa Senevirathne (tsenevir)" <[email-masked]>
Date: Thu, 22 Dec 2011 08:49:51 -0800

Hi Jared

It appears your objections are for authentication part. This is an
optional element. Having said that, the methods used for authentication
are similar to that used for authentication of routing protocols such as
OSPF, so it is not any different than that.

Secondly proposed approach is not meant to replace SNMP as such, It
meant for folks to obtain some vital information through the same Ping
console to improve the productivity efficiency.

IESG anyways have stressed the importance of security. Having thought
about including security even if it is optional is the right direction
by the authors. We need to congratulate them.

Thanks
Tissa

-----Original Message-----
From: Jared Mauch [mailto:[email-masked]] 
Sent: Thursday, December 22, 2011 3:46 AM
To: Tissa Senevirathne (tsenevir)
Cc: Naiming Shen (naiming); Thomas Narten; Carlos Pignataro (cpignata);
[email-masked]; [email-masked]; Enke Chen (enkechen)
Subject: Re: [Int-area] I-D Action:
draft-shen-traceroute-ping-ext-03.txt

this is proposing taking the SNMP browsers/Network Management
applications
and pushing them into an ICMP reply based on an arbitrary set of
authentication
criteria that makes responding in hardware (e.g.: Juniper does this on
the FPC)
harder or impossible, requiring additional IPC within a distributed
platform
to populate the data.

If I didn't experience crashes from vendors having a hard time
processing 20+
year old regular IPv4 options this year, I may consider giving you
access to
add complexity to my distributed platforms, but adding complexity for
the sake
of complexity here seems to be the wrong path forward.  Improving the
network
management software and tools would make operators life better vs adding
complexity.  (And this is a SIRT issue we experienced in Aug 2011 that
is
not yet public, responding to a TTL expiry).

Call me biased against this complexity, but when your routers die
because
of a well-formed packet processing defect, adding more unused
capabilities
to the code is unwanted by network operators.

- Jared

---

From: Naiming Shen <[email-masked]>
Date: Thu, 22 Dec 2011 11:27:17 -0800

Jared,

When we say replying we really meant rate limited replying of course. When we first noticed
we are overwhelmed we should send a message, but should be only once a while. I understand
your point of devices are not designed perfectly to deal with them. And people can use vanilla
traceroute and ping to attack also, they don't have to utilize our extension for those purpose
in particular. I actually have not talked to large network security team recently, although I used to
manage tier 1 IP network myself so I have some idea for the ISP operations.

thanks.
- Naiming

---

From: Naiming Shen <[email-masked]>
Date: Thu, 22 Dec 2011 11:38:25 -0800

Hi folks,

Another option is that we can take out the authentication and information-request
objects out of this document. Let this document to be only the trace-ping extension
framework document, but does not specify any specific object (similar to rfc 4884
is only the framework for other icmp extensions of mpls, interface, etc).

thanks.
- Naiming

---

From: "Tissa Senevirathne (tsenevir)" <[email-masked]>
Date: Thu, 22 Dec 2011 11:56:17 -0800

Hi Naiming

My opinion is to keep authentication as an optional element. Especially
given the fact that we need to address security considerations. Also
with enhanced ICMP extensions we are exposing more information.

Also in some applications such as VM, where the same Ping can be used by
both customers and the provider. May be the information provided in
response can be different based on the originator credentials.

Thanks
Tissa

-----Original Message-----
From: Naiming Shen (naiming) 
Sent: Thursday, December 22, 2011 11:38 AM
To: Tissa Senevirathne (tsenevir); Jared Mauch; Thomas Narten
Cc: Carlos Pignataro (cpignata); [email-masked]; [email-masked];
Enke Chen (enkechen)

Hi folks,

Another option is that we can take out the authentication and
information-request
objects out of this document. Let this document to be only the
trace-ping extension
framework document, but does not specify any specific object (similar to
rfc 4884
is only the framework for other icmp extensions of mpls, interface,
etc).

thanks.
- Naiming

---

From: Naiming Shen <[email-masked]>
Date: Thu, 22 Dec 2011 12:10:59 -0800

Hi Tissa,

I agree with you the authentication is important, in particular in the time of
Cloud and SDN development for the Internet and data-centers. But since
it's optional, I meant to say to document the authentication and other useful
objects in a separate document. Then we can concentrate on at least the
framework portion to make sure they are right.

thanks.
- Naiming

---

From: Carlos Pignataro <[email-masked]>
Date: Thu, 22 Dec 2011 17:23:54 -0500

Hello, Thomas,

> that I would like to see the community understand and weigh in on.

Thanks for this. You provided a critical review and understood the
mechanism well.

> would be interesting presumably varies depending on what is being

I wonder if it would make sense here, instead of having a bit-vector, to
include a list of all the Class-Num and C-Types that are expected to be
returned.

So for example, specifying in the probe a Class-Num = 1 and C-Types = 1
would mean return the "MPLS Label Stack Object". But this gives the
granularity that you alluded to, as well as flexibility, and
forward-compatibility. So continuing the example, adding Class-Num of 2
would mean "some Interface Information Objects", and the C-Types can,
with finer granularity, specify what exactly: C-Type = 00001010b would
mean "Incoming Interface: By ifIndex and Name".

> place, easily located) seems easy to me.

This would certainly simplify a bit. Having the Extension offset gives a
bit more flexibility, though might not be needed for its cost?

> evaluate whether allowing for such private information  is useful.

In my experience, the most typical example is traceroute-nanog or
TrACESroute that include a timestamp in the body.

As a datapoint, can see
<http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf#page=9>
(URI Fragment pointing to page 9 that says):

"• Many implementations encode the original launch timestamp
into the probe packet, to increase accuracy and reduce state"

Or code like this:

    if (ttl == -1)
        ip->ip_ttl = 0;
    else
        ip->ip_ttl = ttl;
    ip->ip_v = IP_VERSION;
    ip->ip_hl = sizeof(*ip) >> 2;

    if (proto == IPPROTO_UDP) {
        up = &op->ip_payload.udp_probe.udp;
        up->uh_sport = htons(ident);
        up->uh_dport = htons(port + seq);
        up->uh_ulen = htons((u_short) (datalen - sizeof(struct ip) -
_optlen));
        op->ip_payload.udp_probe.seq = seq;
        op->ip_payload.udp_probe.ttl = ttl;

#ifndef __linux__
        (void) gettimeofday(&op->ip_payload.udp_probe.tv, NULL);
#else                           /* __linux__ */
        (void) gettimeofday(&op->ip_payload.udp_probe.tv, &tz);
#endif                          /* __linux__ */

> Pick 2 or 3 examples of what vendors have done.

Many Thanks,

-- Carlos.

---

From: Carlos Pignataro <[email-masked]>
Date: Fri, 23 Dec 2011 10:02:08 -0500

Hello, Jared,

Thanks much for the comments, please find some comments inline.

On 12/21/2011 5:57 PM, Jared Mauch wrote:
> administrative control have this level of visibility is interesting
> but unworkable.

Have you read the Security Considerations section? The first paragraph
reads:

http://tools.ietf.org/html/draft-shen-traceroute-ping-ext-03#section-6

6.  Security Considerations

   This extension enhances the security of traceroute and ping operation
   in a backwards-compatible fashion.  The mechanism allows the receiver
   to verify the sender of the traceroute/ping packet such that certain
   sensitive application, interface and network related information can
   be supplied in the internal network or across trusted networks.

Additionally, the existing framework of RFC 4884 + associated extensions
(e.g., RFC 4950 and RFC 5837) already address this point:

http://tools.ietf.org/html/rfc4950#section-6

   interaction between MPLS and ICMP.  However, this document defines an
   extension that allows an MPLS router to append MPLS information to
   multi-part ICMP messages, and therefore can provide the user of the
   TRACEROUTE application with additional information.  Consequently, a
   network operator may wish to provide this information selectively
   based on some policy; for example, only include the MPLS extensions
   in ICMP messages destined to addresses within the network management
   blocks with administrative control over the router.  An

http://tools.ietf.org/html/rfc5837#section-6

   It may be desirable to provide this information to a particular
   network's operators and not to others.  If such policy controls are
   desirable, then an implementation could determine what sub-objects to
   include based upon the destination IP address of the ICMP message
   that will contain the sub-objects.  The implementation of policy
   controls could also be based upon the mechanisms described in
   [TRACEROUTE-EXT] for those limited cases supported.

The authentication in this I-D attempts to make it easier and more scalable.

> per-packet processing cost be high.

And I agree with Alia in that this check could be performed only before
including extra info.

> public looking glasses and graphs. The major networks will not reveal

My personal experience when troubleshooting, which is not a universal
statement but a single datapoint, is that having information in my CLI
in my workflow in my traceroute was most useful. Graphs are not too
readable when on a CLI only.

And since much of this data is spec'ed on a reply already (e.g., RFC
4950 and RFC 5837) we are proposing to augment the framework to have
more control and security on the request side.

Thanks,

-- Carlos.

>> Sure you can certainly use tools such as SNMP to query on those
>> info, the same thing can be said using SNMP to query on MPLS and
>> interface related information also. Although this is not only for
>> one host, this is generally used by Traceroute application when
>> traverse a list of hosts/routers.
>> 
>> thanks. - Naiming
>> 
>>>> From: [email-masked]
>>>> [mailto:[email-masked]] On Behalf Of Naiming Shen 
>>>> I would think it's useful to return the icmp response like:
>>>> "i'm the VM for location tracking service with SQL database,
>>>> CPU 80% busy"
>>>> 
>>> 
>>> [WEG] No, absolutely not. We have multiple protocols to handle
>>> this information already, with significantly better granularity
>>> of information as well as access control permissions (who is
>>> allowed to query/receive this information), most notably SNMP.
>>> There is *zero* reason to try to bolt this sort of information
>>> into an ICMP datagram. Please stick to information that is not
>>> available via alternate means if you are trying to justify its
>>> existence.
>>> 
>>> Thanks Wes George
>>> 
>>> Cable proprietary information, which is privileged, confidential,
>>> or subject to copyright belonging to Time Warner Cable. This
>>> E-mail is intended solely for the use of the individual or entity
>>> to which it is addressed. If you are not the intended recipient
>>> of this E-mail, you are hereby notified that any dissemination,
>>> distribution, copying, or action taken in relation to the
>>> contents of and attachments to this E-mail is strictly prohibited
>>> and may be unlawful. If you have received this E-mail in error,
>>> please notify the sender immediately and permanently delete the
>>> original and any copy of this E-mail and any printout.
>> 
>> _______________________________________________ Int-area mailing
>> list [email-masked]

---

From: Carlos Pignataro <[email-masked]>
Date: Fri, 23 Dec 2011 10:02:47 -0500

Jared,

This I-D is not proposing an operational security feature. These generic
comments are very interesting but perhaps better fitting Opsec; RFC 6192
<http://tools.ietf.org/html/rfc6192> also comes to mind.

What's your take on
<http://tools.ietf.org/html/draft-shen-traceroute-ping-ext-03#section-6>?

Thanks,

-- Carlos.

---

From: Carlos Pignataro <[email-masked]>
Date: Fri, 23 Dec 2011 10:03:34 -0500

Hello, Jared,

Thanks again, please see inline.

> in an exception packet is necessary.

This is from my home, right now:
# traceroute puck.nether.net
<snip>
 9  ae-6.r20.asbnva02.us.bb.gin.ntt.net (129.250.3.109)  25 ms  31 ms  28 ms
    MPLS Label=445248 Exp=0 TTL=1 S=1

>   address-space 3ffe:2001::/48 {

MTR hits the routers with a lot more packets, no? After a traceroute it
starts to ping, I understand. Just to be clear, do you mean that
processing a set of snmp polls is less processor intensive?

> This combined with existing shipping tools/software would keep excessive traffic from degrading the device.

The challenge is that, often times, troubleshooters do not have access
to a full-blown NMS. Or the NMS does not have access to the devices.
Or... This is embedding a response in the traceroute path.

Thanks,

-- Carlos.

---

From: Carlos Pignataro <[email-masked]>
Date: Fri, 23 Dec 2011 10:05:22 -0500

Hi, Naiming,

I think this a terrific idea. Keep the TLV-less framework only like RFC
4884.

Others?

Thanks,

-- Carlos.

---

From: Joel jaeggli <[email-masked]>
Date: Sat, 24 Dec 2011 10:28:13 -0800

> information from the responder.

So, something targeted through the forwarding plane that filters up to
the control plane will be filtered first either by source address or
passed through a rate limiter or both because those are the protections
we have that actually scale. Authentication increases the vulnerability
to some kinds of abuse rather that decreasing it.

---

From: Jared Mauch <[email-masked]>
Date: Tue, 27 Dec 2011 19:38:46 -0500

Happy Holidays!

> Have you read the Security Considerations section? The first paragraph

I have not, because any additional complexity in the distributed forwarding plane
or centralized cpu utilization is undesirable in SP operations..

Any code where you perform additional checks prior to replying adds load to the
device and with minimal cpu provided on line cards, this is my main problem.

Authentication and other mechanisms take away valuable microcode space and
increase the complexity.  

Tier-1 networks have likely evolved significantly since Naiming was in the
space.  A single network (Level3) today announces about 50% of the DFZ rib on
a peering session.  When this interface goes down, the time spent
processing the rib -> fib -> lc events can be significant on a platform.

A commonly used platform can only populate about 8k entries per second under
load, making a 150k prefix change take 18 seconds or more.  This is a significant
amount of time.  Any IPC contention during this time results in further 
customer observed downtime.

These are operational challenges that are faced today with modern hardware.
I'm concerned that the innovation provided by this draft is unnecessary for
the network operator and will place undue load upon the network.

- Jared

---

From: Jared Mauch <[email-masked]>
Date: Tue, 27 Dec 2011 19:52:15 -0500

> <http://tools.ietf.org/html/draft-shen-traceroute-ping-ext-03#section-6>?

Carlos,

Including the text for reference:

-- snip --
   A router needs to protect against purposefully-bogus Traceroute
   packets with extensions that fail the authentication, as a high rate
   of messages can require significant processing time.  [
RFC1812
]
   specifies how rate-limiting is applied to the generation of ICMP
   messages, and this rate-limiting deters the threat when applied
   before checking the Authentication.  Additionally, when using
   Cryptographic authentication, the HMAC includes the source IP
   address, which means the HMAC will not validate if the traceroute/
   ping packet is sent over a NAT.
-- snip --

Traceroute packets (ie: ICMP Exception of TTL Expiry).  If the authentication is provided in hardware prior to the application of rate-limiting, this can be used to attack the limited CPU resources in a line card.  If rate-limited after the authentication, which is more expensive, replying or performing the authentication?  While this is an implementation detail, it's important to measure the inherent cost.  Sometimes it is more expensive to authenticate vs "just reply".

Has this cost been measured in the network elements?  Reducing performance of the network hardware is not a desired outcome, and a major concern of mine here.

I'm also concerned that referencing rfc1812 4.3.2.8 could be overly simplistic in talking about rate-limits involved, are you specifically talking about the Cisco implementation referenced here:

http://www.cisco.com/en/US/docs/ios/ipv6/configuration/guide/ip6-addrg_bsc_con.html#wp1373663

-- snip --
In Cisco IOS Release 12.2(8)T or later releases, the IPv6 ICMP rate limiting feature implements a token bucket algorithm for limiting the rate at which IPv6 ICMP error messages are sent out on the network. The initial implementation of IPv6 ICMP rate limiting defined a fixed interval between error messages, but some applications, such as traceroute, often require replies to a group of requests sent in rapid succession. The fixed interval between error messages is not flexible enough to work with applications such as traceroute and can cause the application to fail.
-- snip --

Where will this rate-limiting be implemented?  Hardware?  Software?  Both?  The 7600 and 6500 platform (for example) have a hardware based rate-limiter on a per-lc basis, but the traffic that is punted to the cpu is the aggregate across all linecards and must be policed a second time before being processed for the punt/exception cases.  Processing this policy in actual implementations has proved to be more expensive than "just replying" in operations.

I'm concerned that without implementation requirements we will see troubles in the future with this.

I'm also reminded of Solaris 2.6 and the 500ms rate-limit it provided to TTL expiry handling that confused network and sysadmins to think the host may be experiencing loss.  The traceroute processing would show the 1st and 3rd packets fine, but the 2nd would be dropped due to rate-limit.

- Jared

---

From: Jared Mauch <[email-masked]>
Date: Tue, 27 Dec 2011 19:59:36 -0500

> Thanks again, please see inline.

Sure,

>    MPLS Label=445248 Exp=0 TTL=1 S=1

Yes, I'm unaware of a way to exclude this information from replies, but this data
is already there on the FPC in the local FIB.  Placing the HMAC and other crypto
in the FIB path seems problematic and would increase the cost of replying.

There was also the case (around JunOS 5.5 iirc) they implemented a hard-limit of
icmp exception handling that caused numerous issues with clueless network admins
thinking they were observing packet loss.  The optics of this issue does matter
to operators, even if the forwarding plane is minimally impacted.

> processing a set of snmp polls is less processor intensive?

Certainly not.  Anyone spending time with ASN.1 will know it's very expensive.
(I used to own one of the internal SNMP tools at a major carrier).

But providing the information to an authorized set of people (network operators/NOC
would seem to be more appropriate than possibly exposing private data that may be
considered proprietary to ones network).

> Or... This is embedding a response in the traceroute path.

I think trying to solve the trouble of operators without access to manage their
equipment via traceroute extensions is the wrong way to approach the problem. 

This sounds more like a business issue.  If the people doing the troubleshooting
don't have access, then there are many jobs available today for qualified network
people.  Natural selection and market forces will work against those carriers
without the skill/access to resolve their troubles.

- Jared

---

From: Jared Mauch <[email-masked]>
Date: Tue, 27 Dec 2011 20:03:22 -0500

Joel,

> to some kinds of abuse rather that decreasing it.

Thanks.  I think I mentioned this in another message, but captures an important
concern about the network elements being managed.  Implementation details matter.

- Jared