Subject: joe, turn it off!

From: Randy Bush <[email-masked]>
Date: Mon, 15 Jul 2002 13:19:58 +0900

joe touch and crew, please turn it off

Jul 15 13:01:45 roam /kernel: arp: unknown hardware address format (0x0800)

---

From: Joe Touch <[email-masked]>
Date: Sun, 14 Jul 2002 21:38:12 -0700

> Jul 15 13:01:45 roam /kernel: arp: unknown hardware address format (0x0800)

As I said at 10:30, and will repeat, this is the result of an arp cache 
that hasn't been flushed. An arp cache that we don't control, most 
likely on a router that has decided to proxy arp.

Randy - if you have evidence to the contrary, we've been waiting since 
10:30.

Joe

---

From: Randy Bush <[email-masked]>
Date: Mon, 15 Jul 2002 13:45:19 +0900

> Randy - if you have evidence to the contrary, we've been waiting since 

this is an engineering group.  we are seeing problems on a production
network.  your experiment has been shown to once be part of the problem.
please turn from lawyer into an engineer and turn that thing off so we
can see if part of the problems go away

from draft-ymbk-termroom-op-07.txt

       While it is tempting for a host/vendor to show off fancy
       technology at an IETF, this audience runs and uses the most
       arcane assortment of services, and is a very poor place to find
       out that your fancy new switch breaks when someone tries to run
       IPv42 through it.  Run a simple production network.  If one must
       run a technology demo, isolate it onto a separate network segment
       so that it is unable to interfere with the production network.

randy

---

From: Joe Touch <[email-masked]>
Date: Sun, 14 Jul 2002 21:53:34 -0700

> can see if part of the problems go away

We did (at 9:30am, minutes after the problem was detected). It didn't go 
away. Now be an engineer and show us a trace.

>        technology at an IETF,

The routers have a bug in their implementation of ARP. Yes, we tripped 
it, at 9:45am. We have no idea who is still jabbering. It is definitely 
possible that our 9:45am bug tripped the jabbering of our IP address by 
one of the routers. They are not ours to shut off, however.

ARP, and our use of it, isn't experimental. We aren't running anything now.

Joe

---

From: Randy Bush <[email-masked]>
Date: Mon, 15 Jul 2002 14:10:20 +0900

> away. Now be an engineer and show us a trace.

a bit hard when it is broken arp packets

Jul 15 11:47:27 roam /kernel: arp: unknown hardware address format (0x0800)
Jul 15 11:47:27 roam /kernel: arp: unknown hardware address format (0x0800)
Jul 15 13:01:45 roam /kernel: arp: unknown hardware address format (0x0800)
Jul 15 13:49:08 roam /kernel: arp: unknown hardware address format (0x0800)

> The routers have a bug in their implementation of ARP.

yet to be known

> ARP, and our use of it, isn't experimental.

your use of it was clearly demonstrated to have broken things this
morning.  we're just asking you to turn your stuff off so we can
isolate the problem.  thanks to lars for being cooperative and
doing so.

randy

---

From: Joe Touch <[email-masked]>
Date: Sun, 14 Jul 2002 22:18:21 -0700

> Jul 15 13:49:08 roam /kernel: arp: unknown hardware address format (0x0800)

That totals 4 packets that are ill-formed in a period of 2 hours, 
packets which are causing the router to crash. That's a bug, period.

This has nothing to do with what happened this morning - those were 
incorrect source addresses.

>>The routers have a bug in their implementation of ARP.

A router that doesn't drop an ill-formed address in an ARP packet has a 
bug. A router that crashes due to this has a fatal bug.

Let's replace that router, and debug THAT offline.

> isolate the problem.

If you had bothered to ask, it isn't even my stuff. It's Lars'. I'm 
running (still, since this morning, and haven't stopped)- WinXP (gasp).

Joe

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 15 Jul 2002 07:59:58 +0200

Somebody is running a number of ad-hoc nodes (or one chameleon-style node) in 
the 802.11b airspace of the IETF meeting.  This is bad enough as it is not 
coordinated with the site's frequency plan.  It is even worse when at least one 
of the nodes uses the SSID "IETF", which causes all Macs (and a sizable number 
of Lucent drivers) to oscillate between their ad-hoc network and the IETF network.

Log attached.  The culprit nodes apparently use generated local addresses (02 
bit in MAC address set, rest seems somewhat random).

Please turn it off.

Since the most frequent SSID is "pulver.com", I interpret this as the knife 
dripping with blood (but then Jeff could still be innocent even if the knife is 
engraved with his DNS name :-).

Gruesse, Carsten
IETF              00:02:2D:00:09:DC	7	46	Managed
IETF              00:02:2D:00:0A:84	6	43	Managed
IETF              00:02:2D:00:14:5E	1	25	Managed
IETF              00:02:2D:00:19:6C	9	27	Managed
IETF              00:02:2D:2D:A3:9C	10	47	Managed
IETF              00:02:2D:4F:99:42	3	57	Managed
IETF              00:02:2D:4F:99:56	6	24	Managed
IETF              00:02:2D:50:22:5D	10	25	Managed
IETF              00:60:1D:23:78:13	5	42	Managed
IETF              00:60:1D:23:AA:D7	1	28	Managed
IETF              02:A0:41:A0:61:1C	11	79	Ad-hoc
<<<
Wireless Network  02:05:5D:2C:6C:3D	1	26	Ad-hoc
pulver.com        02:0A:56:0B:50:A9	10	46	Ad-hoc
pulver.com        02:15:E3:14:E5:B6	10	49	Ad-hoc
pulver.com        02:18:09:19:0F:BB	10	41	Ad-hoc
pulver.com        02:1D:4F:1C:49:BE	10	45	Ad-hoc
pulver.com        02:1F:0B:1E:0D:BC	10	46	Ad-hoc
pulver.com        02:21:95:20:93:82	10	51	Ad-hoc
pulver.com        02:28:7A:29:7C:8B	10	48	Ad-hoc
pulver.com        02:55:00:54:06:F6	10	46	Ad-hoc
pulver.com        02:58:6D:59:6B:FB	10	42	Ad-hoc
pulver.com        02:5D:2A:5C:2C:FE	10	46	Ad-hoc
pulver.com        02:6A:10:6B:16:C9	10	49	Ad-hoc
pulver.com        02:79:C4:78:C2:DA	10	49	Ad-hoc
pulver.com        02:96:3A:97:3C:35	10	45	Ad-hoc
pulver.com        02:AB:55:AA:53:08	10	47	Ad-hoc
pulver.com        02:C2:97:C3:91:61	10	48	Ad-hoc
pulver.com        02:C2:98:C3:9E:61	10	39	Ad-hoc
pulver.com        02:C5:8B:C4:8D:66	10	41	Ad-hoc
pulver.com        02:CC:21:CD:27:6F	10	40	Ad-hoc
pulver.com        02:CD:CB:CC:CD:6E	10	48	Ad-hoc
pulver.com        02:DD:16:DC:10:7E	10	41	Ad-hoc
pulver.com        02:E9:97:E8:91:4A	10	49	Ad-hoc
pulver.com        02:EB:18:EA:1E:48	10	42	Ad-hoc
pulver.com        9E:01:B7:01:C4:03	6	41	Ad-hoc

---

From: Matt Crawford <[email-masked]>
Date: Mon, 15 Jul 2002 11:06:20 -0500

> Since the most frequent SSID is "pulver.com", I interpret this as
> the knife dripping with blood (but then Jeff could still be
> innocent even if the knife is engraved with his DNS name :-).

You mean, it could be a case of media-layer FRAMING?

---

From: Jeff Pulver <[email-masked]>
Date: Mon, 15 Jul 2002 12:20:40 -0400 (EDT)

Matt,

What are you talking about?

This is news to me...Jeff

---

From: Jun-ichiro itojun Hagino <[email-masked]>
Date: Sat, 20 Jul 2002 10:41:02 +0900

I looked through RFC826 and it seems that the operation performed by
	Lars was a Bad Thing.  RFC826 input processing explicitly suggests us
	to update ARP cache entry without checking arp operation type.

	therefore, it is unsafe to transmit ARP_REQUEST with spoofed IP
	source address - it will overwrite ARP entries of neighbors.

itojun

---

From: [email-masked]
Date: Fri, 19 Jul 2002 23:36:36 -0400

> 	source address - it will overwrite ARP entries of neighbors.

This is, of course, a major security hole...

---

From: Lars Eggert <[email-masked]>
Date: Sat, 20 Jul 2002 17:26:07 +0900

> 	source address - it will overwrite ARP entries of neighbors.

I re-read it, too, and you are of course right.

It's sad though how easy a DoS attack can be - as easy as mistyping the 
IP address of your machine.

It might be worthwhile to investigate if 826 should be updated. Someone 
at IETF mentioned that Linux explicitly violates 826 and does not update 
the local cache based on the contents of spoofed packets, and thus seems 
more resilient than the BSDs (against this particular bug).

Lars
-- 
Lars Eggert <[email-masked]>           USC Information Sciences Institute

---

From: Joe Touch <[email-masked]>
Date: Sat, 20 Jul 2002 14:36:31 -0700

> 	source address - it will overwrite ARP entries of neighbors.

Agreed- this was known at 9:30am, and fixed very shortly thereafter. It 
is not related to this thread.

The incomplete ARP packets that were posted to this thread had no effect 
on the router (we asked); there is no information yet on which machine 
was causing them.

The connectivity problems experienced esp. Monday around 1pm were 
wireless related. The NOC had been rebooting some of the access points 
while reconfiguring them (about 10 times in 30 minutes). Other problems 
included users running ad-hoc access points, and general congestion of 
the access points.

When the problems occurred, we checked with the NOC, rather than 
continue this thread. We invite others to do the same in the future.

Joe

---

From: Matt Crawford <[email-masked]>
Date: Mon, 22 Jul 2002 05:50:25 -0500

> > 	source address - it will overwrite ARP entries of neighbors.
 (He meant sender address, of course)
> This is, of course, a major security hole...

Gee, if only ARP's funcntion was performed above the IP layer somehow ...

---

From: "C. M. Heard" <[email-masked]>
Date: Mon, 22 Jul 2002 08:35:22 -0700 (PDT)

> more resilient than the BSDs (against this particular bug).

How does one tell, in principle, that the source IP address (ar$spa) in
an ARP packet is in fact spoofed?

//cmh

---

From: Thomas Narten <[email-masked]>
Date: Mon, 22 Jul 2002 15:02:08 -0400

> It might be worthwhile to investigate if 826 should be updated.

Somewhat related, there was a "SEND" BOF last week on securing IPv6's
Neighbor Discovery. 

This AD has been asked to advance the individual submission
draft-cheshire-ipv4-acd-01.txt on the standards track. That document
proposes backwards-compatable updates to RFC 826.

Thomas

---

From: Lars Eggert <[email-masked]>
Date: Tue, 23 Jul 2002 06:11:03 -0400

> an ARP packet is in fact spoofed?

Not without cryptographic authentication, in general.

But for this particular issue, not updating the local cache based on 
snooped ARP exchanges (i.e. what Linux does) may make sense. Also, under 
this particular misconfiguration, there'll very likely be two ARP 
responses for a lookup of the IP address in question, so maybe could be 
used as an indicator that there's a problem.

Lars

---

From: Vernon Schryver <[email-masked]>
Date: Tue, 23 Jul 2002 08:17:33 -0600 (MDT)

> From: Lars Eggert <[email-masked]>

> used as an indicator that there's a problem.

If you ignore gratuitous ARP, then what happens when a station goes down
and then comes back up with a different MAC address?  That happens when
the station is given new hardware or in some fail-over schemes.

Vernon Schryver    [email-masked]

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 23 Jul 2002 10:14:51 -0500

On 7/23/02, Vernon Schryver wrote:

>> > How does one tell, in principle, that the source IP
>> > address (ar$spa) in an ARP packet is in fact spoofed?
>>
>> But for this particular issue, not updating the local
>> cache based on snooped ARP exchanges (i.e. what Linux
>> does) may make sense. Also, under this particular
>> misconfiguration, there'll very likely be two ARP
>> responses for a lookup of the IP address in question, so
>> maybe could be used as an indicator that there's a
>> problem.
>
>If you ignore gratuitous ARP, then what happens when a
>station goes down and then comes back up with a different
>MAC address?  That happens when the station is given new
>hardware or in some fail-over schemes.
>

There are some simple confidence counter techniques that
would filter out most spoofs but still allow fail-overs.

Basically, you use some simple counters and state variables
to detect when an IP address is being repeatedly switched to
one MAC address, and then back to the original value.

When this is detected the original mac address could then be
queried to confirm that it indeed thought it was the IP
address in question (just in case this is some sort of
migration feature and the flip-flopping was valid).

You could probe the original MAC address on *any* change,
but that could prove nasty on a large subnet. Having a
designated ARP validator on a subnet might make sense if a
network administrator was trying to prevent unauthorized use
of IP addresses on their network.

It's much easier to deal with spoofed source IP addresses on
a network basis than from inside one host. At the network
level the source IP address can be cross-checked against the
physical link it arrived on. Even with automatic fail-over,
the network administrator should know where an IP address
*could* come from. This is especially true of the IP
addresses most worth stealing.