Subject: [rddp] Let's discuss markers

From: [email-masked]
Date: Tue, 3 Dec 2002 01:26:08 -0500

Since an important technical issue underlying this debate
about not-quite-bufferless NICs is markers, let's take a
closer look at markers.  This entire message is with my WG
Chair hat off for the purpose of provoking discussion ...

... so, for the sake of discussion, let's assume that we can
run a framing-modified TCP transmitter in aligned mode - the TCP
ULP framing draft that was ready to go experimental not that
long ago proposed to do this, so this is not an entirely
unreasonable assumption.

Under this assumption, what's the purpose of markers?  MPA was
apparently designed under this assumption, and the MPA draft says:

	7.4.1  Re-segmenting Middle boxes and non-conforming senders

Non-conforming senders sure sounds like a matter for connection setup
negotiation - get the other side to say "yes, I do aligned FPDUs"
on setup, and then close the connection in error if the receiver
ever detects that the sender broke this promise (e.g., as if a CRC
error not recoverable via retransmits occurred).

That leaves re-segmenting middle boxes.  First of all, it appears
to be clear from list discussion that the designers of (and advocates
for) MPA view them as a rare or error case.  Even so, many statically
resegmenting middleboxes ought to be detectable via some sort of
"training sequence" test run at connection setup designed to cause
resegmentation - if resegmentation happens, disable framing before
sending any data that depends on it.

This would appear to reduce the scenarios of concern to a subset
of a rare error case.  Given this, the impact of markers on the
operation of framing seems to be rather disproportionate -
e.g., in their current "every 512 bytes" form, they all but
exclude software implementations.

Nonetheless, there's still a problem here.  What if TCP drops a packet
and at that precise point in the bytestream, a dynamic resegmenting
middlebox decides to kick in and screw up everything?  That should be
sufficiently rare that it may be ok to just close the connection ...
and tell the poor user to get a real network ... ;-).

... but a whole bunch of data is going to come in before the retransmit
arrives that indicates the receiver's been fooled by the middlebox - what
gets done with it?  One possibility is to process it on the assumption
that it's all aligned frames, and close the connection in error when
the retransmit shows up indicating that they're not, and the receiver
was fooled by what looked like frames in data (in most cases, errors
will happen very quickly, but the "frame in data" case has to work
correctly) ...

... so what about the frames that were processed incorrectly between the
original drop and the retransmit?  Well, they could only have been placed,
because the missing retransmit prevents delivery to the ULP (TCP delivers
in order, and framing can't change that).  So, when the retransmit arrives,
close the connection, and complete all the pending receives with errors
indicating that the buffers are invalid.  This has to be implemented
anyway to deal with a CRC error that can't be recovered via retransmits,
so it's almost "free" :-) ...

... but what if the frames that were processed incorrectly affected
memory that wasn't supposed to be affected by any pending receive?  Any
memory that it could have modified has to be declared invalid - this is
somewhat worse than the CRC error as any exposed memory in the protection
domain gets invalidated ... but I think I've seen folks headed towards
a design approach of being rather conservative in how much memory is
exposed when (e.g., to provide application assurances that the memory
from completed receives can't be overwritten).  This might not be an
important difference, but it is clearly fodder for discussion, and
reinforces the importance of precisely specifying protection domain
access control functionality.

One more thing - anytime one comes up with an error mechanism that
closes a connection, denial of service becomes a possibility via
the attacker causing the error.  In this case it's a definite possibility,
as a clever attacker could corrupt a TCP packet and then send
a fake one that looks like a retransmit that's been resegmented by a
middlebox, causing the connection to close.  This doesn't look like
a major increase in risk, as an attacker who can send the latter packet
can probably also send a TCP RST which will close the connection far
more effectively (yes, one could in theory do TCP RST filtering at
[some] routers, but in practice this is not done, and hence can't be
relied upon).  IPsec is needed to defend against this class of attacks.

Ok, that should be more than enough to provoke discussion ...
... comments?

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953 **NEW**     FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 3 Dec 2002 12:24:47 -0600

On 12/3/02, [email-masked] wrote:

>
>That leaves re-segmenting middle boxes.  First of all, it
>appears to be clear from list discussion that the
>designers of (and advocates for) MPA view them as a rare
>or error case.  Even so, many statically resegmenting
>middleboxes ought to be detectable via some sort of
>"training sequence" test run at connection setup designed
>to cause resegmentation - if resegmentation happens,
>disable framing before sending any data that depends on it.
>

Rare would be an appropriate term, but error is clearly
incorrect. If such a middlebox is in place, it would tend to
be there on a persistent basis. That is, unlike a router,
the inclusion of a manipulating middlebox in a path is
probably *not* an accident but a deliberate network policy
presumably towards the client-end.

Supporting such connections is vital. But I do not think
that it is vital that it be done in an optimized fashion.
A product development goal that I would likely set is that
non-alligned performance should be at least 99% of TCP
performance, while I would expect aligned performance to be
several magnitudes better.

The question then becomes whether markers can be suppressed
for the overwhelming majority of cases where they are not
needed. Such a solution is actually messier than MPA as
proposed (you must implement both markers *and* their
suppression) but it would virtually eliminate the "marker
tax" on bandwidth.

The caveat is that I do not believe a re-aligning middlebox
can be presumed to show itself immediately.

Consider a middlebox that attempts to consolidate TCP
segments before forwarding them. If it saw a long sequence
with nothing but full-sized tagged messages interleaved with
occasional small untagged messages it might decline to
"optimize" the connection. But then if it saw two short
untagged messages in a row, or a short tailing tagged
message followed by a short untagged message it might jump
in and combine the two small segments.

Rejecting the connection, and then re-establishing it in
"marker mode" would probably be acceptable  for exceptional
connections (the fact that it is annoying to those stuck
behind meddlesome middleboxes can even be viewed as a
benefit, in that it might lead to elimination of said
middleboxes).

However, tearing down a connection on which real work had
already begun is something totally different, and I believe
totally unacceptable.

This suggests an MPA-layer "alignment marker required nack"
which would cause the peer MPA-layer to retransmit the data
with markers (and from that point forward).

As complexity goes such a solution can at best be described
as 'feasible'. It is clearly more complex than desirable.
The question would be whether eliminating the need for
hardware processing of markers, and eliminating the marker
tax on payload are sufficient justifications.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 3 Dec 2002 12:33:47 -0600

As long as we are exploring markers in more depth, I was
wondering if anyone could explain to what degree the
optional fixed interval markers in iSCSI and MPA markers are
compatible with each other. If there are differences, I
would hope they could be documented and justified.

Caitlin Bestler