Subject: Re: FCIP and wandering duplicates

From: "Raj Bhagwat" <[email-masked]>
Date: Mon, 11 Sep 2000 11:31:07 -0700

Doug,

Your point on TTL is well taken. It is there to prevent a packet from
getting stuck in a routing loop. Duplicates of unicast packets can happen
when the sender retransmits a packet and both the original and the
retransmitted packets reach the destination. If the sender does not
retransmit for an adequate amount of time, duplicates of unicast packets
should not happen. This is true even when there are routing loops or
topology changes in the network.

As you have pointed out, there is no direct correlation between E_D_TOV and
TTL. And that could be a problem. However, setting the TTL to a relatively
smaller value (larger than the # of hops between the source and destination
FCIP gateways) can reduce the possibility of duplicates when the sender of
the FC frame waits for at least E_D_TOV before retransmitting the frame.

I don't think SCTP at the FCIP gateway will solve the duplicate problem even
with a single retry. There is no synchronization between the FC device and
the FCIP gateway for error recovery. So, the FC device can retransmit
independent of the SCTP at the FCIP gateway. One way to solve this problem
is to fine tune the timeout values on the FC side and at the SCTP level on
the FCIP gateway.

Regards,

Raj Bhagwat
LightSand Communications

----- Original Message -----
From: Douglas Otis <[email-masked]>
To: Raj Bhagwat <[email-masked]>; Vivek Kashyap <[email-masked]>;
<[email-masked]>
Cc: <[email-masked]>
Sent: Monday, September 11, 2000 10:14 AM
Subject: RE: FCIP and wandering duplicates

> Raj,
>
> Unless you are willing to wrap the FC frame within some protocol,
duplicates
> can happen and this has nothing to do with TTL.  TTL prevents circular
paths
> from forever sending a packet within a loop.  A modified SCTP protocol
that
> will solve both of these problems, if restricted one time retry.  A
> Metro-Area-Network should induce about 5ms RTT delay such that a 10ms skew
> becomes possible but is limited by the single retry.  TTL buys you nothing
> with this E_D_TOV, Error_Detect Timeout.
>
> Doug
>
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]]On Behalf Of
> > Raj Bhagwat
> > Sent: Sunday, September 10, 2000 10:57 PM
> > To: Vivek Kashyap; [email-masked]
> > Cc: [email-masked]
> > Subject: Re: FCIP and wandering duplicates
> >
> >
> > Vivek,
> >
> > In a properly implemented FC-only fabric, duplicates should not happen.
A
> > frame that enters a fabric should either be delivered within E_D_TOV
> > (typically 2 sec) or be discarded. FC devices don't retransmit a frame
> > before the E_D_TOV timer expires.
> >
> > You do bring up an interesting point when two SAN islands are
> > bridged across
> > an IP network. Once an FC frame enters the IP network, it is hard
> > to say how
> > the E_D_TOV is enforced by that part of the network. As authors of the
> > FCoverIP draft, we have given some thought to the possibility of
> > delivering
> > duplicate frames when both the TCP stack at the FCIP gateway and the
> > original FC device initiate recovery/retransmission procedure in an
> > overlapping time period. This is one of the major reasons why we
initially
> > chose to make the FCIP gateway a stateless device with no buffering and
no
> > retransmission.
> >
> > The good news is that the {IP encapsulated} FC frame will be protected
by
> > the IP TTL once it enters the IP network. The bad news is that
> > detecting the
> > E_D_TOV timeout while the frame is inside the IP network may be
difficult.
> >
> > Regards,
> >
> > Raj Bhagwat
> > LightSand Communications
> >
> >
> > ----- Original Message -----
> > From: Vivek Kashyap <[email-masked]>
> > To: <[email-masked]>
> > Sent: Sunday, September 10, 2000 3:17 PM
> > Subject: FCIP and wandering duplicates
> >
> >
> > > I read the FCIP document recently. It does not address the issue of
> > > duplicates.
> > >
> > > Does fibre channel protect itself from duplicates ? I looked at the
> > > header and it does not have a time to live field. Thus a duplicate
> > > caught in a loop and then released at the 'right' time could be
> > > accepted by the receiver. Even if the FC fabric path algorithms do
> > > ensure loop free paths transient loops due to errors, switch
> > > misconfigurations or failures are possible. Does FC handle such a case
?
> > >
> > > If it does not then FCIP will exacerbate the problem because the IP
> > > network could have 'transient' loops. Such a 'wandering duplicate'
> > > could be re-injected into the FC island and be accepted by a newer
> > > incarnation of a FC inter-island exchange. Furthermore with no TTL one
> > > can't really provide a timout between starting two conversations.
> > >
> > >
> > >
> > > Vivek
> > > --
> > > Vivek Kashyap
> > > IBM-NumaQ
> > > [email-masked]
> > >
> >
>
>

---

From: "Raj Bhagwat" <[email-masked]>
Date: Mon, 11 Sep 2000 12:25:50 -0700

Doug,

I understand how exactly TTL is used. That is why I stated that TTL should
be set to a value larger than the number of hops between the source and
destination. An IP end station may not know this and it may choose to set
the TTL to the max value of 255. However, an FCIP gateway, with some
intelligence, can perhaps know the # of hops to other FCIP gateways it talks
to, especially in enterprise networks.

It is not clear to me how there can be duplicate frames without the source
sending twice. Could you please explain that?

Thanks,

Raj.

----- Original Message -----
From: Douglas Otis <[email-masked]>
To: Raj Bhagwat <[email-masked]>; Vivek Kashyap <[email-masked]>;
<[email-masked]>
Cc: <[email-masked]>
Sent: Monday, September 11, 2000 11:53 AM
Subject: RE: FCIP and wandering duplicates

> Raj,
>
> You can see duplications without the source sending twice.  Reducing TTL
> value may prevent the packet from ever arriving.  The number of times TTL
is
> decremented depends on the path and not time in transit.
>
> Doug
>
> > From: Raj Bhagwat [mailto:[email-masked]]
> > Sent: Monday, September 11, 2000 11:31 AM
> > To: Douglas Otis; Vivek Kashyap; [email-masked]
> > Cc: [email-masked]

---

From: Vivek Kashyap <[email-masked]>
Date: Tue, 12 Sep 2000 15:26:43 -0700 (PDT)

Raj,

I am, after reading your and Doug's replies, still unclear whether FC handles 
duplicates or not. It appears to me that it does not.

E_D_TOV seems to be a time out that is set for retransmission between
end points. However, it is possible to introduce transient loops in
the fabric when a topology change (addtion of a new switch or a
link/switch going down) occurs, due to the re-calculation of paths
unless such changes are made atomic to the traffic. In such a case
there could be duplicates.

Of course, if the FC switches are quiesced during the switch updates
then loops won't matter. Or is there a different mechanism or an
algorithm that guarantees looplessnes?

TTL retuning will at most reduce the duplicates but can't guarantee
elimination of duplicates. So if FC somehow guarantees duplicate
free fabrics and does not handle duplicates then it appears to me that
FCIP always will have the danger of corrupting data due to duplicates
inserted by IP.

Vivek

> 
> Raj,
> 
> I do not understand what tuning TTL does to benefit any situation.  The path
> between locations is not fixed so such tuning may induce lost packets and is
> unrelated to time in transit.  Depending on the domain, a packet may be
> repeated by equipment that does not necessarily decrement TTL.  Although not
> a desired feature, duplication may happen if the packet transmit status is
> made uncertain by local signaling.
> 
> Doug
> 
> > Sent: Monday, September 11, 2000 12:26 PM
> > > > FCIP gateways) can reduce the possibility of duplicates when
> > the sender
> > of
> > > > is to fine tune the timeout values on the FC side and at the
> > SCTP level
> > on
> > > > the FCIP gateway.
> > > >
> > > > Regards,
> > > >
> > > > Raj Bhagwat
> > > > LightSand Communications
> > > >
> > > >
> > > > > > From: [email-masked]
> > [mailto:[email-masked]]On Behalf
> > Of
> > > > > > Raj Bhagwat

-- 
Vivek Kashyap
Advisory Engineer
IBM-NumaQ
[email-masked]
[email-masked]
503 578 3422 (o)