Subject: Re: [storm] Removing iSCSI Markers?

From: "Pat Thaler" <[email-masked]>
Date: Fri, 2 Apr 2010 10:39:05 -0700

We also don't use markers.

I don't recall Randy's presentation, but markers only partially allow out of order placement. If you didn't receive the header of the PDU, you can't place. By the time you take that into account plus the cost of implementing markers, they aren't worth whatever memory savings they might allow. And for various reasons, a 10G offload NIC needs a lot more than 2K RAM.

Regards,
Pat

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark S. Edwards
Sent: Tuesday, March 30, 2010 9:22 AM
To: [email-masked]
Subject: Re: [storm] Removing iSCSI Markers?

Asgeir,

Yes, I do understand and remember all the arguments.  Indeed, I remember one of Randy's theoretical presentations positing that a marker aware 10GB offload NIC might only require as little as 2K onboard buffering RAM.

The point is, that at least for iSCSI the technology that arrived seems to marginalised the need for anyone to implement markers.  Given the fact that running code creates IETF consensus we have an opportunity at this time to remove unnecessary complications, markers are a candidate for being made optional or even being removed completely.

Personally I would be happy to see them removed.  My original note on this topic was to try to be a good citizen by asking anyone who might be affected, or know someone who would be affected, to speak up.

Mark.

At 16:28 30/03/2010, Asgeir Eiriksson wrote:
Paul, Mark

The main selling point for markers is that they enable out or order placement
(on receive) while preserving in order completions and markers therefore have the
potential of decreasing buffering requirements in RNIC and to a less extent in
iSCSI HBA.

'Asgeir

________________________________
Date: Tue, 30 Mar 2010 15:49:11 +0100
To: [email-masked]
From: [email-masked]
Subject: Re: [storm] Removing iSCSI Markers?

Paul,

That's pretty much my recollection, too.

One of those things that was thought to be a reasonable solution to a foreseeable change in the technology.  In the end, the technology found a different solution.

They were fascinating presentations, though.

Mark.

At 15:22 30/03/2010, Paul Koning wrote:

Thanks Asgeir.

As I recall, the original idea behind markers is to make it possible to build 10G HBAs that can run at wire speed, which was believed to be impossible otherwise.

The subsequent record indicates that this was in fact not the case; 10G HBAs are feasible and have been built without resorting to markers.  There is no other reason for using markers.  So if the one reason that they were thought to be needed in fact turned out not to be real, the obvious thing to do is to remove the unused complications from the spec.

I suppose one could argue that, placed in an appendix and optional to implement they do no harm.  Thats a fair point.  If there is still a chance that they will turn out to be needed in the future we may want to go that way.  I personally would bet against that chance.

                paul

From: Asgeir Eiriksson [ mailto:[email-masked]]
Sent: Tuesday, March 30, 2010 9:24 AM
To: Paul Koning; [email-masked]; [email-masked]; [email-masked]
Subject: RE: [storm] Removing iSCSI Markers?

Hello Paul,

The Chelsio RNIC do support the marker feature, but as far as I know the feature
has never been used in the field, and it isn't supported by all RNIC
implementations.

I periodically ask our AE and developers about this feature and so far the answer
is that no one uses it, and no one is asking for it (4 years of data at this point).

Regards,

Asgeir Eiriksson
CTO
Chelsio Communications Inc.

> Date: Mon, 29 Mar 2010 21:01:49 -0400
> From: [email-masked]
> To: [email-masked]; [email-masked]; [email-masked]
> Subject: Re: [storm] Removing iSCSI Markers?
>
> I sure would like markers to go away. Rumors of their use are somewhat
> interesting, but substantiated data would be more so.
>
> paul
>
> > -----Original Message-----
> > From: [email-masked] [ mailto:[email-masked]] On Behalf
> > Of Mallikarjun Chadalapaka
> > Sent: Monday, March 29, 2010 8:33 PM
> > To: 'Mark S. Edwards'; [email-masked]
> > Subject: [storm] Removing iSCSI Markers?
> >
> > Just to clarify...
> >
> > > On another removal topic, I seem to recall that Mallikarjun also
> said
> > > that he was removing markers.
> >
> > I had only said that it's one of the items I had heard prior requests
> > on
> > (that it be removed). Thanks for initiating the list discussion
> > though!
> >
> > > but I do wonder if this will affect any HBA implementations ?
> >
> > Good question, I don't know. HBA vendors, especially iSCSI/iSER/RNIC
> > "roto-tilled" implementations, please chime in.
> >
> >
> > Mallikarjun
> >
> >
> >
> > > -----Original Message-----
> > > From: [email-masked] [ mailto:[email-masked]] On
> > Behalf Of
> > Mark
> > > S. Edwards
> > > Sent: Saturday, March 27, 2010 6:56 AM
> > > To: [email-masked]
> > > Subject: Re: [storm] Draft minutes from Anaheim
> > >
> > > Regarding the feature removal discussion, can I add SLP to the list
> ?
> > >
> > > The RFC 3721 states
> > >
> > > "iSCSI equipment that
> > > need discovery functions beyond SendTargets should at least
> > > implement SLP, and then consider iSNS when extended discovery
> > > management capabilities are required such as in larger
> storage
> > > networks. It should be noted that since iSNS will support
> > SLP,
> > > iSNS can be used to help manage the discovery information
> > returned
> > > by SLP."
> > >
> > > The implication is that targets and initiators should expect to find
> > > support for SLP before considering iSNS.
> > >
> > > I remember our first iSCSI appliance and we spent ages trying to get
> > > SLP working because it the above wording effectively made it
> > > mandatory. SLP turned out to be a complete bust and was effectively
> > > killed off when Microsoft refused to support it in their initiator
> > > and in their target logo tests.
> > >
> > > The result is that today I doubt you could find a target or
> initiator
> > > out there supporting SLP.
> > >
> > > For anybody that does still implement SLP we could change the
> wording
> > > for SLP a little to remove the implied hierarchy, or just admit that
> > > running code has created IETF consensus.
> > >
> > >
> > > On another removal topic, I seem to recall that Mallikarjun also
> said
> > > that he was removing markers. I don't particularly object to this
> > > but I do wonder if this will affect any HBA implementations ?
> > >
> > > Regards,
> > >
> > > Mark.
> > >
> > >
> > > At 06:59 27/03/2010, [email-masked] wrote:
> > > >Draft minutes are attached - please comment, correct, etc.
> > > >
> > > >Also, in the absence of objection on this mailing list, decisions
> > > >recorded in the minutes are considered to be the rough consensus of
> > > >this WG, *except* that two issues were identified as sufficiently
> > > >important to discuss separately on the list (see separate
> messages):
> > > > - Text negotiation key for new iSCSI features (discussion
> > > > in progress)
> > > > - Features to remove from iSCSI (discussion to be started)
> > > >
> > > >Many thanks to Craig Carlson for taking notes during the meeting.
> > > >
> > > >Thanks,
> > > >--David
> > > >----------------------------------------------------
> > > >David L. Black, Distinguished Engineer
> > > >EMC Corporation, 176 South St., Hopkinton, MA 01748
> > > >+1 (508) 293-7953 FAX: +1 (508) 293-7786
> > > >[email-masked] Mobile: +1 (978) 394-7754
> > > >----------------------------------------------------
> > > >
> > > >
> > > >
> > > >
> > > >_______________________________________________
> > > >storm mailing list
> > > >[email-masked]
> > > > https://www.ietf.org/mailman/listinfo/storm
> > >
> > > _______________________________________________
> > > storm mailing list
> > > [email-masked]
> > > https://www.ietf.org/mailman/listinfo/storm
> >
> > _______________________________________________
> > storm mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/storm
> _______________________________________________
> storm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/storm
________________________________
Hotmail is redefining busy with tools for the New Busy. Get more from your inbox. Sign up now.<http://www.windowslive.com/campaign/thenewbusy?ocid=PID27925::T:WLMTAGL:ON:WL:en-US:WM_HMP:032010_2>

________________________________
Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox. Learn More.<http://www.windowslive.com/campaign/thenewbusy?ocid=PID27925::T:WLMTAGL:ON:WL:en-US:WM_HMP:032010_1>

---

From: "Mark Bakke (mbakke)" <[email-masked]>
Date: Fri, 2 Apr 2010 16:26:52 -0500

Cisco doesn't use markers, either, and has no plans to do so.  I agree
that with the RDMA/iSER work that could be used similar purposes,
there's no need for them in the iSCSI protocol.

Mark

From: [email-masked] [mailto:[email-masked]] On Behalf
Of Pat Thaler
Sent: Friday, April 02, 2010 12:39 PM
To: Mark S. Edwards; [email-masked]
Subject: Re: [storm] Removing iSCSI Markers?

I don't recall Randy's presentation, but markers only partially allow
out of order placement. If you didn't receive the header of the PDU, you
can't place. By the time you take that into account plus the cost of
implementing markers, they aren't worth whatever memory savings they
might allow. And for various reasons, a 10G offload NIC needs a lot more
than 2K RAM.

Regards,

Pat

From: [email-masked] [mailto:[email-masked]] On Behalf
Of Mark S. Edwards
Sent: Tuesday, March 30, 2010 9:22 AM
To: [email-masked]
Subject: Re: [storm] Removing iSCSI Markers?

Asgeir,

Yes, I do understand and remember all the arguments.  Indeed, I remember
one of Randy's theoretical presentations positing that a marker aware
10GB offload NIC might only require as little as 2K onboard buffering
RAM.

The point is, that at least for iSCSI the technology that arrived seems
to marginalised the need for anyone to implement markers.  Given the
fact that running code creates IETF consensus we have an opportunity at
this time to remove unnecessary complications, markers are a candidate
for being made optional or even being removed completely.

Personally I would be happy to see them removed.  My original note on
this topic was to try to be a good citizen by asking anyone who might be
affected, or know someone who would be affected, to speak up.

Mark.

Paul, Mark
 
The main selling point for markers is that they enable out or order
placement
(on receive) while preserving in order completions and markers therefore
have the
potential of decreasing buffering requirements in RNIC and to a less
extent in
iSCSI HBA.
 
'Asgeir 

Paul,

One of those things that was thought to be a reasonable solution to a
foreseeable change in the technology.  In the end, the technology found
a different solution.

Mark.

Thanks Asgeir.

As I recall, the original idea behind markers is to make it possible to
build 10G HBAs that can run at wire speed, which was believed to be
impossible otherwise.

The subsequent record indicates that this was in fact not the case; 10G
HBAs are feasible and have been built without resorting to markers.
There is no other reason for using markers.  So if the one reason that
they were thought to be needed in fact turned out not to be real, the
obvious thing to do is to remove the unused complications from the spec.

I suppose one could argue that, placed in an appendix and optional to
implement they do no harm.  Thats a fair point.  If there is still a
chance that they will turn out to be needed in the future we may want to
go that way.  I personally would bet against that chance.

                paul 

To: Paul Koning; [email-masked]; [email-masked];
[email-masked]

  

Hello Paul,

The Chelsio RNIC do support the marker feature, but as far as I know the
feature

implementations. 

I periodically ask our AE and developers about this feature and so far
the answer

is that no one uses it, and no one is asking for it (4 years of data at
this point).

Regards,

Asgeir Eiriksson

CTO

  

> > From: [email-masked] [ mailto:[email-masked]] On
Behalf

Hotmail is redefining busy with tools for the New Busy. Get more from
your inbox. Sign up now.
<http://www.windowslive.com/campaign/thenewbusy?ocid=PID27925::T:WLMTAGL
:ON:WL:en-US:WM_HMP:032010_2> 

Hotmail has tools for the New Busy. Search, chat and e-mail from your
inbox. Learn More.
<http://www.windowslive.com/campaign/thenewbusy?ocid=PID27925::T:WLMTAGL
:ON:WL:en-US:WM_HMP:032010_1>

---

From: "David Harrington" <[email-masked]>
Date: Sun, 11 Apr 2010 11:15:58 -0400

<as an individual contributor>
inline 

> From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of Mallikarjun Chadalapaka
> Sent: Tuesday, March 30, 2010 12:42 PM
> I believe out of order placement continues to be critical for 
> efficient
> RNIC/DDP implementations, and markers play a role there.
> 
> Having said that, IMHO, that is not exactly the question we 
> should tackle on
> this thread.
> 
> We should focus on these two iSCSI-centric questions: 
> 
> 1) Are there implementations out there that implement iSCSI 
> Markers *as
> defined by RFC 3720*? (Asgeir may have answered this question 
> as "yes", but
> he referenced an RNIC so I am not sure if he's referring to 
> the MPA-version
> of markers or iSCSI key-driven markers)
> 
> 2) If "yes" to #1, if we drop iSCSI Markers in the new 
> Consolidated draft,
> would that cause problems to any "applications" - i.e. iSCSI 
> and SCSI stacks
> in either commercial O/S or proprietary embedded implementations?
> 
> 
> If the answer to the second question is "No", we can go ahead 
> and drop it
> from the iSCSI Consolidated draft, independent of MPA/DDP/RDMAP.

Even if the answer to the second question is "Yes", we can consider
whether
the feature is being used interoperably across vendors. If each
application 
is designed to work specifically with the same vendor's stack, then an
IETF
standard for multi-vendor interoperability isn't needed, not even in
an appendix. 
Dropping the feature from the standard would not necessarily impact
existing 
interoperability.

This type of approach of dropping features is supported by the
RFC2026-defined
"bar" for moving from Proposed Standard to Draft Standard:

   The requirement for at least two independent and interoperable
   implementations applies to all of the options and features of the
   specification.  In cases in which one or more options or features
   have not been demonstrated in at least two interoperable
   implementations, the specification may advance to the Draft
Standard
   level only if those options or features are removed.

I am just showing that dropping a feature is justified by IETF
process.
I would focus on the "interoperable implementation" question, not the
"any implementation" question.

(and I am not suggesting going for Draft Standard; that would be a
separate discussion.
You can drop features without advancing the draft.)

> 
> Thanks.
> 
> Mallikarjun
> 
> 
> 
> 
> 
> 
> From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of
> Mark S. Edwards
> Yes, I do understand and remember all the arguments.  Indeed, 
> I remember one
> of Randy's theoretical presentations positing that a marker aware
10GB
> offload NIC might only require as little as 2K onboard buffering
RAM.
> 
> The point is, that at least for iSCSI the technology that 
> arrived seems to
> marginalised the need for anyone to implement markers.  Given 
> the fact that
> running code creates IETF consensus we have an opportunity at 
> this time to
> remove unnecessary complications, markers are a candidate for 
> being made
> optional or even being removed completely.
> 
> Personally I would be happy to see them removed.  My original 
> note on this
> topic was to try to be a good citizen by asking anyone who might be
> potential of decreasing buffering requirements in RNIC and to 
> a less extent
> in
> iSCSI HBA.
>  
> 'Asgeir 
>  
> ________________________________________
> foreseeable change in the technology.  In the end, the 
> technology found a
> different solution.
> 
> The subsequent record indicates that this was in fact not the 
> case; 10G HBAs
> are feasible and have been built without resorting to 
> markers.  There is no
> other reason for using markers.  So if the one reason that 
> they were thought
> to be needed in fact turned out not to be real, the obvious 
> thing to do is
> to remove the unused complications from the spec.
>  
> I suppose one could argue that, placed in an appendix and “optional
to
> implement” they do no harm.  That’s a fair point.  If there is still
a
> chance that they will turn out to be needed in the future we 
> may want to go
> that way.  I personally would bet against that chance.
>  
>                 paul 
>  
> I periodically ask our AE and developers about this feature 
> and so far the
> answer
> is that no one uses it, and no one is asking for it (4 years 
> of data at this
> point).
>  
> Regards,
>  
> Asgeir Eiriksson
> CTO
> ________________________________________
> Hotmail is redefining busy with tools for the New Busy. Get 
> more from your
> inbox. Sign up now.
> 
> ________________________________________
> Hotmail has tools for the New Busy. Search, chat and e-mail 
> from your inbox.
> Learn More. 
>

---

From: "Mallikarjun Chadalapaka" <[email-masked]>
Date: Mon, 12 Apr 2010 15:44:02 -0700

Hi David,

> "any implementation" question.

True, I agree.  Thanks for pointing this out.  

On this specific thread, the resounding answer to question #1 so far is
"No".  So I believe the WG is in agreement on removing iSCSI Markers.  Once
David is back from vacation on the 19th, I assume he will make the formal
consensus call, unless someone reports news of existing interoperable
implementations in the interim.

Thanks.

Mallikarjun

> From: David Harrington [mailto:[email-masked]]
> Sent: Sunday, April 11, 2010 8:16 AM
> To: 'Mallikarjun Chadalapaka'; 'Mark S. Edwards'; [email-masked]