Subject: Re: [dtn] TCPCLv4 -07 updates and simplification

From: "Clark, Gilbert J. (GRC-LCA0)" <[email-masked]>
Date: Wed, 28 Mar 2018 15:11:44 +0000

Ah, I forgot: " Segments from different bundles are never interleaved."  That does indeed mean the only time this is an issue would be on the return, and keeps the complexity manageable ... so sure, that makes more sense.

I loathe the idea of necessitating one TCPCL connection per unidirectional transfer channel.  We’re building this on top of a reliable, bidirectional communication channel that itself involves nontrivial state and link overhead – there should be a way to frame and flow bidirectional data without having two individual connections in order to do it.

I’ll also raise an additional objection here because it seems like this would break support for reactive fragmentation during the time the return transfer is in progress, unless there’s something else I’m missing?

-Gilbert

The views expressed in this mail reflect the opinions of the author.  They are, therefore, not intended to reflect official positions of NASA or the U.S. Government.

From: dtn <[email-masked]> on behalf of Brian Sipos <[email-masked]>
Date: Wednesday, March 28, 2018 at 10:00 AM
To: "Burleigh, Scott C (JPL-312B)[Jet Propulsion Laboratory]" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [dtn] TCPCLv4 -07 updates and simplification

Scott,

The issue is in the case the two nodes are both attempting a transmission (in opposite directions), and it's a matter of bundle data transmission monopolizing the TCP stream which prevents XFER_REFUSE / XFER_ACK messages from being sent.

For example: Imagine node A sending a large 1GiB bundle "A" to node B within a single segment. Once the start of that segment transmission begins it is committed and there are only two possible outcomes: the node continues to send the single large message to completion, or something interrupts the message and the entire TCPCL session is failed. Halfway through that transfer, node B attempts its own transfer of a small 1KiB bundle "B" to node A within a single segment. The transfer of bundle B is quick and completes in a short time, but the ACK for bundle B has to wait until the entirety of bundle A has completed sending, since it was all sent in a single message, TCPCL messages are atomic, and TCP is single-flow in each direction.

This is not a horrible situation, the ACK will arrive eventually but it makes timing of when to be able to expect an ACK less predictable, and there's a liability of several transfer's ACKs to stack up waiting for a large transfer. An implementation work-around in this case is to only use one TCPCL session for unidirectional transfers (which introduces its own complexity at the BP agent of how to distinguish what purpose/direction a session is 'supposed to be' used for).

Victoria,

If the TCPCL is to be a reliable CL, then XFER_ACK is absolutely necessary. It is there to inform the sender that the receiving BP agent itself has fully received a bundle (not just the OS fully receiving all of its associated TCP packets).

To the WG,

My feeling right now is that segmentation, as already defined and as an implementation decision, is a valuable mechanism to be able to tune some performance aspects of the TCPCL. It's also a direct inheritance from TCPCLv3 so makes logical transition straightforward. I don't see segment reassembly as a large implementation issue, because of the in-order guarantee of segments and the mandatory XFER_INIT allows pre-allocating bundle storage. I can add an explanatory sub-section under Section 2 for recommendations on tuning segmentation for different network goals.

From: Burleigh, Scott C (312B) <[email-masked]>
Sent: Wednesday, March 28, 2018 1:56:35 AM
To: Brian Sipos; [email-masked]
Subject: RE: TCPCLv4 -07 updates and simplification

Brian, is it possible that this is an implementation matter?  I am thinking that the sending node could have one thread continuously transmitting segment data over the socket while another thread running concurrently could be reading XFER_ACK messages (sent by the receiving node) from the same socket.  Am I overlooking some reason this won’t work?

Scott

From: dtn <[email-masked]> On Behalf Of Brian Sipos
Sent: Monday, March 26, 2018 12:36 PM
To: [email-masked]
Subject: [dtn] TCPCLv4 -07 updates and simplification

All,

After looking into simplifying TCPCLv4 as discussed at the IETF, I see that if a transfer is always simplified into a single XFER_INIT message followed by a single "segment" of data this effectively turns the TCPCL session unidirectional by preventing any other XFER_ACK messages from acknowledging transfers in the other direction within the session. One side effect of segmented transfer data is to allow other messages from the transmitting node between segments by avoiding monopolizing the TCP connection with a single huge message.

I feel like segmentation, as an optional behavior as currently specified, gives the network / node management a way to avoid this kind of message queue blocking behavior and allows more deterministic timing of when an ACK can be expected based on path delays.

For a simple-as-possible workflow, an implementation could choose to disallow a Segment MRU any smaller than the Transfer MRU and could choose to always transfer within a single segment which would minimize the message framing overhead. If interoperability was not necessary then it could also refuse any transfer that includes a segment smaller than the transfer size, so that there would be no need for receive segment concatenation.

Any thoughs on keeping segmentation as a negotiated behavior in order to allow for better administrative control over worst-case session timing?

---

From: Rick Taylor <[email-masked]>
Date: Wed, 28 Mar 2018 15:56:00 +0000

Hi Gilbert,

The consensus in the room at last week's IETF was that Reactive Fragmentation
was a can of worms not yet to be opened.  There will be extension headers in
TCPCL to allow it's inclusion at a later date, but for now it is out of scope.

The rationale is that TCPCL is a critical document that is currently blocking
progress of BPbis and BPsec.  Simplification was seen as a pragmatic way to make
progress.

Cheers,

Rick

On Wed, 2018-03-28 at 15:11 +0000, Clark, Gilbert J. (GRC-LCA0) wrote:
> Ah, I forgot: " Segments from different bundles are never interleaved."  That
> does indeed mean the only time this is an issue would be on the return, and
> keeps the complexity manageable ... so sure, that makes more sense.
>  
> I loathe the idea of necessitating one TCPCL connection per unidirectional
> transfer channel.  We’re building this on top of a reliable, bidirectional
> communication channel that itself involves nontrivial state and link overhead
> – there should be a way to frame and flow bidirectional data without having
> two individual connections in order to do it.
>  
> I’ll also raise an additional objection here because it seems like this would
> break support for reactive fragmentation during the time the return transfer
> is in progress, unless there’s something else I’m missing?
>  
> -Gilbert
>  
> The views expressed in this mail reflect the opinions of the author.  They
> are, therefore, not intended to reflect official positions of NASA or the U.S.
> Government.
>  
> To: "Burleigh, Scott C (JPL-312B)[Jet Propulsion Laboratory]" <scott.c.burleig
> [email-masked]>, "[email-masked]" <[email-masked]>
> The issue is in the case the two nodes are both attempting a transmission (in
> opposite directions), and it's a matter of bundle data transmission
> monopolizing the TCP stream which prevents XFER_REFUSE / XFER_ACK messages
> from being sent.
>  
> For example: Imagine node A sending a large 1GiB bundle "A" to node B within a
> single segment. Once the start of that segment transmission begins it is
> committed and there are only two possible outcomes: the node continues to send
> the single large message to completion, or something interrupts the message
> and the entire TCPCL session is failed. Halfway through that transfer, node B
> attempts its own transfer of a small 1KiB bundle "B" to node A within a single
> segment. The transfer of bundle B is quick and completes in a short time, but
> the ACK for bundle B has to wait until the entirety of bundle A has completed
> sending, since it was all sent in a single message, TCPCL messages are atomic,
> and TCP is single-flow in each direction.
>  
> This is not a horrible situation, the ACK will arrive eventually but it makes
> timing of when to be able to expect an ACK less predictable, and there's a
> liability of several transfer's ACKs to stack up waiting for a large transfer.
> An implementation work-around in this case is to only use one TCPCL session
> for unidirectional transfers (which introduces its own complexity at the BP
> agent of how to distinguish what purpose/direction a session is 'supposed to
> be' used for).
>  
> Victoria,
> If the TCPCL is to be a reliable CL, then XFER_ACK is absolutely necessary. It
> is there to inform the sender that the receiving BP agent itself has fully
> received a bundle (not just the OS fully receiving all of its associated TCP
> packets).
>  
> To the WG,
> My feeling right now is that segmentation, as already defined and as an
> implementation decision, is a valuable mechanism to be able to tune some
> performance aspects of the TCPCL. It's also a direct inheritance from TCPCLv3
> so makes logical transition straightforward. I don't see segment reassembly as
> a large implementation issue, because of the in-order guarantee of segments
> and the mandatory XFER_INIT allows pre-allocating bundle storage. I can add an
> explanatory sub-section under Section 2 for recommendations on tuning
> segmentation for different network goals.
> 
> Brian, is it possible that this is an implementation matter?  I am thinking
> that the sending node could have one thread continuously transmitting segment
> data over the socket while another thread running concurrently could be
> reading XFER_ACK messages (sent by the receiving node) from the same socket. 
> Am I overlooking some reason this won’t work?
>  
> Scott
>  
> After looking into simplifying TCPCLv4 as discussed at the IETF, I see that if
> a transfer is always simplified into a single XFER_INIT message followed by a
> single "segment" of data this effectively turns the TCPCL session
> unidirectional by preventing any other XFER_ACK messages from acknowledging
> transfers in the other direction within the session. One side effect of
> segmented transfer data is to allow other messages from the transmitting node
> between segments by avoiding monopolizing the TCP connection with a single
> huge message.
>  
> I feel like segmentation, as an optional behavior as currently specified,
> gives the network / node management a way to avoid this kind of message queue
> blocking behavior and allows more deterministic timing of when an ACK can be
> expected based on path delays.
>  
> For a simple-as-possible workflow, an implementation could choose to disallow
> a Segment MRU any smaller than the Transfer MRU and could choose to always
> transfer within a single segment which would minimize the message framing
> overhead. If interoperability was not necessary then it could also refuse any
> transfer that includes a segment smaller than the transfer size, so that there
> would be no need for receive segment concatenation.
>  
> Any thoughs on keeping segmentation as a negotiated behavior in order to allow
> for better administrative control over worst-case session timing?
> _______________________________________________
> dtn mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dtn

---

From: "Clark, Gilbert J. (GRC-LCA0)" <[email-masked]>
Date: Wed, 28 Mar 2018 16:16:30 +0000

That is extremely unfortunate.  I will note that I was under the impression that such decisions normally would go through the list in addition to the meeting.

I will note that this decision does leave tcpcl unsuitable for my immediate use cases, so ... my objections are withdrawn, and I'll simply need to come up with something else entirely :)

Cheers,
Gilbert

The views expressed in this mail reflect the opinions of the author.  They  are, therefore, not intended to reflect official positions of NASA or the U.S.
Government.
________________________________________
From: dtn [[email-masked]] on behalf of Rick Taylor [[email-masked]]
Sent: Wednesday, March 28, 2018 11:56 AM
To: [email-masked]; Clark, Gilbert J. (GRC-LCA0); Burleigh, Scott C (JPL-312B)[Jet Propulsion Laboratory]; [email-masked]
Subject: Re: [dtn] TCPCLv4 -07 updates and simplification

Hi Gilbert,

Cheers,

Rick

---

From: William Ivancic <[email-masked]>
Date: Wed, 28 Mar 2018 14:14:34 -0400

I also did no not recall and discussion on the list regarding eliminating reactive fragmentation.   IMHO, TCP-CL without reactive fragmentation will eliminate many use cases.  

Will 

=======

On 3/28/18, 12:16 PM, "dtn on behalf of Clark, Gilbert J. (GRC-LCA0)" <[email-masked] on behalf of [email-masked]> wrote:

    
    Cheers,
    Gilbert

    Hi Gilbert,

    
    Cheers,
    
    Rick

    On Wed, 2018-03-28 at 15:11 +0000, Clark, Gilbert J. (GRC-LCA0) wrote:
    > From: dtn <[email-masked]> on behalf of Brian Sipos <BSipos@rkf-eng..com>

---

From: "Burleigh, Scott C (312B)" <[email-masked]>
Date: Wed, 28 Mar 2018 19:05:57 +0000

Hi, Will.  In practice, I don't think that many important use cases will be eliminated; I don't know of any operational deployments of DTN that currently rely on reactive fragmentation.  But as Marc points out, what we are talking about is defining reactive fragmentation in a separate document, as a TCPCL extension, not discarding it.  This will let us address its complexities in a focused way, without the distraction of other TCPCL design issues and without delaying TCPCL (and BPbis and bpsec) standardization.  I think it's the best move at this point.

Scott

-----Original Message-----
From: William Ivancic <[email-masked]> 
Sent: Wednesday, March 28, 2018 11:15 AM
To: Clark, Gilbert J. (GRC-LCA0) <[email-masked]>; Rick Taylor <[email-masked]>; [email-masked]; Burleigh, Scott C (312B) <[email-masked]>; [email-masked]
Subject: Re: [dtn] TCPCLv4 -07 updates and simplification

Will 

=======

    
    
    Cheers,
    Gilbert

    Hi Gilbert,

    
    Cheers,
    
    Rick

---

From: "R. Atkinson" <[email-masked]>
Date: Wed, 28 Mar 2018 17:58:43 -0400

> On Mar 28, 2018, at 12:16, Clark, Gilbert J. (GRC-LCA0) <[email-masked]> wrote:
> 

It is not merely an “impression”.

IETF process rules are very clear that no matter what an in-person meeting
consensus might be, it does __not__ count officially.

The WG Chairs are _required_ to solicit an official consensus call on the 
WG mailing list AND the mailing list consensus is the official one even if
different from the in-person meeting.

Yours,

Ran

---

From: "Marc Blanchet" <[email-masked]>
Date: Wed, 28 Mar 2018 18:06:00 -0400

On 28 Mar 2018, at 17:58, R. Atkinson wrote:

>> That is extremely unfortunate.  I will note that I was under the 
>> impression that such decisions normally would go through the list in 
>> addition to the meeting.
>

Ran, please read my response to that post.

Marc.

---

From: "Marc Blanchet" <[email-masked]>
Date: Wed, 28 Mar 2018 18:54:17 -0400

(resending as my original message was not using the right From address)

> impression that such decisions normally would go through the list in 

sure. we are awaiting meeting minutes so that we can then ask for 
confirmation of « decisions » but the thread started faster!

>
> I will note that this decision does leave tcpcl unsuitable for my 
> immediate use cases,

important to know.

> so ... my objections are withdrawn, and I'll simply need to come up 
> with something else entirely :)

note that the concensus was to put it aside as a separate document. It 
does not say that we remove the functionality. We did similarly with 
custody transfer with BP.

Marc.

>
> Cheers,
> Gilbert
>
> The views expressed in this mail reflect the opinions of the author.  
> They  are, therefore, not intended to reflect official positions of 
> NASA or the U.S.
> Government.
> To: [email-masked]; Clark, Gilbert J. (GRC-LCA0); Burleigh, Scott C 
> (JPL-312B)[Jet Propulsion Laboratory]; [email-masked]

---

From: "Marc Blanchet" <[email-masked]>
Date: Wed, 28 Mar 2018 18:56:41 -0400

> Ran, please read my response to that post.

sorry. someone told me that he did not find my response on the mailing 
list. He was right. It was sent by my mail client with the wrong From 
address, so never hit the mailing list. So resent it.

Marc.