Subject: Re: [dtn] TCPCLv4 -07 updates and simplification

From: "Clark, Gilbert J. (GRC-LCA0)" <[email-masked]>
Date: Wed, 28 Mar 2018 12:39:18 +0000

Maybe I’m misunderstanding, but I think the issue here is more about multiplexing data streams across the TCPCL.

The issue I understand here is that, assuming one has a single massive bundle in the approach proposed, one would need to wait for that bundle to be *completely* sent before other messages could be sent … whereas segments allow TCPCL-based links to be more evenly (and arguably fairly) utilized.  No matter the size of the bundle passing through the CL, the segments would ensure that TCPCL could schedule other messages in-between large bundles.

Segments avoid the case where e.g. someone needs to send a small (3 KB) message, but that message is generated halfway through the transmission of a 3 GB bundle through a TCPCL running across a 30 Kbps downlink.  Assuming an 8K segment size, one would have to wait at most around 2.2 seconds to get that small 3 KB message scheduled across the link (vs. up to something like 10 days in the event one must wait for the entire 3 GB bundle to be transferred before that little 3 KB message can).

-Gilbert

The views expressed in this mail reflect the opinions of the author.  They are, therefore, not intended to reflect official positions of NASA or the U.S. Government.

From: dtn <[email-masked]> on behalf of "Burleigh, Scott C (312B)" <[email-masked]>
Date: Wednesday, March 28, 2018 at 1:56 AM
To: Brian Sipos <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [dtn] TCPCLv4 -07 updates and simplification

Brian, is it possible that this is an implementation matter?  I am thinking that the sending node could have one thread continuously transmitting segment data over the socket while another thread running concurrently could be reading XFER_ACK messages (sent by the receiving node) from the same socket.  Am I overlooking some reason this won’t work?

Scott

From: dtn <[email-masked]> On Behalf Of Brian Sipos
Sent: Monday, March 26, 2018 12:36 PM
To: [email-masked]
Subject: [dtn] TCPCLv4 -07 updates and simplification

All,

After looking into simplifying TCPCLv4 as discussed at the IETF, I see that if a transfer is always simplified into a single XFER_INIT message followed by a single "segment" of data this effectively turns the TCPCL session unidirectional by preventing any other XFER_ACK messages from acknowledging transfers in the other direction within the session. One side effect of segmented transfer data is to allow other messages from the transmitting node between segments by avoiding monopolizing the TCP connection with a single huge message.

I feel like segmentation, as an optional behavior as currently specified, gives the network / node management a way to avoid this kind of message queue blocking behavior and allows more deterministic timing of when an ACK can be expected based on path delays.

For a simple-as-possible workflow, an implementation could choose to disallow a Segment MRU any smaller than the Transfer MRU and could choose to always transfer within a single segment which would minimize the message framing overhead. If interoperability was not necessary then it could also refuse any transfer that includes a segment smaller than the transfer size, so that there would be no need for receive segment concatenation.

Any thoughs on keeping segmentation as a negotiated behavior in order to allow for better administrative control over worst-case session timing?

---

From: Rick Taylor <[email-masked]>
Date: Wed, 28 Mar 2018 13:54:45 +0000

As I see it, the segmentation allows 'full-duplex' transmission of
bundles.

Without segmentation, a receiver cannot send an ACK to begin/end a
bundle transmission while it is sending its own bundle the other way. 
It has to wait until it's current transmission is complete before
sending an ACK, due to the pipelined nature of TCP.  Threads don't help
here Scott, the pipe is busy.

So, my questions to the working group are:

1. Do we like full-duplex behaviour so much that we are willing to
accept some kind of segmentation complexity in TCP-CL?

2. Can we live with half-duplex behaviour, as it simplifies the
protocol?

3. Do we want to strip TCP-CL all the way back to uni-directional: one
end sends bundles, the other end sends ACKs?  If you want
bidirectionality, open another connection the other way.

Options I have excluded:

a) Multi-streaming:  Yes, this can be made to work, but at it's
basically a variation on segmentation.

b) SCTP: This is a different CL.  If someone wants to write a draft, it
would be welcome, and perhaps QUIC is also worth a look.

c) Rsync-style chunking:  This should be a different CL in my mind. 
Again, if anyone fancies knocking up a quick draft on how to use
rsysnc, I imagine it would be interesting.  (Good student/intern
project?)

On Wed, 2018-03-28 at 12:39 +0000, Clark, Gilbert J. (GRC-LCA0) wrote:
> Maybe I’m misunderstanding, but I think the issue here is more about
> multiplexing data streams across the TCPCL. 
>  
> The issue I understand here is that, assuming one has a single
> massive bundle in the approach proposed, one would need to wait for
> that bundle to be *completely* sent before other messages could be
> sent … whereas segments allow TCPCL-based links to be more evenly
> (and arguably fairly) utilized.  No matter the size of the bundle
> passing through the CL, the segments would ensure that TCPCL could
> schedule other messages in-between large bundles. 
>  
> Segments avoid the case where e.g. someone needs to send a small (3
> KB) message, but that message is generated halfway through the
> transmission of a 3 GB bundle through a TCPCL running across a 30
> Kbps downlink.  Assuming an 8K segment size, one would have to wait
> at most around 2.2 seconds to get that small 3 KB message scheduled
> across the link (vs. up to something like 10 days in the event one
> must wait for the entire 3 GB bundle to be transferred before that
> little 3 KB message can).
>  
> -Gilbert
>  
> The views expressed in this mail reflect the opinions of the author. 
> They are, therefore, not intended to reflect official positions of
> NASA or the U.S. Government.
>  
> Brian, is it possible that this is an implementation matter?  I am
> thinking that the sending node could have one thread continuously
> transmitting segment data over the socket while another thread
> running concurrently could be reading XFER_ACK messages (sent by the
> receiving node) from the same socket.  Am I overlooking some reason
> this won’t work?
>  
> Scott
>  
> After looking into simplifying TCPCLv4 as discussed at the IETF, I
> see that if a transfer is always simplified into a single XFER_INIT
> message followed by a single "segment" of data this effectively turns
> the TCPCL session unidirectional by preventing any other XFER_ACK
> messages from acknowledging transfers in the other direction within
> the session. One side effect of segmented transfer data is to allow
> other messages from the transmitting node between segments by
> avoiding monopolizing the TCP connection with a single huge message.
>  
> I feel like segmentation, as an optional behavior as currently
> specified, gives the network / node management a way to avoid this
> kind of message queue blocking behavior and allows more deterministic
> timing of when an ACK can be expected based on path delays.
>  
> For a simple-as-possible workflow, an implementation could choose to
> disallow a Segment MRU any smaller than the Transfer MRU and could
> choose to always transfer within a single segment which would
> minimize the message framing overhead. If interoperability was not
> necessary then it could also refuse any transfer that includes a
> segment smaller than the transfer size, so that there would be no
> need for receive segment concatenation.
>  
> Any thoughs on keeping segmentation as a negotiated behavior in order
> to allow for better administrative control over worst-case session
> timing?
> _______________________________________________
> dtn mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dtn

---

From: "Velt, R. (Ronald) in 't" <[email-masked]>
Date: Wed, 28 Mar 2018 16:26:31 +0000

Rick, all,

> -----Original Message-----
> From: dtn [mailto:[email-masked]] On Behalf Of Rick Taylor
> Sent: woensdag 28 maart 2018 15:55
> To: [email-masked]; [email-masked]; [email-masked];
> [email-masked]
> As I see it, the segmentation allows 'full-duplex' transmission of bundles.
> 
> Without segmentation, a receiver cannot send an ACK to begin/end a bundle
> transmission while it is sending its own bundle the other way.
> It has to wait until it's current transmission is complete before sending an
> ACK, due to the pipelined nature of TCP.  Threads don't help here Scott, the
> pipe is busy.
> 
> 1. Do we like full-duplex behaviour so much that we are willing to accept
> some kind of segmentation complexity in TCP-CL?

Yes, in order to maximize the use of potentially limited contact time between nodes.

> 
> 2. Can we live with half-duplex behaviour, as it simplifies the protocol?

No 😊

> 
> 3. Do we want to strip TCP-CL all the way back to uni-directional: one end
> sends bundles, the other end sends ACKs?  If you want bidirectionality, open
> another connection the other way.

In my view, that amounts to shifting the complexity elsewhere.

> 
> Options I have excluded:
> 
> a) Multi-streaming:  Yes, this can be made to work, but at it's basically a
> variation on segmentation.
> 
> b) SCTP: This is a different CL.  If someone wants to write a draft, it would be
> welcome, and perhaps QUIC is also worth a look.
> 
> Again, if anyone fancies knocking up a quick draft on how to use rsysnc, I
> imagine it would be interesting.  (Good student/intern
> project?)

Is there any point in asking the authors of RFC 7242 (to the extent they are still reachable, which should not be a problem with Simon Perreault and Joerg Ott) what the intended purpose of segmentation was?

BR,
Ronald

> > The issue I understand here is that, assuming one has a single massive
> > bundle in the approach proposed, one would need to wait for that
> > bundle to be *completely* sent before other messages could be sent …
> > whereas segments allow TCPCL-based links to be more evenly (and
> > arguably fairly) utilized.  No matter the size of the bundle passing
> > through the CL, the segments would ensure that TCPCL could schedule
> > other messages in-between large bundles.
> >
> > transmission of a 3 GB bundle through a TCPCL running across a 30 Kbps
> > downlink.  Assuming an 8K segment size, one would have to wait at most
> > around 2.2 seconds to get that small 3 KB message scheduled across the
> > link (vs. up to something like 10 days in the event one must wait for
> > the entire 3 GB bundle to be transferred before that little 3 KB
> > message can).
> >
> > -Gilbert
> >
> > transmitting segment data over the socket while another thread running
> > concurrently could be reading XFER_ACK messages (sent by the receiving
> > node) from the same socket.  Am I overlooking some reason this won’t
> > work?
> >
> > Scott
> >
> > After looking into simplifying TCPCLv4 as discussed at the IETF, I see
> > that if a transfer is always simplified into a single XFER_INIT
> > other messages from the transmitting node between segments by avoiding
> > monopolizing the TCP connection with a single huge message.
> >
> > choose to always transfer within a single segment which would minimize
> > the message framing overhead. If interoperability was not necessary
> > then it could also refuse any transfer that includes a segment smaller
> > than the transfer size, so that there would be no need for receive
> > segment concatenation.
> >
This message may contain information that is not intended for you. If you are not the addressee or if this message was sent to you by mistake, you are requested to inform the sender and delete the message. TNO accepts no liability for the content of this e-mail, for the manner in which you use it and for damage of any kind resulting from the risks inherent to the electronic transmission of messages.

---

From: Stan Ratliff <[email-masked]>
Date: Wed, 28 Mar 2018 13:04:43 -0400

On Wed, Mar 28, 2018 at 12:26 PM, Velt, R. (Ronald) in 't <
[email-masked]> wrote:

> sending an

My guess is that I'm way late to the conversation, and this has already
been thrashed out, but I've gotta ask:  Why be concerned about
application-level ACKs when this is built on top of a reliable, "guaranteed
delivery" Layer 4 like TCP? I think that the TCP window size could be
manipulated (for example, set to, say, 8K) in order to drive intermediate
ACKs, and also "open the pipe" for bi-directional data flow. If the concern
is the consistency of packet flow across multiple CLs, (again, for example,
we want the ACKs in TCPCL because there are other, unreliable transports
being considered, and we'd like the protocol to be 'the same'), I'd argue
that ACKs are a legitimate extension to a "base protocol" that runs on
transports with guaranteed delivery.

My apologies if this is a naive question.

Regards,
Stan

> end
> would be
> Is there any point in asking the authors of RFC 7242 (to the extent they
> are still reachable, which should not be a problem with Simon Perreault and
> Joerg Ott) what the intended purpose of segmentation was?
>
> BR,
> Ronald
>
> >
> >
> >
> >
> This message may contain information that is not intended for you. If you
> are not the addressee or if this message was sent to you by mistake, you
> are requested to inform the sender and delete the message. TNO accepts no
> liability for the content of this e-mail, for the manner in which you use
> it and for damage of any kind resulting from the risks inherent to the
> electronic transmission of messages.

---

From: Brian Sipos <[email-masked]>
Date: Wed, 28 Mar 2018 17:20:53 +0000

Stan,
This had been discussed earlier; the only thing that TCP guarantees is that the stream data makes it into an OS buffer, not into an actual application use. For a reliable BP CL, the guarantee needs to be that the bundle actually made it to the BP agent (and presumably the agent was able to do something with it). This is what XFER_ACK provides.
I'm going to take a note to mention that the XFER_ACK shall only be sent after BP agent inspection and approval of the transfer data.

________________________________
From: Stan Ratliff <[email-masked]>
Sent: Wednesday, March 28, 2018 13:04
To: Velt, R. (Ronald) in 't
Cc: Rick Taylor; [email-masked]; [email-masked]; [email-masked]; Brian Sipos
Subject: Re: [dtn] TCPCLv4 -07 updates and simplification

On Wed, Mar 28, 2018 at 12:26 PM, Velt, R. (Ronald) in 't <[email-masked]<mailto:[email-masked]>> wrote:
Rick, all,

> From: dtn [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Rick Taylor
> To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>;
> [email-masked]<mailto:[email-masked]>

My guess is that I'm way late to the conversation, and this has already been thrashed out, but I've gotta ask:  Why be concerned about application-level ACKs when this is built on top of a reliable, "guaranteed delivery" Layer 4 like TCP? I think that the TCP window size could be manipulated (for example, set to, say, 8K) in order to drive intermediate ACKs, and also "open the pipe" for bi-directional data flow. If the concern is the consistency of packet flow across multiple CLs, (again, for example, we want the ACKs in TCPCL because there are other, unreliable transports being considered, and we'd like the protocol to be 'the same'), I'd argue that ACKs are a legitimate extension to a "base protocol" that runs on transports with guaranteed delivery.

Regards,
Stan

No 😊

BR,
Ronald

> > From: dtn <[email-masked]<mailto:[email-masked]>> on behalf of "Burleigh, Scott C
> > (312B)" <[email-masked]<mailto:[email-masked]>>
> > To: Brian Sipos <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
> > From: dtn <[email-masked]<mailto:[email-masked]>> On Behalf Of Brian Sipos
> > To: [email-masked]<mailto:[email-masked]>
> > [email-masked]<mailto:[email-masked]>
> [email-masked]<mailto:[email-masked]>
This message may contain information that is not intended for you. If you are not the addressee or if this message was sent to you by mistake, you are requested to inform the sender and delete the message. TNO accepts no liability for the content of this e-mail, for the manner in which you use it and for damage of any kind resulting from the risks inherent to the electronic transmission of messages.
_______________________________________________
dtn mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/dtn

---

From: "Burleigh, Scott C (312B)" <[email-masked]>
Date: Wed, 28 Mar 2018 17:50:06 +0000

Here I disagree, Brian.  For a reliable BP CL, the guarantee needs to be that the bundle actually made it to the BP node – the aggregate entity that includes the BP agent, the application agent, and the convergence-layer adapters – not specifically to the BP agent.

All that the CL adapter can be expected to do is say “I got it, and I passed it to the BP agent.”  The BP agent is then responsible, on its own, for examining the received bundle and deciding what to do with it; it’s not required to tell the CL adapter anything about that.

Scott

From: dtn <[email-masked]> On Behalf Of Brian Sipos
Sent: Wednesday, March 28, 2018 10:21 AM
To: Stan Ratliff <[email-masked]>
Cc: [email-masked]
Subject: Re: [dtn] TCPCLv4 -07 updates and simplification

Stan,
This had been discussed earlier; the only thing that TCP guarantees is that the stream data makes it into an OS buffer, not into an actual application use. For a reliable BP CL, the guarantee needs to be that the bundle actually made it to the BP agent (and presumably the agent was able to do something with it). This is what XFER_ACK provides.
I'm going to take a note to mention that the XFER_ACK shall only be sent after BP agent inspection and approval of the transfer data.
________________________________
From: Stan Ratliff <[email-masked]<mailto:[email-masked]>>
Sent: Wednesday, March 28, 2018 13:04
To: Velt, R. (Ronald) in 't
Cc: Rick Taylor; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Brian Sipos
Subject: Re: [dtn] TCPCLv4 -07 updates and simplification

Regards,
Stan

No 😊

BR,
Ronald

---

From: Brian Sipos <[email-masked]>
Date: Wed, 28 Mar 2018 20:18:02 +0000

Scott,

For the TCPCLv3, some use cases of BUNDLE_REFUSE message require (or at least allow) feedback from the BP agent. I beleive that those behaviors have been inherited for v4.

I feel like the CL allowing the agent at least a once-over when the full transfer has completed, but before the XFER_ACK / XFER_REFUSE has been sent is reasonable SHOULD/MAY guidance for an implementation.

________________________________
From: Burleigh, Scott C (312B) <[email-masked]>
Sent: Wednesday, March 28, 2018 1:50:06 PM
To: Brian Sipos; Stan Ratliff
Cc: [email-masked]
Subject: RE: [dtn] TCPCLv4 -07 updates and simplification

Scott

Stan,

Rick, all,

Regards,

Stan

> 2. Can we live with half-duplex behaviour, as it simplifies the protocol?

No ??

BR,
Ronald

---

From: "Burleigh, Scott C (312B)" <[email-masked]>
Date: Wed, 28 Mar 2018 20:41:42 +0000

Thanks, Brian, but I still disagree; I think that sort of tight coupling between BP and TCPCL is both unnecessary (because we have other Bundle-layer mechanisms that enable the receiving BP agent to tell the sending BP agent that the bundle was ditched) and problematic (because it introduces additional complexity and additional behavioral dependencies).

Scott

From: Brian Sipos <[email-masked]>
Sent: Wednesday, March 28, 2018 1:18 PM
To: Burleigh, Scott C (312B) <[email-masked]>; Stan Ratliff <[email-masked]>
Cc: [email-masked]
Subject: Re: [dtn] TCPCLv4 -07 updates and simplification

Scott,

________________________________
From: Burleigh, Scott C (312B) <[email-masked]<mailto:[email-masked]>>
Sent: Wednesday, March 28, 2018 1:50:06 PM
To: Brian Sipos; Stan Ratliff
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: [dtn] TCPCLv4 -07 updates and simplification

Scott

From: dtn <[email-masked]<mailto:[email-masked]>> On Behalf Of Brian Sipos
Sent: Wednesday, March 28, 2018 10:21 AM
To: Stan Ratliff <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [dtn] TCPCLv4 -07 updates and simplification

Stan,

Rick, all,

Regards,

Stan

No 😊

BR,
Ronald