Subject: Re: [IPsec] iptfs publication request

From: Christian Hopps <[email-masked]>
Date: Fri, 03 Sep 2021 08:51:59 -0400

Christian Hopps <[email-masked]> writes:
>
> I'm saying we should add new text that mentions the use of this drop timer to
> drop missing packets after a short waiting time instead of just waiting for it
> to slide out of the reorder window. Then there is no issue to discuss anymore
> AFAICT.

A new version has been published which modifies the text in question and changes it to RECOMMENDING the use of a drop timer.

I believe this is a good change, and addresses Tero's concern about long delays if a timer is not used.

New text follows:

   Please note when IP-TFS sends a continuous stream of packets, there
   is no requirement for an explicit drop timer; however, using a drop
   timer is RECOMMENDED.  If an implementation does not use a drop timer
   and only considers an outer packet lost when the reorder window moves
   by it, the inner traffic can be delayed by up to the reorder window
   size times the per packet send rate.  This amount of delay could be
   significant for slower send rates or when larger reorder window sizes
   are in use.

Thanks,
Chris.

---

From: Tero Kivinen <[email-masked]>
Date: Mon, 6 Sep 2021 22:06:51 +0300

Christian Hopps writes:
> > I'm saying we should add new text that mentions the use of this
> > drop timer to drop missing packets after a short waiting time
> > instead of just waiting for it to slide out of the reorder window.
> > Then there is no issue to discuss anymore AFAICT.
> 
> A new version has been published which modifies the text in question
> and changes it to RECOMMENDING the use of a drop timer.
> 

No it does only notes there is an issue, and proposes a partial
solution to it, but still does not really address it.

I still do not understand why you insist on in order processing of the
outer frames?

I mean IPsec is never ever tried to protect you from the packet
reordering, but IPTFS adds this new protection and this causes delays
and dropped frames even at the small reordering.

Why do you not want to allow processing outer frames as they come in,
and sending all the fully reassmebled (or fully received) inner frames
out as soon as you have them in your posession?

I.e., if you say receiver is allowed to send fully received inner
frames out as soon as they are fully received, and is allowed to
process outer frames in the order they are received that would solve
the problem.

I mean that also solves the memory requirement, as you do not need to
keep all the packets in the reorder window in the memory, you only
need to keep the those parts of the outer frames which have not yet
been able to be reassembled in memory, and immediately when you can
reassmeble the frame, you can do that, send it out and free the
memory.

IPsec replay protection will guarantee that you are not seeing any of
those outer frames again, even if attacker would retransmit them, so
there is no need to keep outer frames which have been fully processed
in the memory at all.

With your new text you are just limiting the number of frames you need
to keep in the memory to smaller amount by setting the reorder window
smaller (drop timers just make reorder window smaller with IP-TFS as
packets are sent as continuous stream).

The the issue is still in the text which says:

   Finally, the receiver processes the now in-order AGGFRAG_PAYLOAD
   payload stream to extract the inner-packets (Section 2.2.3,
   Section 6.1).

I.e., the one which do require in-order processing of AGGFRAG_PAYLOAD
payloads.

I.e., looking at the Appendix A, if the ESP1 is lost (or delayed
because of reordering), and we see the ESP2 packet first, why the
received cannot send the 60 octet and 240 octet inner frames out
immediately when ESP2 frame is received, and keep the first 100 bytes
of the ESP2 in memory just in case the ESP1 packet is received later
(and of course also keep the last 1100 at the end of ESP2 frame i.e.,
the beginning of the 4000 octet frame as that is not fully received at
that point).

Then if we next received ESP3, and ESP4, then we can forawrd the 4000
octet frame to the network (and after that we only still need to keep
the 100 bytes in the reassembly buffer, we can free everything else
from the queue). If we then happen to receive the ESP1 (which was not
dropped, only reordered), we can then forward the first 800 octet
frame out, and reassemble the second 800 octet frame and send that out
also to the network.

>    significant for slower send rates or when larger reorder window sizes

I think the amout of delay will also completely mess up any kind of
video and audio streams, and most likely mess up the TCP
retransmission and ack timers.

And even when we do have the congestion information inside the IP-TFS,
I do not see any way we can actually transmit that information from
the security gateway doing the IP-TFS to the actual end node that
would need to get that information, thus the congestion information is
just used to limit the IP-TFS data rate, it does not help at all for
traffic inside the tunnel.

Note, that this issue was also pointed out as the "most significant
concern" in transport area review:

https://datatracker.ietf.org/doc/review-ietf-ipsecme-iptfs-03-tsvart-early-touch-2020-12-03/

    The most significant concern is not at the transport layer – it is
    the assumption of in-order delivery and insufficient consideration
    of the impact of loss at the network layer. Loss could orphan
    received fragments – for which a timeout should be recommended.
    Loss or reordering could generate faulty reassembly at the egress
    – which is actually very likely here, e.g., when a short packet is
    followed by a sequence of packets that are exactly the tunnel MAP
    (as defined in draft-ietf-intarea-tunnels). As noted below,
    persistent fragmentation could make this situation worse, esp. in
    the presence of frequent reordering or loss.

I do not want to go back to see old diffs before this review, but how
did you address that concern from the tsv-review. I just see that
there was a text section addded which do say we do order frames before
doing reassmebly process, i.e., adding in-order processing requirement
(which is still needed for frames which cross multiple frames), but
having some restrictions to frames which are already fully received
inside the inner frame will mess up things.
-- 
[email-masked]

---

From: Christian Hopps <[email-masked]>
Date: Mon, 06 Sep 2021 15:08:40 -0400

> I still do not understand why you insist on in order processing of the

Because IMO this is one of those micro-optimizations for a pathalogical case that are never worth the complexity that they add.

If the packets are coming in out of order then even if they were *not* in this tunnel they would be being delivered down stream out of order and thus they would be DELAYED. All that we are doing, again for a pathalogical case, is adding a bit of bursty-ness to things.

But more importantly, you are optimizing for a case that is simply NOT going to happen in the real world. The reason packets get out of order is b/c you have multiple paths along the way. *ALL* modern routers at LEAST look at source and destination IP when they do ECMP selection. They do this exactly so that things like tunnels don't get delivered out of order!

At this point please suggest the text that you would like changed or added to this document. I object to having to add it, but you are the chair and you are blocking this document so we must accept.

Thanks,
Chris.

---

From: Tero Kivinen <[email-masked]>
Date: Mon, 6 Sep 2021 23:16:24 +0300

Christian Hopps writes:
> Because IMO this is one of those micro-optimizations for a
> pathalogical case that are never worth the complexity that they add.

I think dropped packets and even reordering is thing that will happen
on the internet. I can see you assume that those are rare cases, but
if you try to work over mobile networks those things happen very
often. Also at least in Finland it seems to be harder and harder to
get copper or fiber, ISPs usually just offer mobile connections
nowadays.

Of course if this is only to be used between security gateways between
two datacenters, or similar, then good networks might be something
that can be expected, on the other hand when you are talking about
long distance links, even they are not that good, and I do see packets
dropping in those situations too. 

> If the packets are coming in out of order then even if they were
> *not* in this tunnel they would be being delivered down stream out
> of order and thus they would be DELAYED. All that we are doing,
> again for a pathalogical case, is adding a bit of bursty-ness to
> things.

No, one delayed packet does not delay any of the other frames in
normal case.

I.e., if I have audio stream with packets A1, A2, A3, A4... A50, etc,
and one of them gets delayed (or dropped), i.e., lets say A3 gets hold
up by some buffering or got dropped. this means I do get A1, A2, A4,
A5 ... A50 to the final destination and audio codecs usually take care
of that and you do not loose that much of audio. Going over IP-TFS
with replay and reorder window of 32, I get A1, A2, and then 32 outer
IP-TFS frame long delay and then my end node gets burst of frames from
A4..A32 (if we assume IP-TFS frame rate is same as audio frame rate).
Now end node can either delay the whole audio link by 32-frames so
there is extra delay on the link, or it needs to throw away frames
A4..A32, as they are useless to him as they arrived too late.

> But more importantly, you are optimizing for a case that is simply
> NOT going to happen in the real world.

I tell you packets are lost in real world. I do not know in which
world you live, but I do see packets getting dropped quite often,
especially over mobile links. 

> The reason packets get out of order is b/c you have multiple paths
> along the way. *ALL* modern routers at LEAST look at source and
> destination IP when they do ECMP selection. They do this exactly so
> that things like tunnels don't get delivered out of order!

This same thing happens also when you have any dropped frames. It is
not only the case about the reordered frames. Yes, reordering happens
much less frequently but even that can happen, but this issue is
exactly same for each dropped frame.

> At this point please suggest the text that you would like changed or
> added to this document. I object to having to add it, but you are
> the chair and you are blocking this document so we must accept.

I am not talking here as chair, I am talking here as an developer who
see no point of making crappy protocol. Thats why I said you either
need to properly explain what are the real effects of requring all
outer packets to processed in-order (and combined that with the case
that packets are assumed to be lost only after they drop out from the
re-ordering buffer, which means every single dropped frame completely
stalls the in-order processing of outer frames), or I would prefer
that you remove the in-order processing requirement and allow making
usable and efficient protocol. 

I would expect that we would get similar comments from others also
during the last call or even during the IESG telechat, and fixing
those things during the working group processing usually makes things
faster. 
-- 
[email-masked]

---

From: Graham Bartlett <[email-masked]>
Date: Mon, 6 Sep 2021 21:18:08 +0100

Hi

>From my experience there's lots of network devices that can easily process
traffic out of order.

If you have multiple cores, different packets can be anchored to different
cores. If one core is busier this can result in backpressure and traffic
lost or delayed.

I've seen other devices that supposedly guarantee packets to flow out, in
the order they flowed in, but in reality they didn't.. Wireshark captures
either side of the device proving this.

I spent nearly 10 years in professional services, working with customers
and saw this more than not..

cheers

On Mon, Sep 6, 2021 at 8:17 PM Christian Hopps <[email-masked]> wrote:

> Because IMO this is one of those micro-optimizations for a pathalogical
> case that are never worth the complexity that they add.
>
> If the packets are coming in out of order then even if they were *not* in
> this tunnel they would be being delivered down stream out of order and thus
> they would be DELAYED. All that we are doing, again for a pathalogical
> case, is adding a bit of bursty-ness to things.
>
> But more importantly, you are optimizing for a case that is simply NOT
> going to happen in the real world. The reason packets get out of order is
> b/c you have multiple paths along the way. *ALL* modern routers at LEAST
> look at source and destination IP when they do ECMP selection. They do this
> exactly so that things like tunnels don't get delivered out of order!
>
> At this point please suggest the text that you would like changed or added
> to this document. I object to having to add it, but you are the chair and
> you are blocking this document so we must accept.
>
> Thanks,
> Chris.
>
> _______________________________________________
> IPsec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipsec
>

---

From: Christian Hopps <[email-masked]>
Date: Mon, 6 Sep 2021 16:27:20 -0400

> On Sep 6, 2021, at 4:16 PM, Tero Kivinen <[email-masked]> wrote:
> 
> Christian Hopps writes:

You are asking for the changes after WGLC before you send the document to the IESG otherwise it would already be at the IESG. This is specifically a chair-only action you are taking.

Look, it sounds like you have some idea for a simple change that would satisfy you, I’ve tried to address your concerns, but I keep falling short.

Please, instead of writing another long email, just propose the change you want and we will put it in the document, and we can move forward.

Thank you,
Chris.

---

From: Paul Wouters <[email-masked]>
Date: Tue, 7 Sep 2021 10:13:04 +0300

On Mon, Sep 6, 2021 at 11:27 PM Christian Hopps <[email-masked]> wrote:

Christian Hopps writes:

You are asking for the changes after WGLC before you send the document to
the IESG otherwise it would already be at the IESG. This is specifically a
chair-only action you are taking.

Process note: Tero has been vocal about this issue all through the process.
I do not think it is fair to say this appeared after WGLC. And it was also
raised by the transport area expert review.

I'd prefer to see this discussed out properly. Not just "submit text to
make the document go forward". It seems rather a core issue of the protocol.

Paul

---

From: Christian Hopps <[email-masked]>
Date: Tue, 7 Sep 2021 03:43:21 -0400

> On Sep 7, 2021, at 3:13 AM, Paul Wouters <[email-masked]> wrote:
> 
> 
> On Mon, Sep 6, 2021 at 11:27 PM Christian Hopps <[email-masked] <mailto:[email-masked]>> wrote:
>> On Sep 6, 2021, at 4:16 PM, Tero Kivinen <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Christian Hopps writes:
> Process note: Tero has been vocal about this issue all through the process. I do not think it is fair to say this appeared after WGLC. And it was also raised by the transport area expert review.

Unfortunately this is not true. This point was only raised after WGLC was completed (otherwise it wouldn’t have completed :) and after I submitted the shepherd write-up. The fact that the document has been blocked on this for more than 1 IETF cycle is perhaps creating some fog here.

> I'd prefer to see this discussed out properly. Not just "submit text to make the document go forward". It seems rather a core issue of the protocol.

It was resolved with the transport area by adding the drop timer. I added more text to highlight and even recommend the drop timer in the latest revision, but it is apparently not enough.

I believe Tero really wants to allow for sending packets as soon as he can, even if it means complexity in the implementation (keeping 1/2 processed outer packets around), I believe this is unneeded, but see no reason to disallow it either — and certainly not to block the document.

I’m just asking Tero to provide text that will make him happy, rather than keep going around and around with me not getting the text correct.

Thanks,
Chris.

---

From: "Bottorff, Paul" <[email-masked]>
Date: Tue, 7 Sep 2021 17:59:07 +0000

Hi:

I’d agree that out of order handling is an important aspect of a protocol to make it viable in multi-pathing environments. Multi-pathing attempts to find enough entropy in the outer packet header to distribute over multi-path meshes. It is not a pathological case for a tunnel to be distributed over multiple paths. For instance the multi-tenant encapsulations VxLAN, GENEVE, and GUE are using source port to add entropy to their tunnels specifically to allow the tunnel packets to be distributed over multiple paths. With these protocols the flows encapsulated in the tunnel are kept on the same path while the tunnel is spread over multiple paths.

Cheers,

Paul

From: IPsec [mailto:[email-masked]] On Behalf Of Graham Bartlett
Sent: Monday, September 6, 2021 1:18 PM
To: Christian Hopps <[email-masked]>
Cc: Koning, Paul <[email-masked]>; [email-masked]; [email-masked]; Tero Kivinen <[email-masked]>
Subject: Re: [IPsec] iptfs publication request

Hi

From my experience there's lots of network devices that can easily process traffic out of order.

If you have multiple cores, different packets can be anchored to different cores. If one core is busier this can result in backpressure and traffic lost or delayed.

I've seen other devices that supposedly guarantee packets to flow out, in the order they flowed in, but in reality they didn't.. Wireshark captures either side of the device proving this.

I spent nearly 10 years in professional services, working with customers and saw this more than not..

cheers

On Mon, Sep 6, 2021 at 8:17 PM Christian Hopps <[email-masked]<mailto:[email-masked]>> wrote:

Tero Kivinen <[email-masked]<mailto:[email-masked]>> writes:

> Christian Hopps writes:
>>
>> Christian Hopps <[email-masked]<mailto:[email-masked]>> writes:
>> >

Thanks,
Chris.

> https://datatracker.ietf.org/doc/review-ietf-ipsecme-iptfs-03-tsvart-early-touch-2020-12-03/<https://datatracker.ietf.org/doc/review-ietf-ipsecme-iptfs-03-tsvart-early-touch-2020-12-03/>
>

_______________________________________________
IPsec mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ipsec<https://www.ietf.org/mailman/listinfo/ipsec>

---

From: Christian Hopps <[email-masked]>
Date: Tue, 7 Sep 2021 16:41:16 -0400

We *do* deal with re-ordering so I should have just left that alone — I’m sorry I said anything and made it a distraction.

In any case I suspect that Tero can pretty easily provide the change he’s looking for and we can move forward.

Thanks,
Chris.

P.S. There’s a big difference between GRE and IP-TFS is that IP-TFS packets are paced and exactly evenly apart from each other. If you are getting packets out of order then you have very large differences between RTTs (relative to the tunnel speed). I’m not saying it can’t happen, but it’s not going to be the “normal" re-ordering along ECMP (they are supposed to be equal-cost :) even when the hash algorithms fail you or the routers otherwise make really poor ECMP choices.

> On Sep 7, 2021, at 1:59 PM, Bottorff, Paul <[email-masked]> wrote:
> 
> Hi:
>  
> On Mon, Sep 6, 2021 at 8:17 PM Christian Hopps <[email-masked] <mailto:[email-masked]>> wrote:
> 
> Tero Kivinen <[email-masked] <mailto:[email-masked]>> writes:
> 
> > Christian Hopps writes:
> >>
> >> Christian Hopps <[email-masked] <mailto:[email-masked]>> writes:
> >> >
> > https://datatracker.ietf.org/doc/review-ietf-ipsecme-iptfs-03-tsvart-early-touch-2020-12-03/ <https://datatracker.ietf.org/doc/review-ietf-ipsecme-iptfs-03-tsvart-early-touch-2020-12-03/>
> >
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/ipsec <https://www.ietf.org/mailman/listinfo/ipsec>