Subject: Re: [IPsec] iptfs publication request

From: Lou Berger <[email-masked]>
Date: Tue, 2 Nov 2021 07:49:39 -0400

I'd prefer to see the SHOULD and MAY reversed -- intentionally 
introducing additional reordering is generally considered something to 
avoid.  I'd also be fine with both being a MAY and a recommendation for 
this to be configurable.

Lou

On 10/31/2021 3:53 PM, Michael Richardson wrote:
> Tero Kivinen <[email-masked]> wrote:
>      > Christian Hopps writes:
>      >> Will you be able to provide the text changes that would cover the
>      >> issue you have? Would really like to get this submitted to IESG
>      >> before another IETF cycle completes.
>
>      > How about following:
>
> works for me.
>
> _______________________________________________
> IPsec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipsec

---

From: Christian Hopps <[email-masked]>
Date: Thu, 4 Nov 2021 06:28:50 -0400

I agree with these suggestions.

Tero?

Thanks,
Chris.

> On Nov 2, 2021, at 7:49 AM, Lou Berger <[email-masked]> wrote:
> 
> I'd prefer to see the SHOULD and MAY reversed -- intentionally introducing additional reordering is generally considered something to avoid.  I'd also be fine with both being a MAY and a recommendation for this to be configurable.
> 
> Lou
>

---

From: Tero Kivinen <[email-masked]>
Date: Fri, 5 Nov 2021 21:09:30 +0200

Lou Berger writes:
> avoid.

Yes, intentionally introducing reordering or delay SHOULD be avoided,
thats why it is important to keep the SHOULD and MAY as they are.

The current text does NOT add any reordering, it just does NOT fix the
reordering that has already happened. I.e., if the outer packets come
in-order there is no reordering happening. The question is what
happens when there is reordering happening in the outer packets.
Keeping the SHOULD in first case says that we do not want to
intentionally add any new delay, thus we process the outer packets
immediately when they come in, but to do that we allow the reordering
that happened in the outer packets to be propagated to the inner
packets too. 

> I'd also be fine with both being a MAY and a recommendation for 
> this to be configurable.

I strongly belive that we should not intentionally add reordering OR
delay, and we SHOULD try to keep the conditions as they are, i.e., if
there is reordering happening in the outer packets, lets allow it to
be propagated to inner packets.

If there is lost packet in the outer packets we SHOULD NOT cause extra
delay when trying to fix reordering... I myself think that lost
packets are much more common than reordered packets, and thus causing
extra delays for each lost packets just to be able to fix reordering
cases is much worse than allowing reordering to go through.
-- 
[email-masked]

---

From: Lou Berger <[email-masked]>
Date: Sun, 7 Nov 2021 12:18:23 -0500

Hi Tero,

see below

On 11/5/2021 3:09 PM, Tero Kivinen wrote:
> in-order there is no reordering happening.

Following this SHOULD will actually *amplify* reordering as seen by the 
end user/application.

Consider a TFS tunnel that aggregates (on average) 3 inner packets.  A 
single tunnel reorder (packet O2 followed by packet O1 in reception 
sequence, i.e. [O2, O1]) will result in each of the miss-ordered inner 
packets being N packets out of order, (for a resulting example sequence 
of [I4,I5,I6], [I1,I2,I3]).  A miss-order of 2 outer packets, would 
result in a 2N miss-order (O2,O3,O1 --> [I4,I5,I6], [I7,I8,I9], 
[I1,I2,I3]).

So in this case a *SHOULD* is likely to result in significant change in 
the number of out of order packets seen in any application that doesn't 
use large packets -- and this might break such applications.  For such 
applications seeing added delay may also have negative impact, but 
variable latency on the internet is a pretty normal event that 
applications already have to allow for.

> The question is what

I think adding delay is safe for general internet applications, i.e., 
existing end to end applications will continue to function albeit with, 
perhaps, more jitter due to intermediate reordering. Adding more 
reordering is something that applications may not be able to tolerate, 
so this seems quite risky as the RECOMMENDED option.

FWIW I'm basing my comments on my routing area experience where a huge 
amount of work has been put into maintaining ordering experienced by 
user traffic at significant implementation expense, i.e., in support of 
ECMP and other multipath solutions in protocols and hardware.

> cases is much worse than allowing reordering to go through.

I think this really depends on your operational environment.  If you are 
operating over fiber networks, loss really only happens due to 
congestion and in many place is pretty rare.  On the other hand if your 
tunnels are operating over wireless networks, loss will be quite 
common.  So we may really be heading towards a deployment choice, with 
the recommendation to do both and support configuration.

Lou

---

From: Tero Kivinen <[email-masked]>
Date: Mon, 8 Nov 2021 09:17:50 +0200

> Following this SHOULD will actually *amplify* reordering as seen by the 

Nope. It keeps the same reordering that has already happened and does
not fix it. IPsec does not try to provide ordered delivery, that is
not something it is supposed to provide, and thats why IPTFS should
not even try to provide something that is not requested.

I.e., if the two outer frames are already coming in to the gateway in
different order when they were sent, that means reordering has already
happened. When the gateway processes them, if there is no ipsec or
IPsec without IPTFS it will simply forward those frame in the order
they are received. Doing same when the IPTFS is enabled does not make
things worse, it simply keeps the reordering same. 

> result in a 2N miss-order (O2,O3,O1 --> [I4,I5,I6], [I7,I8,I9], 

Yes, if you assume there is only a single ordering happening ever.

If there is certain procentage chance that there will be reordering
happening, then over long run the effect is same. I.e., i.e. if there
is 1% chance that two packets are reordered and we send 100 aggregated
packets each containing 3 packets, one of them might get reordered,
thus 3 packets out of 300 inner packets will get reordered. If we send
those 300 packets separately 3 of them on average will get reordered,
which is same amount.

> applications already have to allow for.

On the other hand I would assume that lost packets are more frequent
than reordered packets. Each lost frame will cause delay that is same
as reordering timer, and if we assume that reordering is happening
because of multipath over the internet then I would assume the reorder
window should need to be large enough to cover route transit time
differences between those two paths.

What would you expect to be suitable time for it? My feeleing would be
that few seconds or so, but if we add few seconds delay for every
single lost frame that is way too much. On the other hand having
reordering window only 10 milliseconds long does not really allow
that much time difference between routes.

And the issue is that if this is really configurable value the
adminstrator setting this timer might not realize that making it
larger also causes similar delay for every lost frame...

> so this seems quite risky as the RECOMMENDED option.

Internet applications should work both with delay and reordering (and
lost frames), but all of them should be kept minimal by network, and I
think we should not add any of them more than is needed. Thats why
recommending the processing which do not add delay and keeps the
reordering properties same is best.

> ECMP and other multipath solutions in protocols and hardware.

So you can probably then tell me what is the expected time delay
between two reordered frames, and how often that is happening in
general. And compare that to how often you have frames dropped.

My normal feeling is that reordering is not very common, but over last
mile copper, and especially over wireless links (4G etc) there can be
significant (up to 10%) packet drop chance, especially when the link
quality is bad. And usually the direction of packet makes huge
difference, i.e., uplink is quite often much worse than downlink.

> the recommendation to do both and support configuration.

And what would be your recommendations for the drop timer values for
different networks? I have no idea what would be suitable values for
it, and I would expect that most adminstrators are in same situation.
-- 
[email-masked]