Subject: [tsvwg] Network congestion, dynamic lossless compression?

From: "David G. Pickett" <[email-masked]>
Date: Sat, 3 Aug 2019 17:13:46 +0000 (UTC)

Wandering out of lurker land with an idea to deal dynamically with network congestion:
If a router or similar network point discovers congestion in packets going to a specific next hop or final destination, it might divert the flow of those packets into a compressed stream of max sized packets under a wrapper protocol, so that network congestion from smaller packets and bulk data transfer can be sped up.  The speed of modern processors has so outstripped much of the transmission media that bandwidth and even latency might be improved by speeding up the link.  Lossless compression could be applied without any effect on the transmitted data.  I used to manually do something similar for bulk data flows in the 56K international WAN days: find ...|cpio ...|gzip -9|rsh somewhere 'gunzip|cpio ...", and later with 'ssh -C9' even over local networks.  Since the data is not going anywhere soon, the network node even has time to consider various standard compression algorithms, much like zip does for each file, perhaps even for each packet or IP port.  The compression algorithms, taken from the world of open source, would be supported by all nodes exchanging a supporting protocol version or higher.

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 3 Aug 2019 19:34:00 +0100

> On 3 Aug, 2019, at 6:13 pm, David G. Pickett <dgpickett=[email-masked]> wrote:
> 
> Lossless compression could be applied without any effect on the transmitted data.

The chief problem with that idea is that most data transmitted these days is already compressed and/or encrypted, making it impossible to further compress transparently.  Even Web traffic, which used to be relatively compressible back when PPP options and in-modem compression were more relevant, is now encrypted more often than not (HTTPS replaces HTTP), and the encryption system itself applies compression to reduce exploitable entropy in the plaintext.

In the end, anyone can do better than your suggestion by "just" building a bigger pipe.  But that isn't a solution by itself, because demand grows to meet and exceed supply.  ("The bureaucracy is expanding, to meet the needs of the expanding bureaucracy.")  To actually improve quality of service in the long run means finding ways to reduce latency and packet loss, not to improve capacity.

 - Jonathan Morton

---

From: "David G. Pickett" <[email-masked]>
Date: Sat, 3 Aug 2019 19:02:56 +0000 (UTC)

Jonathan:
Already compressed: You may not have noticed that zip detects not compressible parts and just copies them into the archive.  Similarly, a packet stream compressor could send such blocks on verbatim in the stream, or leave the packets alone entirely.  There is a slight overhead for headers on no-compression data, so the protocol would need a threshold for deciding which path a packet takes.  Backlogs give the node time to detect such and make decisions.  You still get the potential for combining smaller packets into larger ones, gaining a reduction of overhead in many media with larger local link MTU and inter-packet lost media use.  There still needs to be a packet discard mechanism when the compressed flow falls too far behind.  What it gives us is a softer saturation.  And lost packets are not free, as the overhead of retransmission is a cost, as well as the pause in transmission hurting both latency and bandwidth.
Discussion is great: I wonder if the compression choice could also be to use a lower, faster mode when the backlog is light, and a slower, higher compression mode when things get worse?  In some cases, streaming small packets in big packets might increase link speed enough without any compression!
Demand grows: Parkins law does not justify no innovation for improvements.  Softer link saturation means less packet loss, both due to packet discard and overdue packet timeout.  While compression at first glance increases latency, when the link is saturated, the increased bandwidth translates into lower latency as the packet gets sent sooner.  And for momentary overload, the link would switch back to uncompressed packet by packet forwarding as soon as the bubble of load is disposed of, for lowest latency!

There is always room at the top!
Best,
David
-----Original Message-----
From: Jonathan Morton <[email-masked]>
To: David G. Pickett <[email-masked]>
Cc: tsvwg <[email-masked]>
Sent: Sat, Aug 3, 2019 2:34 pm
Subject: Re: [tsvwg] Network congestion, dynamic lossless compression?

 - Jonathan Morton

---

From: Emmanuel Lochin <[email-masked]>
Date: Sun, 4 Aug 2019 09:13:51 +0200

Dear David, all,

Le 03/08/2019 à 21:02, David G. Pickett a écrit :
> Already compressed: You may not have noticed that zip detects not 
> compressible parts and just copies them into the archive. Similarly, a 
> packet stream compressor could send such blocks on verbatim in the 
> stream, or leave the packets alone entirely.  There is a slight 
> overhead for headers on no-compression data, so the protocol would 
> need a threshold for deciding which path a packet takes.  Backlogs 
> give the node time to detect such and make decisions.  You still get 
> the potential for combining smaller packets into larger ones, gaining 
> a reduction of overhead in many media with larger local link MTU and 
> inter-packet lost media use.  There still needs to be a packet discard 
> mechanism when the compressed flow falls too far behind.  What it 
> gives us is a softer saturation.  And lost packets are not free, as 
> the overhead of retransmission is a cost, as well as the pause in 
> transmission hurting both latency and bandwidth.
What kind of gain factor do you target? The idea is interesting but 
considering that Internet traffic is mostly compressed (huge video part 
for instance), BREACH attack, latency resulting of on the fly 
compression, I reckon you won't gain anything, may be epsilon?

EL

-- 
Emmanuel LOCHIN
Professeur ISAE
ISAE SUPAERO - Institut Supérieur de l'Aéronautique et de l'Espace
10 avenue Edouard Belin - BP 54032 - 31055 TOULOUSE CEDEX 4 FRANCE -
http://www.isae-supaero.fr
Tel +33 5 61 33 84 85 - Fax (+33) 5 61 33 83 45
Web : http://personnel.isae.fr/emmanuel-lochin/

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 4 Aug 2019 13:08:06 +0300

It should be straightforward to set up an experiment to figure out how much
compression can be expected.  Use tcpdump, capture a session containing
some traffic of interest, then run say gzip -9 over the capture file and
observe the compression ratio.  Or for a more live analysis, Linux PPP
supports two different compression options, so set up a rate limited link
between two instances.

- Jonathan Morton