Subject: [Cellar] compression libraries in IETF specifications (was Re: [cellar-wg/matroska-specification] deprecate LZO compression (#376))

From: Michael Richardson <[email-masked]>
Date: Sun, 03 May 2020 12:10:27 -0400

Hi, I wanted to bring this item to the list's attention.

Spencer: I seem to remember that the IETF has had compression IPR issues in
specifications in the past.  IPCOMP comes to mind, but I am guessing that
other RTP/SIP groups have gone through this.. Is there possibly an IESG
statement that might help us here?

But, first, the software license on the libraries is not entirely germane.
An IETF specification can well contain algorithms which are currently only
available in software licensed in ways that might not be useable by all
implementers.
Someone just will have to write a new library with a license that suits them!

What matters if that the specification (and any related patents!) is
available to all.  The IETF's minimum IPR requirement is generally RAND,
which is often not good enough for open source, which requires RF.
The decision as to which IPR claims to tolerate is a WG-level decision, with
the proviso that the WG (and IETF) is informed about the issue.

So even in the absense of the jackoalan and zivillian LZO implementations,
the WG should be free to standardize LZO, if it the IPR claims on the
algorithm itself are well understood.

{In the compression space, my understanding is that there have been formats
where there were no IPR claims on decompressors, but there was significant
IPR-encumbered innovation on doing compression.  An encoder might need a
specific license in order to get 70% compression rather than just 40%, but
this had no affect on the decoder.  But, I don't recall the specific
situations where this occured.  Maybe back in the old ZOO/ARC/ZIP days...}

Steve Lhomme <[email-masked]> wrote:
    > Matroska Blocks can be compressed with zlib, bz2 and lzo compression.

    > zlib and bz2 have decompression libraries available under permisssive
    > licenses (BSD like for bz2 and OSI approved, GPL compatible, non
    > copyleft for zlib).

    > lzo doesn't have such a permissive library. The [official
    > library](https://www.oberhumer.com/opensource/lzo/) is GPLv2, there is
    > also decompression code that doesn't rely on this library [in
    > FFmpeg](https://github.com/FFmpeg/FFmpeg/blob/master/libavutil/lzo.c)
    > which is LGPLv2 which means grabbing the whole libavutil to compile.

    > This license is not permissive enough to be used in many closed source
    > products where they don't want to touch any GPL-based license. That
    > means a file that is created with LZO in a GPL/LGPL software may not be
    > read universally because of licensing issues. I think we should avoid
    > it and mark LZO as deprecated.

    > The LZO compression advantage over others is that it is [very fast to
    > compress/decompress](https://catchchallenger.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO). But
    > the compression ratio is not good. That is good for realtime
    > transmission but bad for long term storage. In the end automatic tools
    > that test each compression (like mkclean) will never pick it anyway.

...
    > I found 2 other options with better licenses:

    > - a [C++ version in MIT license](https://github.com/jackoalan/lzokay) with compression
    > - a [C# version in MIT license](https://github.com/zivillian/lzo.net) without decompression

    > They are both based on the [Linux Kernel documentation for
    > LZO](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/lzo.txt).

    > In libmatroska2 I've been using minilzo GPLv2 but I may try to port one
    > of these in C so I can use it (or simply write a C wrapper around the
    > C++ one). That means there would be an alternative that is MIT license,
    > so freely used by anyone.

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Tue, 5 May 2020 15:43:17 -0500

Hi, Michael,

On Sun, May 3, 2020 at 11:10 AM Michael Richardson <[email-masked]> wrote:

> other RTP/SIP groups have gone through this..

I had forgotten IPCOMP, but was remembering RoHC and RoHCv2. I think you're
remembering SIGCOMP.

The complete list of disclosures that mention "compression" is at
https://datatracker.ietf.org/ipr/search/?draft=&rfc=&doctitle=compression&submit=doctitle&group=&holder=&iprtitle=&patent=

> Is there possibly an IESG

I believe the only relevant IESG statement on licensing for repositories is
to make it clear that they need to include standard boilerplate (
https://www.ietf.org/about/groups/iesg/statements/open-source-repositories-license/
).

> algorithm itself are well understood.

This is exactly right. The IESG cares that the working group made the
decision to request publication on a document without being "submarined" by
undisclosed patents or patients with vague or confusing licensing
provisions. If we have our eyes open, we'll be fine.

> situations where this occured.  Maybe back in the old ZOO/ARC/ZIP days...}

I have the same vague memories (perhaps I'm remembering that licensing was
different, rather than decompressors hat no known IPR, but that's pretty
close).

I hope this helps!

Spencer

>     > compress/decompress](
> https://catchchallenger.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO).
> But
> ....
>     > LZO](
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/lzo.txt
> ).
>

---

From: Adam Roach <[email-masked]>
Date: Tue, 5 May 2020 16:18:29 -0500

On 5/5/2020 3:43 PM, Spencer Dawkins at IETF wrote:
> Hi, Michael,
>
> On Sun, May 3, 2020 at 11:10 AM Michael Richardson <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>
> I had forgotten IPCOMP, but was remembering RoHC and RoHCv2. I think 
> you're remembering SIGCOMP.

Probably not: SIGCOMP sidestepped this specific issue altogether by 
normatively specifying precisely zero compression algorithms. It instead 
defined a virtual machine that message recipients would implement, and 
to which message senders would send bytecodes sufficient to decode 
subsequent messages.

/a

---

From: Steve Lhomme <[email-masked]>
Date: Fri, 22 May 2020 15:23:04 +0200

I hope we don't have to get to this point.

In Matroska we (currently) have 4 compressions:
- header compression: stripping the same bytes at the beginning of
each frame, not sure if there's a patent/license issue on that. It was
clearly designed in "clean room" (no knowledge of existing other
formats doing that).
- zlib: designed to have patent issues (and used widely for
everything, even PNG ?) https://www.zlib.net/zlib_faq.html#faq31
- LZO which at least has some documentation in the Linux Kernel. I
wouldn't imagine they would use that it there was any issue.
- bzlib, which is fact bz2 an evolution of bzlib to avoid patents
(https://en.wikipedia.org/wiki/Bzip2), also used very widely.

My main concern was that LZO did not have a freely usable
implementation (official being GPL or commercial) but then I found
some more libs capable of that. So at least for implementation they
all have free (as in everything) implementations.

As for patents do we have to put some text that we believe their not
covered by patents ?

Le mar. 5 mai 2020 à 23:18, Adam Roach <[email-masked]> a écrit :
> I had forgotten IPCOMP, but was remembering RoHC and RoHCv2. I think you're remembering SIGCOMP.
>
>
> Probably not: SIGCOMP sidestepped this specific issue altogether by normatively specifying precisely zero compression algorithms. It instead defined a virtual machine that message recipients would implement, and to which message senders would send bytecodes sufficient to decode subsequent messages.
>
> /a
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

-- 
Steve Lhomme
Matroska association Chairman

---

From: Adam Roach <[email-masked]>
Date: Fri, 22 May 2020 10:12:10 -0500

> I hope we don't have to get to this point.

Agreed -- I don't think it would be a good model for CELLAR.

<snip>

> As for patents do we have to put some text that we believe their not

No, and in fact, you shouldn't. The IETF is very careful about not 
taking specific positions about the presence or validity of patents that 
apply to technologies. If someone _does_ know about patents, they're 
either required or strongly encouraged (depending on their relationship 
to the patent in question) to make a disclosure to the IETF, but these 
appear on https://www.ietf.org/ipr rather than in the documents 
themselves. There's no real way to assert "believed to be unencumbered."

/a