Subject: JPEG-XL as Content-Encoding?

From: Yoav Weiss <[email-masked]>
Date: Thu, 20 Aug 2020 16:11:54 +0200

Hey HTTPers!

I was reviewing an intent to add JPEG-XL as a content-encoding
<https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CACZPrBjS75%3D%2B%3DSexFkJ7ewkBXDOF6FhkpwQYBeWn6sT57n1-_w%40mail.gmail.com?utm_medium=email&utm_source=footer>
for JPEG images, and thought I'd get some feedback from y'all regarding it.

The team behind JPEG-XL think that since it can be used to
seamlessly transform JPEGs, it might be interesting to enable that
transformation to happen at the HTTP layer and get ~20% smaller images
*automagically*, as compression would be done by supporting web servers and
image compression providers.

I'm slightly more skeptical about how automatic that would be, and find it
somewhat strange to have image-specific content-encoding, while other image
formats are served as separate mime types and negotiated with `Accept`.

So, a few questions:

   - Would it make sense to have an image specific content-encoding?
   - Are web servers more likely to perform JPEG=>JPXL transformations if
   JPXL is a content encoding, compared to browser support for it as an image
   format?
      - Note that those transformations are CPU heavy, so will need to be
      cached, or done at "build" time.
   - Same question for CDNs
   - Finally for CDNs who already transform images - have you run into
   compatibility issues coming from web content manipulating raw image bytes
   directly, and failing to do that post-transformation?

I'd highly appreciate y'all's opinions!

Cheers :)
Yoav

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 20 Aug 2020 17:14:01 +0200

Hi Yoav,

> I'd highly appreciate y'all's opinions!

Well, I don't know about this JPEG-XL but I'd argue that one very
important aspect of the content-coding is that it must absolutely
be possible to recover the original data by applying the inverse
transformation, as it's not supposed to transform the content but
only its representation. And intermediary might want to cache such
contents and use them to deliver the original image to clients not
compatible with this format, just as can be done with compressed
contents for example. I have no idea if this is the case here.
Otherwise I don't see how it could qualify as a content-encoding.

Just my two cents,
Willy

---

From: Dmitri Tikhonov <[email-masked]>
Date: Thu, 20 Aug 2020 14:30:09 -0400

> > formats are served as separate mime types and negotiated with `Accept`.

I agree that this would be strange.

> >       cached, or done at "build" time.

The transformation cost can be amortized over time by using a file
cache in the web server.

On Thu, Aug 20, 2020 at 05:14:01PM +0200, Willy Tarreau wrote:
> contents for example.

I subscribe here as well.  In my mind, the way to switch the response
from JPEG to JPXL is as follows:

    Request:

        GET /image.jpg
        Accept: image/jpxl

    Response:

        OK...
        Content-Type: image/jpxl
        Vary: accept

If 'Accept' does not specify image/jpxl, the server returns the original
JPEG image.  Note that there is still the white lie of the image's .jpg
extension, but this lie is smaller than the much more elaborate dance
with the Content-Encoding.

  - Dmitri.

---

From: Yoav Weiss <[email-masked]>
Date: Thu, 20 Aug 2020 23:01:04 +0200

> with the Content-Encoding.

Yeah, this is what I referred to as "exposing JPXL as an image format", and
the way we typically ship new formats (e.g. webp).

---

From: Yoav Weiss <[email-masked]>
Date: Thu, 20 Aug 2020 22:59:05 +0200

On Thu, Aug 20, 2020 at 5:14 PM Willy Tarreau <[email-masked]> wrote:

> > <
> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CACZPrBjS75%3D%2B%3DSexFkJ7ewkBXDOF6FhkpwQYBeWn6sT57n1-_w%40mail.gmail.com?utm_medium=email&utm_source=footer
> >

I should've made that point clearer - JPEG-XL (in legacy mode, which is
what's discussed for Content-Encoding) can be losslessly transformed to
JPEG and vice-versa.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 20 Aug 2020 21:51:15 +0000

--------

> with the Content-Encoding.

Agreed.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Sergey Ponomarev <[email-masked]>
Date: Fri, 21 Aug 2020 01:57:28 +0300

1. If the encoding is backward compatible for old clients then it does
not need a separate Content-Encoding i.e. it's not like gzipped text can't
be read.
2. Browsers can't even support concatenated gzip and xz compression ( lz4
and zst doesn't make sense). I wondered how quickly they integrated Brotli.
3. Intermediate proxies (i.e. Cloudlfare) can check file metadata/magic and
they don't really need for a separate mime type. But some simpler like
Nninx may need it to determine the processor. Basically mime types are
needed just to determine if file is text or binary, download or show it,
and basic type (e.g. image/video/archive)
4. I'm a regular developer and I saw that even png, ico mime type often is
added to filters and configs: e.g. upload file, compression filters like
Ziplet etc. On my current project favicon.ico is always auto compressed to
gzip. WebP files are still not widely supported and you can't even upload
screenshots.
IMHO if we can avoid a separate mime and content-encoding that will
simplify life for almost all except of intermediate proxies

-- 
Sergey Ponomarev <https://linkedin.com/in/stokito>, skype:stokito

---

From: Alex Deymo <[email-masked]>
Date: Fri, 21 Aug 2020 14:00:18 +0200

Hi,
I'm Alex and I'm working in the JPEG XL project. The transcoding being
discussed for this is indeed the lossless recompression of JPEG files
feature in JXL.

JXL has many more features as a codec than what the lossless JPEG
recompression can offer, you would get a lot better compression density for
similar visual quality if you don't need to be able to reconstruct the
exact same original JPEG file (or if your original was not a JPEG to start
with but RAW data from a camera), so it makes a lot of sense to add jxl as
an image codec on its own.

However, on top of that, the lossless recompression of JPEG files allows
you to get this ~20% gain for existing files. When you deploy a new lossy
codec there is the question of what to do with the existing images. If you
have a website with photos and want to convert your already lossy JPEG
files to a new codec to save storage and bandwidth and you decide to decode
them to pixels and encode them back to the new format you will end up with
more artifacts or worse compression density trying to accurately represent
the JPEG artifacts in the new codec, whatever the codec is. It's
impractical to do this lossy transcoding to a new codec at large scale on
existing images, each application would need to evaluate whether they want
to do this for existing images. This story is different if you start with a
large and high quality image (like a JPEG from a camera) and want to encode
in a smaller form for the web, since there you already have a high quality
file.

Instead, using jxl lossless recompression of JPEG files as content-encoding
gets you this ~20% benefit for existing files at large scale since you only
need browser support and webserver or intermediary support; you don't need
applications to evaluate whether the new codec fits their needs or if the
new kind of artifacts of a new codec are suitable for their use case. Think
about the JS developer who uses the hash of the file as a key somewhere and
needs to do some work to update their app to a new codec while still deal
with browsers that don't support the new codec. A CDN could decide to
deliver JPEG files encoded with the lossless recompression feature to
supporting browsers and take advantage of these gains today even if the
codec is not widely supported at the beginning like we see with many new
codecs without needing to modify the web application .

Regarding compatibility with browsers that don't support the
content-encoding and how to decide when to use the encoding, I personally
think that a reasonable way to implement that would be that for static
content you store the file already encoded with JXL to save that ~20% on
storage and decode it at serving time to browsers that don't support it,
and for dynamic content encode on the fly, however this would depend on the
cost model of the server/CDN.

I think the only shocking thing about a content-encoding for JPEGs is that
it can't encode any arbitrary file only JPEGs, but if you look at "general
purpose" compressors like Brotli they still can't compress to a smaller
file every file; many binary files that are already compressed like .zip or
even a JPEG files (unless they have a large ICC) won't compress to a
smaller file so you just don't do it even if Brotli is able to compress
them to a ~similar size file.

Let us know if you have any questions. We are happy to discuss ways to
improve.

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Aug 2020 14:23:19 +0200

Am 21.08.2020 um 14:00 schrieb Alex Deymo:
> recompression can offer, you would get a lot better compression density
> for similar visual quality if you don't need to be able to reconstruct
> the exact same original JPEG file (or if your original was not a JPEG to
> start with but RAW data from a camera), so it makes a lot of sense to
> add jxl as an image codec on its own.
>
> you to get this ~20% gain for existing files. When you deploy a new
> lossy codec there is the question of what to do with the existing
> images. If you have a website with photos and want to convert your
> already lossy JPEG files to a new codec to save storage and bandwidth
> and you decide to decode them to pixels and encode them back to the new
> format you will end up with more artifacts or worse compression density
> trying to accurately represent the JPEG artifacts in the new codec,
> whatever the codec is. It's impractical to do this lossy transcoding to
> a new codec at large scale on existing images, each application would
> need to evaluate whether they want to do this for existing images. This
> story is different if you start with a large and high quality image
> (like a JPEG from a camera) and want to encode in a smaller form for the
> web, since there you already have a high quality file.

That makes it sound a bit as if a losslessly-re-encoded JPG file is not
a valid JXL file. Is that the case?

> ...
> I think the only shocking thing about a content-encoding for JPEGs is
> that it can't encode any arbitrary file only JPEGs, but if you look at
> "general purpose" compressors like Brotli they still can't compress to a
> smaller file every file; many binary files that are already compressed
> like .zip or even a JPEG files (unless they have a large ICC) won't
> compress to a smaller file so you just don't do it even if Brotli is
> able to compress them to a ~similar size file.
> ...

That's indeed a concern. For the other currently registered encodings,
you *can* apply them, but they do not necessarily help.

This one can't be applied to any file type. One way to address this
would be to tune the format that it *can* handle any file type (by just
adding a tiny wrapper around it and preserving the actual octet stream
within).

> ...

Best regards, Julian

---

From: Alex Deymo <[email-masked]>
Date: Fri, 21 Aug 2020 16:36:46 +0200

Le ven. 21 août 2020 à 14:27, Julian Reschke <[email-masked]> a

> a valid JXL file. Is that the case?

A losslessly recompressed JPEG is a valid JXL file. There's value in
conserving your JPEG files as lossless recompressed versions outside the
Content-Encoding world (like, converting your existing library in your
hard-drive).

What I meant here is that if you start with a large high quality image
(JPEG or RAW) and you encoded it in the past long time ago to a lower
resolution or lower quality JPEG for the web application, you introduced
certain specific JPEG-artifacts and discarded information about the
original file. In some sense, the damage to the image is done. If you
already did this, then you are limited in your options on how to further
compress this file because you don't know what the original file looked
like so you might be trying to accurately reproduce JPEG artifacts with a
new codec instead of accurately reproduce original image features, this is
where lossless recompression is a good idea.
Instead, if you still have the original file, you can produce a lower
quality or lower resolution JXL that's visually similar to the original
file (not visually similar to the low res JPEG in the previous case). This
would give you a better compression ratio for the visual quality (but it
would not give you a JPEG file right away).

What's not true is the opposite statement, and maybe that's where the
confusion is. Not every JXL is a losslessly-re-encoded JPEG, although you
can always do stuff like decode any JXL to pixels and encode it back to
JPEG but it would largely depend on how you encode back to JPEG what file
you end up with. The lossless recompression feature limits the options when
encoding the JXL and adds extra information to be able to deterministically
produce a certain JPEG file.

> adding a tiny wrapper around it and preserving the actual octet stream

Yes you could add a tiny frame around to tell whether this was lossless
recompressed or not (maybe paying ~1 more byte), but isn't this basically
what the Content-Encoding header in the response is for anyway? I don't see
an application where this frame would help, the server side is not forced
to use the content-encoding and sending a file wrapped into another format
that adds no benefit would be a bit of a waste:
1. If we don't have this frame, you can call the function to do the
lossless encoding, if it returns with an error (like if the file is not a
JPEG) then you don't set Content-Encoding to jxl.
2. If we do have this frame, you always set the content encoding as jxl,
and then the function that would do the encoding does exactly the same
logic but stores the "jxl or raw" bit of information in the first byte
depending on whether it was able to re-encode it.
There's very little difference in how much you can already send to the
client before the encoding is done in either case and in general you know
very quickly whether the file can be encoded or not. Maybe all we need is a
function to tell very quickly whether we *can* encode it. I think this is
possible and relatively easy.  But I understand that this limitation may
need changes in how your server integrates a new content encoding since it
is not the same way that brotli for example was integrated; this is
something that can be addressed at the time of implementing support for
this content encoding in your server-side software.
My idea of how this would be implemented is more along the lines of already
having the jxl lossless file for static content and just serving it on
request or decoding+serving for clients not supporting it, given that you
get a significant benefit in storage size of static content (similar to
brotli_static setting in Nginx brotli).

That said, I should probably mention that according to the spec draft a
valid JPEG-1 file is also a valid JXL file, so it is really the non-JPEG
files that you can't re-encode and that part you can tell by looking at the
first few bytes, so we already have this frame information for old JPEG1 vs
JXL file.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 21 Aug 2020 23:11:51 +0200

> produce a certain JPEG file.

So that means that you cannot reproduce the original JPEG byte-for-byte ?
So it's not an encoding, it's a different image format. Even if it *looks*
visually identical, just like a PNG may look like a JPG, you can't turn
one into the other and conversely and hope that the original SHA256 of
the delivered file that was announced along the object in a header will
validate once a client wants to verify it. When playing only with encoding
you can always recover the original bytes and verify them because only the
representation changes, not the contents. Thus here it definitely is a
different Content-Type and not a different Content-Encoding.

Willy

---

From: Alex Deymo <[email-masked]>
Date: Fri, 21 Aug 2020 23:22:00 +0200

> different Content-Type and not a different Content-Encoding.

If you start with a JPEG file, you can convert it to a ~20% smaller JXL
file such that if you take that JXL file you get back the exact
byte-for-byte original JPEG file you started with, same data, same hash.
This is what lossless recompression of JPEG files is all about and what's
being proposed as a Content-Encoding.

My statement here was about the fact that there are other JXL files that
are not a lossless recompressed JPEG, which is fine, those simply won't
show up as payloads in a Content-Encoding context.

---

From: Tim Bray <[email-masked]>
Date: Fri, 21 Aug 2020 14:30:34 -0700

All that granted, If I send it to a browser or app or whatever in the
expectation that it's going to be rendered, image/jpxl seems like the right
choice.  Among other things, there are a lot of dumb HTTP clients out there
that just dispatch based on the Content-type value and in the case of
image/* probably don't even look at Content-encoding.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 23 Aug 2020 22:00:54 +0000

--------
Alex Deymo writes:

> recompression can offer [...]

I'm sure JXL is fantastic, but despite that, a special-case
(re)compression belongs in the application-layer, not in the
transport layer.

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 24 Aug 2020 10:04:05 +1000

Hi Yoav,

My personal .02 - 

The Intent to Prototype starts with the phrase 'JPEG XL is a next generation image format.' From the discussion, it seems like this is indeed a new format, and not a content encoding. If the proponents think that it will somehow ease the transition, that doesn't seem likely; it's more likely that they'll avoid well-understood problems introduced by format negotiation with brand-new problems related to format-specific content encoding. 

Just register a media type (or make it backwards-compatible to existing JPEGs).

Cheers,

> On 21 Aug 2020, at 12:11 am, Yoav Weiss <[email-masked]> wrote:
> 
> Hey HTTPers!
> 
> I was reviewing an intent to add JPEG-XL as a content-encoding for JPEG images, and thought I'd get some feedback from y'all regarding it.
> 
> The team behind JPEG-XL think that since it can be used to seamlessly transform JPEGs, it might be interesting to enable that transformation to happen at the HTTP layer and get ~20% smaller images *automagically*, as compression would be done by supporting web servers and image compression providers.
> 
> I'm slightly more skeptical about how automatic that would be, and find it somewhat strange to have image-specific content-encoding, while other image formats are served as separate mime types and negotiated with `Accept`.
> 
> So, a few questions:
> 	• Would it make sense to have an image specific content-encoding?
> 	• Are web servers more likely to perform JPEG=>JPXL transformations if JPXL is a content encoding, compared to browser support for it as an image format?
> 		• Note that those transformations are CPU heavy, so will need to be cached, or done at "build" time.
> 	• Same question for CDNs
> 	• Finally for CDNs who already transform images - have you run into compatibility issues coming from web content manipulating raw image bytes directly, and failing to do that post-transformation?

--
Mark Nottingham   https://www.mnot.net/