Subject: [Cbor] CBOR in QRcodes

From: Michael Richardson <[email-masked]>
Date: Tue, 22 Jun 2021 12:16:16 -0400

I was asked to summarize my experiences with encoding CBOR in QRcodes.
The short is that I really have none.  I have created QRcodes that contain
raw binary CBOR, and I've decoded them on a desktop, but I haven't round
tripped them onto paper and into a smartphone via an app (custom or generic).

In writing https://datatracker.ietf.org/doc/draft-richardson-mud-qrcode/ I
interacted with the Return Logistics Alliance (RLA.org), and I generated
QRcodes and scanned them with their app as as well as generic ones.
Their app has the limitation in that it doesn't skip codes it does not know,
instead it hangs.  Their specification is at: https://rla.org/resource/12n-documentation
It is recapped in my document.  As per 12N and ANSI MH10.8.2 , the encoding
uses ASCII control codes: RS, EoT, FS, GS, US characters in its TLV structure.
The Labels are 4-character strings, like "B001", and MUD was assigned "M180"

Thus my surprise at needing to encode CBOR in BASE-45.
I wouldn't have thought that necessary, that one could go directly to binary.
I imagine that the issue is one of what are the existing scanning equipment
out there, in airports, customs booths, etc. Many of them pretend to be
keyboards, and many of the systems they are connected to expect alphanumeric
inputs into a form field (whether an HTML form, or a 3270 terminal).

%diag2cbor.rb voucher_jada123456789.diag | qrencode -8 -o jada123456789.png
%file jada123456789.png
jada123456789.png: PNG image data, 219 x 219, 1-bit colormap, non-interlaced

I'd paste the results of:
%diag2cbor.rb voucher_jada123456789.diag | qrencode -8 -o - -t ASCII

in, but it's wider than 77 columns, so it would get munged.
It's really quite brilliant that we can do QRcodes into ASCII and ANSI block
characters.  We should have done SMIME signature that way :-)

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Christian Amsüss <[email-masked]>
Date: Wed, 23 Jun 2021 19:52:36 +0200

> inputs into a form field (whether an HTML form, or a 3270 terminal).

Have you had any success with binary data on end user QR code readers?

To my understanding, the limitation is in what can be used with generic
cell-phone based QR code scanners. These, according to what I gathered
from around the base45 spec, seem to sometimes[citation needed] reject
binary data that is not UTF-8. The dispatch into apps of the OS appears
to be focused around interpreting the QR code values as URIs and then
registering prefixes (which may not be registered schemes but look like
them, or may be schemes including host names) for an installed app (or
even to go into the app store).

For example (just picking a random tutorial I found), [1] implies that a
snippet like

<intent-filter>
  <!-- ... -->
  <data android:host="example.com" android:scheme="httpsz" />
</intent-filter>

is used to make a particular class of QR codes open in a to-be-designed
app. Whether or not that is good use of URIs, or whether the schemes
used there are registered, is unfortunately secondary (given the
deployment base of The Two Operating Systems, and the user expectation
to point their OS's camera to the code and go). Practically, that
requires the QR codes to conform to (someone's view of) URI syntax.

To avoid wasting too much of the QR code's encoding space, given
that encoding a URI in ASCII alredy wastes >1bit/byte, and that one
available encoding provides 45 (largely URI-safe) symbols in an
efficient encoding, base45 (or some enhanced version thereof) does
appear useful.

BR
c

[1]: https://techpearl.com/blog/qr-code/index.html

-- 
To use raw power is to make yourself infinitely vulnerable to greater powers.
  -- Bene Gesserit axiom

---

From: Michael Richardson <[email-masked]>
Date: Wed, 23 Jun 2021 20:55:22 -0400

> even to go into the app store).

Android's intent system can handle binary data, but quite possibly the way
that the stock QRcode reader works, that can never make it in.

I guess if one intends to make the COVID certificate usable across the widest
number of scanners, then it needs to be ASCII happy.   So from bouncers at
night clubs with throw-away 2013 era iPhone3s, to PCs running 3270 emulators
at the immigration booth.

I'm sure that a custom QRcode app could process binary if it didn't intend to
dispatch through android intents.

For reasons too stupid to explain, I actually have been walking around with
Avery labels with some self-generated QRcodes in my laptop bag for like two years.
I should see if I can convince some small store with a scanner to let me scan
my QRcode, with Notepad open, and see what happens.

I suspect that the keyboard interface is the critical, smallest path for
QRcode input.

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: "Doug Ewell" <[email-masked]>
Date: Thu, 24 Jun 2021 10:16:43 -0600

Christian Amsüss wrote:

> base45 (or some enhanced version thereof) does appear useful.

I'm not sure what the bit in parentheses means. I'm thinking of the various "enhanced versions" of base32, base64, and base85, all of which are intended to solve some localized problem, and all of which add confusion and incompatibility to what should be a straightforward conversion. I hope this isn't a suggestion to do something similar to base45.

--
Doug Ewell, CC, ALB | Lakewood, CO, US | ewellic.org

---

From: 'Christian Amsüss' <[email-masked]>
Date: Fri, 25 Jun 2021 11:10:10 +0200

Hello Doug,
(CC'ing the bse45 authors as the thread is moving there)

On Thu, Jun 24, 2021 at 10:16:43AM -0600, Doug Ewell wrote:
> I'm not sure what the bit in parentheses means. I'm thinking of the
> various "enhanced versions" of base32, base64, and base85, all of
> which are intended to solve some localized problem, and all of which
> add confusion and incompatibility to what should be a straightforward
> conversion. I hope this isn't a suggestion to do something similar to
> base45.

As I understand the base45 mechanism is very recent and not widespread
yet -- it has seen significant use in the digital health certificates
application (thread at [1]), but that was rolled out so fast that little
discussion was had given the time pressure the project is under what
they use is baked[2].

Outside the context of this, the draft has received no pertinent
discussion I could find[3], so I think it's more in the "there is a
draft and the mechanisms can be discussed" stage than in the "There is
an RFC that has become The Word" stage (in which any alterations might
become confusing).

Even if it were,

* given it is aimed at QR code readers but *still* gets trapped in some
  of the URI-but-not-really pitfalls,

* given that the proposed enhancements[4] would make it a base41, base42 or
  base43 and the use in combination with scheme-like dispatch there is
  little chance of confusion,

I think it'd be premature to call for a moratorium (as I'd understand
your comment).

BR
Christian

(with my chair hat off, in case that needs clarification in this context)

[1]: https://mailarchive.ietf.org/arch/msg/cbor/M07MvOOyQlw-0P9i2GYYFd8hSbM/
[2]: https://github.com/ehn-dcc-development/hcert-spec/issues/64#issuecomment-861226115
[3]: https://mailarchive.ietf.org/arch/search/?q=faltstrom-base45
[4]: https://github.com/ehn-dcc-development/hcert-spec/issues/64#issuecomment-860974626

---

From: Michael Richardson <[email-masked]>
Date: Fri, 25 Jun 2021 11:51:45 -0400

I don't have enough caffeine in me to figure out how much more efficient
base45 is over base32.  I think it's log(45)/log(32), right?

In places where base64 or base85 won't work, it seems to me that base32 is
probably safer than base45.   As I wrote, I think that the limitation for
QRcode entry has to do with forms on web pages or 3270 terminals, where the
scanner is a bump-in-the-cord on the keyboard interface.  Whether it's a
PS2/PS2 or a second USB input.

I'm surprised that base45 includes space, + and / :-)

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 25 Jun 2021 18:06:49 +0200

On 2021-06-25, at 17:51, Michael Richardson <[email-masked]> wrote:
> Signed PGP part

Base 32 is ld(32) = 5 bits per character.
Base 4[12345] is 16 bits per three characters, i.e. 5.33 bits per character.
(Base 45 wastes most of another half bit that it might have.)

> I'm surprised that base45 includes space, + and / :-)

QR code has an alphabet.  That *happens* to have 45 characters, because it uses half an 11-bit code (2**5.5 = 45.25), i.e., 5.5 bits off the raw encoding.

Of course you want to be able to have spaces etc. in the human-readable messages that QR-Code might convey.  But using all these for a basenn encoding is not needed, just because you have them.

That 45 is not a meaningful basexx number, as shown above, as Math.log2(45) = 5.49; you can’t do a lot with that.  
(That 5.49 is Math.log2((2**5.5).floor) if you want to know why).
Including the weird characters (“ “ , ”%”, choose another two) in a basenn encoding is, er, not so bright, as the poor DGC people had to find out.

Grüße, Carsten

---

From: Christian Amsüss <[email-masked]>
Date: Fri, 25 Jun 2021 18:08:32 +0200

> base45 is over base32.  I think it's log(45)/log(32), right?

The efficiency of base45 on its own is pretty irrelevant, because it's
something that'd be only used inside an environment where these 45
characters are encoded in 5.5 bits of the QR code.

base45 does a 2-to-3 encoding, so one 8 bit of binary data gets spread
to 24/2=12 bit of ASCII base45 string, but those 1.5 characters are
encoded in 1.5*5.5 = 8.25 bit, so in the end it's a 3% loss.

(Same would be the case for 2-to-3 encodings possible with
base41,..base44).

> PS2/PS2 or a second USB input.

The RFC4648 base32 alphabet does work, but it's a 5-to-8 encoding, so
it'd give 1.6*6.6 bytes payload binary per byte QR-internal binary, so
it's 10% instead of 3% overhead.

Whether that really matters all that much is, as far as generic
recommendations for QR codes go, probably up for debate. (Even in the
health certificate use case there was discussion that if border
conditions were different, base32 would have been a good choice).

> I'm surprised that base45 includes space, + and / :-)

And the percent character; that's part of the trouble (but then again, 4
can be removed without loss of efficiency).

BR
Christian

-- 
There's always a bigger fish.
  -- Qui-Gon Jinn

---

From: "Doug Ewell" <[email-masked]>
Date: Fri, 25 Jun 2021 13:38:22 -0600

My understanding is also that base45 is intended for the specific alphanumeric encoding for QR codes defined in ISO 18004:2015, sections 7.3.4 and 7.4.4.

Given that, whether an alphabet of size 45 is maximally efficient, or whether a "better" alphabet could have been chosen, is really not at issue. There is an installed base (QR codes have been around for more than two decades), so there is a significant and probably prohibitive cost to changing it.

I do note, however, that the specific application described in draft-faltstrom-base45 — encoding 16-bit integral values in three base45 code units, and encoding a trailing 8-bit value in two — does not appear to be part of the standard, so that part could lend itself to "enhancement," although I'm not clear exactly what that would be.

Creating incompatible branches of base45 for different applications (such as URIs) still seems like a decision not well supported by experience.

-----Original Message-----
From: Christian Amsüss <[email-masked]> 
Sent: Friday, June 25, 2021 10:09
To: Michael Richardson <[email-masked]>
Cc: Doug Ewell <[email-masked]>; [email-masked]
Subject: Re: Use and development of draft-faltstrom-base45 (was: Re: [Cbor] CBOR in QRcodes)

> base45 is over base32.  I think it's log(45)/log(32), right?

The efficiency of base45 on its own is pretty irrelevant, because it's something that'd be only used inside an environment where these 45 characters are encoded in 5.5 bits of the QR code.

base45 does a 2-to-3 encoding, so one 8 bit of binary data gets spread to 24/2=12 bit of ASCII base45 string, but those 1.5 characters are encoded in 1.5*5.5 = 8.25 bit, so in the end it's a 3% loss.

(Same would be the case for 2-to-3 encodings possible with base41,..base44).

> QRcode entry has to do with forms on web pages or 3270 terminals, 
> where the scanner is a bump-in-the-cord on the keyboard interface.  
> Whether it's a

The RFC4648 base32 alphabet does work, but it's a 5-to-8 encoding, so it'd give 1.6*6.6 bytes payload binary per byte QR-internal binary, so it's 10% instead of 3% overhead.

Whether that really matters all that much is, as far as generic recommendations for QR codes go, probably up for debate. (Even in the health certificate use case there was discussion that if border conditions were different, base32 would have been a good choice).

> I'm surprised that base45 includes space, + and / :-)

And the percent character; that's part of the trouble (but then again, 4 can be removed without loss of efficiency).

BR
Christian

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 25 Jun 2021 23:34:12 +0200

> Given that, whether an alphabet of size 45 is maximally efficient, or whether a "better" alphabet could have been chosen, is really not at issue. There is an installed base (QR codes have been arou…

Of course, just like base32 and base64 are not out to change RFC20 (aka ASCII).

Both do take a well-deliberated subset of ASCII (32 and 64 characters, respectively) and define a representation of byte strings in these.

The debate is about a sensible choice of a selection of characters out of the 45-character QR-code character set for a basexx that is useful for representing byte strings in the QR-code character set.

> I do note, however, that the specific application described in draft-faltstrom-base45 — encoding 16-bit integral values in three base45 code units, and encoding a trailing 8-bit value in two — does…

That is the part we are talking about.  Just as we usually don’t use base128, even if that is well-supported by the 128-character ASCII character set (which, as I said, we are not planning to change with base64 and friends), we shouldn’t use base45 for the 45-character QR-code character set.

> Creating incompatible branches of base45 for different applications (such as URIs) still seems like a decision not well supported by experience.

Well, having base45 in the first place is, er, not really a well thought-out decision.

Grüße, Carsten