Subject: RFC 6234 code

From: "Dearlove, Christopher (UK)" <[email-masked]>
Date: Thu, 27 Jun 2013 15:49:47 +0000

RFC 6234 contains, embedded in it, code to implement various functions, including SHA-2.

Extracting that code from the RFC is not a clean process. In addition the code must have existed unembedded before being embedded.

Is that code available from the IETF or elsewhere?

(I have tried some approaches to finding such code before posting here, but none successful.)

-- 
Christopher Dearlove
Senior Principal Engineer, Communications Group
Communications, Networks and Image Analysis Capability
BAE Systems Advanced Technology Centre
West Hanningfield Road, Great Baddow, Chelmsford, CM2 8HN, UK
Tel: +44 1245 242194 |  Fax: +44 1245 242124
[email-masked] | http://www.baesystems.com

BAE Systems (Operations) Limited
Registered Office: Warwick House, PO Box 87, Farnborough Aerospace Centre, Farnborough, Hants, GU14 6YU, UK
Registered in England & Wales No: 1996687

********************************************************************
This email and any attachments are confidential to the intended
recipient and may also be privileged. If you are not the intended
recipient please delete it from your system and notify the sender.
You should not copy it or use it for any purpose nor disclose or
distribute its contents to any other person.
********************************************************************

---

From: Hector Santos <[email-masked]>
Date: Thu, 27 Jun 2013 12:26:56 -0400

What language, OS? There are plenty of rich hashing/encrypting C/C++ 
libraries out there. Windows has CAPI, even OPENSSL has these libraries.

---

From: "Dearlove, Christopher (UK)" <[email-masked]>
Date: Thu, 27 Jun 2013 16:33:16 +0000

I want the specific code that is in the RFC (which happens to be in C) rather than some other implementation.

-----Original Message-----
From: Hector Santos [mailto:[email-masked]] 
Sent: 27 June 2013 17:27
To: Dearlove, Christopher (UK)
Cc: [email-masked]
Subject: Re: RFC 6234 code

----------------------! WARNING ! ----------------------
This message originates from outside our organisation,
either from an external partner or from the internet.
Keep this in mind if you answer this message.
Follow the 'Report Suspicious Emails' link on IT matters
for instructions on reporting suspicious email messages.
--------------------------------------------------------

---

From: "Eggert, Lars" <[email-masked]>
Date: Thu, 27 Jun 2013 16:47:14 +0000

Hi,

On Jun 27, 2013, at 17:49, "Dearlove, Christopher (UK)" <[email-masked]> wrote:
> Extracting that code from the RFC is not a clean process. 

https://tools.ietf.org/tools/rfcstrip/ can take the headers/footers out. Extracting the code from its output should be relatively easy.

Lars

---

From: Joe Abley <[email-masked]>
Date: Thu, 27 Jun 2013 13:22:04 -0400

> (I have tried some approaches to finding such code before posting here, but none successful.)

Turns out that extracting the code from the RFC is only twenty minutes' work. Probably faster just to do it than to talk about it. If you'd like a copy of the files I extracted from the RFC text, I can easily send you a tarball.

In general I think that maintaining code repositories is non-trivial, and also has been known (especially with respect to crypto/hash algorithms) to run into export licensing issues. Maintaining an archives of RFCs is already a core competency for the IETF, and (at least in some cases) exporting text is less problematic than exporting code.

Joe

[krill:~]% mkdir 6234
[krill:~]% cd 6234
[krill:~/6234]% curl -O 'http://tools.ietf.org/rfc/rfc6234.txt'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  231k  100  231k    0     0   190k      0  0:00:01  0:00:01 --:--:--  190k
[krill:~/6234]% vi rfc6234.txt 
[krill:~/6234]% ls
hkdf.c			sha.h			stdint-example.h
hmac.c			sha1.c			test-driver.c
rfc6234.txt		sha224-256.c		usha.c
sha-private.h		sha384-512.c
[krill:~/6234]% vi *.h *.c
10 files to edit
[krill:~/6234]% date
Thu 27 Jun 2013 13:14:53 EDT
[krill:~/6234]% 
[krill:~/6234]% cc -c hmac.c
[krill:~/6234]% cc -c sha1.c
[krill:~/6234]% cc -c sha224-256.c
[krill:~/6234]% cc -c sha384-512.c 
[krill:~/6234]% cc -c usha.c
[krill:~/6234]% cc -c test-driver.c
[krill:~/6234]% ls
hkdf.c			sha.h			sha384-512.o
hkdf.o			sha1.c			stdint-example.h
hmac.c			sha1.o			test-driver.c
hmac.o			sha224-256.c		test-driver.o
rfc6234.txt		sha224-256.o		usha.c
sha-private.h		sha384-512.c		usha.o
[krill:~/6234]% cc -o test-driver *.o
[krill:~/6234]% ./test-driver| egrep '(PASS|FAIL)'
SHA1 sha standard test 1: PASSED
SHA1 sha standard test 2: PASSED
SHA1 sha standard test 3: PASSED
SHA1 sha standard test 4: PASSED
SHA1 sha standard test 5: PASSED
SHA1 sha standard test 6: PASSED
SHA1 sha standard test 7: PASSED
SHA1 sha standard test 8: PASSED
SHA1 sha standard test 9: PASSED
SHA1 sha standard test 10: PASSED
SHA1 random test 0: PASSED
SHA1 random test 1: PASSED
SHA1 random test 2: PASSED
SHA1 random test 3: PASSED
SHA224 sha standard test 1: PASSED
SHA224 sha standard test 2: PASSED
SHA224 sha standard test 3: PASSED
SHA224 sha standard test 4: PASSED
SHA224 sha standard test 5: PASSED
SHA224 sha standard test 6: PASSED
SHA224 sha standard test 7: PASSED
SHA224 sha standard test 8: PASSED
SHA224 sha standard test 9: PASSED
SHA224 sha standard test 10: PASSED
SHA224 random test 0: PASSED
SHA224 random test 1: PASSED
SHA224 random test 2: PASSED
SHA224 random test 3: PASSED
SHA256 sha standard test 1: PASSED
SHA256 sha standard test 2: PASSED
SHA256 sha standard test 3: PASSED
SHA256 sha standard test 4: PASSED
SHA256 sha standard test 5: PASSED
SHA256 sha standard test 6: PASSED
SHA256 sha standard test 7: PASSED
SHA256 sha standard test 8: PASSED
SHA256 sha standard test 9: PASSED
SHA256 sha standard test 10: PASSED
SHA256 random test 0: PASSED
SHA256 random test 1: PASSED
SHA256 random test 2: PASSED
SHA256 random test 3: PASSED
SHA384 sha standard test 1: PASSED
SHA384 sha standard test 2: PASSED
SHA384 sha standard test 3: PASSED
SHA384 sha standard test 4: PASSED
SHA384 sha standard test 5: PASSED
SHA384 sha standard test 6: PASSED
SHA384 sha standard test 7: PASSED
SHA384 sha standard test 8: PASSED
SHA384 sha standard test 9: PASSED
SHA384 sha standard test 10: PASSED
SHA384 random test 0: PASSED
SHA384 random test 1: PASSED
SHA384 random test 2: PASSED
SHA384 random test 3: PASSED
SHA512 sha standard test 1: PASSED
SHA512 sha standard test 2: PASSED
SHA512 sha standard test 3: PASSED
SHA512 sha standard test 4: PASSED
SHA512 sha standard test 5: PASSED
SHA512 sha standard test 6: PASSED
SHA512 sha standard test 7: PASSED
SHA512 sha standard test 8: PASSED
SHA512 sha standard test 9: PASSED
SHA512 sha standard test 10: PASSED
SHA512 random test 0: PASSED
SHA512 random test 1: PASSED
SHA512 random test 2: PASSED
SHA512 random test 3: PASSED
[krill:~/6234]%

---

From: Joe Abley <[email-masked]>
Date: Thu, 27 Jun 2013 13:24:27 -0400

Oh, I missed the first "date" line in my paste, which makes the second one a bit mysterious. Here it is :-)

[krill:~]% date
Thu 27 Jun 2013 12:56:35 EDT
[krill:~]% mkdir 6234
[krill:~]% cd 6234

...

> SHA1 sha standard test 5: PASSED

[etc]

---

From: Hector Santos <[email-masked]>
Date: Thu, 27 Jun 2013 15:38:26 -0400

Ok, other than time, it should be easy to extract, clean up and cross 
your fingers that it compiles with your favorite C compiler.  But I 
would write to the authors to get the original source. Or google:

     C source crypto libraries API hashing functions

among the first hit:

http://www.cryptopp.com/

---

From: Joe Abley <[email-masked]>
Date: Thu, 27 Jun 2013 16:06:37 -0400

On 2013-06-27, at 15:38, Hector Santos <[email-masked]> wrote:

> Ok, other than time, it should be easy to extract, clean up and cross your fingers that it compiles with your favorite C compiler.

Having just done it, I'm happy to report that there was little finger-crossing involved. The fact that there's a fairly comprehensive test suite included in the source code in the RFC makes it quite easy to gain confidence that the non-test code was extracted intact, beyond the simple lack of compiler warnings.

Joe

---

From: "Dearlove, Christopher (UK)" <[email-masked]>
Date: Fri, 28 Jun 2013 08:53:09 +0000

I'd actually tried the authors, but no reply yet (only a few days). I also tried the RFC Editor thinking they might have e.g. XML from which extraction might have been easier, but also no response yet. And I had found several libraries, but not the RFC code. I can't see any suggestion that the library you indicate includes that code, it also bills itself as a C++ library, which the RFC code is not (and also not what I want, but that's not a subject for this list).

But the broader point is that if it's worth the IETF publishing the code as an RFC, it's worth making the code available straightforwardly.

For me, a thanks to Tony Hansen, who did the extraction for me. (That makes me feel a little guilty, why should he do my work I could have done?) But the point of posting on this list was to say that the code should be available so that each person wanting that code doesn't have to do that again.

Christopher

-----Original Message-----
From: Hector Santos [mailto:[email-masked]] 
Sent: 27 June 2013 20:38
To: Dearlove, Christopher (UK)
Cc: [email-masked]
Subject: Re: RFC 6234 code

among the first hit:

http://www.cryptopp.com/

---

From: "Eggert, Lars" <[email-masked]>
Date: Fri, 28 Jun 2013 08:57:06 +0000

Hi,

> But the broader point is that if it's worth the IETF publishing the code as an RFC, it's worth making the code available straightforwardly.

some WGs are good at this. RFC5662 for example includes the shell commands to extract the sources it contains.

It would certainly be nice if other WGs did this, too, but I'm not sure we need to make it a requirement.

Lars

---

From: "Dearlove, Christopher (UK)" <[email-masked]>
Date: Fri, 28 Jun 2013 08:59:23 +0000

I think we must agree to differ on whether the IETF should do this. If it takes twenty minutes work for everyone who has to do it, either that stacks up, or no one is actually interested, so is the RFC worth it?

As for maintaining the code, I'm just suggesting the same snapshot as the RFC is maintained. It could even be another link off the tools page: here's the text, here's the XML, here's any code.

-----Original Message-----
From: Joe Abley [mailto:[email-masked]] 
Sent: 27 June 2013 18:22
To: Dearlove, Christopher (UK)
Cc: [email-masked]
Subject: Re: RFC 6234 code

Joe

[krill:~]% mkdir 6234
[krill:~]% cd 6234

---

From: Tony Hansen <[email-masked]>
Date: Fri, 28 Jun 2013 10:11:19 -0400

On 6/28/2013 4:53 AM, Dearlove, Christopher (UK) wrote:
> I'd actually tried the authors, but no reply yet (only a few days). ....
>

Hmmm, as one of the authors of RFC6234, I'm not sure how to respond to
that, other than to say, You're welcome. :-) (And no, you didn't get the
files extracted from the RFC, but instead got the files as used to
generate the RFC.)

> I also tried the RFC Editor thinking they might have e.g. XML from which extraction might have been easier, but also no response yet. And I had found several libraries, but not the RFC code. ...
> But the broader point is that if it's worth the IETF publishing the
> code as an RFC, it's worth making the code available straightforwardly. 

I've suggested on a couple occasions to the RFC Editor that, when an RFC
provides source code, they should allow rfcXXXX.tar or rfcXXXX.tgz to be
provided as well.

There's only a handful of RFCs that do provide source code, for whatever
reason, so this should not be an onerous additional feature.

I've CC'd the RFC Interest mailing list, where this would be more
properly discussed.

    Tony Hansen

---

From: Riccardo Bernardini <[email-masked]>
Date: Fri, 28 Jun 2013 16:34:57 +0200

> I've CC'd the RFC Interest mailing list, where this would be more

This came to my mind while reading: why not embedding the archive in
the text with some special "line header" that makes automatic
extraction possible?  For example, this could be a first rough draft
of the procedure

  1.  Create an archive of the source files with ar.  This command has
the characteristic that if the files are text files, also the archive
is pure text
  2.  Embed the archive in the RFC by adding at the beginning of each
line a '!' (or another marker).  Lines too long for an RFC can be
split and the marker changed to (say) '?'

In order to extract the archive should suffice to process it with an
awk script, piped to ar to extract the files (and if you have Windows,
you install cygwin :-).  With this approach the source code would also
remain human-readable in the RFC text.  The only drawback is the
length of the resulting RFC if the embedded code is very large.

Riccardo Bernardini

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 28 Jun 2013 07:53:03 -0700

> some WGs are good at this. RFC5662 for example includes the shell commands to extract the sources it contains.

Earlier example: RFC 4134.

> It would certainly be nice if other WGs did this, too, but I'm not sure we need to make it a requirement.

Fully agree. This is a service that the RFC Editor should provide to the community.

--Paul Hoffman

---

From: Hector Santos <[email-masked]>
Date: Fri, 28 Jun 2013 11:56:17 -0400

I believe this is all part of improving the IETF Electronic Diversity 
picture. Just like we have to deal with greater people personal 
globalization diversity issues, there is also greater technology and 
legal diversity issues to deal with. So many tools, so many languages, 
so many OSes, so many devices and communications API platforms, where 
are the proposals for better, new "IETF Global Commons?"  Guidelines for 
technical writing for the new world engineers to use, etc.

For me, when I saw this RFC, the things crossed my mind:

- I have trouble with the licensing statement. The RFC describes public 
domain technology.  It requires passing this thru your lawyer(s) to see 
if it can used in our commercial product lines.

- Far too big to distribute via a RFC.  Provide a link to some RFC site. 
Note, I'm just saying in general. I did not read in detail if
the RFC already included links to the official source code.

- Because it was too big, it requires a stripper/parser, although a good 
power programmer can quickly macro-clean it up.  The RFCSTRIP tool, 
well, what language is that? I'm not an *nix person. So this adds to the 
complexity for the Windows shops to get at these hashing functions.  Of 
course, its a piece of cake for a sharp programmer, but even the 
sharpest knives eventually get dull.

--
HLS

---

From: John C Klensin <[email-masked]>
Date: Fri, 28 Jun 2013 15:10:37 -0400

--On Friday, June 28, 2013 10:11 -0400 Tony Hansen
<[email-masked]> wrote:

>> I also tried the RFC Editor thinking they might have e.g. XML
>> from which extraction might have been easier, but also no
>> response yet. And I had found several libraries, but not the
>> RFC code. ... But the broader point is that if it's worth the
>> IETF publishing the code as an RFC, it's worth making the
>> code available straightforwardly. 
> 
> I've suggested on a couple occasions to the RFC Editor that,
> when an RFC provides source code, they should allow
> rfcXXXX.tar or rfcXXXX.tgz to be provided as well.
> 
> There's only a handful of RFCs that do provide source code,
> for whatever reason, so this should not be an onerous
> additional feature.

Folks, IANAL, but please be _very_ careful about the comment Joe
made about the potential difference between publishing a paper
or article that contains code and exporting the code itself or
making it generally available for export.   I have no reason to
believe that this particular case is a problem given how widely
the details of SHA-2 has been published but, if we want the RFC
Editor to go into the code distribution business, we should be
very sure that an attorney with the right specialties has looked
at the situation and either cleared it generally or advised on
what needs to be examined case by case.

   john

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 28 Jun 2013 12:19:21 -0700

> what needs to be examined case by case.

The RFC Editor is publishing code in a text file that is formatted like an RFC. The proposal is for the RFC Editor to publish *the exact same code* in a file without the RFC wrapping.

If you really think you see a legal difference in doing the second, fine; I propose that you are just searching for problems that do not exist.

--Paul Hoffman

---

From: Joe Abley <[email-masked]>
Date: Fri, 28 Jun 2013 16:41:05 -0400

> If you really think you see a legal difference in doing the second, fine; I propose that you are just searching for problems that do not exist.

Quite possibly they don't, and I'm not presuming to talk for John. But the vague thoughts that crossed my mine were the issues that allowed Phil Zimmerman to publish PGP source code in book form and avoid conviction for munitions export without a licence.

Perhaps this is no longer an issue in the US following Bernstein vs. United States and Junger vs. Daley.

Just thinking aloud, really.

I will observe that far more than 20 minutes of combined person-minutes have been consumed by this thread already, and the approach of minimum collective effort is still "scrape the code from the RFC yourself and compile it".

Joe

---

From: Donald Eastlake <[email-masked]>
Date: Fri, 28 Jun 2013 17:19:27 -0400

Hi Hector,

On Fri, Jun 28, 2013 at 11:56 AM, Hector Santos <[email-masked]> wrote:
> globalization diversity issues, there is also greater technology and legal
> diversity issues to deal with. So many tools, so many languages, so many
> OSes, so many devices and communications API platforms, where are the
> proposals for better, new "IETF Global Commons?"  Guidelines for technical
> writing for the new world engineers to use, etc.

I think that computational code written in C, pretty much the lowest
common denominator, is just fine. I don't think there is any greater
diversity now that then was 10 or 20 years ago at that level. In fact,
I think there is less.

> domain technology.  It requires passing this thru your lawyer(s) to see if
> it can used in our commercial product lines.

Sorry, the standard simplified BSD license is required for code in
RFCs. The authors couldn't change it. But if you are doing a serious
commercial product and you don't have a lawyer checking such things
anyway, you are a fool.

> the RFC already included links to the official source code.

Nonsense. An RFC should be self contained and IO don't see what's "too
big" about this one. The code is in a relatively compact region of the
RFC, so you can read the text and skip the code, if you want, without
undue effort.

> power programmer can quickly macro-clean it up.  The RFCSTRIP tool, well,
> what language is that? I'm not an *nix person. So this adds to the
> course, its a piece of cake for a sharp programmer, but even the sharpest
> knives eventually get dull.

This is just nonsense. If you are going to do anything serious with
code, you need a source code editor. With that, it is no big deal to
extract and test the code. Although I would agree that making the
RFCSTRIP tool available over the web, as many other tools are, would
be reasonable.

Thanks,
Donald
=============================
 Donald E. Eastlake 3rd   +1-508-333-2270 (cell)
 155 Beaver Street, Milford, MA 01757 USA
 [email-masked]

>> I'd actually tried the authors, but no reply yet (only a few days). I also
>> tried the RFC Editor thinking they might have e.g. XML from which extraction
>> might have been easier, but also no response yet. And I had found several
>> libraries, but not the RFC code. I can't see any suggestion that the library
>> you indicate includes that code, it also bills itself as a C++ library,
>> which the RFC code is not (and also not what I want, but that's not a
>> subject for this list).
>>
>> But the broader point is that if it's worth the IETF publishing the code
>> as an RFC, it's worth making the code available straightforwardly.
>>
>> For me, a thanks to Tony Hansen, who did the extraction for me. (That
>> makes me feel a little guilty, why should he do my work I could have done?)
>> But the point of posting on this list was to say that the code should be
>> available so that each person wanting that code doesn't have to do that
>> again.
>>
>> Christopher
>>
>

---

From: [email-masked] (Martin Rex)
Date: Sat, 29 Jun 2013 02:59:13 +0200 (CEST)

Dearlove, Christopher (UK) wrote:
>
> I'd actually tried the authors, but no reply yet (only a few days).
> from which extraction might have been easier, but also no response yet.

Extracting code from text is pretty trivial.

Use copy&paste from the output of below simple perl script
(which removes the pagebreaks):

-Martin

#!/usr/bin/perl
#
$rfcnum="6234";
$footerpattern="^Eastlake";
$headerpattern="^RFC ${rfcnum}";

$url = "http://tools.ietf.org/rfc/rfc${rfcnum}.txt";

open(IN,"curl '${url}'|")
  || die("Download of \"$url\" failed: $!\n");
@doc  = ();
$show = 2;
while($line = <IN>) {
    $line =~ tr/\r\n//d;
    if ( $line =~ m/${footerpattern}/io ) {
        $show = 0;
        while ( $doc[$#doc] eq "" ) {
            pop(@doc);
        }
    }
    if ( $show>0 ) {
        if ( 2==$show || $line ne "" ) {
            push(@doc, $line);
            $show = 2;
        }
    }
    if ( 0==$show && $line =~ m/^${headerpattern}/ ) {
        $show = 1;
    }
}

close(IN);

print join("\n",@doc), "\n";

---

From: [email-masked] (Martin Rex)
Date: Sat, 29 Jun 2013 03:46:31 +0200 (CEST)

> (which removes the pagebreaks):

Modified script which also does the extraction of the files
into the current directory as well (below).

  cc -c shatest.c sha1.c sha224-256.c sha384-512.c usha.c hmac.c hkdf.c
  cc -o shatest shatest.o sha1.o sha224-256.o sha384-512.o usha.o hmac.o hkdf.o
  ./shatest

it seems to work (for me on Linux, at least)

-Martin

#!/usr/bin/perl
#
$rfcnum="6234";

@origdoc = <IN>;
close(IN);

@doc  = ();
$show = 2;
$fname = "";
@file = ();

foreach $line ( @origdoc ) {

   $line =~ tr/\r\n//d;
   if ( $line =~ m/${footerpattern}/io ) {
      $show = 0;
      while ( $#doc >=0 && "" eq $doc[$#doc] ) {
         pop(@doc);
      }
      while ( $#file>=0 && "" eq $file[$#file] ) {
         pop(@file);
      }
   }
   if ( $show>0 ) {
      if ( 2==$show || $line ne "" ) {
         push(@doc, $line);
         $show = 2;
         if ( $fname eq "" && $line =~ m#^/\*\*.* (\w.+) \*\*#io ) {
            $fname = $1;
            $fname =~ tr/a-zA-Z0-9._-//cd;
            @file  = ();
         }
         if ( "" ne $fname ) {
            if ( $line =~ m/^\d/io ) {
               $filedata = join("\n",@file) . "\n";
               $filelen  = length($filedata);
               printf("Writing file \"$fname\" (len=${filelen})\n");
               open(OUT,">$fname")
                 || die("Could not write file \"$fname\": $!\n");
               print OUT $filedata;
               close(OUT);
               $fname = "";
            } else {
               push(@file, $line);
            }
         }
      }
   }
   if ( 0==$show && $line =~ m/${headerpattern}/io ) {
      $show = 1;
   }
}

# print join("\n",@doc), "\n";

---

From: John C Klensin <[email-masked]>
Date: Sat, 29 Jun 2013 17:25:08 -0400

--On Friday, June 28, 2013 16:41 -0400 Joe Abley
<[email-masked]> wrote:

>> If you really think you see a legal difference in doing the
>> second, fine; I propose that you are just searching for
>> problems that do not exist.
> 
> Quite possibly they don't, and I'm not presuming to talk for
> John. But the vague thoughts that crossed my mine were the
> issues that allowed Phil Zimmerman to publish PGP source code
> in book form and avoid conviction for munitions export without
> a licence.

That was one of the examples I had in mind, along with a few
others.  

However, given Paul's extensive legal experience and knowledge
of the details and applicability of export regulations and
associated case law, I apologize for wasting the time of people
on the list by raising this obviously spurious issue.  I am
delighted that, on the basis of that knowledge and experience,
Paul was willing to step in on this issue and assure us that
there is nothing to worry about.

   john