Subject: [sip-clf] I-D Action: draft-ietf-sipclf-format-07.txt

From: [email-masked]
Date: Thu, 04 Oct 2012 18:56:20 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories.
 This draft is a work item of the SIP Common Log Format Working Group of the IETF.

	Title           : Format for the Session Initiation Protocol (SIP) Common Log Format (CLF)
	Author(s)       : Gonzalo Salgueiro
                          Vijay Gurbani
                          Adam Roach
	Filename        : draft-ietf-sipclf-format-07.txt
	Pages           : 30
	Date            : 2012-10-04

Abstract:
   The SIPCLF Workgroup has defined a common log format framework for
   Session Initiation Protocol (SIP) servers.  This common log format
   mimics the successful event logging format found in well-known web
   servers like Apache and web proxies like Squid.  This document
   proposes an indexed text encoding format for the SIP Common Log
   Format (CLF) that retains the key advantages of a text-based format,
   while significantly increasing processing performance over a purely
   text-based implementation.  This file format adheres to the SIP CLF
   data model and provides an effective encoding scheme for all
   mandatory and optional fields that appear in a SIP CLF record.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-sipclf-format

There's also a htmlized version available at:
http://tools.ietf.org/html/draft-ietf-sipclf-format-07

A diff from the previous version is available at:
http://www.ietf.org/rfcdiff?url2=draft-ietf-sipclf-format-07

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Gonzalo Salgueiro <[email-masked]>
Date: Thu, 4 Oct 2012 22:13:22 -0400

Folks:

I have posted the -07 version of the SIP CLF Format Draft and it is now in the archives at [1].

This draft was meant to address the few remaining AD review items that were raised. The diffs between the -06 and the -07 versions are at [2]

The principal changes include:

- Introduced Base64 Encoded Byte (BEB) to deal with optional logging of non-UTF-8 message bodies
- Added an example using BEB for logging a binary body
- Clarified and expanded the text dealing with encoding, character escaping, and usage of SIP CLF and put it all into its own section
- Addressed several minor issues and nits
 
[1] http://tools.ietf.org/html/draft-ietf-sipclf-format
[2] http://www.ietf.org/rfcdiff?url2=draft-ietf-sipclf-format-07

Regards,

Gonzalo

On Mar 12, 2012, at 3:13 PM, [email-masked] wrote:

> 
> A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the SIP Common Log Format Working Group of the IETF.
> 
> 	Filename        : draft-ietf-sipclf-format-06.txt
> 	Pages           : 28
> 	Date            : 2012-03-12
> 
> A URL for this Internet-Draft is:
> http://www.ietf.org/internet-drafts/draft-ietf-sipclf-format-06.txt
> 
> This Internet-Draft can be retrieved at:
> ftp://ftp.ietf.org/internet-drafts/draft-ietf-sipclf-format-06.txt
> 
> _______________________________________________
> sip-clf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sip-clf
> 

On Oct 4, 2012, at 9:56 PM, [email-masked] wrote:

> _______________________________________________
> I-D-Announce mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/i-d-announce
> Internet-Draft directories: http://www.ietf.org/shadow.html
> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>

---

From: Robert Sparks <[email-masked]>
Date: Tue, 23 Oct 2012 16:07:05 -0500

The list has been silent on this. It needs review and comment please.

There is one part of the new text that is not right yet.

" The Value field for the optional element being logged MUST be Base64
    encoded if it has any characters outside of the UTF-8 character set."
doesn't make sense.

This should be saying something like  "if it contains any octets with 
values between 0 and 31 inclusive, or 127".

RjS

---

From: Gonzalo Salgueiro <[email-masked]>
Date: Tue, 23 Oct 2012 17:28:26 -0400

On Oct 23, 2012, at 5:07 PM, Robert Sparks wrote:

> This should be saying something like  "if it contains any octets with values between 0 and 31 inclusive, or 127".

Yep, this was a stupid oversight by me on the non-printable characters within UTF-8. I'll replace the above quoted text with the following:

The Value field for the optional element being logged MUST be Base64 encoded if it has any non-printable UTF-8 characters or characters outside of the UTF-8 character set.

Does that sound fine?

Gonzalo

>> [1] 
>> http://tools.ietf.org/html/draft-ietf-sipclf-format
>> 
>> [2] 
>> On Mar 12, 2012, at 3:13 PM, 
>> [email-masked]
>>  wrote:
>> 
>> 
>> On Oct 4, 2012, at 9:56 PM, 
>> [email-masked]
>>  wrote:
>> 
>> 
>>> or 
>>> ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>>> 
>>> 
>>> 
>

---

From: Adam Roach <[email-masked]>
Date: Tue, 23 Oct 2012 16:31:36 -0500

> - Introduced Base64 Encoded Byte (BEB) to deal with optional logging of non-UTF-8 message bodies

I think the text is a bit confused on what UTF-8 means. The phrase "any 
characters outside the UTF-8 character set" appears many times, and I'm 
afraid it doesn't really make any sense.

What you probably want to do is define a term somewhere in the document 
to formally specify "Those Things That Must Be Base-64 Encoded". I would 
recommend something like:

"For the purposes of this document, we define 'unprintable' to mean a 
string of octets that: (a) contains an octet with a value in the range 
of 0 to 31, inclusive; (b) contains an octet with a value of 127; or (c) 
contains any series of octets greater than or equal to 128 which do not 
form a valid UTF-8 sequence, as specified by [UNICODE]."

Then, your statements become far simpler; for example: "The Value field 
for the optional element being logged MUST be Base64 encoded if it is 
unprintable." And: "For an optionally logged message body (Tag=01) the 
BEB is set according to whether the message body is unprintable." And: 
"When optionally logging an entire SIP message (Tag=02) the BEB is set 
according to whether the message body portion is unprintable."

You get the idea.

/a

P.S. The above definition would require adding a reference for 
[UNICODE], like so:

    [UNICODE] The Unicode Consortium. The Unicode Standard,
              Version 6.2.0, (Mountain View, CA: The Unicode
              Consortium, 2012. ISBN 978-1-936213-07-8)
              <http://www.unicode.org/versions/Unicode6.2.0/>

---

From: Gonzalo Salgueiro <[email-masked]>
Date: Tue, 23 Oct 2012 17:56:18 -0400

Thanks Adam. The approach seems reasonable.  I'll make the suggested change.  

Just a few comments inline...

<snip>

> I think the text is a bit confused on what UTF-8 means. The phrase "any characters outside the UTF-8 character set" appears many times, and I'm afraid it doesn't really make any sense.
> 
> What you probably want to do is define a term somewhere in the document to formally specify "Those Things That Must Be Base-64 Encoded". I would recommend something like:
> 
> "For the purposes of this document, we define 'unprintable' to mean a string of octets that: (a) contains an octet with a value in the range of 0 to 31, inclusive; (b) contains an octet with a value of 127; or (c) contains any series of octets greater than or equal to 128 which do not form a valid UTF-8 sequence, as specified by [UNICODE]."

I'll have to expand the definition a bit since there are octets greater than 128 that are non-printable but still valid UTF-8 control characters.  I can go with something like:

"For the purposes of this document, we define 'unprintable' to mean a string of octets that: (a) contains an octet with a value in the range of 0 to 31, inclusive; (b) contains an octet with a value of 127, (c) contains any octet greater than or equal to 128 which is a formatting or control character (such as 128 to 159) within the UTF-8 character set; or (d) falls outside the UTF-8 character range, as specified by [UNICODE]."

Does that sound ok?

> Then, your statements become far simpler; for example: "The Value field for the optional element being logged MUST be Base64 encoded if it is unprintable." And: "For an optionally logged message body (Tag=01) the BEB is set according to whether the message body is unprintable." And: "When optionally logging an entire SIP message (Tag=02) the BEB is set according to whether the message body portion is unprintable."

I'll make the appropriate replacements once we agree on the definition.
> You get the idea.

I do. Thanks.

Cheers,

Gonzalo

> 
> /a
> 
> 
> P.S. The above definition would require adding a reference for [UNICODE], like so:
>

---

From: Adam Roach <[email-masked]>
Date: Tue, 23 Oct 2012 17:26:57 -0500

> "For the purposes of this document, we define 'unprintable' to mean a string of octets that: (a) contains an octet with a value in the range of 0 to 31, inclusive; (b) contains an octet with a valu…

I think we're still talking past each other here.

"Outside the UTF-8 character range" simply isn't a sensible thing to 
say. What we're talking about putting into a log record is a series of 
*octets*, not a series of *characters*. UTF-8 is an encoding that 
defines how octets are put together to make characters.

Once you start talking about the octets as if they *are* characters, 
you're conflating two very different things. So, for example, you can't 
talk about "a string of octets that... falls outside the UTF-8 character 
range."

You can talk about a string of bytes that does not form a valid UTF-8 
sequence, and that's almost certainly what you want to say here.

I'm also getting a bit lost in what you mean when you say "which is a 
formatting or control character (such as 128 to 159)." Keep in mind that 
we're still talking about *octets* here, not characters. In UTF-8, 
there's nothing special about an octet with a value of 128. There's 
nothing special about an octet with a value of 159. Both can appear as 
the second octet in a two-octet character. Or the second or third octet 
in a three-octet character. And so on. The same goes for everything 
between 128 and 191.

Now, octet values of 192, 193, and 245-255 won't appear in valid UTF-8. 
If we wanted to be abundantly careful, we could call those out as being 
invalid. But I think we catch those just fine if we talk about octets 
that form valid UTF-8 sequences.

Or are you meaning to call out UTF-8 code points like U+0080 (the 
Latin-1 padding character)? Because that has nothing to do with an 
*octet* with a value of 128. It would be encoded as a two-octet sequence 
starting with 194. However, if we're intending to go down the rabbit 
hole of making decisions about whether to Base-64 encode based on which 
UTF-8 codepoints we want to consider "printable," then we've got years 
of draft refinement ahead of us (I can already imagine the right-to-left 
mark arguments). That way lies madness.

All of which is a very long winded way to say: octets are not characters 
and characters are not octets; and you need to write the text in a way 
that does not mix them with each other.

/a

---

From: Gonzalo Salgueiro <[email-masked]>
Date: Wed, 24 Oct 2012 02:30:42 -0400

Adam -

Let me first state that I'm certainly no expert in UTF-8 and will happily defer to you on this. I certainly get your point about octets versus characters, especially since UTF-8 allows for multi-byte characters.  When I made the statement "outside the UTF-8 character range" I was referring to the discrete set of characters from U+0000 to U+7FFFFFFF  (or U+10FFFF if using the restricted RFC 3629 definition). I assumed this was a clearly bounded set.

To be clear, my only intent was to expand your original definition of 'unprintable' beyond the C0 control codes (U+0000 to U+001F, U+007F) to also include the well known C1 control codes (U+0080 to U+009F). To avoid confusion, I should have used the Unicode code point notation.  I certainly don't want to try and go down the road of specifying what characters are 'printable', but I thought C1 control codes were worth stating explicitly in the same way you did C0 control codes. Your original definition seemed unnecessarily restrictive since the C1 control codes, for example, are not printable but are certainly a valid UTF-8 sequence of octets greater than or equal to 128. Does that make sense?

If you think your original definition of 'unprintable' is broad enough, then I will use that.

Cheers,

Gonzalo

On Oct 23, 2012, at 6:26 PM, Adam Roach wrote:

> "Outside the UTF-8 character range" simply isn't a sensible thing to say. What we're talking about putting into a log record is a series of *octets*, not a series of *characters*. UTF-8 is an encoding that defines how octets are put together to make characters.
> 
> Once you start talking about the octets as if they *are* characters, you're conflating two very different things. So, for example, you can't talk about "a string of octets that... falls outside the UTF-8 character range."
> 
> You can talk about a string of bytes that does not form a valid UTF-8 sequence, and that's almost certainly what you want to say here.
> 
> I'm also getting a bit lost in what you mean when you say "which is a formatting or control character (such as 128 to 159)." Keep in mind that we're still talking about *octets* here, not characters. In UTF-8, there's nothing special about an octet with a value of 128. There's nothing special about an octet with a value of 159. Both can appear as the second octet in a two-octet character. Or the second or third octet in a three-octet character. And so on. The same goes for everything between 128 and 191.
> 
> Now, octet values of 192, 193, and 245-255 won't appear in valid UTF-8. If we wanted to be abundantly careful, we could call those out as being invalid. But I think we catch those just fine if we talk about octets that form valid UTF-8 sequences.
> 
> Or are you meaning to call out UTF-8 code points like U+0080 (the Latin-1 padding character)? Because that has nothing to do with an *octet* with a value of 128. It would be encoded as a two-octet sequence starting with 194. However, if we're intending to go down the rabbit hole of making decisions about whether to Base-64 encode based on which UTF-8 codepoints we want to consider "printable," then we've got years of draft refinement ahead of us (I can already imagine the right-to-left mark arguments). That way lies madness.
> 
> All of which is a very long winded way to say: octets are not characters and characters are not octets; and you need to write the text in a way that does not mix them with each other.
> 
> /a
>

---

From: Peter Musgrave <[email-masked]>
Date: Wed, 24 Oct 2012 13:01:10 -0400

Sorry to be so absent.

Great that we're gonna get this puppy done.

I will read through front to back this weekend.

Can I ask others to look at the diffs and comment by Monday Oct 29th?

Then it seems we'll need a minor fix up to -08 and we're good to write this
up again.

If anyone wants more time, please let me know.

Regards,

Peter Musgrave (as WG chair)

On Wed, Oct 24, 2012 at 2:30 AM, Gonzalo Salgueiro <[email-masked]>wrote:

> Adam -
>
> Let me first state that I'm certainly no expert in UTF-8 and will happily
> defer to you on this. I certainly get your point about octets versus
> characters, especially since UTF-8 allows for multi-byte characters.  When
> I made the statement "outside the UTF-8 character range" I was referring to
> the discrete set of characters from U+0000 to U+7FFFFFFF  (or U+10FFFF if
> using the restricted RFC 3629 definition). I assumed this was a clearly
> bounded set.
>
> To be clear, my only intent was to expand your original definition of
> 'unprintable' beyond the C0 control codes (U+0000 to U+001F, U+007F) to
> also include the well known C1 control codes (U+0080 to U+009F). To avoid
> confusion, I should have used the Unicode code point notation.  I certainly
> don't want to try and go down the road of specifying what characters are
> 'printable', but I thought C1 control codes were worth stating explicitly
> in the same way you did C0 control codes. Your original definition seemed
> unnecessarily restrictive since the C1 control codes, for example, are not
> printable but are certainly a valid UTF-8 sequence of octets greater than
> or equal to 128. Does that make sense?
>
> string of octets that: (a) contains an octet with a value in the range of 0
> to 31, inclusive; (b) contains an octet with a value of 127, (c) contains
> any octet greater than or equal to 128 which is a formatting or control
> character (such as 128 to 159) within the UTF-8 character set; or (d) falls
> outside the UTF-8 character range, as specified by [UNICODE]."
> >>
> >> Does that sound ok?
> >
> *octets*, not a series of *characters*. UTF-8 is an encoding that defines
> how octets are put together to make characters.
> >
> we're still talking about *octets* here, not characters. In UTF-8, there's
> nothing special about an octet with a value of 128. There's nothing special
> about an octet with a value of 159. Both can appear as the second octet in
> a two-octet character. Or the second or third octet in a three-octet
> character. And so on. The same goes for everything between 128 and 191.
> >
> invalid. But I think we catch those just fine if we talk about octets that
> form valid UTF-8 sequences.
> >
> Latin-1 padding character)? Because that has nothing to do with an *octet*
> with a value of 128. It would be encoded as a two-octet sequence starting
> with 194. However, if we're intending to go down the rabbit hole of making
> decisions about whether to Base-64 encode based on which UTF-8 codepoints
> we want to consider "printable," then we've got years of draft refinement
> ahead of us (I can already imagine the right-to-left mark arguments). That
> way lies madness.
> >
> and characters are not octets; and you need to write the text in a way that
> does not mix them with each other.
> >
> > /a
> >
>

---

From: Adam Roach <[email-masked]>
Date: Wed, 24 Oct 2012 14:04:11 -0500

So, I think I understand your point. (And, to be clear, I don't intend 
to hold myself out as a Unicode expert either). The rationale behind 
what I'm proposing has two main assumptions:

  * The whole point of choosing a non-binary record format is allowing
    the use of commandline tools to work with records. Legacy
    commandline tools will frequently make assumptions about what
    octetes that correspond to ASCII control characters mean (e.g. 10
    and 13), and take certain actions based on them. By and large, I
    wouldn't expect the same kind of reaction to octetes that represent
    extended control characters, like those in the C1 set. I would
    expect normal commandline tools to simply treat octets in the range
    of 128-255 as (relatively) non-special, and process them normally.
    Cursory experimentation with the commandline tools I have easily
    available leads me to believe that such is the case.

  * I presume that we do not want to force Base-64 encoding due to the
    simple presence of an accented Latin character. Similarly, we
    probably don't want a small number of characters from the
    CJK/Cyrillic/Arabic/etc codepages to trigger Base-64 either.

If I'm wrong about either of these assumptions, then we need to re-visit 
what I'm proposing.

/a

---

From: Gonzalo Salgueiro <[email-masked]>
Date: Wed, 24 Oct 2012 19:41:17 -0400

Thanks, Adam.  I agree with both your assumptions. I think where we diverged is whether C1 was on an equal footing with C0 as far as their 'unprintability'.  If C1 is handled nicely by text processing tools then we can omit them. We all have a vested interest in minimizing the amount we Base64 encode these optionally logged bodies and whole messages within SIP CLF since it drastically reduces its usefulness and can introduce information loss. Thus, I absolutely agree that we don't want an accented Latin character or other valid UTF-8 non-control characters (CJK/Cyrillic/Arabic/etc) to trigger a Base64 encoding.  At this point I think I will go with your original text for defining 'unprintable' within the context of this document.

Cheers,

Gonzalo

On Oct 24, 2012, at 3:04 PM, Adam Roach wrote:

> So, I think I understand your point. (And, to be clear, I don't intend to hold myself out as a Unicode expert either). The rationale behind what I'm proposing has two main assumptions:
> 	• The whole point of choosing a non-binary record format is allowing the use of commandline tools to work with records. Legacy commandline tools will frequently make assumptions about what octetes that correspond to ASCII control characters mean (e.g. 10 and 13), and take certain actions based on them. By and large, I wouldn't expect the same kind of reaction to octetes that represent extended control characters, like those in the C1 set. I would expect normal commandline tools to simply treat octets in the range of 128-255 as (relatively) non-special, and process them normally. Cursory experimentation with the commandline tools I have easily available leads me to believe that such is the case.
> 
> 	• I presume that we do not want to force Base-64 encoding due to the simple presence of an accented Latin character. Similarly, we probably don't want a small number of characters from the CJK/Cyrillic/Arabic/etc codepages to trigger Base-64 either. 
> If I'm wrong about either of these assumptions, then we need to re-visit what I'm proposing.
> 
> /a
> 
>

---

From: Peter Musgrave <[email-masked]>
Date: Sun, 28 Oct 2012 18:29:28 -0400

<individual>
Ok - I know less about UTF-8 than either Adam or Gonzalo, but the rationale of being able to use text base tools is one I support. 

I read through the diffs and did not find anything else I wanted to bring up. 
</individual>

<as chair>
Once we agree on "printable" is there anything else people want to see altered?
</as chair>

Thanks, 

Peter

One nit: 
4.3 s/macine/machine/