Subject: Re: [Sipping] Alternate CLF syntax proposal

From: Hisham Khartabil <[email-masked]>
Date: Thu, 2 Apr 2009 12:42:56 +1100

Yeah, its for troubleshooting, but I suspect people will use it for charging.

Hisham

2009/3/27 Hadriel Kaplan <[email-masked]>:
>
> If the purpose of it is troubleshooting, or threat analysis type stuff, I think the following format satisfies the needs as well:
> http://wiki.wireshark.org/Development/LibpcapFileFormat
>
> The advantages:
> 1) There's running code
> 2) There's open source, on all operating systems, and also for commercial use
> 3) It's used by many people
> 4) There are many tools which accept/process it
> 5) It's fast to write/save
> 6) It supports a sub-second timestamp
> 7) It supports length encoding of packets, so you can skip past them
> 8) It supports truncated saving of packets, so you don't have to save all of very big ones
> 9) It records the method name or response code very early in the saved log entry for each packet
>
> Disadvantages:
> 1) Nothing much new to specify, except to document it?
> 2) It's a little tricky for SIP/TLS, where you basically have to create fake segments/packets for the low layers, and same may be true for SIP/TCP depending on when you record the log
> 3) It doesn't provide a way to report internal system events/actions/info (although we could fix that)
> 4) Afaik, there is no specific remote push/streaming mechanism for it defined (there was an attempt at it, but not final)
>
> -hadriel
>
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]] On Behalf
>> Of Vijay K. Gurbani
>> Sent: Wednesday, March 25, 2009 10:58 PM
>>
>> Adam Roach wrote:
>> > In the spirit of "send text," I've put together a straw-man proposal for
>> > an easy-to-generate and fast-to-process extensible format for saving SIP
>> > log messages:
>> >
>> > http://www.ietf.org/internet-drafts/draft-roach-sipping-clf-syntax-
>> 00.txt
>> [...]
>>
>> Adam: Essentially you are advocating for a table-of-content
>> type of approach where you read the ToC and index straight
>> to where you want to go.  I have worked on SIP parsers
>> designed this way.
>>
>> The parsing is optimized, yes, when compared to the ASCII
>> version -- though perl can do wonders, but not to outperform
>> binary parsing.  The disadvantage is that you loose readability
>> and would need specialized tools to, say, grep through such
>> a file.
>>
>> It will be interesting to see what others think...
>>
>> Thanks,
>>
>> - vijay
>> --
>> Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
>> 1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60566 (USA)
>> Email: vkg@{alcatel-lucent.com,bell-labs.com,acm.org}
>> Web:   http://ect.bell-labs.com/who/vkg/
>> _______________________________________________
>> Sipping mailing list  https://www.ietf.org/mailman/listinfo/sipping
>> This list is for NEW development of the application of SIP
>> Use [email-masked] for questions on current sip
>> Use [email-masked] for new developments of core SIP
> _______________________________________________
> Sipping mailing list  https://www.ietf.org/mailman/listinfo/sipping
> This list is for NEW development of the application of SIP
> Use [email-masked] for questions on current sip
> Use [email-masked] for new developments of core SIP
>

---

From: Vijay Gurbani <[email-masked]>
Date: Wed, 01 Apr 2009 23:19:39 -0500

> Yeah, its for troubleshooting, but I suspect people will use it for charging.

I suppose people could use it for whatever purposes they deem
necessary -- I don't think we can stop them.  However, the
draft expressly notes that the CLF format is not a replacement
for a CDR system.  Now, if people will use it as such, then at least
they've been duly notified.

- vijay
-- 
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60566 (USA)
Email: vkg@{alcatel-lucent.com,bell-labs.com,acm.org}
WWW:   http://www.alcatel-lucent.com/bell-labs

---

From: "Dale Worley" <[email-masked]>
Date: Fri, 10 Apr 2009 15:59:34 -0400

On Tue, 2009-03-31 at 12:42 -0400, Hadriel Kaplan wrote:
> That depends on what you think a "Common Log Format" would be useful
> for.  The premise was that it's for troubleshooting and analysis
> systems.  AFAICT, nothing short of the entire SIP message sequence,
> including contents, and even IP/UDP or TCP headers, would be
> sufficient for that in practice.

In our PBX, we've discovered that that is correct.  We do include the
entire SIP message, of course, though we restrict the IP/UDP/TCP
information to the IP address/port of the far end.

Dale

---

From: Adam Roach <[email-masked]>
Date: Wed, 15 Apr 2009 10:08:21 -0500

Theo Zourzouvillys wrote:
> On Tue, Mar 31, 2009 at 5:42 PM, Hadriel Kaplan<[email-masked]>  wrote:
>    
>> But anyway, attached is an example CLF file using only the currently defined fields from draft-gurbani-sipping-clf-01, I think, encoded in a different format from the draft.
>>      
>
> The problem is this isn't really any different to the plain text
> format, except it includes an initial packet length header and thus
> you can skip over the whole record easily.  You still need to parse
> the contents of the pacp record to match any information such as a
> specific via header or to tag.

Precisely. Using PCAP captures straight up makes pretty much all of the 
problems that I'm trying to solve worse instead of better. You still 
need to dredge through long records to find the fields that you're 
filtering on, but now the records are entire SIP messages instead of 
extracted data. Instead of taking the time to find records of interest 
from o(m) to o(0.3m), you've INCREASED it to o(n), where n >> m.

I'll point out that my proposal incorporates the ability to include 
entire SIP messages, if necessary (with the decision about whether to 
include the message possible on a record-by-record basis), so you can 
retain any debugging ability you have with PCAP -- but you get fast 
indexing, and the ability to store meta-information that appears in 
neither PCAP headers nor in the SIP message itself.

/a

---

From: Adam Roach <[email-masked]>
Date: Wed, 15 Apr 2009 10:16:14 -0500

Adam Roach wrote:
>
> Instead of taking the time to find records of interest from o(m) to 
> o(0.3m), you've INCREASED it to o(n), where n >> m.

Sorry... that should have been o(0.03m), not o(0.3m).

/a