Subject: Mobile headers

From: "Martin Nilsson" <[email-masked]>
Date: Fri, 11 Oct 2013 18:42:30 +0200

So I've finally scrubbed the mobile HTTP header captures that I mentioned  
earlier. The capture was made on one of our Opera Mini download servers in  
March. The request headers were then parsed and the most broken requests  
removed. After that all requests coming from internal systems, monitoring  
and Opera Mini servers were removed. I'm aware that there are some cases  
where the headers are not parsed correctly when the linebreaks keep  
changing between different header lines. Finally the headers are scrubbed  
where everything resembling IP numbers and longer sequences of digits are  
replaced with random, similar looking data. Also the header values of  
y-msisdn, referer, authorization and proxy-authorization are replaced with  
random data. All request payload data is stripped as well. The resulting  
data is written into a a json structure with one object per request where  
lower case header names maps to a string, or an array of strings in the  
case of multiple headers. Order is preserved in the array, but not amongst  
the different headers (they are serialized sorted). The file contains  
203586 requests and is 154MB uncompressed, 20MB compressed.

http://people.opera.com/nilsson/headers.json.gz

Please report report any issues or concerns.

/Martin Nilsson

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 16 Oct 2013 12:54:59 -0700

Hi Martin,

Thanks for that.

The file isn't valid JSON; while some issues are easy to fix (e.g., wrapping the whole thing in an array), there appear to be a number of other issues; e.g., at line 918:

{
  "accept" : "application/vnd.wap.xhtml+xml, application/xhtml+xml, text/html, $
  "accept-encoding" : {
    "deflate, gzip, x-gzip, identity, *;q=0"
    "deflate, gzip"
,
  "connection" : "Keep-Alive",
  "content-length" : "19",
  "content-type" : "text/plain",
  "host" : "m.opera.com",

Notice the missing delimiter after the first accept-encoding value, and what appears to be a missing end bracket.

This appears to happen a lot (I guess I could fix it up with a regex, but ew).

I *think* what you mean to do is:

{
  "accept" : "application/vnd.wap.xhtml+xml, application/xhtml+xml, text/html, $
  "accept-encoding" : [
    "deflate, gzip, x-gzip, identity, *;q=0",
    "deflate, gzip"
  ],
  "connection" : "Keep-Alive",
  "content-length" : "19",
  "content-type" : "text/plain",
  "host" : "m.opera.com",

to denote when there are multiple instances of a single header field?

Cheers,

On 11/10/2013, at 9:42 AM, Martin Nilsson <[email-masked]> wrote:

> 
> So I've finally scrubbed the mobile HTTP header captures that I mentioned earlier. The capture was made on one of our Opera Mini download servers in March. The request headers were then parsed and the most broken requests removed. After that all requests coming from internal systems, monitoring and Opera Mini servers were removed. I'm aware that there are some cases where the headers are not parsed correctly when the linebreaks keep changing between different header lines. Finally the headers are scrubbed where everything resembling IP numbers and longer sequences of digits are replaced with random, similar looking data. Also the header values of y-msisdn, referer, authorization and proxy-authorization are replaced with random data. All request payload data is stripped as well. The resulting data is written into a a json structure with one object per request where lower case header names maps to a string, or an array of strings in the case of multiple headers. Order is preserved in the array, but not amongst the different headers (they are serialized sorted). The file contains 203586 requests and is 154MB uncompressed, 20MB compressed.
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 16 Oct 2013 22:14:21 +0200

* Mark Nottingham wrote:
>Notice the missing delimiter after the first accept-encoding value, and 
>what appears to be a missing end bracket.

(The `accept` field value also appears to be truncated...)
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 16 Oct 2013 13:15:15 -0700

On 16/10/2013, at 1:14 PM, Bjoern Hoehrmann <[email-masked]> wrote:
> 
> (The `accept` field value also appears to be truncated…)

I think that's my editor ;)

Cheers,

---

From: "Martin Nilsson" <[email-masked]>
Date: Thu, 17 Oct 2013 01:51:06 +0200

On Wed, 16 Oct 2013 21:54:59 +0200, Mark Nottingham <[email-masked]> wrote:

> Hi Martin,
>
> Thanks for that.
>
> The file isn't valid JSON; while some issues are easy to fix (e.g.,  
> wrapping the whole thing in an array), there appear to be a number of  
> other issues; e.g., at line 918:

I've added a version header, a wrapper and fixed the mutliple header  
encoding. There are still some issues with 8-bit characters encoded the  
wrong way, or not at all.

/Martin Nilsson

-- 
Using Opera's mail client: http://www.opera.com/mail/