Subject: [http-state] Our first test

From: Adam Barth <[email-masked]>
Date: Fri, 7 Aug 2009 00:27:22 -0700

I've added a test harness as well as our first test to the git
repository.  I looked around at various test harnesses, and the
simplest thing appeared to be to appropriate testserver.py from
Chromium.  (Note that I might be biased by having worked on Chromium.)
 We can, of course, change our minds later.  Currently our test
harness is optimized for parser tests, which I think is a good place
to start.  When we move on to testing the various attributes, we'll
probably need something fancier.

== How to Write a Test ==

Writing a parsing test is super simple.  To make a test, create two files:

foo-test
foo-expected

In foo-test, put however many Set-Cookie headers you like.  In
foo-expect, put what you expect the Cookie header to be.  You can see
the example 0001 test at:

http://github.com/abarth/http-state/tree/52b8b5c24282e36c8c080a563df18074865bb79c/tests/data/parser

I'm happy to report that this test passes in Firefox 3.5, Safari 4,
Google Chrome 3, and Opera 9.64.  If you'd like to contribute tests,
please send them to this mailing lists.

== How to Run a Test ==

Running a single test is also easy:

1) Install python and git
2) $ git clone git://github.com/abarth/http-state.git
3) $ ./http-state/tests/run-test-server
4) Visit http://localhost:8888/cookie-parser?foo in the user agent
you'd like to test

If the page says PASS, then the test passed.  If the test failed, the
page should say what header it saw and what header it was expecting.
Once we have a few tests, we should probably add a way to run all the
tests at once, but I haven't gotten that far.

Enjoy,
Adam

---

From: Daniel Stenberg <[email-masked]>
Date: Fri, 7 Aug 2009 09:36:10 +0200 (CEST)

On Fri, 7 Aug 2009, Adam Barth wrote:

Nice!

> 3) $ ./http-state/tests/run-test-server

./http-state/tests/run-test-server
python: can't open file '../tools/testserver/testserver.py': [Errno 2] No such 
file or directory

but...

  $ cd http-state/tests && ./run-test-server

works

> 4) Visit http://localhost:8888/cookie-parser?foo in the user agent

I suspect there's something further wrong here since I get a 404 from that!

> If the page says PASS, then the test passed.  If the test failed, the page 
> should say what header it saw and what header it was expecting. Once we have 
> a few tests, we should probably add a way to run all the tests at once, but 
> I haven't gotten that far.

A few comments:

I want to be able to test command line driven tools against it so I would like 
the plain URLs listed/extracted somehow.

This test seems limited to only test cookies on 'localhost' or how are you 
going to test other host names without using the test server as a proxy?

Since we're discussing different cookie implementations and different 
behaviors, I have a hard time to see that we can have a single file with 
"expected" results for all test cases. I mean who's to tell which is right and 
which is wrong?

-- 

  / daniel.haxx.se

---

From: Adam Barth <[email-masked]>
Date: Fri, 7 Aug 2009 00:50:50 -0700

>  $ cd http-state/tests && ./run-test-server

Ah, thanks.  That's what I get for not testing my testing instructions.  :)

> I suspect there's something further wrong here since I get a 404 from that!

Yeah, that was for the hypothetical foo test that you wrote in the
previous section (and put in the tests/data/parser folder).  If you
want to run the test we have, try

http://localhost:8888/cookie-parser?0001

> A few comments:
>
> I want to be able to test command line driven tools against it so I would
> like the plain URLs listed/extracted somehow.

You can get this information using this command:

$ ls http-state/tests/data/parser/*-test

> going to test other host names without using the test server as a proxy?

Currently, these tests are must useful for the parser.  If we want to
test the behavior of attributes, like Domain, we'll need something
fancier.  My plan is to cross that bridge when we come to it.  I'd
certainly welcome patches that make this possible.  :)

> "expected" results for all test cases. I mean who's to tell which is right
> and which is wrong?

That, I'm afraid, is our job as a working group.

If we like, we can document the behavior of various user agents by
adding results like foo-expected-ie7 or foo-expected-firefox-3.5.
Ultimately, though, we need to decide which is the correct behavior
according to the spec.

Adam

---

From: Julian Reschke <[email-masked]>
Date: Fri, 07 Aug 2009 09:54:57 +0200

Adam Barth wrote:
> ...
> ...

Disagreed.

If there's no interoperability for a specific aspect anyway then it's 
totally acceptable to just state that the behavior is undefined (at 
least as far as the IETF is concerned).

BR, Julian

---

From: Adam Barth <[email-masked]>
Date: Fri, 7 Aug 2009 00:58:34 -0700

On Fri, Aug 7, 2009 at 12:54 AM, Julian Reschke<[email-masked]> wrote:
> totally acceptable to just state that the behavior is undefined (at least as
> far as the IETF is concerned).

Fair enough.  Perhaps we should use a file named foo-undefined to mean
the behavior on a particular test is undefined.

Adam

---

From: Daniel Stenberg <[email-masked]>
Date: Fri, 7 Aug 2009 10:20:28 +0200 (CEST)

On Fri, 7 Aug 2009, Adam Barth wrote:

> Yeah, that was for the hypothetical foo test that you wrote in the previous 
> section (and put in the tests/data/parser folder).  If you want to run the 
> test we have, try
>

Ah yes, that works. I even tried adding a few new tests and it worked fine. 
Although I had problems to add a test that ended up with no cookies sent (like 
all was sent expired) or with multiple cookie sent in the test file.

But I won't push you, I figure you can proceed and add some of those simple 
test cases and then I'll just see how you've done them when I want to add more 
cases later on.

> $ ls http-state/tests/data/parser/*-test

Nice. I'll write up an automated test for my own pleasure with this.

I figure we can also help those who wants to test this with a browser, to make 
a script like:

for f in tests/data/parser/*-test; do
   c=`echo $f | sed -e 's:.*/::g'`;
   num=`echo $c | sed -e 's:[^0-9]::g'`

   echo "<a href=\"http://localhost:8888/cookie-parser?$num\">test $num</a>"
done

To get an HTML page with links to all existing tests.

-- 

  / daniel.haxx.se

---

From: Adam Barth <[email-masked]>
Date: Fri, 7 Aug 2009 09:52:59 -0700

On Fri, Aug 7, 2009 at 1:20 AM, Daniel Stenberg<[email-masked]> wrote:
>> previous section (and put in the tests/data/parser folder).  If you want to
>> run the test we have, try
>>
> Although I had problems to add a test that ended up with no cookies sent
> (like all was sent expired) or with multiple cookie sent in the test file.

If you'd be willing to share those tests, that would be great.

> test cases and then I'll just see how you've done them when I want to add
> more cases later on.

Patches welcome.  I bet there are some simple changes to testserver.py
that would fix these bugs.

> I figure we can also help those who wants to test this with a browser, to
> make a script like:
>

I've added this code to run-test-server so it prints a list of test
URLs when you start it.  We may need another solution as we add more
tests, but it should be useful for now.

Adam

---

From: Daniel Stenberg <[email-masked]>
Date: Fri, 7 Aug 2009 23:33:57 +0200 (CEST)

> If you'd be willing to share those tests, that would be great.

It was really nothing particular to boast about yet. See attachment. I want to 
start adding cases for specific stuff. This is two cases of cookies with one 
sett of expires date string syntax used. With expired and non-expired cookies.

We could continue and add the same test with all date formats we believe are 
used in the wild. At least if we can manage to make tests that can send 
multiple Set-Cookie: headers and that can deal with expected to be blank.

Then we can continue to try simple domain and path matching tests, and so on. 
I wouldn't mind working on this, but I don't want to step on anyone's toes.

Oh, btw. Perhaps we should add a 3rd file for each test that describes the 
test (and result?) in plain english. When we add more tests, they'll soon 
start to overlap and just be slight variants unless we document them properly.

-- 

  / daniel.haxx.se

diff --git a/tests/data/parser/0002-expected b/tests/data/parser/0002-expected
new file mode 100644
index 0000000..b14d4f6
--- /dev/null
+++ b/tests/data/parser/0002-expected
@@ -0,0 +1 @@
+Cookie: foo=bar
diff --git a/tests/data/parser/0002-test b/tests/data/parser/0002-test
new file mode 100644
index 0000000..a3eaff0
--- /dev/null
+++ b/tests/data/parser/0002-test
@@ -0,0 +1 @@
+Set-Cookie: foo=bar; Expires=Fri, 07 Aug 2019 08:04:19 GMT
diff --git a/tests/data/parser/0003-expected b/tests/data/parser/0003-expected
new file mode 100644
index 0000000..96768f9
--- /dev/null
+++ b/tests/data/parser/0003-expected
@@ -0,0 +1 @@
+foo2=bar2 
diff --git a/tests/data/parser/0003-test b/tests/data/parser/0003-test
new file mode 100644
index 0000000..5d4fcf0
--- /dev/null
+++ b/tests/data/parser/0003-test
@@ -0,0 +1,2 @@
+Set-Cookie: foo=bar; Expires=Fri, 07 Aug 2007 08:04:19 GMT
+Set-Cookie: foo2=bar2; Expires=Fri, 07 Aug 2017 08:04:19 GMT

---

From: Adam Barth <[email-masked]>
Date: Fri, 7 Aug 2009 15:22:46 -0700

On Fri, Aug 7, 2009 at 2:33 PM, Daniel Stenberg<[email-masked]> wrote:
> It was really nothing particular to boast about yet. See attachment.

Great thanks.  Committed as 213905b24bce835a0b547663d324995491194f79.

> I want
> to start adding cases for specific stuff. This is two cases of cookies with
> one sett of expires date string syntax used. With expired and non-expired
> cookies.

By the way, 0003-expected wasn't properly formated.  You need to add
the "Cookie: " bit before the contents of the cookie.

> multiple Set-Cookie: headers and that can deal with expected to be blank.

These should be working now.

> Then we can continue to try simple domain and path matching tests, and so
> on. I wouldn't mind working on this, but I don't want to step on anyone's
> toes.

Please do.  If you like, you can name them something like
expires0023-test and path0012-test to make it easier to see what
they're trying to test.

If you'd like to test path stuff, you should be able to use URLs like

http://localhost:8888/cookie-parser/foo/bar/baz?path0012

> start to overlap and just be slight variants unless we document them

Go for it.  I recommend naming them something like "path0012-notes".

BTW, I'm about to write a script that runs all the tests on CURL to
make sure we don't add bogus expected results.

Adam

---

From: Adam Barth <[email-masked]>
Date: Fri, 7 Aug 2009 15:35:40 -0700

> make sure we don't add bogus expected results.

Ok, this is done now.  You can run all the test on CURL after starting
the test server using this command:

run-all-tests-for-curl

As with everything, this will probably require iteration as we start
testing more things.

Adam

---

From: Daniel Stenberg <[email-masked]>
Date: Sat, 8 Aug 2009 22:32:21 +0200 (CEST)

> the "Cookie: " bit before the contents of the cookie.

Hah, silly me. Thanks for being gentle! ;-)

Attached to this mail is a patch with 7 additional tests for max-age, secure 
and a custom assigned attribute (with and without quotes and with and without 
trailing semicolon).

> http://localhost:8888/cookie-parser/foo/bar/baz?path0012

Ah, that's nice. But how do we tell the caller this so that we can generate 
the URL properly that would be used to run this imaginary test 0012? Maybe 
some extra hint in a file for the specific test or something?

> BTW, I'm about to write a script that runs all the tests on CURL to make 
> sure we don't add bogus expected results.

Cool. But it wouldn't really surprise me if we soon figure out a case that 
actually does make the curl results deviate from that of one or more 
browsers...

And to make things really interesting for domain matching etc, we need to be 
able to run against servers claiming to be something else than localhost. It 
seems the test server already partly supports full URIs in the request-URI 
field, but not quite? It doesn't seem to return the correct response when I 
try:

   curl -L -c /dev/null "http://localhost:8888/cookie-parser?0001" -x localhost:8888

-- 

  / daniel.haxx.se

diff --git a/tests/data/parser/0005-expected b/tests/data/parser/0005-expected
new file mode 100644
index 0000000..b14d4f6
--- /dev/null
+++ b/tests/data/parser/0005-expected
@@ -0,0 +1 @@
+Cookie: foo=bar
diff --git a/tests/data/parser/0005-test b/tests/data/parser/0005-test
new file mode 100644
index 0000000..4de576a
--- /dev/null
+++ b/tests/data/parser/0005-test
@@ -0,0 +1 @@
+Set-Cookie: foo=bar; max-age=10000;
diff --git a/tests/data/parser/0006-expected b/tests/data/parser/0006-expected
new file mode 100644
index 0000000..e69de29
diff --git a/tests/data/parser/0006-test b/tests/data/parser/0006-test
new file mode 100644
index 0000000..83e41ec
--- /dev/null
+++ b/tests/data/parser/0006-test
@@ -0,0 +1 @@
+Set-Cookie: foo=bar; max-age=0;
diff --git a/tests/data/parser/0007-expected b/tests/data/parser/0007-expected
new file mode 100644
index 0000000..b14d4f6
--- /dev/null
+++ b/tests/data/parser/0007-expected
@@ -0,0 +1 @@
+Cookie: foo=bar
diff --git a/tests/data/parser/0007-test b/tests/data/parser/0007-test
new file mode 100644
index 0000000..af7b672
--- /dev/null
+++ b/tests/data/parser/0007-test
@@ -0,0 +1 @@
+Set-Cookie: foo=bar; version=1;
diff --git a/tests/data/parser/0008-expected b/tests/data/parser/0008-expected
new file mode 100644
index 0000000..b14d4f6
--- /dev/null
+++ b/tests/data/parser/0008-expected
@@ -0,0 +1 @@
+Cookie: foo=bar
diff --git a/tests/data/parser/0008-test b/tests/data/parser/0008-test
new file mode 100644
index 0000000..02baa00
--- /dev/null
+++ b/tests/data/parser/0008-test
@@ -0,0 +1 @@
+Set-Cookie: foo=bar; version=1000;
diff --git a/tests/data/parser/0009-expected b/tests/data/parser/0009-expected
new file mode 100644
index 0000000..b14d4f6
--- /dev/null
+++ b/tests/data/parser/0009-expected
@@ -0,0 +1 @@
+Cookie: foo=bar
diff --git a/tests/data/parser/0009-test b/tests/data/parser/0009-test
new file mode 100644
index 0000000..83c83fe
--- /dev/null
+++ b/tests/data/parser/0009-test
@@ -0,0 +1 @@
+Set-Cookie: foo=bar; customvalue=1000;
diff --git a/tests/data/parser/0010-expected b/tests/data/parser/0010-expected
new file mode 100644
index 0000000..e69de29
diff --git a/tests/data/parser/0010-test b/tests/data/parser/0010-test
new file mode 100644
index 0000000..1408056
--- /dev/null
+++ b/tests/data/parser/0010-test
@@ -0,0 +1 @@
+Set-Cookie: foo=bar; secure;
diff --git a/tests/data/parser/0011-expected b/tests/data/parser/0011-expected
new file mode 100644
index 0000000..b14d4f6
--- /dev/null
+++ b/tests/data/parser/0011-expected
@@ -0,0 +1 @@
+Cookie: foo=bar
diff --git a/tests/data/parser/0011-test b/tests/data/parser/0011-test
new file mode 100644
index 0000000..ad11eab
--- /dev/null
+++ b/tests/data/parser/0011-test
@@ -0,0 +1 @@
+Set-Cookie: foo=bar; customvalue="1000 or more";
diff --git a/tests/data/parser/0012-expected b/tests/data/parser/0012-expected
new file mode 100644
index 0000000..b14d4f6
--- /dev/null
+++ b/tests/data/parser/0012-expected
@@ -0,0 +1 @@
+Cookie: foo=bar
diff --git a/tests/data/parser/0012-test b/tests/data/parser/0012-test
new file mode 100644
index 0000000..cf9e712
--- /dev/null
+++ b/tests/data/parser/0012-test
@@ -0,0 +1 @@
+Set-Cookie: foo=bar; customvalue="no trailing semicolon"

---

From: Adam Barth <[email-masked]>
Date: Sat, 8 Aug 2009 14:51:28 -0700

On Sat, Aug 8, 2009 at 1:32 PM, Daniel Stenberg<[email-masked]> wrote:
> and a custom assigned attribute (with and without quotes and with and
> without trailing semicolon).

Thanks Daniel.  These tests are great.  I've added them to the repository.

One thing that might be good to test is the interaction between the
quote character and the cookie name and value.  For example:

Set-Cookie: foo="bar;baz"
Set-Cookie: "foo=bar"=baz

etc.

> some extra hint in a file for the specific test or something?

Not sure.  That could work, or we could put the tests in
http-state/test/data/path and have a list of URLs to query the test
server for.  We also have a similar problem with Domain cookies in
that they need to say which URL to redirect o.

> actually does make the curl results deviate from that of one or more

Sure, but at least it's a sanity check.  :)

>  curl -L -c /dev/null "http://localhost:8888/cookie-parser?0001" -x

I haven't investigated this, but you should feel free to send me a
patch for testserver.py.  This page seems to suggest that it's
possible with the HTTPServer library we're using:

http://www.okisoft.co.jp/esc/python/proxy/

Adam