Subject: HTTP Testing Resources

From: Mark Nottingham <[email-masked]>
Date: Tue, 9 Apr 2019 09:39:25 +1000

Hi everyone,

We've started collecting resources that might be helpful to test HTTP implementations here:
  https://github.com/httpwg/wiki/wiki/HTTP-Testing-Resources

Thoughts / additions?

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 9 Apr 2019 10:13:15 +1000

Hi Sawood,

That sounds interesting; have you published them yet? If so, I don't see any harm in adding them.

For what it's worth, the approach you outline reminds me of how cache-tests.fyi works; it defines test cases in JavaScript data structures, and you can run them using a NodeJS CLI or in a browser. 

Also, part of the problem of server testing is that many behaviours are dependent upon a particular resource, because of how it interfaces with the server (filesystem vs. CGI, for example) and also because of resource-specific handling. REDbot tries to address that by running checks against individual URLs, but I think there's also space for a tool that can be used to check how "back-ends" such as filesystems, CGI, FastCGI, as well as Web frameworks (e.g., Drupal, Wordpress) handle various aspects of the protocol.

Cheers,

> On 9 Apr 2019, at 10:03 am, Sawood Alam <[email-masked]> wrote:
> 
> Hi,
> 
> Last year I wrote a web server tester library along with a handful of custom test suites to automate testing/grading of students' assignments of the Web Server Design course. While I think it is relevant here, I would wait for some feedback before I decide to add it to the wiki. Here is how it works:
> 
> First, write a template file (say, "malformed-header.http") for the request data (which can be a single HTTP message or multiple messages put together for pipeline requests) as following:
> 
> GET /foo HTTP/1.1
> Host: <HOSTPORT>
> Header with missing colon
> 
> Then write a test method in a test suite class, inherited from the base tester class, as following:
> 
> @make_request("malformed-header.http")
> def test_bad_request_header(self, report):
> """Test whether the server recognizes malformed headers"""
>     self.check_status_is(report, 400)
> 
> From there, the test can be run against any HTTP server using the included CLI or Web UI.
> 
> I am planning to improve it further when I will offer this course in the Fall 2019.
> 
> Best,
> 
> --
> Sawood Alam
> Department of Computer Science
> Old Dominion University
> Norfolk VA 23529
> 
> 
> 
> On Mon, Apr 8, 2019 at 7:43 PM Mark Nottingham <[email-masked]> wrote:
> Hi everyone,
>

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 9 Apr 2019 15:25:57 +1000

Hi Willy,

Thanks. I think those are slightly out of scope for this; the intent is to focus on tests, rather than debugging tools. 

Cheers,

> On 9 Apr 2019, at 12:34 pm, Willy Tarreau <[email-masked]> wrote:
> 
> Hi Mark,
> 
> On Tue, Apr 09, 2019 at 09:39:25AM +1000, Mark Nottingham wrote:
>> Hi everyone,
>> 
> Thanks for creating this. I'd add these ones :
> 
>  h2c: https://github.com/fstab/h2c
>  h2i: https://github.com/bradfitz/http2/tree/master/h2i
> 
> I'm using the first one a lot. The second one less since by the time I
> figure the syntax of certain commands my connections tend to time out,
> but it's very convenient to test for edge cases.
> 
> Cheers,
> Willy

---

From: Sawood Alam <[email-masked]>
Date: Mon, 8 Apr 2019 20:03:53 -0400

Hi,

Last year I wrote a web server tester library along with a handful of
custom test suites to automate testing/grading of students' assignments of
the Web Server Design course. While I think it is relevant here, I would
wait for some feedback before I decide to add it to the wiki. Here is how
it works:

First, write a template file (say, "malformed-header.http") for the request
data (which can be a single HTTP message or multiple messages put together
for pipeline requests) as following:

GET /foo HTTP/1.1
Host: <HOSTPORT>

Then write a test method in a test suite class, inherited from the base
tester class, as following:

>From there, the test can be run against any HTTP server using the included
CLI or Web UI.

I am planning to improve it further when I will offer this course in the
Fall 2019.

Best,

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 9 Apr 2019 07:59:09 +0200

On Tue, Apr 09, 2019 at 03:25:57PM +1000, Mark Nottingham wrote:
> Hi Willy,
> 
> Thanks. I think those are slightly out of scope for this; the intent is to
> focus on tests, rather than debugging tools. 

The border between the two is often quite thin, especially when such
debugging tools are involved in test scripts, but OK.

Cheers,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 9 Apr 2019 04:34:59 +0200

Hi Mark,

Cheers,
Willy

---

From: Sawood Alam <[email-masked]>
Date: Mon, 8 Apr 2019 20:50:28 -0400

Yes, the code was open-sourced from the day one under MIT license at
https://github.com/ibnesayeed/webserver-tester (sorry, I forgot to provide
the link in the previous email).

I agree that it is almost impossible to create one tool that can test every
aspect of a web server. In the case of our class test suites, we provided
static files and CGI scripts with predefined directory structure for
students to serve with their web server implementations.

Before writing this library we looked for many other ways people have
implemented HTTP testing, but we did not find anything quite what we were
looking for, i.e., something that takes a raw HTTP message as input (which
was essential for the course). Many implementations require some sort of
representation or a data structure or provide helper methods to build
request piece by piece.

We have an instance of our custom tester instance tailored for the course
still deployed at https://cs531.cs.odu.edu/ where one can enter a host
(e.g., "odu.edu") in the "test a Web Server" block and run provided test
suits against it. Alternatively, first enter "msiddique" (which is the ID
of one of the students) in the first text input field under the "Deploy a
Web Server" block then deploy it. This will fetch the code of that
student's implementation from the private repo at GitHub, build a Docker
image, deploy it, and made that instance accessible at "cs531-msiddique"
host name from our tester instance (which itself is running in a
container). The block where tests are performed will automatically be
populated with this host name. Running those tests will show what aspects
were tested, what test cases passed, where things failed, and any request
and response data will be rendered for manual inspection. Similar
information is available from the CLI as well.

Best,

On Mon, Apr 8, 2019 at 8:13 PM Mark Nottingham <[email-masked]> wrote:

> For what it's worth, the approach you outline reminds me of how
> cache-tests.fyi works; it defines test cases in JavaScript data structures,
> and you can run them using a NodeJS CLI or in a browser.
>
> Also, part of the problem of server testing is that many behaviours are
> dependent upon a particular resource, because of how it interfaces with the
> server (filesystem vs. CGI, for example) and also because of
> resource-specific handling. REDbot tries to address that by running checks
> against individual URLs, but I think there's also space for a tool that can
> be used to check how "back-ends" such as filesystems, CGI, FastCGI, as well
> as Web frameworks (e.g., Drupal, Wordpress) handle various aspects of the
> protocol.
>
> Cheers,
>
>
> > First, write a template file (say, "malformed-header.http") for the
> request data (which can be a single HTTP message or multiple messages put
> together for pipeline requests) as following:
> >
> > GET /foo HTTP/1.1
> > Host: <HOSTPORT>

---

From: Alcides Viamontes E <[email-masked]>
Date: Tue, 9 Apr 2019 15:16:01 +0200

Not sure how relevant is this, but we have found nghttp (the cli utility
that comes with ngttp2) together with the `-v` option invaluable as part of
our functional test suite; mainly to verify that the server does HTTP/2
priorities and flow control correctly.

On Tue, Apr 9, 2019 at 3:08 PM Sawood Alam <[email-masked]> wrote:

> https://github.com/ibnesayeed/webserver-tester
> <https://github..com/ibnesayeed/webserver-tester> (sorry, I forgot to
> provide the link in the previous email).
>
> I agree that it is almost impossible to create one tool that can test
> every aspect of a web server. In the case of our class test suites, we
> provided static files and CGI scripts with predefined directory structure
> for students to serve with their web server implementations.
>

-- 
Alcides Viamontes E.
ShimmerCat AB, CTO
(+46) 722294542

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 9 Apr 2019 10:29:25 -0600

>   https://github.com/httpwg/wiki/wiki/HTTP-Testing-Resources

Besides nghttp mentioned earlier, I can suggest two open source projects
I am involved with:

* Web Polygraph (HTTP proxy/server performance testing suite)
  http://www.web-polygraph.org/

* Daft (HTTP functionality testing toolbox)
  https://github.com/measurement-factory/daft

Alex.