Subject: Experience with Online Protocol Testing

From: Hannes Tschofenig <[email-masked]>
Date: Fri, 28 Jun 2013 19:54:06 +0300

[I posted this question a little while ago to the WG chairs mailing list and got no response.
Maybe my question is too trivial but I thought I should try it on the [email-masked] list as well to 
get some feedback.]

Hi all,

when concerns about the lack of interoperability surfaced mid last year 
in the OAuth working group we (Derek, and myself) tried to figure out 
whether we should schedule a face-to-face interop and/or to develop an 
online test suite. We got in touch with Lucy Lynch (ISOC) and she helped 
us to find developers to work with us on the test software.

Roland Hedberg, one of the guys working on the project for OAuth 
testing, presented his ongoing work in the OAuth working group, see
http://www.ietf.org/proceedings/86/slides/slides-86-oauth-2.pdf

OAuth is a bit more complex since it involves more than two parties and 
we were looking for a test framework that could be re-used to develop 
the desired results more quickly. To our surprise we couldn't find 
a test framework that we could easily re-use since most test frameworks 
really focus on different types of tests. Of course, we might 
have looked in the wrong direction.

Here is how it works at the moment:
* Imagine you have developed an OAuth-based identity management server 
(that contains an OAuth 2.0 authorization server) and it runs somewhere 
on the Internet (or in your lab). You don't need to have access to the 
source code to execute the tests.
* You download the scripts that Roland & Co had developed and configure 
them. Of course you will have to create an account at your IdP as well.
* You run the test scripts against the authorization server and the 
script plays the other OAuth 2.0 parties in the exchange. The script contains a number 
of test cases (around 60+ at the moment) and determines whether the IdP 
responds correctly in the exchanges.

I know that these ideas have come up in other working groups in the past 
already (such as in SCIM, which also has a test server up and 
running).

It would be interesting to hear what others have been doing and what 
worked for you or what didn't.

Ciao
Hannes

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 28 Jun 2013 19:05:50 +0200

On Jun 28, 2013, at 6:54 PM, Hannes Tschofenig <[email-masked]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
> 
So it sounds like you are doing some sort of conformance testing...

For SCTP we did a number of interoperability tests, which were
face to face meetings and the people who were developing stacks
we there. This events were always very helpful not only for improving
the stacks but also for improving the IETF documents.
I also developed a test tool for conformance testing based on some
test descriptions provided by ETSI. However, it would have made sense to
specify also some tests within the IETF. That can also help to clarify
some protocol aspects and to focus on "common mistakes".
Providing tests is a very good thing in my experience. Unfortunately,
at the point we did this for SCTP, the IETF position was that testing
isn't an objective. Maybe it is time to change that...

Best regards
Michael
> 
> Ciao
> Hannes
> 
> 
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.19 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> 
> iQEcBAEBCgAGBQJRzb+vAAoJEGhJURNOOiAtZBwIAISKHjD7gv8irkL4yaBR31K8
> KLZCr/1n0n1OcXl3rE9MFOyA85hYNplZFd1giJLLgEX3UyofYXg/L2QOOLqtP0lT
> JgnW2CvR0WWKfIT1iKjAwAodCVLsHF8MdPE4tl0LBlCeqhA1waj/oCLkBzZrrhhq
> oWnZzP0I9nFdlSxV9EAHQ62RAxLUVQmBEqgMxl7A+iC9fGD8IhWSNSqqsaF0WOaB
> 6bHdwCFLYYAyqKhiuJAo/f6YFGEzIbPgpHPGjwBZzBIjwP/EGiFnAliyF8WATHzF
> RM+OWg6QASh1cNwzc0dbMcrcr1L1ve7amATMc4uPN7sRjhv0s62iguWfGRhQhHw=
> =YT5M
> -----END PGP SIGNATURE-----
>

---

From: Hannes Tschofenig <[email-masked]>
Date: Fri, 28 Jun 2013 20:14:41 +0300

> isn't an objective. Maybe it is time to change that...

I guess the code you wrote for your online test cases was your own work and you didn't re-use some else's test framework? 
The SCTP case is also a bit simpler than the OAuth scenario since in SCTP might have only tested client-server interactions (right?)

Ciao
Hannes

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 28 Jun 2013 19:58:37 +0200

> I guess the code you wrote for your online test cases was your own work and you didn't re-use some else's test framework? 
Not a test framework, but a scheme interpreter (guile) was extended with the
necessary stuff for sending/receiving packets, such that tests can be implemented
as simple scheme functions. Doing a test is calling a scheme function which
returns whether the corresponding test was passed or failed. Some shell scripts
allows you running a test suite.
> The SCTP case is also a bit simpler than the OAuth scenario since in SCTP might have only tested client-server interactions (right?)
Well, the testtool only deals with network layer events. Taking interactions
with the application (API) into account is a different story. We also have
an API tester, which is also self written. We currently don't have something
which takes network and API events into account.

Best regards
Michael
> iQEcBAEBCgAGBQJRzcSCAAoJEGhJURNOOiAt064H/3iowM2lBBunk58jOq1/eAfK
> Cy+y+dpEqJoXeqMI7RLtTsXXDbblxI83cxO6OXol7HU37elmomdiD7qGg2KPeeOJ
> 1GyEWnXXJ1oaC9AcWAkPiRan2EhdXGxu6yWG1iuHniOSkC3WvU9nhVU8PLUkSPwB
> WSzh4hsj6tnTpR67oYoHkDwtQsFuvcvxiYkNV9rI3fy+FIXJ5ygf9UPmgVJpNqIx
> azmc8W9iWUUeGGnaU+/2APRybe8PpCZf6y+QKVFA/6Zkai86/HfTaDFdAnL/eOXV
> xhdXjHsn8Ci4CkCXJ7Pn2JRpIYX1y7ccMOvu8WHuvJnSQ2S091/EFHBY2w/Wydk=
> =vU1M