Subject: tsp interop test vector (proposal)

From: tho <[email-masked]>
Date: Mon, 7 Jan 2002 14:12:48 +0100

hello everybody,
in trying to organize some interop testing between all the publicly available 
rfc3161 implementations I have begun sketching down a set of test vectors and
I'd like to discuss the matter with you ;-)

the first (TSRQ-G_i) is made up of correctly formatted TimeStampReqs which 
we expect to be fulfilled by the TSA. 
Starting from a very basic request, we make vary some parameters:

  o TSRQ-G_001 -> basic request (none of the optional fields)
  o TSRQ-G_002 -> TSRQ-G_001 + nonce 
  o TSRQ-G_003 -> TSRQ-G_001 + reqPolicy{=one supported by the actual TSA} 
  o TSRQ-G_004 -> TSRQ-G_001 + certReq=TRUE 
  ... other

  for all these TimeStampReqs we expect to get the signed receipt of the 
  TSTInfo back from the TSA (which must be then verified in the many ways 
  stated by the rfc)

the second (TSRQ-E_i) is a vector of malformed (in a number of ways) 
TimeStampReqs:

  o TSRQ-E_001 -> bad asn.1 encoding (expected badDataFormat)
  o TSRQ-E_002 -> policy id not recognized (expected unacceptedPolicy)
  o TSRQ-E_003 -> weak or unknown hash algorithm (expected badAlg)
  o TSRQ-E_004 -> wrong protocol version (expected badRequest or badDataFormat)
  o TSRQ-E_005 -> length mismatch in messageImprint (expected badDataFormat)
  o TSRQ-E_006 -> unrecognized extension (expected unacceptedExtension)
  ... other

the third (SBP-E_i) is `Socket Based Protocol' specific (really there should 
be one of these vectors for each of the transport protocols including HTTP, 
SMTP, etc. but for now we can start with `Socket Based Protocol' which seems 
to be the most widely deployed) and consists of the following:

  o SBP-E_001 -> packet with pollRep flag set
  o SBP-E_002 -> packet with pollReq flag set
  o SBP-E_003 -> packet with negPollRep flag set
  o SBP-E_004 -> packet with partialMsgRep flag set
  o SBP-E_005 -> packet with finalMsgRep flag set
  o SBP-E_006 -> packet with errorMsgRep flag set

  for all those above we expect a rejection with failInfo=badRequest 

  o SBP-E_007 -> packet with length discrepancy (or a similar trick to
                 test if there is a time-out mechanism on I/O)
  ... other

I have just tried to use this framework for testing against my own 
implementation, the one from SIA and Peter Sylvester's (no SBP-E_i for him 
since the exposed interface is HTTP) and I've found it quite useful as it 
offers an _unified_ approach and the possibility to automate the procedure.

Surely we can go more in-depth than this, and in fact this is just a starting
point.

tho
--

---

From: [email-masked] (Peter Gutmann)
Date: Tue, 8 Jan 2002 18:04:49 +1300 (NZDT)

>I'd like to discuss the matter with you ;-)

I don't know how well some of the negative-response testing will work.  For
example most of the policies I've seen are either unknown or just randomly-
invented OIDs, so my implementation ignores policies because otherwise it
couldn't interop with anything at all.  Based on the incredible number of bugs
in related stuff like certs (see for example the X.509 style guide) I'm also
extremely flexible with message contents, for example if the field looks more
or less right or doesn't affect the entire operation I just skip it and move
on.  In particular I ignore most of the stream-protocol flags based on the
confusion I saw in CMP interop testing when, no matter what flags were
(presumably erroneously) set, the other side usually just wanted the most basic
request-response operation.  Trying to fiddle with polling when the other side
has lost interest and gone home doesn't make your software look very functional
to the user.

(Aside: I have probably spent about an order of magnitude more time adding
 workarounds for bugs to my cert-handling code than in writing the original
 code itself.  Over time, I've also removed more and more checks on (non-
 security-related) items in order to get it to work with broken certs coming
 from other software.  There are still (security-related) checks in there which
 are rejecting certs which result in people sending me "bug" reports, but I'm
 not going to remove those... as a non-commercial vendor I can afford to lose a
 few users if they think it's a problem).

I would expect that other implementors are also writing code which tries to
DTRT in the face of broken requests and implementations.  This means that a lot
of the negative-response tests you're proposing would fail, because the other
side would look at the problem, decide it's irrelevant to the operation of the
protocol, and continue.  For example my policy OID is:

/* Dummy policy OID for the TSA ('snooze policy, "Anything that arrives, we
   sign") */

and I've never found an implementation which has any problems with this.

Peter.