Subject: RE: CRLF on POST requests, where/how specified (repost from prior bad subject line)

From: "Fred Bohle" <[email-masked]>
Date: Tue, 20 Jun 2000 12:51:11 -0500

Fred Bohle@NEON
06/20/2000 12:51 PM

Scott,

     No, not the CRLF between the headers and the body.  We are asking
about a
CRLF that follows the body.  It does not seem to be in the 1.0 spec, and
the 1.1 spec
seems to specifically prohibit it.  And yet, IE5 and Netscape both will
send a CRLF
after the body for Content-type: application/x-www-url-encoded.

     What do other web servers do to handle this CRLF?  We find that if we
decide to
close the connection with the end of the response we generate, (Connection:
close)
and THEN the CRLF arrives from the client, the TCP layer will generate a
Reset packet.
This causes (IE5 at least) the client to fail processing the response we
just sent.  So the
application stops, dead in the water.

Fred Bohle

"Scott Lawrence" <[email-masked]> on 06/20/2000 12:36:45 PM

To:   Brad Taylor/Neon, [email-masked]
cc:    (bcc: Fred Bohle/Dev/Neon)

Subject:  RE: CRLF on POST requests, where/how specified (repost from prior
      bad subject line)

See RFC 2616 ( http://www.innosoft.com/rfc/rfc2616.html#sec-4.1 )

Quoting:

----
Request (section 5) and Response (section 6) messages use the
generic message format of RFC 822 [9] for transferring entities (the
payload of the message). Both types of message consist of a
start-line, zero or more header fields (also known as "headers"), an
empty line (i.e., a line with nothing preceding the CRLF) indicating
the end of the header fields, and possibly a message-body.

        generic-message = start-line
                          *(message-header CRLF)
                          CRLF
                          [ message-body ]
----
The CRLF that separates the headers from the body is not part of
either.  Since it is not part of the body, it is not included in the
content length.

--
Scott Lawrence      Director of R & D        <[email-masked]>
Agranat Systems   Embedded Web Technology   http://www.agranat.com/

---

From: "Life is hard, and then you die" <[email-masked]>
Date: Tue, 20 Jun 2000 11:21:07 -0700

On Tue, Jun 20, 2000 at 12:51:11PM -0500, Fred Bohle wrote:
>      about a CRLF that follows the body.  It does not seem to be in
>      the 1.0 spec, and the 1.1 spec seems to specifically prohibit it.
>      And yet, IE5 and Netscape both will send a CRLF after the body
>      for Content-type: application/x-www-url-encoded.

The spec clearly disallows this extra CRLF, i.e. it's a bug in those
browsers. However, there is an easy workaround which many people use:
when reading the request line, ignore all whitespace (or empty lines)
until you hit the actual request line.

>      What do other web servers do to handle this CRLF?  We find that
>      if we decide to close the connection with the end of the response
>      we generate, (Connection: close) and THEN the CRLF arrives from
>      the client, the TCP layer will generate a Reset packet.  This
>      causes (IE5 at least) the client to fail processing the response
>      we just sent.  So the application stops, dead in the water.

This is a little tricky, but not much. Basically you need to do a
shutdown on the socket, i.e. only close for sends; then you keep
reading from the socket (and just discard the data) until either you get
an EOF (in which case the client did the close), or until some timeout
(a few seconds) in case the client doesn't close the connection. Note
that this problem is a general problem (i.e. not limited to the extra
CRLF), because you can get the same effect when a client pipelines
requests.

  Cheers,

  Ronald

P.S. take a look at the Apache source: read_request_line() in http_protocol.c
     for the first problem, and lingering_close() in http_main.c for
     the second - the comments are enlightening.