Subject: Re: Termination for SENDDATA command

From: Doug Royer <[email-masked]>
Date: Wed, 12 May 1999 11:06:13 -0700 (PDT)

> From: [email-masked]
> Date: Wed, 12 May 1999 17:50:18 GMT
> 
> One of our folks passed this to me:
> 
> >1]  Terminate the data with CRLF "." CRLF, escaping "."'s at the
> >    beginning of a line a la SMTP.
> >[Snip]
> >2]  Provide a byte count when issuing the command,
> >    eg "SENDDATA 2345".
> >[Snip]
> >3]  Provide a termination string when issuing the command,
> >    eg "SENDDATA :=="
> >[Snip]
> >4]  Send data across another connection, a la FTP.
> 
> Don't forget to consider the performance characteristics of server's with
> size constraints. If a server is unwilling to receive data greater than
> size X, then options 1 and 3 require the server to receive all the data
> before it can complain. Options

The same is true for SMPT and USENET. Things happen. And it is still usable.
I opt for [1].

-Doug

---

From: John Stracke <[email-masked]>
Date: Thu, 13 May 1999 01:59:45 +0000

> The same is true for SMPT and USENET. Things happen. And it is still usable.

SMTP and NNTP are 17 and 13 years old, respectively.  We've learned a few
things since then.  Let's go for "good" rather than "usable".

--
/=============================================================\
|John Stracke    | My opinions are my own | S/MIME & HTML OK  |
[email-masked]|============================================|
|Chief Scientist | NT's lack of reliability is only surpassed |
|eCal Corp.      |  by its lack of scalability. -- John Kirch |
\=============================================================/

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 13 May 1999 10:27:25 +0200

>things since then.  Let's go for "good" rather than "usable".

And you belive "good" is? Sorry if I have missed your opinion on this 
matter, but please repeat.

   paf

---

From: John Stracke <[email-masked]>
Date: Thu, 13 May 1999 15:32:44 +0000

Patrik Fältström wrote:

> And you belive "good" is?

Right, good point.  :-) I believe that byte-count-based systems are more efficient
than terminators.  With a terminator, the receiver has to parse the data as it
arrives, even if it already knows it can't honor the request (say, if the data is
1M long and the receiver has a 100K limit), or if it intends to forward the
request without examining it (if the receiver is a proxy--I don't think we have
proxies in CAP yet, but a good design should make them feasible).  You also have
to worry about encoding the terminator (if the terminator is fixed) or
guaranteeing uniqueness (if it isn't).

The objection I heard (from Steve Mansour) to byte counts seems to be that they
require the sender to marshal the data before sending it.  However, I believe that
this will turn out not to be a real problem.  I can think of three broad
implementation strategies:

  1. Marshal the entire protocol message into a single block of memory before
     sending it.  I think this is what Steve was thinking of.  This obviously
     makes it easy to calculate the length, but it costs buffer space and incurs
     performance costs via extra memory-to-memory copies.
  2. Pluck each piece of data out of the database on the fly, with a separate
     query for each component of the data structure.  This does indeed offer
     savings in buffer space, but it wrecks database performance.  I don't expect
     anybody is going to adopt this strategy.
  3. Load the data from the database into a set of objects, which are components
     of the protocol message, and have those objects send themselves in sequence.
     This is probably the most reasonable strategy--it reduces memory-to-memory
     copies, saves on buffer space, and may (with some good buffer management)
     even allow sharing of buffer space between separate protocol connections.  It
     does seem to lead to trouble generating byte counts, but I think there's a
     solution: each object can have a size() method, which calculates the number
     of bytes it will take to send.  (This may be recursive, if the object
     structure is tree; a compound object would add up its components' sizes and
     its own overhead.) Note that this strategy would have trouble with a
     terminator-based protocol, because either the objects have to escape the
     terminator on the fly (if the terminator is fixed), or they have to be
     searched individually (if it isn't).

Does anybody have another method that they prefer that would have trouble with
byte counts?

> Sorry if I have missed your opinion on this
> matter,

I think I expressed it only on the off-list thread that sprang up about this among
the CAP design team a couple of months ago.

I apologize for being snippy about the "good' vs. "usable" thing; it's just that
I've seen a strong push in this group in the past couple of months to get
something written down by July, and ignore the long-term problems we'll have if we
get locked into a clumsy protocol.

---

From: John C Klensin <[email-masked]>
Date: Thu, 13 May 1999 12:05:03 -0400

--On Thursday, 13 May, 1999, 15:32 +0000 John Stracke
<[email-masked]> wrote:

> Right, good point.  :-) I believe that byte-count-based
> systems are more efficient than terminators.  With a
> terminator, the receiver has to parse the data as it arrives,
> even if it already knows it can't honor the request (say, if
> the data is 1M long and the receiver has a 100K limit), or if
> it intends to forward the request without examining it (if the
> receiver is a proxy--I don't think we have proxies in CAP yet,
>...

Without recommending one strategy over another --the tradeoffs
are complex and, while my design instincts cause me to favor
byte counts, I've seen far fewer practical interoperability
problems with terminators-- it is probably worth noting that...

if the strongest argument in favor of byte counts is, as you
have expressed it, warning the receiver about how much stuff is
coming so it can decline, then there is a "have it both ways"
option that should be considered, i.e.,

   * announce an estimated byte count with the command, noting
that senders are required to estimate either exactly or high and
that a receiver may decline to give a "go ahead" if it doesn't
like the estimate.

   * send with a delimiter.

That eliminates substantially all of the problems potentially
associated with making exact counts, while having probable size
information available to the receiver before data is sent.   And
it saves a system that already has things in more or less the
transmission format, but with different, e.g., end of line
delimiters or character sets, from having to make a second copy
in order to count rather than doing the conversions to "on the
wire" format on the fly.

      john

---

From: John Stracke <[email-masked]>
Date: Thu, 13 May 1999 16:08:10 +0000

> that a receiver may decline to give a "go ahead" if it doesn't

But this doesn't save the receiver from having to parse for the
delimiter.

---

From: [email-masked] (Steve Mansour)
Date: Thu, 13 May 1999 09:17:58 -0700

The current proposal in CAP is to terminate SENDDATA with <CRLF>.<CRLF> and quote
lines that begin with '.' .  The only parsing involved is:

   * look for <CRLF>.<CRLF> and when you find it you're done.
   * if you find a line that begins with '.' and the next characters are not <CRLF>,
     then remove the initial '.' .

It seems a whole lot easier than what is described below. Seems like it would scale
well too.

Also, note that the keywords and line folding in iCal are such that '.' is never the
first character of the line. The only time it could happen is when we have other mime
parts (attachments, etc.).

Are there problems with this approach?

-Steve

---

From: John C Klensin <[email-masked]>
Date: Thu, 13 May 1999 13:01:33 -0400

--On Thursday, 13 May, 1999, 16:08 +0000 John Stracke
<[email-masked]> wrote:

> > have expressed it, warning the receiver about how much stuff
> > ...

Which part of "if the strongest argument in favor..." did you
have trouble parsing?  

If "save the receiver from having to parse..." is considered the
key argument, then, of course, approximate sizes aren't useful.

As long as processing speeds are a lot greater than likely
line/transmission speeds, as long as what is being transmitted
is basically textual material that has to be parsed anyway (and
especially, as others have pointed out, calendar transmissions
are unliikely to contain leading periods), you have trouble
convincing many of us about the importance of this bit of
overhead.  But, if it is important, then byte (or bit or
character) counts are clearly the only solution.

    john

---

From: John Stracke <[email-masked]>
Date: Fri, 14 May 1999 18:39:46 +0000

> Which part of "if the strongest argument in favor..." did you

Ack.  You're right, of course.  (Nice phrasing, by the way.  :-) I read
your message too quickly.

> line/transmission speeds,

This is not always the case--if it were, then nobody would ever have an
overloaded Web server.

> as long as what is being transmitted
> is basically textual material that has to be parsed anyway

This is the main point I disagree with.  In the current protocol, with
no proxies, it is true; but, if we want to be able to have proxies
eventually, we should watch out for design choices that will make them
less efficient.  A proxy does not need to parse the message it's
relaying.

---

From: John Stracke <[email-masked]>
Date: Fri, 14 May 1999 18:35:43 +0000

Steve Mansour wrote:

> It seems a whole lot easier than what is described below.

I don't understand what you're considering to be difficult.  I described how a sender
would implement byte counts under three different approaches for implementing the
sender.  The only implementation approach that made byte counts difficult was one that
nobody would use (I included it only for completenes).  Do you have a fourth
implementation approach in mind?

> Seems like it would scale
> well too.

But there are cases where it will not scale as well as byte counts--for example, in
proxies.  A proxy should be able to buffer blindly, without having to parse its input at
all.

> parts (attachments, etc.).

Which are an extremely important case.  eCal may be unable to adopt CAP at all unless we
can have attachments (since our customers are Web sites, who want their ads delivered to
the users).

---

From: [email-masked] (Steve Mansour)
Date: Fri, 14 May 1999 14:13:18 -0700

All,

My comments are below.

We need to nail this point and move on. I'm interested in the opinion of others.  Shall we
do byte counts or not?

-Steve

> implementation approach in mind?

None of the options you described are difficult. I pointed out that it is easier to stream
the data out without counting bytes than it is to count bytes in any of the ways you
described.

In order to support attachments and ALTREP variants of text, code will have to be written to
count bytes. Attachments could be large. Altreps could be large. As you point out, it can be
done.  But there is overhead.

The overhead in parsing the '.' seems small.

> > Seems like it would scale well too.

This is true. The parsing seems trivial, but the point is taken.

> can have attachments (since our customers are Web sites, who want their ads delivered to

This is a non-issue. You can easily quote the '.' so attachments are not a problem.

---

From: Dave Crocker <[email-masked]>
Date: Tue, 18 May 1999 06:46:42 -0700

>that a receiver may decline to give a "go ahead" if it doesn't

Or, permit the receiver to pre-announce it's limit.  Then the client can 
terminate when it reaches it.

d/

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Dave Crocker                                         Tel: +1 408 246 8253
Brandenburg Consulting                               Fax: +1 408 273 6464
675 Spruce Drive                             <http://www.brandenburg.com>
Sunnyvale, CA 94086 USA                 <mailto:[email-masked]>