Subject: second message
From: [email-masked] (David A. Borman)
To: [email-masked]
Date: Mon, 22 Jun 92 11:13:29 -0500

And here's a follow-up message that also got bounced by
our mail gateway...
			-David Borman, [email-masked]

Date: Mon, 22 Jun 92 11:09:41 -0500
From: [email-masked] (David A. Borman)
To: [email-masked]
Subject: Message from Van

Van's attached message got bounced when going to over half
of the tcplw mailing list.  Seems a patch that was installed
to sendmail on our mail gateway broke MX record processing...
They claim it is now fixed, so I am re-sending Van's message for
those of you who didn't get it.  Sorry to those of you who
get it twice...
		-David Borman, [email-masked]

From: [email-masked] (Jim Hughes)
Subject: Re: Checksumming on TCP
To: [email-masked] (Van Jacobson)
Date: Mon, 22 Jun 92 9:06:59 EDT
Cc: [email-masked], [email-masked]
X-Mailer: ELM [version 2.3 PL11]

> 
> > Yes, 4.4 and the long fat pipe mods are available.
> 
> Tom Skibo's long fat pipe mods were made to the BSD Net-2
> release, not 4.4.  The architecture of Net-2 is fairly close to
> that of 4.2/3 BSD and doesn't include the changes that Craig
> mentioned.  The code that I hope will be in the 4.4 release is a
> ground-up re-architecting of the BSD networking kernel & is very
> different from Net-2.  E.g., mbufs are gone, netipl is gone,
> ip_output is gone, the arp table is gone, all the xxx_usrreq are
> gone, xxx_usrsend (the protocol specific replacement for sosend
> that Craig mentioned) and xxx_usrrecv (the equivalent
> replacement for soreceive) have been added, ...   The 4.4 code
> isn't available yet.  I had hoped to have it on the 4.4alpha
> tape coming out in July but my personal life went through major
> trauma starting last December & I've been catatonic most of the
> last 6 months.  Although I'm not really functioning technically
> yet, most of the remaining work is just cleanup & integration
> (e.g., the OSI stack won't compile with the new architecture
> yet) and I'm pretty sure an alpha of the networking code will be
> available by the end of the summer.

Van, I apologize for my incorrect statement. The confusion is based
on the definition of 4.4. There is a tape called 4.4free and Thomas
did code to that. I don't have any knowledge of what level of networking
is in that release.

>  - Van

jim

From: [email-masked] (Jim Hughes)
Subject: Re: Checksumming on TCP
To: [email-masked] (Van Jacobson)
Date: Mon, 22 Jun 92 9:06:59 EDT
Cc: [email-masked], [email-masked]

> 
> > Yes, 4.4 and the long fat pipe mods are available.
> 
> Tom Skibo's long fat pipe mods were made to the BSD Net-2
> release, not 4.4.  The architecture of Net-2 is fairly close to
> that of 4.2/3 BSD and doesn't include the changes that Craig
> mentioned.  The code that I hope will be in the 4.4 release is a
> ground-up re-architecting of the BSD networking kernel & is very
> different from Net-2.  E.g., mbufs are gone, netipl is gone,
> ip_output is gone, the arp table is gone, all the xxx_usrreq are
> gone, xxx_usrsend (the protocol specific replacement for sosend
> that Craig mentioned) and xxx_usrrecv (the equivalent
> replacement for soreceive) have been added, ...   The 4.4 code
> isn't available yet.  I had hoped to have it on the 4.4alpha
> tape coming out in July but my personal life went through major
> trauma starting last December & I've been catatonic most of the
> last 6 months.  Although I'm not really functioning technically
> yet, most of the remaining work is just cleanup & integration
> (e.g., the OSI stack won't compile with the new architecture
> yet) and I'm pretty sure an alpha of the networking code will be
> available by the end of the summer.

Van, I apologize for my incorrect statement. The confusion is based
on the definition of 4.4. There is a tape called 4.4free and Thomas
did code to that. I don't have any knowledge of what level of networking
is in that release.

>  - Van

jim

To: [email-masked] (Jim Hughes)
Cc: [email-masked] (Steve Wise), [email-masked]
Subject: Re: Checksumming on TCP
Date: Mon, 22 Jun 92 04:01:35 PDT
From: Van Jacobson <[email-masked]>

> Yes, 4.4 and the long fat pipe mods are available.

Tom Skibo's long fat pipe mods were made to the BSD Net-2
release, not 4.4.  The architecture of Net-2 is fairly close to
that of 4.2/3 BSD and doesn't include the changes that Craig
mentioned.  The code that I hope will be in the 4.4 release is a
ground-up re-architecting of the BSD networking kernel & is very
different from Net-2.  E.g., mbufs are gone, netipl is gone,
ip_output is gone, the arp table is gone, all the xxx_usrreq are
gone, xxx_usrsend (the protocol specific replacement for sosend
that Craig mentioned) and xxx_usrrecv (the equivalent
replacement for soreceive) have been added, ...   The 4.4 code
isn't available yet.  I had hoped to have it on the 4.4alpha
tape coming out in July but my personal life went through major
trauma starting last December & I've been catatonic most of the
last 6 months.  Although I'm not really functioning technically
yet, most of the remaining work is just cleanup & integration
(e.g., the OSI stack won't compile with the new architecture
yet) and I'm pretty sure an alpha of the networking code will be
available by the end of the summer.

 - Van

To: [email-masked] (Jim Hughes)
Cc: [email-masked] (Steve Wise), [email-masked]
Subject: Re: Checksumming on TCP
Date: Mon, 22 Jun 92 04:01:35 PDT
From: Van Jacobson <[email-masked]>

> Yes, 4.4 and the long fat pipe mods are available.

Tom Skibo's long fat pipe mods were made to the BSD Net-2
release, not 4.4.  The architecture of Net-2 is fairly close to
that of 4.2/3 BSD and doesn't include the changes that Craig
mentioned.  The code that I hope will be in the 4.4 release is a
ground-up re-architecting of the BSD networking kernel & is very
different from Net-2.  E.g., mbufs are gone, netipl is gone,
ip_output is gone, the arp table is gone, all the xxx_usrreq are
gone, xxx_usrsend (the protocol specific replacement for sosend
that Craig mentioned) and xxx_usrrecv (the equivalent
replacement for soreceive) have been added, ...   The 4.4 code
isn't available yet.  I had hoped to have it on the 4.4alpha
tape coming out in July but my personal life went through major
trauma starting last December & I've been catatonic most of the
last 6 months.  Although I'm not really functioning technically
yet, most of the remaining work is just cleanup & integration
(e.g., the OSI stack won't compile with the new architecture
yet) and I'm pretty sure an alpha of the networking code will be
available by the end of the summer.

 - Van

Date: Fri, 19 Jun 92 13:09:40 -0600
From: [email-masked] (Vernon Schryver)
To: [email-masked], [email-masked]
Subject: TCP checksums, bcopys, etc

I composed and flushed responses to Craig Partridge and others along
these lines 3 times yesterday.  I can't help myself today.

-Both input and output data is often (but not always) not copied
    when sending or receiving TCP buffers in Silicon Graphics' UNIX
    system.  At speed, the data sees each system bus between the fiber
    and user-space one (1) time.  It does see the local, on-board
    or on-chip buses inside the FDDI hardware more often.

    In other words, there are no byte copies between user space and
    kernel space when things are running at speed.

-It is possible to finesse the checksum either
	1. during the user-space/system-buffer copies
	2. in the driver and below
    Since I choose to make byte copies run fast by not doing them,
    you might guess that I prefer (2).

-The calculation of checksum for TCP/IP/FDDI does not consume most of the
    cycles in systems I know about, despite the fact that the copies
    consume zero (0) CPU cycles.  However, I will admit we do a little
    more than 17-22 Mb/s.  (reported in the ATM mailing list).
    SHEEEESH!  That's only twice ethernet.  You can do better than that
    with a 16MHz CPU and a stupid DMA engine even the designer won't
    acknowledge, and while computing the checksum and using the nearly
    the old number of bcopy()'s on the host.

-Any system now in a lab that does not consume substantially more than
    50% of the ~98Mb/s FDDI speed of light with a single TCP virtual
    circuit should be cancelled to avoid embarrassment.  After all, we
    demonstrated 50Mb/s at InterOP91 with FDDI hardware we are not proud
    of, while simultaneously doing CFD simulations.

-Neither the checksum nor the bcopy() should matter in modern systems.
    Modern CPU's can compute the TCP checksum in <= 1 clock/byte, and can
    copy data in <0.5 clocks/byte.  What makes bcopy() or the
    checksumming slow are the cache misses and pollution.  Depending on
    the upper layer protocol, you may have to pay the cache regardless,
    making the cost of byte copies and checksumming 5-20% of the
    inevitable cache cost.

-The TCP checksum has some handy properties.  The checksum of
    the concatenation of two buffers (containing 16-bit words) is a sum
    of their checksums.  The checksum of part of a buffer is a difference
    of the checksum of the entire buffer and the checksum of the other
    part.

    These two properties computing the checksum in drivers or in
    user-space copies much easier.  You need not worry about computing
    the checksum of the psuedo-header in copyin().

-Vector machines may find the calculation of the checksum more onerous.
-I do not know about Gb/s TCP, such as Cray has reported.

Vernon Schryver,  [email-masked]

Date: Fri, 19 Jun 92 13:29:08 CDT
From: [email-masked] (Randy Butler)
To: [email-masked], [email-masked]
Subject: Re: Checksumming on TCP
Cc: [email-masked]

>Yes, 4.4 and the long fat pipe mods are available.  The mods can be
>gotten from Thomas Skibo @UIUC.edu

Thomas has moved on now but the mods can still be gotten from the
anonymous ftp server "uxc.cso.uiuc.edu" and found in the pub directory.

Randy Butler
National Center for Supercomputing Applications

From: [email-masked] (Jim Hughes)
Subject: Re: Checksumming on TCP
To: [email-masked] (Steve Wise)
Date: Fri, 19 Jun 92 14:16:04 EDT
Cc: [email-masked], [email-masked]

> 
> 
> | What has been done in 4.4 and in the modifications to 4.3 that I'm
> | aware of to support rolling checksum and copy loops is to add protocol
> | specific versions of sosend() to the protosw structure and have sendit()
> | call these routines (if they exist) instead of sosend().
> 
> 	Is the BSD 4.4 code available?  

Yes, 4.4 and the long fat pipe mods are available.  The mods can be
gotten from Thomas Skibo @UIUC.edu
 
> | 
> | far simpler.  Also, by being in protocol-specific code when copying data
> | from application space, one can make more intelligent memory placement
> | decisions (e.g. Jacobson's idea to copy direct from application space
> | to interface memory requires that one know which interface one will send
> | out -- which is a protocol-specific decision).  Finally, one can easily
> 
> 	About Van Jacobson's stuff.  Does the TCP segment stay in 
> 	interface memory until the ACK is received?  This memory's on the
> 	adapter, right?
> 
> | roll together checksum and copy because one knows how big a segment one
> | is going to send and which checksum algorithm to use.
> | 
> | Hope this is useful.
>  
> 	It is...thanx
> Stevo.
> 

This is exactly what is being worked on in the Gigabit Nectar testbed.

Outboard memory for ACK storage, and also for data that the Appl has
not issued a read for, so that the data can be copied directly to the
appl from network memory and vice versa on receive. Also the checksum
is calculated as the packet is moved out to adapter memory, and also
as it is being received into adapter memory.

The software issues are indeed large, and we are working on that as
well. Hopefully it will be on 4.4 large winow TCP.

jim

From: [email-masked] (Steve Wise)
To: Craig Partridge <[email-masked]>
Cc: [email-masked]
Subject: Re: Checksumming on TCP 
Date: Fri, 19 Jun 92 12:20:32 -0600

| What has been done in 4.4 and in the modifications to 4.3 that I'm
| aware of to support rolling checksum and copy loops is to add protocol
| specific versions of sosend() to the protosw structure and have sendit()
| call these routines (if they exist) instead of sosend().

	Is the BSD 4.4 code available?  

| 
| far simpler.  Also, by being in protocol-specific code when copying data
| from application space, one can make more intelligent memory placement
| decisions (e.g. Jacobson's idea to copy direct from application space
| to interface memory requires that one know which interface one will send
| out -- which is a protocol-specific decision).  Finally, one can easily

	About Van Jacobson's stuff.  Does the TCP segment stay in 
	interface memory until the ACK is received?  This memory's on the
	adapter, right?

| roll together checksum and copy because one knows how big a segment one
| is going to send and which checksum algorithm to use.
| 
| Hope this is useful.
 
	It is...thanx
Stevo.

To: [email-masked] (Steve Wise)
Cc: [email-masked]
Cc: [email-masked]
Subject: re: Checksumming on TCP
Reply-To: Craig Partridge <[email-masked]>
From: Craig Partridge <[email-masked]>
Date: Fri, 19 Jun 92 09:32:28 -0700
Sender: [email-masked]

> One argument, which I agree with, is that you should reduce your
> copies/moves of the data to a minimum.  If you're doing more than one
> copy, you might as well not worry about the checksum time...And,
> assuming you must move from user space to system memory (or adapter
> memory), do the checksum during this move.
> 
> Question: 
> On a BSD 4.3 RENO based system, is this possible?  My understanding is
> that you move it into a mbuf chain in the socket layer.  Since there is
> no guarantee that the exact chain will be shipped as a TCP segment,
> what good is checksumming this data during the copy into the
> socketbuff?
> 
> My guess would be to only move in chunks of TCP segments in at a time,
> checksum while you move it into the socket layer, then make sure TCP
> sends exactly that segment using the checksum computed during the
> move.
> 
> Since (I believe) the checksum includes parts of the IP and TCP
> headers, then this move from user space to kernel space would need to
> be done in tcp_output() instead of sosend()...hmmm

Steve:
    
    Well, you could modify sosend to checksum each mbuf separately and
store the IP checksum of that mbuf's worth of data in the mbuf.  Then
when you assemble the TCP segment, you simply add together the IP checksums
of the constituent mbufs.  However, to make this practice useful you'll
want to keep mbufs small (i.e. no clusters) which will hurt performance
in other ways.

    What has been done in 4.4 and in the modifications to 4.3 that I'm
aware of to support rolling checksum and copy loops is to add protocol
specific versions of sosend() to the protosw structure and have sendit()
call these routines (if they exist) instead of sosend().

    This practice is a win on multiple levels -- a lot of sosend()'s code
is not relevant to most protocols so the protocol specific sosend() can be
far simpler.  Also, by being in protocol-specific code when copying data
from application space, one can make more intelligent memory placement
decisions (e.g. Jacobson's idea to copy direct from application space
to interface memory requires that one know which interface one will send
out -- which is a protocol-specific decision).  Finally, one can easily
roll together checksum and copy because one knows how big a segment one
is going to send and which checksum algorithm to use.

    One other point (perhaps of interest to the POSIX types).  When rolling
the checksum and copy together on the inbound side, the semantics of
recv*() routines have to change slightly, to wit:

    * EWOULDBLOCK must be a valid return code even after select() indicates
    there is data to be read.  [The data to be read has not yet been
    checksummed because we don't know what application buffer to put it
    in until a recv*() call tells us.  But on checksumming we may discover
    the data is bad...]

    * The entire buffer provided may be modified even if the number of bytes
    read is smaller than the buffer size.  (In UDP it may be the first datagram
    we tried to copy filled the entire buffer but failed the checksum, so
    we copied in the following datagram which was smaller but checksummed
    correctly...]

Hope this is useful.

Craig Partridge
[email-masked]

From: [email-masked] (Steve Wise)
To: [email-masked]
Cc: [email-masked]
Subject: Checksumming on TCP
Date: Fri, 19 Jun 92 10:00:40 -0600

Hi!

I have been hearing lots of discussion about turning off the TCP
checksum over links that provide checksumming (like HiPPI, FCS, ATM,
etc).  I understand somewhat the arguments pro and con.

One argument, which I agree with, is that you should reduce your
copies/moves of the data to a minimum.  If you're doing more than one
copy, you might as well not worry about the checksum time...And,
assuming you must move from user space to system memory (or adapter
memory), do the checksum during this move.

Question: 
On a BSD 4.3 RENO based system, is this possible?  My understanding is
that you move it into a mbuf chain in the socket layer.  Since there is
no guarantee that the exact chain will be shipped as a TCP segment,
what good is checksumming this data during the copy into the
socketbuff?

My guess would be to only move in chunks of TCP segments in at a time,
checksum while you move it into the socket layer, then make sure TCP
sends exactly that segment using the checksum computed during the
move.

Since (I believe) the checksum includes parts of the IP and TCP
headers, then this move from user space to kernel space would need to
be done in tcp_output() instead of sosend()...hmmm

Any thoughts, comments?

********
Steve Wise 	IBM PSP - Austin, TX.
Internal: 	[email-masked]
VNET:		AUSVMQ(SWISE)
Internet:	[email-masked]

X400-Received: by mta mhs-relay.cs.wisc.edu in /PRMD=XNREN/ADMD= /C=US/; Relayed; Fri, 5 Jun 1992 04:40:39 +0000
X400-Received: by /PRMD=uninett/ADMD= /C=no/; Relayed; Fri, 5 Jun 1992 04:40:26 +0000
X400-Received: by /PRMD=uninett/ADMD= /C=no/; Relayed; Fri, 5 Jun 1992 04:40:21 +0000
Date: Fri, 5 Jun 1992 04:40:21 +0000
X400-Originator: [email-masked]
X400-Recipients: [email-masked]
X400-Mts-Identifier: [/PRMD=uninett/ADMD= /C=no/;920605114021]
X400-Content-Type: P2-1984 (2)
Content-Identifier: 3369
From: [email-masked]
To: [email-masked] (Non Receipt Notification Requested)
Subject: Implementations of RFC1323

Hello,

I am doing a performace study of Supernett, a 34 Mbps inter-city IP-network  in Norway, having FDDI access MAN's using NSC and cisco routers. I have discovered 
that the current 64KB TCP window is to small for filling the
network with one session due to larger bandwitdth*delay product.

We have DECstations, Suns, a Cray and possibly an Alliant available.
Is there interest for testing any of the upcoming trial implementations
in such an environment ?

Also - if possible I would like to listen in on the discussions by joining the
list.

Regards
  Olav Kvittem

      RFC Address     [email-masked]
      OR Address      C=no;ADMD=" ";PRMD=uninett;
                      O=sintef;OU=delab;S=Kvittem;G=Olav
      Postal Address  SINTEF DELAB
                      N-7034 Trondheim
      Phone           +47-7-532586(FAX)
      Phone           +47-7-596981
      Description     Research Scientist  -  Group  for  Communication
                      Networks