Subject: Re: We do have a marketing problem ...

From: Gunther Schadow <[email-masked]>
Date: Tue, 20 Apr 1999 15:04:32 -0500 (EST)

Thank you, Carl, for your list of arguments. It will help me in the
ballot reconsiliation procedure where my task is to talk to people to
get them withdraw their negative votes.

I now take Kit's concerns to practice a little bit :-)

Kit makes the following points:

(1) with FTP (same for HTTP or custom direct TCP) you kow immediately
whether the transfer succeeded or failed.

(2) which has the advantage that you can act on the problem right
away.

(3) with e-mail you are sort of disconnected from what's going on. You
don't know whether your party has got the message and you have to deal
with longer timeouts for the receipt.

(4) when the timeout hits (because the receipt took long to get back)
and you send the message again, you sent it twice and it may
eventually get there twice.

(5) with e-mail, therefore, you have to care to resend the messages
later, so you have to queue them and check them off when the receipt
comes in later.

(6) with FTP you can see whether your file resides on your party's
system so you don't have to keep the queue at your's.

(7) since there is so much that can go wrong with e-mail, it is less
reliable and more difficult, probably even expensive, to set up e-mail
communication.

In summary: e-mail is asynchronous, not real-time so it's slow,
unreliable, you have to retry delivery yourselves.

This is not just Kit's argument, this is the exact same argument that
one can hear everywhere.  One almost wonders whether there is a common
source, some article in a magazine that originated this
argumentation. Something one can respond to once and for all?

So, what's wrong with this argumentation?

(1) FTP (HTTP, custom direct TCP) does much less for you than e-mail,
which is why the little that those other technologies do seems to be
so reliable. The truth is, if you expect from e-mail the same
functionality that you expect from FTP, e-mail is at least as reliable
as FTP. Why?

With FTP you get the following:

a) Transfer a file from your file system into your party's file system,
b) using a TCP virtual circuit
c) that connects directly from your machine to your party's machine
d) and tells you right away whether the transmission succeeded or
   failed.

When does an FTP transmission fail?

f1) when you can not set up a direct TCP connection to your party's
    server because of one of the following reasons:

      - your party's machine is behind a dialup line and is
        disconnected most of the time.

      - your party's machine is in a non-IP network such as a
        proprietory LAN or a peer to peer network (like UUCP.)

      - your party's machine is behind a firewall.
   
f2) when your party's FTP server is down.

f3) when you do not know in which directory you should drop off
    your messages and what the file naming conventions are.

f4) when the transmission fails because of a transient network
    problem (e.g., people stumbling over the wire, a clogged
    up network, etc.)

f5) when the receiver's file system is full.

f6) when after dropping of your file, your party's system crashes
    and suffers data loss.

f7) when your party ignores to actually process your message and
    clears it from its file system.

f8) when the line termination translation screwed up (How often have
    you downloaded a 500 kByte ZIP file in ASCII mode only to find out
    you forgot to choose binary transfer mode?)

How does e-mail compare?  Remember the goal was to drop a message off
in a receiver's file system, using a direct TCP connect, and be sure
it's there before you disconnect. The truth is, a direct SMTP
connection performs the same service, in the same time, with the same
major failure possibilities (except for f3.)  The SMTP protocol is
actually very similar to the FTP protocol.  It's no wonder, since
Jonathan Postel has designed both both protocols FTP and SMTP during
the years 1980 to 1982.

We can go into all the details if you want, to find out that raw and
direct SMTP meets all your goals that you meet with FTP.  There is a
little difference, of course:

 - with SMTP you can not see what your party's other trading
   partners have sent.
 
 - with SMTP you can not delete messages from you party's
   file system.

 - with SMTP you do not have the choice between different
   drop-off directories.

 - with SMTP you can not choose the file name of your message.

If you want any of those things, you have to ask yourself why? Why
would you want to let your trading partner's see that you are in
negociations with one of their competitors? Why would you want your
party to be able to delete messages from your file system?  Why would
you want to worry about file names and directories? Why the additional
negociations? Why bother with enforcing the additional rules about
directories and file names? Why would you enjoy to manually fix
problems that would not need to occured in the first place?

Of course, you can render your drop-off directory unreadable for any
of your partners.  But then your partner's can not even see whether
their own stuff went trough.  Of course you can build more complex
file structures in your drop-off area, but what for?

SMTP gives you all you need for EDI messaging without the additional
negociations and worries you have with FTP.

Of course, there is a practical point. People know how to set up FTP.
There are nice GUI FTP clients so people can visually drop messages
off in your inbox. At least at this point you have to make clear your
business environment: are you using EDI for occasional purchase orders
or do you work on a basis of 10000 health care claims per day?  If you
are in the real business, you probably want to save on the human labor
involved in operating a nice looking GUI, don't you?  If you deal with
mass EDI communication, you don't want to fill out forms and click the
mouse. You want to pump the stuff through from your data base backend
to your trading partner's backend.

So is it all wrong? All the concerns about unreliable e-mail just
plain wrong? You would not believe me if I said so, would you? So let
me explain. Why does e-mail exist in the first place?  Does it exist
because Jonathan Postel, after having published the FTP specification
in June 1980, thought: "Gee, this net its getting too easy, lets mess
it up a little bit." ---and so he published SMTP in 1982?  You
wouldn't believe that either, would you?

So, what is the point in SMTP and Internet e-mail?  The point is that
FTP has the eight reasons to fail as I listed above.  Direct raw SMTP
has the same reasons to fail.  However SMTP does not just exist
"direct and raw."  E-mail and SMTP are value added services for the
Internet.  What e-mail does for you is to take a bite out of this long
list of problems.  We know those problems are serious in all
communication, if you don't have your cell phone on, how can I reach
you?  Serious problems demand serious treatment.  And serious
treatment comes with some adverse side effects.  Do we not treat sick
patients because of fear for possible side effects?

Well, sometimes we do refrain from treatment, but not for the mere
*fear* for side effects but because we *know* that the chances of cure
do not outweigh the risk of adverse effects.  If physicians where as
lucky as technicians, life would be a lot easier.  Technicians can
track a problem to its first origin, can understand side effects.
Physicians can mostly only count side effects without too much of
understanding why they occur and how to prevent them.  So let's take
advantage of being technicians and let's solve our problems with
knowledge, not fear.

The main two cures that Internet e-mail provides is message queuing
and message relaying.  Message queuing is the idea that outgoing
messages that can not be sent directly are saved and retried at a
later point in time.  This attacks problems f2, f4, and f5.  An e-mail
system, such as Eric Allman's sendmail, will try to resend mail in
certain intervals and for a certain time that you can configure. It
will list you the queue and allows you to manually intervene, since
all messages are stored at a particular place on your file system.
But on a basis of 10000 health care claims per day, you don't want to
manually intervene.  

That's what computers are for: retry, wait, retry, wait, and finally
to succeed or fail.  With 10000 EDI messages per day, you don't want a
polished dialog box telling you "sending message, please hang on..."
If sendmail gives up after the timeout you set, it'll notify you by
sending an error message to a designated user, commonly called the
"postmaster."  You can configure sendmail to re-route error messages
to your phone or FAX or pager, so you can sleep well knowing that the
system will notify you of any serious problems.

So, the reason you get an error message from an e-mail system every
now and then does not mean that the system is less reliable. It does
mean that the direct connection failed, and thus, that FTP, HTTP, or
custom TCP all would have failed too. So, who needs to keep a queue
manually? An e-mail user? Wasn't e-mail about queuing? Yes it was and
therefore an e-mail user does not need to keep a queue to make sure
the message gets delivered. The system does it all for you. So, who
needs the separate queue attention? FTP, HTTP direct TCP, they all
need the additional queue. Sendmail/SMTP comes with message queue
feature out of the box.

Message sequencing problem, resent more than once?  Those problems can
exist with all kinds of resending. It doesn't depend on whether you
choose SMTP or FTP.  But there are low risk cures: sequence numbering
and unique message identification. Internet e-mail comes with unique
message id's so you can check for replays. Most EDI protocols come
with a sequence number feature, so you can cope with keeping the
messages in line. It all is independent of SMTP or FTP though. So, I
won't dive too deeply into that. If you want to know more, you are
invited to take a look into section 6.1 of the current HL7 draft
available at:

http://aurora.rg.iupui.edu/security/draft-ietf-ediint-hl7-01.txt
http://aurora.rg.iupui.edu/security/draft-ietf-ediint-hl7-01.pdf

The second cure that e-mail has is relaying.  Relaying e-mail is a
very powerful service that treats mainly problem f1 but also f2, f3,
f4, and f5.  Where direct TCP connections including FTP, HTTP, and raw
SMTP only go over IP-Networks, e-mail can reach far beyond.  E-mail
can be delivered over X.400 WANs, proprietory LANs (e.g., Novell/IPX),
and peer to peer networks, like UUCP, which can reach even in the
remotest communication desert.  E-mail relaying through firewalls is a
standard task that your firewall administrator will do anyway.

I have heard of serious problems people had with setting up an e-mail
system in their company for human users. Interestingly when it comes
to the report of companies being bogged down by their e-mail system I
hear that Lotus Notes was involved. There is no doubt, setting up a
relaying an queuing system requires you to know what you want and to
know what you do. But it is still way better than to handcraft
software around FTP and HTTP to implement queuing an relaying
at your site.  You need to write a lot of program and go through
debugging and mistakes, you even create your own security loopholes,
which have been discovered and removed from sendmail by tenthousands
of users all over the world since 15 years.  You still think SMTP is
unreliable? 

I believe that overcomming this concern about unreliable e-mail is
something you have to do actively.  If you want to be worried about it
and choose to go your own way with FTP and HTTP, protocols that were
not designed to run back-end to back-end on a basis of 10000 messages
a day, you certainly can do so.  But if you rather want to deliver EDI
messages with the stuff that has been designed for messaging, you can
come to the confidence that it can work very well.  However, what you
have to do is stop carrying your feeling that e-mail is no good and
start nailing down the arguments. 

The best way for patients to cope with their disease is to learn
something about it and to understand what it is doing.  So if you are
worried about e-mail and you can't trust it, learn more about it and
make your decision from knowing what's going on and what's involved in
those protocols.

And I have not even talked about non-repudiation receipts and the
world of message delivery notifications (MDN.)  If we look at this,
than the claim e-mail is unreliable is even more false. With a
processing level MDN you can prove that your party has actually run
the message through its processor. This is something the FTP will
never give you, unless you design an e-mail system using FTP instead
of SMTP. Imagine all the work, all the bugs, all the security
loopholes you aren't even aware of? Do you still want to go alone with
FTP instead of joining the world of e-mail?

regards
-Gunther

Kit C. J. Lueder wrote:
> I agree that there is a perception problem that e-mail is
> unreliable.  That is why EDIINT was developed in the first place.
>
> However... with FTP you know immediately at the end of the session
> whether the FTP transfer succeeded or failed, so you can act on the
> failure now or archive the document that was successfully sent. With
> e-mail, even using the EDIINT secure receipt, you don't know untill
> later whether it succeeded or failed (including the use of a timeout if
> you don't receive a response at all). So EDIINT takes more effort to
> implement, since you have to keep track of it and save the original
> files, and not know whether they are "checked off" until later.
>
> There is also the risk that your EDIINT transmission succeeded, but the
> receipt that was sent back in response failed, in which case you resend
> your original transmission after a timeout of waiting for the receipt.
> This would result in the recipient receiving duplicate copies of the
> transaction, which some VANs or users would consider to be a problem.
> This generally does not happen with FTP, since you can see if the file
> resides on the destination system at the end of the transfer.
>
> > It is incorrect to state that SMTP is not real-time whereas HTTP is.
> > Compared with traditional VANs, EDIINT offers:
> >   - Real-time messaging
>
> I don't see why e-mail is "real-time" (unless the only alternative is
> "batch"). You send and wait for an asynchronous reply. Maybe we need a
> definition of "real-time". You don't have a session or a virtual
> connection with e-mail. Can't you have a session with http (maybe that
> is what cookies are used for), in which case http is real-time?
>
> Kit.
>
> Carl Hage wrote:
> > 
> > Date sent:              Mon, 19 Apr 1999 17:18:17 -0500 (EST)
> > From:                   Gunther Schadow <[email-masked]>
> > 
> > > ...The hardest argument to
> > > break is the dull feeling that "e-mail is unreliable."
> > 
> > I think you are right. In fact email is more reliable than http (email
> > has error-retry whereas www broken/inaccessable connections
> > simply fail).
> > 
> > Perhaps the problem is using the term "email". There is a big
> > difference between uucp email gatewayed into a proprietary LAN
> > email system and EDIINT. Why not use the term "EDIINT
> > Messaging", which has the properties:
> >   - real time message transfer (if recipient has a dedicated
> > connection)
> >   - guaranteed delivery confirmed by receipts
> >   - legal non-repudiation
> >   - ability to use redundant network providers and processing sites
> >    for ultra-high reliability
> >   - compatible with dedicated (real-time) or dial-up net connections
> >   - compatible with standard Internet software
> > 
> > The irony is that traditional EDI using vans _is_ email, but
> > proprietary email.
> > 
> > Compared with HTTPS (SSL Secure-HTTP), EDIINT offers:
> >   - Legal nonrepudiation
> >   - Automated recovery on delivery failures
> >   - Support for backup sites/networks
> > 
> > It is incorrect to state that SMTP is not real-time whereas HTTP is.
> > 
> > Compared with traditional VANs, EDIINT offers:
> >   - Real-time messaging
> >   - Interoperability across all network providers
> >   - Increased security
> >   - Increased reliability
> >   - Lower network cost
> > 
> > The main problem with VANs (other than cost) is potential poor
> > reliability across VANs. I lost my reference (US government
> > electronic filing) where cross-VAN deliveries took several days.
> > Simply changing VANs creates major problems since it requires
> > manual intervention with all TPs.
> > 
> > Also, a VAN is a single point of failure. Two internet-compatible
> > network providers (ISPs) with independent networks can be
> > obtained for less than the cost of one VAN. Changes in ISPs or
> > routing across alternate providers is transparent. (The DNS system
> > replaces proprietary VAN email addresses.)
> > 
> > Security is improved since data can be exchanged encrypted and
> > cannot be intercepted or altered by VANs/ISPs/wiretaps. Non-
> > repudiation is provided by digital signatures, not the just the
> > testimony of a third-party VAN which can offer logging service.

Gunther Schadow ----------------------------------- http://aurora.rg.iupui.edu
Regenstrief Institute for Health Care
1001 W 10th Street RG5, Indianapolis IN 46202, Phone: (317) 630 7960
[email-masked] ---------------------- #include <usual/disclaimer>

---

From: "Gary Morin" <[email-masked]>
Date: Thu, 22 Apr 1999 09:01:19 -0400

I'm sorry but I just don't buy your argument.  HTTP is the only reasonable way
to do high volume EDI over TCP/IP.  When you are sending thousands of EDI
message per day the  whole AS-1 thing just breaks down.  Dealing with the
thousands of disjoint MDN's is just a total pain.  FTP doesn't work mostly for
the arguments you presented and only HTTP as implemented on today's high volume
web servers like IIS and Netscape have the "programmability" to deal with the
inflow, subsequent internal routing and their delivery of the appropriate
documents to internal processing tools.

Building a processing environment to deal with this bi-directional flow of
documents is now quite straightforward using the E-5 or GISB specs and MS
Commerce Interchange Pipeline or any of several Java toolkits now available.
(For an example go to http://www.scinteract.com.)

We need to start thinking or these document interchanges agents as silicon
employees which are in constant contact with one another exchanging information
on a real time basis.  This is the only way to get your supply chain up to
speed.  The automotive companies expect shipping acknowledgements within 30
minutes of the truck leaving the dock. (in some cases this has been found to be
too long)  In that environment any processing delay is unaccepatable.

    -- Gary

---

From: "Carl Hage" <[email-masked]>
Date: Thu, 22 Apr 1999 12:21:12 -0800

From:           	"Gary Morin" <[email-masked]>
Date sent:      	Thu, 22 Apr 1999 09:01:19 -0400
> HTTP is the only reasonable
> way to do high volume EDI over TCP/IP.  When you are sending thousands of
> EDI message per day the  whole AS-1 thing just breaks down.  Dealing with
> the thousands of disjoint MDN's is just a total pain.  FTP doesn't work
> mostly for the arguments you presented and only HTTP as implemented on
> today's high volume web servers like IIS and Netscape have the
> "programmability" to deal with the inflow, subsequent internal routing and
> their delivery of the appropriate documents to internal processing tools.

I think there is a problem in perception because people associate 
AS-1 with something like an EDIINT server using sendmail/cc-
mail/lotus-notes/exchange routed to an SMTP gateway, routed to 
some other gateway, routed to sendmail, ... Yes, you _can_ do 
that, but that doesn't make sense. Heck, you could write it as a 
Word macro and run it by double clicking on an attached .doc file!

IIS/Netscape have much less programmability and reliability than a 
decent EDIINT server implementation. The normal http servers have 
poor error detection and recovery.

Whether you use SMTP, HTTP, FTP, or some proprietary protocol, 
an EDI messaging subsystem must deal with errors. If you ask 
me, a thin SMTP client/server would be the easiest to write for.

The ideal solution is a thin EDIINT server, which would be highly 
reliable, have built-in redundancy, and would handle the receipt 
tracking and error recovery. It would use SMTP (or HTTP) direct to 
the recipient's system, with automatic error retry and performance 
monitoring.

AS-1 offers equivalent (or better) performance when interoperating 
with TPs which have net-connections equivalent to that required for 
an HTTP server. However, AS1 can also interoperate with TPs 
having non-dedicated net connections and gateways though VANs, 
etc.

--------------------------------------------------------------------------
Carl Hage                                              C. Hage Associates
<mailto:[email-masked]> Voice/Fax: 1-408-244-8410      1180 Reed Ave #51
<http://www.chage.com/chage/>                          Sunnyvale, CA 94086