Subject: Re: OPES Status Update, Drafts, ICAP

From: Ralf Horstmann <[email-masked]>
Date: Wed, 18 Sep 2002 22:39:59 +0200

On Mon, 16 Sep 2002, Markus Hofmann wrote:

> time for a brief status update on our OPES activities and on 
> immediate 
> next steps.

[...]

> As such, it's time for all of us to carefully read the ICAP 
> Internet Draft
> 
>    "ICAP - the Internet Content Adaptation Protocol"
>     draft-elson-icap-01.txt
> 
> and to provide comments on this protocol specification in the context 
> of OPES. Please be aware that the ICAP specification is *not* a 
> product of the OPES WE, but represents an individual submission, 
> documenting a "status quo". Nevertheless, it's of quite some 
> relevance 
> for our WG, and we better have a careful look at and comment 
> on it. In 
> particular, it might be interesting to discuss:
> 
>   (a) How does ICAP fit into the OPES context and how does it stack
>       up to the OPES protocol requirements?
>   (b) If there are open issues, would it make sense for the WG trying
>       to recommend extensions toICAP so that it'll meet all
>       requirements?
>   (c) What are the strengths, the weaknesses, and the limitations of
>       the current ICAP specification? Are there lesson to be learned
>       from the spec and from the work on ICAP?
>   (d) Given that there are several (commercial) implementations
>       of ICAP out and in use - are there any lesses learned from these
>       implementations? Is the current specification stable and clear,
>       or are there any open issues or issues to clarify?

I'm working on the ICAP client implementation for Squid. See
http://icap-server.sourceforge.net/squid.html for details. 
The ICAP specs were quite useful to get the implementation
ICAP/1.0-compliant. In my opinion the specification is clear and ensures
that different ICAP implementations will work just fine together.

Regards, Ralf.

--
ruby -h | ruby -e 'a=[];readlines.join.\
scan(/-(.)\[e|Kk(\S*)|le.l(..)e|#!(\S*)/) \
{|r| a << r.compact.first };puts "\n>#{a.join(%q/ /)}<\n\n"'

---

From: "Martin Stecher" <[email-masked]>
Date: Tue, 24 Sep 2002 12:21:43 +0200

Hi.

> 
> So far, it seems that the ICAP protocol specification in its current 
> form does not have major hiccups that would leave room for 
> misinterpretation and interoperability problems. However, did 
> somebody 
> actually do some interoperability tests of different ICAP 
> implementations? Do different, independent ICAP implementations that 
> conform to the current spec easily interoperate with each other, or 
> are there parts that leave room for different interpretations?
> 

I tested the WebWasher ICAP server against five different ICAP clients and I tested the WebWasher ICAP client against three different ICAP servers.
During testing (or later in productive environments) we found some communication problems. But for all of these problems the protocol draft gave us a clear statement which side was correct and which side was wrong. After fixing the problems we run with all the mentioned clients and servers without having any special workaround code except of one setting that addresses a typical problem I found several ICAP clients have in the beginning and which is usually solved in later revisions of the ICAP client: Handling an ICAP response header which is sent in multiple TCP packets. (This problem is beyond the protocol specifications)

So from my experience the protocol draft is really clear in how to design interoperable implementations.
On the other hand I already saw a switch in another ICAP client to set the type of ICAP server. I have no testing experience with the servers mentioned there. So there may be protocol problems that I am not aware of or just bugs that have been worked around that way.

I hope that the guys that experienced the reason for such a switch will speak up in this forum and share their experience with this group.

> Now, we also need to look beyond questions about the 
> clearness and the 
> stability of the current ICAP specification. I'd like to get some 
> feedback on how people feel of ICAP with respect to the laid out OPES 
> architecture and the OPES protocol requirements. How does ICAP fit 
> into this picture? Are their major roadblocks, minor issues 
> that could 
> be "fixed", or is the current ICAP perfectly fine?

ICAP is good but not perfect I think. Please check again my evaluation paper (draft-stecher-opes-icap-eval-00.txt)
It could be a good starting base. For me the OPES protocol could be an ICAP/2.0 implementation.

> 
> > 2. The PREVIEW header support in the protocol is very useful in
> > optimizing response modification. 
> 
> I can see and agree that the "preview" mechanism is quite useful in 
> several secnarios. However, it requires that the preview size is 
> statically agreed on a-priori. I could imagine that there are 
> services 
> that can deliver a final response before receiving the entire 
> message, 
> but without knowing how many bytes (i.e. preview size) they need to 
> see before being able to do so. Simple example: A filtering service 
> searching for certain keywords in the message. As soon as a forbidden 
> keyword is detected, there's no need to further analyze the reminder 
> of the message, the transaction can be finished immediately. So, it 
> would be nice to have a more general mechanism that would allow the 
> callout server to present a final response to a transaction at any 
> given time, and not only after receiving the preview or after 
> receivingthe entire message.
> 

I fully agree: Having a flexible number of decision points in the protocol would be a big plus.

Beside this I believe that although ICAP is simple, the protocol could even be simpler. It's one and a half year ago that I started the discussion about the Encapsulated headers and a proposal to simplify things using more chunk extensions.

Meanwhiletime I have strong interest to have a callout protocol that it able to support many different application protocols. OPES wants to concentrate on HTTP and RTP first which is fine but we should prepare the protocol from the very beginning to be easily extendable.

We have some experience how to encapsulate FTP and even SMTP into ICAP although ICAP is specifically designed for HTTP and while FTP encapsulation is still simple, the SMTP stuff is much harder because of one important difference: The HTTP and FTP data is sent to one recipient (the requesting client) while a mail message is sent to many different recipients that could have different filtering needs. ICAP is not prepared to handle this.

Martin

---

From: Arumugam K <[email-masked]>
Date: Tue, 24 Sep 2002 19:24:23 +0530

Hi all,

   The following are the few points about the ICAP protocol
   we would like to share.

1. A single client request-response flow could include both reqmod and
   respmod modifications. This could be explicitly mentioned in the
draft.
   An example could be the decompression service.
   ( This service is mentioned in the 4 point)

2. One concern over the request modification.

   Icap client sends  the client request to icap server. The icap server
   could sent an adapted request , an HTTP error message
   indicating the client request is prohibited or ICAP error msg.

    The icap server sends ICAP error msg due to an error in ICAP
    semantics such as ICAP request is not correct or service not
available.
    In this case , the icap client can either forward the original
    request without any modification to OS or send an error msg to 
    client stating that the service could not be provided.

    It would be useful if the draft says how to handle this scenario.

3. The draft is not posing any prerequisite to the tools used to provide
   the actual services. Based on the protocol , it seems to be valid to
   have few prerequisite on the service tools.

   1. The tool has to be run in non-interactive mode. GUI interface is
      not needed.

   2. The tool should be started by cli. Allows to pass command line
      arguments.

   3. In the simplest form, each instance of the tool serves only one
      request.Considering the overhead due to the size of the tool and 
      time to load and make it ready , it would be fine to make a single
      instance of the tool to serve multiple requests either
      sequentially or parallely. I am not sure how far this is
achievable.

4. It could be possible to make use of the ICAP protocol for real
   bandwidth savings.This is possible by providing a service 
   called decompression.

   For example, consider a scenario.
   The client makes a request to html page in OS. Due to ICAP ,a request 
   with  a url which points to the compressed copy of the html page is
   sent to OS. This is done in REQMOD mode.

   OS sends back the compressed data. The compressed data is sent to
   ICAP server which can decompress and send the data in the format 
   requested by the end-user. This is done in RESPMOD mode.

   Thus the bandwidth needed to transfer the html page is reduced to the
   level of compression.

   If this service is considered as a valid one, then this service could
   be mentioned in the draft.

Regards
Arumugam

-- 
Visit us at http://cdn.hcltech.com