Subject: Re: Resend: HTTP Performance extension for NAV(network-based anti-virus) systems

From: Adrien de Croy <[email-masked]>
Date: Fri, 03 Aug 2007 13:11:52 +1200

Hi

A few months back I submitted an I-D which contained proposals for 
progress notifications.

the reason I raise that, is because it considers the same scenarios as 
your draft, i.e. it was specifically conceived to deal with issues 
around gateway scanning.

There is a problem with the approach taken by your I-D.

It is concerned around the issue of event sending any of the resource to 
the client prior to it having been verified as safe.

Trying to get the client to ignore the file after it has downloaded a 
whole heap of it isn't safe.  The file may have in fact been padded out 
to avert such techniques, so an entire workable image may have been 
already received by the client, which would run if executed.

There's not really any amount of an unscanned resource that can be 
guaranteed to be safe to send (well maybe just a few bytes).

This is why we're trying to deprecate "drip-feeding" in WinGate, since 
it's fundamentally a flawed concept, albeit necessary to appease the 
users of browsers.

I just submitted a revised draft which I'll also post here, which just 
focuses on using interim response messages to report progress.

Cheers

Adrien

[email-masked] wrote:
> Hi,
>
> I have submitted a draft on "HTTP Performance extension for NAV
> (network-based anti-virus) systems":
> http://www.ietf.org/internet-drafts/draft-babu-navmime-00.txt
>
> This draft attempts to also solve issues in reporting download progress
> and avoiding client connection timeout in presence of network-based
> anti-virus/anti-spam proxies.
>
> Like to hear feedback on this.
>
>
> Thanks,
> Babu
>
> PS: Appologies for resending this. Forgot to indicate the subject in my
> last email, hence re-sending.
>
>
>
> ********************************************************************************
> This email message (including any attachments) is for the sole use of the intended recipient(s) 
> and may contain confidential, proprietary and privileged information. Any unauthorized review, 
> use, disclosure or distribution is prohibited. If you are not the intended recipient, 
> please immediately notify the sender by reply email and destroy all copies of the original message. 
> Thank you.
>  
> Intoto Inc. 
>
>
>   

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: "Babu.N" <[email-masked]>
Date: Fri, 03 Aug 2007 13:47:37 +0530

Hi,

Thanks for the comments.

Please see inline.

At 06:41 AM 8/3/2007, Adrien de Croy wrote:

>the reason I raise that, is because it considers the same scenarios 
>as your draft, i.e. it was specifically conceived to deal with 
>issues around gateway scanning.
>
>It is concerned around the issue of event sending any of the 
>resource to the client prior to it having been verified as safe.
>
>Trying to get the client to ignore the file after it has downloaded 
>a whole heap of it isn't safe.  The file may have in fact been 
>padded out to avert such techniques, so an entire workable image may 
>have been already received by the client, which would run if executed.

This is the reason I have the following statements in the draft:

"
    For some reason, if a client doesn't receive second and third (when
    applicable) sub-parts, it MUST never store/process the already
    received data.
"

"
    Some programs like browsers' "Download Manager" start storing the
    data to disk even before the complete response is received. Such
    applications should ensure that they delete any partial data in the
    event of "Proxy response IGNORE" or when second/third body (when
    applicable) parts are not received.
"

So implementations confirming to this strictly should never 
run/process a response till the response is completely received. So 
they should not have an issue ? Please suggest.

Thanks,
Babu

>This is why we're trying to deprecate "drip-feeding" in WinGate, 
>since it's fundamentally a flawed concept, albeit necessary to 
>appease the users of browsers.
>
>I just submitted a revised draft which I'll also post here, which 
>just focuses on using interim response messages to report progress.
>
>Cheers
>
>Adrien
>
>
>[email-masked] wrote:
>>Hi,
>>
>>This email message (including any attachments) is for the sole use 
>>of the intended recipient(s) and may contain confidential, 
>>proprietary and privileged information. Any unauthorized review, 
>>use, disclosure or distribution is prohibited. If you are not the 
>>intended recipient, please immediately notify the sender by reply 
>>email and destroy all copies of the original message. Thank you.
>>
>>Intoto Inc.
>>
>>
>
>--

Intoto Inc.

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 07 Aug 2007 09:27:46 +1200

Hi

The underlying issue IMO is that of the human user knowing what's going 
on.  If you have some sort of progress notification, that allows the 
progress bar on the bottom of the browser window to show movement, and 
the status area to show activity, then users won't hang up waiting.  
Download managers could show this information as well.

The actual transmission of the file from the proxy to the client is then 
secondary.  It will happen at LAN speed once the file is scanned.  Since 
responses to date have been interim, the result of the scan (e.g. if 
failure) can simply be returned as the response to the request.

Your proposed method seems to assume that transfer of the resource is 
necessary for there to be progress to show.  Having a separate specific 
progress mechanism however does this, and furthermore is applicable to 
more cases than just gateway scanning, in fact is useful any time a 
request will take a while to process, and can be generated by gateways 
and/or servers.

Furthermore the Progress mechanism is transparent to proxies that don't 
understand it, and is much less complicated to implement than a 
multi-part post-abort mechanism, so would find much easier deployment.  
It's arguably safer as well, since a problem in post-aborting a request 
could still leave compromised data on the client system, puts the client 
to more work as well (esp considering client-side processing such as AV, 
caching etc).

Regards

Adrien

Babu.N wrote:
>> as your draft, i.e. it was specifically conceived to deal with issues 
>> around gateway scanning.
>>
>> It is concerned around the issue of event sending any of the resource 
>> to the client prior to it having been verified as safe.
>>
>> whole heap of it isn't safe.  The file may have in fact been padded 
>> out to avert such techniques, so an entire workable image may have 
>> been already received by the client, which would run if executed.
>
>> <http://www.wingate.com/>
>

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 06 Aug 2007 23:47:29 +0200

> Download managers could show this information as well.

Indeed.

I like the 1xx Progress extension. It's a natural use of 1xx type of
messages.

But imho it should also be obvious to all client implementators that
processing a partial response as if it was a complete and valid response
should generally not be done.

Regards
Henrik

---

From: "Babu.N" <[email-masked]>
Date: Fri, 10 Aug 2007 15:56:51 +0530

Hi,

Thanks for the comments. Please see inline.

At 02:57 AM 8/7/2007, Adrien de Croy wrote:

>Hi
>
>The underlying issue IMO is that of the human user knowing what's 
>going on.  If you have some sort of progress notification, that 
>allows the progress bar on the bottom of the browser window to show 
>movement, and the status area to show activity, then users won't 
>hang up waiting.

Agree. But I would like to clarify that solutions to "Progress 
notifications" or "clients inactivity timeout" is not the focus of 
this draft. This was only an offshoot of the approach used to solve 
the "performance issue". I will remove progress notification 
statements from the draft, if necessary.

>The actual transmission of the file from the proxy to the client is 
>then secondary.  It will happen at LAN speed once the file is scanned.

This is the main focus of this draft. Though it is perceived that 
downloading file at LAN speed is faster, some practical observations 
(under typical LAN congestion level & server load/processing 
capacity) indicated nearly 7.3% improvement in HTTP response time, if 
responses are sent to client while scanning is still in progress.

Consider this scenario:

HTTP client --LAN-- Edge -----1Mbps IPLC-----Edge--LAN--Edge ------ Internet
                |              from ISP              |
             BrachOffice                          MainOffice
             webServer                            Webserevr
<---branch office in -->                     <-----Main ---->
     INDIA(100 Mbps LAN speed)                  office in US
                                                (100 Mbps LAN)

In this scenario, it is observerd that
     - A 12.4 MB file download from an internet webserver to "Branch 
WebServer" took 154 sec.
     - clamd took 11 sec the 12.4 MB file.
     - Client took 13 sec to download the same 12.4 MB file.

These figures indicate that
     - HTTP response time without a NAV proxy: 154 sec
     - HTTP response time with NAV proxy: 154 + 11(clamd scan time) + 
13 (proxy->client transfer time) = 178 sec

The focus of this draft is to lessen the 13 sec from this HTTP 
response time, by sending the HTTP response from Internal web proxy 
to the client even before scanning is complete.

In this scenario 13 sec amounts to nearly 7.3 %.

Consider the following scenarios:
     - Branch office and main office are connected though IPLC. NAV 
scanning gets applied at a proxy in main office. Here the time it 
takes for the proxy to send HTTP response to a client in brach office 
will be more than that seen in the above example.
     - Branch office and main office are connected via a L3 VPN. NAV 
scanning gets applied at a proxy in main office. Again, the time it 
takes for the proxy to send HTTP response to a client in brach office 
will be more than that seen in the above examples.

This percentage may increase/decrease depending on NAV proxy->Client 
network speed, congestion level, NAV proxy load & capacity.

Please let me know you thoughts.

Thanks,
Babu

>Since responses to date have been interim, the result of the scan 
>(e.g. if failure) can simply be returned as the response to the request.
>
>Your proposed method seems to assume that transfer of the resource 
>is necessary for there to be progress to show.  Having a separate 
>specific progress mechanism however does this, and furthermore is 
>applicable to more cases than just gateway scanning, in fact is 
>useful any time a request will take a while to process, and can be 
>generated by gateways and/or servers.
>
>Furthermore the Progress mechanism is transparent to proxies that 
>don't understand it, and is much less complicated to implement than 
>a multi-part post-abort mechanism, so would find much easier deployment.
>It's arguably safer as well, since a problem in post-aborting a 
>request could still leave compromised data on the client system, 
>puts the client to more work as well (esp considering client-side 
>processing such as AV, caching etc).
>
>Regards
>
>Adrien
>
>
>Babu.N wrote:
>>Hi,
>>
>>Thanks for the comments.
>>
>>Please see inline.
>>
>>>the reason I raise that, is because it considers the same 
>>>scenarios as your draft, i.e. it was specifically conceived to 
>>>deal with issues around gateway scanning.
>>>
>>>Trying to get the client to ignore the file after it has 
>>>downloaded a whole heap of it isn't safe.  The file may have in 
>>>fact been padded out to avert such techniques, so an entire 
>>>workable image may have been already received by the client, which 
>>>would run if executed.
>>
>>    For some reason, if a client doesn't receive second and
>>third (when    applicable) sub-parts, it MUST never store/process the
>>already    received data.
>>
>>"
>>
>>"
>>    Some programs like browsers' "Download
>>Manager" start storing the    data to disk even before the complete 
>>response is received.
>>Such    applications should ensure that they delete any partial data
>>in the    event of "Proxy response IGNORE" or when
>>second/third body (when    applicable) parts are not received.
>>"
>>
>>>>This email message (including any attachments) is for the sole 
>>>>use of the intended recipient(s) and may contain confidential, 

Intoto Inc.