Subject: estimated Content-Length with chunked encoding

From: "Greg Dean" <[email-masked]>
Date: Mon, 20 Oct 2008 13:52:19 -0700

*I sent this once before, but I think to the wrong address, excuse me if
it's a duplicate*

I am new to this mailing list.  I wanted to bounce an idea that I've had for
a while off everyone/anyone.  Basically I'd like to provide the sender of a
chunked encoding message to indicate an estimate of the content.  In the
following example resource.zip is created dynamically, however the
approximate size is known.

C: Get /resource.zip HTTP/1.1

S: HTTP/1.1 200 OK

S: Transfer-Encoding: chunked

S: Estimated-Content-Length: 300000

This would allow for the recipient of such a message to prepare for a
message of a certain size.  The best example of this at work would be when a
browser displays the estimated time remaining.  Today browsers indicate "Not
Known", or show nothing.  This could also be useful for PUT requests, where
it would allow the recipient to make a decision based on what the sender
expects the size to be (possibly allocating resources, or rejecting the
request altogether).

If something like this already exists in 1.1 or is already being considered,
please accept my apologies in advance.  Otherwise does, anyone have any
feedback?

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 20 Oct 2008 22:59:57 +0200

On mån, 2008-10-20 at 13:52 -0700, Greg Dean wrote:
> 
> Transfer-Encoding: chunked
> Estimated-Content-Length: 300000

> message of a certain size.

Looks useful to me, for the reasons you outlined and a bunch more. See
for example the NTLM auth chunked discussion a year ago or so..

For responses it's meaningful even without chunked.

I don't see anything wrong with it, especially not with the length being
an estiamte and not the exact expected lenght.

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 20 Oct 2008 23:21:21 +0100

> an estiamte and not the exact expected lenght.

In some circumstances you may be able to refine the estimate as the
message is being transmitted.

Chunk extensions ("chunk-extension") would suit that:

    1000;estimated-remaining=299000
    (1000 bytes)
    1000;estimated-remaining=298000
    (1000 bytes)

I don't know if chunk extensions break in the real world, though.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 21 Oct 2008 02:14:18 +0200

> I don't know if chunk extensions break in the real world, though.

Probably, but it's not the specs fault.

I know I have made & released one implemenation which broke on chunk
extensions due to a silly typo. It's fixed since quite some time back,
and was/is in an HTTP/1.0 client so I sleep well... But since there is
no chunk extensions in use it's quite likely there is some other broken
implementations around..  But the majority of implementations most
likely just works. Failing on chunk extensions requires one in most
languages to actively care if there is chunk extensions or other
"unknown" data in the chunk header.

Regards
Henrik

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Mon, 20 Oct 2008 19:36:49 -0500

> I don't know if chunk extensions break in the real world, though.

Or, permit

     1000;completed=25%
     (1000 bytes)
     1000;completed=25%
     (1000 bytes)

My only concern about implementing an 'estimated' extension is what is
your pain tolerance in terms of deviation?  Would a min or max be a better
fit to the problem set?

Given that precomputing the result size is often suboptimal for todays'
assembled responses, but there is a desire to have something to share with
the user, the % notation might work well.  I'm concerned that the estimate
will become an expectation, also costly to precompute, but rarely useful
in practice.

---

From: David Morris <[email-masked]>
Date: Mon, 20 Oct 2008 18:11:22 -0700 (PDT)

> > I don't know if chunk extensions break in the real world, though.

Either % completed or estimated remaining requires computing an estimate
of the total data to be transfered. I don't see a difference in the impact
of either choice on a server. I think there are many cases where a
generated result can be estimated to a 95%+ accuracy, but not to the exact
size needed for content-length. There is no incentive to do it today, but
if it could be utilized to improve the user's experience, many web
application developers would be happy to do so. In addition, a jsp/php/asp
engine could even watch the actual size generated for each request and
recognize some pages with a small standard deviation in generated size.
Use that value automatically.

I'd rather see raw sizes from a recipient's perspective as I might
be able manage resources better. A percentage is only useful for end
user presentation without interpolating from the amount of data already
received. Raw numbers make computation of a percentage trivial while more
easily supporting other use cases.

David Morris

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Mon, 20 Oct 2008 20:48:27 -0500

> Use that value automatically.

The example I had in my head was more oriented to work required.  E.g. if
this is a database access or map/reduce problem, representing what % of
the records had been searched (with an understanding that the amount of
bytes then associated with individual records, the actual amount of work
to be performed per record, or the number of actual record matches that
remain can vary widely).  Still, humans seem to want some measure of how
much might still remain to be completed.

> easily supporting other use cases.

My point is that managing local resources by imaginary numbers is probably
an exercise in futility.  Again, what is your threshold for pain?  10x an
allocation that you didn't consume?  Or underestimating the data requirements
by a factor of 10?  These have to be understood before a space-oriented false
advertisement is announced to the client.

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 21 Oct 2008 14:52:52 +1300

another key use of such a field is for policy control.

For instance a proxy that doesn't want people downloading files over a 
certain size, or uploading items over a certain size, but for various 
reasons the transfer mode needs to be chunked (e.g. NTLM auth + POST).

In such cases the worth of such a value and how important its accuracy 
is, is related to what it will be used for.  A proxy may not care really 
how accurate the estimate is, if it will use it for blocking, although 
if the reality shows the estimate was very inaccurate, some other 
processing / consequence may be required.  E.g if we are to use such a 
thing for policy (which I know users wish for since I have had many such 
user requests) then we can't just trust the input from the client - or 
at least need to take a conservative approach.

So in short, I'd like to see the capability for agents to indicate a 
length as well as using chunking, whether that's an estimate or not.  
Perhaps a max size would be most useful in most cases (e.g. will not 
exceed X bytes), or has been mentioned, a range.

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 21 Oct 2008 15:10:49 +1300

> much might still remain to be completed.
for this point, you want my draft ID on progress notifications :)

http://tools.ietf.org/id/draft-decroy-http-progress-04.txt

regards

Adrien

---

From: "William A. Rowe, Jr." <[email-masked]>
Date: Mon, 20 Oct 2008 21:32:10 -0500

> http://tools.ietf.org/id/draft-decroy-http-progress-04.txt

Although I agree with a desire to keep the request lightweight, so only
to pass on the information when desired, the chunked scheme is nearly
always used for progressive data.  If you amended the draft to use an
attribute extention on the chunk header, it might be very interesting.

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 21 Oct 2008 15:44:52 +1300

> attribute extention on the chunk header, it might be very interesting.

could be interesting.  My original use case was to prevent user timeouts 
due to lack of perceived progress, so all the 1xx responses would be 
sent prior to any data (chunked or otherwise)... but I can see some good 
potential for being able to write to the UI of a UA during chunked data 
download as well - like a multiplexed data stream.

Although I thought chunk extension headers could only occur on the final 
chunk?  Or are they different to attribute extensions... bit rusty on 
those...

---

From: "Brian Smith" <[email-masked]>
Date: Mon, 20 Oct 2008 23:56:15 -0500

Adrien de Croy wrote:
> could be interesting.  My original use case was to prevent user
> timeouts due to lack of perceived progress, so all the 1xx responses
> would be sent prior to any data (chunked or otherwise)... but I can
> see some good potential for being able to write to the UI of a UA
> during chunked data download as well - like a multiplexed data stream.

By far the most common case, in my experience, is when the server is 
dynamically compressing the response (e.g. mod_deflate). In that case, the 
server often knows exactly the unencoded length but doesn't know the encoded 
length. If the server could send the unencoded length in the header then the 
client could provide a perfectly accurate download progress bar as long as it 
decodes the content-encoding as it downloads. I would be very interested in 
such an extension and I think UA implementers would be quick to implement it.

I'm not sure that UA builders are going to care so much about the other 
proposed uses. There are already workarounds for those use cases (using AJAX 
and with the progress information encoded in custom message formats). In order 
to be useful for AJAX applications, the information would have to be exposed 
in the XHR object. Further, it is really confusing or the end user to see an 
estimate of progress that is inaccurate, so I doubt UAs are going to be eager 
to display anything that is just an estimate. It seems like a lot of 
specification work, followed by a lot of implementation work, just to show the 
user information that is often likely to be inaccurate and misleading.

> Although I thought chunk extension headers could only occur on the
> final chunk?  Or are they different to attribute extensions... bit
> rusty on those...

I think you are thinking about trailers. Chunk extensions can occur on any 
chunk, AFAICT, according to the specification. However, I wouldn't be 
surprised to see real world interoperability problems with them.

Regards,
Brian

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 21 Oct 2008 22:38:37 +1300

the use case relating to scanning at a proxy is actually very widespread.

For instance AV scanners that do heuristic scanning can't function 
without the entire resource, therefore at a gateway, the entire download 
must be spooled before it's safe to send any of it.  This means the user 
sits waiting until the proxy has downloaded the entire file and scanned 
it before they see ANY progress.

This affects millions of people, it's a big problem, and historically a 
big support issue for us.  Workarounds are almost worse than the problem 
- sending part of the file to keep the human attached to the browser 
happy that something is happening.  Trading security against convenience 
ad costing complexity and performance.

Another area we see problems in is in requesting resources which take a 
very long time to generate.  E.g. logging into a webmail server with a 
huge index and number of folders on the back end, it can take several 
minutes before a single byte is sent to the UA.  Humans don't wait that 
long.  They hit the refresh button several times before that.

---

From: "Brian Smith" <[email-masked]>
Date: Tue, 21 Oct 2008 06:52:22 -0500

> the use case relating to scanning at a proxy is actually very

I think that is an important use case, but the progress notification proposal 
makes more sense for it, since the client won't even receive the response 
header until the scanning is complete. Plus, you aren't really estimating the 
number of bytes remaining in the response but the amount of time the user will 
have to wait; usually the proxy will know exactly how many bytes will 
eventually be returned.

> long.  They hit the refresh button several times before that.

There are already lots of ways of solving that problem without any extensions. 
But, I can see how the progress notification proposal makes sense for it too.

However, I don't think it makes sense for the server to say "hey, I think I 
will return this amount--no wait, this amount--no wait I was right the first 
time--oh no, I was way off, sorry."

- Brian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 27 Oct 2008 15:43:20 +0100

Greg Dean wrote:
> I am new to this mailing list.  I wanted to bounce an idea that I’ve had 
> for a while off everyone/anyone.  Basically I’d like to provide the 
> sender of a chunked encoding message to indicate an estimate of the 
> content.  In the following example resource.zip is created dynamically, 
> however the approximate size is known.
> 
>  
> 
> ...

Interesting enough, the MacOS X WebDAV Client already seems to use 
something like that:

> C-68-#000000 -> [PUT /content/dam/STROMBERG_2_2.mp4 HTTP/1.1 ]
> C-68-#000045 -> [User-Agent: WebDAVFS/1.7 (01708000) Darwin/9.5.1 (i386) ]
> C-68-#000102 -> [Accept: */* ]
> C-68-#000115 -> [X-Expected-Entity-Length: 302108459 ]
> C-68-#000152 -> [If: (<3ef131eb-eb74-47a4-ae97-a72d94ee6ac9-1>) ]
> C-68-#000200 -> [Authorization: Basic YWRtaW46YWRtaW4= ]
> C-68-#000239 -> [Connection: close ]
> C-68-#000258 -> [Host: localhost:1234 ]
> C-68-#000280 -> [Transfer-Encoding: Chunked ]
> C-68-#000308 -> [ ]
> C-68-#000310 -> [800 ]

(see 
<https://issues.apache.org/jira/browse/JCR-1837?focusedCommentId=12642918#action_12642918>)

BR, Julian

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 28 Oct 2008 08:03:44 +0100

> > C-68-#000280 -> [Transfer-Encoding: Chunked ]

Oh, cool. A client daring to use chunked encoding in requests!

Regards
Henrik

---

From: Daniel Stenberg <[email-masked]>
Date: Tue, 28 Oct 2008 09:29:37 +0100 (CET)

> Oh, cool. A client daring to use chunked encoding in requests!

Is that really so unusual? The "HTTP Implementations" spreadsheet lists at 
least 4 clients supporting it, but it seems the info is lacking for most of 
them:

http://spreadsheets.google.com/pub?key=pAZLaupBqGqXrtsaexTjPYg&gid=0

-- 

  / daniel.haxx.se

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 28 Oct 2008 20:23:06 +0100

> http://spreadsheets.google.com/pub?key=pAZLaupBqGqXrtsaexTjPYg&gid=0

Supporting and using is slightly different things.. Most of the Yes:es
is on HTTP libraries where I would be a bit disappointed if this wasn't
supported, not user agents. But on the other hand there is no telling of
how many applications ontop of those libraries use the feature. But this
is the first reference I have seen to chunked encoding requests "out in
the wild". And it makes me happy that someone do find it useful despite
all negative talk about how broken the current state is. As you probably
remember I consider chunked encoding of requests as one way of solving
some of the problems we see today with connection management in response
to errors such as authentication required.

Regards
Henrik