Subject: Re: proposal for issue #178

From: Yves Lafon <[email-masked]>
Date: Thu, 3 Jun 2010 13:08:29 -0400 (EDT)

On Mon, 17 May 2010, Henrik Nordström wrote:

> mån 2010-05-17 klockan 07:37 -0400 skrev Yves Lafon:
>> On Sun, 9 May 2010, Henrik Nordström wrote:
>>
>>> ons 2010-03-24 klockan 20:47 -0400 skrev Yves Lafon:
>>>
>>>>     If the 206 response is the result of an If-Range request, the
>>>>     response SHOULD NOT include other entity-headers.  Otherwise, the
>>>>     response MUST include all of the entity-headers that would have been
>>>>     returned with a 200 (OK) response to the same request, except for
>>>>     Content-MD5 that MUST NOT be present.
>>>
>>> For compatibility reasons this can not be a MUST NOT, only a SHOULD NOT.
>>>
>>>>     Recipients MUST ignore Content-MD5 on 206 responses; Caches MUST strip
>>>>     Content-MD5 when caching or combining 206 responses with existing
>>>>     cached content.
>>>
>>> Same here.
>>
>> Issue with a SHOULD NOT is that Content-MD5 may be present, and in that
>> case we have a real interoperability issue. It is one of the few areas
>> where we need to break compatibility (but as not that many servers are
>> sending Content-MD5, and even less for Content-MD5 for 206 responses, it
>> is quite unlikely that it will break implementations)
>
>
> I am OK with the client side SHOULD NOT requirement. As you say the
> likelyhood of breaking any existing implementations is very slim there.
>
> But not sure about the server side. For one thing there is is existing
> server implementations of both alternatives for how to generate
> Content-MD5 in partial responses, MD5(partial entity-body) or MD5(full
> entity-body). Also having this requirement on servers complicates server
> implementations even more.
>
> If we are to break server implementations then I would very much prefer
> the specs clarify which of the two should be used, and my preference
> there is clearly for MD5(full entity-body).
>
> Is there any other known servers than older Jigsaw versions which
> implements Content-MD5 as MD5(partial entity-body)?
>
> I ask this because I see an alternative resolution to this problem by
> defining Content-MD5 as MD5(full entity-body), and instead of ignoring
> Content-MD5 in partial responses requiring clients to NOT merge 206
> responses with conflicting Content-MD5, and instead if needed retry the
> request without the Range specifier.

I agree that those are the two major options, forbid the use of C-MD5 in 
206, or explicitely say that it is only about the full content, but will 
defeat the purpose of acting as a message integrity check...

Forbidding its use is the safe side, as there are known implementation 
issues (but not widely visible, as C-MD5 is not used that much anyway).

> But I do not really have a strong opinion on this. Any change is to the
> better, including moving Content-MD5 as a whole to historic noting that
> it SHOULD NOT be used by either servers or clients in this
> specification. There exists a standards track alternative to Content-MD5
> ready to replace it, it's not used much, and have conflicting
> implementations.
>
> Regards
> Henrik
>
>

-- 
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 7 Jun 2010 15:24:26 +0200 (CEST)

On Tue, 8 Jun 2010, Adrien de Croy wrote:

>> In the end, you might sit there with a huge file with a bad checksum 
>> without being able to pinpoint the single small chunk that had the error. 
>> So now you need to redownload the whole thing again, instead of just 
>> regetting the small chunk that contained the error.
>
> I didn't think that was really supported by HTTP, since you can't know 
> without some meta information that parts from different servers belong to 
> the same entity.

Well, that meta information can still exist. See RFC5854 for inspiration!

> in which case it's an extension, in which case may as well add a new header 
> to explicitly cover the part entity.  E.g. Content-Part-MD5 or something. 
> And leave Content-MD5 as it is (could use both).

Yes it can, I was not really advocating for any particular header. I was 
merely suggesting that there are use-cases where getting a checksum on a range 
can be useful.

-- 

  / daniel.haxx.se

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 08 Jun 2010 00:17:45 +1200

On 4/06/2010 5:08 a.m., Yves Lafon wrote:
> I agree that those are the two major options, forbid the use of C-MD5 
> in 206, or explicitely say that it is only about the full content, but 
> will defeat the purpose of acting as a message integrity check...

I don't see any point in having an integrity check for a message 
containing only a partial range.  Surely you want to accumulate the 
entire entity by piecing together all the parts, and then you use the 
MD5 to check the total.

On that note I don't see any point in range extensions either, other 
than to make the job of intermediaries impossible.

Making the MD5 based on the entire entity gives a very simple test for a 
receiver.  If it gets any 206 response with a different C-MD5, it knows 
it's not a part of the same entity.  Similar to an ETag but with the 
benefit of being able to be used to verify integrity of the entire entity.

So I can see a clear value in making it based on entire content, and I 
can't see any value in making it the MD5 of the part.

Therefore it seems unnecessary to forbid it as an alternative to simply 
clarifying it.

Regards

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 08 Jun 2010 01:33:51 +1200

when serving a part of an entity, having to provide an MD5 for that 
part, the server would just calculate it at the time it served it, in 
which case if the file is corrupt, the MD5 won't help you.

Otherwise the server would have to pre-calculate a bunch of MD5s of 
different part sizes and different parts at the time it knew the parts 
were not corrupted...

In which case the server could just check the MD5 itself, and return an 
error if it couldn't serve an uncorrupted file... so this leans towards 
having the MD5 on the total entity.

Adrien

On 8/06/2010 1:24 a.m., Daniel Stenberg wrote:
>>> without being able to pinpoint the single small chunk that had the 
>>> error. So now you need to redownload the whole thing again, instead 
>>> of just regetting the small chunk that contained the error.
>>
>> I didn't think that was really supported by HTTP, since you can't 
>> know without some meta information that parts from different servers 
>> belong to the same entity.
>
>> in which case it's an extension, in which case may as well add a new 
>> header to explicitly cover the part entity.  E.g. Content-Part-MD5 or 
>> something. And leave Content-MD5 as it is (could use both).
>
> Yes it can, I was not really advocating for any particular header. I 
> was merely suggesting that there are use-cases where getting a 
> checksum on a range can be useful.
>

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 08 Jun 2010 01:20:03 +1200

On 8/06/2010 1:07 a.m., Daniel Stenberg wrote:
> Imagine downloading a large file in many small chunks from many 
> different servers.
>
> error. So now you need to redownload the whole thing again, instead of 
> just regetting the small chunk that contained the error.
>

I didn't think that was really supported by HTTP, since you can't know 
without some meta information that parts from different servers belong 
to the same entity.

However isn't this just doing the same job as TCP checksums?  Or are you 
trying to code against corrupt parts coming from a server?

Forgive me but I don't see the point in getting different bits from 
different servers either.

If load is that high on a system that you need to deploy multiple 
servers to satisfy it, you'd be better multiplexing by client, rather 
than by part of an entity.  E.g. spread the load by spreading the 
clients around, rather than spreading entity parts around.

?

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 7 Jun 2010 15:07:41 +0200 (CEST)

On Tue, 8 Jun 2010, Adrien de Croy wrote:

> I don't see any point in having an integrity check for a message containing 
> only a partial range.  Surely you want to accumulate the entire entity by 
> piecing together all the parts, and then you use the MD5 to check the total.

Imagine downloading a large file in many small chunks from many different 
servers.

In the end, you might sit there with a huge file with a bad checksum without 
being able to pinpoint the single small chunk that had the error. So now you 
need to redownload the whole thing again, instead of just regetting the small 
chunk that contained the error.

-- 

  / daniel.haxx.se

---

From: Henrik Nordström <[email-masked]>
Date: Mon, 07 Jun 2010 16:13:37 +0200

> entire entity by piecing together all the parts, and then you use the 

My view also, but others have read it differently in past, and the spec
is ambigious on which reading is right with a bit blurred definition of
206, response entity, full response entiry and partial response entity.

But as far as I know there has been only one actual server
implementation using the Content-MD5 on the partial response entity and
it's no longer sending Content-MD5 in 206 responses.  Unknown if there
is any client implementation using it, or even other server
implementations making use of Content-MD5 at all.

> than to make the job of intermediaries impossible.

Do you refer to the non-linear data discussion some many months ago
here?

Regards
Henrik

---

From: David Morris <[email-masked]>
Date: Mon, 7 Jun 2010 08:05:12 -0700 (PDT)

> chunk that contained the error.

Fact of the matter is that the probablitly of getting one chunk in error 
is very low in comparison with the origin servers being out of sync being 
the reason for the error. Without an MD5 for the whole entity there is no
was to check the complete entity for correctness.

It should have been clear from the beginning that the MD5 applied to the
whole entity.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 07 Jun 2010 17:35:14 +0200

On 07.06.2010 17:05, David Morris wrote:
> ...
> ...

Hmmm. Etag?

---

From: David Morris <[email-masked]>
Date: Mon, 7 Jun 2010 09:10:05 -0700 (PDT)

Etag proves nothing if the servers are out of sync data wise but the 
Etags agree for what ever reason. Which I consider much higher probability
than corruption in a single segment response. An MD5 or other 'checksum' 
of similar or better strength when applied to the reconstructed entity 
will 'prove' it to be completely received and reconstructed correctly.

Etag only indicates that the servers believe they are serving the same
binary bits, not that they did, or the the reconstruction was bug free,
etc.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Mon, 7 Jun 2010 13:02:05 -0700

> It should have been clear from the beginning that the MD5 applied to the

No, Content-MD5 is a MIME header field for message content.  Always has been.

....Roy

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 08 Jun 2010 08:54:25 +1200

> Do you refer to the non-linear data discussion some many months ago

yes.  There was some discussion a few months back about it - even an I-D?

Regards

Adrien