Subject: Standardizing Firefox's Implementation of Link Fingerprints

From: "Edward Lee" <[email-masked]>
Date: Mon, 2 Jul 2007 16:21:16 -0700

For Firefox 3, there are patches [1] that implement Link Fingerprints,
which provide automatic resource verification for URIs that look like
http://site.com/file#hash(sha256:abc123) so that link providers can be
sure that end users download the exact file that the provider intended
(and not a trojaned download).

The fragment identifier portion of the URI is used for backwards
compatibility with existing clients while allowing for extended usage
across protocols (e.g., http, ftp) and resource contexts (e.g., a
href, img src). Additionally, fragment identifiers are not sent as
part of a HTTP request, so the network and servers do not need to be
changed. With the backwards compatibility, incremental deployment is
feasible with some clients supporting Link Fingerprints, and end users
don't need to do anything unless there's a fingerprint failure.

This is not the same as the Content-MD5 header because that hash is
generated by the server providing the file, while the Link Fingerprint
would be provided by the link provider. This is especially useful for
mirroring files, so that the centralized site providing links can be
kept more secure while less secure mirrors can host the file.

An initial draft to standardize Link Fingerprints is available online..

https://people.mozilla.com/~edilee/draft-lee-uri-linkfingerprints-00.txt

Feedback is welcome about the design, syntax, supported hashes,
failure cases, etc.

Ed

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=377245

---

From: Jeffrey Mogul <[email-masked]>
Date: Mon, 02 Jul 2007 16:27:14 -0700

You may want to look at:

   http://www.w3.org/TR/NOTE-drp-19970825

from almost 10 years ago, which (I think) had some similar goals
(but it tried to do a lot of other things, too.)

FYI, I had no involvement with the DRP design; I'm not the
right person to ask questions about it.

-Jeff

---

From: Eric Lawrence <[email-masked]>
Date: Mon, 2 Jul 2007 17:51:43 -0700

Interesting stuff, a few thoughts...

- RFC 2396 cited in the draft was obsoleted by RFC 3986.

- The definition of this behavior across "all" media types is a significant departure from the XPointer syntax registered for XML fragments, and does present some level of compatibility concern.

- The draft may benefit from a more explicit statement of the algorithm (for at least HTTP) that specifies the steps taken in the process of evaluation of the hash.  For HTTP, I'd imagine that this includes removing any HTTP Compression (gzip/deflate) and Transfer-Encoding, before hashing the resulting body?

- Does this mechanism permit use of the "hash fragment" alongside the Media-Type specific fragment (that is to say, can I have a hyperlink which contains both a Hash, and a HTML anchor ID)?

- What is the expected behavior of a user-agent when it follows a redirection that returns, for instance, a 404 page?  The hash will obviously not be valid at that point.

- In order for the hashing algorithm to be effective, the entire body must first be retrieved, correct?  This could significantly change the behavior of a user-agent rendering the response.

Thanks,

Eric Lawrence
Program Manager
Internet Explorer

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Edward Lee
Sent: Monday, July 02, 2007 4:21 PM
To: [email-masked]
Subject: Standardizing Firefox's Implementation of Link Fingerprints

Ed

---

From: "Roy T. Fielding" <[email-masked]>
Date: Mon, 2 Jul 2007 18:04:22 -0700

> (and not a trojaned download).

Identifiers should not be abused in this way.  Adding metadata to a URI
that is orthogonal to its identifying purpose duplicates the space of
references and splits the power of the resulting resources.  The same
task can be accomplished better by specifying the hash in an attribute
of the link/anchor instead, and deploying that is far less likely to
confuse existing clients.

....Roy

---

From: "Edward Lee" <[email-masked]>
Date: Mon, 2 Jul 2007 18:32:05 -0700

> - The definition of this behavior across "all" media types is a significant departure from the XPointer syntax registered for XML fragments, and does present some level of compatibility concern.
Are you referring to #hash() potentially taking up the fragment
identifier space of a MIME-type? I.e., a MIME-type might want to
define its own fragment identifier starting with "hash("? Hopefully
this won't be an issue in the future if other fragment identifiers
follow similar syntax as xpointer() and hash(). But currently the only
"major" conflict is with text/html, which will have user-agents trying
to jump to a "hash()" identifier (not that xhtml 'id's allow parens
and colons).

> - Does this mechanism permit use of the "hash fragment" alongside the Media-Type specific fragment (that is to say, can I have a hyperlink which contains both a Hash, and a HTML anchor ID)?
There is potential for conflict with the fragment identifier in that
it has to be either Link Fingerprints or the MIME-type specific
fragment identifier. Currently, the main uses of the fragment
identifier is for html, xml, pdf, (and potentially text/plain [1]).
The main use case of Link Fingerprints is for checking content that is
not displayed by the web browser, i.e., file downloads. However, this
edge case can occur, and the link provider would need to make a
decision to use the original fragment identifier or Link Fingerprints.

There were previous discussions when designing Link Fingerprints about
allowing multiple fragment identifiers in a URI, but that would either
require changing the URI Generic Syntax or defining the "multiple
fragment identifier" syntax for "all media types" as well.

> - The draft may benefit from a more explicit statement of the algorithm (for at least HTTP) that specifies the steps taken in the process of evaluation of the hash.  For HTTP, I'd imagine that this…
Right. The bits used to compute the hash would be the bits just before
being made available to the user. The implementation for Firefox
basically uses the same gzip->uncompressed interface to "convert" the
Link Fingerprinted data on-the-fly (because hashes can be computed
incrementally) and return an error code at the end of data transfer if
the fingerprint does not match. The Link Fingerprints converter had to
be placed correctly after the decompressing and before the consumer of
the data.

> - What is the expected behavior of a user-agent when it follows a redirection that returns, for instance, a 404 page?  The hash will obviously not be valid at that point.
The 404 page would be treated as if it was the result of the download,
so the user-agent would indicate that there was a Link Fingerprint
failure. (Unless the link provider wanted to put a Link Fingerprint
for the 404 page..)

> - In order for the hashing algorithm to be effective, the entire body must first be retrieved, correct?  This could significantly change the behavior of a user-agent rendering the response.
Right. Again, because this was designed for file downloads, Firefox's
network layer will make the Link Fingerprinted data available to the
higher level consumers (e.g., download manager or page displaying
engine) so that progress bars are updated correctly and pages can
incrementally display. If some content is to be displayed in the
user-agent and not saved to a file, it would need to retract the data
already shown (e.g., an image that partially loaded would instead show
a broken image). An open issue with the current implementation is that
scripts for a HTML page could redirect away before the original page
finishes loading to compute the hash.

Ed

[1] http://www.ietf.org/internet-drafts/draft-wilde-text-fragment-06.txt

---

From: "Edward Lee" <[email-masked]>
Date: Mon, 2 Jul 2007 18:37:43 -0700

On 7/2/07, Roy T. Fielding <[email-masked]> wrote:
> The same task can be accomplished better by specifying the hash
> in an attribute of the link/anchor instead
Link Fingerprints allow for fingerprint checks for all resource
requests and not just links from anchor tags. For example, embedded
content like images or objects or scripts can be checked as well.

Internally, Link Fingerprints can be used by applications requesting
resources not necessarily in the context of HTML such as downloading
patches or updates without needing to re-implement this verification
step at a higher-than-network level.

> deploying that is far less likely to confuse existing clients.
There is the potential for clients getting confused by a string it
doesn't understand, but in those cases, the client would likely do
nothing - as if there was no fragment identifier used for the URI.

Ed

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 3 Jul 2007 11:59:09 +1000

+1; standardising the format of metadata in a URI is bad practice.  
Yes, making the hash external to the link means you need some way to  
carry it -- such as another attribute -- but link metadata is a  
problem in other domains too, and piling it all into the URL is going  
to end up as a mess.

Doing it this way makes the Web much more brittle, and just pushes  
the problem one step along -- if the links are compromised, the same  
risks are present.

Please don't do this.

--
Mark Nottingham       [email-masked]

---

From: "Larry Masinter" <[email-masked]>
Date: Tue, 3 Jul 2007 01:15:39 -0700

The most obvious complaint is that you are usurping the
fragment identifier syntax of URIs for a purpose other than
what was intended for it.  I believe many applications of URIs
will strip the fragment identifier off at a relatively high level
in the application stack. Conforming fragment requires determining
the MIME type and interpreting the fragment identifier according
to the MIME type of the result. http://site.com/file#hash is
defined to determine the meaning of 'hash' by the MIME type
of the result of accessing http://site.com/file .

You could, instead, define a new URI scheme, e.g.,

hashcheck:sha256:abc123:http://site.com/file

hashcheck:<hashscheme>:<hashvalue>:originalURI

which says 'retrieve data from originalURI but reject it if it doesn't
hash to <hashvalue>. This would be an extension with its own
clear meaning, rather than trying to redirect something that is
already defined.

Even then, you might have attacks which replace the
content-type or other headers in a result. Are you only
hashing the content body or the entire HTTP result message?
It sounds harder to create a Trojan just by supplying a different
header, but there might be some circumstances where that would
be the case.

I'm not clear what the threat is that this mechanism is
blocking. If someone wants to insert a Trojan, how is it
they can do that but not also modify the hash to match
the malicious content?

---

From: "Mark Baker" <[email-masked]>
Date: Tue, 3 Jul 2007 09:32:52 -0400

> confuse existing clients.

Exactly my thoughts.  It might look like this;

<a href="http://site.com/file" hash="sha256:abc123">the file</a>

Mark.
-- 
Mark Baker.  Ottawa, Ontario, CANADA.         http://www.markbaker.ca
Coactus; Web-inspired integration strategies  http://www.coactus.com

---

From: "Joris Dobbelsteen" <[email-masked]>
Date: Tue, 3 Jul 2007 16:19:49 +0200

>From: [email-masked] 
>[mailto:[email-masked]] On Behalf Of Larry Masinter
>Sent: dinsdag 3 juli 2007 10:16
>To: 'Edward Lee'
>Cc: [email-masked]
>Subject: RE: Standardizing Firefox's Implementation of Link 
>Fingerprints
>
>
>what was intended for it.

The most strongest argument against usage.
Exactly as Roy T. Fielding already pointed out.

[snip]
>It sounds harder to create a Trojan just by supplying a 
>different header, but there might be some circumstances where 
>that would be the case.
>
>blocking. If someone wants to insert a Trojan, how is it they 
>can do that but not also modify the hash to match the 
>malicious content?

Agreed,

In fact, RFCs are required to have "Security Consideration", which it
has. However this section is saying things regarding 'buffer overflow'
for unobvious reasons (it's a generic problem, not specific to this
RFC). Yet it even fails, IMHO, to address any possible problems with the
scheme (which there are sufficient). In fact, it might miss many of the
practical problems that are presently true for the suggested uses (as
mirrors with immediate pages like cnet and download.com have).

Taking these considerations in general, I do not see how the scheme
provides any added protection in general. A much better way seems the
'signed code' packages, like is currently done with Windows executables
(AuthenticiCode, .NET code signing) and some Linux distrubutions
(package signing with PGP as done with current versions of APT package
system).

As I see it the scheme breaks more standards and fails to solve the
practical problems. In fact, the scheme might also give a completely
false sense of security.

- Joris Dobbelsteen

---

From: Travis Snoozy <[email-masked]>
Date: Tue, 3 Jul 2007 07:31:56 -0700

On Tue, 3 Jul 2007 09:32:52 -0400, "Mark Baker" <[email-masked]> wrote:

> > > For Firefox 3, there are patches [1] that implement Link
> > > Fingerprints, which provide automatic resource verification for
> > > URIs that look like http://site.com/file#hash(sha256:abc123) so
> > > that link providers can be sure that end users download the exact
> > > file that the provider intended (and not a trojaned download).
> >
> > Identifiers should not be abused in this way.  Adding metadata to a
> > URI that is orthogonal to its identifying purpose duplicates the
> > space of references and splits the power of the resulting
> > resources.  The same task can be accomplished better by specifying
> > the hash in an attribute of the link/anchor instead, and deploying
> > that is far less likely to confuse existing clients.
> 

You're missing the point, insofar as the URL no longer has the
requisite data -- the link can only be made via HTTP, so that (e.g.) I
couldn't right-click copy-link and paste it into another application.

That said, URLs are already _way_ to long within their regular problem
space. We wouldn't need services like tinyurl if that weren't the case
-- making links even longer is counterproductive if there is any goal
to keep the darn things human readable.

As to why it's bad to try and add a feature to the URL syntax in this
way -- see the metalink discussion that rattled through the list back
in April[1]. Now, imagine that syntax, plus the hashing syntax. How
would you make them work together? Could they? Would *you* want to look
at a link like that?

Really, what it comes down to, is we need a full file format for richly
describing links. And, in fact, we have such a format -- RDF. The
entire point of RDF is to say "Hey, see this resource? Here's a whole
bunch of information about it." The point of the URL, on the other
hand, is to provide a handle to get a hold of something -- preferably in
such a way that the URL has some semi-discernible human-readable
meaning to it -- or, barring that, is at least _short_.
"http://example.com/2007/07/03/What-I-Had-For-Breakfast/"
is much nicer than
"http://example.com/19863"
is much nicer than
"http://example.com/1875,253,098676/ba2fcc201de92c8b.html?arb09="<...>

Yes, if metadata is put into a file, it's not part of the URL anymore.
However, URLs are just bad, bad, bad ways to shuffle _lots_ of data
around. Ideally, URLs that correlate to a document with complicated
extra-URL information would actually point at a metadata file. This is
how, e.g., ASX works: you get a link to the file, then your media
player opens that file, sees a few more links and follows them. The
process can repeat, and metadata is thrown in all along the way. It's a
little extra work for humans who want to chase down the "last hop" link,
but it's still all human-readable.

-- 
Travis

[1]http://lists.w3.org/Archives/Public/ietf-http-wg/2007AprJun/0042.html

---

From: "Edward Lee" <[email-masked]>
Date: Tue, 3 Jul 2007 12:01:08 -0700

On 7/2/07, Mark Nottingham <[email-masked]> wrote:
> standardizing the format of metadata in a URI is bad practice.
> Doing it this way makes the Web much more brittle

That seems valid, but I'm not entirely sure where the cause for
concern originated. Are there example uses of fragment identifiers
that started to show signs of these potential problems? Currently, it
seems like the majority of fragment identifier usage is for text/html,
and there has not been much spreading usage of fragment identifiers
elsewhere.

Ed

---

From: "Edward Lee" <[email-masked]>
Date: Tue, 10 Jul 2007 14:49:54 -0700

On 7/2/07, Roy T. Fielding <[email-masked]> wrote:
> Adding metadata to a URI that is orthogonal to its identifying purpose
> duplicates the space of references and splits the power of the resulting
> resources.

Link Fingerprints are used to identify a particular version of a
resource; if the Link Fingerprint does not match up, it's as if the
resource was not found. In that sense, it strengthens the URI's
ability to identify resources.

Ed

---

From: "Edward Lee" <[email-masked]>
Date: Tue, 10 Jul 2007 14:58:09 -0700

> hashcheck:sha256:abc123:http://site.com/file

One main advantage of the proposed Link Fingerprints syntax is that it
is backwards compatible and abides by the URI Generic Syntax. This
allows user agents that don't recognize the Link Fingerprints to
continue to retrieve resources correctly (they just remove the
#fragment-id when requesting). This also satisfies the common case
where Link Fingerprints do not result in a mismatch.

Ed

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 10 Jul 2007 15:13:41 -0700

> ability to identify resources.

No, it does not.  A "resource" is not what is sent to the client in
response to a retrieval request.  Fragments are an extra level of
indirection that refers to a different resource defined by the
representation returned by the resource.  The only component in
the request chain that is even aware of the fragment is the UA.

This whole feature is completely unnecessary.  If you want the
resource to be named as a static representation, then use the hash
in its real identifier -- the URI path.  That way, both the server
and the client can verify that the representation fits the hash
prior to its delivery and the origin server can set the appropriate
metadata for cache-control, content-md5, etc.

....Roy

---

From: "Edward Lee" <[email-masked]>
Date: Tue, 10 Jul 2007 17:09:18 -0700

On 7/10/07, Roy T. Fielding <[email-masked]> wrote:
> If you want the resource to be named as a static representation,
> then use the hash in its real identifier -- the URI path.  That way,
> both the server and the client can verify that the representation
> fits the hash prior to its delivery and the origin server can set
> the appropriate metadata for cache-control, content-md5, etc.
There's actually 3 players in the picture: server, client, and link
provider. For example, mozilla.com can link to mirror.net for an end
user to download a file. If the hash was contained in the URI path, a
link provider could not make sure the client gets the intended file if
the file on the server is not under the link provider's control. Also
important to note, the link provider can enforce stronger security and
be trusted while safely linking to external resources.

As you pointed out earlier..
> The only component in the request chain that is even aware of
> the fragment is the UA.
This means the servers and networks don't need to be modified to
support Link Fingerprints - in fact, they don't even know if a request
uses Link Fingerprints. Modifying the servers to include a hash in the
URI path requires deploying new software on both ends - a task that
requires more resources than implementing Link Fingerprints in some
browsers or download managers.

Link Fingerprints is bar-raising exercise that requires little/no
effort from the parties involved. Servers don't change. Links have an
additional #hash(). And in the common case of no failure, the end user
sees nothing different from before, while existing clients function
normally.

Ed

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 11 Jul 2007 10:23:31 +1000

Just a thought -- it might be more appropriate to discuss this on the  
URI mailing list <http://lists.w3.org/Archives/Public/uri/>.

Cheers,

---

From: Travis Snoozy <[email-masked]>
Date: Tue, 10 Jul 2007 17:31:30 -0700

On Tue, 10 Jul 2007 17:09:18 -0700, "Edward Lee" <[email-masked]>
wrote:

<snip>
> sees nothing different from before, while existing clients function

There is a tool for that. It's called RDF. Failing that, there's
metalinks XML files, which from what I recall support hashes of the
content (and sections of the content). Failing even that, there's the
thing that everybody's done for forever... putting MD5/SHA hashes and
PGP signatures of the file on the linking site. URLs are not supposed to
handle unique identification, nor are they supposed to handle integrity
checks. There are other technologies that are way better suited for
that job.

Everyone always argues "Oh, but it's a really easy backwards-compatible
way to implement feature X!" when trying to cram more cruft into the
meaning of a URL. That's because URLs are very much meant to be a
flexible, extensible, generic tool. However, you have to stay within
the framework for them to stay flexible, extensible and generic -- and
add-ons like this break as soon as you try and do two at once (e.g.,
fingerprints and metalinks).

It's a one trick pony. That trick might be a really big "wow" the first
time, but when you realize that's all it can do, you kinda get bummed
out.

Pick a single standard to describe the contents of a link. Stick
with it. If there isn't one (and it sounds like that's the case),
maybe there needs to be a new WG to make one. But making a million
different standards to do the same job, and overriding existing
standards to do a job they weren't meant to do, is not in anyone's best
interest (despite excellent intentions).

-- 
Travis