Subject: [Int-area] Comments on draft-bryan-metalink-00

From: "Ian Macfarlane" <[email-masked]>
Date: Wed, 27 Aug 2008 12:33:59 +0100

A few comments regarding the draft at
http://tools.ietf.org/html/draft-bryan-metalink-00

(1) With regards to the "type "attribute of the metalink:url element
in 4.2.17.2, I think it should be made clear that this overrides any
file extension sniffing specified in 4.2.17.

(2) With regards to the metalink:url element in 4.2.17, it is not
clear if the IRI must end with ".torrent", or if the path should, e.g.
does http://example.com/file.torrent?id=1 count? What about
http://example.com/generate.php?file.torrent

(3) Also with regards to the "type "attribute of the metalink:url
element in 4.2.17.2, it's slightly inconsistent to allow both
http/https/ftp/etc as well as "bittorrent" as types, as a .torrent
file itself can be sent over any of these protocols. There is explicit
information about the protocol from the scheme in these URLs. I would
suggest "direct" (or omit altogether) for this type of file, and the
Metalink Processor should infer the protocol from the scheme used.

(4) With regards to the metalink:url element "preference" attribute in
4.2.17.1, it is not entirely clear if a Metalink Processor which can
download simultaneously should download from two locations where one
has a lower priority. A comment such as "A Metalink Processor SHOULD
do xxx" would be helpful.

(5) With regards to the "name" attribute in 4.1.3.1, instead of "Only
relative paths are allowed" I think using the formal restrictive
language of the standards process is a good idea here, e.g."The path
MUST be relative". It might also be a good idea to add that the path
MUST NOT begin with a "/" or contain "../" (and possibly ./")
[technically this should be starting with ../ or ./ or containing /..
or /. I think].

(6) Under 4.1.4 where it says "This specification assigns no
significance to the order of metalink:url elements" it might be useful
to include a reference to the "preference" attribute.

(7) With regards to verification (4.1.6.1 and 4.2.4.1) there is no
information as to how a Metalink Processor should deal with one (can
it ignore it) or deal with multiple hash types (e.g. if there is MD5
and SHA1, MUST / MAY / MUST NOT it check more than one?). Also, it
might be useful to extend metalink documents with new verification
methods before they arrive in the standard. Perhaps unknown types
could be allowed here? The same comments mostly apply to digital
signatures too (4.2.13).

(8) Formatting nit last - the use of spacing in the nesting of XML
elements is pretty inconsistent - so instead of this on page 4:

<?xml version="1.0" encoding="UTF-8" ?>
  <metalink version="3.0" xmlns="http://metalinker.org">
    <published>2008-05-15T12:23:23Z</published>
    <files>
      <file name="example.ext">
      <description>A description of the example file for download.
      </description>
      <verification>
        <hash type="md5">83b1a04f18d6782cfe0407edadac377f</hash>
        <hash type="sha1">80bc95fd391772fa61c91ed68567f0980bb45fd9
        </hash>
      </verification>
      <resources>
        <url>ftp://ftp.example.com/example.ext</url>
        <url>http://example.com/example.ext</url>
        <url>http://example.com/example.ext.torrent</url>
      </resources>
      </file>
    </files>
  </metalink>

It would be much nicer if it were nested something like this:

<?xml version="1.0" encoding="UTF-8" ?>
  <metalink version="3.0" xmlns="http://metalinker.org">
    <published>2008-05-15T12:23:23Z</published>
    <files>
      <file name="example.ext">
        <description>A description of the example file for
download.</description>
        <verification>
          <hash type="md5">83b1a04f18d6782cfe0407edadac377f</hash>
          <hash type="sha1">80bc95fd391772fa61c91ed68567f0980bb45fd9
          </hash>
        </verification>
        <resources>
          <url>ftp://ftp.example.com/example.ext</url>
          <url>http://example.com/example.ext</url>
          <url>http://example.com/example.ext.torrent</url>
        </resources>
      </file>
    </files>
  </metalink>

Kind regards

Ian Macfarlane
_______________________________________________
Int-area mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/int-area

---

From: "Anthony Bryan" <[email-masked]>
Date: Wed, 27 Aug 2008 23:39:12 -0400

Hi Ian,

Great comments, thank you so much for taking the time to examine this!
These are issues that needed to be addressed.

Do you mind if I forward this to the metalink-discussion list?

I'll put the changes here, let me know if they are an improvement, or
suggest a change.

> file extension sniffing specified in 4.2.17.

4.2.17.2.  The "type" Attribute

  metalink:url elements MAY have a "type" attribute that indicates the
  IRI type.  The "type" attribute overrides any file extension sniffing
  specified above.

> http://example.com/generate.php?file.torrent

  6.  The value "bittorrent" signifies that the IRI leads to a
      BitTorrent .torrent file as specified in [BITTORRENT].  Metalink
      Processors that do not support BitTorrent should ignore this type
      and also ignore metalink:url elements which retrieve files that
      end with the extension ".torrent".

> Metalink Processor should infer the protocol from the scheme used.

What about requiring that "bittorrent" is used as a "type" attribute
since .torrent files can be acquired from multiple methods, & just
examining the IRI as you mentioned can be misleading?

> do xxx" would be helpful.

I'm not sure what to put here :)

This sentence would accurately describe what they do now. In our
pre-ID version, we have a "maxconnections" attribute where you can
limit the amount of segments for a download.

"A Metalink Processor MAY download different segments of a file from
more than one IRI simultaneously, and when doing so SHOULD first use
the highest priority IRIs and then use lower ones."

> [technically this should be starting with ../ or ./ or containing /..

  In this example, a subdirectory debian-amd64/sarge/ will be created
  and a file named Contents-amd64.gz will be created inside it.  The
  path MUST be relative.  The path MUST NOT begin with a "/" or contain
  "../" or "./" Metalink Processors MUST NOT allow directory traversal.

> to include a reference to the "preference" attribute.

  This specification assigns no significance to the order of metalink:
  url elements.  Significance is determines by the value of the
  "preference" attribute of the metalink:url elements.

> could be allowed here? The same comments mostly apply to digital

I agree that unknown hash types or digital signatures should be allowed.

  This document defines nine initial values for hash types.  It may be
  useful to extend Metalink documents with new verification methods, so
  unknown types are allowed.

and

  metalink:signature elements MUST have a "type" attribute.  The inital
  value of "type" is the string that is non-empty and matches "pgp".
  It may be useful to extend Metalink documents with new types of
  digital signatures, so unknown types are allowed.

What do you suggest about dealing with multiple hash types?

An issue could be if someone malicious makes a metalink where the MD5
matches that of something published by a legit group, but also
includes a SHA-256 checksum, and if clients prefer & only verify
SHA-256, then the file could appear to be good even if the downloader
looked inside the metalink & compared the MD5 (if the legit group
didn't also use & publish SHA-256 checksums too).
- Show quoted text -

>          <url>http://example.com/example.ext.torrent</url>

Done, if:

is the only thing changed?

The updated ID is draft-bryan-metalink-01pre.txt

http://groups.google.com/group/metalink-discussion/files

http://www.metalinker.org/ID/draft-bryan-metalink-01pre.txt

-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
 )) Easier, More Reliable, Self Healing Downloads
_______________________________________________
Int-area mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/int-area

---

From: "Anthony Bryan" <[email-masked]>
Date: Thu, 28 Aug 2008 16:44:18 -0400

On Thu, Aug 28, 2008 at 7:14 AM, Ian Macfarlane <[email-masked]> wrote:
> Hi Anthony,
>
> Thanks for your reply. A few comments about these changes:

Thanks again for the patience & taking the time to help out.

I'm keeping up to date versions at
http://metalinks.svn.sourceforge.net/viewvc/metalinks/internetdraft/draft-bryan-metalink-01.txt?view=markup

> (1) With regards to this new wording:
>
> "  6.  The value "bittorrent" signifies that the IRI leads to a
>     end with the extension ".torrent"."
>
> This implies that the file extension still overrides the type
> attribute even if the type is not "bittorrent" - I might suggest
> adding to the end:
>
> ", unless the metalink:url element has a type attribute which the
> Metalink Processor supports".
>
> It's definitely a real corner case, but it's good to specify the
> correct behavior for future proofing (what if a new file format comes
> out called "bittorrent2" which extends bittorrent and uses .torrent
> files, but which existing "bittorrent1" processors can't handle?)

Yes, better to be clear. I've added it.

> Also my original point regarding the location of the ".torrent" text
> in the IRI isn't dealt with by this new text - I would suggest
> explicitly stating that this means when the IRI path ends with the
> characters ".torrent" (or alternatively, as you suggest, require the
> "type" attribute for bittorrent).

I think it's good to require the "type" attribute for bittorrent so
I've changed it. (In practice, it has always been used). This way, as
you say, FTP or HTTP etc IRIs that don't obviously lead to a torrent
can be disregarded by Metalink processors, even tho they would be most
likely by not having a matching file size or the correct hash as the
other files.

> (2):
>
> "What about requiring that "bittorrent" is used as a "type" attribute
> examining the IRI as you mentioned can be misleading?"
>
> The difference between the ".torrent" naming issue and the "http://"
> naming issue is that the first is simply part of the path, and doesn't
> really mean anything (there's no reason you couldn't serve a web page
> with a .torrent extension, if you have the right Content-Type).

I didn't think of that.

> However, for "http://" etc, this is the IRI's scheme itself, which has
> an explicit unalterable meaning. They're really two very different
> things.
>
> (technically ed2k/magnet/rsync URIs don't need a type either, as the
> scheme provides the required information - it's only the BitTorrent
> protocol which is different as there is not a 'torrent' URI scheme per
> se).
>
> No strong objection either way to requiring "type" for "bittorrent",
> so long as any explicit "type" attribute specified overrides any file
> type "sniffing", but I'm slightly in favour of requiring the type
> attribute for where it can't be inferred from the scheme and dropping
> sniffing altogether.

Ok, "type" attribute for bittorrent is required.

> (3):
> I agree that this is a difficult one. Some possible suggestions:
>
> - When one or more resources have a value of "100", no other resources
> should be used, unless these cannot be processed (e.g. are bittorrent
> etc and this is not supported, or the servers are down).
>
> - Any resources with a value of "1" should not be used unless all
> other resources cannot be processed (e.g. are bittorrent etc and this
> is not supported, or the servers are down).
>
> I think at least those two are valuable enough to include (probably a SHOULD).

   metalink:url elements MAY have a preference attribute, whose value
   MUST be a number from 1 to 100 for priority, with 100 used first and
   1 used last.  Multiple metalink:url elements can have the same
   preference, i.e. ten mirrors could have preference="100".  A Metalink
   Processor MAY download different segments of a file from more than
   one IRI simultaneously, and when doing so SHOULD first use the
   highest priority IRIs and then use lower ones.

   When one or more metalink:url elements have a preference attribute
   value of "100", other metalink:url elements SHOULD NOT be used,
   unless these cannot be processed (e.g. are "bittorrent" etc, and this
   is not supported by the Metalink Processor, or the servers are down).

   Any metalink:url elements with a preference attribute value of "1"
   SHOULD NOT be used unless all other metalink:url elements cannot be
   processed (e.g. are "bittorrent" etc and this is not supported by the
   Metalink Processor, or the servers are down).

> Lastly, it might be possible to do something based on the 'initial
> digit', e.g. if the initial digit is higher, all servers with lower
> digits should not be used (unless the higher ones cannot be
> processed), and the others should have their work distributed evenly
> based on the minor digit. For example if you have three resources with
> preferences of 89, 91 and 95 - the one with 89 would not be used
> (unless the other two can't be used), and the processor would try and
> distribute more work to the resource with a value of 95 than the one
> with 91 (e.g. 5 times more, or something along those lines - or you
> could leave the exact distribution down to the metalink processor). I
> think this sort of behavior could be no stronger than a SHOULD though.

This could be interesting, I want to consult the authors of Metalink
clients first though.

> (4):
>
> " In this example, a subdirectory debian-amd64/sarge/ will be created
>  "../" or "./" Metalink Processors MUST NOT allow directory traversal."
>
> I think the actual correct form for this should be:
>
> " In this example, a subdirectory debian-amd64/sarge/ will be created
>  path MUST be relative.  The path MUST NOT begin with a "/", "./" or
> "../", contain
>  "/../", or end with "/..". Metalink Processors MUST NOT allow
> directory traversal."
>
> (./ at the start could cause some badly written applications to change
> to their current directory, but /./ anywhere else should be fine I
> think).
>
> I think it would be good if you could get a second opinion on this
> wording from someone who knows this a bit better than I.

I've fixed it & hopefully we'll have corrections :)

> (5) It might also be worth adding information as to how to deal with
> characters which are invalid in the filesystem - I'd suggest something
> like:
>
> "A Metalink Processor MAY alter the name of the subdirectory or file
> if they contain characters which are invalid in the destination
> filesystem."
>
> (that way it can be left to the processor itself to decide what to
> rename it to on any particular filesystem, or even reject it if
> desired).

That sounds good, added.

> (6) "What do you suggest about dealing with multiple hash types?" -
> obviously it would be better for a processor to check multiple hashes,
> as it's a good way to prevent malicious altering of the files. This
> needs to be left down to the metalink processor though. Something
> like:
>
> "When multiple hash types methods are provided, a Metalink Processor
> MAY verify using more than one of these hash types".

Added. Currently, I think most only do one.

> Also you write:
>
> "An issue could be if someone malicious makes a metalink where the MD5
> didn't also use & publish SHA-256 checksums too)."
>
> That's an interesting case, but if their metalink processor didn't
> support md5 and only supported SHA-256, it'd be the same scenario too.
> I don't think this is too much of a concern. It's not unreasonable to
> change the "MAY" suggested above for checking multiple hash types to a
> "SHOULD", but for reasons such as performance, this might not always
> be desirable, and I'm not sure you'd gain as much security as you
> might think from doing this. Most people don't even know what md5 etc
> are, and a user who cares about this will probably make sure their
> metalink processor is one which checks all hashes.
>
> It's quite in keeping with standards to write something like this:
>
> "Metalink processors are encouraged to check all hash types given
> which they are able to process"
>
> This will probably lead to checking all as the common behavior while
> not preventing people choosing not to do this for whatever reason.

Yes, I think it ultimately comes down to downloading .metalinks from
someone you trust, just like any other type of download.

> (7): "is the only thing changed?" - also the indentation changed (e.g.
> <description> should be indented one more level, as it's a child of
> <file>).

Hopefully I've got this right now, let me know.

> (8) An additional point I've just noticed - it specifies that it
> should ignore resources with a "type" of bittorrent if it is
> unsupported. ("Metalink Processors that do not support BitTorrent MUST
> ignore" ...) [as changed in your additional email from should]. This
> should probably be removed from the bittorrent subsection (point 6 in
> 4.2.17.2), and moved to above the list in 4.2.17.2, and state
> something like "Metalink Processors that do not support a specified
> type of resource MUST ignore that resource". This is both future
> proof, and handles the case of Metalink Processors not supporting ed2k
> etc.

Good point.

> Best wishes
>
> Ian Macfarlane
>
> ps: No objections to forwarding any of this to the metalink list.

Thanks!

> 2008/8/28 Anthony Bryan <[email-masked]>:
>> Hi Ian,
>>