Subject: Re: [Int-area] Comments on draft-bryan-metalink-00

From: "Anthony Bryan" <[email-masked]>
Date: Tue, 2 Sep 2008 20:55:59 -0400

On Tue, Sep 2, 2008 at 8:23 AM, Ian Macfarlane <[email-masked]> wrote:
> Dear Anthony,
>
> One extra alteration where I think the wording could be slightly tidied up:
>
> "When one or more metalink:url elements have a preference attribute
>  value of "100", other metalink:url elements SHOULD NOT be used,
>  unless these cannot be processed (e.g. are "bittorrent" etc, and this
>  is not supported by the Metalink Processor, or the servers are down)."
>
> Here "these" could potentially be misread as the "non-100" elements
> rather than the "100" elements. I think slightly clarifying the
> wording here would be beneficial. I suggest something along the lines
> of:
>
> "When one or more metalink:url elements have a preference attribute
> value of "100", other metalink:url elements SHOULD NOT be used, unless
> the elements with a preference of 100 cannot be processed (e.g. if
> they are of a type which is not supported by the Metalink Processor,
> such as bittorrent, or if the servers are unavailable)."

Changed.

> Also, I still think the "type" definitions such as "http", "https" etc
> should be removed, as per the reasons given in the previous emails.

It looks like you're getting your way. :)

You & others have made good points against it.

> Thank you for taking the time to look at my suggestions.

Thanks again for taking the time to make them. Your comments were the
first & very helpful!

> ps: I have not had a chance to look through the entire revised
> document - these comments are based on the revisions described in your
> email.
>
> 2008/8/28 Anthony Bryan <[email-masked]>:
>> On Thu, Aug 28, 2008 at 7:14 AM, Ian Macfarlane <[email-masked]> wrote:
>>> Hi Anthony,
>>>
>>> Thanks for your reply. A few comments about these changes:
>>
>> Thanks again for the patience & taking the time to help out.
>>
>> I'm keeping up to date versions at
>> http://metalinks.svn.sourceforge.net/viewvc/metalinks/internetdraft/draft-bryan-metalink-01.txt?view=markup
>>
>>> (1) With regards to this new wording:
>>>
>>> "  6.  The value "bittorrent" signifies that the IRI leads to a
>>>     BitTorrent .torrent file as specified in [BITTORRENT].  Metalink
>>>     Processors that do not support BitTorrent should ignore this type
>>>     and also ignore metalink:url elements which retrieve files that
>>>     end with the extension ".torrent"."
>>>
>>> This implies that the file extension still overrides the type
>>> attribute even if the type is not "bittorrent" - I might suggest
>>> adding to the end:
>>>
>>> ", unless the metalink:url element has a type attribute which the
>>> Metalink Processor supports".
>>>
>>> It's definitely a real corner case, but it's good to specify the
>>> correct behavior for future proofing (what if a new file format comes
>>> out called "bittorrent2" which extends bittorrent and uses .torrent
>>> files, but which existing "bittorrent1" processors can't handle?)
>>
>> Yes, better to be clear. I've added it.
>>
>>> Also my original point regarding the location of the ".torrent" text
>>> in the IRI isn't dealt with by this new text - I would suggest
>>> explicitly stating that this means when the IRI path ends with the
>>> characters ".torrent" (or alternatively, as you suggest, require the
>>> "type" attribute for bittorrent).
>>
>> I think it's good to require the "type" attribute for bittorrent so
>> I've changed it. (In practice, it has always been used). This way, as
>> you say, FTP or HTTP etc IRIs that don't obviously lead to a torrent
>> can be disregarded by Metalink processors, even tho they would be most
>> likely by not having a matching file size or the correct hash as the
>> other files.
>>
>>> (2):
>>>
>>> "What about requiring that "bittorrent" is used as a "type" attribute
>>> since .torrent files can be acquired from multiple methods, & just
>>> examining the IRI as you mentioned can be misleading?"
>>>
>>> The difference between the ".torrent" naming issue and the "http://"
>>> naming issue is that the first is simply part of the path, and doesn't
>>> really mean anything (there's no reason you couldn't serve a web page
>>> with a .torrent extension, if you have the right Content-Type).
>>
>> I didn't think of that.
>>
>>> However, for "http://" etc, this is the IRI's scheme itself, which has
>>> an explicit unalterable meaning. They're really two very different
>>> things.
>>>
>>> (technically ed2k/magnet/rsync URIs don't need a type either, as the
>>> scheme provides the required information - it's only the BitTorrent
>>> protocol which is different as there is not a 'torrent' URI scheme per
>>> se).
>>>
>>> No strong objection either way to requiring "type" for "bittorrent",
>>> so long as any explicit "type" attribute specified overrides any file
>>> type "sniffing", but I'm slightly in favour of requiring the type
>>> attribute for where it can't be inferred from the scheme and dropping
>>> sniffing altogether.
>>
>> Ok, "type" attribute for bittorrent is required.
>>
>>> (3):
>>>
>>> "A Metalink Processor MAY download different segments of a file from
>>> more than one IRI simultaneously, and when doing so SHOULD first use
>>> the highest priority IRIs and then use lower ones."
>>>
>>> I agree that this is a difficult one. Some possible suggestions:
>>>
>>> - When one or more resources have a value of "100", no other resources
>>> should be used, unless these cannot be processed (e.g. are bittorrent
>>> etc and this is not supported, or the servers are down).
>>>
>>> - Any resources with a value of "1" should not be used unless all
>>> other resources cannot be processed (e.g. are bittorrent etc and this
>>> is not supported, or the servers are down).
>>>
>>> I think at least those two are valuable enough to include (probably a SHOULD).
>>
>>   metalink:url elements MAY have a preference attribute, whose value
>>   MUST be a number from 1 to 100 for priority, with 100 used first and
>>   1 used last.  Multiple metalink:url elements can have the same
>>   preference, i.e. ten mirrors could have preference="100".  A Metalink
>>   Processor MAY download different segments of a file from more than
>>   one IRI simultaneously, and when doing so SHOULD first use the
>>   highest priority IRIs and then use lower ones.
>>
>>   When one or more metalink:url elements have a preference attribute
>>   value of "100", other metalink:url elements SHOULD NOT be used,
>>   unless these cannot be processed (e.g. are "bittorrent" etc, and this
>>   is not supported by the Metalink Processor, or the servers are down).
>>
>>   Any metalink:url elements with a preference attribute value of "1"
>>   SHOULD NOT be used unless all other metalink:url elements cannot be
>>   processed (e.g. are "bittorrent" etc and this is not supported by the
>>   Metalink Processor, or the servers are down).
>>
>>> Lastly, it might be possible to do something based on the 'initial
>>> digit', e.g. if the initial digit is higher, all servers with lower
>>> digits should not be used (unless the higher ones cannot be
>>> processed), and the others should have their work distributed evenly
>>> based on the minor digit. For example if you have three resources with
>>> preferences of 89, 91 and 95 - the one with 89 would not be used
>>> (unless the other two can't be used), and the processor would try and
>>> distribute more work to the resource with a value of 95 than the one
>>> with 91 (e.g. 5 times more, or something along those lines - or you
>>> could leave the exact distribution down to the metalink processor). I
>>> think this sort of behavior could be no stronger than a SHOULD though.
>>
>> This could be interesting, I want to consult the authors of Metalink
>> clients first though.
>>
>>> (4):
>>>
>>> " In this example, a subdirectory debian-amd64/sarge/ will be created
>>>  and a file named Contents-amd64.gz will be created inside it.  The
>>>  path MUST be relative.  The path MUST NOT begin with a "/" or contain
>>>  "../" or "./" Metalink Processors MUST NOT allow directory traversal."
>>>
>>> I think the actual correct form for this should be:
>>>
>>> " In this example, a subdirectory debian-amd64/sarge/ will be created
>>>  and a file named Contents-amd64.gz will be created inside it.  The
>>>  path MUST be relative.  The path MUST NOT begin with a "/", "./" or
>>> "../", contain
>>>  "/../", or end with "/..". Metalink Processors MUST NOT allow
>>> directory traversal."
>>>
>>> (./ at the start could cause some badly written applications to change
>>> to their current directory, but /./ anywhere else should be fine I
>>> think).
>>>
>>> I think it would be good if you could get a second opinion on this
>>> wording from someone who knows this a bit better than I.
>>
>> I've fixed it & hopefully we'll have corrections :)
>>
>>> (5) It might also be worth adding information as to how to deal with
>>> characters which are invalid in the filesystem - I'd suggest something
>>> like:
>>>
>>> "A Metalink Processor MAY alter the name of the subdirectory or file
>>> if they contain characters which are invalid in the destination
>>> filesystem."
>>>
>>> (that way it can be left to the processor itself to decide what to
>>> rename it to on any particular filesystem, or even reject it if
>>> desired).
>>
>> That sounds good, added.
>>
>>> (6) "What do you suggest about dealing with multiple hash types?" -
>>> obviously it would be better for a processor to check multiple hashes,
>>> as it's a good way to prevent malicious altering of the files. This
>>> needs to be left down to the metalink processor though. Something
>>> like:
>>>
>>> "When multiple hash types methods are provided, a Metalink Processor
>>> MAY verify using more than one of these hash types".
>>
>> Added. Currently, I think most only do one.
>>
>>> Also you write:
>>>
>>> "An issue could be if someone malicious makes a metalink where the MD5
>>> matches that of something published by a legit group, but also
>>> includes a SHA-256 checksum, and if clients prefer & only verify
>>> SHA-256, then the file could appear to be good even if the downloader
>>> looked inside the metalink & compared the MD5 (if the legit group
>>> didn't also use & publish SHA-256 checksums too)."
>>>
>>> That's an interesting case, but if their metalink processor didn't
>>> support md5 and only supported SHA-256, it'd be the same scenario too.
>>> I don't think this is too much of a concern. It's not unreasonable to
>>> change the "MAY" suggested above for checking multiple hash types to a
>>> "SHOULD", but for reasons such as performance, this might not always
>>> be desirable, and I'm not sure you'd gain as much security as you
>>> might think from doing this. Most people don't even know what md5 etc
>>> are, and a user who cares about this will probably make sure their
>>> metalink processor is one which checks all hashes.
>>>
>>> It's quite in keeping with standards to write something like this:
>>>
>>> "Metalink processors are encouraged to check all hash types given
>>> which they are able to process"
>>>
>>> This will probably lead to checking all as the common behavior while
>>> not preventing people choosing not to do this for whatever reason.
>>
>> Yes, I think it ultimately comes down to downloading .metalinks from
>> someone you trust, just like any other type of download.
>>
>>> (7): "is the only thing changed?" - also the indentation changed (e.g.
>>> <description> should be indented one more level, as it's a child of
>>> <file>).
>>
>> Hopefully I've got this right now, let me know.
>>
>>> (8) An additional point I've just noticed - it specifies that it
>>> should ignore resources with a "type" of bittorrent if it is
>>> unsupported. ("Metalink Processors that do not support BitTorrent MUST
>>> ignore" ...) [as changed in your additional email from should]. This
>>> should probably be removed from the bittorrent subsection (point 6 in
>>> 4.2.17.2), and moved to above the list in 4.2.17.2, and state
>>> something like "Metalink Processors that do not support a specified
>>> type of resource MUST ignore that resource". This is both future
>>> proof, and handles the case of Metalink Processors not supporting ed2k
>>> etc.
>>
>> Good point.
>>
>>> Best wishes
>>>
>>> Ian Macfarlane
>>>
>>> ps: No objections to forwarding any of this to the metalink list.
>>
>> Thanks!
>>
>>> 2008/8/28 Anthony Bryan <[email-masked]>:
>>>> Hi Ian,
>>>>
>>>> Great comments, thank you so much for taking the time to examine this!
>>>> These are issues that needed to be addressed.
>>>>
>>>> Do you mind if I forward this to the metalink-discussion list?
>>>>
>>>> I'll put the changes here, let me know if they are an improvement, or
>>>> suggest a change.
>>>>
>>>> On Wed, Aug 27, 2008 at 7:33 AM, Ian Macfarlane <[email-masked]> wrote:
>>>>> A few comments regarding the draft at
>>>>> http://tools.ietf.org/html/draft-bryan-metalink-00
>>>>>
>>>>> (1) With regards to the "type "attribute of the metalink:url element
>>>>> in 4.2.17.2, I think it should be made clear that this overrides any
>>>>> file extension sniffing specified in 4.2.17.
>>>>
>>>> 4.2.17.2.  The "type" Attribute
>>>>
>>>>  metalink:url elements MAY have a "type" attribute that indicates the
>>>>  IRI type.  The "type" attribute overrides any file extension sniffing
>>>>  specified above.
>>>>
>>>>> (2) With regards to the metalink:url element in 4.2.17, it is not
>>>>> clear if the IRI must end with ".torrent", or if the path should, e.g.
>>>>> does http://example.com/file.torrent?id=1 count? What about
>>>>> http://example.com/generate.php?file.torrent
>>>>
>>>>  6.  The value "bittorrent" signifies that the IRI leads to a
>>>>      BitTorrent .torrent file as specified in [BITTORRENT].  Metalink
>>>>      Processors that do not support BitTorrent should ignore this type
>>>>      and also ignore metalink:url elements which retrieve files that
>>>>      end with the extension ".torrent".
>>>>
>>>>> (3) Also with regards to the "type "attribute of the metalink:url
>>>>> element in 4.2.17.2, it's slightly inconsistent to allow both
>>>>> http/https/ftp/etc as well as "bittorrent" as types, as a .torrent
>>>>> file itself can be sent over any of these protocols. There is explicit
>>>>> information about the protocol from the scheme in these URLs. I would
>>>>> suggest "direct" (or omit altogether) for this type of file, and the
>>>>> Metalink Processor should infer the protocol from the scheme used.
>>>>
>>>> What about requiring that "bittorrent" is used as a "type" attribute
>>>> since .torrent files can be acquired from multiple methods, & just
>>>> examining the IRI as you mentioned can be misleading?
>>>>
>>>>> (4) With regards to the metalink:url element "preference" attribute in
>>>>> 4.2.17.1, it is not entirely clear if a Metalink Processor which can
>>>>> download simultaneously should download from two locations where one
>>>>> has a lower priority. A comment such as "A Metalink Processor SHOULD
>>>>> do xxx" would be helpful.
>>>>
>>>> I'm not sure what to put here :)
>>>>
>>>> This sentence would accurately describe what they do now. In our
>>>> pre-ID version, we have a "maxconnections" attribute where you can
>>>> limit the amount of segments for a download.
>>>>
>>>> "A Metalink Processor MAY download different segments of a file from
>>>> more than one IRI simultaneously, and when doing so SHOULD first use
>>>> the highest priority IRIs and then use lower ones."
>>>>
>>>>> (5) With regards to the "name" attribute in 4.1.3.1, instead of "Only
>>>>> relative paths are allowed" I think using the formal restrictive
>>>>> language of the standards process is a good idea here, e.g."The path
>>>>> MUST be relative". It might also be a good idea to add that the path
>>>>> MUST NOT begin with a "/" or contain "../" (and possibly ./")
>>>>> [technically this should be starting with ../ or ./ or containing /..
>>>>> or /. I think].
>>>>
>>>>  In this example, a subdirectory debian-amd64/sarge/ will be created
>>>>  and a file named Contents-amd64.gz will be created inside it.  The
>>>>  path MUST be relative.  The path MUST NOT begin with a "/" or contain
>>>>  "../" or "./" Metalink Processors MUST NOT allow directory traversal.
>>>>
>>>>> (6) Under 4.1.4 where it says "This specification assigns no
>>>>> significance to the order of metalink:url elements" it might be useful
>>>>> to include a reference to the "preference" attribute.
>>>>
>>>>  This specification assigns no significance to the order of metalink:
>>>>  url elements.  Significance is determines by the value of the
>>>>  "preference" attribute of the metalink:url elements.
>>>>
>>>>> (7) With regards to verification (4.1.6.1 and 4.2.4.1) there is no
>>>>> information as to how a Metalink Processor should deal with one (can
>>>>> it ignore it) or deal with multiple hash types (e.g. if there is MD5
>>>>> and SHA1, MUST / MAY / MUST NOT it check more than one?). Also, it
>>>>> might be useful to extend metalink documents with new verification
>>>>> methods before they arrive in the standard. Perhaps unknown types
>>>>> could be allowed here? The same comments mostly apply to digital
>>>>> signatures too (4.2.13).
>>>>
>>>> I agree that unknown hash types or digital signatures should be allowed.
>>>>
>>>>  This document defines nine initial values for hash types.  It may be
>>>>  useful to extend Metalink documents with new verification methods, so
>>>>  unknown types are allowed.
>>>>
>>>> and
>>>>
>>>>  metalink:signature elements MUST have a "type" attribute.  The inital
>>>>  value of "type" is the string that is non-empty and matches "pgp".
>>>>  It may be useful to extend Metalink documents with new types of
>>>>  digital signatures, so unknown types are allowed.
>>>>
>>>>
>>>> What do you suggest about dealing with multiple hash types?
>>>>
>>>> An issue could be if someone malicious makes a metalink where the MD5
>>>> matches that of something published by a legit group, but also
>>>> includes a SHA-256 checksum, and if clients prefer & only verify
>>>> SHA-256, then the file could appear to be good even if the downloader
>>>> looked inside the metalink & compared the MD5 (if the legit group
>>>> didn't also use & publish SHA-256 checksums too).
>>>> - Show quoted text -
>>>>
>>>>> (8) Formatting nit last - the use of spacing in the nesting of XML
>>>>> elements is pretty inconsistent - so instead of this on page 4:
>>>>>
>>>>> <?xml version="1.0" encoding="UTF-8" ?>
>>>>>  <metalink version="3.0" xmlns="http://metalinker.org">
>>>>>    <published>2008-05-15T12:23:23Z</published>
>>>>>    <files>
>>>>>      <file name="example.ext">
>>>>>      <description>A description of the example file for download.
>>>>>      </description>
>>>>>      <verification>
>>>>>        <hash type="md5">83b1a04f18d6782cfe0407edadac377f</hash>
>>>>>        <hash type="sha1">80bc95fd391772fa61c91ed68567f0980bb45fd9
>>>>>        </hash>
>>>>>      </verification>
>>>>>      <resources>
>>>>>        <url>ftp://ftp.example.com/example.ext</url>
>>>>>        <url>http://example.com/example.ext</url>
>>>>>        <url>http://example.com/example.ext.torrent</url>
>>>>>      </resources>
>>>>>      </file>
>>>>>    </files>
>>>>>  </metalink>
>>>>>
>>>>> It would be much nicer if it were nested something like this:
>>>>>
>>>>> <?xml version="1.0" encoding="UTF-8" ?>
>>>>>  <metalink version="3.0" xmlns="http://metalinker.org">
>>>>>    <published>2008-05-15T12:23:23Z</published>
>>>>>    <files>
>>>>>      <file name="example.ext">
>>>>>        <description>A description of the example file for
>>>>> download.</description>
>>>>>        <verification>
>>>>>          <hash type="md5">83b1a04f18d6782cfe0407edadac377f</hash>
>>>>>          <hash type="sha1">80bc95fd391772fa61c91ed68567f0980bb45fd9
>>>>>          </hash>
>>>>>        </verification>
>>>>>        <resources>
>>>>>          <url>ftp://ftp.example.com/example.ext</url>
>>>>>          <url>http://example.com/example.ext</url>
>>>>>          <url>http://example.com/example.ext.torrent</url>
>>>>>        </resources>
>>>>>      </file>
>>>>>    </files>
>>>>>  </metalink>
>>>>
>>>> Done, if:
>>>>
>>>>      <description>A description of the example file for
>>>> download.</description>
>>>>
>>>> is the only thing changed?
>>>>

-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
 )) Easier, More Reliable, Self Healing Downloads
_______________________________________________
Int-area mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/int-area