Subject: Re: [APPS-REVIEW] Metalink XML Download Description Format (draft-bryan-metalink-02)

From: "Anthony Bryan" <[email-masked]>
Date: Thu, 4 Sep 2008 16:09:26 -0400

On Wed, Sep 3, 2008 at 2:54 PM, Julian Reschke <[email-masked]> wrote:
> Anthony Bryan wrote:
>>
>> ...
>>>
>>> One way out of that would by to "grab" special type names like "torrent",
>>> and hardwire them. That should be ok as long as they can't collide with
>>> registered names (thus no "/" allowed).
>>
>> Great, thanks Julian! Thanks to the other people that mentioned this
>> as well, it seems much cleaner & tightened up now. :)
>>
>> Do I need to explicitly state no "/" allowed, or just know any that
>> ones I'm defining can't be "/" ?
>
> I'd say that types without "/" are reserved, and the only reserved value for
> now is "torrent".

Done.

>> And would it be better to just allow the unofficial MIME type, instead
>> of essentially creating an unofficial registry with one or a handful
>> of entries? Or is MIME types plus hardwired entries better?
>> ...
>
> I think you'll see pushback once you start to make normative requirements
> related to unregistered mime types. So, you can also try to get a type
> registered, which of course would be the cleaner solution...

Understood. That would be nice if it already was registered.

I've incorporated these comments into a new draft at
http://tools.ietf.org/html/draft-bryan-metalink-02

There are some issues I would like help from people:

- Inclusion of digital signatures other than PGP.
- Ability to specify if a file is in one or multiple languages.
- Ability to specify if a file is for one or more operating systems.
Maybe using the IANA Operating System Names registry at
http://www.iana.org/assignments/operating-system-names , but these
seem quite outdated. Or possibly using
http://nvd.nist.gov/schema/nvdcve.xsd ?

-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
 )) Easier, More Reliable, Self Healing Downloads
_______________________________________________
Apps-Discuss mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: "Anthony Bryan" <[email-masked]>
Date: Wed, 17 Sep 2008 20:31:45 -0400

On Tue, Sep 9, 2008 at 7:27 PM, Nathan Myers <[email-masked]> wrote:
> On Tue, Sep 09, 2008 at 06:11:35PM -0400, Anthony Bryan wrote:
>> On Tue, Sep 9, 2008 at 5:29 PM, Nathan Myers <[email-masked]> wrote:
>> > Anthony,
>> >
>> > I have skimmed the Metalink draft.
>>
>> Hi Nathan. Thanks for taking the time to do so. Do you mind ccing
>> [email-masked] or allowing me to? Public comments are good. :)
>
> You may cc with whatever bits of sense remain after you've
> eviscerated me enough.

It's all friendly discussion, sorry for the delay!

>> > My first concern is that there probably should be a sparse hash
>> > defined, for cases where hashing the whole file is prohibitively
>> > expensive.  One practical approach is to do a regular checksum
>> > (md5, sha) on a fixed-size subset of the file, distributed in
>> > chunks throughout.  This could be implemented with attributes,
>> > e.g. "sparsechunksize=" and "numsparsechunks".  The spacing
>> > between chunks would need to be defined rigorously, particularly
>> > near EOF, such that the beginning and end of the file always
>> > get summed, and for small files, so that it degenerates cleanly
>> > to a regular full-file checksum.
>>
>> I haven't heard of this before, is it used elsewhere?
>
> I know it only from proprietary products.
>
>> If I understand correctly, would it be enough to use the already
>> defined chunk/piece checksums and just omit some from the file?
>>
>> Here's what this looks like now, minus most of the chunks.
>>
>> <verification>
>>         <hash type="md5">579be26a5dd7957394c4d69e72e0127d</hash>
>>         <hash type="sha-1">0156e8d36971ae2bae50dfc14c96ac47675ee5a6</hash>
>>         <hash type="sha-256">c65613b5bf9853773b68e2498616e1499fc0b9e41f73a5726c95a8fe55f44650</hash>
>>         <pieces type="sha-1" length="2097152">
>>           <hash piece="0">cdc92a8fd1477850e9fd466334769e040ba313bb</hash>
>>           <hash piece="16">2828c06d1034949787f4606bd73435551606fd4f</hash>
>>           <hash piece="31">0527ddda2ae716cf56ac8a55f5bec1489e6f1b18</hash>
>>           <hash piece="46">46e433a1fc5d067ffe82c930f9a2dc9607560f4c</hash>
>>           <hash piece="55">42c92d20fdc9c05a147be910c6f0840c5c4fbc86</hash>
>>           <hash piece="56">ac23f5f017e5ba53456129c661285025f6366a22</hash>
>>         </pieces>
>>       </verification>
>
> Let's see if I understand this... Your fragment above has md5, sha-1,
> and sha-256 hashes for the whole file, and then an sha-1 hash for each
> range of bytes (0 to 2097151), (16*2097152 to 17*2097152-1), etc.,
> correct?  I suppose the difference in what I had suggested was to
> provide a way to deduce the list of pieces programmatically, along
> with a summation of all the individual piece hashes, so the pieces
> need not all be listed severally.

Yes, all the hashes are optional, whether full file hashes or just for
the chunks.

I hadn't heard of that before, but I suppose you could just include
the hashes for some chunks. The full file hashes aren't required if
you don't want them or have the time to compute them.

> My employer uses this to determine whether to send a file that might
> already exist at the destination.  When the files are 300GB, time
> spent summing a whole file that will not actually be sent can be
> prohibitively expensive.

Would including only some of the chunk checksums be sufficient for
your purposes?

>> > I suspect it will be important to allow names for pieces.
>> > It seems the natural way to implement file forks/streams.
>>
>> You mean being able to mix & match named parts of different files, or
>> something else?
>
> Perhaps I am misunderstanding what pieces are about.
>
> On "modern" operating systems (including MSWin, Macosix, and Linux)
> a file may contain not just bytes, but act as a sort of subdirectory
> containing what amount to additional, oddly-named files.  On MSWin,
> these are called "streams".  Apple calls them "forks".  They record
> additional bytes that a call "read(open(fname,0),buf,~0)" would not
> reveal.  The regular file contents amount to a default stream/fork,
> one of potentially many.  To copy such a file correctly, you must copy
> its regular data, and also all its associated sub-files.
>
> For your purposes, it might suffice to list the names and metadata
> of the sub-files as separate files, using their odd names, but the
> odd names and delimiters aren't portable in the way "/" is.
> Furthermore, most of the metadata for the sub-files is shared with
> the parent file.
>
> This also suggests a need for more checksums; the present that
> record the output of "cat f | md5sum" etc., and more that sum
> the contents of all the file and subfile components.

You're the first to mention this. Perhaps it can be dealt with as an
extension to metalink?

Unless other people feel that it should be dealt with in the metalink "core".

>> > I wonder, idly, why to bother offering DES3 in-this-day-and-age.
>> > Is it for uses that have hardware support for DES, but not AES,
>> > or is it just a general policy that everything IETF should
>> > support DES3?
>>
>> Where is DES3 mentioned? It's not in the Metalink Draft, so in the
>> references? I'm trying to use references that mesh w/ IETF documents.
>> If there is a more recently updated one, please let me know.
>> Otherwise, I don't know about IETF policy enough.
>
> Maybe I'm misunderstanding section 5.2, Encryption: "Section 5.1 of
> [REC-xmlenc-core] requires support of TripleDES, AES-128, and AES-256."

I'm using the same references as the Atom XML format (RFC 4287). I
don't see an updated version or replacement for [REC-xmlenc-core].