Subject: Re: comments on draft-barth-mime-sniffing

From: Ian Hickson <[email-masked]>
Date: Mon, 1 Jun 2009 08:18:22 +0000 (UTC)

On Sun, 31 May 2009, Adam Barth wrote:
> > 
> > I see no justification whatsoever for allowing conforming user agents 
> > to sniff types for new elements such as <video>, or encouraging such 
> > behavior, which is just opening the door for whole other categories of 
> > spoofing.  Certainly this isn't represented by any deployed 
> > infrastructure.
> 
> The current draft doesn't take a position on this issue.  Is there 
> something you'd like changed in the draft pursuant to the above?

The current HTML5 spec's position on this issue is that the Content-Type 
header is completely ignored in the processing of <video>, I believe. 
(Like with <img>, though less explicitly.) There's no sniffing algorithm 
defined or referenced; it is assumed that video formats define how to 
recognise themselves.

I'd be happy to require the browsers to obey Content-Type rigorously here 
if they are willing to implement such requirements. Can any browser 
vendors comment on this issue?

(In practice, many video files seem to be transmitted with completely 
bogus MIME types; indeed video, possibly primarily pornographic video, is 
the primary motivator for the "text vs binary" sniffing algorithm in the 
content sniffing draft, which is invoked when navigating to files, as 
opposed to when using <video>.)

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: "Robert O'Callahan" <[email-masked]>
Date: Mon, 1 Jun 2009 21:39:32 +1200

> header is completely ignored in the processing of <video>, I believe.

Hmm, really? In our media element implementation, we honor Content-Type
completely and do no sniffing whatever.

Rob
-- 
"He was pierced for our transgressions, he was crushed for our iniquities;
the punishment that brought us peace was upon him, and by his wounds we are
healed. We all, like sheep, have gone astray, each of us has turned to his
own way; and the LORD has laid on him the iniquity of us all." [Isaiah
53:5-6]

---

From: Adam Barth <[email-masked]>
Date: Fri, 5 Jun 2009 01:02:25 -0700

On Sun, May 31, 2009 at 1:18 PM, Adam Barth <[email-masked]> wrote:
> On Sun, May 31, 2009 at 10:22 AM, Larry Masinter <[email-masked]> wrote:
>> In the interest of monitoring this, and possibly removing content
>> type sniffing in the future, is it possible to publish (and reference)
>> the methodology used?
>
> The methodology is described in
> http://www.adambarth.com/papers/2009/barth-caballero-song.pdf, which
> has been published in archival form in the IEEE Symposium on Security
> and Privacy.  I suppose I can add a reference if you'd find that
> helpful.

I've added a TODO to insert this reference.  I have to learn how to
add references to I-Ds before I can complete this task.

Adam

---

From: Adam Barth <[email-masked]>
Date: Fri, 5 Jun 2009 01:04:40 -0700

> completely and do no sniffing whatever.

I've removed all mention of audio or video formats from the I-D.  I
think it's wise to wait and see how <video> and <audio> get deployed.
If we can follow Content-Types strictly, that would be great.

Adam

---

From: Adam Barth <[email-masked]>
Date: Fri, 5 Jun 2009 01:55:34 -0700

> add references to I-Ds before I can complete this task.

Julian Reschke has graciously taught me how to add a reference, which
I have now done.

Adam

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 8 Jun 2009 03:06:39 +0100

> If we can follow Content-Types strictly, that would be great.

Video formats are incredibly diverse, and typically consist of:

    - The container file format, or sometimes no container.

    - The container (if there is one) has 1 or more streams of video
      (and audio).

    - Each video stream may be any common video encoding (codec).

E.g. you can have MPEG video inside an AVI, MPEG video inside an MPEG
container, WMV video inside a WMV container, or WMV inside an AVI
container, etc.  Although it's a hideous mess, there's a certain
amount of orthogonality between the container formats and the video
encodings they can contain.

Am I right that the Content-Type merely says the type of the container
format, and says nothing about the format of encoded video(s) within?

-- Jamie

---

From: "Robert O'Callahan" <[email-masked]>
Date: Mon, 8 Jun 2009 14:15:11 +1200

> format, and says nothing about the format of encoded video(s) within?

No.

http://tools.ietf.org/html/rfc4281

Rob

---

From: Boris Zbarsky <[email-masked]>
Date: Sun, 07 Jun 2009 22:18:49 -0400

> format, and says nothing about the format of encoded video(s) within?

I believe this is correct.  Certainly the MIME type Gecko looks for 
where video is concerned is video/ogg or application/ogg.

I would hope that container formats define how to figure out the types 
of media streams they contain.  If they don't, then that's unfortunate....

-Boris

---

From: Boris Zbarsky <[email-masked]>
Date: Sun, 07 Jun 2009 22:19:45 -0400

> http://tools.ietf.org/html/rfc4281

Ah, I stand corrected.  ;)

-Boris

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 8 Jun 2009 03:57:10 +0100

> of media streams they contain.  If they don't, then that's unfortunate....

There's always a way to find out the subtypes, of course, but it can
be quite complicated.

E.g. the unix "file" command does not try, because it's more
complicated than a fixed header in the file.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 8 Jun 2009 03:58:51 +0100

Robert O'Callahan wrote:
> 
>    On Mon, Jun 8, 2009 at 2:06 PM, Jamie Lokier <[1][email-masked]>
>    wrote:
> 
>    [2]http://tools.ietf.org/html/rfc4281

Thanks!  I hadn't heard of that.

*chuckles* I will be *astonished* if Apache and other web servers
consistently report the correct, full RFC4281 Content-Type for video
files :-)

-- Jamie

---

From: "Robert O'Callahan" <[email-masked]>
Date: Mon, 8 Jun 2009 15:32:22 +1200

> consistently report the correct, full RFC4281 Content-Type for video

As long as they get the container type right, no sniffing will be needed.

Rob

---

From: Ian Hickson <[email-masked]>
Date: Fri, 12 Jun 2009 18:55:29 +0000 (UTC)

On Mon, 1 Jun 2009, Robert O'Callahan wrote:
> > The current HTML5 spec's position on this issue is that the 
> > Content-Type header is completely ignored in the processing of 
> > <video>, I believe.
> 

I've updated HTML5 to require that Content-Types of types that are not 
supported cause the resource to be ignored (even if it would otherwise be 
supported).

---

From: "Simon Pieters" <[email-masked]>
Date: Mon, 15 Jun 2009 08:23:33 +0200

> supported cause the resource to be ignored (even if it would otherwise be

If a UA does not know what is not supported, is it reasonable to consider  
anything that is not video/* or audio/* to be not supported?

-- 
Simon Pieters
Opera Software

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 16 Jun 2009 16:25:43 +0100

> anything that is not video/* or audio/* to be not supported?

For video, content negotiation is probably going to end up being done
in Javascript rather than HTTP, or by User-Agent recognition.

The Content-Type of video/* simply doesn't provide enough information,
in practice to tell if the file can be played or not by a particular
browser / video player.  The file itself must be read, to determine
the codecs used.  Of course the client must understand the file
format, but the tough part is understanding the codec format inside
the file format, which is just a container.

Although there is an RFC specifying MIME sub-types to video/*
describing the codecs, I don't see any way to sensibly fit that into
HTTP's Accept: content-negotiation.

-- Jamie

---

From: "Anne van Kesteren" <[email-masked]>
Date: Tue, 16 Jun 2009 17:32:13 +0200

On Tue, 16 Jun 2009 17:25:43 +0200, Jamie Lokier <[email-masked]> wrote:
>>> supported cause the resource to be ignored (even if it would otherwise  
>>> be supported).
>>
>> If a UA does not know what is not supported, is it reasonable to  
>> consider anything that is not video/* or audio/* to be not supported?
>

Still, if the user agent does a request, should it abort the request if the media type of the response is not video/* or audio/*?

-- 
Anne van Kesteren
http://annevankesteren.nl/

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 16 Jun 2009 19:23:04 +0100

Anne van Kesteren wrote:
> Still, if the user agent does a request, should it abort the request
> if the media type of the response is not video/* or audio/*?

I'm thinking if the response is an animated GIF, of type image/gif,
why forbid a <video> element from being allowed to play it as a video.

I'm also thinking, why forbid other image/* types from being played as
static videos.  After all, sometimes single-frame video/* files (such
as an MPEG single I-Frame) are sometimes created for this purpose on
real video players.  It would seem odd to allow the display of
single-frame video files, but forbid image/* files from behaving
exactly the same way.

-- Jamie

---

From: David Singer <[email-masked]>
Date: Tue, 16 Jun 2009 13:45:17 -0700

>supported cause the resource to be ignored (even if it would otherwise be

We're concerned about this, though we support the hope that we can 
get away from doing content-type sniffing in this area.

1) if you use a multimedia framework that you don't control, and it 
does content-type sniffing, then you get it whether you want it or 
not.  QuickTime does, so Safari does, right now.

2) End-users posting AV content rarely have access to the MIME type 
config. of their server (often, their ISP's server).  Getting the 
MIME type table updated can take ages.

3) Once one browser gets a competitive 'edge' by sniffing, the rest 
usually follow.  It's hard to see why this would not happen in this 
case.

Our data suggests that the amount of AV content on the net, today, 
that is served under 'wrong' MIME types is perhaps 10-15%, which 
doesn't seem ignorable.  One of the 'wrong' types is 
application/octet-stream, which seems to invite sniffing;  and of 
course a lot of the rest is text/plain, which is (I believe) Apache's 
default.

So, we're with the motivation, but not sure we're with the spec. as 
written. Hm.
-- 
David Singer
Multimedia Standards, Apple Inc.

---

From: "Joe D Williams" <[email-masked]>
Date: Tue, 16 Jun 2009 17:00:12 -0700

> get away from doing content-type sniffing in this area.

I hope so too. For <video> and <audio> it seems like a fairly 
specified contract about what can be expected to play in a W3C HTML 5 
browser. Is sniffing always required? For <object> it has seemed like 
@data targets were sniffed, but for a param URL the url was sent to 
the player and the player essentially got to manage getting and 
running the content.

If the allowed by the spec file extensions are listed then that will 
probably be enough to start out with.
Trust the author to test before deployment.

I would not expect to be using any particular media player, but one 
essentially built-in that is tuned to work with this spec. If I want 
_all_ the bells and whistles and the specific or optimized or 
(streaming?) file types and codecs provided by the major competitors 
in media players, then I will 'fallback' to <object> for possible 
access to maybe proprietary file extensions.
What I as an author am looking for in these elements is a sure-fire 
way to get something running. From abstraction I would not expect to 
get the current default win/ie or apple/safari or realplayer or adobe 
whatever optioned media player, for example, or something else that 
the user may have installed and uses for the system default for the 
registered mimes, but instead the 'standard' simple, possibly 
restricted "no brand" player 'built-in' to the HTML 5 browser. This 
will not handle everything, but if it runs and I need more or 
different then maybe I can get user cooperation to bootstrap me into 
what might be more complex or effective interactive multiple media 
environment.

So, me as the author knows the rules and what is supposed to be where 
I say it is so I would expect the host to get that <video> or <audio> 
player going with what I have put out there without a lot of delay and 
questions. On the other hand, if I don't know what is coming in and I 
wanted to try <video> first and I got an unsupported (by <video> or 
<audio>) MIME or file extension, then I would have to fallback to 
<object> to try and handle it by @type and/or @classid selection with 
what might end up to be the actual current user specified default 
player for the browser host platform .

I think the reason for host browser needing to sniff content streams 
in general is not so much needed in this case. Perhaps the file 
extensions for these element's content are more well known and better 
defined than their MIMEs. Since the author should know (I mean if 
tested at home; it should work on the WWW!). So I would say in this 
case it is not possible to recommend actual sniffing by the host 
browser past the file extension. Maybe in these cases of <audio> and 
<video> if the file extension is recognized, treat it if @type was 
+xml and just pass the URL to the handler and let it start to work it 
out?

Thank You and Best Regards,
Joe

---

From: "Robert O'Callahan" <[email-masked]>
Date: Wed, 17 Jun 2009 14:28:39 +1200

On Wed, Jun 17, 2009 at 8:45 AM, David Singer <[email-masked]> wrote:

> We're concerned about this, though we support the hope that we can get away
> from doing content-type sniffing in this area.
>
> 1) if you use a multimedia framework that you don't control, and it does
> content-type sniffing, then you get it whether you want it or not.
>  QuickTime does, so Safari does, right now.

But you (Apple) do control Quicktime, and that's no accident. There are
already many reasons why it's undesirable for a browser to depend on a
multimedia framework it doesn't directly or indirectly control.

But even if we end up allowing sniffing, we'll still definitely want
sniffing to be done consistently according to some spec, and media
frameworks will have to be changed to match.

2) End-users posting AV content rarely have access to the MIME type config.
> of their server (often, their ISP's server).  Getting the MIME type table
> updated can take ages.

That's OK, getting <video> support universally deployed will take ages too.
The old-school ISPs probably won't let you serve video from those accounts
anyway, and real hosting is cheap these days.

3) Once one browser gets a competitive 'edge' by sniffing, the rest usually
> follow.  It's hard to see why this would not happen in this case.

Sheer determination?

At least we can try being strict. If that doesn't work out, we can start
sniffing. It's impossible to go the other way.

Our data suggests that the amount of AV content on the net, today, that is
> served under 'wrong' MIME types is perhaps 10-15%, which doesn't seem
> ignorable.  One of the 'wrong' types is application/octet-stream, which
> seems to invite sniffing;  and of course a lot of the rest is text/plain,
> which is (I believe) Apache's default.

It's ignorable since that data is not currently served for consumption by
the <video> tag.

Rob

---

From: Boris Zbarsky <[email-masked]>
Date: Tue, 16 Jun 2009 19:44:58 -0700

Robert O'Callahan wrote:
>     course a lot of the rest is text/plain, which is (I believe)
>     Apache's default.
> 
> 
> It's ignorable since that data is not currently served for consumption 
> by the <video> tag.

One other interesting data point here is that upcoming versions of 
Apache will no longer send a default MIME at all, if I understood the 
comments in the relevant bug correctly.  That is, if there is no type 
configured for the file via some mechanism the server knows about, it 
will simply send no Content-Type header and let the consumer decide what 
to do.

-Boris

---

From: "Robert O'Callahan" <[email-masked]>
Date: Wed, 17 Jun 2009 14:55:25 +1200

I should also point out that so far we have had approximately zero
complaints from authors about the fact that Firefox doesn't sniff.

Rob

---

From: David Singer <[email-masked]>
Date: Tue, 16 Jun 2009 21:04:19 -0700

How many formats, off how many hosting services, do you support? We  
looked at movies, mp4 files, MPEG files, and so on (many thousand,  
fielded by the existing plugin). We have years of experience with the  
vagaries here, alas.

Sent from my iPhone

On Jun 16, 2009, at 19:55, Robert O'Callahan <[email-masked]>  
wrote:

> "He was pierced for our transgressions, he was crushed for our  
> iniquities; the punishment that brought us peace was upon him, and  
> by his wounds we are healed. We all, like sheep, have gone astray,  
> each of us has turned to his own way; and the LORD has laid on him  
> the iniquity of us all." [Isaiah 53:5-6]

---

From: Shane McCarron <[email-masked]>
Date: Tue, 16 Jun 2009 23:31:24 -0500

This is my favorite comment in this thread, bar none!  Why, oh why are 
you all trying so hard to continue to support broken behavior instead of 
slapping down the people who insist on doing it wrong?  If the user 
agents (go Firefox!) just refused to sniff, then no one would send 
nonsense content.  'cause it wouldn't work. 

Robert O'Callahan wrote:
> iniquities; the punishment that brought us peace was upon him, and by 
> his wounds we are healed. We all, like sheep, have gone astray, each 
> of us has turned to his own way; and the LORD has laid on him the 
> iniquity of us all." [Isaiah 53:5-6]

-- 
Shane P. McCarron                          Phone: +1 763 786-8160 x120
Managing Director                            Fax: +1 763 786-8180
ApTest Minnesota                            Inet: [email-masked]

---

From: Dave Singer <[email-masked]>
Date: Tue, 16 Jun 2009 21:42:32 -0700

I suspect that the problem is partly economic.  Faces with bug 
reports etc. saying "it doesn't work right" we can either explain 
endlessly what people have got wrong...or we can sharply reduce the 
number of such calls (which cost money) by sniffing.

Don't get me wrong, I don't like sniffing.  But I don't think a 
stroke of Ian's pen will make it go away.

(It's also odd to have a spec. which bars the user agent from 'going 
the extra mile'. Specs classically say all the things you must do to 
be conformant, and generally can't stop you from doing more.)

At 23:31 -0500 6/16/09, Shane McCarron wrote:
>This is my favorite comment in this thread, bar none!  Why, oh why 
>are you all trying so hard to continue to support broken behavior 
>instead of slapping down the people who insist on doing it wrong? 
>If the user agents (go Firefox!) just refused to sniff, then no one 
>would send nonsense content.  'cause it wouldn't work.
>Robert O'Callahan wrote:

-- 
Dave Singer
Apple Computer/QuickTime 408 974 3162

---

From: Shane McCarron <[email-masked]>
Date: Wed, 17 Jun 2009 00:01:48 -0500

> be conformant, and generally can't stop you from doing more.)

Exactly!  Most modern standards and "recommendations" define what 
happens when presented with conforming input, and implicitly or 
explicitly state that non-conforming input results unspecified or 
undefined behavior (see *all* the IEEE POSIX standards, for example).  
It is an unparalleled act of hubris that this work attempts not only to 
define good behavior, but also to define *positive* behavior in the face 
of bad data.  That's just insane.  If it is critical to define the 
behavior of implementations when handed invalid input, then say that the 
implementation is required to tell the user "the input is broken"!  
Defining that an implementation will somehow cleverly deal with every 
broken piece of input is not just silly, it's impossible.

Of course, that's just my opinion.  I could be wrong (with apologies to 
Dennis Miller).

>> instead of slapping down the people who insist on doing it wrong? If 
>> the user agents (go Firefox!) just refused to sniff, then no one

---

From: "Robert O'Callahan" <[email-masked]>
Date: Wed, 17 Jun 2009 17:58:56 +1200

On Wed, Jun 17, 2009 at 4:04 PM, David Singer <[email-masked]> wrote:

> How many formats, off how many hosting services, do you support?

Only Ogg and Wave, as you know. We've got users at big sites, some personal
sites, blogs --- self-hosting, hosted servers, Amazon S3, not sure what
else.

We have had plentiful feedback about other kinds of deployment issues. When
we were talking about having a default same-origin restriction plus CORS,
that generated a lot of negative feedback. Requiring HTTP byte-range
requests for seeking is a problem in some cases. But sending valid MIME
types hasn't come up yet.

Rob

---

From: Adam Barth <[email-masked]>
Date: Tue, 16 Jun 2009 23:12:14 -0700

I think roc was specifically referring to content for the <video> tag.
 Because no such content currently exists, the legacy compatibility
requirements are much less onerous than, say, for HTML.

Adam

On Tue, Jun 16, 2009 at 9:31 PM, Shane McCarron<[email-masked]> wrote:
> This is my favorite comment in this thread, bar none!  Why, oh why are you
> all trying so hard to continue to support broken behavior instead of
> slapping down the people who insist on doing it wrong?  If the user agents
> (go Firefox!) just refused to sniff, then no one would send nonsense
> content.  'cause it wouldn't work.
>
> Robert O'Callahan wrote:
>>

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 17 Jun 2009 00:15:24 -0700

> requirements are much less onerous than, say, for HTML.

I think Dave's point was that if you support codecs and containers in  
<video> that are currently supported by widely used plugins, then you  
are more likely to face these legacy issues with broken MIME types  
when deploying <video>. Content authors might well expect that already  
published MP4 files which work in the QuickTime plugin or a Flash- 
based player, should continue to work if embedded via <video>. But  
there isn't a similar volume of already-deployed Ogg Theora files  
being served with a broken MIME type, so it's not likely there would  
be a similar expectation. To be fair, though, we didn't go through the  
exercise of being stricter about types, so we can't say definitively  
if there would be complaints.

  - Maciej

>> are you
>> agents
>>> iniquities;

---

From: "Anne van Kesteren" <[email-masked]>
Date: Wed, 17 Jun 2009 09:45:11 +0200

> will simply send no Content-Type header and let the consumer decide what  

Per the current rules that would give the same result as a bogus media type though. I.e. lack of Content-Type does not give you the right to start sniffing for <video> / <audio>.

---

From: Adam Barth <[email-masked]>
Date: Tue, 16 Jun 2009 23:18:19 -0700

On Tue, Jun 16, 2009 at 10:01 PM, Shane McCarron<[email-masked]> wrote:
> Dave Singer wrote:
>> (It's also odd to have a spec. which bars the user agent from 'going the
>> extra mile'. Specs classically say all the things you must do to be
>> conformant, and generally can't stop you from doing more.)
>
> Exactly!  Most modern standards and "recommendations" define what happens
> when presented with conforming input, and implicitly or explicitly state
> that non-conforming input results unspecified or undefined behavior (see
> *all* the IEEE POSIX standards, for example).  It is an unparalleled act of
> hubris that this work attempts not only to define good behavior, but also to
> define *positive* behavior in the face of bad data.  That's just insane.  If
> it is critical to define the behavior of implementations when handed invalid
> input, then say that the implementation is required to tell the user "the
> input is broken"!  Defining that an implementation will somehow cleverly
> deal with every broken piece of input is not just silly, it's impossible.

For content sniffing, this approach causes both security and
compatibility issues.  Ultimately, user agents have to decide how to
act when presented with non-conformant input.  You're welcome to
construct a user agent that displays a "the input is borken" message,
but I suspect your user agent will not gain an appreciable market
share.  Given that user agents do process this input, the world is
better off if they all do so in the same way.

Adam

---

From: Boris Zbarsky <[email-masked]>
Date: Wed, 17 Jun 2009 01:09:12 -0700

> Per the current rules that would give the same result as a bogus media type though. I.e. lack of Content-Type does not give you the right to start sniffing for <video> / <audio>.

So it's also not possible to deliver <video>/<audio> content over 
HTTP/0.9, or over other protocols that have no way to deliver a 
Content-Type header (file:// comes to mind)?

If that's the case, then it sounds like the rules are badly broken and 
we should fix them.  If not, then presumably we have some rules on how 
to treat <audio>/<video> in situations when no MIME type is delivered 
via the network protocol, and should presumably be applying those rules 
in all such cases, no?

-Boris

---

From: "Anne van Kesteren" <[email-masked]>
Date: Wed, 17 Jun 2009 10:21:47 +0200

On Wed, 17 Jun 2009 10:09:12 +0200, Boris Zbarsky <[email-masked]> wrote:
> Anne van Kesteren wrote:
>> Per the current rules that would give the same result as a bogus media  
>> type though. I.e. lack of Content-Type does not give you the right to  
>> start sniffing for <video> / <audio>.
>

File system works. Content-Type metadata is defined here:

  http://tools.ietf.org/html/draft-abarth-mime-sniff-01#section-2

> via the network protocol, and should presumably be applying those rules  

It seems like a bad idea to me, though I agree that the media type situation is not quite ideal.

---

From: Boris Zbarsky <[email-masked]>
Date: Wed, 17 Jun 2009 01:47:46 -0700

Anne van Kesteren wrote:
> File system works.

Maybe.... Let's assume it foes for now.

> Content-Type metadata is defined here:
> 

So per this draft, it is in fact not possible to send <video> over 
HTTP/0.9, or ftp, or in fact any protocol that doesn't provide an 
explicit MIME type....

> It seems like a bad idea to me, though I agree that the media type situation is not quite ideal.

Thinking about this more, having the rule be "no sniffing" in this case 
might not be too bad.  Yes, if an Apache server is not configured in any 
way for your video it'll get sent with no type at all and not show up as 
video... but it also won't show up as text (which is where we are now). 
  And it'll be pretty obvious to you that it's not working and that you 
should get your server fixed, especially if web browsers provide nice 
informative error messages here.

That all assumes that people actually stick to not sniffing, of course.

-Boris

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 17 Jun 2009 22:50:36 +1000

Speaking personally -- that's all fine and good and understandable,  
but why can't I as a user turn it off, either as a preference or on a  
case-by-case basis? Sometimes "going the extra mile" gets it wrong,  
and as with any heuristic, there needs to be a way to say "no, don't  
do that!"

E.g., how hard is it to have an unintrusive non-modal indication that  
the type was sniffed, giving the user the option to view it without  
sniffing (as FF does when it asks you if you want to remember a  
password)? This isn't perfect (e.g., because there can still be  
unwanted side effects of loading the document with the incorrect media  
type), but it's a good step in the right direction...

I know that implementing that take developer time, but how much  
developer time has been sunk on these discussions already? ;)

Cheers,

--
Mark Nottingham     http://www.mnot.net/

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 17 Jun 2009 07:45:24 -0700

On Jun 17, 2009, at 1:09 AM, Boris Zbarsky wrote:

> Anne van Kesteren wrote:
>> Per the current rules that would give the same result as a bogus  
>> media type though. I.e. lack of Content-Type does not give you the  
>> right to start sniffing for <video> / <audio>.
>
> If that's the case, then it sounds like the rules are badly broken  
> and we should fix them.  If not, then presumably we have some rules  
> on how to treat <audio>/<video> in situations when no MIME type is  
> delivered via the network protocol, and should presumably be  
> applying those rules in all such cases, no?

It seems to me like it would make sense to treat the no MIME type case  
differently, since the server is giving the client a license to sniff  
per the HTTP protocol spec when it sends no Content-Type header. This  
doesn't have the same kinds of problems as sniffing out of text/plain.

Regards,
Maciej

---

From: Jonas Sicking <[email-masked]>
Date: Wed, 17 Jun 2009 08:47:45 -0700

On Wed, Jun 17, 2009 at 5:50 AM, Mark Nottingham<[email-masked]> wrote:
> Speaking personally -- that's all fine and good and understandable, but why
> can't I as a user turn it off, either as a preference or on a case-by-case
> basis? Sometimes "going the extra mile" gets it wrong, and as with any
> heuristic, there needs to be a way to say "no, don't do that!"
>
> E.g., how hard is it to have an unintrusive non-modal indication that the
> type was sniffed, giving the user the option to view it without sniffing (as
> FF does when it asks you if you want to remember a password)? This isn't
> perfect (e.g., because there can still be unwanted side effects of loading
> the document with the incorrect media type), but it's a good step in the
> right direction...
>
> I know that implementing that take developer time, but how much developer
> time has been sunk on these discussions already? ;)

UI is very very rarely the answer. I believe chrome as a rule to only
add UI if they think that over 90% of their users will use it. In
firefox I don't think we have an explicit rule like that, however we
are still very conservative about adding things to the UI. The reason
is that if you add UI for too many scenarios you'll very quickly end
up with a cluttered UI. While your proposal is something that users
that don't understad/care what it is can simply ignore, it's still
something that takes attention away for such users and leaves them
wondering what is going on. Imagine getting one or two of such an
"information bar" on every page you visit, for things like popup
blockers, content sniffers, security notifications, use of deprecated
features etc. Very quickly this will become very annoying and unlikely
to no longer fulfill its purpose.

One think that we often tell people that want something added to
firefox UI is: Build an extension that does that, then get people
excited about it and build a user base. Once we see that there is a
real desire for this, we have much better data to make an informed
decision if we should add it to the default firefox configuration.

I'll also point to the success of firefox compared to the success of
the old mozilla suite. Basically the two were the same browser, the
main difference was that firefox had a severely simplified UI by
removing buttons and menu options.

So it's not at all about how hard something is to implement, it's
about if it'll make a good product or not.

/ Jonas

---

From: "Joe D Williams" <[email-masked]>
Date: Wed, 17 Jun 2009 10:42:39 -0700

> but why can't I as a user turn it off, either as a preference or on 
> a case-by-case basis? Sometimes "going the extra mile" gets it 
> wrong, and as with any heuristic, there needs to be a way to say 
> "no, don't do that!"

Well, lets give both author and the actual browser pilot a chance.
First, from authoring, for <audio> and <video> the author can present 
from a small, safe group of content models using simple reference to 
file MIME and file extension. These are first class objects; the 
element is a trusted context that can negotiate directly with the host 
DOM. These are active, live nodes but their content is not active.

Until these spec <audio> and <video> MIMEs are officially registered 
and the servers are actually updated, or the author can produce 
.htaccess file  to control the served ContentType, the server response 
is uncertain. So at this time and in the future, the server may 
actually produce the target content with no content type string.

In fact, the only information that needs to be be relied upon at this 
point is that if the author says audio me.ogg then the UA either:
1. fetches the resource for sniffing then sends it to the media 
handler, or
2  sends the handler the absolute URL and leaves until the handler 
context needs it again.
I think 2 is preferred.

Next, from the browser user, since <audio> and <video> are first class 
objects, there are no special permissions required to render asap, In 
particular, there are no "Security" issues. Default is as the author 
specified for the current object.
Although UA might provide some common means of defining use 
preferences to allow local control of the browing environment produced 
by particular pieces of the document it is on same order as the 
choices provide by the author with @sandbox. or CSS. If any halt or 
significant delay is produced by these user-selected UA actions, the 
fallback content must be rendered, then replaced by the parent content 
when approved.

As for UA sniffing of this particular <audio> or <video> content, it 
is useless and counter-productive. First, for these types, no matter 
what goes into the handler. the content can only confuse or break the 
handler not its container. Next, although a fine idea, we are unlikely 
to find a MIME-like string in the essentially binary file we want the 
handler to be dealing with. There are metadata in there, and the whole 
thing could be XML someday but only the handler needs to know about 
that in this case. For example, no matter what the served or sniffed 
MIME, if the author said ogg then the handler will treat the content 
only as ogg. Can there be scripts or any other 'security' issues in 
these content types?

I will check more about ogg, but for .wav, I find that any suffix to 
.wav to tell more about the particular internal data is for internal 
production sorting convenience in authoring tools, maybe. Otherwise, 
you do have to actually read some file front matter to determine some 
actual characteristics, such as sample rate, and others. These are of 
no value to the UA or the handler in this instance because a handler 
that does .wav is not going to have trouble with any spec form of the 
.wav. PDM is just the best open choice of other mod alternatives. 
Again, if it works on a couple of HTML 5 browsers at home, then it 
should just work out there in the HTML 5 WWW.

---

From: Adam Barth <[email-masked]>
Date: Wed, 17 Jun 2009 11:11:57 -0700

Hi Joe,

I had a lot of difficulty understanding your message.  Are you
suggesting we sniff the media type from the file extension for audio /
video content?

Adam

On Wed, Jun 17, 2009 at 10:42 AM, Joe D Williams<[email-masked]> wrote:
>> case-by-case basis? Sometimes "going the extra mile" gets it wrong, and as
>> with any heuristic, there needs to be a way to say "no, don't do that!"
>
> First, from authoring, for <audio> and <video> the author can present from a
> small, safe group of content models using simple reference to file MIME and
> file extension. These are first class objects; the element is a trusted
> context that can negotiate directly with the host DOM. These are active,
> live nodes but their content is not active.
>
> Until these spec <audio> and <video> MIMEs are officially registered and the
> servers are actually updated, or the author can produce .htaccess file  to
> control the served ContentType, the server response is uncertain. So at this
> time and in the future, the server may actually produce the target content
> with no content type string.
>
> In fact, the only information that needs to be be relied upon at this point
> is that if the author says audio me.ogg then the UA either:
> 1. fetches the resource for sniffing then sends it to the media handler, or
> 2  sends the handler the absolute URL and leaves until the handler context
> needs it again.
> I think 2 is preferred.
>
> Although UA might provide some common means of defining use preferences to
> allow local control of the browing environment produced by particular pieces
> of the document it is on same order as the choices provide by the author
> with @sandbox. or CSS. If any halt or significant delay is produced by these
> user-selected UA actions, the fallback content must be rendered, then
> replaced by the parent content when approved.
>
> As for UA sniffing of this particular <audio> or <video> content, it is
> useless and counter-productive. First, for these types, no matter what goes
> into the handler. the content can only confuse or break the handler not its
> container. Next, although a fine idea, we are unlikely to find a MIME-like
> string in the essentially binary file we want the handler to be dealing
> with. There are metadata in there, and the whole thing could be XML someday
> but only the handler needs to know about that in this case. For example, no
> matter what the served or sniffed MIME, if the author said ogg then the
> handler will treat the content only as ogg. Can there be scripts or any
> other 'security' issues in these content types?
>
> I will check more about ogg, but for .wav, I find that any suffix to .wav to
> tell more about the particular internal data is for internal production
> sorting convenience in authoring tools, maybe. Otherwise, you do have to
> actually read some file front matter to determine some actual
> characteristics, such as sample rate, and others. These are of no value to
> the UA or the handler in this instance because a handler that does .wav is
> not going to have trouble with any spec form of the .wav. PDM is just the
> best open choice of other mod alternatives. Again, if it works on a couple
> of HTML 5 browsers at home, then it should just work out there in the HTML 5
> WWW.
>

---

From: "Joe D Williams" <[email-masked]>
Date: Wed, 17 Jun 2009 11:57:51 -0700

> suggesting we sniff the media type from the file extension for audio 
> /video content?

Sorry Adam, I am saying any UA sniffing except the file extension is 
fruitless in the cases of <audio> and <video> given the spec group of 
content types allowed for these elements. In particular, the served 
content type is unpredicable and there is nothing inside the file that 
UA needs to know about before passing this type of file to the 
handler.

>> In particular, there are no "Security" issues.

If that is true for this set of elements using specified content 
models, then any sniffing that needs to be done is just the UA trying 
to shield the handler, which in this case should not be necessary. In 
particular, if there is any user interaction required to get these 
going, it should not be based on any security issure, just agreement 
with the users requested browsing environment.

But really, I'm just trying to make sure that it is a easy and 
reliable as feasible to use these. I think in this case the model says 
no need to sniff the file for anything else than it has the proper 
file extension for the element. Mostly, just load and run the thing 
asap.

Thanks and Best Regards,
Joe

---

From: Boris Zbarsky <[email-masked]>
Date: Wed, 17 Jun 2009 12:23:05 -0700

Joe D Williams wrote:
> UA needs to know about before passing this type of file to the handler.

1) I'm not quite clear on the distinction between "UA" and "handler" 
that you're drawing here.  Can you clarify what you mean?  In 
particular, if you claim that data-sniffing is not needed, then why is 
extension-sniffing needed?

2) Sniffing the file extension instead of the content significantly 
complicates server-side solutions that, say, send different content 
based on query params in the URL.  Of course these should arguably send 
a Content-Type header anyway.

-Boris

---

From: "Joe D Williams" <[email-masked]>
Date: Wed, 17 Jun 2009 13:09:10 -0700

>> Sorry Adam, I am saying any UA sniffing except the file extension 
>> is fruitless in the cases of <audio> and <video> given the spec 
>> group of content types allowed for these elements. In particular, 
>> the served content type is unpredicable and there is nothing inside 
>> the file that UA needs to know about before passing this type of 
>> file to the handler.
>
> particular, if you claim that data-sniffing is not needed, then why 
> is extension-sniffing needed?

Hi Boris,
Given that I am looking at it from an author side, how about if the 
current browsing context or whatever higher order process does the 
top-level sniffing before it goes to the renderer is called the UA, 
which contains a nested browsing context.<video> or <audio> element. 
So in these cases I will call the browser the host UA and the <video> 
element the home of the handler which actually does the rendering.

I would like to say that data-sniffing is not needed. Can the target 
files contain any security problems?
I don't think you will find a MIME string in there unless specially 
authored into the file.
.
The author should include both the mime and the target url in the 
element. If the type and file extension do not match the content model 
for the element then fallback. If the type and file extension match, 
then either send the file or URL to the player and let it do what it 
needs to do. At this time it is (may not) not reliable to rely upon 
the served ContentType so this cannot be used to reject the file. 
Data-sniffing of file internal structures by the UA could take place, 
but I don't think this would give the "standard" handler any 
additional info. Literally, the handler should be able to handle any 
.wav or .ogg spec files. The handler may need to do some data-sniffing 
to get set up. but all it needs from the UA is the content.

Again, the load on the author should be no more than getting the type 
and file name correct for the element and it should just work.

> based on query params in the URL.  Of course these should arguably 
> send a Content-Type header anyway.

The file is there with the right extension but maybe I can't get 
control of the server and its defaults are not sufficient for the UA 
sniffer (like maybe I can't use htaccess). Is my document success 
going to depend on the fact that the server must be configured to send 
the specified MIME? If it is served wrong or even absent a MIME am I 
going to be saved by the UA data-sniffing to figure out that the 
misserved file is really OK?

> say, send different content based on query params in the URL.

Are we going to see this with these elements? We have already 
negotiated the content model down to a simple set. Many times those 
query forms are built from scripts that have determined or are 
determining some client capabilites? The application and content types 
of these elements seems quite clear as defined by the kestrokes that 
actually appear in the element code at run time. .

---

From: "Robert O'Callahan" <[email-masked]>
Date: Thu, 18 Jun 2009 16:32:52 +1200

On Wed, Jun 17, 2009 at 7:15 PM, Maciej Stachowiak <[email-masked]> wrote:

> <video> that are currently supported by widely used plugins, then you are
> more likely to face these legacy issues with broken MIME types when
> deploying <video>. Content authors might well expect that already published
> MP4 files which work in the QuickTime plugin or a Flash-based player, should
> continue to work if embedded via <video>.

They may expect that, but breaking that expectation is less of a problem
than breaking actual existing content.

Rob

---

From: David Singer <[email-masked]>
Date: Thu, 18 Jun 2009 12:05:35 -0700

At 16:32  +1200 18/06/09, Robert O'Callahan wrote:
>On Wed, Jun 17, 2009 at 7:15 PM, Maciej Stachowiak 
><<mailto:[email-masked]>[email-masked]> wrote:
>
>I think Dave's point was that if you support codecs and containers 
>in <video> that are currently supported by widely used plugins, then 
>you are more likely to face these legacy issues with broken MIME 
>types when deploying <video>. Content authors might well expect that 
>already published MP4 files which work in the QuickTime plugin or a 
>Flash-based player, should continue to work if embedded via <video>.
>
>
>They may expect that, but breaking that expectation is less of a 
>problem than breaking actual existing content.

sorry, you've lost me.  breaking that expectation does break existing 
content;  and I can't see how doing content sniffing breaks more 
content than not.  can you explain?
-- 
David Singer
Multimedia Standards, Apple Inc.

---

From: "Robert O'Callahan" <[email-masked]>
Date: Fri, 19 Jun 2009 12:01:49 +1200

On Fri, Jun 19, 2009 at 7:05 AM, David Singer <[email-masked]> wrote:

> content;  and I can't see how doing content sniffing breaks more content
> than not.  can you explain?
>

Currently there is almost no existing content that uses the <video> element
to play media files that are being served with incorrect Content-Types.
(Please correct me if I'm wrong about this.) Therefore, not sniffing for
<video> should break almost no existing content.

Maciej suggests there is a significant amount of content using plugins
(Flash or Quicktime, I suppose) to play media files served with incorrect
Content-Types, and therefore authors will be surprised if they attempt to
use <video> to play those files and they don't work.

I say that "author surprise" is much less of a problem than "existing Web
pages stop working".

Rob

---

From: David Singer <[email-masked]>
Date: Thu, 18 Jun 2009 17:08:26 -0700

At 12:01  +1200 19/06/09, Robert O'Callahan wrote:
>On Fri, Jun 19, 2009 at 7:05 AM, David Singer 
><<mailto:[email-masked]>[email-masked]> wrote:
>
>>in <video> that are currently supported by widely used plugins, 
>>then you are more likely to face these legacy issues with broken 
>>MIME types when deploying <video>. Content authors might well 
>>expect that already published MP4 files which work in the QuickTime 
>>plugin or a Flash-based player, should continue to work if embedded 
>>via <video>.
>>
>
>
>sorry, you've lost me.  breaking that expectation does break 
>existing content;  and I can't see how doing content sniffing breaks 
>more content than not.  can you explain?
>
>
>Currently there is almost no existing content that uses the <video> 
>element to play media files that are being served with incorrect 
>Content-Types. (Please correct me if I'm wrong about this.) 
>Therefore, not sniffing for <video> should break almost no existing 
>content.
>
>Maciej suggests there is a significant amount of content using 
>plugins (Flash or Quicktime, I suppose) to play media files served 
>with incorrect Content-Types, and therefore authors will be 
>surprised if they attempt to use <video> to play those files and 
>they don't work.
>
>I say that "author surprise" is much less of a problem than 
>"existing Web pages stop working".

But I may well be able to re-author my HTML but completely unable to 
influence my hosting provider to provide the correct MIME types.  I 
don't want users trapped into using the old because we've made the 
new inaccessible to them.
-- 
David Singer
Multimedia Standards, Apple Inc.

---

From: Boris Zbarsky <[email-masked]>
Date: Thu, 18 Jun 2009 20:21:37 -0400

Joe D Williams wrote:
> top-level sniffing before it goes to the renderer

To which renderer?

> is called the UA, 
> which contains a nested browsing context.<video> or <audio> element. So 
> in these cases I will call the browser the host UA and the <video> 

OK...  I'm not sure how you're separating these, exactly, but under hte 
only reasonable split that comes to mind for me (the UA is what fetches 
the data while the handler is what actually renders it), you need to 
know what kind of data you have so you can select the right handler. 
Agreed, or not?  If not, is the problem one of disagreement on the 
definitions, or on the conclusion?

> files contain any security problems?

The target files (assuming you mean the <video> and <audio> data the 
server is sending) are always assumed to be malicious and trying to 
exploit the UA or the renderer to gain access to the user's system.

> I don't think you will find a MIME string in there unless specially 

Of course not.  Data sniffing doesn't look for MIME strings; it looks 
for initial bytes identifying the file type.  Many common file types 
must start with a particular byte sequence.

> element.

Actually, no.  The author can just include the target url in the 
<video>.  If using a <source> element, a type hint (non-authoritative) 
can be provided; this is to be used only for deciding whether to load 
the <source> at all, not for what to do with it after loading it.

> If the type and file extension do not match the content model 
> for the element then fallback.

Uh... What content model?  And how did extension come in?  Extension is 
not a reliable indicator of type; the mapping from types to extensions 
is many-to-many...

> At this time it is (may not) not reliable to rely upon the 
> served ContentType so this cannot be used to reject the file. 

On the contrary, we should in fact be relying on the served 
Content-Type.  Anything else means the server has no control over how we 
handle the content, which can be a huge security problem for the server...

> but I don't think this would give the "standard" handler any additional 
> info.

Sure it would.  It'd tell it which of a number of container formats or 
codecs is being used, if done correctly.

> Literally, the handler should be able to handle any .wav or .ogg 
> spec files.

That's not very reasonable for .wav (see the earlier mention in this 
thread of the number of diffever WAVE codecs out there).  And it's 
completely unreasonable for .ogg, since that's just a container 
indicator.  Anyone can make up a new codec and stick it in a .ogg.

> The handler may need to do some data-sniffing to get set up. 
> but all it needs from the UA is the content.

I think you're drawing the line between "handler" and "UA" very 
arbitrarily here... and I can't tell exactly where you're doing it.  Can 
we please define _very_ clearly where the line is being drawn?

> and file name correct for the element and it should just work.

The load on the author should be pointing to the right URI; after that 
it should Just Work assuming the server and UA don't screw up.

> The file is there with the right extension but maybe I can't get control 
> of the server

Sorry, then you have a buggy server.

> and its defaults are not sufficient for the UA sniffer 
> (like maybe I can't use htaccess).

Again, sounds like you have a buggy server.

> Is my document success going to 
> depend on the fact that the server must be configured to send the 
> specified MIME? 

Yes.  It generally is already; why should this case be different?

> If it is served wrong or even absent a MIME am I going 
> to be saved by the UA data-sniffing to figure out that the misserved 
> file is really OK?

Under the current proposal, no.  If it's served wrong or with no MIME 
type, you will get no video.  Then you will realize that you need to fix 
your server setup.

> Are we going to see this with these elements?

I don't see why not.  We see it all the time now with objects, iframes, 
etc...

> We have already negotiated 
> the content model down to a simple set. Many times those query forms are 
> built from scripts that have determined or are determining some client 
> capabilites?

No, often enough they're just built from web pages that give the user a 
list of choices and ask him to pick one, then just have one script on 
the server that reads the data from somewhere (database, filesystem, 
whatever) and returns it.

> The application and content types of these elements seems 
> quite clear as defined by the kestrokes that actually appear in the 
> element code at run time. .

I have no idea what you're trying to say here.

-Boris

---

From: Boris Zbarsky <[email-masked]>
Date: Thu, 18 Jun 2009 20:26:40 -0400

David Singer wrote:
> influence my hosting provider to provide the correct MIME types.

How likely is this inability to influence if browsers in fact do not sniff?

Quite honestly, it seems to me that such behavior on the part of the 
provider would be breach of contract if the contract is at all 
reasonably written, and that if browsers don't bend over to make it work 
the provider won't be able to say "but look, it's not our fault; it 
works in browser X" as they tend to do now.

-Boris

---

From: "Joe D Williams" <[email-masked]>
Date: Thu, 18 Jun 2009 18:53:11 -0700

Joe >> Given that I am looking at it from an author side, how about if 
the
>> top-level sniffing before it goes to the renderer
Boris > To which renderer?

sorry, in the case of <audio> it would be whatever derives what is 
supposed to drive the speakers.
There, I meant renderer to mean the player or whatever that friendly 
thing behind the <audio> element is: the thingee that plays me.ogg and 
knows literallyl nothing else about anything but .ogg and .wav.:)

>> is called the UA, which contains a nested browsing context.<video> 
>> or <audio> element. So in these cases I will call the browser the 
>> host UA and the <video> element the home of the handler which 
>> actually does the rendering.
>
> OK...  I'm not sure how you're separating these, exactly, but under 
> hte only reasonable split that comes to mind for me (the UA is what 
> fetches the data while the handler is what actually renders it), you 
> need to know what kind of data you have so you can select the right 
> handler. Agreed, or not?  If not, is the problem one of disagreement 
> on the definitions, or on the conclusion?

No, in my little fantasy of this case, there is only one renderer for 
.ogg and .wav and that is all the host browser knows about for 
<audio>. There is no choice, If I <audio my.ogg> all I care about is 
that the host browser picks the 'built-in'  or native one that only 
knows about these file types.

>> I would like to say that data-sniffing is not needed. Can the 
>> target files contain any security problems?
>

What can me.ogg do to corrupt the system using the simple native 
player. ogg or .wav does not have and active form, I think.

> Of course not.  Data sniffing doesn't look for MIME strings; it 
> looks for initial bytes identifying the file type.  Many common file 
> types must start with a particular byte sequence.

'I see in the general sniffing algoritm that the file is read to look 
for the internal contenttype in the header. If it finds this, then 
that is the mime.

Yes, you can find the data structures in the detailed specs for the 
streams but lookie here mr browser maker, I don't want to fool with 
all that stuff. I put me.ogg out there on my server somehow and then I 
have an HTML 5 web page that says play what I put out there, please. 
The browser should let this happen with  no built-in security or other 
delays except perhaps some user preferrence settings about multiple 
media, on the level of a show images choice somdwhere.

And even that is OK as long as you give me me fallback content while 
you are asking questions.

> <video>.  If using a <source> element, a type hint 
> (non-authoritative) can be provided; this is to be used only for 
> deciding whether to load the <source> at all, not for what to do 
> with it after loading it.

Oh, I will look at that. If type is there, that should be clue enough 
to start the player and start fetching the content

There should be no choice. <video> might be a bit different than 
<audio> although I better look more closly because why aren't they the 
same?
but ok.
Anyway, the correct thing to do is for the host browser not to worry 
about it all. Just send the url to the player behind the <audio> or 
<video> element and let it figure it out and when it is ready, let it 
ask you for the content by url and really, just let the handler deal 
witht the loading. I would hope the host has better things to do than 
spoonfeed content to the audio or video handler.

>
>> If the type and file extension do not match the content model for 
>> the element then fallback.
>
> Uh... What content model?  And how did extension come in?  Extension 
> is not a reliable indicator of type; the mapping from types to 
> extensions is many-to-many...

'OK, let's make it like that in the spec. Right now are listing the 
acceptable file types identified mainly by file extension. Maybe 
listing MIME first, then. But heck as I see the web from my text 
editor, I know I must know about file extensions first. I see a list 
of allowable ones and I use those. If you tell me the mime type, I 
will use that also. But really, I mainly gotta care about the file 
extension first.

>
>> At this time it is (may not) not reliable to rely upon the served 
>> ContentType so this cannot be used to reject the file.
>
> Content-Type.  Anything else means the server has no control over 
> how we handle the content, which can be a huge security problem for 
> the server...

That is very enlightened and forward looking. However in this case 
where all I want to do is play me.ogg I don't really care about all 
that. I know what I put out there. Trust me.
Anyway, with these file extensions, what is the problem? If I can feed 
some string of bits from me.ogg after using an appropriate MIME in my 
user code and and a retrievable file of me.ogg data from the server 
with whatever contenttype and feed it directly to the hanler for 
<audio> element and cause a problem then the problem is not mine, but 
the browser makers who hasn't given me a safe place to play me.ogg.

>> Data-sniffing of file internal structures by the UA could take 
>> place, but I don't think this would give the "standard" handler any 
>> additional info.
>
> Sure it would.  It'd tell it which of a number of container formats 
> or codecs is being used, if done correctly.

OK, I must step back a notch and focus just on <audio>.
No problem for audio, right. Send the url or the file content as 
instructed in the user code no sniffing needed because we trust the 
author to have done the right thing. Aftera ll, it must have worked at 
home, so it must be OK out there in the WWW.

For <video> there are also a few specifc forms that we will allow, but 
we just don't know them now. In the greater scope than <video>. LIke 
when I open to all the big wonderful world, then yes there are alot of 
formats out there and some that can be dangerous.

> indicator.  Anyone can make up a new codec and stick it in a .ogg.

I willhave to look at that in more detail. Are you saying there are 
.wev files out there that this simple player is not going to play. 
Please show me.

As for the ogg, I will look more closely at that one.

However, in either of these, there is nothing that would give the 
player the idea that there was some script it was supposed to execute 
or anything wlse hazardous.

>> The handler may need to do some data-sniffing to get set up. but 
>> all it needs from the UA is the content.
>
> arbitrarily here... and I can't tell exactly where you're doing it. 
> Can we please define _very_ clearly where the line is being drawn?

For me it is easy to draw the line: I'm just the simple little <audio> 
player. I know I don't do much, but give me a chance. Host, get out of 
the way and let me do it.
For these file types, give me the bits straight for the server as fast 
as you can and I will handle it.

>> Again, the load on the author should be no more than getting the 
>> type and file name correct for the element and it should just work.
>
> The load on the author should be pointing to the right URI; after 
> that it should Just Work assuming the server and UA don't screw up.
>

Great. But how is the server going to screw it up. I ask for the file 
and he sends it, but uhoh he sends the wrong contenttype. Maybe not, 
maybe he is up to date or the webmaster has used .htaccess corretly. 
For <audio> why do I care?

>>> 2) Sniffing the file extension instead of the content 
>>> significantly complicates server-side solutions that, say, send 
>>> different content based on query params in the URL.  Of course 
>>> these should arguably send a Content-Type header anyway.
>>
>> control of the server

Sorry, I don't. My htaccess is just fine, thank you. Somebody else may 
be torched because the mime has not passed ietf or whoever and/or the 
server has not been updated. Do I gotta take the wrap for that?

>
>> and its defaults are not sufficient for the UA sniffer (like maybe 
>> I can't use htaccess).
>

I will ask more nicely, then.

>> Is my document success going to depend on the fact that the server 
>> must be configured to send the specified MIME?
>

If it was, you wouldn't need to sniff for that. Sure, as we see, 
mostly a user can manage to get .html served correctly as text/html or 
whatever the recommendation.

>
>> If it is served wrong or even absent a MIME am I going to be saved 
>> by the UA data-sniffing to figure out that the misserved file is 
>> really OK?
>
> Under the current proposal, no.  If it's served wrong or with no 
> MIME type, you will get no video.  Then you will realize that you 
> need to fix your server setup.

Maybe I will be luckler with <audio> and you won't try to oursmart me 
as a author and user.
But how do I fix my server?

I pick a file extension, like .x3d for instance, then I tell my server 
(maybe via .htaccess) to serve that file extension as model/x3d+xml.
Simple but effective and sure there are other ways to do it by 
standadrd bulletin from ietf but that is it. You figure out and 
designate which file extension you want to seved with which MIME.

Why not fix that in the host browser by me telling it that if you see 
a .x3d always treat it as being served as model/x3d+xml. Well, I 
already can sort of do that in some browsers. But really, from the 
server, as far as I know, it keys on file extension. Sure there are 
some fancy extensionless negotiation concepts out there, but that is 
complex stuff and this is easy.

> I don't see why not.  We see it all the time now with objects, 
> iframes, etc...

Some of those may be fallback content, or the <audio> may be fallback 
content for object to get audio playing that wouldn't in an object 
that asked for "house player" (not the <audio> player because the UA 
did so much sniffing to protext the house player that it wouldn't go. 
Now, for the fallback, we pick the simple littl ogg/wav player that 
always just works because I tested it at home and it worked so it must 
be ok out there on the World Wild Web.

>
>> We have already negotiated the content model down to a simple set. 
>> Many times those query forms are built from scripts that have 
>> determined or are determining some client capabilites?
>
> No, often enough they're just built from web pages that give the 
> user a list of choices and ask him to pick one, then just have one 
> script on the server that reads the data from somewhere (database, 
> filesystem, whatever) and returns it.

Well, I think when you ask the sever for what to send, you will ask 
something in a way that ends up with a file name and certain 
extension. Then, once the server knows what file it is going to send, 
it looks up the mime that is supposed to be in the head stuffs.

But even so, I don't think you are going to have success with that 
scale of use for these simple elements. What you describe will work I 
guess, but that sort of more side open stuff may work better with 
<object> and a more full-featured player.

>
>> The application and content types of these elements seems quite 
>> clear as defined by the keystrokes that actually appear in the 

Basically, please give me what I am asking for.
A simple way to play <audio> and <video> from a very limited list of 
input fomats. I shouldn' have to promise not to break anything because 
they should be bulletproof if the file is within the specs.

> -Boris

Thanks, Boris,
Joe

---

From: Boris Zbarsky <[email-masked]>
Date: Thu, 18 Jun 2009 23:02:55 -0400

> knows literallyl nothing else about anything but .ogg and .wav.:)

OK.  So you don't mean the two different things one of which knows 
nothing about anything but .ogg and the other of which knows nothing 
about anything but .wav.  Gotcha.

So the "renderer" in your definition still needs to decide whether to 
handle the data as an OGG or a WAV.  With this definition, the UA does 
not indeed need to know anything about the kind of data it's handing the 
renderer.  The renderer does need to know what sort of data it's looking 
at, though.

> .ogg and .wav and that is all the host browser knows about for <audio>. 
> There is no choice, If I <audio my.ogg> all I care about is that the 
> host browser picks the 'built-in'  or native one that only knows about 
> these file types.

The "native renderer" as you call it then has to delegate to two 
completely different subsystems for the OGG and WAV cases.  How do you 
envision it picking which one to use?

> What can me.ogg do to corrupt the system using the simple native player. 
> ogg or .wav does not have and active form, I think.

You're assuming a completely non-buggy player.  That's desirable, but 
unlikely in practice.

> for the internal contenttype in the header. If it finds this, then that 
> is the mime.
> 
> streams but lookie here mr browser maker, I don't want to fool with all 
> that stuff.

No one is asking you to.

> I put me.ogg out there on my server somehow and then I have 
> an HTML 5 web page that says play what I put out there, please.

Sure.  And the browser will take that data, and treat it as what you 
tell it that data is.  If you lie to it... then it'll still treat it as 
what you tell it that data is.  Ideally, at least.

> Oh, I will look at that. If type is there, that should be clue enough to 
> start the player and start fetching the content

What's "the player"?  OGG and WAV need different "players".

> There should be no choice.

Choice by whom of what?

> spoonfeed content to the audio or video handler.

What sort of media player are you using, exactly, that has a rich 
network stack built in?  In practice, it is exactly the browser's job to 
get the data and "spoonfeed" it to the player; the player just knows 
what to do with bytes, not with URLs.

> acceptable file types identified mainly by file extension.

Where, exactly?

> That is very enlightened and forward looking. However in this case where 
> all I want to do is play me.ogg I don't really care about all that. I 
> know what I put out there. Trust me.

Sure. We do! "You" in this case is the web server.  We'll handle the 
data based on what you tell us it is.

> Anyway, with these file extensions, what is the problem?

With which ones?

> If I can feed some string of bits from me.ogg after using an appropriate MIME in my 
> user code and and a retrievable file of me.ogg data from the server with 
> whatever contenttype and feed it directly to the hanler for <audio> 
> element and cause a problem then the problem is not mine, but the 
> browser makers who hasn't given me a safe place to play me.ogg.

Sure.  I'm still not sure what your point is...

> OK, I must step back a notch and focus just on <audio>.

OK...

> author to have done the right thing.

Uh...  The sniffing is just needed to pick the right player, in the case 
when the author hasn't bothered to tell us what kind of media he's 
sending.  Assuming sniffing is done at all.

> I willhave to look at that in more detail. Are you saying there are .wev 
> files out there that this simple player is not going to play.

Sure.  Firefox, or example, is shipping a WAV player in version 3.5 that 
plays only a subset of the WAV files out there. 
https://bugzilla.mozilla.org/show_bug.cgi?id=475110 has some example WAV 
files attached that will not play in that player.

> However, in either of these, there is nothing that would give the player 
> the idea that there was some script it was supposed to execute or 
> anything wlse hazardous.

I'm not sure what script has to do with this whole thing....

> as you can and I will handle it.

OK. So you are a chunk of code that will take a byte stream and the MIME 
type and then do the right thing.  Fine.

> and he sends it, but uhoh he sends the wrong contenttype.

Well, define "wrong".  Not the one the author intended?  The browser 
can't read your mind; it can't tell whether the server is sending what 
you intended or not.  We can only assume that it is or isn't.  I'm 
arguing we should assume it is.  Others want to assume it isn't...

> Maybe not, maybe he is up to date or the webmaster has used .htaccess corretly. For 
> <audio> why do I care?

Because you need to decide whether to play it as a WAV or an OGG?

> server has not been updated. Do I gotta take the wrap for that?

You're sort of responsible for what you put on the wire, yes.

> If it was, you wouldn't need to sniff for that. Sure, as we see, mostly 
> a user can manage to get .html served correctly as text/html or whatever 
> the recommendation.

Sure.  And CSS as text/css (and if he doesn't, a number of modern 
browsers won't treat it as CSS).  Why is this case different?

I can't tell what your point is, honestly....

> Maybe I will be luckler with <audio> and you won't try to oursmart me as 
> a author and user.

Right.  That's what I'm saying browsers should do: assume the 
author+server know what they're talking about and not try to 
second-guess them.

> But how do I fix my server?

That depends on the exact server you're using; I suggest contacting your 
hosting service (or server vendor if you configure your own server).

> (maybe via .htaccess) to serve that file extension as model/x3d+xml.

That's up to you.  Some servers use this model for configuration; others 
use other models.  Some servers support more than one approach to this 
problem (e.g. Apache).

> Why not fix that in the host browser by me telling it that if you see a 
> .x3d always treat it as being served as model/x3d+xml.

Because the host browser is not being run by you.  It's being run by the 
user...  Which means that you can't tell it anything directly.  All it 
knows is what the web server tells it.

> But really, from the server, as far as I know, it keys on file extension.

Some servers, sometimes, do that.  I can assure you that if you have a 
PHP script that serves an OGG file you do NOT want your server keying 
off the ".php" extension for the type.  Nor does it.

> Sure there are some fancy 
> extensionless negotiation concepts out there, but that is complex stuff 
> and this is easy.

And the point is that one shouldn't break people using the complex stuff.

> content for object to get audio playing that wouldn't in an object that 
> asked for "house player"

What does fallback content have to do with anything here?

>> No, often enough they're just built from web pages that give the user 
>> a list of choices and ask him to pick one, then just have one script 
>> on the server that reads the data from somewhere (database, 
> something in a way that ends up with a file name and certain extension.

That's really the server's business...

> Then, once the server knows what file it is going to send, it looks up 
> the mime that is supposed to be in the head stuffs.

Again, that's the server's business.

> But even so, I don't think you are going to have success with that scale 
> of use for these simple elements. What you describe will work I guess, 
> but that sort of more side open stuff may work better with <object> and 
> a more full-featured player.

I have no idea what you're talking about here, honestly.

> Basically, please give me what I am asking for.

What are you asking for?

> input fomats.

You as an author?  We're giving you that.  You put the tag in your HTML, 
point it to your file, send the file with the right Content-Type.

What's the problem you're trying to solve here, exactly?

-Boris

---

From: "Robert O'Callahan" <[email-masked]>
Date: Fri, 19 Jun 2009 15:33:08 +1200

On Fri, Jun 19, 2009 at 12:08 PM, David Singer <[email-masked]> wrote:

> influence my hosting provider to provide the correct MIME types.  I don't
> want users trapped into using the old because we've made the new
> inaccessible to them.
>

Out of all the things video hosting providers have to do, sending correct
Content-Types seems like one of the smaller ones.

I would like to see how big a problem this turns out to be before we embrace
sniffing.

Rob

---

From: "Joe D Williams" <[email-masked]>
Date: Fri, 19 Jun 2009 03:58:09 -0700

>> There, I meant renderer to mean the player or whatever that 
>> friendly thing behind the <audio> element is: the thingee that 
>> plays me.ogg and knows literallyl nothing else about anything but 
>> .ogg and .wav.:)

> about anything but .wav.  Gotcha.

Yes, my depth of vision into the process is text of the <audio> 
element in this case. I neither know about nor care about the 
machinery behind it. All I have is a list of file extensions and MIMEs 
that are acceptable. I dutifully prepare my files with that 
information in my user code.

> So the "renderer" in your definition still needs to decide whether 
> to handle the data as an OGG or a WAV.  With this definition, the UA 
> does not indeed need to know anything about the kind of data it's 
> handing the renderer.  The renderer does need to know what sort of 
> data it's looking at, though.

I told it what the file extension and type. I got them from the spec 
list then I made my content in some tool that gave me the right file 
extension. I put it the file on the server. What else does it need to 
know?

>> No, in my little fantasy of this case, there is only one renderer 
>> for .ogg and .wav and that is all the host browser knows about for 
>> <audio>. There is no choice, If I <audio my.ogg> all I care about 
>> is that the host browser picks the 'built-in'  or native one that 
>> only knows about these file types.

> completely different subsystems for the OGG and WAV cases. How do 
> you envision it picking which one to use?

I told it the file extension and the type in genuine HTML 5 
keystrokes. What do I have to do, hold its hand?

> You're assuming a completely non-buggy player.  That's desirable, 
> but unlikely in practice.

That is too bad and not really acceptable at introduction. Seems like 
all the browser makers should join forces on this one and make it 
work. It is the right thing to do.

>> 'I see in the general sniffing algoritm that the file is read to 
>> look for the internal contenttype in the header. If it finds this, 
>> then that is the mime. Yes, you can find the data structures in the 
>> detailed specs for the streams but lookie here mr browser maker, I 
>> don't want to fool with all that stuff.

> No one is asking you to.

Good. I need a break.

>> I put me.ogg out there on my server somehow and then I have an HTML 
>> 5 web page that says play what I put out there, please.

> tell it that data is.  If you lie to it... then it'll still treat it 
> as what you tell it that data is.  Ideally, at least.

I won't lie, at least I will really try. If I lie, maybe it shouldn't 
work but I don't think you are going to find that out by looking at 
the served MIME.

>> The browser should let this happen with  no built-in security or 
>> other delays except perhaps some user preferrence settings about 
>> multiple media, on the level of a show images choice somdwhere.

> I have no idea what you're trying to say here.

I mean such as a warning or caution that hey lookout we are starting a 
player that could damage your system. In this web the <audio> and 
<video> are first class and are rendered just like other first class 
content like <p> and <em> and as easy to use if you stay in the spec.

>> Oh, I will look at that. If type is there, that should be clue 
>> enough to start the player and start fetching the content
>

Another implementation detail There is no <audioogg ...> <audiowav 
...>, the player is whatever the UA sets up for the <audio> element 
which may be requested to play some wav or hopefully some ogg.

"4.8.8.1 Audio codecs for audio elements

User agents may support any audio codecs and container formats.

User agents must support the WAVE container format with audio encoded 
using the 16 bit PCM (LE) codec, at sampling frequencies of 11.025kHz, 
22.050kHz, and 44.100kHz, and for both mono and stereo. [WAVE]"

I think it is more reasonable to drop the first sentence and update 
this spec as the proven cross-browser capability increases. Ogg next. 
Also, what is the need to restrict the kHz? wav also does 12/24/48/96, 
and ogg does a fixed Q or variable rates. and all rates should work 
fine, I think.

>> There should be no choice.

I think that had something to do with <video>

>> Anyway, the correct thing to do is for the host browser not to 
>> worry about it all. Just send the url to the player behind the 
>> <audio> or <video> element and let it figure it out and when it is 
>> ready, let it ask you for the content by url and really, just let 
>> the handler deal witht the loading. I would hope the host has 
>> better things to do than spoonfeed content to the audio or video 
>> handler.
>
> network stack built in?  In practice, it is exactly the browser's 
> job to get the data and "spoonfeed" it to the player; the player 
> just knows what to do with bytes, not with URLs.

OK, I guess this player shouldn't need to go get URLs by itself. Maybe 
spoonfeeding it is best, but I really don't know. Yes, I do get to 
play with a couple of 'em that expose a very rich network stack in 
some environments. Nothing really proprietaryd, though. .

> Where, exactly?

Well, now that I look again, it seems like <audio> can be anything 
(must have wav) and <video> can be anything so hey, I guess browser 
makers will just have to keep doing what they were doing in the old 
days before HTML 5.  I hope it works out better.

> data based on what you tell us it is.

That is fine. If the server is right.

> With which ones?

Well, for <audio> the only thing we tell them is that at least certain 
wav is OK.
Anything else is browsers dependent.

> Uh...  The sniffing is just needed to pick the right player, in the 
> case when the author hasn't bothered to tell us what kind of media 
> he's sending.  Assuming sniffing is done at all.

>> I will have to look at that in more detail. Are you saying there 
>> are .wev files out there that this simple player is not going to 
>> play.

> Sure.  Firefox, or example, is shipping a WAV player in version 3.5 
> that plays only a subset of the WAV files out there. 
> https://bugzilla.mozilla.org/show_bug.cgi?id=475110 has some example 
> WAV files attached that will not play in that player.

Great saga. Looks like the right goals. Where is ogg?
That should be the <audio> player for everyone. All you browser makers 
get that thing working right and turn it loose as a simple handler 
that works in <audio> everywhere for that certain set of required 
support files. Certain wav(s) but if you want anything else, go to 
'house' player which the user may need to get from someplace like QT 
or Win or Real and use <object>. That is the secret to doing something 
good with <audio>  -  a limited easy to use player that works with 
these file types in all the browsers.

>> player the idea that there was some script it was supposed to 
>> execute or anything else hazardous.

> I'm not sure what script has to do with this whole thing....

Right, I meant a hidden script in the <audio> content. That shouldn't 
have anything to do with it because it can't happen.

>> For me it is easy to draw the line: I'm just the simple little 
>> <audio> player. I know I don't do much, but give me a chance. Host, 
>> get out of the way and let me do it. For these file types, give me 
>> the bits straight for the server as fast as you can and I will 
>> handle it.

> OK. So you are a chunk of code that will take a byte stream and the 
> MIME type and then do the right thing.  Fine.

Not just any one. Only the byte stream(s) required by the spec. I just 
know that if you put a wav in an <audio> I will get picked first.

>> Great. But how is the server going to screw it up. I ask for the 
>> file and he sends it, but uhoh he sends the wrong contenttype.

> can't read your mind; it can't tell whether the server is sending 
> what you intended or not.  We can only assume that it is or isn't. 
> I'm arguing we should assume it is.  Others want to assume it 
> isn't...

It is, it is. It is what I told you it was going to be in my <audio> 
or <source> statement.
If the server does not agree, the fact that the server doesn't agree 
with the user request (I think I requested it by file name and 
extension) that is just a sad coincidence

>> Maybe not, maybe he is up to date or the webmaster has used 
>> .htaccess corretly. For <audio> why do I care?

> Because you need to decide whether to play it as a WAV or an OGG?

So the host is going to tell me here comes a wav? If I can only do 
wav, i think I can figure out what to do when I start getting the 
stream.

>>  Somebody else may  be torched because the mime has not passed ietf 
>> or whoever and/or the server has not been updated. Do I gotta take 
>> the wrap for that?

> You're sort of responsible for what you put on the wire, yes.

Well, I want to be. That is why I do it, I guess

>> ... as we see, mostly a user can manage to get .html served 
>> correctly as text/html or whatever the recommendation.

> I can't tell what your point is, honestly....

I think it was about how sniffing can help solve problems, mainly 
security issues.

> That depends on the exact server you're using; I suggest contacting 
> your hosting service (or server vendor if you configure your own 
> server).

>> I pick a file extension, like .x3d for instance, then I tell my 
>> server (maybe via .htaccess) to serve that file extension as 
>> model/x3d+xml.

> That's up to you.  Some servers use this model for configuration; 
> others use other models.  Some servers support more than one 
> approach to this problem (e.g. Apache).

Above. linking up a mime with a file extension is one way to do it for 
Apache. specifically, like
AddType model/x3d+xml .x3d

>> Why not fix that in the host browser by me telling it that if you 
>> see a .x3d always treat it as being served as model/x3d+xml.
>
> Because the host browser is not being run by you.  It's being run by 
> the user...  Which means that you can't tell it anything directly. 
> All it knows is what the web server tells it.

Well, I meant in some W3C browsers that I personally use I can sort of 
experiment around in some of that.

> Some servers, sometimes, do that.  I can assure you that if you have 
> a PHP script that serves an OGG file you do NOT want your server 
> keying off the ".php" extension for the type.  Nor does it.

I am glad. I hoped I would never see or hear of anything like that 
again.
But really,  the file reqested from somewhere, maybe home, as .ogg. If 
home did not request it and does not know which guest is coming, then 
maybe the served ct will tell, or maybe the file meta ct will tell. If 
the request is from home, you may know more about what to expect back. 
Please note that if that request ends up wanting a new audio([src]) 
then wherefore the MIME?

This is why the types of files played at intro of this feature has to 
be a known subset of the avworld with some good hints about creation 
tool config.

Another message close to this advised trying to find out what the 
actual situation is out there with the server settings.
So, if I give the UA
<audio autobuffer autoplay><source src='me.ogg' type='audio/ogg' 
 ></audio>
are you going out the the server to look for MIME or name.extension?

The sniffing routine as shown in the referenced barth-mime-sniffing 
text is fine and it mentions looking for the the content type coming 
from the server with the content and also possibly looking inside the 
file at the meta content type string. Maybe I should look again, but 
the step of looking inside in that way is generally not needed for 
<audio> and <video> media types because something different but 
equally identfiable is probably in there.
This does remind me that in X3D we had decided not to include a 
requirement or even suggestion to include a specific meta content type 
declaration. Maybe because we did not have an approved MIME at that 
time. With this mime-sniffing algorithm that finally gets down into 
the internal meta, then that might be the last chance to get on board 
if the server ct messes up.
Anyway, I have seen mimes at work. All browsers are using them better 
and better all the time. I think I even know in a special example, how 
to test whether or not the W3C browser even cares or knows. 
Fortunately, bad or non-existent behavior is diminishing.

What I probably want to do is start it like this:
<audio autobuffer autoplay><source type='audio/ogg' ></audio>
and wait to hear back from the DOM when the handler is up and running 
and ready to be sent a src URL. At this point, the handler still may 
not know what it is going to play, it is just ready with currentSrc; 
empty (or null) and HAVE_NOTHING. When it is ready and I am ready, 
then maybe try to send it something to play. Then the player will 
somehow ask its parent context for some content by name URL; does it 
need to include a type in the request?

So as I look at the HTMLMediaElement in more detail I remember again 
how much fun this multiple media stuff really is. It should be as 
simple and reliable as me filling a div with new text. Looks like all 
the controls and stati are in there and it only remains to actually be 
able to stuff the thing with some content. By golly I might be ready 
to actually look around and find an <audio> element to work on.

Thanks Again and Best Regards,
Joe

---

From: David Singer <[email-masked]>
Date: Fri, 19 Jun 2009 09:31:35 -0700

At 15:33  +1200 19/06/09, Robert O'Callahan wrote:
>On Fri, Jun 19, 2009 at 12:08 PM, David Singer 
>Out of all the things video hosting providers have to do, sending 
>correct Content-Types seems like one of the smaller ones.
>
>I would like to see how big a problem this turns out to be before we 
>embrace sniffing.

we seem to have come full circle here. what we're saying is that our 
experience with the plugin content is that it's non-trivial (in 
excess of 10%).  Whether the service providers would have fixed it if 
we *didn't* provide sniffing, I don't know, but they seem remarkably 
resistant to pressure...
-- 
David Singer
Multimedia Standards, Apple Inc.

---

From: "Joe D Williams" <[email-masked]>
Date: Fri, 19 Jun 2009 11:56:50 -0700

>  ... they seem remarkably resistant to pressure..

The best way is to show them exactly what you wish to be done.
One step in the cycle is getting the servers to provide an expected or 
at least approriate content type with the content.
I think in some servers this may be done by some level of automation 
that follows the main MIME list. In general, the community process 
keeps track of mimes that people and organizations are interested in 
registering for the maintained list. Then, I think servers are free to 
update themselves re the 'official' list. Usually it seems to take a 
while longer for the mime to be offical than for content to be 
published. In this time, the server guy can easily configure the 
server to deliver most any mime for any file extension as the web 
author wishes.

Browsers know about MIMEs and the connection with file extensions or 
other content model definitions when some are native and some are 
added by all sorts of web features and applications that get loaded 
into people's computers. I think that when content publishers are 
convinced that the W3C host browsers care and are actually using the 
things to add some production value (or even security and reliability) 
they will be responsive. But these are  seasoned producers and you may 
have to go there and tell tell tell them using actual representative 
working examples of flexible and highly reusable cross-browser 
behaviors using HTML 5 code and content.

For instance, I think text/html uses the mime definition to help 
because the mime helps predict that the sniffer may find a content 
type string somewhere up close to the beginning. Tool makers and 
authors use special and standard mimes, extensions to mimes, and file 
extensions for many reasons, even to distinguish 'public' from 
'special' content.

Thank You and Best Regards,
Joe
"There is good stuff built into the MIME scheme of doing things that 
we don't even know about anymore."

---

From: "Robert O'Callahan" <[email-masked]>
Date: Sat, 20 Jun 2009 10:07:08 +1200

On Sat, Jun 20, 2009 at 4:31 AM, David Singer <[email-masked]> wrote:

> experience with the plugin content is that it's non-trivial (in excess of
> 10%).  Whether the service providers would have fixed it if we *didn't*
> provide sniffing, I don't know, but they seem remarkably resistant to
> pressure...
>

And I'm saying that our experience is different. Hence the impasse.

Rob

---

From: "Joe D Williams" <[email-masked]>
Date: Mon, 22 Jun 2009 07:55:23 -0700

Hi Adam,

> suggesting we sniff the media type from the file extension for 
> audio/video content?

I have done some wandering on this topic and in this thread, but to me 
an important point is that I am still thinking in terms of audio and 
video as an embedded (nested) or auxiliary context. That is, the audio 
or video might be embedded content using the HTML 5 <audio> or <video> 
element, and that the UA will open an auxiliary context if a standard 
hyperlink with an audio or video file extension as @href is selected. 
Either way, I get access to the host DOM through interfaces granted by 
the host context.

What does the host, or source, context need to know about the served 
content type and encoding and maybe contents of the file in order to 
send it to the <audio> or <video> requester, or to the auxiliary 
context?

If we knew that, then the mime-sniffing table used to find Sniffed 
Type by the Unknown algorithm might get longer, or maybe more entries 
in Image, or better yet, an added main title for Audio and Video 
objects. I think this is where a real contribution will be made by 
HTML 5, is when there is a standard or highly recommended (evolving) 
set of mimes that will play 'standard' audio and video (@src, @type, 
@codecs) in W3C HTML 5 UAs. This  list needs to be readily available 
to web authors.

For example, for audio media, I have commented that the spec should 
not recommend 'any audio codecs' but instead state a selected subset 
of available content types/file extensions as 'native' to the <audio> 
element. I think it would be best to just say current typical .wav and 
Ogg Vorbis (.oog) - if those are as open and free and available as we 
believe and hope. I wish I knew more about this and thus aid my point 
of view here with conclusive argument that allowing 'any' 
container/codec in the <audio> node is not an improvement in what was 
already there in <object>. Well, fine that the media style interfaces 
are being specified in <audio> and <video>; that is a great step 
forward for simplified implementation-independent authoring, for sure. 
However, if there are no 'native' audio media types in HTML 5 then it 
is just like it was where the author gets to depend upon the client 
determination of which proprietary media player system (legacy audio 
plugin) gets used. Basically, that just means a level of inconvenience 
for everyone.

Also, looking at:  [ myemphasis ]  DIFF1

Metatdata
... only the [first last] Content-Type HTTP header, ...
http://www.ietf.org/internet-drafts/draft-abarth-mime-sniff-01.txt
May 31, 2009
and
http://tools.ietf.org/html/draft-abarth-mime-sniff-00
January 9, 2009

Adam, please update the link on your site with the latest draft?
I know now which one to read = 01 but it distracted me for a while:)

The choice of which of multiple content type headers to use seems 
important and interesting.
Is there some background on the final choice of last/first?

ps. can that entry down at the bottom of that unknown type table:| FF 
FF FF          | 49 44 33          | audio/mpeg      | Safe       |
| Comment: The string "ID3", the MP3 signature.have anything to do 
with this conversation?

---

From: Adam Barth <[email-masked]>
Date: Mon, 22 Jun 2009 12:08:25 -0700

2009/6/22 Joe D Williams <[email-masked]>:
> I have done some wandering on this topic and in this thread, but to me an important point is that I am still thinking in terms of audio and video as an embedded (nested) or auxiliary context. That is, the audio or video might be embedded content using the HTML 5 <audio> or <video> element, and that the UA will open an auxiliary context if a standard hyperlink with an audio or video file extension as @href is selected. Either way, I get access to the host DOM through interfaces granted by the host context.
>
> What does the host, or source, context need to know about the served content type and encoding and maybe contents of the file in order to send it to the <audio> or <video> requester, or to the auxiliary context?

The media type.  That's the purpose of the media type: to give
semantics to the bytes coming over the network.

> If we knew that, then the mime-sniffing table used to find Sniffed Type by the Unknown algorithm might get longer, or maybe more entries in Image, or better yet, an added main title for Audio and Video objects. I think this is where a real contribution will be made by HTML 5, is when there is a standard or highly recommended (evolving) set of mimes that will play 'standard' audio and video (@src, @type, @codecs) in W3C HTML 5 UAs. This  list needs to be readily available to web authors.
>
> For example, for audio media, I have commented that the spec should not recommend 'any audio codecs' but instead state a selected subset of available content types/file extensions as 'native' to the <audio> element. I think it would be best to just say current typical .wav and Ogg Vorbis (.oog) - if those are as open and free and available as we believe and hope. I wish I knew more about this and thus aid my point of view here with conclusive argument that allowing 'any' container/codec in the <audio> node is not an improvement in what was already there in <object>. Well, fine that the media style interfaces are being specified in <audio> and <video>; that is a great step forward for simplified implementation-independent authoring, for sure. However, if there are no 'native' audio media types in HTML 5 then it is just like it was where the author gets to depend upon the client determination of which proprietary media player system (legacy audio plugin) gets used. Basically, that just means a level of inconvenience for everyone.

The codec issue is indeed thorning, but thankfully separate from this
discussion.

> I know now which one to read = 01 but it distracted me for a while:)

Thanks.  I'll update my site.

> The choice of which of multiple content type headers to use seems important and interesting.

Yes.  Please see the discussion on the WHAT WG mailing list.

> ps. can that entry down at the bottom of that unknown type table:| FF FF FF          | 49 44 33          | audio/mpeg      | Safe       |
> | Comment: The string "ID3", the MP3 signature.have anything to do with this conversation?

I've removed this from my working draft.

Adam

---

From: Ian Hickson <[email-masked]>
Date: Tue, 30 Jun 2009 11:11:05 +0000 (UTC)

> consider anything that is not video/* or audio/* to be not supported?

The way the spec is phrased, yes.

On Tue, 16 Jun 2009, Jamie Lokier wrote:
> 
> For video, content negotiation is probably going to end up being done in 
> Javascript rather than HTTP, or by User-Agent recognition.

HTML5 actually has a built-in way of doing fallback for <video>.

On Tue, 16 Jun 2009, Anne van Kesteren wrote:
> 
> Still, if the user agent does a request, should it abort the request if 
> the media type of the response is not video/* or audio/*?

Assuming the type is one "that the user agent knows it cannot render", 
yes.

On Tue, 16 Jun 2009, Jamie Lokier wrote:
> 
> I'm thinking if the response is an animated GIF, of type image/gif, why 
> forbid a <video> element from being allowed to play it as a video.

I believe the spec allows this.

> static videos.  After all, sometimes single-frame video/* files (such as 
> an MPEG single I-Frame) are sometimes created for this purpose on real 
> video players.  It would seem odd to allow the display of single-frame 
> video files, but forbid image/* files from behaving exactly the same 
> way.

I believe a browser is allowed to do this as currently written.

---

From: Ian Hickson <[email-masked]>
Date: Tue, 30 Jun 2009 11:17:21 +0000 (UTC)

On Tue, 16 Jun 2009, David Singer wrote:
> We're concerned about this, though we support the hope that we can get 
> away from doing content-type sniffing in this area.
> 

There are many requirements in the spec that require that libraries do 
particular things. If the library you use doesn't do the right thing, you 
will either need to fix it or reimplement it. (This is why some browsers, 
for instance, don't use their system HTTP stack -- it's often the case 
that the system's HTTP stack isn't standards-compliant enough.)

> config. of their server (often, their ISP's server).  Getting the MIME 
> type table updated can take ages.

Indeed.

> usually follow.  It's hard to see why this would not happen in this 

Once it does, I guess we'll define how the sniffing must occur.

---

From: Ian Hickson <[email-masked]>
Date: Tue, 30 Jun 2009 11:23:09 +0000 (UTC)

> start sniffing for <video> / <audio>.

This doesn't appear to be the case with the way the spec is phrased now. 
Can you confirm that this problem is now fixed?

---

From: Boris Zbarsky <[email-masked]>
Date: Tue, 30 Jun 2009 08:54:22 -0700

> Can you confirm that this problem is now fixed?

I would be glad to, if I could find the right things to read.  The HTML5 
draft has MIME sniffing removed and replaced with the [MIMESNIFF] 
reference (not defined anywhere that I can see).

Or is the relevant text somewhere in the 
http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html#concept-media-load-resource 
section?

-Boris

---

From: Ian Hickson <[email-masked]>
Date: Tue, 30 Jun 2009 21:40:51 +0000 (UTC)

On Tue, 30 Jun 2009, Boris Zbarsky wrote:
> > This doesn't appear to be the case with the way the spec is phrased now. Can
> > you confirm that this problem is now fixed?
> 
> I would be glad to, if I could find the right things to read.  The HTML5 draft
> has MIME sniffing removed and replaced with the [MIMESNIFF] reference (not
> defined anywhere that I can see).

The MIME sniffing section should hopefully be irrelevant in this case; 
does the video section refer to it somehow? If so I should fix that.

> http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html#concept-media-load-resource

That is the relevant algorith, yes; specifically the second and third 
cases in the first set of cases in the list in step 3.

---

From: Boris Zbarsky <[email-masked]>
Date: Tue, 30 Jun 2009 15:27:13 -0700

> cases in the first set of cases in the list in step 3.

OK, then agreed that this allows sniffing of the video data if the type 
is not provided.

Note that Gecko, at least, sniffs all HTTP responses without a 
Content-Type header, and after that treats them all effectively as 
metadata-provided.  It sounds like doing that is fundamentally 
incompatible with this specification, though, unless we teach our 
sniffer to recognize video/audio types by data, not just by extension...

-Boris

---

From: Ian Hickson <[email-masked]>
Date: Tue, 30 Jun 2009 23:52:53 +0000 (UTC)

> sniffer to recognize video/audio types by data, not just by extension...

Generally I would encourage you to sniff based on data and not extension 
for anything sent over HTTP anyway.

---

From: Boris Zbarsky <[email-masked]>
Date: Tue, 30 Jun 2009 17:01:19 -0700

> for anything sent over HTTP anyway.

In that case we'd typically sniff the videos as application/octet-stream.

Which raises an interesting question.  If a <video> points to data that 
has Content-Type metadata that says it's application/octet-stream.... 
I'd say that the UA should ignore that and look at the actual data just 
like it would if the MIME type is not set.

-Boris