Subject: [Cellar] Storage of WebVTT subtitles in Matroska

From: Moritz Bunkus <[email-masked]>
Date: Mon, 14 Mar 2016 11:24:41 +0100

Hey,

I'm currently looking into storing WebVTT subtitles[1] in Matroska due
to user requests and Youtube's usage of WebVTT. The format is similar
to SRT but unfortunately much more powerful and therefore not as
straightforward to store.

Things we have to specify how to store include:

1. The actual CodecID and encoding
2. Codec private data (all global blocks like STYLE, REGION and NOTEs
   appearing before the first entry)
3. NOTE blocks between entries
4. tags/numbers preceding an entry
5. cue settings
6. cue timestamps in entries

Here's an example including all of the aforementioned problems:

------------------------------------------------------------
WEBVTT

STYLE
::cue {
  background-image: linear-gradient(to bottom, dimgray, lightgray);
  color: papayawhip;
}
/* Style blocks cannot use blank lines nor "dash dash greater than" */

NOTE comment blocks can be used between style blocks.

STYLE
::cue(b) {
  color: peachpuff;
}

REGION
id:bill
width:40%
lines:3
regionanchor:0%,100%
viewportanchor:10%,90%
scroll:up

NOTE
Notes always span a whole block and can cover multiple
lines. Like this one.
An empty line ends the block.

hello
00:00:00.000 --> 00:00:10.000
Example entry 1: Hello <b>world</b>.

NOTE style blocks cannot appear after the first cue.

00:00:25.000 --> 00:00:35.000
Example entry 2: Another entry.
This one has multiple lines.

4
00:00:40.000 --> 00:00:45.000
Example entry 3: Entries can be numbered (like this one)…

00:00:46.000 --> 00:00:47.000
Example entry 4: …but don't have to be.

00:01:03.000 --> 00:01:06.500 position:90% align:right size:35%
Example entry 5: That stuff to the right of the timestamps are cue settings.

00:02:02.500 --> 00:02:22.500 region:bill align:right
Example entry 6: <v Bill>Hi, I’m Bill. I'm using a region defined above.

00:03:10.000 --> 00:03:20.000 region:bill align:right
Example entry 7: Entries can even include timestamps.
For example:<00:03:15.000>This becomes visible five seconds
after the first part.
------------------------------------------------------------

Implementation details:

1. CodecID and encoding

Easy enough. I propose S_WEBVTT and using UTF-8.

2. Codec private data

WebVTT files consist of blocks. Blocks are separated by blank
lines. Each WebVTT file starts with a block consisting solely of the
line "WEBVTT".

There are several blocks that can only occur before the first
entry. Examples of those blocks are "STYLE" or "REGION".

There are other blocks that may occur both before the first entry and
before any other entry. This is mostly the comment block "NOTE".

I propose to store all blocks that occur before the first entry in
CodecPrivate without modifying them. This includes the "WEBVTT" file
magic block.

3. NOTE blocks between entries

A comment block ("NOTE …") can appear between subtitle entries (see
above between example entries 1 and 2). We have to decide whether or
not we want to keep them when muxing into Matroska. If we do we need
to store them somehow.

I defer a proposal until I've mentioned the remaining points.

4. tags/numbers preceding an entry

Each entry always has a timestamp line, but that line may be preceded
by a line containing an entry number (example entry 3) or a free-form
tag of some kind (example entry 1). Those tags/numbers are irrelevant
for playback, but they may convey information to a person editing the
entries.

We have to decide whether or not we want to keep them when muxing into
Matroska. If we do we need to store them somehow.

I defer a proposal until I've mentioned the remaining points.

5. cue settings

These are the things listed after the timestamps (example entries 5
and 6). They are very relevant to playback and must be included,
either out of band or in band.

I defer a proposal until I've mentioned the remaining points.

6. cue timestamps in entries

Cue timestamps occur within an entry (example entry 7) and tell the
player to process parts of the entry only at a certain point in
time. Cue timestamps are absolute, unfortunately, and not relative to
the start of the entry itself. The example entry 7 contains two parts,
one that is shown at 3:10, and the second part that's shown five
seconds later.

Storing such absolute timestamps in-band makes manipulation at the
container level (e.g. applying some kind of delays or splitting and
joining files) very difficult.

I therefore propose that a muxer MUST change cue timestamps to be
relative to the start of the entry during muxing and that a demuxer
MUST change the cue timestamps back to be absolute during demuxing.
As several other modifications of each entry's content will be
necessary (see below) this should be OK.

Proposed entry storage format:

As can be seen above there are several pieces of information for each
entry that must be kept apart from the actual text to show. Matroska
currently doesn't provide block elements for these kinds of
information. The only element coming close to the purpose is
CodecState. The problem with CodecState is, though, that CodecState is
supposed to _replace_ CodecPrivate from the point of its occurrence in
the file. Therefore it cannot really be used for storing things like
"NOTE" blocks between entries or the entry tag/number lines.

I also favor keeping as much information as possible. This means
keeping the "NOTE" blocks between entries as well as keeping the
tag/number lines.

Therefore I propose to use the following storage format for entries:

1. (optional) all non-global blocks preceding an entry (the "NOTE"
   blocks); each block is followed by a blank line marking its end.

2. (optional) the tag/number line

3. (required) the timestamp line with start and end timestamps removed
   but including the cue settings; leading/trailing whitespaces are
   removed; the entry's start timestamp is stored as the Matroska
   block's start timestamp; the difference between the entry's end and
   start timestamps is stored as the Matroska block's duration

4. (required) the entry's content lines with all cue timestamps being
   shifted to be relative to the Matroska block's start timestamp

A demuxer will have to re-create the entry by inserting the start/end
timestamps and by shifting the embedded cue timestamps back to their
absolute values.

Rationale for removing the start/end timestamps and for shifting
embedded cue timestamps: duplicating container-level information
(start timestamp, end timestamp/duration) in-band makes any kind of
container-level transformation extremely difficult.

Here's how several example entries from above would be modified:

---start---------------------------------------------------------
hello
-->
Example entry 1: Hello <b>world</b>.
---end-----------------------------------------------------------

---start---------------------------------------------------------
NOTE style blocks cannot appear after the first cue.

-->
Example entry 2: Another entry.
This one has multiple lines.
---end-----------------------------------------------------------

---start---------------------------------------------------------
4
-->
Example entry 3: Entries can be numbered (like this one)…
---end-----------------------------------------------------------

---start---------------------------------------------------------
--> region:bill align:right
Example entry 7: Entries can even include timestamps.
For example:<00:00:05.000>This becomes visible five seconds
after the first part.
---end-----------------------------------------------------------

I'm open to all kinds of suggestions.

Kind regards,
mosu

[1]  https://w3c.github.io/webvtt/

---

From: Moritz Bunkus <[email-masked]>
Date: Mon, 14 Mar 2016 16:57:27 +0100

Hey,

aaaand I just noticed that the WebM guys already have a document about
storing WebVTT in WebM:
http://wiki.webmproject.org/webm-metadata/temporal-metadata/webvtt-in-webm

I haven't reviewed it yet but using the same way to store WebVTT seems
highly sensible to me :)

Kind regards,
mosu

---

From: Moritz Bunkus <[email-masked]>
Date: Sun, 20 Mar 2016 19:17:42 +0100

Hey,

thanks for your input, Denis.

I've looked over the document from the WebM project[1] thinking about
WebVTT inclusion. Unfortunately it's rather old (from 2012), and its
text leads me to believe that WebVTT has changed considerable since
then. For one the WebM proposal states that no CodecPrivate content will
be used; however, several global blocks in WebVTT are not covered at
all.

Additionally the WebM project proposes splitting the WebVTT content into
two tracks, one for the styling, one for the content. I don't like this
approach at all as we currently don't have any method of linking two
tracks in a way that clearly indicates that those tracks must be kept
and used together.

I therefore don't think we can use the WebM proposal.

That being said the WebM proposal contains similar thoughts to mine
about keeping the entry tags and the timecode line

Additionally using the BlockAdditions is a good idea; thanks, Denis. I'm
incorporating it.

Denis had an objection:

> That's basically useless... I'd remove it too, the less useless stuff
> the decoder/muxer has to handle, the better.

My goal is still to keep as much information from the WebVTT file as
possible without compromising Matroska's general ideas (and hopefully
without jumping through hoops). I aspire to the same for other source
container formats, not just for WebVTT. The reason is that Matroska is
not solely used for storing content for later playback but also as an
intermediate format. And keeping as much information as possible makes
it a lot easier to edit the content later.

> Wouldn't it be easier to just create another block starting with the
> new timestamp and ending with containing block end?

Maybe. My problem is that I don't fully understand where they're trying
to go with allowing cue timestamps within the content in the first
place. Would something like this be valid?

------------------------------------------------------------
00:02:00.000 --> 00:02:10.000
<v Professor Farnsworth>No fair!<00:02:01.500>You changed the outcome by
measuring it!
------------------------------------------------------------

I guess yes, and splitting this up into two entries would require
duplicating at leas the <v…> tag like. Additionally the muxer would have
to calculate where exactly the text after the embedded cue time stamp
would have to appear.

It quickly gets pretty complicated for a muxer. Therefore (and for
keeping the original structure as intact as possible) I'm against
splitting entries on embedded cue timestamps.

Here's my updated storage format proposal:

(A) CodecID: S_WEBVTT

(B) CodecPrivate: This element contains all global blocks before the
    first subtitle entry. The »WEBVTT« file identification marker is NOT
    part of CodecPrivate.

(C) Non-global blocks (e.g. »NOTE«) before an entry are stored in
    Matroska's BlockAddition element together with the entry they
    precede

(D) Each entry consists of three or more lines:

1. The first line contains the entry's cue identifier if present in the
   source file followed by a WebVTT line terminator. If no cue
   identifier was used then only the WebVTT line terminator is used.

2. The second line contains the entry's timestamp line with the actual
   timestamps removed followed by cue settings if present followed by a
   WebVTT line terminator. The start timestamp is used as the block's
   start timestamp, and the difference between the block's end and start
   timestamps are used as the block's duration.

3. All following lines are the content lines from the entry. If the
   content lines contain cue timestamps then those timestamps MUST be
   adjusted to be relative to the entry's start timestamp (and a demuxer
   has to reverse this process again).
------------------------------------------------------------

Here's an example how an entry could be converted. If a WebVTT file
looks like this:

---start---------------------------------------------------------
WEBVTT

example entry 1
00:03:10.000 --> 00:03:20.000 region:bill align:right
Entries can even include timestamps. For example:
<00:03:15.000>This becomes visible five seconds after the first part.

NOTE This is a comment block.

00:05:22.000 --> 00:05:28.700
Another entry, this one with neither a cue identifier nor cue settings.
---end-----------------------------------------------------------

then the converted first entry would look line this:

---start---------------------------------------------------------
example entry 1
--> region:bill align:right
Entries can even include timestamps. For example:
<00:00:05.000>This becomes visible five seconds after the first part.
---end-----------------------------------------------------------

The second Matroska block will contain a BlockAddition element:

---start---------------------------------------------------------
NOTE This is a comment block.
---end-----------------------------------------------------------

and the actual block:

---start---------------------------------------------------------

-->
Another entry, this one with neither a cue identifier nor cue settings.
---end-----------------------------------------------------------

Kind regards,
mosu

[1]  http://wiki.webmproject.org/webm-metadata/temporal-metadata/webvtt-in-webm

---

From: Moritz Bunkus <[email-masked]>
Date: Sun, 20 Mar 2016 19:39:29 +0100

Hey,

> (A) CodecID: S_WEBVTT

I'd be OK with S_TEXT/WEBVTT, too, as the other text subtitle formats
all use some variation of S_TEXT/…

Kind regards,
mosu