Subject: [AVTCORE] Magnus Westerlund's Discuss on draft-ietf-payload-rtp-ttml-03: (with DISCUSS and COMMENT)

From: Magnus Westerlund via Datatracker <[email-masked]>
Date: Tue, 15 Oct 2019 07:41:21 -0700

Magnus Westerlund has entered the following ballot position for
draft-ietf-payload-rtp-ttml-03: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-payload-rtp-ttml/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

James and WG,

I do have a couple of issues I want to have your feedback on if they should be
corrected or not before proceeding to publication. Note they are for discussion
and in cases where things have been discussed and there is consensus please
reference that so that I can take that into consideration when we resolve these.

1. Section 4.1:
        Timestamp:
        The RTP Timestamp encodes the time of the text in the packet.

        As timed text is a media that has duration, from a start time to an end
        time, and the RTP timestmap is a single time tick in the chose clock
        resolution the above text is not clear. I would think the start time of
        the document would be the most useful to include?

        I think the text in 4.2.1.2 combined with the above attempts to imply
        that the RTP timestamp will be the 0 reference for the time-expression?

        I think this needs a bit more clarification. Not having detailed
        studied TTML2/1 I might be missing important details. But some more
        information how the document timebase:media time line connects to the
        RTP timestamp appears necessary.

2. A Discuss Discuss: As Timed Text is directly associated with one or more
video and audio streams and requires synchronization with these other media
streams to function correct. This leads to two questions.

        First of all is application/ttml+xml actually the right top-level media
        type? If using SDP that forces one unless one have BUNDLE to use a
        different RTP session. Many media types having this type of properties
        of being associated with some other media types have registered media
        types in all relevant top-level media types.

        Secondly, this payload format may need some references to mechanisms in
        RTP and signalling that has the purpose of associating media streams? I
        also assume that we have the interesting cases with localization that
        different languages have different time lines for the text and how long
        it shows as there are different tranditions in different countries and
        languages for how one makes subtitles.

        This may also point to the need for discussing the pick one out of n
        mechanism that a manifest may need.

3. Section 7.1:

        It may be appropriate to use the same Synchronization
   Source and Clock Rate as the related media.

        Using the same SSRC as another media stream in the same RTP Session is
        no-no. If you meant to use multiple RTP sessions and associate them
        using the same SSRC in diffiernt, yes it works but is not recommended.
        This points to the need for a clearer discussion of how to achieve
        linkage and the reasons for why same RTP timestamp may be useful or not.

4. Fragmentation:
        I think the fragmentation of an TTML document across multiple RTP
        payloads are a bit insufficiently described. I have the impression that
        it is hard to do something more clever than to fill each RTP payload to
        MTU limtiation, and send them out insequence. However, I think a firm
        requirement to apply RTP sequence number for a single document in
        consecutive numbers. Also the re-assebly process appear to have to
        parts for detecting what belongs together, same timestamp and last
        packet of document should have marker bit set. As a receiver can loose
        the last packet in the previous document, still know that it has
        received everything for the following document. However, if the losses
        are multiple, inspection of the re-assemblied document will be
        necessary to determine if the correct beginning is present. I have the
        impression that a proper section discussing these matter of
        fragmentation and re-assembly are necessary for good interoperability
        and function.

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

A. Section 6.
        To my understanding the TTML document is basically not possible to
        encode better. A poor generator can create unnecessary verbose XML
        which could be shorter, but there are no possibility here to trade-off
        media quality for lower bit-rate. I think that should be made more
        explicit in Section 6.

B. Section 7.
        Wouldn't using 90kHz be the better default? 1kHz is the minimal from
        RTCP report that will work decently. However, if the timed text is
        primarily going to be synchronized with video 90k do ensure that
        (sub-)frame precise timing is possible to express. I don't see any need
        raster line specific for time text so the SMPTE 27 MHz clock is not
        needed. And using non default for subtitling radio etc appears fine.

C. Repair operations and relation to documents. Based on basic properties of
TTML documents, I do think the repair operations should be highly targeting
single documents as there is likely seconds between documents, while the
fragments of a document will be sent in a rather short interval. That
recommendation would be good to include.

---

From: James Sandford <[email-masked]>
Date: Wed, 16 Oct 2019 08:52:24 +0000

Hello,
Sorry for a slight non-response and I am conscious of the urgency of a reply. But I wanted to assure you that I am preparing a response but I feel some of the points are quite nuanced. I should have it to you later today but I may reply to Adam first as I feel some of his suggestions address your concerns. 

Regards,
James

James and WG,

1. Section 4.1:
        Timestamp:

3. Section 7.1:

4. Fragmentation:

A. Section 6.

B. Section 7.

---

From: James Sandford <[email-masked]>
Date: Wed, 16 Oct 2019 15:17:37 +0000

Thank you for the detailed review, Magnus. I've responded to your points in-line.

Regards,
James

Suggested updated text for the Timestamp definition: 

    The RTP Timestamp encodes the start time of the document in User Data Words

>        RTP timestamp appears necessary.

Suggested updated text for the last sentence of Paragraph 2 of 4.2.1.2: 

    Computed TTML media times are offset relative to E in accordance with Section I.2 of [TTML2].

I'm hesitant to expand on the calculation of timing beyond that in this document. It is discussed at length in TTML2 which is included as a normative reference in this document. I'd like to avoid blurring the scope of this document into that of TTML2 if possible.

>        types in all relevant top-level media types.

I think we need to be careful with assumptions here. Timed Text MAY be associated with Video and/or Audio streams but there is no requirement to do so. Just as there is no requirement for Video to be associated with Audio or visa-versa.

I'm cautious about opening a can of worms with regards to media types. But my personal opinion is that TTML is not video and it is not audio. It may be associated with them but it is not those. It therefore shouldn't have registered types for video, audio, etc. If the association of different top-level media types is currently difficult, that is an issue I believe should be addressed outside of the scope of this document.

>        mechanism that a manifest may need.

I believe this is outside of the scope of this document in the same way it is for audio equivalents and other text formats.

>        linkage and the reasons for why same RTP timestamp may be useful or not.

Thanks for spotting this. It should refer to the Reference Clock, not Synchronization Source.

>        fragmentation and re-assembly are necessary for good interoperability

This has been discussed in response to Adam Roach's Discuss.

Again, this has been discussed in response to Adam's Discuss.

>        needed. And using non default for subtitling radio etc appears fine.

Could you justify a 90k default? Such a high rate isn't appropriate for most timed text use cases. As stated in my response to "2." above, there is no requirement for TTML to be used alongside video so defining a default rate based on video would be inappropriate. Just as we don't expect video to use a 48k rate or audio a 90k rate. The default for TTML should be appropriate to timed text and should not make assumptions about specific implementations.

>recommendation would be good to include.

Also discussed in response to Adam's Discuss.

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 21 Oct 2019 13:27:54 +0000

Hi James, 

Please see inline for response and comments. 

>     The RTP Timestamp encodes the start time of the document in User Data

Okay, and this corresponds to "E" mentioned below? 

> I'm hesitant to expand on the calculation of timing beyond that in this
> document. It is discussed at length in TTML2 which is included as a normative
> reference in this document. I'd like to avoid blurring the scope of this
> document into that of TTML2 if possible.

Okay, I am all fine with keeping that in the TTML2 specs. However, the point of
linking the timescales must be clear. And to me it wasn't clear that RTP
Timestamp value is equal to the instant E that is the basis for document. 

This also leads to another warning that likely need to be in the RTP payload
time. As the document "start time" is not strongly correlated with transmission
time of the RTP packet there should be a mention of that as the RTCP
measurements based on the RTP timestamp field will contain significant error
components. 

Section 5.2 of RFC 8088 says this: 

   RTP payload formats with a timestamp definition that results in no or
   little correlation between the media time instance and its
   transmission time cause the RTCP jitter calculation to become
   unusable due to the errors introduced on the sender side.  A common
   example is a payload format for a video codec where the RTP timestamp
   represents the capture time of the video frame, but frames are large
   enough that multiple RTP packets need to be sent for each frame
   spread across the framing interval.  It should be noted whether or
   not the payload format has this property.

> I think we need to be careful with assumptions here. Timed Text MAY be
> associated with Video and/or Audio streams but there is no requirement to do
> so. Just as there is no requirement for Video to be associated with Audio or
> visa-versa.
> 
> I'm cautious about opening a can of worms with regards to media types. But my
> personal opinion is that TTML is not video and it is not audio. It may be
> associated with them but it is not those. It therefore shouldn't have
> registered types for video, audio, etc. If the association of different top-
> level media types is currently difficult, that is an issue I believe should be
> addressed outside of the scope of this document.

Okay, if the needs exists additional media types can be registred. It is very
much a question of which RTP usage world have TTML2 being used. In a world that
supports multiple media types on the same RTP session this is not an issue. The
m= line or equivalent for declaring media sources will simply say that there is
one TTML2 media source using top type application. 

For the older style one media type per RTP session, the implication of
application/ttml2 is that one need to have an RTP session extra in parallel if
one don't already have an application/foo media type in the multi-media session.

I just want to ensure that people intended to use this don't throw a fit when
they realize that their system spec suddenly have to add another RTP session. 

> I believe this is outside of the scope of this document in the same way it is
> for audio equivalents and other text formats.

Depends if the very basic mechanisms are sufficient. I do see that RTCP CNAME
work when all media sources from one endpoint shall be synchronized. And in more
advanced case Lipsynch signalling should work as long as all TTML2 tracks belong
to a single LS group and there are no overlapp. 

However, I think there are interesting cases that would occur if one have
multiple TTML2 media source, one for each language one provide and potentially
combined with multiple audio tracks for different dubbings. The question is if
one have a manifest description work for this case. I am uncertain if SDP can
actually properly express the relations between all these media sources. Maybe
someone else on the AVTCORE mailing list can answer this. 

> This has been discussed in response to Adam Roach's Discuss.

Yes, Adam brought up one aspect that I didn't think about, the splitting of UTF
characters. My focus was on the transport aspects of having a clear description
of the behavior of senders and receivers when TTML documents gets split over
multiple RTP packets. 

> Could you justify a 90k default? Such a high rate isn't appropriate for most
> timed text use cases. As stated in my response to "2." above, there is no
> requirement for TTML to be used alongside video so defining a default rate
> based on video would be inappropriate. Just as we don't expect video to use a
> 48k rate or audio a 90k rate. The default for TTML should be appropriate to
> timed text and should not make assumptions about specific implementations.

Sure, leave one 1kHz clock in as default. Howver, if one synchronize to another
media or so, using the same time based has its benefits as it enables precise
synchronization between the two. For example to get the subtitles to synchronize
with the correct video frame. Otherwise I see that one frame errors could occur.
Not that they necessary are super visible to the user, but especially as scene
cuts if the subitle shows up on the last frame of the previous scene rather then
first of the new can be visible. Thus, I think you should consider a
recommendation for the cases where one like to synchronize the TTML with other
media to select matching clock frequencies for the RTP timestamp to avoid these
issues. 

> Also discussed in response to Adam's Discuss.

Fine

Cheers

Magnus Westerlund 

----------------------------------------------------------------------
Network Architecture & Protocols, Ericsson Research
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Torshamnsgatan 23           | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 21 Oct 2019 13:30:28 +0000

Hi James, 

Fine

Cheers

Magnus Westerlund 

-- 
Cheers

Magnus Westerlund