Subject: Re: [AVT] Working group last call: JPEG-2000 payload format

From: Chuck Harrison <[email-masked]>
Date: Wed, 06 Jun 2007 09:15:22 -0700

Suppose

1. A compliant J2K receiver MUST accept 90kHz timestamps
2. A compliant J2K receiver MAY accept other timestamp rates
3. A compliant J2K sender SHOULD transmit 90kHz timestamps
4. The RFC warns that senders violating the SHOULD of item 3
    risk interoperability problems.

Then Dave is permitted to send a stream between matching
compliant applications using an 89999 Hz media clock, 12857
ticks per frame, exactly 7 frames per second. Or: a 70000 Hz
clock, 10000 ticks per frame.

An alternate video source, playing nice and sending at
90 kHz, is guaranteed to be able to talk successfully to
Dave's receiver application.

Dave's special source may not be able to send successfully
to the majority of J2K receiver applications.

Is this a high enough level of interoperability?

Regards,
  Chuck

Magnus Westerlund wrote:
> 
> Dave Singer skrev:
> > RTP payload formats may be used for all sorts of purposes and (varying)
> > frame rates.  I disagree that we in the AVT group 'know best' and that
> > any payload format should mandate a particular clock frequency.  It's
> > perfectly legitimate for me to send material at say 7 fps, and 90000 is
> > not suitable for that -- the frames would appear to have varying
> > duration, and there is risk of round-off.
> 
> I know that AVT is not knowing best. However, I asked this questions so
> that we could be educated and take an informed decision. However I think
> there are definitely an trade-off between using a fixed timestamp clock
> rate and allow any rate to be used.
> 
> >
> > Any payload format should recommend ('should') something, but mention
> > that other possibilities exist.
> 
> As this have shown to create problems in implementations. For example it
> showed up in an interop test creating problems for the packet-based
> streaming service where people sent MPEG-4 content using RFC3016
> packetization using non 90kHz clock. This occurs because the wiggle room
> was given and most implementors do no understand the implications of
> using another clock. Sure, we didn't document well enough all the
> downsides with choosing something else. Also we really should provide
> some lower and upper boundaries on these values.
> 
> >
> > Very high clock frequencies (e.g. 27 MHz) bring their own problems
> > (frequent wrap-around being one).  For a given program, it is usually
> > the case that one can easily find a number that
> > a) doesn't wrap around too often
> > b) is high enough for accurate jitter calculations
> > c) is an integer multiple of the frame rate, or, for variable frame
> > rate, allows each frame to have an integer timestamp which has not been
> > rounded-off
> >
> 
> Yes, and that have so far been 90kHz to satisfy this. If there are cases
> when this is not true I would like to know about them. But I think we
> should be clear on why we are allowing any wiggle room at all. 90kHz do
> support quite a lot of different frame rates. I am not saying that there
> doesn't exist such rate. But given a blank card on what rate to use adds
> complexities. If we rather could enumerate a small set of possible rates
> to cover the known cases that would make me much happier.
> 
> But Ali's example is not something that can be solved using the RTP
> timestamp rate if I understand this correctly. Simply because 27 MHz
> gets us into the problem with frequent wrap around.
> 
> Cheers
> 
> Magnus Westerlund
> 
> IETF Transport Area Director & TSVWG Chair
> ----------------------------------------------------------------------
> Multimedia Technologies, Ericsson Research EAB/TVM/M
> ----------------------------------------------------------------------
> Ericsson AB                | Phone +46 8 4048287
> Torshamsgatan 23           | Fax   +46 8 7575550
> S-164 80 Stockholm, Sweden | mailto: [email-masked]
> ----------------------------------------------------------------------
> 
> _______________________________________________
> Audio/Video Transport Working Group
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/avt

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: Dave Singer <[email-masked]>
Date: Fri, 8 Jun 2007 16:30:50 -0700

>     risk interoperability problems.

I think timestamp conversion tends to be a common function in an RTP 
stack, doesn't it?  Be that as it may, it's no good allowing me to 
send non-90KHz if I can't know it'll work -- standards are about 
interoperability.  So I think that it should say

* a 90 KHz clockrate is recommended (you could even use SHOULD be 
used) for JPEG 2000 streams
* receivers MUST be able to handle other clock rates

The RFC warns that if you choose an unsuitable rate, or a rate whose 
conversion is difficult, error-prone (e.g. to round-off errors), or 
even unusual, you risk playability problems.

-- 
David Singer
Apple/QuickTime

---

From: Colin Perkins <[email-masked]>
Date: Sat, 9 Jun 2007 22:03:31 +0100

Okay, some questions for the working group, since this issue is  
broader than just the JPEG-2000 payload format:

1) Is there utility in allowing non-90kHz clock rates for video  
payload formats, and if so, does this utility out-weigh potential  
interoperability problems with existing code which assumes the common  
case of a 90kHz clock? Why?

2) Is there something special about the JPEG-2000 payload format,  
requiring it to use a different clock rate to other formats (e.g. as  
RFC 3497 uses an usual clock rate)?

Cheers,
Colin

On 9 Jun 2007, at 00:30, Dave Singer wrote:
> I think timestamp conversion tends to be a common function in an  
> RTP stack, doesn't it?  Be that as it may, it's no good allowing me  
> to send non-90KHz if I can't know it'll work -- standards are about  
> The RFC warns that if you choose an unsuitable rate, or a rate  
> whose conversion is difficult, error-prone (e.g. to round-off  
> errors), or even unusual, you risk playability problems.
>
>>

-- 
Colin Perkins
http://csperkins.org/

---

From: Colin Perkins <[email-masked]>
Date: Sun, 17 Jun 2007 17:40:56 +0100

On 9 Jun 2007, at 00:30, Dave Singer wrote:
> interoperability.

Right - this is the problem with allowing multiple rates.

> So I think that it should say
>

SHOULD, I think.

> errors), or even unusual, you risk playability problems.

...and the offer/answer considerations section in the draft explains  
how to negotiate the desired rate, and how to interact with receivers  
that only support 90 kHz.

-- 
Colin Perkins
http://csperkins.org/

---

From: Chuck Harrison <[email-masked]>
Date: Sun, 17 Jun 2007 10:42:45 -0700

> > * receivers MUST be able to handle other clock rates

The trouble with a MUST like this is that it is untestable;
therefore although compliance with the RFC *in theory*
guarantees interoperability, i.e. that a receiver will be
able to handle anything you throw at it, this may not
occur in practice. Also untestable compliance MUSTs could
interfere with progressing the document to standard (is
that so, chairs?)

If we choose a small number of specific alternate clock rates,
then the MUST becomes testable. But I do not think this is
what Dave was asking for.

> that only support 90 kHz.

If 90kHz capability is made a MUST at both senders and
receivers, then there is a baseline rate at which two
compliant implementations are guaranteed to interoperate.
This capability can be tested.

If a sender and receiver are able to negotiate (e.g. thru
offer/answer) an alternate rate that is more suitable to
the particular application environment, so much the better.

But I do not think Dave's original proposal -- that a sender
can choose an arbitrary rate and be guaranteed that any
compliant receiving implementation will handle it -- is
viable.

So is the drift of the proposal currently on the table as
follows?

- clockrate is considered as an SDP offer/answer parameter
- receiver MUST be capable of 90kHz
- sender MUST be capable of 90hKz 
- sender SHOULD use 90kHz
- receiver SHOULD accept other clockrates

Some guidance on possible alternate clockrates (e.g. a range
for the SHOULD) would make it more likely that interoperability
of independent implementations might occur. Any suggestions
for a range, Dave?

Cheers,
  Chuck

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 18 Jun 2007 14:18:42 +0200

> - receiver SHOULD accept other clockrates

I think so. I would prefer that the last SHOULD to be a MAY. That makes 
it 90kHz is normative to support, you may support others if you have 
reasons for it. Without good reasons, I don't see the need to support 
other clockrates, thus I am in favor of only a MAY.

> of independent implementations might occur. Any suggestions

Yes, it is important to be clear about how one does negotiate this and 
the impact this negotiation has.

I would also include a range for possible values. The lower limit I 
would say is 1kHz. This due to that RTCP would starting to get into the 
range where the actual useful measurement resolution is affected.

Cheers

Magnus Westerlund

---

From: Dave Singer <[email-masked]>
Date: Tue, 19 Jun 2007 10:19:14 +0200

At 14:18  +0200 18/06/07, Magnus Westerlund wrote:
>>- sender MUST be capable of 90hKz - sender SHOULD use 90kHz
>I think so. I would prefer that the last SHOULD to be a MAY. That 
>makes it 90kHz is normative to support, you may support others if 
>you have reasons for it. Without good reasons, I don't see the need 
>to support other clockrates, thus I am in favor of only a MAY.

you cannot have anything other than a MUST for receivers if you want 
interoperability.  Unless I am required to use SIP and negotiate 
always.

>Yes, it is important to be clear about how one does negotiate this 
>and the impact this negotiation has.
>
>would say is 1kHz. This due to that RTCP would starting to get into 
>the range where the actual useful measurement resolution is affected.
>
>Cheers
>
>Magnus Westerlund
>

-- 
David Singer
Apple/QuickTime

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 19 Jun 2007 10:32:07 +0200

Dave Singer skrev:
>> makes it 90kHz is normative to support, you may support others if you 
>> have reasons for it. Without good reasons, I don't see the need to 
>> support other clockrates, thus I am in favor of only a MAY.
> 
> interoperability.  Unless I am required to use SIP and negotiate always.
> 

Yes, either you use negotiation through SIP or other ways. Other ways 
include capability declarations from client towards RTSP servers for 
example. But sure, there are cases when you like to have this 
functionality for certain use cases. But I think it is very reasonable 
in these cases to have the specific usage requirements then to upgrade 
this requirement to a MUST. This is something that people writing 
umbrella standards can do when they need it. But the fact is that we do 
have some negotiation possibilities in most cases, except 
broadcast/multicast scenarios.

The reason I have this point of view is that you still haven't explained 
why it is important for everyone to support something else than 90kHz. 
Based on the lack of concrete motivation or complaints from the industry 
of the missing support in other video codecs, like H.264, I think this 
is a requirement of limited need. And until you provide a more 
convincing case, that this is something that really everyone needs, I 
think it is fine providing one guaranteed interoperability point (90kHz) 
and then allow others.

Cheers

Magnus Westerlund

---

From: Colin Perkins <[email-masked]>
Date: Mon, 25 Jun 2007 23:14:49 +0100

Wrapping up, I believe the rough consensus is that the JPEG-2000  
draft should be updated to allow non-90kHz timestamp rates, as follows:

- Senders and receivers MUST support a 90kHz RTP timestamp rate, and MAY
   support other rates.

- Senders SHOULD use 90kHz timestamp where possible, and the draft  
needs to
   be updated to include strong guidance on choosing the timestamp  
rate, and
   the implications of using a rate other than 90kHz.

- The timestamp rate MUST be negotiated. The offer/answer considerations
   should explain how this is done.

- Senders that wish to use a non-90kHz rate SHOULD also offer the  
same stream
   using a 90kHz timestamp rate, with a different RTP payload type,  
e.g.:

	m=video 5004 RTP/AVP 98 99
	a=rtpmap:98 jpeg2000/90000
	a=rtpmap:99 jpeg2000/27000000

   allowing graceful fallback to 90kHz if the receiver doesn't  
support the high
   rate.

To check the consensus, please reply to this message by the end of  
this week, stating if you agree or disagree with this proposal, and  
giving your reasons. Please do this even if you believe you have  
already stated your preference.

Colin (as chair)

On 19 Jun 2007, at 09:32, Magnus Westerlund wrote:
>>> you have reasons for it. Without good reasons, I don't see the  
>>> need to support other clockrates, thus I am in favor of only a MAY.
>> you cannot have anything other than a MUST for receivers if you  
>> want interoperability.  Unless I am required to use SIP and  
>> negotiate always.
>
> Yes, either you use negotiation through SIP or other ways. Other  
> ways include capability declarations from client towards RTSP  
> servers for example. But sure, there are cases when you like to  
> have this functionality for certain use cases. But I think it is  
> very reasonable in these cases to have the specific usage  
> requirements then to upgrade this requirement to a MUST. This is  
> something that people writing umbrella standards can do when they  
> need it. But the fact is that we do have some negotiation  
> possibilities in most cases, except broadcast/multicast scenarios.
>
> The reason I have this point of view is that you still haven't  
> explained why it is important for everyone to support something  
> else than 90kHz. Based on the lack of concrete motivation or  
> complaints from the industry of the missing support in other video  
> codecs, like H.264, I think this is a requirement of limited need.  
> And until you provide a more convincing case, that this is  
> something that really everyone needs, I think it is fine providing  
> one guaranteed interoperability point (90kHz) and then allow others.
>
> Cheers
>
> Magnus Westerlund
>

-- 
Colin Perkins
http://csperkins.org/

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 26 Jun 2007 10:03:37 +0200

Colin Perkins skrev:
> Wrapping up, I believe the rough consensus is that the JPEG-2000 draft 
> should be updated to allow non-90kHz timestamp rates, as follows:
> 
> - Senders SHOULD use 90kHz timestamp where possible, and the draft needs to
>   be updated to include strong guidance on choosing the timestamp rate, and
> - Senders that wish to use a non-90kHz rate SHOULD also offer the same 
> stream
>   using a 90kHz timestamp rate, with a different RTP payload type, e.g.:
> 
>   allowing graceful fallback to 90kHz if the receiver doesn't support 
> the high
>   rate.
> 
> To check the consensus, please reply to this message by the end of this 
> week, stating if you agree or disagree with this proposal, and giving 
> your reasons. Please do this even if you believe you have already stated 
> your preference.
> 

I am fine with this. However, I think the SDP example should switch the 
priority order of the two payload formats to indicate that one does like 
to use 27 MHz rather then 90k. So if including an example with this, 
change the m= line to:
m=video 5004 RTP/AVP 99 98

Cheers

Magnus Westerlund

---

From: Dave Singer <[email-masked]>
Date: Tue, 26 Jun 2007 12:00:55 +0300

OK, most of this isn't really a point about JPEG 2000, but about the 
evolution of RTP, and perhaps it needs to be disconnected from the 
JPEG 2000 draft.

* * * *

The problem with the proposed text is that it is SIP-centric;  it 
assumes negotiation and choice, neither of which are present in SAP 
or RTSP (for example).  In a multicast or RTSP context, can I rely on 
using the correct non-90kHz rate for my 7 fps stream, or not?  The 
requirement "must be negotiated" and "alternatives must be offered" 
are not always achievable.

The consequences of doing timestamp conversion badly vary greatly 
between the sender and receiver, also.  If a receiver does poor 
timestamp conversion (e.g. it introduces drift as a result of 
round-off), it affects only itself.  If a sender does it badly, the 
RTP packets have poorly formed timestamps and all receivers are 
affected (e.g. in a multicast).

Also, if the sender does conversion correctly for something that 
isn't an integer division of 90kHz, the delta between successive 
timestamps will vary, suggesting that the frame rate is variable, 
when it is, in fact, constant.

I still question the whole principle that RTP payload formats should 
mandate anything at all about clock rates.  RTP receivers are 
supposed to cope with arbitrary, suitable, clock rates, and the RTP 
spec. has a statement that the resolution must be high enough for the 
desired synchronization accuracy and for measuring packet arrival 
jitter.

Way back when a payload type (small number) was all you needed to 
know to decode an RTP session (you didn't need an SDP file or 
anything), it was necessary that the clockrate for a given payload 
format be statically defined.  We don't have static payload types any 
more and we don't need to statically associate clockrates with 
payloads.

The MPEG payload format uses 90 kHz because that clock rate is 
mandated in the MPEG spec., so the payload format reasonably says 
that it should match.  Not all applications of JPEG2000 RTP come from 
a 90 kHz source, so that does not apply here.

At 10:03  +0200 26/06/07, Magnus Westerlund wrote:
>>- Senders that wish to use a non-90kHz rate SHOULD also offer the same stream
>I am fine with this. However, I think the SDP example should switch 
>the priority order of the two payload formats to indicate that one 
>does like to use 27 MHz rather then 90k. So if including an example 
>with this, change the m= line to:

-- 
David Singer
Apple/QuickTime

---

From: sona chalam <[email-masked]>
Date: Tue, 26 Jun 2007 06:26:08 -0700 (PDT)

Hi all
 
How to identify a start and end of  Frame in H264.

In RTSP streaming session I will get the SPS and PPS
NAL in sdp itself.
 
And in the  RTP payload has either the VCL
NAlunits(Nal type 1 or 5) or the Filler data
filler_data(Nal type 12). 
 
How to identify the Start and End of Frame from the
NALs?.

      ____________________________________________________________________________________
Park yourself in front of a world of choices in alternative vehicles. Visit the Yahoo! Auto Green Center.
http://autos.yahoo.com/green_center/

---

From: Dave Singer <[email-masked]>
Date: Thu, 28 Jun 2007 09:56:33 +0300

>How to identify the Start and End of Frame from the

You follow the marker bit, check for lost packets by looking at 
sequence numbers, and look at the timestamp changes:

Marker bit (M): 1 bit : Set for the very last packet of the access 
unit indicated by the RTP timestamp
-- 
David Singer
Apple/QuickTime

---

From: "Jaehwan Kim" <[email-masked]>
Date: Fri, 29 Jun 2007 14:39:06 +0900

Hi,
Yes, generally Market is like that.
However, for H.264 RTPPayload format and RTP marker bit:

A few years ago in IMTC, we had informal technical discussion about
this:

   In case of H.264, the meaning of RTP Marker bit seems unclear.
   Rather than following legacy, we should follow H.264 payload format.

If we make a future H.264 RFC, this should be clarified.

Cheers,
Jaehwan
Vidiator

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 29 Jun 2007 11:49:59 +0200

Hi,

Please, explain which part of the text in RFC 3984 that is unclear.

Cheers

Magnus Westerlund

---

From: Randell Jesup <[email-masked]>
Date: Fri, 29 Jun 2007 07:39:04 -0400

>unit indicated by the RTP timestamp

However, you don't *need* to use the Mark bit to reassemble H.264 RTP
packets into NALUs to pass to a decoder (certainly not for packetization
modes 0 or 1), at least if the de-packetizer knows it's H.264.  Mode 0 is
particularly easy.

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)