Subject: [AVT] FW: H264 Frame identification

From: "K Sonachalam-a22339" <[email-masked]>
Date: Tue, 26 Jun 2007 21:29:43 +0800

Hi all
 
How to identify a start and end of  Frame in H264.

In RTSP streaming session I will get the SPS and PPS NAL in sdp itself.
 
And in the  RTP payload has either the VCL NAlunits(Nal type 1 or 5) or
the Filler data
filler_data(Nal type 12). 
 
How to identify the Start and End of Frame from the NALs?.
 
Thanks
Regards
Sona

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: Randell Jesup <[email-masked]>
Date: Wed, 27 Jun 2007 10:24:49 -0400

While investigating some RFC 2833/4733 stuff, I ran across this discussion
at the Asterisk bug system.  The relevant piece is the discussion exerpted
from email with a "major PBX vendor" about simultaneous 2833 and audio
packets (which is explicitly allowed by RFC 2833).  

http://bugs.digium.com/view.php?id=6667

Search for this message, about 1/2-way down:
(0042832)
damin - manager
03-17-06 11:07

There's a bunch more from the engineers at this "Major PBX Vendor", but
this is the summary:

> We checked the RFC 2833, it dose not work, because your DTMF event and
> voice event has the same time stamp. In this case, we are going to drop
> the "duplicate" packets. I have several email exchanges with our
> engineers. They think logically, DMTF events and voice packets should not
> have the same time stamps. We have been in SIPIT events and test with 10+
> vendors, we don't have problems to recieve DTMF through RFC 2833. I don't
> know if it's easy to change some implementation in your side. Attatched
> are some email exchanges among our engineers.

My reading of 2833 and 4733 is that duplicate timestamps between audio
packets and 2833/4733 packets *are* allowed.  Encoded audio and Events
are allowed to overlap, and from the below I'd assume they can overlap
for the entire event.

>From RFC 2833:
   3.2 Simultaneous Generation of Audio and Events

   A source MAY send events and coded audio packets for the same time
   instants, using events as the redundant encoding for the audio
   stream, or it MAY block outgoing audio while event tones are active
   and only send named events as both the primary and redundant
   encodings.

   Note that a period covered by an encoded tone may overlap in time
   with a period of audio encoded by other means. This is likely to
   occur at the onset of a tone and is necessary to avoid possible
   errors in the interpretation of the reproduced tone at the remote
   end.  Implementations supporting this payload format must be prepared
   to handle the overlap. It is RECOMMENDED that gateways only render
   the encoded tone since the audio may contain spurious tones
   introduced by the audio compression algorithm. However, it is
   anticipated that these extra tones in general should not interfere
   with recognition at the far end.

I also note that this paragraph has been removed from RFC 4733, which makes
no direct comment on simultaneous/overlapping audio and Event payloads.
(It does imply in the DTMF discussion that commonly devices will send
DTMF events interspersed with audio, and Events allow them to avoid codec
mangling and they don't have to generate precise tone waveforms.)

So....
1) Can I have 2833/4733 Events with the same timestamp(s) as coded audio
   packets (G711/etc)?  Both in theory (per the RFC), and in practice (will
   receiving devices in the real world break on it, per the PBX vendor
   above)?  If it breaks gateways/PBXs to have packets with the same
   timestamp (BAD jitter-buffer implementation - should use sequence
   numbers for duplicate detection), can I increment the timestamp of
   the Event by (say) 1?  Or do I need to increment it by a packetization-
   interval?  (Such that, barring re-ordering(!), the evil jitter-buffer
   will throw away the coded audio pakcet instead of the Event.)
2) What do other devices do?  Do they send both in-band (tones in audio)
   and out-of-band (RFC 2833/4733) continuously when there's a press?
3) If a device doesn't send both throughout, what about overlap between the
   last audio codec packet and first DTMF packet?  and the same at the end
   of the DTMF sequence?  (In both theory and practice - for example, it
   would seem to be valid in theory to overlap at the start, and at the end
   restart the audio codec at a point which would be in the "middle" of
   a frame from the pre-Event timing.  However, many receivers might get
   upset at that, even if it's legal.)
4) (not really related to the above)
   RFC 4733 (and 2833) say:
       An audio source SHOULD start transmitting event packets as soon as
       it recognizes an event and continue to send updates until the event
       has ended. 
   But the examples show transmission of the Start packet for the Event
   occurring 50ms after the Event started, with a duration of 50ms.  I
   know Cisco (and others) send a start packet with a duration of 0ms,
   then increase the duration in future packets.  Per RFC 4733, 0 for 
   duration is a special value only allowed for "states", and DTMF events
   are "tones".  RFC 2833 seems to allow a 0 duration at the start of
   a DTMF event.

The RFC 4733 example doesn't show coded audio or overlaps.  It does
strongly imply that the starting timestamp is the time the button is
pressed, with no caveats expressed.

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)

---

From: "Mark Brown" <[email-masked]>
Date: Wed, 27 Jun 2007 18:14:43 +0100

>    and out-of-band (RFC 2833/4733) continuously when there's a press?

I've seen both choices but it is very much more common to suppress the audio
when RFC2833/4733 is being transmitted.

>    upset at that, even if it's legal.)

In practice it seems much more reliable to transmit with no overlap. The
case that tends to break is that where the RTP stream is converted into
audio and DTMF detection is then performed on the result. This is especially
problematic if there is a possibility that there may be DTMF encoded as
audio outside the RFC 2833/4733 frames. It also helps DTMF detectors to
leave some time after the event before restarting audio.

>    occurring 50ms after the Event started, with a duration of 50ms.

The example is specifically referring to a gateway - the 50ms delay looks
like the time taken to identify that the audio stream contains a DTMF tone
and start encoding it as such.

 Per RFC 4733, 0 for
>   are "tones".  RFC 2833 seems to allow a 0 duration at the start of

Section 2.3.5 does explicitly state that zero length durations ought to be
ignored for no states.
_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: David R Oran <[email-masked]>
Date: Wed, 27 Jun 2007 13:31:02 -0400

> packets (which is explicitly allowed by RFC 2833).

In fact, this can be used to advantage. Check out US Patent 6,775,265.

---

From: Randell Jesup <[email-masked]>
Date: Thu, 28 Jun 2007 00:22:32 -0400

David R Oran <[email-masked]> writes:
>> While investigating some RFC 2833/4733 stuff, I ran across this
>> discussion at the Asterisk bug system.  The relevant piece is the
>> discussion exerpted from email with a "major PBX vendor" about
>> simultaneous 2833 and audio packets (which is explicitly allowed by RFC
>> 2833).
>

Hmmpf.  Hardly patentable, IMHO, and in fact I'd say it was forecast by
the 2833 draft itself.  Not to mention many of the claims are "the world is
ours!" claims.  :-/

For example, this claim:
    13. A method for processing signaling in a packet telephony system,
        comprising: encoding a telephony data stream into voice packets;
        identifying signaling in the telephony data stream; transmitting
        the voice packets regardless of whether the voice packets contain
        signaling; encoding identified signaling into signaling packets and
        transmitting the signaling packets and the voice packets; and
        encoding and transmitting the voice packets and the signaling
        packets for the same time reference values.

Is pretty much "we implement 2833, which allows us to send both audio and
DTMF for the same time period".  And it's pretty darn obvious.  Some of the
others are a little stronger, but overall it's pretty weak - like most
software patents.

Ah well.

---

From: Randell Jesup <[email-masked]>
Date: Thu, 28 Jun 2007 00:42:24 -0400

>when RFC2833/4733 is being transmitted.

Ok, thanks.  My preference would be to send both, but that's me.  Sounds
like "not sending both" is likely to be better tested.  Do they round
start/end times to packet boundaries?

>leave some time after the event before restarting audio.

Ick.  My guess is the above is more focused on RFC 2833 generated from
audio data (such as an ATA attached to a POTS phone), as opposed to a pure
digital SIP phone, which knows exactly when DMTF events are generated.

This also implies that start times would get delayed by up to one audio
frame time, and durations might be rounded (up I assume) to a multiple of
frame time.  So a 40ms tone might become a 60ms tone (with a 30ms
packetization).  If you don't delay, but round start down and end up (to
avoid overlaps with audio packets) you could end up with a 40ms tone
becoming a 90ms tone.  It could also eat a lot into inter-tone gaps.

When you say "more reliable", do you mean when DTMF is generated by
detection in an audio stream (ATA+POTS example), or that "it's less likely
to blow up or confuse someone receiving it"?  In my case, there's no POTS
phone or detection to be done.

>Section 2.3.5 does explicitly state that zero length durations ought to be

Which would be a slight problem if they did this, since if you ignore the
start-packet, you'd have to notice a later packet (with non-0 duration),
and decide that the start packet had been "lost" (really it was ignored),
and back-fill the tone.  It should work, in most cases, but it increases
vulnerability to packet loss (i.e. if the first update packet gets really
lost or delayed, for example).

This really is just a case of something valid (if unmentioned) in RFC 2833
that's invalid in RFC 4733, but if you do something "reasonable" there's
not likely to be a bad side-effect (per above, though with additional
risk).

---

From: Randell Jesup <[email-masked]>
Date: Fri, 29 Jun 2007 09:45:50 -0400

One other thing about RFC 2833/4733:

There's a bunch of implications that there can't be an overlap between DTMF
events, and other implications that they can.  Which is correct?  I realize
that in the analog DTMF world, two (or especially more) simultaneous DTMF
digits may not be possible to (properly) recognize.  But with a direct IP
endpoint, overlapping digits certainly can be recognized and sent.

My guess is that sending overlapped DTMF events would seriously confuse
many gateways and IVR systems, though others might happen to handle it.

>From RFC 4733:

   Typically, rather than a series of tone signals each with its own
   meaning, the content will consist of a single tone played out
   continuously or a single sequence of tones and possibly silence,
   repeated cyclically for some period of time.
(implies no overlap)

   Sending of a packet with the E bit set is OPTIONAL if the packet
   reports two events that are defined as mutually exclusive states, or
   if the final packet for one state is immediately followed by a packet
   reporting a mutually exclusive state.  (For events defined as states,
   the appearance of a mutually exclusive state implies the end of the
   previous state.)

(Note that DTMF digits are not states, they are type "tone").  Implies
overlap for tones.

   2.5.1.5.  Packing Multiple Events into One Packet

   Multiple named events can be packed into a single RTP packet if and
   only if the events are consecutive and contiguous, i.e., occur
   without overlap and without pause between them, and if the last event
   packed into a packet occurs quickly enough to avoid excessive delays
   at the receiver.

   This approach is similar to having multiple frames of frame-based
   audio in one RTP packet.

   The constraint that packed events not overlap implies that events
   designated as states can be followed in a packet only by other state
   events that are mutually exclusive to them.  The constraint itself is
   needed so that the beginning time of each event can be calculated at
   the receiver.

Implies overlap

   In the first algorithm, the receiver simply places a tone of the
   given duration in the audio playout buffer at the location indicated
   by the timestamp.  As additional packets are received that extend the
   same tone, the waveform in the playout buffer is extended
   accordingly. 

Implies no overlap

   Alternatively, the receiver can start a tone and play it until one of
   the following occurs:

   o  it receives a packet with the E bit set;

   o  it receives the next tone, distinguished by a different timestamp
      value (noting that new segments of long-duration events also
      appear with a new timestamp value);

   o  it receives an alternative non-event media stream (assuming none
      was being received while the event stream was active); or

Strongly implies no overlap

   2.5.2.3.  Long-Duration Events

   If an event report is received with duration equal to the maximum
   duration expressible in the duration field (0xFFFF) and the E bit for
   the report is not set, the event report may mark the end of a segment
   generated according to the procedures of Section 2.5.1.3.  If another
   report for the same event type is received, the receiver MUST compare
   the RTP timestamp for the new event with the sum of the RTP timestamp
   of the previous report plus the duration (0xFFFF).  The receiver uses
   the absence of a gap between the events to detect that it is
   receiving a single long-duration event.

Implies overlap