Subject: RE: [AVT] H264 interleaving method

From: "K Sonachalam-a22339" <[email-masked]>
Date: Wed, 2 May 2007 17:11:35 +0800

Hi All
 
I have some doubts in implementing H264 Depacketizer.Can some body can
help me regarding  this and also help me with some sample code for
depacketizer.
 
I am trying to implement for a H264 depacketizer for a streaming client.

Doubts are. 
 
1. In H264 a complete NAL  contains one frame or an Access unit contains
one Frame?

2.When we get FUs  and if any FU unit is lost according to RFC it says
all the following FUs should be discarded. Is that means should we
discard the complete FU-A?
 
3.If you get FUs with seq number as 1 2 3 5  and assume 1 is the satrt
of FU and 5 is the End of the FU how the depacketizer should handle this
case?
 
4. If  the decoder needs Access unit not interms of NAL units? how the
the NALs units should be arranged or how to identify the start of the
new access unit? 
 
5.Within an access unit do we need to arrange interms of Picture
parameter set or sequence parameter set NAL units  as the  first  and
the rest follows?
 
It will be great if  I get to know how a typical H264  decoder would
have been designed to accept the NAL units?

Thank You
Regards
K.Sonachalam
Ph:-2601-4017

-----Original Message-----
From: K Sonachalam-a22339 
Sent: Friday, April 27, 2007 6:55 PM
To: 'Magnus Westerlund'
Subject: RE: [AVT] H264 interleaving method

Hi Magnus

Thank you very much for your quick repsonse and support.

Thanks

Thank You
Regards
K.Sonachalam
Ph:-2601-4017

-----Original Message-----
From: Magnus Westerlund [mailto:[email-masked]]
Sent: Friday, April 27, 2007 5:58 PM
To: sona chalam
Cc: K Sonachalam-a22339; [email-masked]
Subject: Re: [AVT] H264 interleaving method

sona chalam skrev:
> 
> Hi Mganus
> 
> In my previous mail i was refering some doubts in AbsDon.
> 
> I think i did a mistake while calculating AbsDon.
> It never becomes negative. It won't wrap around like DON.
> 
> So in that case I can always send the NALs in AbsDon order once my
> AbsDon-diff(m,n) is greates than sprop-max-don-diff or Number of VCL 
> NAL units greater sprop-interleaving-depth.
> 
> So while calcualting sprop-deinterleaving-depth do i ignore NON VCL 
> NAL units?.
> 
> So please clarify my doubts?
> 
> 1.For streaming session will the spop-max-don-diff should always be 
> present?

No, guarantees. As you say this is optional.

> 
> 2.If I arrange the NALs in AbsDon order is it same as decoding order?

decoding order is decoding order. The DON values are simply a wrapped
representation of this.

> 
> 3. Do I need to exclude the Non VCL NAL units while counting the 
> sprop-deinterleaving-depth?
> 

Good question, I can't remember. Does the RFC says something on this?

cheers

Magnus Westerlund

IETF Transport Area Director & TSVWG Chair
----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM/M
----------------------------------------------------------------------
Ericsson AB                | Phone +46 8 4048287
Torshamsgatan 23           | Fax   +46 8 7575550
S-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: "Franceschini Guido" <[email-masked]>
Date: Thu, 3 May 2007 10:13:03 +0200

Dears,

I have a case in which an RTP flow (for H.263) shows no SN gaps, but contains H263 partial frames. When the terminating portion of a frame is missing, this can be detected by observing that for TS X no Marker bit (set in the last RTP packet of a video frame) has been received. When instead an intermediate portion of a frame is lost, this is not detectable by inspecting just the RTP headers.

My assumption so far was that such a flow should be declared as non conformant to the H263/RFC2429/RFC3550 stack, however I failed to find a sentence stating this, i.e. that in the absence of SN gaps the media content shall conform to the respective payload specification (H.263 in this case).

Could you clarify this?

Thanks in advance
Guido

--------------------------------------------------------------------

CONFIDENTIALITY NOTICE

This message and its attachments are addressed solely to the persons above and may contain confidential information. If you have received the message in error, be informed that any use of the content hereof is prohibited. Please return it immediately to the sender and delete the message. Should you have any questions, please contact us by replying to [email-masked].

        Thank you

                                        www.telecomitalia.it

--------------------------------------------------------------------

---

From: "Even, Roni" <[email-masked]>
Date: Thu, 3 May 2007 11:58:02 +0300

Guido,
I am not sure I understand. If the sequence numbers have no gaps for the same time stamp and you identify the last packet when the marker bit is set, why do you say that you lost information or that this is H263 invalid information (did you look at the H263 payload header to deduct that).
Roni

> From: Franceschini Guido [mailto:[email-masked]]
> Sent: Thursday, May 03, 2007 11:13 AM
> To: [email-masked]
> Subject: [AVT] Question on RTP SN and conformance
> 
> Dears,
> 
> I have a case in which an RTP flow (for H.263) shows no SN gaps, but
> contains H263 partial frames. When the terminating portion of a frame is
> missing, this can be detected by observing that for TS X no Marker bit
> (set in the last RTP packet of a video frame) has been received. When
> instead an intermediate portion of a frame is lost, this is not detectable
> by inspecting just the RTP headers.
> 
> My assumption so far was that such a flow should be declared as non
> conformant to the H263/RFC2429/RFC3550 stack, however I failed to find a
> sentence stating this, i.e. that in the absence of SN gaps the media
> content shall conform to the respective payload specification (H.263 in
> this case).
> 
> Could you clarify this?
> 
> Thanks in advance
> Guido
> 
> 
> 
> This message and its attachments are addressed solely to the persons above
> and may contain confidential information. If you have received the message
> in error, be informed that any use of the content hereof is prohibited.
> Please return it immediately to the sender and delete the message. Should
> you have any questions, please contact us by replying to
> [email-masked].
> 
>         Thank you
> 
>                                         www.telecomitalia.it
>

---

From: "Franceschini Guido" <[email-masked]>
Date: Thu, 3 May 2007 11:21:39 +0200

I make an example to better describe my case.

I have 3 frames of 4500 bytes each. The first frame is fully mapped into 5 RTP packets, no errors.
The second frame misses the last 500 bytes: this is easily detected by the absence of a MarkerBit.
The third frame misses bytes 2001-3000, and this is not detectable by just looking at the RTP headers.
The missing payload portions are not reflected in SN gaps: SN are contiguos instead.

Thus, for this example, the sequence of packets observed would look like:

SN = 100 TS = 1000 MarkerBit = 0 Payload contains bytes    1-1000 of video frame 1
SN = 101 TS = 1000 MarkerBit = 0 Payload contains bytes 1001-2000 of video frame 1
SN = 102 TS = 1000 MarkerBit = 0 Payload contains bytes 2001-3000 of video frame 1
SN = 103 TS = 1000 MarkerBit = 0 Payload contains bytes 3001-4000 of video frame 1
SN = 104 TS = 1000 MarkerBit = 1 Payload contains bytes 4001-4500 of video frame 1
SN = 105 TS = 2000 MarkerBit = 0 Payload contains bytes    1-1000 of video frame 2
SN = 106 TS = 2000 MarkerBit = 0 Payload contains bytes 1001-2000 of video frame 2
SN = 107 TS = 2000 MarkerBit = 0 Payload contains bytes 2001-3000 of video frame 2
SN = 108 TS = 2000 MarkerBit = 0 Payload contains bytes 3001-4000 of video frame 2
SN = 109 TS = 3000 MarkerBit = 0 Payload contains bytes    1-1000 of video frame 3
SN = 110 TS = 3000 MarkerBit = 0 Payload contains bytes 1001-2000 of video frame 3
SN = 111 TS = 3000 MarkerBit = 0 Payload contains bytes 3001-4000 of video frame 3
SN = 112 TS = 3000 MarkerBit = 1 Payload contains bytes 4001-4500 of video frame 3

The question is: can we assert that this flow is NOT conformant?
Which part of RFC3550/RFC2429 or others would state this?

Thanks in advance
Guido

> From: Even, Roni [mailto:[email-masked]]
> Sent: giovedì 3 maggio 2007 10.58
> To: Franceschini Guido; [email-masked]
> Subject: RE: [AVT] Question on RTP SN and conformance
> 
> 
> Guido,
> I am not sure I understand. If the sequence numbers have no 
> gaps for the same time stamp and you identify the last packet 
> when the marker bit is set, why do you say that you lost 
> information or that this is H263 invalid information (did you 
> look at the H263 payload header to deduct that).
> Roni
> 
> > From: Franceschini Guido 
> [mailto:[email-masked]]

CONFIDENTIALITY NOTICE

        Thank you

                                        www.telecomitalia.it

---

From: "Even, Roni" <[email-masked]>
Date: Thu, 3 May 2007 12:33:14 +0300

Guido,
In case 3 you need to look at the h263 payload header to understand that you really miss part of the information.

I can see two reason and both are wrong.

1. The source did not build the packet correctly skipping part of it for some reason (not very likely)

2. A translator or mixer in the middle changed the SN incorrectly according to RFC3550

The way to build consecutive RTP packets for the same decoding time like a video frame is in RFC 3550. (look at timestamp)
How to break big H263 frames is in RFC 2429 (now replaced by RFC 4629)

Roni

> Sent: Thursday, May 03, 2007 12:22 PM
> To: Even, Roni; [email-masked]

---

From: "Yuval Nissan" <[email-masked]>
Date: Thu, 3 May 2007 12:44:12 +0300

This can also happen in recording and streaming application:

If a server records H.263 RTP stream from the network (and there are packet loss), and than play it back to some device.

Yuval

-----Original Message-----
From: Even, Roni [mailto:[email-masked]] 
Sent: Thursday, May 03, 2007 12:33 PM
To: Franceschini Guido; [email-masked]
Subject: RE: [AVT] Question on RTP SN and conformance

Guido,

Roni

---

From: "Even, Roni" <[email-masked]>
Date: Thu, 3 May 2007 12:46:17 +0300

Yuval,
This is my second reason. If he keeps the RTP and transmit it back, the SN should be correct.
Roni

> From: Yuval Nissan [mailto:[email-masked]]
> Sent: Thursday, May 03, 2007 12:44 PM
> To: Even, Roni; Franceschini Guido; [email-masked]
> If a server records H.263 RTP stream from the network (and there are
> packet loss), and than play it back to some device.
> 
> Yuval
> 
> The way to build consecutive RTP packets for the same decoding time like a
> video frame is in RFC 3550. (look at timestamp)
> > I have 3 frames of 4500 bytes each. The first frame is fully mapped into
> 5
> > RTP packets, no errors.
> > The second frame misses the last 500 bytes: this is easily detected by
> the
> > absence of a MarkerBit.
> > The third frame misses bytes 2001-3000, and this is not detectable by
> just
> > looking at the RTP headers.
> > SN = 100 TS = 1000 MarkerBit = 0 Payload contains bytes    1-1000 of
> video
> > frame 1
> > SN = 101 TS = 1000 MarkerBit = 0 Payload contains bytes 1001-2000 of
> video
> > frame 1
> > SN = 102 TS = 1000 MarkerBit = 0 Payload contains bytes 2001-3000 of
> video
> > frame 1
> > SN = 103 TS = 1000 MarkerBit = 0 Payload contains bytes 3001-4000 of
> video
> > frame 1
> > SN = 104 TS = 1000 MarkerBit = 1 Payload contains bytes 4001-4500 of
> video
> > frame 1
> > SN = 105 TS = 2000 MarkerBit = 0 Payload contains bytes    1-1000 of
> video
> > frame 2
> > SN = 106 TS = 2000 MarkerBit = 0 Payload contains bytes 1001-2000 of
> video
> > frame 2
> > SN = 107 TS = 2000 MarkerBit = 0 Payload contains bytes 2001-3000 of
> video
> > frame 2
> > SN = 108 TS = 2000 MarkerBit = 0 Payload contains bytes 3001-4000 of
> video
> > frame 2
> > SN = 109 TS = 3000 MarkerBit = 0 Payload contains bytes    1-1000 of
> video
> > frame 3
> > SN = 110 TS = 3000 MarkerBit = 0 Payload contains bytes 1001-2000 of
> video
> > frame 3
> > SN = 111 TS = 3000 MarkerBit = 0 Payload contains bytes 3001-4000 of
> video
> > frame 3
> > SN = 112 TS = 3000 MarkerBit = 1 Payload contains bytes 4001-4500 of
> video
> > frame 3
> >
> >

---

From: "Yuval Nissan" <[email-masked]>
Date: Thu, 3 May 2007 12:52:18 +0300

Roni,

If it keeps the original sequence numbers the RTCP statistics that present the network impairments in the time of streaming won't be correct. I am not sure this is the right thing to do - the decoder can detect and handle the missing parts without the RTP help.

Regards,
Yuval 

-----Original Message-----
From: Even, Roni [mailto:[email-masked]] 
Sent: Thursday, May 03, 2007 12:46 PM
To: Yuval Nissan; Franceschini Guido; [email-masked]
Subject: RE: [AVT] Question on RTP SN and conformance

Yuval,

---

From: "Franceschini Guido" <[email-masked]>
Date: Thu, 3 May 2007 14:50:20 +0200

Yuval, Roni,

I think that by means of the recording scenario you are even better highlighting the problem I am facing.

Trying to recap:

1) On one side, if a receiver gets a flow with no SN gaps, it can assume that the media it would extract from that sequence of packets conforms to the respective payload specification (e.g.: a clean H.263 stream)

2) On the other side, SNs apply at the RTP session, and describe a transport property. If packets are lost within the RTP session, we have SN gaps, and RTCP statisticts that reflect that loss. If however packets are not lost in the RTP session, SN should show no gaps.

3) To meet both 1) and 2), a sender shall ensure that the media content it packetizes into RTP is conforming to the appropriate specification. In the recording and streaming application scenario this would imply that the streamer should discard the incomplete frames and deliver only the complete frames.

Actually, I was looking for a specific sentence stating 1), but I couldn't find any in the RFCs.
It seems to me that 2) is non controversial.
Then, it seems to me that if we assert that rules 1) and 2) both applies, then also rule 3) applies.

Therefore my question: does rule 1) above exist at all? If yes, where is it stated?

PS: I would love to see such a rule.

Best regards
Guido

> Sent: giovedì 3 maggio 2007 11.52
> If it keeps the original sequence numbers the RTCP statistics 
> that present the network impairments in the time of streaming 
> won't be correct. I am not sure this is the right thing to do 
> - the decoder can detect and handle the missing parts without 
> the RTP help.
> 
> Regards,
> Yuval 
> 
> > In case 3 you need to look at the h263 payload header to 
> understand that
> > you really miss part of the information.
> > 
> > 1. The source did not build the packet correctly skipping 
> part of it for
> > some reason (not very likely)
> > 
> > > Thus, for this example, the sequence of packets observed 
> would look
> > like:
> > >
> > > This message and its attachments are addressed solely to 
> the persons
> > above

CONFIDENTIALITY NOTICE

        Thank you

                                        www.telecomitalia.it

---

From: "Yuval Nissan" <[email-masked]>
Date: Thu, 3 May 2007 16:10:41 +0300

Guido,

As you say your assumption lead to discarded of valuable data. This is again unnecessary because you should send the uncompleted frames to the video decoder (in the received side) and let it do the best it could.

The important functions of RTP - time sync and packet ordering are still kept in this case.

So again in practice (and I think that also according to the RFCs) you should be able to handle it.     

Regards,
Yuval

-----Original Message-----
From: Franceschini Guido [mailto:[email-masked]] 
Sent: Thursday, May 03, 2007 3:50 PM
To: Yuval Nissan; Even, Roni; [email-masked]
Subject: RE: [AVT] Question on RTP SN and conformance

Yuval, Roni,

Trying to recap:

Best regards
Guido

> > > > the persons above

CONFIDENTIALITY NOTICE

        Thank you

                                        www.telecomitalia.it

---

From: Dave Singer <[email-masked]>
Date: Thu, 3 May 2007 09:06:05 -0700

At 16:10  +0300 3/05/07, Yuval Nissan wrote:
>Guido,
>
>As you say your assumption lead to discarded of valuable data. This 
>is again unnecessary because you should send the uncompleted frames 
>to the video decoder (in the received side) and let it do the best 
>it could.
>

I think Guido's question is not "how can it arise" or even "should I 
be able to handle it" but "can I rely on the sequence numbers to be 
able to detect missing data".

I think that this is 50% of the point of sequence numbers (the 
otehher 50% is, well, putting the packets in sequence).  If you 
cannot do codec-independent lost-packet detection, by using sequence 
numbers, I think the SN value is sharply reduced.

Should you write your re-assembly code to be resilient to this kind 
of loss?  Yes, surely, that's good engineering?

But should the soures of RTP flows take care to indicate loss by 
making sure that there are SN gaps?  yes, I think so.  But like 
Guido, I am not sure I can point at text that says that.

-- 
David Singer
Apple Computer/QuickTime

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 07 May 2007 16:03:05 +0200

> 1. In H264 a complete NAL  contains one frame or an Access unit contains

A NAL unit do not need to be a complete frame.

> discard the complete FU-A?

Yes, as a FU can only contain a single fragment. One may keep the 
partial reconstructed NALU if ones decoder is capable of using the 
broken NALU.

> of FU and 5 is the End of the FU how the depacketizer should handle this

Depends on the implementation. Either discard the complete NALU or keep 
the reconstucted partial NALU from 1 2 3. Using 5 is quite difficult.

> the NALs units should be arranged or how to identify the start of the

I can't answer this. The RTP payload format is written using the NALU 
concept and needs the network abstraction layer to be in place.

> have been designed to accept the NAL units?

I think you need to look at the H.264 standard.

Cheers

Magnus Westerlund