Joel Keller | 19 Mar 15:41 2013

MP4 video trak samples: were do 'non-frame' NAL units belong?

Greetings experts,

Live555 is playing a key role in our industrial machine vision application - thank you for the library!  I am experiencing a problem saving MP4 format video files using openRTSP.  The RTSP/RTP stream is coming from a camera, and I can successfully save the 'raw/elementary' H264 NAL stream with openRTSP.  However, when I save the video as MP4, with something like this:

openRTSP -d 30 -b 1000000 -B 1000000 -4 -y -w 1536 -h 1536 -f 15 rtsp://10.0.3.11/live.sdp > myvideo.mp4


I find that the generated MP4 file contains NAL units in the 'mdat' box which do not correspond to video frames, specifically NAL units of type SPS, PPS, and SEI.  These three NAL units are periodically interspersed with the frame data.  This interspersion itself is of course not a problem as the media data area can contain non-frame data, and this meta data is of course needed to interpret the frame.  However these NAL units are treated as 'MP4 samples' in the MP4 'Sample Table Boxes'.  Specifically, in the stts (Decoding time to sample box), these samples have a "delta" of zero, implying that their presentation time is the same as the samples which precede them (this makes sense, because they are not frames of the video, but it violates ISO spec, which states that "The sample entries are ordered by decoding time stamps; therefore the deltas are all non-negative.").   I find that this causes some (most?) video players to "stutter" when playing back the MP4 file (avidemux2 for example),  because they determine the video's frame rate and then go sample-by-sample through the video.  When the player gets to these non-frame samples, it appears to delay for the frame-duration, but it cannot decode the frame (because the sample is not a frame).  This leads to a video with ~15 smooth frames, then 3 pause frames, then ~15 smooth frames.

My questions are:  

1)  What is the correct thing to do with these NAL units in the MP4 file?

2)  How can I make openRTSP/live555 create an MP4 file without this stuttering?



Possible answers I have to #1 are:

a.  "non-frame" NAL units are appropriate samples in MP4 video traks.  But then what to do with the stts deltas?

b.  "non-frame" NAL units should not show up as samples (see avcC box), and should be 'skipped over' based on MP4 'chunks'.  But how do I get live555 to do this?

c.  "non-frame' NAL units should be "combined" their following "frame" NAL unit to be a single 'MP4 sample'.  (ie a sample is not necessarily a single NAL unit). Again how would I get live555 to do this?



Thank you for reading this long post.  I am most grateful for any help.


Thanks,

Joel Keller

_______________________________________________
live-devel mailing list
live-devel@...
http://lists.live555.com/mailman/listinfo/live-devel
Ross Finlayson | 19 Mar 18:54 2013

Re: MP4 video trak samples: were do 'non-frame' NAL units belong?

My questions are:  

1)  What is the correct thing to do with these NAL units in the MP4 file?

2)  How can I make openRTSP/live555 create an MP4 file without this stuttering?

Unfortunately I don't know the answers to these questions, because I'm not an expert on the "MP4" file format.  What I do know, though, is that this file format is ill-suited for what we are trying to do here: Record a file that properly represents incoming audio and video frames that are time-stamped.  But - in spite of this format's bad design - we try to do the best that we can...

But if you're interested in trying to improve this, the code to look at is our implementation of the "QuickTimeFileSink" class - i.e., "liveMedia/QuickTimeFileSink.cpp".


Ross Finlayson
Live Networks, Inc.
http://www.live555.com/

_______________________________________________
live-devel mailing list
live-devel <at> lists.live555.com
http://lists.live555.com/mailman/listinfo/live-devel
Muhammad Hassan | 19 Mar 19:43 2013
Picon

Re: MP4 video trak samples: were do 'non-frame' NAL units belong?

U can write your own source filter using live555 client side code and connect it to any muxer filter like matroska, mp4 or avi and then save it. Feel free to contact me at hassan.nust <at> gmail.com

On Mar 19, 2013 10:48 PM, "Joel Keller" <jkeller <at> miovision.com> wrote:
Greetings experts,

Live555 is playing a key role in our industrial machine vision application - thank you for the library!  I am experiencing a problem saving MP4 format video files using openRTSP.  The RTSP/RTP stream is coming from a camera, and I can successfully save the 'raw/elementary' H264 NAL stream with openRTSP.  However, when I save the video as MP4, with something like this:

openRTSP -d 30 -b 1000000 -B 1000000 -4 -y -w 1536 -h 1536 -f 15 rtsp://10.0.3.11/live.sdp > myvideo.mp4


I find that the generated MP4 file contains NAL units in the 'mdat' box which do not correspond to video frames, specifically NAL units of type SPS, PPS, and SEI.  These three NAL units are periodically interspersed with the frame data.  This interspersion itself is of course not a problem as the media data area can contain non-frame data, and this meta data is of course needed to interpret the frame.  However these NAL units are treated as 'MP4 samples' in the MP4 'Sample Table Boxes'.  Specifically, in the stts (Decoding time to sample box), these samples have a "delta" of zero, implying that their presentation time is the same as the samples which precede them (this makes sense, because they are not frames of the video, but it violates ISO spec, which states that "The sample entries are ordered by decoding time stamps; therefore the deltas are all non-negative.").   I find that this causes some (most?) video players to "stutter" when playing back the MP4 file (avidemux2 for example),  because they determine the video's frame rate and then go sample-by-sample through the video.  When the player gets to these non-frame samples, it appears to delay for the frame-duration, but it cannot decode the frame (because the sample is not a frame).  This leads to a video with ~15 smooth frames, then 3 pause frames, then ~15 smooth frames.

My questions are:  

1)  What is the correct thing to do with these NAL units in the MP4 file?

2)  How can I make openRTSP/live555 create an MP4 file without this stuttering?



Possible answers I have to #1 are:

a.  "non-frame" NAL units are appropriate samples in MP4 video traks.  But then what to do with the stts deltas?

b.  "non-frame" NAL units should not show up as samples (see avcC box), and should be 'skipped over' based on MP4 'chunks'.  But how do I get live555 to do this?

c.  "non-frame' NAL units should be "combined" their following "frame" NAL unit to be a single 'MP4 sample'.  (ie a sample is not necessarily a single NAL unit). Again how would I get live555 to do this?



Thank you for reading this long post.  I am most grateful for any help.


Thanks,

Joel Keller


_______________________________________________
live-devel mailing list
live-devel <at> lists.live555.com
http://lists.live555.com/mailman/listinfo/live-devel

_______________________________________________
live-devel mailing list
live-devel <at> lists.live555.com
http://lists.live555.com/mailman/listinfo/live-devel
Peng Zheng | 24 Mar 07:24 2013
Picon

Re: MP4 video trak samples: were do 'non-frame' NAL units belong?

> My questions are:
>
> 1)  What is the correct thing to do with these NAL units in the MP4 file?

According to ISO 14496, SPS and PPS should ONLY appears in 'avcC' box.

> 2)  How can I make openRTSP/live555 create an MP4 file without this
> stuttering?

Collect NALUs with the same RTP timestamp and send them as a frame
into any third
party muxer that supporting ISO base media format, e.g. libavformat
from FFmpeg.

If you set avformat context correctly, it will produce what you want.

Peng

Gmane