Add H265RtpDepacketizer

[edmonds]

This commit adds an H265 depacketizer which takes incoming H265 RTP packets and emits H265 access units. It is closely based on the H264RtpDepacketizer added by @Sean-Der in #1082.

I originally started with a version of this commit that was closer to the H264RtpDepacketizer and which emitted individual H265 NALUs in H265RtpDepacketizer::buildFrames(). This resulted in calling my Track::onFrame() callback for each NALU, which did not work well with the decoder that I'm using which wants to see the VPS/SPS/PPS NALUs as a unit before initializing the decoder (https://intel.github.io/libvpl/v2.10/API_ref/VPL_func_vid_decode.html#mfxvideodecode-decodeheader).

So for the H265RtpDepacketizer I've tried to make it emit access units rather than NALUs. An "access unit" is (RFC 7798):

A set of NAL units that are associated with each other according to a specified classification rule, that are consecutive in decoding order, and that contain exactly one coded picture.

"Exactly one coded picture" seems to correspond with what a caller might expect an "onFrame" callback to do. Maybe the H264RtpDepacketizer should be revised to similarly emit H264 access units rather than NALUs, too. At least, I could not find a way to receive individual NALUs from the depacketizer and run the VPL decoder without needing to do my own buffering/copying of the NALUs.

With this commit I can now do the following:

• Generate encoded bitstream output from the Intel VPL encoder.
• Pass the output of the encoder one frame at a time to libdatachannel's Track::send() on a track with an H265RtpPacketizer media handler.
• Transport the video track over a WebRTC connection to a libdatachannel peer.
• Depacketize it with the H265RtpDepacketizer media handler in this commit.
• Pass the depacketized output via my Track::onFrame() callback to the Intel VPL decoder in "complete frame" mode (https://intel.github.io/libvpl/v2.10/API_ref/VPL_enums.html#_CPPv428MFX_BITSTREAM_COMPLETE_FRAME). Each "onFrame" callback corresponds to a single call to the decoder API to decode a frame.

[edmonds]

By the way, producing access units rather than NALUs seems to be what FFmpeg's H265 RTP depacketizer does. It's kind of awkward to set up, though!

Some open items:

• Is it always correct to build an access unit by simply prepending the start code { 0x00, 0x00, 0x00, 0x01 } to each NALU?
• H265RtpDepacketizer::incoming() is identical to H264RtpDepacketizer::incoming() and doesn't even look all that specific to H264/H265. Should that function be factored out somehow?
• DONL/DOND fields are always assumed to be absent. Does this need to be handled?
• RFC 7798 § 4.4.3: "If an FU is lost, the receiver SHOULD discard all following fragmentation units in transmission order corresponding to the same fragmented NAL unit, unless the decoder in the receiver is known to be prepared to gracefully handle incomplete NAL units."

[Sean-Der]

Agree on NALU API being awkward!

do you have bandwidth to update H264 depacketizer I can test + approve? We should make that change ASAP before people depend on it.

or maybe this is controlled via constructor

[edmonds]

do you have bandwidth to update H264 depacketizer I can test + approve? We should make that change ASAP before people depend on it.

Yeah, I should be able to take a look later today.

[paullouisageneau]

"Exactly one coded picture" seems to correspond with what a caller might expect an "onFrame" callback to do. Maybe the H264RtpDepacketizer should be revised to similarly emit H264 access units rather than NALUs, too. At least, I could not find a way to receive individual NALUs from the depacketizer and run the VPL decoder without needing to do my own buffering/copying of the NALUs.

I think you are right, it corresponds better to what the user would expect the callback to do. Additionally, it makes the API more symmetrical with packetizers.

Is it always correct to build an access unit by simply prepending the start code { 0x00, 0x00, 0x00, 0x01 } to each NALU?

This is the most common, but a short start sequence { 0x00, 0x00, 0x01 } also exists. Packetizers have an NalUnit::Separator constructor parameter to specify what they should expect, maybe depacketizers could have the same.

H265RtpDepacketizer::incoming() is identical to H264RtpDepacketizer::incoming() and doesn't even look all that specific to H264/H265. Should that function be factored out somehow?

Yes it should be, but I can do it in a second time.

[paullouisageneau]

Thank you for adding, I have only a couple comments and suggestions.

[edmonds]

Re-based on master, added fix for empty RTP packet payloads, made the start sequence configurable.

[paullouisageneau]

Thanks, it looks good, feel free to add you name in new files headers.

[edmonds]

Thanks, it looks good, feel free to add you name in new files headers.

Done.

[paullouisageneau]

@edmonds Sorry, you need to make a couple changes after rebasing as FrameInfo now requires the payload type after #1156 was merged.

[edmonds]

OK, no problem. Should be able to take a look in a couple days.

[paullouisageneau]

This would need a fix similar to #1167 to properly handle multiple NAL units.

[edmonds]

Sorry for the delay getting back to this!

I rebased on master, added the missing <algorithm> include, made the FrameInfo changes (due to #1156), and made a fix similar to #1167 to handle multiple NALUs.

[paullouisageneau]

Great, thank you for the work!