Repository navigation
Worker: Complete dependency descriptor implementation - #2015
Conversation
|
rust CHANGELOG missing. |
|
As said in the issue ticket, I don't think that we should forward the DD extension verbatim. I'm 65% sure that we talked about this long ago and it was not correct to forward it (as it is) when we are filtering layers in the Consumer. |
|
As per discussion in the issue ticket in #2013, let's mark this PR as draft until we do further investigation to validate the fix. |
|
While implementing AV1 simulcast for versatica/mediasoup#2011, I also implemented the DD rewriting this PR’s TODO mentions (filtered layers). The earlier failure with a rewritten DD (versatica/mediasoup#1625) may come from the old implementation rather than rewriting itself: WriteExtendedDescriptorFields() wrote the custom dtis/fdiffs/chains flags as 0 and dropped them. libwebrtc uses these whenever a frame doesn’t match its template. In my branch: DependencyDescriptor::Write() rewrites only active_decode_targets_bitmask (plus frame number and template id offset on simulcast switches), keeping the structure and custom fields. It’s mixed with the simulcast changes right now, but I can split the DD rewriting part into a separate PR if that helps. Disclaimer: this branch was implemented by AI and hasn’t been verified on real devices yet. |
|
Thanks @satoren, we'll check it as reference. |
|
This PR is outdated compared to v3. Please take into account the TODO items in the above PR description. |
|
Some thoughts:
|
|
I would also recommend revert all changes in this PR, merge v3 and start from there, otherwise ugly conflicts may happen specially in Rust. |
Yes, this is what we're doing. |
The Dependency Descriptor header extension is 'sendrecv' now, so it's also forwarded to non pipe Consumers. AV1 receivers have no other way to tell the frames of the different spatial layers apart.
5722c8f to
68bb1b4
Compare
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@satoren, if you don't mind I'll take that part as reference and use it in this PR. |
…ight packet A cloned packet shares the payload descriptor handler with the original, and RtpPacketChanged() made the handler's single DependencyDescriptor write into the clone from then on. Since Router hands the very same packet to every Consumer and clones it only once, every Consumer after the one that created the clone had its rewritten descriptor written into the clone instead of into the packet it was about to send. Those Consumers therefore transmitted the descriptor as received, advertising decode targets that they were in fact filtering out, so the endpoint waited for frames that never arrived and counted them as loss. Their retransmissions were not affected, since RetransmitPendingPackets() re-applies the stored encoder to the clone, which is the packet the listener happened to point at. Process(), Encode() and Restore() already receive the packet they act on, so point the descriptor at it on every call rather than remembering the last clone. RtpPacketChanged() only ever existed to do that and every implementation of it is now empty, so remove it. Extend the AV1 section of TestRtpStreamSend so that it also checks the packet being sent and not just the retransmitted clone.
It returns true if the data rewrite took place, false otherwise.
The descriptor is now given the packet to write into on every call rather than remembering one, so it cannot write into the wrong packet.
UpdateDependencyDescriptor() only took a descriptor fitting in the current length of the RTP header extension. A rewritten descriptor shorter than the received one therefore shrunk the extension for good, and Restore() could no longer write the original back, so the packet kept the descriptor of the Consumer that rewrote it first. It now takes the padding bytes that follow it, which the Producer leaves behind by telling the length the descriptor really has instead of the room made for it. Packets are no longer forwarded with trailing zeroes inside the Dependency Descriptor. The AV1 test read the retransmitted packet after it had been restored, so it checked a descriptor that only looked right while Restore() was failing. It now keeps what the packet carried when it was sent.
ibc
left a comment
There was a problem hiding this comment.
Too hard to review since it's huge and complex stuff but I trust you. I've requested cosmetic changes (or at least commented some concerns).
Please tell the robot to honor this strictly: #2020
This is, clang-tidy doesn't complain if we use auto instead of const auto (but const auto looks much better). And not just that. Pointers:
auto* ptr const = xxxxand things like that are good to have (see #2020).
Co-authored-by: Iñaki Baz Castillo <ibc@aliax.net>
…iptor.hpp Co-authored-by: Iñaki Baz Castillo <ibc@aliax.net>
Co-authored-by: Iñaki Baz Castillo <ibc@aliax.net>
|
Please update branch just in case. |
ibc
left a comment
There was a problem hiding this comment.
NOTE: If a consumer does not support DD, we use the codec native payload handler for it, so the needed payload re-writes are done accordingly.
Is this tested with VP9 and H264? (I know it cannot work with AV1 and we don't use DD in VP8).
Yes, given the new mediasoup-demo URL argument |
Done here 8122cfe @satoren, may I ask you for a last review? I've tested everything (tomorrow I'll do a last test). mediasoup-demo has now a URL param |
LGTM, thanks for addressing all the feedback! One non-blocking note, which |
The Dependency Descriptor RTP extension given to a Consumer is written out of the one in the packets of this Producer, so don't announce it to Consumers unless this Producer negotiated it. Otherwise a Consumer would negotiate an extension that is never going to arrive, and it would wrap its frame ids at the 16 bits of the Dependency Descriptor rather than at the 15 of the VP9 picture ids it is actually given.
This is a good observation. Handled in the last commit. Basically we are now filtering out DD from the Consumer RTP extensions if the producer does not offer it, the same way we did with Tested with and without pipe transports. The outcome is:
|
| continue; | ||
| } | ||
|
|
||
| // The Dependency Descriptor RTP extension given to a Consumer is written out |
When/where did we do that with |
Hello @ibc
|
I forgot. But it's a hack. Maybe not. |
|
While working on ongoing PR #2046 I've found a problem that ALSO affects this branch (I've also added a bullet about it in the above TODO list). Problem
How does libwebrtc solve it?In
What must be done in this branch?
|
|
@ibc, that's already fixed in the branch forward-dependency-descriptor. |
NOTE: At least part of it is fixed as a retransmission buffer packet holds the template structure of the original RTP packet and that is never overwritten. That was a bug that has been fixed in the DD PR. I'll check for the complete claim here. |
… in use Fixes the issue mentioned here: #2015 (comment) DependencyDescriptor::Parse() handed the stream the structure of every descriptor it parsed. A retransmitted or reordered packet of an old key frame thus replaced the one in use, leaving every following packet parsed with outdated templates — wrong layers and wrong decode target indications — until the next key frame. It is now taken only from the first packet of a frame that is not older than the one that brought the structure in use, which takes following the unwrapped frame numbers of the stream along with it. The packet is forwarded all the same, since it’s the endpoint who decides what to do with it.
… in use (#2047) Fixes the issue mentioned here: #2015 (comment) DependencyDescriptor::Parse() handed the stream the structure of every descriptor it parsed. A retransmitted or reordered packet of an old key frame thus replaced the one in use, leaving every following packet parsed with outdated templates — wrong layers and wrong decode target indications — until the next key frame. It is now taken only from the first packet of a frame that is not older than the one that brought the structure in use, which takes following the unwrapped frame numbers of the stream along with it. The packet is forwarded all the same, since it’s the endpoint who decides what to do with it.

Details
Encoder. Remove it fromRTP::Packetas it is solely important perConsumer. It should have never lived inRTP::Packet.DependencyDescriptorPayloadDescriptorwhich handles any codec using DD. It's codec agnostic.New
DependencyDescriptorPayloadDescriptorclass.active_decode_targets_bitmask.NOTE: If a consumer does not support DD, we use the codec native payload handler for it, so the needed payload re-writes are done accordingly.
TODO
routerRtpCapabilitiesargument inortc.getPipeConsumerRtpParameters()(also in Rust) and the TODO comment at the end of that function. Basically these commits:RTC::RTP::Packet::SetExtensions()return abool. It returnsfalsewithout modifying the packet when the packet buffer has no room for the given extensions.Producer::MangleRtpPacket()now doesif (!packet->SetExtensions(...)) { ...; return false; }, so the packet is discarded. When resolving the potential conflict, the newpacket->UpdateDependencyDescriptor(...)call must go after that check. Otherwise it would try to update the Dependency Descriptor of a packet whose extensions were never set.Producer::MangleRtpPacket()(if (extenValue && extenLen > 0)). When resolving the conflict in the Dependency Descriptor block, keep theextenLen > 0check. Otherwise a zero length Dependency Descriptor would be proxied again.