Skip to content

Worker: Complete dependency descriptor implementation - #2015

Merged
jmillan merged 72 commits into
v3from
forward-dependency-descriptor
Nov 9, 2026
Merged

jmillan merged 72 commits into
v3from
forward-dependency-descriptor

Conversation

@jmillan

@jmillan jmillan commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

Details

  • Fixes AV1 SVC nor working at all #2013
  • The Dependency Descriptor header extension is 'sendrecv' now, so it's also forwarded to non pipe Consumers.
  • DD header extension is rewritten accordingly and forwarded.
  • Rework/Simplify Codec's Encoder. Remove it from RTP::Packet as it is solely important per Consumer. It should have never lived in RTP::Packet.
  • Created a new DependencyDescriptorPayloadDescriptor which handles any codec using DD. It's codec agnostic.

New DependencyDescriptorPayloadDescriptor class.

  • Its encoder writes the outgoing DD with the correct active_decode_targets_bitmask.
  • AV1 uses it rather than the specific AV1 codec one, since we don't parse AV1.
  • H264:
    • Uses it if the packet comes with DD.
    • Uses current H264::PayloadDescriptor otherwise.
  • VP9:
    • Uses it if the packet comes with DD.
    • Uses current VP9::PayloadDescriptor otherwise.
  • K-SVC:
    • Respect it by droping frames absent from the Consumer’s decode target. This is expressed in the Dependency Descriptor, so we don't even need to explicitly check if we are doing ksvc or not.

NOTE: If a consumer does not support DD, we use the codec native payload handler for it, so the needed payload re-writes are done accordingly.

TODO

  • Must remove routerRtpCapabilities argument in ortc.getPipeConsumerRtpParameters() (also in Rust) and the TODO comment at the end of that function. Basically these commits:
  • Use 'codec' log tag when appropriate.
    • Document it.
  • Add VP9 in mediasoup-client DependencyDescriptorCodecs
  • Consumers not supporting DD RTP extension MUST keep using the corresponding codec handler rather than DD.
  • CAUTION: Ongoing PR Worker: Fix abort due to big received packet (and more) #2022 makes RTC::RTP::Packet::SetExtensions() return a bool. It returns false without modifying the packet when the packet buffer has no room for the given extensions. Producer::MangleRtpPacket() now does if (!packet->SetExtensions(...)) { ...; return false; }, so the packet is discarded. When resolving the potential conflict, the new packet->UpdateDependencyDescriptor(...) call must go after that check. Otherwise it would try to update the Dependency Descriptor of a packet whose extensions were never set.
  • CAUTION: Ongoing PR Worker: Fix abort due to big received packet (and more) #2022 also skips proxied header extensions with length 0 in Producer::MangleRtpPacket() (if (extenValue && extenLen > 0)). When resolving the conflict in the Dependency Descriptor block, keep the extenLen > 0 check. Otherwise a zero length Dependency Descriptor would be proxied again.
  • Consider https://issues.webrtc.org/issues/560293075
  • Parsing a DD overwrites the stream's template structure with one from any parsed packet, even an old retransmitted keyframe, which breaks the following packets. We must ignore attached structures older than the current one, as libwebrtc does. See details in this comment below.

@ibc

ibc commented Oct 26, 2026

Copy link
Copy Markdown
Member

rust CHANGELOG missing.

@ibc

ibc commented Oct 26, 2026

Copy link
Copy Markdown
Member

As said in the issue ticket, I don't think that we should forward the DD extension verbatim. I'm 65% sure that we talked about this long ago and it was not correct to forward it (as it is) when we are filtering layers in the Consumer.

@ibc

ibc commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

As per discussion in the issue ticket in #2013, let's mark this PR as draft until we do further investigation to validate the fix.

@ibc
ibc marked this pull request as draft October 26, 2026 11:34
@satoren

satoren commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

While implementing AV1 simulcast for versatica/mediasoup#2011, I also implemented the DD rewriting this PR’s TODO mentions (filtered layers).

The earlier failure with a rewritten DD (versatica/mediasoup#1625) may come from the old implementation rather than rewriting itself:

WriteExtendedDescriptorFields() wrote the custom dtis/fdiffs/chains flags as 0 and dropped them. libwebrtc uses these whenever a frame doesn’t match its template.
Restore() was a no-op, so a DD rewritten for one Consumer leaked to the others and to retransmissions.

In my branch:

DependencyDescriptor::Write() rewrites only active_decode_targets_bitmask (plus frame number and template id offset on simulcast switches), keeping the structure and custom fields.
RewriteDependencyDescriptor() writes the bitmask of the forwarded layers. The encoder re-applies it on retransmission, and Restore() writes the original back.
It only runs for Consumers that negotiated the DD, so with this PR’s sendrecv it would apply to SVC Consumers as is.
UpdateDependencyDescriptor() can grow into trailing padding, which the Producer reserves.
Tests: TestAV1.cpp.

It’s mixed with the simulcast changes right now, but I can split the DD rewriting part into a separate PR if that helps.

Disclaimer: this branch was implemented by AI and hasn’t been verified on real devices yet.

@jmillan

jmillan commented Oct 27, 2026

Copy link
Copy Markdown
Member Author

Thanks @satoren, we'll check it as reference.

@ibc

ibc commented Oct 27, 2026

Copy link
Copy Markdown
Member

This PR is outdated compared to v3. Please take into account the TODO items in the above PR description.

@ibc

ibc commented Oct 27, 2026

Copy link
Copy Markdown
Member

Some thoughts:

  1. In PR AV1 Issues Investigation (II) #1625 we made DD be recvonly (probably because the rewriting of DD extension was wrong as said 2 comments above) and it kinda "solved" the issue. I mean, AV1 SVC did work.
  2. However, nowadays it no longer works (with AV1 SVC of Chrome) unless we make DD extension be sendrecv, which clearly means that something has changed in libwebrtc/Chrome.
  3. Forwarding the DD extension verbatim (the way it arrives to mediasoup) cannot be right since it may tell a receiver peer that it should receive layers that we are filtering in its mediasoup Consumer on purpose.
  4. So IMHO the only proper way to go here is to make DD sendrecv, address the TODO comments in the PR description above, and do proper DD rewriting based on current preferred layers in the Consumer.

@ibc

ibc commented Oct 27, 2026

Copy link
Copy Markdown
Member

I would also recommend revert all changes in this PR, merge v3 and start from there, otherwise ugly conflicts may happen specially in Rust.

@jmillan

jmillan commented Oct 27, 2026

Copy link
Copy Markdown
Member Author

Some thoughts:

Yes, this is what we're doing.

The Dependency Descriptor header extension is 'sendrecv' now, so it's also forwarded to non pipe Consumers. AV1 receivers have no other way to tell the frames of the different spatial layers apart.
@jmillan
jmillan force-pushed the forward-dependency-descriptor branch from 5722c8f to 68bb1b4 Compare October 27, 2026 10:10

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@jmillan

jmillan commented Oct 27, 2026

Copy link
Copy Markdown
Member Author

but I can split the DD rewriting part into a separate PR if that helps.

@satoren, if you don't mind I'll take that part as reference and use it in this PR.

…ight packet

A cloned packet shares the payload descriptor handler with the original,
and RtpPacketChanged() made the handler's single DependencyDescriptor
write into the clone from then on. Since Router hands the very same
packet to every Consumer and clones it only once, every Consumer after
the one that created the clone had its rewritten descriptor written into
the clone instead of into the packet it was about to send.

Those Consumers therefore transmitted the descriptor as received,
advertising decode targets that they were in fact filtering out, so the
endpoint waited for frames that never arrived and counted them as loss.
Their retransmissions were not affected, since RetransmitPendingPackets()
re-applies the stored encoder to the clone, which is the packet the
listener happened to point at.

Process(), Encode() and Restore() already receive the packet they act on,
so point the descriptor at it on every call rather than remembering the
last clone. RtpPacketChanged() only ever existed to do that and every
implementation of it is now empty, so remove it.

Extend the AV1 section of TestRtpStreamSend so that it also checks the
packet being sent and not just the retransmitted clone.
It returns true if the data rewrite took place, false otherwise.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

The descriptor is now given the packet to write into on every call rather
than remembering one, so it cannot write into the wrong packet.
Comment thread worker/include/Utils.hpp Resolved
Comment thread CHANGELOG.md Resolved
UpdateDependencyDescriptor() only took a descriptor fitting in the
current length of the RTP header extension. A rewritten descriptor
shorter than the received one therefore shrunk the extension for good,
and Restore() could no longer write the original back, so the packet
kept the descriptor of the Consumer that rewrote it first.

It now takes the padding bytes that follow it, which the Producer leaves
behind by telling the length the descriptor really has instead of the
room made for it. Packets are no longer forwarded with trailing zeroes
inside the Dependency Descriptor.

The AV1 test read the retransmitted packet after it had been restored,
so it checked a descriptor that only looked right while Restore() was
failing. It now keeps what the packet carried when it was sent.

@ibc ibc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too hard to review since it's huge and complex stuff but I trust you. I've requested cosmetic changes (or at least commented some concerns).

Please tell the robot to honor this strictly: #2020

This is, clang-tidy doesn't complain if we use auto instead of const auto (but const auto looks much better). And not just that. Pointers:

auto* ptr const = xxxx

and things like that are good to have (see #2020).

Comment thread worker/include/Utils.hpp Resolved
Comment thread worker/meson.build Resolved
jmillan and others added 4 commits November 5, 2026 13:02
Co-authored-by: Iñaki Baz Castillo <ibc@aliax.net>
…iptor.hpp

Co-authored-by: Iñaki Baz Castillo <ibc@aliax.net>
Co-authored-by: Iñaki Baz Castillo <ibc@aliax.net>
@ibc

ibc commented Nov 5, 2026

Copy link
Copy Markdown
Member

Please update branch just in case.

@ibc ibc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE: If a consumer does not support DD, we use the codec native payload handler for it, so the needed payload re-writes are done accordingly.

Is this tested with VP9 and H264? (I know it cannot work with AV1 and we don't use DD in VP8).

@jmillan

jmillan commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

If a consumer does not support DD, we use the codec native payload handler for it, so the needed payload re-writes are done accordingly.

Is this tested with VP9 and H264? (I know it cannot work with AV1 and we don't use DD in VP8).

Yes, given the new mediasoup-demo URL argument disableDD.

@jmillan

jmillan commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

and things like that are good to have (see #2020).

Done here 8122cfe

@satoren, may I ask you for a last review? I've tested everything (tomorrow I'll do a last test). mediasoup-demo has now a URL param disableDD (Remove "dependency-descriptor" RTP header extension from router RTP capabilities before loading the mediasoup-client), so there can be a producer with DD, a consumer with DD and another consumer without DD. Just in case you'd like to test.

@satoren

satoren commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

@satoren, may I ask you for a last review?

LGTM, thanks for addressing all the feedback!

One non-blocking note, which disableDD makes easy to try: if the Producer doesn't send DD (e.g. VP9 produced with disableDD) but the Consumer negotiated it, EncodingContext still uses the 16-bit DD frame id length, while the VP9 handler records its 15-bit pictureIds there. The only effect is that reordered packets around a pictureId wrap (~18 min at 30 fps) can be judged by the current layers instead of the ones of their picture, so it's fine as a follow-up or to leave as is.

The Dependency Descriptor RTP extension given to a Consumer is written out
of the one in the packets of this Producer, so don't announce it to
Consumers unless this Producer negotiated it. Otherwise a Consumer would
negotiate an extension that is never going to arrive, and it would wrap its
frame ids at the 16 bits of the Dependency Descriptor rather than at the 15
of the VP9 picture ids it is actually given.
@jmillan

jmillan commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

One non-blocking note, which disableDD makes easy to try: if the Producer doesn't send DD (e.g. VP9 produced with disableDD) but the Consumer negotiated it, EncodingContext still uses the 16-bit DD frame id length, while the VP9 handler records its 15-bit pictureIds there. The only effect is that reordered packets around a pictureId wrap (~18 min at 30 fps) can be judged by the current layers instead of the ones of their picture, so it's fine as a follow-up or to leave as is.

This is a good observation. Handled in the last commit. Basically we are now filtering out DD from the Consumer RTP extensions if the producer does not offer it, the same way we did with abs-capture-time cc:@ibc.

Tested with and without pipe transports.

The outcome is:

  • Producer NOT offering DD.
    • Frame length of 15 is used regardless of the Consumer supporting DD.
  • Producer offering DD.
    • Frame length of 16 used if Consumer supports DD or 15 otherwise.

Comment thread node/src/ortc.ts
continue;
}

// The Dependency Descriptor RTP extension given to a Consumer is written out

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"written out"

🤖

Comment thread node/src/ortc.ts Resolved
@ibc

ibc commented Nov 6, 2026

Copy link
Copy Markdown
Member

This is a good observation. Handled in the last commit. Basically we are now filtering out DD from the Consumer RTP extensions if the producer does not offer it, the same way we did with abs-capture-time

When/where did we do that with abs-capture-time?
BTW, as commented above: #2044

@jmillan

jmillan commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

When/where did we do that with abs-capture-time?

Hello @ibc

Screenshot 2026-11-06 at 11 41 33

@ibc

ibc commented Nov 6, 2026

Copy link
Copy Markdown
Member

When/where did we do that with abs-capture-time?

Hello @ibc

Screenshot 2026-11-06 at 11 41 33

I forgot. But it's a hack. Maybe not.

@ibc

ibc commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

While working on ongoing PR #2046 I've found a problem that ALSO affects this branch (I've also added a bullet about it in the above TODO list).

Problem

DependencyDescriptor::Parse() hands the stream over the template dependency structure attached to any descriptor it parses successfully, regardless of the frame that descriptor belongs to. A packet of an older key frame that arrives late (a retransmission recovered through RtpStreamRecv::ReceiveRtxPacket(), or just a reordered packet) thus replaces the structure in use with an outdated one, and every following packet of the stream is parsed with the wrong templates (wrong spatial and temporal layers, wrong decode target indications) until the next key frame arrives.

How does libwebrtc solve it?

In RtpVideoStreamReceiver2::ParseGenericDependenciesExtension() (video/rtp_video_stream_receiver2.cc):

  • It unwraps the 16-bit frame_number of every descriptor into a 64-bit frame_id with a per-stream SeqNumUnwrapper<uint16_t> frame_id_unwrapper_.
  • It parses the descriptor with the structure in use (video_structure_).
  • If the descriptor carries a structure in a packet that is not the first one of its frame, it drops the packet.
  • If the descriptor carries a structure and video_structure_frame_id_ > frame_id (this is, it belongs to a key frame older than the one that brought the structure in use), it logs a warning and drops the packet.
  • Otherwise it replaces video_structure_ with the new structure and sets video_structure_frame_id_ = frame_id.

What must be done in this branch?

  • Half of it is already done: ReadExtendedDescriptorFields() parses an attached structure into a new std::shared_ptr instead of overwriting the one in use, so the packet that carries it is parsed with its own structure.
  • What is missing is a condition on the hand over in DependencyDescriptor::Parse() (templateDependencyStructure = dependencyDescriptor->templateDependencyStructure;): it must only happen if the frame of the descriptor is not older than the frame that brought the structure in use.
  • To tell that, keep per stream, along with the structure in use:
    • A Utils::UnwrappedSequenceNumber<uint16_t>::Unwrapper (our counterpart of libwebrtc's SeqNumUnwrapper<uint16_t>).
    • A std::optional<Utils::UnwrappedSequenceNumber<uint16_t>> with the unwrapped frame number of the frame that brought the structure in use.
  • For every successfully parsed descriptor, call Unwrap(frameNumber) on that unwrapper, not only for descriptors that carry a structure, so it follows the progress of the frames as libwebrtc does.
  • If the descriptor carries a structure, hand it over only if there is no stored frame number yet or the unwrapped frame number of the descriptor is not lower than the stored one, and then store the new one.
  • The packet itself must not be dropped: an SFU forwards it and the receiver's decoder decides what to do with it.
  • Optionally, and as libwebrtc does, ignore a structure attached to a packet that is not the first one of its frame.

@jmillan

jmillan commented Nov 9, 2026

Copy link
Copy Markdown
Member Author

@ibc, that's already fixed in the branch forward-dependency-descriptor.

@jmillan

jmillan commented Nov 9, 2026

Copy link
Copy Markdown
Member Author

@ibc, that's already fixed in the branch forward-dependency-descriptor.

NOTE: At least part of it is fixed as a retransmission buffer packet holds the template structure of the original RTP packet and that is never overwritten. That was a bug that has been fixed in the DD PR.

I'll check for the complete claim here.

@jmillan
jmillan merged commit 354eedb into v3 Nov 9, 2026
50 checks passed
@jmillan
jmillan deleted the forward-dependency-descriptor branch November 9, 2026 14:46
jmillan added a commit that referenced this pull request Nov 9, 2026
… in use

Fixes the issue mentioned here: #2015 (comment)

DependencyDescriptor::Parse() handed the stream the structure of every descriptor it parsed. A retransmitted or reordered packet of an old key frame thus replaced the one in use, leaving every following packet parsed with outdated templates — wrong layers and wrong decode target indications — until the next key frame.
It is now taken only from the first packet of a frame that is not older than the one that brought the structure in use, which takes following the unwrapped frame numbers of the stream along with it. The packet is forwarded all the same, since it’s the endpoint who decides what to do with it.
jmillan added a commit that referenced this pull request Nov 10, 2026
… in use (#2047)

Fixes the issue mentioned here: #2015 (comment)

DependencyDescriptor::Parse() handed the stream the structure of every descriptor it parsed. A retransmitted or reordered packet of an old key frame thus replaced the one in use, leaving every following packet parsed with outdated templates — wrong layers and wrong decode target indications — until the next key frame.
It is now taken only from the first packet of a frame that is not older than the one that brought the structure in use, which takes following the unwrapped frame numbers of the stream along with it. The packet is forwarded all the same, since it’s the endpoint who decides what to do with it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

AV1 SVC nor working at all

3 participants