obs-outputs: Enable eRTMP Flac and Opus support

[yuriy-chumak]

Description

This PR enables Flac and Opus audio codecs for the eRTMP protocol according the spec.

Note: the eRTMP backend may not accept both codecs.

Motivation and Context

Add more audio codecs to the stream.

How Has This Been Tested?

It has been tested using:

1. OBS Studio as RTMP sender (Linux, Windows, macOS),
2. modified FFmpeg instance as eRTMP server (Linux only),
3. modified FFplay instance as RTMP client (Linux only).

The provided FFMpeg patch makes the modified ffmpeg server accept flac and opus audio codecs inside the eRTMP stream.

Types of changes

• Tweak (non-breaking change to improve existing functionality)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

---

Testing

The PR can be tested using a modified FFmpeg library. Modified FFmpeg accept flac and opus audio codecs inside the eRTMP stream.

The following steps are a working example of the testing pipeline.

Preparation

• Clone the https://github.com/FFmpeg/FFmpeg repository,
• Apply provided patch,
• Build the FFmpeg library including FFmpeg and FFplay.

Configure and Run Backend (Linux)

• Note empty rtmp port for a backend instance. Let it be 8889.
• Create next shell script to run the backend (change the ${port-number} to the port number, change the ${the_folder_with_ffmpeg} to the real modified FFmpeg path).

  #!/bin/sh
  export LD_LIBRARY_PATH=${the_folder_with_ffmpeg}
  export PATH=${the_folder_with_ffmpeg}:$PATH
  
  while true; do
  echo =============================== `date` ============================
  ffmpeg -f flv -listen 1 \
     -v trace -hide_banner \
     -i rtmp://localhost:${port-number}/live/app \
     -c:v copy -c:a copy \
     -f flv - \
  |  ffplay - -autoexit
  
     sleep 1
  done

• Run backends by running rtmp-server.sh.

Configure and Run OBS Studio (any OS)

• Run OBS Studio and change the next "Stream" settings:
  • Service: Custom...
  • Server: rtmp://localhost:8889/live (the port value must be a port of a backend)
  • Stream Key: "app"
    
• Change the next "Output" settings:
  • Select "Audio Encoder" to the "FLAC" or "Opus" with "Output Mode" set to "Advanced".
• Start OBS-Studio Streaming.
  We must see the running logs in the backend console, and a proper status notifications in the OBS-Studio.
  
  We must hear the proper audio with automatically runned FFplay.

[tytan652]

This requires to update the services.json to not allow FLAC and Opus on every RTMP service.

[RytoEX]

Does this require OBS to be built against a patched FFmpeg to work? Or is the patch being used as a demonstration for receiving the stream? Has this patch been submitted to FFmpeg?

As mentioned in the existing review, services.json will require updates to the supported audio codecs key for each service entry that does not support FLAC/Opus, unless it is demonstrated that UX is not impacted without that change.

[yuriy-chumak]

Does this require OBS to be built against a patched FFmpeg to work?

No

Or is the patch being used as a demonstration for receiving the stream?

Yes, just a way to check is a PR code workable.

Has this patch been submitted to FFmpeg?

No.

As mentioned in the existing review, services.json will require updates to the supported audio codecs key for each service entry that does not support FLAC/Opus, unless it is demonstrated that UX is not impacted without that change.

Ok... I need to check. Will comment shortly.

[TroyKomodo]

This pr almost works perfectly just one issue we found while testing against our RTMP Enhanced v2 impl (ScuffleCloud/scuffle#396) using https://github.com/lennartkloock/poc-rtmp-server

Tested using the artifacts generated by https://github.com/TroyKomodo/obs-studio/actions/runs/13859647197/job/38784696062

This implementation is not working correctly with enhanced audio enabled.
See the below for a dump of the binary sent.
stream-dump.zip

Hex Offset	Significance
0x142F	audiocodecid starts in the onMetaData payload
0x143C	The codec sent was 10 which is the codec for AAC

This points to

obs-studio/plugins/obs-outputs/flv-mux.c

Line 231 in 1f70c3f

enc_num_val(&enc, end, "audiocodecid", AUDIODATA_AAC);

[TheB1gG]

There are many media servers which did implement flac via ertmp in the last year e.g. mediamtx and it is supported in ffmpeg without any need for additional patches. So it would nice to get this PR merged, because I really would like to offer flac on my own plattform without the need to have users build a modified OBS version.
Thank you.