Skip to content

replay: fix video decoding from files with audio streams #35715

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jul 14, 2025

Conversation

deanlee
Copy link
Contributor

@deanlee deanlee commented Jul 13, 2025

Fixes FrameReader to properly handle video files that contain audio streams.

resolve: #35711

@github-actions github-actions bot added the tools label Jul 13, 2025
@deanlee deanlee force-pushed the replay_fix_video_decoding branch from 69d8153 to bd85dec Compare July 14, 2025 07:03
@deanlee deanlee marked this pull request as ready for review July 14, 2025 07:17
@sshane sshane requested a review from Copilot July 14, 2025 21:41
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes FrameReader to correctly handle video files that include audio streams by explicitly finding and using the video stream rather than assuming stream 0.

  • Use av_find_best_stream to locate the video stream and store its index in video_stream_idx_
  • Filter packet reading and decoder acquisition to the identified video stream
  • Revise decoding loop to skip non-video packets and return the requested frame

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
tools/replay/framereader.h Added video_stream_idx_ member to track stream
tools/replay/framereader.cc Updated stream selection, packet filtering, and decoding logic
Comments suppressed due to low confidence (1)

tools/replay/framereader.cc:83

  • Consider adding a test with a sample file containing both audio and video streams to verify that loadFromFile correctly identifies the video stream and processes frames without audio interference.
  video_stream_idx_ = av_find_best_stream(input_ctx, AVMEDIA_TYPE_VIDEO, -1, -1, NULL, 0);

break;
}
}

auto pos = reader->packets_info[from_idx].pos;
auto pos = reader->packets_info[current_idx].pos;
int ret = avformat_seek_file(reader->input_ctx, 0, pos, pos, pos, AVSEEK_FLAG_BYTE);
Copy link
Preview

Copilot AI Jul 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The call to avformat_seek_file uses stream index 0, which may refer to an audio stream if present. Replace the hardcoded 0 with reader->video_stream_idx_ to seek the correct video stream.

Suggested change
int ret = avformat_seek_file(reader->input_ctx, 0, pos, pos, pos, AVSEEK_FLAG_BYTE);
int ret = avformat_seek_file(reader->input_ctx, reader->video_stream_idx_, pos, pos, pos, AVSEEK_FLAG_BYTE);

Copilot uses AI. Check for mistakes.

Comment on lines +210 to +211
rError("Failed to decode frame at index %d", current_idx);
return false;
Copy link
Preview

Copilot AI Jul 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

decodeFrame may return null when the packet does not immediately yield a frame; instead of erroring out here, you should continue reading subsequent packets until the requested frame is decoded.

Suggested change
rError("Failed to decode frame at index %d", current_idx);
return false;
rWarning("Frame not decoded, continuing to read subsequent packets...");
continue;

Copilot uses AI. Check for mistakes.

@sshane sshane merged commit 97f6dc6 into commaai:master Jul 14, 2025
15 checks passed
@deanlee deanlee deleted the replay_fix_video_decoding branch July 14, 2025 22:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Replay doesn't properly load new routes with microphone audio
2 participants