OpenGL filter prototype: no frame received

[rom1v]

On-device OpenGL video filters will allow to:

• reimplement --lock-video-orientation on Android >= 14 to fix Android 14 beta lock orientation doesn't work as expected when the device goes into landscape mode. #4011
• reimplement --crop on Android >= 14 to fix Android 14 Beta 5 crop not working (Pixel 6 Pro) #4162
• apply a custom capture rotation, useful to mirror a Meta Quest 3 for example (Add --scale, --position-x-offset, --position-y-offset and --rotation-offset #4658)
• support virtual display rotation (Virtual Display Rotation Listener Problem #5428)
• rotate the camera capture for V4L2 (Feature request: Allow to rotate the camera. #4426)

I wrote a quick&dirty prototype (branch opengl_filter), but I'm blocked because onNewFrameAvailable() is never called:

scrcpy/server/src/main/java/com/genymobile/scrcpy/video/VideoFilter.java

Line 80 in 4397dfb

public void onFrameAvailable(SurfaceTexture surfaceTexture) {

I'm pretty sure this is related to threading (setOnFrameAvailableListener() also accepts a Handler), but I did not manage to make it work.

Once this listener is called on every frame, I can implement the shaders to perform the transforms I want, but I need this to work to get started.

Any help welcome.

[yume-chan]

https://github.com/Genymobile/scrcpy/compare/opengl_filter...yume-chan:scrcpy:opengl_filter_v2?expand=1

I copied some OpenGL code from https://github.com/google/grafika, and mimicked how https://cs.android.com/android/platform/superproject/main/+/main:frameworks/av/cmds/screenrecord/Overlay.cpp initialized it.

updateTexImage needs to be called on the same thread as the SurfaceTexture is created, and the SurfaceEncoder thread is busy in a while loop, so we can only create it in a new thread.

setOnFrameAvailableListener requires the current (or specified) looper is in looping state.

[rom1v]

Thank you very much! It works perfectly.

I have everything I need to start working on it 🎉 ❤️