camera-controls + Android WebXR = `camera.position` not updating + frustum culling breaking

[FrostKiwi]

Describe the bug

This one was a headache to debug, but on React Three Fiber XR, when <CameraControls> is in the scene, then Android WebXR immersive-ar has camera.position stuck at the origin, breaking frustum culling. The visual camera, the view of the Android Phone still moves and updates correctly, but camera.position remains at the origin, breaking things like frustum culling.

Bizzarely, this is fine on Quest 3 WebXR. This only happens with my Pixel 9 Android. Something about Android WebXR and camera-controls doesn't mix well.

Using https://pmndrs.github.io/xr/docs/tutorials/guards to do this:

<IfInSessionMode deny="immersive-ar">
    <Camera />
</IfInSessionMode>

Fixes it for me.

To Reproduce

Steps to reproduce the behavior:

1. Run a immersive-ar WebXR scene on Android Chrome
2. Check to see frustum culling breaking, as camera.position doesn't update.

Code

Sorry for being out of context, but here is my full camera component. It should not interact with WebXR, as WebXR handles the camera on its own, but bizzarely, somehow it does in the specific case of Android Chrome WebXR, where the camera remains nailed at the origin.

import { CameraControls } from '@react-three/drei';
import { useRef, useEffect } from 'react';
import { setCustomData } from 'r3f-perf'
import * as THREE from 'three';
import { useThree, useFrame, invalidate } from '@react-three/fiber';
import { useStore } from 'src/store';
import { CameraControlsImpl } from '@react-three/drei';

const Camera = () => {
	const controlsRef = useRef(null);
	const baseFOVRef = useRef(null);
	const { scene, size, gl } = useThree();
	const currentTab = useStore((state) => state.tabName);
	const viewOffset = useStore((state) => state.viewOffset);
	const sceneLoaded = useStore((state) => state.internal.sceneLoaded);

	/* Default Setup */
	useEffect(() => {
		const controls = controlsRef.current;
		baseFOVRef.current = controls.camera.fov; /* Capture base FOV for portrait adjustment */
		controls.camera.near = 0.01;
		controls.camera.updateProjectionMatrix();

		controls.maxDistance = 5;
		
		// Disable built-in wheel controls - using custom handler instead
		controls.mouseButtons.wheel = CameraControlsImpl.ACTION.NONE;
		
		invalidate();
		requestAnimationFrame(() => controls.normalizeRotations().setLookAt(-2, 1.5, 3, 0.470086, 1.16085, 0, true));
		controls.setBoundary(new THREE.Box3(new THREE.Vector3(-50, 0.1, -50), new THREE.Vector3(50, 50, 50)));

		controls.saveState();
	}, []);

	/* Custom wheel handler - reimplemented from camera-controls */
	useEffect(() => {
		const domElement = gl.domElement;
		const isMac = /Mac/.test(globalThis?.navigator?.platform);
		
		const handleWheel = (event) => {
			const controls = controlsRef.current;
			if (!controls || !controls.enabled) return;

			event.preventDefault();

			/* Delta calculation from original implementation */
			const deltaYFactor = isMac ? -1 : -3;
			const delta = (event.deltaMode === 1 || event.ctrlKey) ? 
				event.deltaY / deltaYFactor : 
				event.deltaY / (deltaYFactor * 10);

			/* The required invalidate */
			invalidate();
			
			/* Wait for the next frame to start the operation */
			requestAnimationFrame(() => {
				const currentControls = controlsRef.current;
				if (!currentControls) return;

				/* Simplified for perspective camera only - always dolly (no cursor tracking) */
				currentControls._dollyInternal(-delta, 0, 0);
				currentControls._isUserControllingDolly = true;

				/* Dispatch control event */
				currentControls.dispatchEvent({ type: 'control' });
			});
		};

		domElement.addEventListener('wheel', handleWheel, { passive: false });
	}, []);

	/* Portrait orientation FOV adjustment */
	useEffect(() => {
		const controls = controlsRef.current;
		const mediaQuery = window.matchMedia('(orientation: portrait)');

		const updateFOV = () => {
			/* Increase FOV portrait orientation */
			controls.camera.fov = mediaQuery.matches ? 95 : 75;
			controls.camera.updateProjectionMatrix();
			invalidate();
		};

		updateFOV();
		mediaQuery.addEventListener('change', updateFOV);

		return () => mediaQuery.removeEventListener('change', updateFOV);
	}, []);

	useEffect(() => {
		const controls = controlsRef.current;
		controls.camera.setViewOffset(size.width, size.height, viewOffset[0], viewOffset[1], size.width, size.height);
		controls.camera.updateProjectionMatrix();
		invalidate();
	}, [size, viewOffset]);

	/* report azimuth during debug */
	useFrame(() => {
		const ctl = controlsRef.current
		if (ctl) setCustomData(THREE.MathUtils.radToDeg(ctl.azimuthAngle))
	})

	/* currentTab Manager */
	useEffect(() => {
		const controls = controlsRef.current;
		if (!controls) return;

		let camera = scene.getObjectByName(currentTab);
		if (!camera && !sceneLoaded) return;
		if (!camera) camera = scene.getObjectByName("Default");
		if (camera) {
			const { position } = camera;
			const target = camera.userData.TargetPos || [];
			const [azimuthMin = 0, azimuthMax = 0] = camera.userData.AzimuthMinMax || [];
			const [distanceMin = 0, distanceMax = 5] = camera.userData.DistanceMinMax || [];

			invalidate();
			requestAnimationFrame(() =>
				controls.normalizeRotations().setLookAt(position.x, position.y, position.z, target[0] || 0, target[1] || 0, target[2] || 0, true)
			);
			controls.minAzimuthAngle = azimuthMin === 0 ? -Infinity : azimuthMin;
			controls.maxAzimuthAngle = azimuthMax === 0 ? Infinity : azimuthMax;
			controls.minDistance = distanceMin;
			controls.maxDistance = distanceMax;
		} else {
			controls.minAzimuthAngle = -Infinity;
			controls.maxAzimuthAngle = Infinity;
			controls.reset(true);
		}
	}, [currentTab, scene, sceneLoaded]);

	return <CameraControls ref={controlsRef} boundaryEnclosesCamera={true} />;
};

export default Camera;

Live example

No response

Expected behavior

ThreeJS's camera.position should update even when in immersive-ar with WebXR on Android.

Screenshots or Video

No response

Device

Mobile

OS

Android

Browser

Chrome

[yomotsu]

Thanks for the report.

I think the solution is to avoid using CameraControls in XR. CameraControls isn't designed for XR: in immersive sessions the camera transform is driven by the XR pose, and per-frame updates from orbit-style controls will conflict with that. Please use controls designed for XR instead.

[FrostKiwi]

@yomotsu

CameraControls isn't designed for XR: in immersive sessions the camera transform is driven by the XR pose, and per-frame updates from orbit-style controls will conflict with that

WebXR takes over the camera, overrides everything, in a new WebXR specific RenderLayer. So it shouldn't matter. In fact, nowhere in the React Three Fiber XR does it says you have to exclude the scene camera from the parent scope.

But specifically for ThreeJS's Frustum culling this "override" goes wrong for some reason, specifically if this package is used. Something is not properly overridden and that causes frustum culling to think the camera is at the origin.