controlling video added with useVideoTexture

[amcc]

I have video successfully working with useVideoTexture as per https://github.com/pmndrs/drei?tab=readme-ov-file#usevideotexture. Its being used in conjunction with MindAR using this library: https://github.com/tommasoturchi/react-three-mind

I want to be able to unmute the video and perhaps start/stop the video. On mobile you must start the video muted or safari in ios displays black video (or errors locally). The AR provides callbacks for the target mounted/unmounted so this would be suitable for me to try to unmute the video. I can't see any way to do this with useVideoTexture. Other option is to use a more standard three.js video texture - but would like to use this

[netgfx]

@amcc You can check this example https://codesandbox.io/p/sandbox/white-smoke-wx5zht?file=%2Fsrc%2FApp.js%3A30%2C49 that shows how you can play/pause or mute/unmute a video

[amcc]

thanks for the help. I'm not sure i need the complexity within the HSL texture you have there. However now i look at that example it does seem that useVideoTexture has a reference to the source video, so i might be able to control it that way.

I have also been trying this via the more default threejs video texture, however iOS is giving me a few issues.

[netgfx]

@amcc You don't need the HLS stuff I just had that already in that sandbox, it should work the same with the default useVideoTexture. The underlying video element is under texture.image (which is unfortunate as a property name :( )

[amcc]

looks like texture.image doesn't exist on useVideoTexture, however the video source is there:
texture.source.data

This contains the html video element, so i can set muted to false on that.

It looks like your example isn't actually using drei useVideoTexture, it uses THREE.VideoTexture which must use the (i agree unfortunate) texture.image

So far this seems to work on ios

[netgfx]

Hm weird if you see the source code for useVideoTexture https://github.com/pmndrs/drei/blob/master/src/core/useVideoTexture.tsx it is almost identical to my component (this is were I extended from). And useVideoTexture actually returns a VideoTexture object

[amcc]

Here's what i've done - i'll need to do some more testing on mobile devices that are very fussy about how video is used (with sound)

Using this hook: https://github.com/pmndrs/drei?tab=readme-ov-file#usevideotexture

i've changed the VideoMaterial in the above example to take a state 'muted' in order to enable the sound when i detect an anchor (using the callback from MindAR 'onAnchorFound' to trigger the statechange:

function VideoMaterial({ muted, url }) {
  const props = {
    // unsuspend: "canplay",
    crossOrigin: "Anonymous",
    muted: muted,
    loop: true,
    playsInline: true,
  };
  const texture = useVideoTexture(url, props);

  if (!muted) {
    texture.source.data.muted = false;
    texture.source.data.play();
  } else {
    texture.source.data.muted = true;
    texture.source.data.pause();
  }
  return <meshBasicMaterial map={texture} toneMapped={false} />;
}

still having issues with ios - works on my phone, but not on another iphone at the moment

[netgfx]

@amcc I have updated my example to work with useVideoTexture from drei directly, and it still works https://codesandbox.io/p/sandbox/white-smoke-wx5zht?file=%2Fsrc%2FApp.js%3A26%2C5

[amcc]

youre right it is there for me too - sorry, also can find it under source.data - i wonder if there's a difference. Still struggling with safari giving an error. This works in chrome:

const texture = useVideoTexture(url, props);
  const videoElement = texture.image;
  if (!muted) {
    videoElement.muted = false;
    videoElement.play();
  } else {
    texture.source.data.muted = true;
  }

safari errors:
The request is not allowed by the user agent or the platform in the current context, possibly because the user denied permission.

[netgfx]

does something like this work? https://arrangeactassert.com/posts/how-to-fix-the-request-is-not-allowed-by-the-user-agent-or-the-platform-in-the-current-context-possibly-because-the-user-denied-permission/

[amcc]

i'm looking at exactly that example - thus far no, but probably doing something wrong. Tried this:

if (!muted) {
    videoElement.addEventListener("canplaythrough", () => {
      console.log("canplaythrough");
      videoElement.muted = false;
      videoElement.play();
    });
  }

going to give it a try tomorrow. will post results and thanks so much for the help so far

[amcc]

Solved the issue - my problem was that i was trying to change properties on the video element (texture.image) within the videoMaterial component:

originally i was passing in a state 'muted'

function VideoMaterial({ url, setVideoElement, muted }) {
  const texture = useVideoTexture(url);
  
  // change textute.image.muted to be false here..
  return <meshBasicMaterial map={texture} toneMapped={false} />;
}

I realised that this is essentially re-rendering this component each time muted state changes, making a new video element, which probably causes safari to need a new click - its not going to be synchronous so we get the error in safari.

The solution is to handle the mute/unmute within the parent component. I think we could do this a few ways, I chose to pass down a state setter to the VideoMaterial like so:

function Plane({ setVideoElement }, ...props) {
  return (
    <mesh {...props}>
      <boxGeometry args={[1, 1, 0.1]} />
      <Suspense fallback={<FallbackMaterial url={fallbackURL} />}>
        <VideoMaterial url={videoUrl} setVideoElement={setVideoElement} />
      </Suspense>
    </mesh>
  );
}

function VideoMaterial({ url, setVideoElement }) {
  const texture = useVideoTexture(url);
  setVideoElement(texture.image);
  return <meshBasicMaterial map={texture} toneMapped={false} />;
}

In the App component i'm doing this:

  const [videoElement, setVideoElement] = useState(null);

  const toggleSound = () => {
    videoElement.muted = !videoElement.muted;
  };

   render (
   // rest of component
   <button onClick={() =>  toggleSound()}>
          turn on sound
   </button>
   <Plane setVideoElement={setVideoElement} />
   // rest of component
   )

[netgfx]

@amcc Why not just use a ref? Mutating ref properties doesn't cause re-render

[amcc]

That’s a much better idea and sounds like the correct thing to do! Does seem to work as it is however. Also worth mentioning is that desktop safari causes a lot of the same issues. Eg try running a Mac laptop on battery and setting the low power mode to always on battery, you find it has the same behaviour as iOS on low power mode and causes a similar react error (this can be dismissed however and is only in development). You see this error on your example in this situation too

…

On Sat, 17 Feb 2024 at 22:44, Michael Dobekidis ***@***.***> wrote: @amcc <https://github.com/amcc> Why not just use a ref? Mutating ref properties doesn't cause re-render — Reply to this email directly, view it on GitHub <#1832 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA4EX5PGDTV3RAONQUVF5TYUEXENAVCNFSM6AAAAABDMGWBAGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DKMBTG4ZTS> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[amcc]

@netgfx Thanks for all the help - got it working, in case the solution is useful i put it all in a repo here:
https://github.com/amcc/react-three-mind-video

See the app.js for how I did it:
https://github.com/amcc/react-three-mind-video/blob/main/src/App.js

[netgfx]

Nice! @amcc
I believe you could isolate some more things like lights etc in a useMemo to avoid re-rendering of all the scene, or wrap the video element in a separate component so only that re-renders, this might come in handy if you add 3D Models or other heavy stuff to the scene.

[amcc]

@netgfx true - i will be moving components to their own files, i wanted to keep this super simple as a test case for now. useMemo is a good call, i'll do that.

I have another issue with this, which you might be able to help with - the built site has the lights working fine - but they're broken in the dev version only - you seen this behaviour before?

I'm including drei in order to use the useVideoMaterial hook, but react-three-fibre and drei are already dependencies of react-three-mind (which is doing the AR stuff) - so i think i've 2 different versions included overall. I'm getting warnings in develop about multiple versions of three and i think there's possibly some issue with the newer version of drei in the App.js

I elaborated more on the exact warnings here if you're interested: tommasoturchi/react-three-mind#21