Trame RCA - Stream Video

[tmarrinan]

Hello!
I am trying to create a Trame app that uses a custom visualization (rather than VTK or ParaView) and streams images/video for display on the web application. I followed this example that uses Trame RCA, but used PIL to create a simple image rather than VTK.
I was able to successfully get the image generated by PIL to show on the web app and get user events back into the Trame app using the RemoteControlledArea. However, if I switched the mime type from image/jpeg it no longer worked.
I am curious how to support streaming video (h264 or h265) instead of image streaming? I also see in the Trame documentation that RCA has ImageDisplayArea, VideoDecoderDisplayArea, and MediaSourceDisplayArea, but do not see any examples or documentation on how to use them. Any help in this area would be greatly appreciated!

[jourdain]

We have one based on async here. Hope that helps.

[tmarrinan]

Thank you - that example did provide a bit more help. I was able to change the media type (<img> for "image", <canvas> for "raw-image", and <video> for "media-source"). I was also able to send raw RGB data and get it to display using "raw-image". However, I am still not quite sure about the video codecs. I saw things like "avc1.64003e" rather than something simple such as "h264". Is there somewhere to look up the valid codecs (and their meanings) that Trame RCA expects?

[jourdain]

For video codec it is trickier and dependent of some browser capabilities. This field get used inside that webworker.

[tmarrinan]

One more question on the topic. By following the examples, I can stream raw RGB data or JPEG images, but I seem to be overwhelming the network.
I see target_fps used to simply delay the rendering and push of the next image from the server. However, I think that the actual time to stream the image is > 1/30 of a second (especially for raw RGB) and thus many images are being backlogged in transit from the server to the client.
Is there any way to get a callback when the client has successfully received the updated image? I do not see anything in the state changing and cannot figure out if any other callbacks are available on the VContainer other than click. Such a callback would enable me to hold off on sending the next image until confirmation that the prior one has been received by the client.

[jourdain]

We have an unfinished PR for that Kitware/trame-rca#13

[tmarrinan]

Thanks! I've been keeping an eye on the PR, but haven't noticed anything for a few months. Is there any update on its status?

[jourdain]

We don't have any active project pushing it right now. If you have a need, you could submit your own PR or funds us to help complete it faster for you.