reactmx WIP

[aureliendavid]

[ use https://github.com/aureliendavid/gpac/tree/mxwithbuild branch and share/reactmx/build/index.html file to run the UI without npm/react ]

attempts and tests towards a mxgraph-based filters bench UI for gpac

(this is very WIP and unstable and unorganized, this branch serves as dev backup mostly for now)

goals:

• display the filters graph of a running gpac session (in poc)
• display "realtime" info about the filters (in poc)
• live edit of filter properties that are editable (in poc, kinda)
• live probing with temporary filter chains for AV pids (e.g. display png preview of decoded frame) (not in poc, some tests started)
• adding/removing filters from live session (not done)

basic implementation ideas:

• hijack remotery LOG messages to exchange text (json) via websocket between native gpac and browser
• use javascript api as a controller of the filter session on the gpac side (reactmx/mx.js)
• use react for "realtime" display on the browser side (reactmx/src/index.js)
• use mxgraphs auto layout to display the filter bench graph (a lot of libs are stomped because filters benches are actually multigraphs)

usage:

• run gpac command with both -rmt and -js share/reactmx/mx.js

e.g.:
gpac -p=0 -rmt -js=mx.js -graph -stats -i bcs.mkv aout vout

• for dev, run npm run start in share/reactmx/ and open localhost:3000 in browser
  OR, for test, just open share/reactmx/build/index.html (from the mxwithbuild branch)

(might need some F5s before something is displayed)

todos:

• continue tests with temporary filters, probes, and FileIO to extract and display png of running session
• review editable status of most filters properties (are they really updatable live? should they?)
• rework remotery / add proper websocket controller in gpac to avoid using LOG messages
• make the whole thing a lot more stable, a lot less ressource consuming, and make the code look less awful (might not be possible to do everything with the current architecture without being a huge ressource hog, so choices would have to be made in this case)
• increase command line usability for end user (with some kind of alias like gpac -webui <args>)
• study feasibility of adding/removing filters live from the UI

[aureliendavid]

Update:

The branch at https://github.com/gpac/gpac/tree/buildbot-rmtws now replaces Remotery with an ad-hoc websocket server.

The main difference for this use case are:

• multi client support
• no remotery protocol encapsulation

A very basic example of gpac js controller for websocket server can be found here: https://github.com/gpac/testsuite/blob/buildbot-rmtws/media/jsf/gpac.js#L17

There are also usage samples for js/node/python in https://github.com/gpac/gpac/tree/buildbot-rmtws/share/rmtws

To use them:

# js
gpac -js=jsrmt.js  -p=0 -rmt  -logs=rmtws@info  avgen reframer:rt=on inspect:deep

# node
node nodermt.js -p=0 -rmt -logs=rmtws@info -f=avgen -f=reframer:rt=on -f=inspect:deep

# python
python3 pyrmt.py -p=0 -rmt -logs=rmtws@info -f=avgen -f=reframer:rt=on -f=inspect:deep

Then open https://github.com/gpac/gpac/blob/buildbot-rmtws/share/rmtws/index.html in a web browser and you should see a json dump of all loaded filters.

The index.html is a barebones websocket client example that dumps what it receives from gpac to a <pre> element.

More sophisticated UIs and data exchange formats can be implemented from there.

---

The branch https://github.com/aureliendavid/gpac/tree/rmtws-reactmx has the same content + the reactmx UI described in the original post.

You can use it with e.g.:

gpac  -js=reactmx/mx2.js -p=0 -rmt  -logs=rmtws@info  avgen reframer:rt=on inspect:deep

Then npm start the UI in share/reactmx.