Client-side sonification for datacubes #3843
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
CC @javerbukh @mariobuikhuizen @kecnry
Reimplementation of javerbukh#33 to avoid complex rebase, see original description below:
Initial implementation with playback handling fully in JS so we can but audio rendering client side.
With the JS-heavy implementation, some diagnostic logging has been added so turn on the JS console in the browser to see this.
Basic summary of behaviour: This creates the sonified cube as before, but flattens the array into a single 1D audio buffer of audio clips stacked end-to-end, representing the entire cube. Each (x,y) pixel then represents an exact audio sample range in the audio clip. To handle chnges between these we operate two audio players where:
For this to work we need to hold two copies of the audio in memory at once. the size of the audio clip depends on the number of pixels. We can catch big cubes. Need to work out tolerable limits for transmitting audio data and whether to load staggered. would also need to generallise player A and B to list of players to handle simultaneously playing audio layers (rather than just most recent)
Outstanding issues / Questions:
Tone.js- an external library. Currently it pulls it when sonify plugin is opened, but could we package with install for offline mode?jdaviz/configs/imviz/plugins/coords_infoandjdaviz/configs/default/plugins/markersfor inspirationSome additional issues noticed since:
mainproblem or specific to the new features yetstraussrequirement: currentlystraussisn't required by default so will say sonification unavailable until it is installed manually - is there a reason not to add?This pull request is to solve client-side sonification, making sonification via platforms viable
Fixes #
Change log entry
CHANGES.rst? If you want to avoid merge conflicts,list the proposed change log here for review and add to
CHANGES.rstbefore merge. If no, maintainershould add a
no-changelog-entry-neededlabel.Checklist for package maintainer(s)
This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.
triviallabel.cache-download.ymlworkflow?