Skip to content

Conversation

@james-trayford
Copy link
Contributor

Description

CC @javerbukh @mariobuikhuizen @kecnry

Reimplementation of javerbukh#33 to avoid complex rebase, see original description below:


Initial implementation with playback handling fully in JS so we can but audio rendering client side.

With the JS-heavy implementation, some diagnostic logging has been added so turn on the JS console in the browser to see this.

Basic summary of behaviour: This creates the sonified cube as before, but flattens the array into a single 1D audio buffer of audio clips stacked end-to-end, representing the entire cube. Each (x,y) pixel then represents an exact audio sample range in the audio clip. To handle chnges between these we operate two audio players where:

  1. initialise two players, A and B with two copies of the fullaudio clip
  2. on entry of the viewer start players A on that pixel and fade up master audio, with crossfade 100% on player A
  3. player A, say, is the active player current looping current pixel sound
  4. On a change in pixel detected, change the loop points in player B, and cross fade audio from A=>B
  5. switch B to be the active player, on the next change repeat 3->5 with A and B transposed.
  6. on exit, fade down master audio and stop players

For this to work we need to hold two copies of the audio in memory at once. the size of the audio clip depends on the number of pixels. We can catch big cubes. Need to work out tolerable limits for transmitting audio data and whether to load staggered. would also need to generallise player A and B to list of players to handle simultaneously playing audio layers (rather than just most recent)

Outstanding issues / Questions:

  • This uses Tone.js - an external library. Currently it pulls it when sonify plugin is opened, but could we package with install for offline mode?
  • Some of the previous functionality is not yet integrated - the volume slider should be easy, but I don;t think choosing audio device wll work without something complicated with cookies etc. I think we ultimately just remove for client-side playback and just provide default audio.
  • Is there a flag to know if we are operating on a server vs local instance? Currently this operates as before simultaneously - so the audio is still playing through the local available sound device if available. This has been useful for development/ debugging, but in operation need to either use one universal method or switch based on server vs local instance flag.
  • I need to make sure I'm using the right viewer coordinates for the player at all times. I followed jdaviz/configs/imviz/plugins/coords_info and jdaviz/configs/default/plugins/markers for inspiration
  • This should be generalised for audio layers: Can extend this to process lists of paired players to play multiple audio layers - mixing would be handled in JS too.

Some additional issues noticed since:

  • Align viewer coords: I think there is a half-spaxel offset between the spectrum-at-spaxel viewer and the heard sonification, would be good to get advice on the best coordinate set to use, rounding pixels etc.
  • Thread errors: I'm getting a lot of thread errors though things seem to still work as expected- I haven't tested if that is main problem or specific to the new features yet
  • strauss requirement: currently strauss isn't required by default so will say sonification unavailable until it is installed manually - is there a reason not to add?

This pull request is to solve client-side sonification, making sonification via platforms viable

Fixes #

Change log entry

  • Is a change log needed? If yes, is it added to CHANGES.rst? If you want to avoid merge conflicts,
    list the proposed change log here for review and add to CHANGES.rst before merge. If no, maintainer
    should add a no-changelog-entry-needed label.

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

  • Are two approvals required? Branch protection rule does not check for the second approval. If a second approval is not necessary, please apply the trivial label.
  • Do the proposed changes actually accomplish desired goals? Also manually run the affected example notebooks, if necessary.
  • Do the proposed changes follow the STScI Style Guides?
  • Are tests added/updated as required? If so, do they follow the STScI Style Guides?
  • Are docs added/updated as required? If so, do they follow the STScI Style Guides?
  • If new remote data is added that uses MAST, is the URI added to the cache-download.yml workflow?
  • Did the CI pass? If not, are the failures related?
  • Is a milestone set? Set this to bugfix milestone if this is a bug fix and needs to be released ASAP; otherwise, set this to the next major release milestone. Bugfix milestone also needs an accompanying backport label.
  • After merge, any internal documentations need updating (e.g., JIRA, Innerspace)?

James Trayford added 2 commits October 17, 2025 09:16
…ented in an older branch between commits ec7b7a9 and 3502fbd (see PR javerbukh#33) to avoid hairy rebase
@github-actions github-actions bot added cubeviz plugin Label for plugins common to multiple configurations labels Oct 17, 2025
@kecnry kecnry added this to the 4.5 milestone Oct 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cubeviz plugin Label for plugins common to multiple configurations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants