Live monitoring

[mishushakov]

it would be nice to be able to run a script that will monitor audio and play back the output of neural net
here are some libs, that could help us accomplish this

• https://github.com/spatialaudio/python-sounddevice
• http://people.csail.mit.edu/hubert/pyaudio/
• https://github.com/bastibe/SoundCard

[mishushakov]

using BlackHole i was able to reroute sound from my computer to virtual audio interface

since both input and output was rerouted, monitoring had to be done through headphones

this could be very useful for testing the proposed feature

[mishushakov]

Multi-Output device feature makes it possible to record & monitor at the same time

[GuitarML]

That all sounds very cool, forgive me for being slow but what will this add from a user standpoint in addition to what a typical DAW provides? Is this for testing the trained model or for recording the initial samples?

[mishushakov]

the messages above were testing-instructions
this feature would let you try out models live

evaluating models would be as simple as:

python monitor.py --list to list audio devices
python monitor.py --model=ts9 --input=1 --output=1 to record input 1 and playback through output 1

(or something like that)

why would we do this?
to run the plugin you need a DAW and the plugin = complexity
this will run out of the box = simplicity

why should we do this?
this would allow people to plug-and-play without having to install additional software

[mishushakov]

yeah recording samples is also cool idea
would solve all the conversion shenanigans

👍

[mishushakov]

there might be as-well is another way to accomplish the goal
you can run Onnx models in Browser

With ONNX.js, web developers can score pre-trained ONNX models directly on browsers with various benefits of reducing server-client communication and protecting user privacy, as well as offering install-free and cross-platform in-browser ML experience

this is like exactly what i was thinking about

lightning has support for onnx built-in
https://pytorch-lightning.readthedocs.io/en/latest/production_inference.html

if we could figure out how to make it work we would be able to offer online experience for evaluating models without any runtime at all

this would be life changing

[GuitarML]

That would be pretty amazing if we could do it all in a browser. I briefly looked into onnx when I was working on converting the PyTorch models into tensorflow, and it seems like a good framework.

Lots of good ideas to try out here, great work!

[mishushakov]

i can do the web part but need help saving models as onnx