Investigate if a neural network (AI) can be used for Edrumulus

[corrados]

It should be investigated if neural networks (AI) can be used for peak detection, positional sensing, etc. In the official documentation of, e.g., the ESP32-S3 they claim to support AI algorithms. There are multiple recordings of pads in https://github.com/corrados/edrumulus/tree/main/algorithm/signals which can be used to learn the network.

Challanges:

• Make the network as small as possible so that it can be run on, e.g., an ESP32-S3.
• What about latency of the algorithm? Can the AI lern to keep the delay below a certain threshold?
• Do we have sufficient recordings or are there much more needed to lern the neural network?

[giff25]

Gotcha, I can start looking into it.

[corrados]

Great :-). I think for a first start you should use the existing peak detection algorithm for usage to learn the AI network. If you run drumtrigger.m in Octave, you will get:

I would start with a pure peak detection. The detected peaks of the existing algorithm are in all_peaks.

In signalsandsettings.m, you can choose between different recordings by enabling the corresponding code line.

[giff25]

Thanks for the direction. I looked into some of the theory of this. It seems we would need 2 different kind of models for this project: A transformer model for processing the type of hit into the correct midi note (or set of midi notes), and a convolutional neural network for positional sensing.

It looks like there are a good amount of external libraries for esp32 as well as espressif having their own DL library. My thoughts are to run a sort of benchmark first by implementing the transformer model with LiteRT to see what the current performance would be like (I wouldn't want to waste a bunch of time training and AI model only to have it take 10 minutes to process 1 hit).

Concerns:
From what I have read most devices that don't have hardware accelerators (NPU's) run realllly slowly. So we may have to consider looking into a chip like the STM32N6 for real-time applicability.

Also, I really think looking into switching to ST chips might be worth it. They have an insanely well documented ecosystem of tools and code for all of their chips; however the dev boards are roughly 2 times more expensive than the ESP's. Just another consideration.

Here are some of the resources I used:
AI usage in embedded
LiteRT
Espressif DL lib
If you are able to access IEEE articles this was a good read
Transformer Model explained
CNN Model Explained
NPU's explained

Let me know if you have any questions, and thoughts. :)

Edit: I misunderstood what TinyML was, and added info about ST.

[corrados]

My thoughts are to run a sort of benchmark first by implementing the transformer model with LiteRT to see what the current performance would be like

That's a good idea. As stated in the title, we should first "investigate" the possibilities.

Also, I really think looking into switching to ST chips might be worth it.

There is no need for a "switch". Edrumulus shall support multiple platforms. Right now it is ESP32 and Teensy. ST would just be another option. Talking about AI, yes, there might be hardware which cannot be used with AI but on other it may work. But at least the "normal" algorithms shall be supported on all platforms.

[giff25]

Also, I really think looking into switching to ST chips might be worth it.

There is no need for a "switch". Edrumulus shall support multiple platforms. Right now it is ESP32 and Teensy. ST would just be another option. Talking about AI, yes, there might be hardware which cannot be used with AI but on other it may work. But at least the "normal" algorithms shall be supported on all platforms.

Ok, that makes sense. I have an STM32 board on hand. I could potentially just start developing the AI stuff on it and see where it goes.

[corrados]

I wouldn't want to waste a bunch of time training and AI model only to have it take 10 minutes to process 1 hit

I was thinking about this recently. I feel like AI will definitely be the future for e-drum triggering. The peak detection (which is basically a waveform pattern recognition) is perfectly suited for AI processing. I am pretty sure in 5 to 10 years all known commercial vendors will use AI for this. I assume vendors like Roland are already working on this.

So, even if it does not work on current micro-processors like ESP32 or STM32, it will run on this type of hardware in the the future. So, having an open source approach to this will be great even if it only runs on, e.g., PC hardware for now.

[giff25]

That makes sense, Ill move forward with Tensorflow then, it seems to be the standard library supported by most workflows for TinyML.

Also, do any of the signal files have notable retriggering? I want to see if there is any noticeable difference between them and actual peaks. I also think this is where a NN could potentially shine compared to the current algorithm.

[corrados]

Also, do any of the signal files have notable retriggering?

Retriggering usually occurs for the kick drum trigger. If you want to find signal with double-trigger potential, you can take a look at the kick trigger pad signals like KD7 or MPS750Kick.

[giff25]

Were you thinking of using the NN to replace the filtering process? I just wanted to clarify this to make sure I'm moving in the right direction. If so, where in the octave code are you getting the wave frequency range? I see you did it somehow within the docs but I cant figure out where it is in the code.

Also I want your input on how to move forward with classification. I think it would be worth it to start with two separate classes: 1.) ADC noise, and 2.) actual hit.
For this to happen we will need different training set containing vectored data of each type. Im not really sure how to accomplish this. we could perhaps feed the signal data that was stopped by the low pass filter into one training set and send the rest into another.

A very big issue I will be looking into is, this is very hardware specific (e.g. using the esp32 adc with the specific analog front end circuit). Unless we train the model with a very large data set ranging across a wide variety of microcontrollers (which would require extensive frequency range analysis.) we will need to retrain and reconfigure the model every time a new piece of hardware is added.

Let me know if you have a different idea on how to train that kind of classification. Or, your thoughts in general.

[corrados]

I have no experience with AI (yet) but my wish would be that we do not do any pre-processing but provide the AI network directly with the ADC data. I thought the AI network would learn itself to only look at the interesting part and ignore the irrelevant part of the signal (implicitly do something similar to what a filter would do).

If so, where in the octave code are you getting the wave frequency range?

All waveform data is recorded with 8 kHz sampling rate. But if your question targets my band-pass-filter, I just looked at the signal spectrum and created an initial version of the filter. Then I fine-tuned the filter parameter by looking at the detection results. But as stated above, it would be great if we would need no pre-filtering at all for AI.

I think it would be worth it to start with two separate classes: 1.) ADC noise, and 2.) actual hit.

As I said, I have no experience and cannot answer this question. The goal is that we have an ADC signal and the AI network tells me where the hit is. How this can be achieved with AI ... that's where you come in :-)

Unless we train the model with a very large data set ranging across a wide variety of microcontrollers (which would require extensive frequency range analysis.) we will need to retrain and reconfigure the model every time a new piece of hardware is added.

That's a good point. But as far as I know, for training e.g. autonomous cars, they use partly artificial training data generated (even by AI) from the original recorded data set. A similar thing we could also do. We could record the pad signals with a very good ADC and then distort the signal, filter the signal and add noise afterwards in, e.g., Octave. This way we could simulate different ADC hardware and train the AI network accordingly.

BTW, my thoughts on AI go even further. Right now we train the AI network on the detection result of the conventional Edrumulus algorithm. But we do not know if the detected hit is correct. A "stretch goal" would be that we record the pad signal playing normal drums and at the same time record a video of the playing. Then feed both signals to the AI learning algorithm and this extracts the actual hit from the video signal and uses this for getting the correct hits. But I am pretty sure this is very very complicated and will not be feasible. But engineers should never stop dreaming ;-).

[giff25]

A similar thing we could also do. We could record the pad signals with a very good ADC and then distort the signal, filter the signal and add noise afterwards in, e.g., Octave. This way we could simulate different ADC hardware and train the AI network accordingly.

Interesting idea for the adc's ill revisit it once i get to this step.

BTW, my thoughts on AI go even further. Right now we train the AI network on the detection result of the conventional Edrumulus algorithm. But we do not know if the detected hit is correct. A "stretch goal" would be that we record the pad signal playing normal drums and at the same time record a video of the playing. Then feed both signals to the AI learning algorithm and this extracts the actual hit from the video signal and uses this for getting the correct hits. But I am pretty sure this is very very complicated and will not be feasible. But engineers should never stop dreaming ;-).

I like the idea of testing the validity of its output, I think we can make it a bit easier by just comparing the NN outputs with the current algorithm; however, I might be completely wrong as this is still just assumption and speculation so I will revisit this once I get closer to testing the results.

[edward-io]

Just chiming in, ESP32 would be too weak for transformers, and the latency would be too high. A small convolutional residual network could work.

[giff25]

Just chiming in, ESP32 would be too weak for transformers, and the latency would be too high. A small convolutional residual network could work.

Yep! I realized this earlier last week. Ill look into ResNets after looking into VAEs :)

[giff25]

UPDATE: I'm continuing this work under a professor so I can also get credit in a college course. I created a little dev blog for this if you would like to keep tabs on progress it will all be here: https://giff25.github.io/

In summary: From the research I did over the summer it doesn't make sense to use any neural network for this purpose. They are only good at classifying the type of signal or its parts and this isnt a classification task; hence, It makes more sense to use a hotspot ML approach that would be used for pos sensing. This would extend the repo code with device support (Stm32f446re) and offer a different approach to positional sensing (assuming I correctly understood how the current one works).