Replies: 12 comments 8 replies
-
What I found, birdnet model contains some pesky layer(s?), that is reason why it isn't convertable into ONNX and doesn't work in tfjs+wasm: But what about tfjs+webgl/webgpu? Seems like all needed layers supported, yet it fails with errors. |
Beta Was this translation helpful? Give feedback.
-
Found that Chirpity converted model into tfjs too, and implemented custom layers |
Beta Was this translation helpful? Give feedback.
-
You may find this an easier starting point. It's a simple example of running BirdNET in the browser (it's set up using flask, but you don't need it): It was deleted from the repo about a month ago. I don't know why. |
Beta Was this translation helpful? Give feedback.
-
Reimplemented Will also throw away all stft imaginary output that is not used by birdnet. https://georg95.github.io/birdnet-web/test-chirpity.html?backend=webgl&profile=off&fast_fft=on |
Beta Was this translation helpful? Give feedback.
-
Removed unused imaginary output calculation |
Beta Was this translation helpful? Give feedback.
-
Tensorflow.js FFT implementation turned out to be naive N², and needs to be reimplemented to achieve industry standard computational complexity of N*log2(N). Measured time spent inferencing all besides fft on my device. |
Beta Was this translation helpful? Give feedback.
-
Finally after some research implemented fast FFT webgl algorithm. Now 2 FFT calls take 100-150ms on my device instead of 2000ms. Other operations take ~650ms. https://georg95.github.io/birdnet-web/test-chirpity.html?backend=webgl&profile=off&fast_fft=on |
Beta Was this translation helpful? Give feedback.
-
WebGPU is faster: 4.2s -> 0.5s, but FFT takes same time as in WebGL. Didn't use any specific optimization, don't know how. https://georg95.github.io/birdnet-web/test-chirpity.html?backend=webgpu&profile=off&fast_fft=on |
Beta Was this translation helpful? Give feedback.
-
Optimizing it all again, now performance is ok, 200ms per 3s chunk is factor of 15. https://georg95.github.io/birdnet-web/test-chirpity.html?backend=webgl&profile=off&fast_fft=on |
Beta Was this translation helpful? Give feedback.
-
Thanks for all the detective work. Is that something we should/could publish or share officially? I was thinking along the lines of "birdnet.js" maybe... |
Beta Was this translation helpful? Give feedback.
-
Built a little demo, listing all birds detected in audio file |
Beta Was this translation helpful? Give feedback.
-
Great work @georg95 ! I've incorporated some of your optimisations into Chirpity, and also adapted your frame kernel and the BirdNET spectrogram layer for batched inputs to the STFT function, which further improves the gains you've made for the Chirpity application. For example, BirdNET inference used to run at c. 150x real time speeds on my M2 MacBook Pro, now it runs at 500x speed. I've had reports that the M1 Pro (with twice the RAM) will do 800x real time inference speed! With a PC equipped with an RTX 3090 (and WAV files on NVME) inference is close to an hour per second! |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Trying to run birdnet inference in browser, idea is to implement birdnet-pi for smartphones
Successfully converted model into tfjs format, and it almost works, but:
CPU inference runs too slow (30s for 3s audio chunk):
https://georg95.github.io/birdnet-web/test.html?backend=cpu&debug_mode=true
2 calls of FFT operations take most of the time, but other operations work slow too (can see in console)
In tensorflow/cpu on same laptop it runs 500-1000 times faster, when warmed up.
WASM errors about Complex numbers, as per issue tensorflow/tfjs#5440:
https://georg95.github.io/birdnet-web/test.html?backend=wasm&debug_mode=true
WebGL gives strange error
GatherV2: the index value 2 is not in [0, 1]
https://georg95.github.io/birdnet-web/test.html?backend=webgl&debug_mode=true
Without debug mode WebGL error is different and I also don't get it:
Size(262143) must match the product of shape 262143,0
https://georg95.github.io/birdnet-web/test.html?backend=webgl&debug_mode=false
WebGPU also:
Cannot read properties of undefined (reading 'length') at e.profileKernel
https://georg95.github.io/birdnet-web/test.html?backend=webgpu&debug_mode=true
Size(262143) must match the product of shape 262143,0
https://georg95.github.io/birdnet-web/test.html?backend=webgpu&debug_mode=false
What I also tried is to convert model into ONNX format (and run with onnx-web), which also doesn't work.
python3 -m tf2onnx.convert --saved-model ./V2.4/audio-model --output model.onnx
Something about not implemented complex numbers
Current implementation of RFFT or FFT only allows ComplexAbs as consumer not {'Cast'}
Checked for OpenVino, even if it doesn't have web inference
ovc ./V2.4/audio-model
Again, complex numbers not supported somewhere inside
[TensorFlow Frontend] internal error: translator for Cast does not support input complex type
What should make it happen?
Have no experience in ML, any suggestions welcome.
Beta Was this translation helpful? Give feedback.
All reactions