Skip to content

Commit 427011f

Browse files
Hoog-VHoog-V
committed
add documentation and implement remove_wakeword function
1 parent 3cd6731 commit 427011f

22 files changed

+497
-2
lines changed

‎docs/about/implementation.md

+70
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# Library implementation details
2+
3+
Here some brief details as how the library is implemented;
4+
5+
## Architecture
6+
7+
The whole runtime consists of three stages:
8+
9+
1. Melspectrogram; A pre-processing model that computes a melspectrogram of the input audio data. This library uses an ONNX implementation of Torch's melspectrogram function with fixed parameters which enables efficient performance across devices. This onnx model is provided by [dscripka](https://github.com/dscripka/openWakeWord).
10+
11+
2. Embedding/feature generation; A shared feature extraction backbone model that converts melspectrogram inputs into general-purpose speech audio embeddings. This model is provided by Google as a TFHub module under an Apache-2.0 license. For openWakeWord, this model was manually re-implemented to separate out different functionality and allow for more control of architecture modifications compared to a TFHub module. The model itself is series of relatively simple convolutional blocks, and gains its strong performance from extensive pre-training on large amounts of data. This model is the core component of openWakeWord, and enables the strong performance that is seen even when training on fully-synthetic data.
12+
13+
3. Feature Classification: A classification model that follows the shared (and frozen) feature extraction model. The structure of this classification model is arbitrary, but in practice a simple fully-connected network or 2 layer RNN works well.
14+
15+
16+
## Code implementation
17+
18+
The general architecture of openwakeword is also reflected in to the code. Each class is one of the stages, and runs one model:
19+
20+
- Class: Melspectrogram -> Runs melspectrogram stage
21+
22+
- Class: Embedding -> Runs embedding stage
23+
24+
- Class: WakeWord -> Runs classifier stage
25+
26+
Yes, it is that simple! the **Lowwi** class itself contains some glue logic that controls the pipeline and calls the right callback functions when classifier model detects wakeword.
27+
28+
For example see the Lowwi->Run function:
29+
30+
```cpp
31+
void Lowwi::run(const std::vector<float> &audio_samples)
32+
{
33+
if(audio_samples.empty()) {
34+
return; /* No samples */
35+
}
36+
37+
_mel_samples = _mel->convert(std::ref(audio_samples));
38+
_feature_samples = _emb->convert(std::ref(_mel_samples));
39+
/* Loop through the classifier models */
40+
for (auto &ww : _wakewords) {
41+
wakeword_result res = ww.ww_inst->detect(_feature_samples);
42+
/* Check if wakeword is triggered */
43+
if(res.detected) {
44+
/*
45+
* Wakeword triggered!
46+
* Construct context variable
47+
* with the triggered wakeword phrase & confidence factor
48+
*/
49+
Lowwi_ctx_t cb = {ww.properties.phrase, res.confidence};
50+
/* Run callback function */
51+
ww.properties.cbfunc(cb, ww.properties.cb_arg);
52+
}
53+
}
54+
/* Clear all processed Melspectrogram & feature samples */
55+
_mel_samples.resize(0);
56+
_feature_samples.resize(0);
57+
}
58+
```
59+
60+
It just runs the steps followed by looping through all registered classifier models. When a wakeword is detected (`res.detected`) it runs the callback function(`ww.properties.cbfunc()`).
61+
62+
63+
## Design decisions
64+
65+
Design decisions that were made (some might change!):
66+
67+
- Everything was designed single threaded (as the biggest c++ alternative of this library, used so much threading it slowed down the code drastically).
68+
Although for small amount of wakewords < 10 it is faster, over > 10 it might be slower or as fast.
69+
70+
- ONNX is used instead of liteRT, even though both are supported by openwakeword project. ONNX has been proven to be faster for this model. The only thing it might hinder is using the Coral edge tpu which is liteRT only.

‎docs/assets/favicon.png

263 KB
Loading

‎docs/assets/logo.png

71.2 KB
Loading

‎docs/assets/stylesheets/logo.css

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
.md-header__button.md-logo {
2+
margin-top: 0;
3+
margin-bottom: 0;
4+
padding-top: 0;
5+
padding-bottom: 0;
6+
}
7+
8+
.md-header__button.md-logo img,
9+
.md-header__button.md-logo svg {
10+
height: 10%;
11+
width: 10%;
12+
}

‎docs/contributing/license.md

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# License
2+
3+
This work is licensed under the apache-2.0 license.
4+

‎docs/contributing/rules.md

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Contribution rules
2+
3+
For contributing we recommend creating a fork and when ready a pull request to merge the changes with this repository.
4+
In the pull request please state:
5+
6+
- What has been added/changed?
7+
- Some reasoning about implementation details
8+
9+
Please don't do refactors without consent of administrators <Ducroq & Hoog-V> as we will not merge them.
10+
11+
## New features
12+
13+
If you have any cool feature/idea to add to the code, please start an issue in GitHub to introduce the idea to the maintainers.
14+
15+
16+

‎docs/index.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Welcome
2+
3+
Hi Welcome to the Lowwi project! This project aims to provide users with a (easily) optimizable lightweight runtime for wakewords using the openwakeword architecture.
4+
5+
**Head over to Getting Started and kickstart your wakeword journey :)**

‎docs/usage/build_environment/img/build_variant.png

24.2 KB
Loading

‎docs/usage/build_environment/img/launch_target.png

23.8 KB
Loading

‎docs/usage/build_environment/img/linux_kit.png

29.8 KB
Loading

‎docs/usage/build_environment/img/vs_build_tools_options.png

116 KB
Loading

‎docs/usage/build_environment/img/vscode_cmake_generator.png

21.1 KB
Loading

‎docs/usage/build_environment/usage_with_linux.md

+64
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Set-up build environment on Linux
2+
3+
Linux is one of the easiest os'es to set-up as most packages and libraries can be found in the package repositories.
4+
5+
## Ubuntu and Debian
6+
7+
```bash
8+
sudo apt-get update && sudo apt-get install build-essential cmake ninja-build libsdl2-2.0-0 libsdl2-dev
9+
```
10+
11+
If you have not installed VSCode yet, do not install using APT in ubuntu as it will install the sandboxed snap version.
12+
13+
**Which has many issues due to the sandbox environment**
14+
15+
Use [this guide](https://code.visualstudio.com/docs/setup/linux) instead, which installs it using the APT repository from Microsoft themselves.
16+
17+
18+
## Arch
19+
20+
```bash
21+
sudo pacman -S sdl2 cmake gcc ninja
22+
```
23+
24+
If you have not installed VSCode yet,
25+
26+
Install the `visual-studio-code-bin` package from AUR.
27+
28+
29+
## Fedora
30+
31+
```bash
32+
sudo dnf install SDL2-devel gcc cmake ninja
33+
```
34+
35+
If you have not installed VSCode yet, use [this guide](https://code.visualstudio.com/docs/setup/linux).
36+
37+
38+
## Compiling and running the example
39+
The library contains an example demonstrating the usage and functionality of this library.
40+
41+
To compile and run this example:
42+
43+
1. Clone this repo:
44+
```
45+
git clone https://github.com/CLFML/lowwi.git
46+
```
47+
48+
2. Open the cloned repo folder in vscode; `File->Open Folder`
49+
50+
3. Select Ninja as build generator by pressing **CRTL+SHIFT+P**->**"CMake: Open CMake Tools Extension Settings"**->**"@ext:ms-vscode.cmake-tools generator"**
51+
Now type Ninja (with capital N into the generator field!).
52+
![CMake extension tool settings; Generator](img/vscode_cmake_generator.png)
53+
54+
4. Select the `GCC kit`by pressing CTRL+SHIFT+p and selecting `CMake: Select a kit`.
55+
![Select a kit, Linux gcc](img/linux_kit.png)
56+
57+
5. CMake will now configure; By default it will configure as Debug build, this has a significant performance hit.
58+
To change to release with debug info (which has optimizations turned on, but is still debuggable). Press CTRL+SHIFT+p again and enter `CMake: Select Variant`-> `RelWithDebInfo`
59+
![Variant](img/build_variant.png)
60+
61+
6. Let CMake Finish configuring your build configuration. **Then click on the Play button on the blue bar on the bottom of screen**, CMake might ask which target to launch, select the `LOWWI_demo_mic` target.
62+
![Launch target](img/launch_target.png)
63+
64+
7. After build is finished, it will launch the demo which uses your microphone to detect the **"Hey Mycroft"** wakeword.

‎docs/usage/build_environment/usage_with_mac.md

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Set-up build environment on Mac Os
2+
3+
For Mac os it is easier to use [Brew](https://brew.sh/) (community-based package manager).
4+
5+
With Brew installed it is relatively easy as it requires one command to install all the required packages and libraries.
6+
7+
## Installing Brew
8+
9+
Follow the instructions on the [webpage](https://brew.sh/).
10+
11+
## Installation of packages and libraries
12+
13+
The command for installing the packages and libraries:
14+
15+
```bash
16+
brew install cmake gcc cmake sdl2 ninja
17+
```
18+
19+
## Installing VSCode
20+
21+
Follow the instructions on the [VSCode website](https://code.visualstudio.com/docs/setup/mac).
22+
23+
## Compiling and running the example
24+
The library contains an example demonstrating the usage and functionality of this library.
25+
26+
To compile and run this example:
27+
28+
1. Clone this repo:
29+
```
30+
git clone https://github.com/CLFML/lowwi.git
31+
```
32+
33+
2. Open the cloned repo folder in vscode; `File->Open Folder`
34+
35+
3. Select Ninja as build generator by pressing **CRTL+SHIFT+P**->**"CMake: Open CMake Tools Extension Settings"**->**"@ext:ms-vscode.cmake-tools generator"**
36+
Now type Ninja (with capital N into the generator field!).
37+
![CMake extension tool settings; Generator](img/vscode_cmake_generator.png)
38+
39+
4. Select the `GCC kit`by pressing CTRL+SHIFT+p and selecting `CMake: Select a kit`.
40+
41+
5. CMake will now configure; By default it will configure as Debug build, this has a significant performance hit.
42+
To change to release with debug info (which has optimizations turned on, but is still debuggable). Press CTRL+SHIFT+p again and enter `CMake: Select Variant`-> `RelWithDebInfo`
43+
![Variant](img/build_variant.png)
44+
45+
6. Let CMake Finish configuring your build configuration. **Then click on the Play button on the blue bar on the bottom of screen**, CMake might ask which target to launch, select the `LOWWI_demo_mic` target.
46+
![Launch target](img/launch_target.png)
47+
48+
7. After build is finished, it will launch the demo which uses your microphone to detect the **"Hey Mycroft"** wakeword.

‎docs/usage/build_environment/usage_with_windows.md

+81
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Set-up build environment on Windows
2+
Before you can compile and run the demo you need to install the following tools and libraries:
3+
4+
- Any C/C++ compiler (MSVC is recommended, but GCC works as well)
5+
- CMake
6+
- Ninja (**Optional**, but recommended as generating for MSBuild is very slow!)
7+
- Any code editor (This guide will use VSCode, as it is free and easy to configure with CMake)
8+
9+
!!! note
10+
11+
Although any recent enough C/C++ compiler can be used.
12+
13+
**This guide will only use the MSVC compiler!**
14+
15+
This choice was made as this compiler is also used for all other CLFML projects.
16+
17+
18+
## Installing MSVC16 (2019 edition) & CMake
19+
The MSVC compiler (and CMake) can be installed by either installing VS BuildTools or Visual Studio 2019.
20+
21+
**This guide will use the VS BuildTools method, as we don't need the Visual Studio IDE**.
22+
23+
There are multiple ways you can download and install VS BuildTools 2019;
24+
25+
- Manually downloading and installing using this [link](https://aka.ms/vs/16/release/vs_buildtools.exe)
26+
- Using chocolatey:
27+
```choco install visualstudio2019buildtools```
28+
- Using winget:
29+
```winget install --id=Microsoft.VisualStudio.2019.BuildTools -e```
30+
31+
### Manually installing VS Build Tools 2019
32+
1. [Download and run this installer](https://aka.ms/vs/16/release/vs_buildtools.exe).
33+
2. Select these options:
34+
![Visual Studio Build Tools Options](img/vs_build_tools_options.png)
35+
3. After installation, reboot your machine!
36+
37+
38+
## Installing Ninja
39+
1. [Download the latest Ninja release for Windows!](https://github.com/ninja-build/ninja/releases)
40+
2. Unzip this ninja-win.zip to `C:\ninja-win`
41+
3. Open the environment variables editor using the Windows Startup Menu ([Try this guide if you can't find it](https://www.imatest.com/docs/editing-system-environment-variables/#Windows))
42+
4. Add the `C:\ninja-win` path to the PATH variable;
43+
5. Open a commandline window and check if Ninja is correctly installed by running the `ninja` command!
44+
45+
46+
## Installing VSCode (with plugins)
47+
VSCode is an easy to use code-editor with CMake support (using the CMake Tools plugin).
48+
49+
To set-up VSCode the follow these steps:
50+
51+
1. [Download and install VSCode using the installer](https://code.visualstudio.com/download)
52+
2. Follow the initial set-up wizard in vscode (if freshly installed)
53+
3. Download and install this plugin pack:
54+
- C/C++ Extension Pack (Microsoft)
55+
56+
## Compiling and running the example
57+
The library contains an example demonstrating the usage and functionality of this library.
58+
59+
To compile and run this example:
60+
61+
1. Clone this repo:
62+
```
63+
git clone https://github.com/CLFML/lowwi.git
64+
```
65+
66+
2. Open the cloned repo folder in vscode; `File->Open Folder`
67+
68+
3. Select Ninja as build generator by pressing **CRTL+SHIFT+P**->**"CMake: Open CMake Tools Extension Settings"**->**"@ext:ms-vscode.cmake-tools generator"**
69+
Now type Ninja (with capital N into the generator field!).
70+
![CMake extension tool settings; Generator](img/vscode_cmake_generator.png)
71+
72+
4. Select the `MSVC amd64 kit`by pressing CTRL+SHIFT+p and selecting `CMake: Select a kit`.
73+
74+
5. CMake will now configure; By default it will configure as Debug build, this has a significant performance hit.
75+
To change to release with debug info (which has optimizations turned on, but is still debuggable). Press CTRL+SHIFT+p again and enter `CMake: Select Variant`-> `RelWithDebInfo`
76+
![Variant](img/build_variant.png)
77+
78+
6. Let CMake Finish configuring your build configuration. **Then click on the Play button on the blue bar on the bottom of screen**, CMake might ask which target to launch, select the `Lowwi_demo_mic` target.
79+
![Launch target](img/launch_target.png)
80+
81+
7. After build is finished, it will launch the demo which uses your microphone to detect the **"Hey Mycroft"** wakeword.

‎docs/usage/lowwi_demo.md

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# LOWWI demo fragment
2+
The Lowwi fragment Demo that comes with this library demonstrates the wakeword api of this library without a microphone. It uses a prerecorded .wav file with the "Hey mycroft" wakeword. To trigger the "Hey mycroft" wakeword.
3+
4+
## Building and compiling
5+
To build and run this example you need all the prerequisites you would normally need to build the library. As these steps differ a lot between platforms/oses, a guide for the three most used platforms was made:
6+
7+
- [Windows Guide](build_environment/usage_with_windows.md)
8+
- [Mac Os Guide](build_environment/usage_with_mac.md)
9+
- [Linux Guide](build_environment/usage_with_linux.md)

‎docs/usage/lowwi_demo_mic.md

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# LOWWI demo microphone
2+
The Lowwi microphone Demo that comes with this library demonstrates the wakeword api of this library by letting you test it with your microphone. You can now trigger the "Hey mycroft" wakeword, by screaming it at your screen! Watch out for any house mates, you might get strange looks.
3+
4+
## Building and compiling
5+
To build and run this example you need all the prerequisites you would normally need to build the library. As these steps differ a lot between platforms/oses, a guide for the three most used platforms was made:
6+
7+
- [Windows Guide](build_environment/usage_with_windows.md)
8+
- [Mac Os Guide](build_environment/usage_with_mac.md)
9+
- [Linux Guide](build_environment/usage_with_linux.md)

0 commit comments

Comments
 (0)