6x Running

6x Running

Audiveris 6.0 prototype demoes a first integration of the page classifier within the pipeline of OMR engine. It also demoes the patch classifier, but this classifier is not yet integrated in the OMR engine, it is used only for interactive display purpose.

Audiveris prototype is available in the dws branch of audiveris repository on github. To build and run the prototype, you can thus use commands similar to the following ones which install the code into a brand new folder, named for example "my-proto", and build the prototype:

  git clone https://github.com/Audiveris/audiveris.git my-proto
  cd my-proto
  git checkout dws
  git pull --all
  ./gradlew clean build

The prerequisites are the same as for 5.1 release, see the related "Building from sources" section in Audiveris handbook.
Mind the fact that you will be able to build the prototype only on a 64-bit architecture. This is due to new dependencies: deeplearning4j relies on nd4j (n-dimensional arrays) whose native binaries are available only in 64-bit (whether you run under Windows, Linux or MacOS).

Page classifier

This classifier is coded in python language, runs in a separate process and provides a local web service (we call it "detection service") through which a client like Audiveris process can interact.

The code for the classifier web service is available on github Detection Service repository. First clone it to a directory of your choice:

git clone https://github.com/tuggeluk/Detection_Service.git

This classifier needs Python 3, (I personally use python 3.5), including the modules: numpy, tensorflow, pandas, pillow and flask. To install those, execute:

pip install numpy tensorflow pandas pillow flask

Then, you need to download the page trained model which is available separately here. It is an archive named trained_models_18_may.zip, ignore the warning message about the file being too large to be previewed, and just download this 1 GB file...
Then expand the archive directly into the detection service folder, resulting in the following hierarchy:

Detection_Service
├── classifier
├── class_utils
├── demo
├── models
├── main.py
├── ...
└── trained_models_18_may
    ├── RefineNet-Res101.data-00000-of-00001
    ├── RefineNet-Res101.index
    └── RefineNet-Res101.meta

Everything is now in place, you can start the detection service with:

python main.py

Some time later, you should get this message, which tells you that the detection service is now waiting for requests on port 5000:

* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

Patch classifier

The runtime code for this classifier is already integrated in the Java code of Audiveris 6.0 prototype.

You need to download the patch classifier trained model data. The model is available as one file named patch_class.h5 of about 90 MB. Rename it as patch-classifier.h5 and store it into Audiveris /res folder.

Running Audiveris 6.0 prototype

Launch Audiveris as usual, for example by:

  ./gradlew run

You should get messages similar to these:

Allocating instances for ShapeClassifier...
No model found at C:\Users\herve\AppData\Roaming\AudiverisLtd\audiveris\config\train\patch-classifier.h5. Searching in resource folder...
Audiveris version 6.0.0

LogUtil. Property logback.configurationFile not defined, skipped.
LogUtil. No C:\Users\herve\AppData\Roaming\AudiverisLtd\audiveris\config\logback.xml, skipped.
LogUtil. Configuration found file:///D:/soft/audiveris-github/new-dws/res/logback.xml
LogUtil. Logging to C:\Users\herve\AppData\Roaming\AudiverisLtd\audiveris\log\20190104T182551.log

Loaded plugins from C:\Users\herve\AppData\Roaming\AudiverisLtd\audiveris\config\plugins.xml
Loaded [CpuBackend] backend
Number of threads used for NativeOps: 4
Backend used: [CPU]; OS: [Windows 10]
Cores: [8]; Memory: [0.9GB];
Blas vendor: [OPENBLAS]
Layer "batchnormalization_1" momentum has been set but will not be applied unless the updater is set to NESTEROVS.
Layer "activation_1" regularization is set to true but l1, l2 or dropout has not been added to configuration.
Layer "batchnormalization_2" momentum has been set but will not be applied unless the updater is set to NESTEROVS.
Layer "activation_2" regularization is set to true but l1, l2 or dropout has not been added to configuration.
...
Layer "batchnormalization_24" momentum has been set but will not be applied unless the updater is set to NESTEROVS.
Layer "activation_24" regularization is set to true but l1, l2 or dropout has not been added to configuration.
Layer "dense_1_loss" regularization is set to true but l1, l2 or dropout has not been added to configuration.
Model cannot be trained: output layer dense_1_loss is not an IOutputLayer (no loss function specified)
Model cannot be trained: output dense_1_loss is not an IOutputLayer (no loss function specified)
ShapeClassifier instances allocated.

Don't get too impressed by the long list of warning messages. They stem from the Keras import of the patch classifier model. We can ignore them for the time being.

If you click on the Step menu, you will notice that the step sequence of the engine pipeline has been modified. There is a new step, named ANNOTATIONS between SCALE and GRID steps, and the order of the other steps is slightly different from 5.x.

Select an input file and launch its transcription (or just select ANNOTATIONS target step). You should observe something like:

[Dichterliebe01] 2 sheets in D:\soft\audiveris-github\release-5.1\data\examples\Dichterliebe01.pdf
[Dichterliebe01#1] Loaded image 1 2549x3299 from D:\soft\audiveris-github\release-5.1\data\examples\Dichterliebe01.pdf
[Dichterliebe01#1] Second combo peak too different (150,150,150), ignored
[Dichterliebe01#1] Beam minHeight:5 maxHeight:17 quorum:2522
[Dichterliebe01#1] Scale{line(2,3,4) interline(18,19,19) beam(10)}
[Dichterliebe01#1] Saved C:\Users\herve\AppData\Roaming\AudiverisLtd\audiveris\log\temp\Dichterliebe01#1.png
[Dichterliebe01#1] Posting image C:\Users\herve\AppData\Roaming\AudiverisLtd\audiveris\log\temp\Dichterliebe01#1.png
[Dichterliebe01#1] Waiting for response...

Then expect your machine resources (memory and CPU) to be fully occupied for a matter of minutes...

[Dichterliebe01#1] Duration= 180 seconds
[Dichterliebe01#1] Annotations: 275
[Dichterliebe01#1] End of sheet step ANNOTATIONS

At this point, a new tab ("Annotations") appears to display all the annotations retrieved by the page classifier.

Move at least to the GRID step and now the usual "Data" tab appears. On the right side, in the boards column, there is a new board named "Patch Classifier". Select any point in the image on Data tab and observe the display of this patch classifier.

Finally, you can complete the transcription. The annotations from the page classifier are taken into account to populate the final OMR data. Several steps have been modified in this prototype, especially the former SYMBOLS step, and many consistency checks had to be modified. This task is not complete yet.

Note you can save the project into its .omr file. The annotations data are saved into the project, hence a reload of the project will avoid the long ANNOTATIONS step. While in 5.x you could reset processing to BINARY step, now you can also reset to ANNOTATIONS step.