|
1 | 1 | # Broadmark |
2 | 2 |
|
3 | | - |
| 3 | + |
4 | 4 |
|
5 | 5 | Our framework consists of a Unity-based scenario generation tool and a C++ environment to execute algorithms upon generated scenes. |
6 | 6 |
|
@@ -33,45 +33,45 @@ Requirements: |
33 | 33 | - Unity 2019.2 (optional, other versions might work) |
34 | 34 | - MVC++ Compiler (e.g., Visual Studio Community) |
35 | 35 |
|
36 | | -The system consists of 3 modules: (1) a Unity-based simulation generator tool, used to bake rigid body sims to disk; (2) the broadmark C++ tool, which houses the algorithms and allows them to be benchmarked on a baked scene; and (3) a python 3 module to run tests and parse the results. Out-of-the-box, we provide pre-built binaries and some baked scenes to help getting the system up-and-running for the first time. |
| 36 | +The system consists of 3 modules: (1) a Unity-based simulation generator tool, used to bake rigid body sims to disk; (2) the broadmark C++ tool, which houses the algorithms and allows them to be benchmarked on a baked scene; and (3) a python module to run tests and parse the results. Out-of-the-box, we provide pre-built binaries and some baked scenes to help getting the system up-and-running for the first time. |
37 | 37 |
|
38 | | -First, download this repository (`git clone https://github.com/ppgia-unifor/Broadmark.git`) and install the lastest Python 3. To get all python dependencies, run `pip install numpy pandas, openpyxl matplotlib seaborn`. At the [releases page](https://github.com/ppgia-unifor/Broadmark/releases), download `SimulationGenerator_Build.exe` file (if you wish to generate the simulations yourself) or download the `FreeFall Cubes N1000 F1000frames.aabbs` pre-generated scene and skip the following paragraph. Then, download the `Broadmark_x64.exe`, which is the pre-built binary of the Broadmark system. In the following, no strict directory structure is needed to run the system, however, it is advisable to put everything on a common folder and to store simulations on a drive with plenty of space. Also, do not change generated scene names, as some portions of the code expect it to be tokenizable. |
| 38 | +First, download this repository (`git clone https://github.com/ppgia-unifor/Broadmark.git`) and install the lastest Python 3. To get all python dependencies, run `pip install numpy pandas, openpyxl matplotlib seaborn`. The repository itself already comes with the Simulation Generator Tool packed within a .zip file inside the `SimulationGenerator_Build` folder. For instructions on how to generate this tool yourself, consider reading the README.md file located at the `SimulationGenerator` folder. Alternatively, we package a sample scene within the `Simulations/Recordings_AABBs/` folder. Finally, pre-built binaries of the Broadmark tool can be found at the [Releases](https://github.com/ppgia-unifor/Broadmark/releases) page. These should be placed at `Broadmark/bin64/`. |
39 | 39 |
|
40 | | -To generate scenes yourself, extract the Simulation Generation tool and run it. You should see the following screen, which allows you to customize which scenes you wish to run, for how many objects and frames, and several other options. For now, keep all default values as is, but mark "Record Simulations" and set the "Output Path" to the folder you wish to save scenes. Now, click Generate Simulations and watch/wait until finished. Finally, check the output folder for the `.aabb` files. |
| 40 | +To generate scenes using the Simulation Generation Tool, extract the .zip file located at the `SimulationGenerator_Build` and run the `Simulation Generator.exe` file. You should see the following screen, which allows you to customize which scenes you wish to run, for how many objects and frames, and several other options. For now, keep all default values as is, but mark "Record Simulations". The default "Output Path" is set to `../Simulations/` and should be left as is. If you move the simulation generation tool to another folder, update this value accordingly. Finally, click Generate Simulations and watch/wait until finished. Finally, check the output folder for the `.aabb` files. |
41 | 41 |
|
42 | 42 | Simulation Generator Wizard and its Settings. |
43 | | - |
| 43 | + |
44 | 44 | Simulation Generator at work producing a Brownian scene. |
45 | | - |
| 45 | + |
46 | 46 | Generated .aabb files. |
47 | | - |
| 47 | + |
48 | 48 |
|
49 | 49 | Having the simulation files ready, we can start running some algorithms. There are two ways of running the Broadmark system, one is the manual way and the other is based on using the Python module to handle the low level work. The Broadmark tool takes as input a .json file which contains the algorithm it should run (and any parameters), the path to the baked scene file and the path where it should put the output json file. The manual execution consists of writing the .json file by hand (or writing a custom tool). Here is a sample .json file to be used with the Broadmark.exe: |
50 | 50 |
|
51 | 51 | ``` |
52 | 52 | { |
53 | 53 | "m_algorithm": "KD", |
54 | 54 | "m_algorithm_prettyName": "KD", |
55 | | - "m_inputScene": "E:/Simulations/Recordings_AABBs/Brownian Cubes N1000 F1000frames.aabbs", |
56 | | - "m_outputLog": "D:/Bitbucket/broadmark/Scripts/Tests/Tests 2020-01-17_13-28-56/Results/Brownian Cubes N1000 F1000frames.aabbs_KD.json_results.json", |
| 55 | + "m_inputScene": "D:/Bitbucket/broadmark/Simulations/Recordings_AABBs/FreeFall Cubes N1000 F1000frames.aabbs", |
| 56 | + "m_outputLog": "D:/Bitbucket/broadmark/Scripts/Tests/My First Broadmark Benchmark/Results/FreeFall Cubes N1000 F1000frames.aabbs_KD.json_results.json", |
57 | 57 | "NoAdditionalProperty": "" |
58 | 58 | } |
59 | 59 | ``` |
60 | 60 |
|
61 | | -To run the benchmark, replace the path strings to fit your directory structure and run `Broadmark.exe config.json`. |
| 61 | +To run the benchmark, replace the path strings to fit your directory structure, save it to a file named "example.json" in the same folder as the broadmark executable and run `Broadmark.exe example.json`. |
62 | 62 |
|
63 | | -To run multiple algorithms and scenes, a `.json` file per algorithm-simulation pair is required. This ensures that every test is a separate execution context, sandboxing each run from potential memory leaks and/or crashes from other executions. To alleviate the amount of work needed, the Python module comes into play as a middleware to manage the .json generation. To use it, inspect the file `Scripts/run.py`. In it, fill-in the variables `broadmark_bin`, `scenes_folder` and `output_folder` with your directory structure and the `algorithms` variable with a list of algorithms you wish to run. Now, if you run `python run.py`, the batch execution will start, testing all supplied algorithms against all `.aabb` files in the `scenes_folder` path. At the end, some `.csv` files are generated, aggregating the outputs of every test. |
| 63 | +To run multiple algorithms and scenes, a `.json` file per algorithm-simulation pair is required. This ensures that every test is a separate execution context, sandboxing each run from potential memory leaks and/or crashes from other executions. To alleviate the amount of work needed, the Python module comes into play as a middleware to manage the .json generation. To use it, inspect the file `Scripts/run.py`. In it, the variables `broadmark_bin`, `scenes_folder` and `output_folder` are pre-set to "../Broadmark/bin64/Broadmark_Release_x64.exe", "../Simulations/Recordings_AABBs/" and "./Tests/". While you are free to change these as needed, these default values ensure that everything will run out-of-the-box. Using the `algorithms` variable, you can set which algorithms are going to be run. Now, navigate to the scripts folder and run `python run.py`, the batch execution will start, testing all supplied algorithms against all `.aabb` files in the `scenes_folder` path. After each successful test, `.csv` files are generated with statistics from all ran tests. |
64 | 64 |
|
65 | 65 | Sample of the Broadmark system performing several benchmark tests. |
66 | | - |
| 66 | + |
67 | 67 | Outputted log files and .csv summaries. |
68 | | - |
| 68 | + |
69 | 69 |
|
70 | 70 | To generate plots of the obtained results, run `python plot.py`. This script will search for the lastest results (at the "Scripts/Tests/" folder) and present some useful plots of the generated data. To perform custom ploting, you can use the generated .csv files or parse the output .json files yourself, for maximum control. |
71 | 71 |
|
72 | 72 | At this point, you should be able to generate scenes using the Simulation Generator tool and run benchmarks for the supplied algorithms on the generated scenes. |
73 | 73 |
|
74 | | -From now on, for a more advanced usage, if you wish to develop your own scenes, you need to install the Unity Game Engine and open the Simulation Generator project, which is pretty much unity-esque. With some Unity experience it should not be hard to use, understand and modify. The same goes for the Broadmark tool. The repository contains the necessary Visual Studio project files and they should compile out-of-the-box without any issues. Right now, we only support Windows x64 builds, however, it is fairly SO agnostic and should not be difficult to run on Linux or MAC under Clang or maybe GCC. The threading portion is completely implemented as either STL threads or OpenMP and all dependencies are either bundled or can be compiled to other OSes. For the CGAL algorithm to work, you will have to compile the CGAL library yourself and link to the application. The easiest way to do this is to use the [Vcpkg](https://github.com/Microsoft/vcpkg) tool, which has the CGAL package. For the OpenCL algorithms, you will need to download and install the OpenCL SDK of your graphics card vendor. Currently, we have only tested the application under NVidia Cards using the CUDA Toolkit 8, 9 and 10. We would be glad to hear if the system worked fine on AMD or Intel cards as well. Regarding the python module, its implementation is pretty straight forward and should also be easy to understand and modify. Since the whole system is based on the .json format, it should not be hard to write any custom tools as well. |
| 74 | +From now on, for a more advanced usage, if you wish to develop your own scenes, you need to install the Unity Game Engine and open the Simulation Generator project, which is pretty much unity-esque. With some Unity experience it should not be hard to use, understand and modify. The same goes for the Broadmark tool. The repository contains the necessary Visual Studio project files and they should compile out-of-the-box without any issues. Right now, we only support Windows x64 builds, however, it is fairly SO agnostic and should not be difficult to run on Linux or Mac under Clang or maybe GCC. The threading portion is completely implemented as either STL threads or OpenMP and all dependencies are either bundled or can be compiled to other OSes. For the CGAL algorithm to work, you will have to compile the CGAL library yourself and link to the application. The easiest way to do this is to use the [Vcpkg](https://github.com/Microsoft/vcpkg) tool, which has the CGAL package. For the OpenCL algorithms, you will need to download and install the OpenCL SDK of your graphics card vendor. Currently, we have only tested the application under NVidia Cards using the CUDA Toolkit 8, 9 and 10. We would be glad to hear if the system worked fine on AMD or Intel cards as well. Regarding the python module, its implementation is pretty straight forward and should also be easy to understand and modify with some Python knowledge. Since the whole system is based on the .json format, it is friendly to those who wish to write custom tools as well. |
75 | 75 |
|
76 | 76 | Justifying some of the design choices, we opted for a Simulation Generator tool decoupled from the execution to make the system more scalable to massive simulations and to enforce that every test is conducted over the exact same data. Since simulations are baked, they are generated only once, despite the number of tests that will be conducted on them, and they are fixed, so each algorithm will receive the exact same set of AABBs. Regarding the Broadmark system being capable of handling only one pair of Algorithm-Scene per execution, this has the desirable properties of sandboxing each benchmark run and of allowing the test management code to live outside of the error-prone C++ world. While this makes using the system a bit harder, most of the necessary setup is performed only once and should not change over time, such as path strings. For more information, please consider reading our work [Broadmark: A Testing Framework for Broad‐Phase Collision Detection Algorithms](https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.13884), which throughly elaborates on these and other design choices. |
77 | 77 |
|
|
0 commit comments