Improving the documentation

Author: nuzrub

.gitignore

@@ -2,3 +2,5 @@
 Scripts/__pycache__/
 
 Scripts/cache/
+
+Scripts/Tests/

README.md

@@ -1,36 +1,81 @@
 # Broadmark
 
-A framework for the development, testing and evaluation of broad phase collision detection algorithms. Our framework consists of a Unity-based scenario generation tool and a C++ environment to execute algorithms upon generated scenes.
+![Broadmark](/images/main-logo.png)
 
-Out-of-the-box, we have currently implemented the following:
+Our framework consists of a Unity-based scenario generation tool and a C++ environment to execute algorithms upon generated scenes. 
+
+Out-of-the-box, we currently implement the following:
 - Scenario Generation Engines:
-  - PhysX: Built-in Unity physics engine
-  - Bullet: Through the Bullet Unity project
-  - Viewer: Used to view previously generated scenes. Useful for movie recordings
-  - Splitter: Views n-objects simulation and outputs a 8n-objects simulation by splitting objects in 8 sub-objects
+  - PhysX: built-in Unity physics engine
+  - Bullet: through the Bullet Unity project
+  - Viewer: used to view previously generated scenes. Useful for movie recordings
+  - Splitter: views a n-objects simulation and outputs a 8n-objects simulation by splitting objects in 8 sub-objects
 
 - Built-in Scenes:
-  - Brownian: A dynamic and chaotic scene with randomly moving boxes. Each object is assigned with a new random speed every now and then. Useful as baseline benchmark and analysis of algorithms under a uniform object distribution.
-  - Free Fall: A coherent simulation of free falling boxes. Useful for testing temporal-optimized algorithms.
-  - Gravity: A coherent simulation of free falling boxes under a strong and moving gravity. This is a stress test scenario which generates many collisions and forces all objects to move significantly within the world. 
+  - Brownian: a dynamic and chaotic scene of randomly moving boxes. Each object is assigned a new random speed every now and then. Useful as baseline benchmark and analysis of algorithms under a uniform object distribution.
+  - Free Fall: a coherent simulation of free falling boxes. Useful for testing temporal-optimized algorithms.
+  - Gravity: a coherent simulation of free falling boxes under a strong and moving gravity. This is a stress test scenario which generates many collisions and forces all objects to move significantly within the world. 
 
 - Implemented Algorithms:
   - Original Implementations: Bruteforce, Sweep-and-Prune and Grid. Several variants of these algorithms are implemented, such as SIMD optimized, Open-MP parallel and SIMD + OpenMP. For the Grid algorithm, we present it using a Grid + BF and Grid + SAP settings as well as using a 2D or 3D grid. Best variants are BF SIMD Parallel, SAP SIMD Parallel and Grid 3D SAP Parallel.
-  - Bullet 2 Algorithms: We include the btAxisSweep3 and btDBVT algorithms. For convenience, they are incorported into the project and compiled with the remaining files for ease of integration and to reduce dependency troubles.
-  - Bullet 3 Algorithms: We include a wrapper to use the b3OpenCL branch algorithms. These contain also a 3D Grid, a LBVH and a SAP implementation on OpenCL. For convenience, they are incorporated into the project and can be compiled-out by altering the Config.h file. This is useful if you do not have a GPU compatible with OpenCL or the necessary SDKs installed for compilation.
-  - CGAL algorithms: We include the ["Intersecting Sequences of dD Iso oriented Boxes"](https://doc.cgal.org/latest/Box_intersection_d/index.html) algorithm. The necessary CGAL and Boost dependencies are not incorporated into this project due to licensing and size. This algorithm can be compiled out by altering the Config.h file. An easy way to set up all the dependencies is to use the [Vcpkg](https://github.com/Microsoft/vcpkg) tool installed and download the cgal:x64-windows package.
+  - Bullet 2 Algorithms: We include the btAxisSweep3 and btDBVT algorithms. For convinience, these are included in the project and compiled with the remaining files for ease of integration and to reduce dependency troubles.
+  - Bullet 3 Algorithms: We include a wrarper to use the b3OpenCL branch algorithms. These include a 3D Grid, a LBVH and a SAP implementation on OpenCL. For convinience, these are included in the project and can be compiled-out by altering the Config.h file. This is useful if you do not have a GPU compatible with OpenCL or does not have the necessary SDKs installed for compilation.
+  - CGAL algorithms: We include the ["Intersecting Sequences of dD Iso oriented Boxes"](https://doc.cgal.org/latest/Box_intersection_d/index.html) algorithm. The necessary CGAL and Boost dependencies are not included in this project due to licensing and size. This algorithm can be compiled out by altering the Config.h file. A easy way to have all the dependencies set is to use the [Vcpkg](https://github.com/Microsoft/vcpkg) tool installed and download the cgal:x64-windows package.
   - Tracy's Algorithm: Original implementation of the [Efficient Large-Scale Sweep and Prune Methods with AABB Insertion and Removal](https://dl.acm.org/citation.cfm?id=1549865) paper. This used to be available at the [author's website](http://www.danieljosephtracy.com/) but it is no longer on-line.
-  - KD-Tree Algorithm: Original implementation of the [Flexible Use of Temporal and Spatial Reasoning for Fast and Scalable CPU Broad‐Phase Collision Detection Using KD‐Trees](https://onlinelibrary.wiley.com/doi/full/10.1111/cgf.13529) algorithm. This is exactly the same implementation as used in the paper.
+  - KD-Tree Algorithm: Original implementation of the [Flexible Use of Temporal and Spatial Reasoning for Fast and Scalable CPU Broad‐Phase Collision Detection Using KD‐Trees](https://onlinelibrary.wiley.com/doi/full/10.1111/cgf.13529) algorithm. This is the exact same implementation as used in the paper.
 
 
 
 ## Runing the Code
 
-The Simulation Generator tool has been implemented using the Unity 2019.2 edition, but should probably work on any 2017+ version. For non-unity users, a pre-built version for Windows x64 is available at the 'releases' section of the GitHub repository.
+Requirements:
+- Python 3 (numpy, pandas, openpyxl, matplotlib, seaborn)
+- Unity 2019.2 (optional, other versions might work)
+- MVC++ Compiler (e.g., Visual Studio Community)
+
+The system consists of 3 modules: (1) a Unity-based simulation generator tool, used to bake rigid body sims to disk; (2) the broadmark C++ tool, which houses the algorithms and allows them to be benchmarked on a baked scene; and (3) a python 3 module to run tests and parse the results. Out-of-the-box, we provide pre-built binaries and some baked scenes to help getting the system up-and-running for the first time. 
+
+First, download this repository (`git clone https://github.com/ppgia-unifor/Broadmark.git`) and install the lastest Python 3. To get all python dependencies, run `pip install numpy pandas, openpyxl matplotlib seaborn`. At the [releases page](https://github.com/ppgia-unifor/Broadmark/releases), download `SimulationGenerator_Build.exe` file (if you wish to generate the simulations yourself) or download the `FreeFall Cubes N1000 F1000frames.aabbs` pre-generated scene and skip the following paragraph. Then, download the `Broadmark_x64.exe`, which is the pre-built binary of the Broadmark system. In the following, no strict directory structure is needed to run the system, however, it is advisable to put everything on a common folder and to store simulations on a drive with plenty of space. 
+
+To generate scenes yourself, extract the Simulation Generation tool and run it. You should see the following screen, which allows you to customize which scenes you wish to run, for how many objects and frames, and several other options. For now, keep all default values as is, but mark "Record Simulations" and set the "Output Path" to the folder you wish to save scenes. Now, click Generate Simulations and watch/wait until finished. Finally, check the output folder for the `.aabb` files.
+
+Simulation Generator Wizard and its Settings.
+![Simulation Generator Wizard](/images/simulation-generator-wizard.png)
+Simulation Generator at work producing a Brownian scene.
+![Simulation Generator in Action](/images/simulation-generator-brownian.png)
+Generated .aabb files.
+![Simulation Generator in Action](/images/simulation-generator-files.png)
+
+Having the simulation files ready, we can start running some algorithms. There are two ways of running the Broadmark system, one is the manual way and the other is based on using the Python module to handle the low level work. The Broadmark tool takes as input a .json file which contains the algorithm it should run (and any parameters), the path to the baked scene file and the path where it should put the output json file. The manual execution consists of writing the .json file by hand (or writing a custom tool). Here is a sample .json file to be used with the Broadmark.exe:
+
+```
+{
+    "m_algorithm": "KD",
+    "m_algorithm_prettyName": "KD",
+    "m_inputScene": "E:/Simulations/Recordings_AABBs/Brownian Cubes N1000 F1000frames.aabbs",
+    "m_outputLog": "D:/Bitbucket/broadmark/Scripts/Tests/Tests 2020-01-17_13-28-56/Results/Brownian Cubes N1000 F1000frames.aabbs_KD.json_results.json",
+    "NoAdditionalProperty": ""
+}
+```
+
+To run the benchmark, replace the path strings to fit your directory structure and run `Broadmark.exe config.json`.
+
+To run multiple algorithms and scenes, a `.json` file per algorithm-simulation pair is required. This ensures that every test is a separate execution context, sandboxing each run from potential memory leaks and/or crashes from other executions. To alleviate the amount of work needed, the Python module comes into play as a middleware to manage the .json generation. To use it, inspect the file `Scripts/run.py`. In it, fill-in the variables `broadmark_bin`, `scenes_folder` and `output_folder` with your directory structure and the `algorithms` variable with a list of algorithms you wish to run. Now, if you run `python run.py`, the batch execution will start, testing all supplied algorithms against all `.aabb` files in the `scenes_folder` path. At the end, some `.csv` files are generated, aggregating the outputs of every test. 
+
+Sample of the Broadmark system performing several benchmark tests.
+![Broadmark Execution](/images/broadmark-execution.png)
+Outputted log files and .csv summaries.
+![Generated Results](/images/broadmark-logs.png)
+
+To generate plots of the obtained results, run `python plot.py`. This script will search for the lastest results (at the "Scripts/Tests/" folder) and present some useful plots of the generated data. To perform custom ploting, you can use the generated .csv files or parse the output .json files yourself, for maximum control. 
+
+At this point, you should be able to generate scenes using the Simulation Generator tool and run benchmarks for the supplied algorithms on the generated scenes. 
+
+From now on, for a more advanced usage, if you wish to develop your own scenes, you need to install the Unity Game Engine and open the Simulation Generator project, which is pretty much unity-esque. With some Unity experience it should not be hard to use, understand and modify. The same goes for the Broadmark tool. The repository contains the necessary Visual Studio project files and they should compile out-of-the-box without any issues. Right now, we only support Windows x64 builds, however, it is fairly SO agnostic and should not be difficult to run on Linux or MAC under Clang or maybe GCC. The threading portion is completely implemented as either STL threads or OpenMP and all dependencies are either bundled or can be compiled to other OSes. For the CGAL algorithm to work, you will have to compile the CGAL library yourself and link to the application. The easiest way to do this is to use the [Vcpkg](https://github.com/Microsoft/vcpkg) tool, which has the CGAL package. For the OpenCL algorithms, you will need to download and install the OpenCL SDK of your graphics card vendor. Currently, we have only tested the application under NVidia Cards using the CUDA Toolkit 8, 9 and 10. We would be glad to hear if the system worked fine on AMD or Intel cards as well. Regarding the python module, its implementation is pretty straight forward and should also be easy to understand and modify. Since the whole system is based on the .json format, it should not be hard to write any custom tools as well. 
 
-The Broadmark tool comes with Visual Studio 2017 solution files but can be run from VS Code without much trouble. The code was originally written and tested only on Windows machines. However, it is fairly SO agnostic and should not be difficult to run on Linux or MAC under Clang or maybe GCC. The threading portion is completely implemented as either STL threads or OpenMP and all dependencies are either bundled or can be compiled to other OSes.
+Justifying some of the design choices, we opted for a Simulation Generator tool decoupled from the execution to make the system more scalable to massive simulations and to enforce that every test is conducted over the exact same data. Since simulations are baked, they are generated only once, despite the number of tests that will be conducted on them, and they are fixed, so each algorithm will receive the exact same set of AABBs. Regarding the Broadmark system being capable of handling only one pair of Algorithm-Scene per execution, this has the desirable properties of sandboxing each benchmark run and of allowing the test management code to live outside of the error-prone C++ world. While this makes using the system a bit harder, most of the necessary setup is performed only once and should not change over time, such as path strings. For more information, please consider reading our work [Broadmark: A Testing Framework for Broad‐Phase Collision Detection Algorithms](https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.13884), which throughly elaborates on these and other design choices. 
 
-Most of the system will work out-of-the-box, except for the CGAL and OpenCL algorithms. For the CGAL algorithm to work, you will have to compile the CGAL library yourself and link to the application. The easiest way to do this is to use the [Vcpkg](https://github.com/Microsoft/vcpkg) tool. For the OpenCL algorithms, you will need to download and install the OpenCL SDK of your graphics card vendor. Currently, we have only tested the application under Nvidia Cards using the CUDA Toolkit 8 and 9. I would be glad to hear if the system worked fine on AMD or Intel cards as well.
+For any questions, issues and enhancements, please contact us by e-mail or open a github issue or pull request. We will be glad to answer and help in any inquiries. 
 
 
 ## Citing our Work

Scripts/bm_plot.py

@@ -47,7 +47,6 @@ def plot_box(results_folder):
     ns = sorted(list(set(frame.index.get_level_values(1))))
     algorithms = list(set(frame.index.get_level_values(2)))
 
-    
     subframe = frame.transpose()[scenes[0]][ns[2]]
 
     sns.set(style="ticks", palette="pastel")

Scripts/plot.py

@@ -1,37 +1,21 @@
 
 
-from os import listdir
-from os.path import isfile, join, basename
+from os import listdir, getcwd
+from os.path import isfile, isdir, join, basename, getmtime
 from bm_plot import *
 from bm_parse import process_results
 from os.path import join, isfile
 
-base_path = "D:/Bitbucket/broadmark/"
-results_folder = base_path + "/Scripts/Tests/1_Artigo_Tool/3 Gravity/GPU + Parallel/"
 
-process_results(results_folder)
-#if not isfile(join(results_folder, "main_frame.pickle")):
-#    process_results(results_folder)
+# You can set the results folder to use directly or let the system find the most recent one
+results_folder = ''
+if results_folder == '':
+    root = getcwd() + '\\Tests\\'
+    all_subdirs = [root + d + '\\Results\\' for d in listdir(root)]
+    latest_subdir = max(all_subdirs, key=getmtime)
+    results_folder = latest_subdir
 
-#plot_lines(results_folder)
-#plot_bars(results_folder)
-#plot_box(results_folder)
 
-
-# starting_folder = "D:/Bitbucket/broadmark/Scripts/Tests/1_Artigo_Tool/"
-# def parse_folder(folder):
-#     print(" -" + folder)
-#     results = [d for d in listdir(folder) if not isfile(join(folder, d))]
-
-#     if len(results) > 0:
-#         for d in results:
-#             d = join(folder, d)
-#             parse_folder(d)
-#     else:
-#         try:
-#             process_results(folder)
-#         except:
-#             print(" -Error processing this folder")
-
-
-# parse_folder(starting_folder)
+plot_lines(results_folder)
+plot_bars(results_folder)
+plot_box(results_folder)

Scripts/run.py

@@ -4,21 +4,20 @@
 from bm_parse import process_results
 
 
-base_path = ""
 
-batch_run_name = "Faltantes/" # use this to name this testing session. If blank, a auto-generated timestamped name will be used
-root_folder = base_path + "D:/Bitbucket/broadmark/Scripts/Tests/"
-scenes_folder = base_path + "E:/Simulations/Recordings_AABBs/"
-algorithms = ["GPU_SAP"]
+batch_run_name = "" # use this to name this testing session. If blank, a auto-generated timestamped name will be used
+broadmark_bin = "D:/Bitbucket/broadmark/Broadmark/bin64/Algorithms_Release_x64.exe"
+scenes_folder = "E:/Simulations/Recordings_AABBs/"
+output_folder = "D:/Bitbucket/broadmark/Scripts/Tests/"
+algorithms = ["KD", "CGAL", "GPU_Grid"]
 #algorithms = ["SAP", "SAP_AVX_Parallel", "DBVT F", "DBVT D", "Tracy", "Tracy_Parallel", "CGAL", "Grid_3D", "Grid_3D_SAP", "Grid_3D_Parallel", "Grid_3D_SAP_Parallel", "KD", "GPU_Grid", "GPU_LBVH", "GPU_SAP"]
 additionalProperty = ""
 additionaPropertyValues = []
 
 
-broadmark_bin = base_path + "D:/Bitbucket/broadmark/Broadmark/bin64/Algorithms_Release_x64.exe"
 
 
-tests_folder = generate_jsons(batch_run_name, root_folder, scenes_folder, algorithms, additionalProperty, additionaPropertyValues)
+tests_folder = generate_jsons(batch_run_name, output_folder, scenes_folder, algorithms, additionalProperty, additionaPropertyValues)
 results_folder = run_algorithms(tests_folder, broadmark_bin)
 
 process_results(results_folder)