cbenchsuite - High accuracy C benchmark suite for Linux

cbenchsuite.org

cbenchsuite offers high accuracy performance measurements. It is written in C to minimize the impact of the suite itself on the running benchmarks. Cbench is also highly configurable and extendable, offering extensive interfaces for additional plugins or storage backends. By default cbenchsuite uses sqlite3 as storage backend.

Features

• Complete framework
  • Benchmarking systems
  • Storage in sqlite3 databases
  • Automatically generate figures and websites for the results
• Efficient benchmark framework
  • written in C
  • unloads unnecessary modules before benchmarking
• High accuracy
  • Efficient framework implementation
  • Setting scheduling priorities to get accurate monitor information
  • Warmup benchmark execution to remove measurements of cache heatup
  • Data is stored with the number of preceding benchmark executions to be able to determine the cache hotness after the benchmarks
  • Repeat benchmarks until the standard error drops below a given threshold
  • Plugins to reset Linux kernel caches/swap etc.
  • Store system information and plugin component versions to seperate different benchmark environments
• Extendable
  • Extensive module/plugin/benchsuite interface that can be used without changing any core source files
  • Storage backend interface
• Configurable
  • Many configuration options like minimum/maximum runs/runtime per benchmark
  • Execute combinations of plugins defined via command line arguments or as a seperate benchsuite inside a module
  • Plugins can have options and different versions

Example figures

• Results as stacked bar chart

• Results as simple bar chart

• Monitor data in line charts

Example website

• Example hackbench website
• Example kernel compile website
• Some other websites generated

Installation

1. First clone cbenchsuite.

    git clone https://github.com/scosu/cbenchsuite.git
    cd cbenchsuite
   

2. Configure

    mkdir build
    cd build
    cmake ..
   

3. Compile cbenchsuite.

    make
   

4. Install

    make install
   

   This will install cbenchsuite to /bin/cbenchsuite and the benchmark modules to /lib/cbenchsuite.

5. Now the benchsuite is ready to use.

    cbenchsuite
   

Requirements

• gcc (>= 4.6, I tested 4.5.3 and it did not compile due to union problems)
• libuuid
• sqlite3
• ncurses
• gperf
• matplotlib for python >= 3 (for plotter)
• python >= 3 (for plotter)

Documentation

• cbenchsuite usage
• Plotter usage
• Identifier specification
• Write a module/plugin
• Database

FAQ

1. Why are there NOTICE: ... failed to set priority ... messages?

   You are probably running cbenchsuite not as root. cbenchsuite is designed to create results that are very accurate, so it tries to set the priority for some threads. Execute the cbenchsuite as root to get rid of those messages. Also be aware that some plugins can not be executed as normal user. If you are not able to start cbenchsuite as root, you can ignore those messages without a problem.

Acknowledgements

• kconfig-frontends (Build configuration)
• Linux (Some datastructures like list etc.)
• Matplotlib
• sha256 C implementation
• statistics.py

License

/*
 * Cbench - A C benchmarking suite for Linux benchmarking.
 * Copyright (C) 2013  Markus Pargmann
 *
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License
 * as published by the Free Software Foundation; either version 2
 * of the License, or (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program; if not, write to the Free Software
 * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
 */