OPENED Extraction Tool

LPC 2022 blurb describing the goal of the tool and an initial prototype is here: https://lpc.events/event/16/contributions/1370/

Dependencies

1. Works on a) kernel verion 5.4.0-131, Ubuntu 22:04, Intel arch x86 arch b) Dockerfile works with Wondows 10, WSL2+Docker Desktop with Ubuntu 22.04 App from MS store. There is a known issue with Apple Silicon based Macbooks with installing a) gcc-multlib b) TXL and c) Codequery, described here
2. git
3. Docker

Download

1. Run git clone [email protected]:eBPFDevSecTools/opened.git followed by git submodule update --init --recursive
2. cd opened
3. To update the submodules a) git submodule update --remote --merge b) cd codequery; git pull

Install

Process 1: Docker

1. mkdir op To store the output of extraction phase (or any other folder name)
2. docker build . -t opened/extract:0.01

Process 2: On Host

1. For now: You will need to parse the Dockerfile and execute the installation steps on your host system.
2. In future we will provide a script for on-host installation (Issue #24).

Updating local branch

1. run git pull
2. run git submodule update --recursive
3. If you have docker for install, you are done.
4. If you have on-host install, you will need to re-install codequery by running the relevant instructions from Dockerfile.

Extraction code and artefacts

Code extraction consists of three phases 1) Determining the necessary functions and data-structures to be copied, 2) (Manual) disambiguation of the target set of functions identified in previous step and 3) Extracting required code from source files to generate an independantly compilable module.

Phase I: Determining necessary functions and data-structures for extracting specific functionality

1. Run annotated function call graph extraction phase,

python3 src/extraction_runner.py --help
usage: extraction_runner.py [-h] -s SRC_DIR -f FUNCTION_NAME [-d DB_FILE_NAME] [-g FUNCTION_CALL_GRAPH_PATH] -r REPO_NAME

optional arguments:
  -h, --help            show this help message and exit
  -s SRC_DIR, --src_dir SRC_DIR
                        directory with source code
  -f FUNCTION_NAME, --function_name FUNCTION_NAME
                        function name to be extracted
  -d DB_FILE_NAME, --db_file_name DB_FILE_NAME
                        Optional sqlite3 database with cqmakedb info
  -g FUNCTION_CALL_GRAPH_PATH, --function_call_graph_path FUNCTION_CALL_GRAPH_PATH
                        directory to put function and map dependency call graph file. Output of phase I
  -r REPO_NAME, --repo_name REPO_NAME
                        Project repository name

NOTE: example is given in run2.sh.

Phase II

1. Open the func.out file and remove the duplicate function and struct definitions. This will output an annotated function call graph in a file named func.out. Note that func.out may have duplicate function defintions. We expect the developer to disambiguate and identify the required set of functions to be extracted in Phase II.

Phase III: Extracting Required Code

2. Run the function extractor to extract and dump required functions and map definitions.

python3 src/function-extractor.py -h
usage: function-extractor.py [-h] -o OPDIR -c CODEQUERYOUTPUTFILE -e EXTRACTEDFILENAME -t STRUCT_INFO -f FUNC_INFO -s SRCDIR -b BASEDIR [--isCilium]

Function Extractor

optional arguments:
  -h, --help            show this help message and exit
  -o OPDIR, --opdir OPDIR
                        directory to dump extracted files to
  -c CODEQUERYOUTPUTFILE, --codequeryOutputFile CODEQUERYOUTPUTFILE
                        Function and Map dependency output from codequery
  -e EXTRACTEDFILENAME, --extractedFileName EXTRACTEDFILENAME
                        Output file with extracted function
  -t STRUCT_INFO, --struct_info STRUCT_INFO
                        json file containing struct definitions in the repo
  -f FUNC_INFO, --func_info FUNC_INFO
                        json file containing function definitions in the repo
  -s SRCDIR, --srcdir SRCDIR
                        Directory containing source files for function to be extraced from
  -b BASEDIR, --basedir BASEDIR
                        Base Directory path relative to which directory structure in opdir will be created
  --isCilium            whether repository is cilium

Note that STRUCT_INFO and FUNC_INFO are generated using the annotator script in the eBPF-projects-annotations repo

Note that extracted.c may contain duplicate eBPF Map defintions within and ATTENTION section. We expect the developer to choose the right map definition and delete the offending defintion.

Compilation

Run make to compile the extracted code.