Python functions to facilitate the pre-processing of data to prepare them for ML tasks, especially suitable for data in a clinical context.
Major functionalities include heuristic based data cleaning and feature engineering like:
- Automatic detection of encoding strings (e.g. 1=m) and application of the corresponding encoding to un-encoded data of the corresponding column
- Automatic detection of date strings of different formats (e.g. 2019-01-01, 01/01/2019, January 2022) and conversion to a unified format
- Encoding of date strings into decomposed date features (e.g. year, month, day, weekday, etc.)
- Heuristics for unification of different number formats, e.g. 1,000.00 vs. 1.000,00 or exponential notations like 1e3 vs 10x10^2
- Detection and replacement of inconsistent data values
Install via pip:
pip install cleandat