demo_missing_compas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "burning-plasma",
   "metadata": {},
   "source": [
    "## Missing values with COMPAS data\n",
    "This notebook demonstrates the effect of MAR and MNAR missing values on fairness using Adult data. <br>\n",
    "In this notebook, we first import packages needed in this file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "phantom-antenna",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(\"models\")\n",
    "sys.path.append(\"AIF360/\")\n",
    "import numpy as np\n",
    "from compas_model import get_distortion_compas, CompasDataset, reweight_df, get_evaluation\n",
    "from aif360.algorithms.preprocessing.optim_preproc import OptimPreproc\n",
    "from aif360.algorithms.preprocessing.optim_preproc_helpers.opt_tools import OptTools\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "careful-impossible",
   "metadata": {},
   "source": [
    "The function below process data and create missing values in the dataset. <br>\n",
    "In COMPAS dataset, we have race as sensitive attribute and use age (binned into 3 bins), priors_count (number of prior criminal cases, binned into 3 bins), c_charge_degree (degree charged by the prosecutor) and score_text (a risk score assigned to the defendant) as features to predict two_year_recid (if person will be re-arrested for a violent offense within two years). <br>\n",
    "In this dataset, we create missing values in the feature priors_count with MNAR and MAR type of missing values. In the function below, the missing value mechanism is MNAR that the missing values depends on the feature itself. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "appointed-chosen",
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_preproc_data_compas(protected_attributes=None):\n",
    "    def custom_preprocessing(df):\n",
    "        \"\"\"The custom pre-processing function is adapted from\n",
    "            https://github.com/fair-preprocessing/nips2017/blob/master/compas/code/Generate_Compas_Data.ipynb\n",
    "        \"\"\"\n",
    "\n",
    "        df = df[['age',\n",
    "                 'c_charge_degree',\n",
    "                 'race',\n",
    "                 'age_cat',\n",
    "                 'score_text',\n",
    "                 'sex',\n",
    "                 'priors_count',\n",
    "                 'days_b_screening_arrest',\n",
    "                 'decile_score',\n",
    "                 'is_recid',\n",
    "                 'two_year_recid',\n",
    "                 'length_of_stay']]\n",
    "\n",
    "        # Indices of data samples to keep\n",
    "        ix = df['days_b_screening_arrest'] <= 30\n",
    "        ix = (df['days_b_screening_arrest'] >= -30) & ix\n",
    "        ix = (df['is_recid'] != -1) & ix\n",
    "        ix = (df['c_charge_degree'] != \"O\") & ix\n",
    "        ix = (df['score_text'] != 'N/A') & ix\n",
    "        df = df.loc[ix, :]\n",
    "        # Restrict races to African-American and Caucasian\n",
    "        dfcut = df.loc[~df['race'].isin(\n",
    "            ['Native American', 'Hispanic', 'Asian', 'Other']), :]\n",
    "\n",
    "        # Restrict the features to use\n",
    "        dfcutQ = dfcut[['sex',\n",
    "                        'race',\n",
    "                        'age_cat',\n",
    "                        'c_charge_degree',\n",
    "                        'score_text',\n",
    "                        'priors_count',\n",
    "                        'is_recid',\n",
    "                        'two_year_recid',\n",
    "                        'length_of_stay']].copy()\n",
    "\n",
    "        # Quantize priors count between 0, 1-3, and >3\n",
    "        def quantizePrior(x):\n",
    "            if x == 0:\n",
    "                return '0'\n",
    "            elif x == 1:\n",
    "                return '1 to 3'\n",
    "            elif x == 2:\n",
    "                return 'More than 3'\n",
    "            elif x == 'missing':\n",
    "                return 'missing'\n",
    "        # Quantize length of stay\n",
    "\n",
    "        def quantizeLOS(x):\n",
    "            if x == 0:\n",
    "                return '<week'\n",
    "            if x == 1:\n",
    "                return '<3months'\n",
    "            else:\n",
    "                return '>3 months'\n",
    "\n",
    "        # Quantize length of stay\n",
    "        def adjustAge(x):\n",
    "            if x == 0:\n",
    "                return '25 to 45'\n",
    "            elif x == 1:\n",
    "                return 'Greater than 45'\n",
    "            elif x == 2:\n",
    "                return 'Less than 25'\n",
    "\n",
    "        def quantizeScore(x):\n",
    "            if x == 1:\n",
    "                return 'MediumHigh'\n",
    "            else:\n",
    "                return 'Low'\n",
    "\n",
    "        def group_race(x):\n",
    "            if x == \"Caucasian\":\n",
    "                return 1.0\n",
    "            else:\n",
    "                return 0.0\n",
    "        \n",
    "        np.random.seed(10)\n",
    "        df1 = dfcutQ[['priors_count', 'c_charge_degree', 'race',\n",
    "                  'age_cat', 'score_text', 'two_year_recid']]\n",
    "        # Here we define a column called mis_prob to assign the probability of each observation \n",
    "        # being missed\n",
    "        dfcutQ['mis_prob'] = 0\n",
    "        # Here, the probability of missing values in priors_count depends on race, two_year_recid\n",
    "        # and priors_count, so in this case the missing values are under MNAR that the probability\n",
    "        # of missing values depends on the feature itself\n",
    "        # To change the distribution of missing values, we can change the probability here\n",
    "        for index, row in dfcutQ.iterrows():\n",
    "            if row['race'] == 'African-American' and row['two_year_recid']==0 and row['priors_count']==0:\n",
    "                dfcutQ.loc[index, 'mis_prob'] = 0.5\n",
    "            elif row['race'] != 'African-American' and row['two_year_recid']==1 and row['priors_count']==2:\n",
    "                dfcutQ.loc[index, 'mis_prob'] = 0.3\n",
    "            elif row['race'] == 'African-American':\n",
    "                dfcutQ.loc[index, 'mis_prob'] = 0.2\n",
    "            else:\n",
    "                dfcutQ.loc[index, 'mis_prob'] = 0.05\n",
    "        new_label = []\n",
    "        for index, row in dfcutQ.iterrows():\n",
    "            if np.random.binomial(1, float(row['mis_prob']), 1)[0] == 1:\n",
    "                new_label.append('missing')\n",
    "            else:\n",
    "                new_label.append(row['priors_count'])\n",
    "        dfcutQ['priors_count'] = new_label\n",
    "        print('Total number of missing values')\n",
    "        print(len(dfcutQ.loc[dfcutQ['priors_count'] == 'missing', :].index))\n",
    "        print('Total number of observations')\n",
    "        print(len(dfcutQ.index))\n",
    "\n",
    "        dfcutQ['priors_count'] = dfcutQ['priors_count'].apply(\n",
    "            lambda x: quantizePrior(x))\n",
    "        dfcutQ['length_of_stay'] = dfcutQ['length_of_stay'].apply(\n",
    "            lambda x: quantizeLOS(x))\n",
    "        dfcutQ['score_text'] = dfcutQ['score_text'].apply(\n",
    "            lambda x: quantizeScore(x))\n",
    "        dfcutQ['age_cat'] = dfcutQ['age_cat'].apply(lambda x: adjustAge(x))\n",
    "        # Recode sex and race\n",
    "        dfcutQ['sex'] = dfcutQ['sex'].replace({'Female': 1.0, 'Male': 0.0})\n",
    "        dfcutQ['race'] = dfcutQ['race'].apply(lambda x: group_race(x))\n",
    "\n",
    "        features = ['two_year_recid', 'race',\n",
    "                    'age_cat', 'priors_count', 'c_charge_degree', 'score_text']\n",
    "\n",
    "        # Pass vallue to df\n",
    "        df = dfcutQ[features]\n",
    "        return df\n",
    "\n",
    "    XD_features = [\n",
    "        'age_cat',\n",
    "        'c_charge_degree',\n",
    "        'priors_count',\n",
    "        'race',\n",
    "        'score_text']\n",
    "    D_features = [\n",
    "        'race'] if protected_attributes is None else protected_attributes\n",
    "    Y_features = ['two_year_recid']\n",
    "    X_features = list(set(XD_features) - set(D_features))\n",
    "    categorical_features = [\n",
    "        'age_cat',\n",
    "        'priors_count',\n",
    "        'c_charge_degree',\n",
    "        'score_text']\n",
    "\n",
    "    # privileged classes\n",
    "    all_privileged_classes = {\"sex\": [1.0],\n",
    "                              \"race\": [1.0]}\n",
    "\n",
    "    # protected attribute maps\n",
    "    all_protected_attribute_maps = {\n",
    "        \"sex\": {\n",
    "            0.0: 'Male', 1.0: 'Female'}, \"race\": {\n",
    "            1.0: 'Caucasian', 0.0: 'Not Caucasian'}}\n",
    "\n",
    "    return CompasDataset(\n",
    "        label_name=Y_features[0],\n",
    "        favorable_classes=[0],\n",
    "        protected_attribute_names=D_features,\n",
    "        privileged_classes=[all_privileged_classes[x] for x in D_features],\n",
    "        instance_weights_name=None,\n",
    "        categorical_features=categorical_features,\n",
    "        features_to_keep=X_features + Y_features + D_features,\n",
    "        na_values=[],\n",
    "        metadata={'label_maps': [{1.0: 'Did recid.', 0.0: 'No recid.'}],\n",
    "                  'protected_attribute_maps': [all_protected_attribute_maps[x]\n",
    "                                               for x in D_features]},\n",
    "        custom_preprocessing=custom_preprocessing)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ethical-rider",
   "metadata": {},
   "source": [
    "The code below is to load the data and run the fairness fixing algorithm proposed by Calmon et al. \\[1\\]. We set missing values as a new category in features containing missing values. <br>\n",
    "Note that we modified the distortion function at ```get_distortion_compas```. In this function, we define the penalty for the fairness fixing algorithm to change values in each feature. In this distortion function, we set penalty to be 0 if the original observation value changes from the missing category to a non-missing category and we set a big penalty if the original value changes from a non-missing category to the missing category or the original values remain at the missing category. <br> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "contemporary-panel",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of missing values\n",
      "906\n",
      "Total number of observations\n",
      "5278\n",
      "Optimized Preprocessing: Objective converged to 0.157471\n"
     ]
    }
   ],
   "source": [
    "privileged_groups = [{'race': 1}]\n",
    "unprivileged_groups = [{'race': 0}]\n",
    "dataset_orig = load_preproc_data_compas(['race'])\n",
    "\n",
    "optim_options = {\n",
    "    \"distortion_fun\": get_distortion_compas,\n",
    "    \"epsilon\": 0.05,\n",
    "    \"clist\": [0.99, 1.99, 2.99],\n",
    "    \"dlist\": [.1, 0.05, 0]\n",
    "}\n",
    "\n",
    "dataset_orig_train, dataset_orig_vt = dataset_orig.split(\n",
    "    [0.7], shuffle=True)\n",
    "\n",
    "OP = OptimPreproc(OptTools, optim_options,\n",
    "                  unprivileged_groups=unprivileged_groups,\n",
    "                  privileged_groups=privileged_groups)\n",
    "\n",
    "OP = OP.fit(dataset_orig_train)\n",
    "\n",
    "dataset_transf_cat_test = OP.transform(dataset_orig_vt, transform_Y=True)\n",
    "dataset_transf_cat_test = dataset_orig_vt.align_datasets(\n",
    "    dataset_transf_cat_test)\n",
    "\n",
    "dataset_transf_cat_train = OP.transform(\n",
    "    dataset_orig_train, transform_Y=True)\n",
    "dataset_transf_cat_train = dataset_orig_train.align_datasets(\n",
    "    dataset_transf_cat_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "known-storm",
   "metadata": {},
   "source": [
    "In this part we use the training data obtained from the fairness fixing algorithm by Calmon et al. \\[1\\] to train a logistic regression classifier and validate the classifier on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "lasting-input",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Without reweight\n",
      "Accuracy\n",
      "0.6502525252525253\n",
      "p-rule\n",
      "0.7667486382146068\n",
      "FPR for unpriv group\n",
      "0.37632135306553915\n",
      "FNR for unpriv group\n",
      "0.34439834024896265\n",
      "FPR for priv group\n",
      "0.25133689839572193\n",
      "FNR for priv group\n",
      "0.4549019607843138\n"
     ]
    }
   ],
   "source": [
    "scale_transf = StandardScaler()\n",
    "X_train = scale_transf.fit_transform(dataset_transf_cat_train.features)\n",
    "y_train = dataset_transf_cat_train.labels.ravel()\n",
    "X_test = scale_transf.fit_transform(dataset_transf_cat_test.features)\n",
    "lmod = LogisticRegression()\n",
    "lmod.fit(X_train, y_train)\n",
    "y_pred = lmod.predict(X_test)\n",
    "print('Without reweight')\n",
    "get_evaluation(dataset_orig_vt,y_pred,privileged_groups,unprivileged_groups,0,1,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "august-structure",
   "metadata": {},
   "source": [
    "After getting the accuracy and fairness results, we apply our reweighting algorithm to train a new logistic regression classifier and validate the classifier on the same test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "interracial-fisher",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With reweight\n",
      "Accuracy\n",
      "0.648989898989899\n",
      "p-rule\n",
      "0.8382674916706331\n",
      "FPR for unpriv group\n",
      "0.34460887949260044\n",
      "FNR for unpriv group\n",
      "0.37344398340248963\n",
      "FPR for priv group\n",
      "0.27005347593582885\n",
      "FNR for priv group\n",
      "0.4392156862745098\n"
     ]
    }
   ],
   "source": [
    "dataset_orig_train.instance_weights = reweight_df(dataset_orig_train)\n",
    "scale_transf = StandardScaler()\n",
    "X_train = scale_transf.fit_transform(dataset_transf_cat_train.features)\n",
    "y_train = dataset_transf_cat_train.labels.ravel()\n",
    "X_test = scale_transf.fit_transform(dataset_transf_cat_test.features)\n",
    "lmod = LogisticRegression()\n",
    "lmod.fit(\n",
    "    X_train,\n",
    "    y_train,\n",
    "    sample_weight=dataset_orig_train.instance_weights)\n",
    "y_pred = lmod.predict(X_test)\n",
    "print('With reweight')\n",
    "get_evaluation(dataset_orig_vt,y_pred,privileged_groups,unprivileged_groups,0,1,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "nervous-resort",
   "metadata": {},
   "source": [
    "By comparing the two results, the fairness scores increase with a very small tradeoff in accuracy (about 0.1-0.2\\% decrease in accuracy) <br>\n",
    "The code chunks below process data and create missing values with MAR missing type. <br>\n",
    "The function below process data and create missing values in the dataset. In the function below, the missing value mechanism is MAR that the missing values do not depend on the feature itself.<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "signed-headline",
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_preproc_data_compas(protected_attributes=None):\n",
    "    def custom_preprocessing(df):\n",
    "        \"\"\"The custom pre-processing function is adapted from\n",
    "            https://github.com/fair-preprocessing/nips2017/blob/master/compas/code/Generate_Compas_Data.ipynb\n",
    "        \"\"\"\n",
    "\n",
    "        df = df[['age',\n",
    "                 'c_charge_degree',\n",
    "                 'race',\n",
    "                 'age_cat',\n",
    "                 'score_text',\n",
    "                 'sex',\n",
    "                 'priors_count',\n",
    "                 'days_b_screening_arrest',\n",
    "                 'decile_score',\n",
    "                 'is_recid',\n",
    "                 'two_year_recid',\n",
    "                 'length_of_stay']]\n",
    "\n",
    "        # Indices of data samples to keep\n",
    "        ix = df['days_b_screening_arrest'] <= 30\n",
    "        ix = (df['days_b_screening_arrest'] >= -30) & ix\n",
    "        ix = (df['is_recid'] != -1) & ix\n",
    "        ix = (df['c_charge_degree'] != \"O\") & ix\n",
    "        ix = (df['score_text'] != 'N/A') & ix\n",
    "        df = df.loc[ix, :]\n",
    "        # Restrict races to African-American and Caucasian\n",
    "        dfcut = df.loc[~df['race'].isin(\n",
    "            ['Native American', 'Hispanic', 'Asian', 'Other']), :]\n",
    "\n",
    "        # Restrict the features to use\n",
    "        dfcutQ = dfcut[['sex',\n",
    "                        'race',\n",
    "                        'age_cat',\n",
    "                        'c_charge_degree',\n",
    "                        'score_text',\n",
    "                        'priors_count',\n",
    "                        'is_recid',\n",
    "                        'two_year_recid',\n",
    "                        'length_of_stay']].copy()\n",
    "\n",
    "        # Quantize priors count between 0, 1-3, and >3\n",
    "        def quantizePrior(x):\n",
    "            if x == 0:\n",
    "                return '0'\n",
    "            elif x == 1:\n",
    "                return '1 to 3'\n",
    "            elif x == 2:\n",
    "                return 'More than 3'\n",
    "            elif x == 'missing':\n",
    "                return 'missing'\n",
    "        # Quantize length of stay\n",
    "\n",
    "        def quantizeLOS(x):\n",
    "            if x == 0:\n",
    "                return '<week'\n",
    "            if x == 1:\n",
    "                return '<3months'\n",
    "            else:\n",
    "                return '>3 months'\n",
    "\n",
    "        # Quantize length of stay\n",
    "        def adjustAge(x):\n",
    "            if x == 0:\n",
    "                return '25 to 45'\n",
    "            elif x == 1:\n",
    "                return 'Greater than 45'\n",
    "            elif x == 2:\n",
    "                return 'Less than 25'\n",
    "\n",
    "        def quantizeScore(x):\n",
    "            if x == 1:\n",
    "                return 'MediumHigh'\n",
    "            else:\n",
    "                return 'Low'\n",
    "\n",
    "        def group_race(x):\n",
    "            if x == \"Caucasian\":\n",
    "                return 1.0\n",
    "            else:\n",
    "                return 0.0\n",
    "        \n",
    "        np.random.seed(10)\n",
    "        df1 = dfcutQ[['priors_count', 'c_charge_degree', 'race',\n",
    "                  'age_cat', 'score_text', 'two_year_recid']]\n",
    "        # Here we define a column called mis_prob to assign the probability of each observation \n",
    "        # being missed\n",
    "        dfcutQ['mis_prob'] = 0\n",
    "        # Here, the probability of missing values in priors_count depend on race and \n",
    "        # two_year_recid, so in this case the missing values are under MAR because the missingness \n",
    "        # does not depend on the feature priors_count\n",
    "        # To change the distribution of missing values, we can change the probability here\n",
    "        for index, row in dfcutQ.iterrows():\n",
    "            if row['race'] == 'African-American' and row['two_year_recid']==0:\n",
    "                dfcutQ.loc[index, 'mis_prob'] = 0.2\n",
    "            elif row['race'] == 'African-American':\n",
    "                dfcutQ.loc[index, 'mis_prob'] = 0.15\n",
    "            else:\n",
    "                dfcutQ.loc[index, 'mis_prob'] = 0.05\n",
    "        new_label = []\n",
    "        for index, row in dfcutQ.iterrows():\n",
    "            if np.random.binomial(1, float(row['mis_prob']), 1)[0] == 1:\n",
    "                new_label.append('missing')\n",
    "            else:\n",
    "                new_label.append(row['priors_count'])\n",
    "        dfcutQ['priors_count'] = new_label\n",
    "        print('Total number of missing values')\n",
    "        print(len(dfcutQ.loc[dfcutQ['priors_count'] == 'missing', :].index))\n",
    "        print('Total number of observations')\n",
    "        print(len(dfcutQ.index))\n",
    "\n",
    "        dfcutQ['priors_count'] = dfcutQ['priors_count'].apply(\n",
    "            lambda x: quantizePrior(x))\n",
    "        dfcutQ['length_of_stay'] = dfcutQ['length_of_stay'].apply(\n",
    "            lambda x: quantizeLOS(x))\n",
    "        dfcutQ['score_text'] = dfcutQ['score_text'].apply(\n",
    "            lambda x: quantizeScore(x))\n",
    "        dfcutQ['age_cat'] = dfcutQ['age_cat'].apply(lambda x: adjustAge(x))\n",
    "        # Recode sex and race\n",
    "        dfcutQ['sex'] = dfcutQ['sex'].replace({'Female': 1.0, 'Male': 0.0})\n",
    "        dfcutQ['race'] = dfcutQ['race'].apply(lambda x: group_race(x))\n",
    "\n",
    "        features = ['two_year_recid', 'race',\n",
    "                    'age_cat', 'priors_count', 'c_charge_degree', 'score_text']\n",
    "\n",
    "        # Pass vallue to df\n",
    "        df = dfcutQ[features]\n",
    "\n",
    "        return df\n",
    "\n",
    "    XD_features = [\n",
    "        'age_cat',\n",
    "        'c_charge_degree',\n",
    "        'priors_count',\n",
    "        'race',\n",
    "        'score_text']\n",
    "    D_features = [\n",
    "        'race'] if protected_attributes is None else protected_attributes\n",
    "    Y_features = ['two_year_recid']\n",
    "    X_features = list(set(XD_features) - set(D_features))\n",
    "    categorical_features = [\n",
    "        'age_cat',\n",
    "        'priors_count',\n",
    "        'c_charge_degree',\n",
    "        'score_text']\n",
    "\n",
    "    # privileged classes\n",
    "    all_privileged_classes = {\"sex\": [1.0],\n",
    "                              \"race\": [1.0]}\n",
    "\n",
    "    # protected attribute maps\n",
    "    all_protected_attribute_maps = {\n",
    "        \"sex\": {\n",
    "            0.0: 'Male', 1.0: 'Female'}, \"race\": {\n",
    "            1.0: 'Caucasian', 0.0: 'Not Caucasian'}}\n",
    "\n",
    "    return CompasDataset(\n",
    "        label_name=Y_features[0],\n",
    "        favorable_classes=[0],\n",
    "        protected_attribute_names=D_features,\n",
    "        privileged_classes=[all_privileged_classes[x] for x in D_features],\n",
    "        instance_weights_name=None,\n",
    "        categorical_features=categorical_features,\n",
    "        features_to_keep=X_features + Y_features + D_features,\n",
    "        na_values=[],\n",
    "        metadata={'label_maps': [{1.0: 'Did recid.', 0.0: 'No recid.'}],\n",
    "                  'protected_attribute_maps': [all_protected_attribute_maps[x]\n",
    "                                               for x in D_features]},\n",
    "        custom_preprocessing=custom_preprocessing)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "illegal-comedy",
   "metadata": {},
   "source": [
    "Same as above, we load the data and run the fairness fixing algorithm proposed by Calmon et al."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "stylish-possession",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of missing values\n",
      "623\n",
      "Total number of observations\n",
      "5278\n",
      "Optimized Preprocessing: Objective converged to 0.102867\n"
     ]
    }
   ],
   "source": [
    "privileged_groups = [{'race': 1}]\n",
    "unprivileged_groups = [{'race': 0}]\n",
    "dataset_orig = load_preproc_data_compas(['race'])\n",
    "\n",
    "optim_options = {\n",
    "    \"distortion_fun\": get_distortion_compas,\n",
    "    \"epsilon\": 0.05,\n",
    "    \"clist\": [0.99, 1.99, 2.99],\n",
    "    \"dlist\": [.1, 0.05, 0]\n",
    "}\n",
    "\n",
    "dataset_orig_train, dataset_orig_vt = dataset_orig.split(\n",
    "    [0.7], shuffle=True)\n",
    "\n",
    "OP = OptimPreproc(OptTools, optim_options,\n",
    "                  unprivileged_groups=unprivileged_groups,\n",
    "                  privileged_groups=privileged_groups)\n",
    "\n",
    "OP = OP.fit(dataset_orig_train)\n",
    "\n",
    "dataset_transf_cat_test = OP.transform(dataset_orig_vt, transform_Y=True)\n",
    "dataset_transf_cat_test = dataset_orig_vt.align_datasets(\n",
    "    dataset_transf_cat_test)\n",
    "\n",
    "dataset_transf_cat_train = OP.transform(\n",
    "    dataset_orig_train, transform_Y=True)\n",
    "dataset_transf_cat_train = dataset_orig_train.align_datasets(\n",
    "    dataset_transf_cat_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "behavioral-chorus",
   "metadata": {},
   "source": [
    "Same as MNAR case, we first train a logistic regression classifier without reweight and train another logistic regression classifier with reweight and validate both of them on the same test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "inner-designation",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Without reweight\n",
      "Accuracy\n",
      "0.6546717171717171\n",
      "p-rule\n",
      "0.7910284406324818\n",
      "FPR for unpriv group\n",
      "0.3568464730290456\n",
      "FNR for unpriv group\n",
      "0.35306553911205074\n",
      "FPR for priv group\n",
      "0.4549019607843138\n",
      "FNR for priv group\n",
      "0.2459893048128342\n"
     ]
    }
   ],
   "source": [
    "scale_transf = StandardScaler()\n",
    "X_train = scale_transf.fit_transform(dataset_transf_cat_train.features)\n",
    "y_train = dataset_transf_cat_train.labels.ravel()\n",
    "X_test = scale_transf.fit_transform(dataset_transf_cat_test.features)\n",
    "lmod = LogisticRegression()\n",
    "lmod.fit(X_train, y_train)\n",
    "y_pred = lmod.predict(X_test)\n",
    "print('Without reweight')\n",
    "get_evaluation(dataset_orig_vt,y_pred,privileged_groups,unprivileged_groups,0,1,0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "bacterial-portfolio",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With reweight\n",
      "Accuracy\n",
      "0.6534090909090909\n",
      "p-rule\n",
      "0.8284982088729678\n",
      "FPR for unpriv group\n",
      "0.3568464730290456\n",
      "FNR for unpriv group\n",
      "0.35306553911205074\n",
      "FPR for priv group\n",
      "0.42352941176470593\n",
      "FNR for priv group\n",
      "0.2727272727272727\n"
     ]
    }
   ],
   "source": [
    "dataset_orig_train.instance_weights = reweight_df(dataset_orig_train)\n",
    "scale_transf = StandardScaler()\n",
    "X_train = scale_transf.fit_transform(dataset_transf_cat_train.features)\n",
    "y_train = dataset_transf_cat_train.labels.ravel()\n",
    "X_test = scale_transf.fit_transform(dataset_transf_cat_test.features)\n",
    "lmod = LogisticRegression()\n",
    "lmod.fit(\n",
    "    X_train,\n",
    "    y_train,\n",
    "    sample_weight=dataset_orig_train.instance_weights)\n",
    "y_pred = lmod.predict(X_test)\n",
    "print('With reweight')\n",
    "get_evaluation(dataset_orig_vt,y_pred,privileged_groups,unprivileged_groups,0,1,0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "measured-repair",
   "metadata": {},
   "source": [
    "Similar to results from MNAR, our reweighting algorithm improves the fairness scores with a very small tradeoff in accuracy. <br>\n",
    "# Reference\n",
    "[1] Optimized Pre-Processing for Discrimination Prevention <br>\n",
    "Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy and Kush R. Varshney.\n",
    "31st Advances in Neural Information Processing Systems (NIPS), Long Beach, CA, December 2017."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}