cah_eda.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1fd1e26b",
   "metadata": {},
   "source": [
    "Each row of the file represents a white card that was given to a player in a round and how it performed. It also contains information about the round like how long it took and what black card was dealt.\n",
    "So columns that describe general round data will be exactly the same for all ten rows of that round. These rows are: fake_round_id, round_completion_seconds, round_skipped, black_card_text, and black_card_pick_num.\n",
    "But columns that describe a property of the white card will change over the ten rows of that round. These rows are: white_card_text, won, and winning_index.\n",
    "\n",
    "\n",
    "The columns:\n",
    "* fake_round_id: a unique integer for every round. This is to help you separate one round from another to find which 10 cards were presented in every round.\n",
    "* round_completion_seconds: the number of seconds it took for the user to pick a winner after being presented the cards for that round. You'll probably want to filter out users that decide too quickly by setting some minimum threshold for this. Some of these numbers can be insanely high as well since the user just left their browser open for hours or possibly days before picking a winner.\n",
    "* round_skipped: we have a red button in the top right of the lab page that says \"No Good Plays!\". If a user clicks this button, round_skipped will be \"True\" and no white card will be marked as a winner.\n",
    "* black_card_text: self explanatory.\n",
    "* black_card_pick_num: The number of white cards that the black card requires. Generally 1, but will be 2 for black cards like \"That's right, I killed _____. How, you ask? _____.\"\n",
    "* white_card_text: self explanatory.\n",
    "* won: True if this white card won the round.\n",
    "* winning_index: This is for storing the order that the winning card(s) were picked in. If the white card didn't win, this will just contain a null value (NULL). If the card won and the black card was a normal \"Pick 1\" this will always be 0. But if the black card was a \"Pick 2\", one of the winners will be 0 and the other will be 1. This is how you figure out which blank in pick 2 black cards was meant to be filled by which winning white card. The first blank is filled by the 0 and the second one by the 1."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa5e49f8",
   "metadata": {},
   "source": [
    "* sample groups (for test);\n",
    "* https://hippocampus-garden.com/pandas_group_sample/\n",
    "```\n",
    "sampled_users = np.random.choice(df[\"user_id\"].unique(), 100)\n",
    "df_sampled = df.groupby('user_id').filter(lambda x: x[\"user_id\"].values[0] in sampled_users)\n",
    "df_sampled[\"user_id\"].nunique()\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "experienced-isolation",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "pd.set_option(\"display.precision\", 4)\n",
    "pd.set_option('display.width', 1024)\n",
    "pd.set_option('display.max_colwidth', None)\n",
    "%matplotlib inline \n",
    "from sklearn.linear_model import LogisticRegression, LogisticRegressionCV\n",
    "from sklearn.model_selection import train_test_split,GridSearchCV, cross_val_score, cross_val_predict\n",
    "from sklearn.preprocessing import StandardScaler, MinMaxScaler\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.metrics import classification_report, roc_auc_score  \n",
    "# https://stackoverflow.com/questions/53784971/how-to-disable-convergencewarning-using-sklearn\n",
    "from warnings import simplefilter\n",
    "from sklearn.exceptions import ConvergenceWarning\n",
    "simplefilter(\"ignore\", category=ConvergenceWarning)\n",
    "\n",
    "from sentence_transformers import SentenceTransformer, util\n",
    "LOAD_CSV = False ## load raw data from csv instead of parquet (which has had some cols dropped)\n",
    "\n",
    "MIN_PAIR_FREQ = 4\n",
    "SAVE_TRAIN_DATA = False\n",
    "\n",
    "LOWERCASE_DATA = False\n",
    "DROP_DOUBLES = True ## drop white cards with pick 2 options ; and their blacks. (TODO: analyze interesting pairs)\n",
    "DROP_SKIPPED = True ## ignore rounds where skipped"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e259e07f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install pyarrow\n",
    "# !conda install -c conda-forge category_encoders -y\n",
    "from category_encoders.wrapper import NestedCVWrapper ## cv\n",
    "from category_encoders import TargetEncoder,WOEEncoder,CountEncoder\n",
    "from category_encoders.leave_one_out import LeaveOneOutEncoder\n",
    "from category_encoders.cat_boost import CatBoostEncoder\n",
    "# cat_encoder = TargetEncoder(min_samples_leaf=3, smoothing=0.2)\n",
    "## note - not split by groups\n",
    "cat_encoder = NestedCVWrapper(feature_encoder=CatBoostEncoder(), cv=5)#CatBoostEncoder()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bridal-paris",
   "metadata": {},
   "source": [
    "Load data\n",
    "\n",
    "* Note there are some near-duplicate cards (e.g. footballers - could minhash - https://github.com/chrisjmccormick/MinHash/blob/master/runMinHashExample.py , LSH/shingles : https://onestopdataanalysis.com/lsh/  (broken library?) , https://github.com/zyocum/dedup/blob/master/dedup.py ,\n",
    "\n",
    "`Datasketch` https://www.learndatasci.com/tutorials/building-recommendation-engine-locality-sensitive-hashing-lsh-python/ ,\n",
    "https://stackoverflow.com/questions/25114338/approximate-string-matching-using-lsh , \n",
    "\n",
    "SKLearn friendly?: http://ethen8181.github.io/machine-learning/recsys/content_based/lsh_text.html\n",
    "\n",
    "`dedupe` library - https://github.com/dedupeio/dedupe\n",
    "\n",
    "* Paraphrase mining using sentence bert - https://www.sbert.net/examples/applications/paraphrase-mining/README.html  (may be too semantic)\n",
    "\n",
    "* Example: \"10 football players with erections *barreling* towards you at full speed\" - 10 football players with erections **barrelling** towards you at full speed.\"\n",
    "\n",
    "Task modelling:\n",
    "* Ranking = most accurate\n",
    "* Regression/classification: can be done just over single cards, or between pairs. \n",
    "    * if pairs-wise: can use sentencebert/SNLI models (which assume pairs of sentences) - https://github.com/UKPLab/sentence-transformers\n",
    "    \n",
    "    \n",
    "Examples:\n",
    "\n",
    "* https://www.sbert.net/examples/training/sts/README.html#training-data\n",
    "* https://github.com/UKPLab/sentence-transformers/blob/master/examples/training/nli/training_nli.py\n",
    "* https://github.com/UKPLab/sentence-transformers/blob/master/examples/training/nli/training_nli_v2.py\n",
    "\n",
    "\n",
    "* Keras example (not sentenceBert): https://keras.io/examples/nlp/semantic_similarity_with_bert/\n",
    "\n",
    "\n",
    "* Interpretability tool : https://github.com/cdpierse/transformers-interpret"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "sound-transcript",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 723 ms\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fake_round_id</th>\n",
       "      <th>round_completion_seconds</th>\n",
       "      <th>round_skipped</th>\n",
       "      <th>black_card_text</th>\n",
       "      <th>black_card_pick_num</th>\n",
       "      <th>white_card_text</th>\n",
       "      <th>won</th>\n",
       "      <th>ID_index</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>False</td>\n",
       "      <td>Hi MTV! My name is Kendra, I live in Malibu, I'm into _____, and I love to have a good time.</td>\n",
       "      <td>1</td>\n",
       "      <td>Going inside at some point because of the mosquitoes.</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>False</td>\n",
       "      <td>Hi MTV! My name is Kendra, I live in Malibu, I'm into _____, and I love to have a good time.</td>\n",
       "      <td>1</td>\n",
       "      <td>Being fat from noodles.</td>\n",
       "      <td>False</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>False</td>\n",
       "      <td>Hi MTV! My name is Kendra, I live in Malibu, I'm into _____, and I love to have a good time.</td>\n",
       "      <td>1</td>\n",
       "      <td>Letting this loser eat me out.</td>\n",
       "      <td>False</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>False</td>\n",
       "      <td>Hi MTV! My name is Kendra, I live in Malibu, I'm into _____, and I love to have a good time.</td>\n",
       "      <td>1</td>\n",
       "      <td>That chicken from Popeyes.®</td>\n",
       "      <td>False</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>False</td>\n",
       "      <td>Hi MTV! My name is Kendra, I live in Malibu, I'm into _____, and I love to have a good time.</td>\n",
       "      <td>1</td>\n",
       "      <td>A sorry excuse for a father.</td>\n",
       "      <td>False</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2989545</th>\n",
       "      <td>298955</td>\n",
       "      <td>7613</td>\n",
       "      <td>False</td>\n",
       "      <td>Oh my god! _____ killed Kenny!</td>\n",
       "      <td>1</td>\n",
       "      <td>Breastfeeding a ten-year-old.</td>\n",
       "      <td>False</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2989546</th>\n",
       "      <td>298955</td>\n",
       "      <td>7613</td>\n",
       "      <td>False</td>\n",
       "      <td>Oh my god! _____ killed Kenny!</td>\n",
       "      <td>1</td>\n",
       "      <td>Happy daddies with happy sandals.</td>\n",
       "      <td>False</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2989547</th>\n",
       "      <td>298955</td>\n",
       "      <td>7613</td>\n",
       "      <td>False</td>\n",
       "      <td>Oh my god! _____ killed Kenny!</td>\n",
       "      <td>1</td>\n",
       "      <td>Jerking off to a 10-second RealMedia clip.</td>\n",
       "      <td>False</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2989548</th>\n",
       "      <td>298955</td>\n",
       "      <td>7613</td>\n",
       "      <td>False</td>\n",
       "      <td>Oh my god! _____ killed Kenny!</td>\n",
       "      <td>1</td>\n",
       "      <td>Getting naked and watching Nickelodeon.</td>\n",
       "      <td>False</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2989549</th>\n",
       "      <td>298955</td>\n",
       "      <td>7613</td>\n",
       "      <td>False</td>\n",
       "      <td>Oh my god! _____ killed Kenny!</td>\n",
       "      <td>1</td>\n",
       "      <td>Awesome pictures of planets and stuff.</td>\n",
       "      <td>False</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2989550 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         fake_round_id  round_completion_seconds  round_skipped                                                                               black_card_text  black_card_pick_num                                        white_card_text    won  ID_index\n",
       "0                    1                        24          False  Hi MTV! My name is Kendra, I live in Malibu, I'm into _____, and I love to have a good time.                    1  Going inside at some point because of the mosquitoes.  False         1\n",
       "1                    1                        24          False  Hi MTV! My name is Kendra, I live in Malibu, I'm into _____, and I love to have a good time.                    1                                Being fat from noodles.  False         2\n",
       "2                    1                        24          False  Hi MTV! My name is Kendra, I live in Malibu, I'm into _____, and I love to have a good time.                    1                         Letting this loser eat me out.  False         3\n",
       "3                    1                        24          False  Hi MTV! My name is Kendra, I live in Malibu, I'm into _____, and I love to have a good time.                    1                            That chicken from Popeyes.®  False         4\n",
       "4                    1                        24          False  Hi MTV! My name is Kendra, I live in Malibu, I'm into _____, and I love to have a good time.                    1                           A sorry excuse for a father.  False         5\n",
       "...                ...                       ...            ...                                                                                           ...                  ...                                                    ...    ...       ...\n",
       "2989545         298955                      7613          False                                                                Oh my god! _____ killed Kenny!                    1                          Breastfeeding a ten-year-old.  False         6\n",
       "2989546         298955                      7613          False                                                                Oh my god! _____ killed Kenny!                    1                      Happy daddies with happy sandals.  False         7\n",
       "2989547         298955                      7613          False                                                                Oh my god! _____ killed Kenny!                    1             Jerking off to a 10-second RealMedia clip.  False         8\n",
       "2989548         298955                      7613          False                                                                Oh my god! _____ killed Kenny!                    1                Getting naked and watching Nickelodeon.  False         9\n",
       "2989549         298955                      7613          False                                                                Oh my god! _____ killed Kenny!                    1                 Awesome pictures of planets and stuff.  False        10\n",
       "\n",
       "[2989550 rows x 8 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "#### Original processing of data, before I saved it as a parquet for faster subsequent loading: \n",
    "if LOAD_CSV:\n",
    "    df = pd.read_csv(DATA)\n",
    "    ## remove punct/newlines\n",
    "    df[\"white_card_text\"] = df[\"white_card_text\"].replace(\"\\n\", \" \",regex=True).replace(\"\\t\", \" \",regex=True) # .replace(\"\\r\", \"\")\n",
    "    df[\"black_card_text\"] = df[\"black_card_text\"].replace(\"\\n\", \" \",regex=True).replace(\"\\t\", \" \",regex=True) # .replace(\"\\r\", \"\")\n",
    "\n",
    "    ## add number within group - may not actually be the read order shown on screen!! \n",
    "    df['ID_index'] = df.groupby('fake_round_id').cumcount() + 1\n",
    "\n",
    "    print(df[['black_card_text', 'black_card_pick_num', 'white_card_text',\"fake_round_id\"]].nunique())\n",
    "    print(df.describe())\n",
    "    \n",
    "    ## save to disk in compressed, binary format\n",
    "    df.drop([\"winning_index\"],axis=1).to_parquet('cah_lab_data.parquet')\n",
    "#     df.drop([\"winning_index\"],axis=1).to_feather('cah_lab_data.feather')\n",
    "    \n",
    "else:  # load data saved in above code previously\n",
    "    df = pd.read_parquet('cah_lab_data.parquet')\n",
    "#     df= pd.read_feather('cah_lab_data.feather')\n",
    "\n",
    "if DROP_SKIPPED:\n",
    "    df = df.loc[~df[\"round_skipped\"]]\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "53b2859a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ### could use the winning_index==1 cases instead to filter these (but then misses skipped rounds)\n",
    "# # df[\"double_card\"] = df[\"black_card_text\"].str.count(\"_____\")>1 \n",
    "# df[\"double_card\"] = df[\"black_card_pick_num\"]>1\n",
    "# print(df[\"double_card\"].describe())\n",
    "# df.loc[df[\"double_card\"]].drop_duplicates(\"black_card_text\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a815bf58",
   "metadata": {},
   "source": [
    "##### analyze rounds skipped\n",
    "* at level of round -  5% of rounds skipped due to all combinations being considered \"unfunny\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "554f408c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.058145874797210284"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.drop_duplicates(\"fake_round_id\")[\"round_skipped\"].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d0d9bb6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # df.loc[~df[\"winning_index\"].isna()]\n",
    "# df.loc[df[\"winning_index\"]>=1][[\"black_card_text\"]].drop_duplicates() ## these are all double cards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "61278679",
   "metadata": {},
   "outputs": [],
   "source": [
    "if LOWERCASE_DATA:\n",
    "    df[\"white_card_text\"] = df[\"white_card_text\"].str.lower()\n",
    "    df[\"black_card_text\"] = df[\"black_card_text\"].str.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e8d5b6f",
   "metadata": {},
   "source": [
    "##### Fit prior odds / encoder\n",
    "* Use targetEncoder or WOE\n",
    "* Do IPW either here, or later on data subset? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b91c5193",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    2128.0000\n",
       "mean        0.1061\n",
       "std         0.0528\n",
       "min         0.0037\n",
       "50%         0.0958\n",
       "max         0.3959\n",
       "Name: prior_white, dtype: float64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## for now, fit on just white cards, instead of black, white  (For black, we could just use mean_skipped as prior and multiply)\n",
    "df[\"prior_white\"] = cat_encoder.fit_transform(X=df[\"white_card_text\"],y=df[\"won\"])\n",
    "\n",
    "# df[\"prior_white\"] = cat_encoder.transform(df[\"white_card_text\"]) ## catboost encoder - doesn't have min 0 (for those with 0 cases)\n",
    "df.drop_duplicates([\"white_card_text\"])[\"prior_white\"].describe(percentiles=[])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6373271e",
   "metadata": {},
   "source": [
    "### check white prior as baseline model\n",
    "\n",
    "* 19% win rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "dc0160e6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.21408907695138063"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values([\"prior_white\"],ascending=False).groupby(\"fake_round_id\")[\"won\"].head(1).mean()##19%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f3b5fb2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# df.sort_values([\"prior_white\"],ascending=False).groupby(\"fake_round_id\")[[\"won\",\"prior_white\"]].head(1).corr()\n",
    "## 0.06 correlation\n",
    "\n",
    "# df[[\"won\",\"prior_white\"]].corr() ## in overall data: 0.16 corr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb5eae76",
   "metadata": {},
   "source": [
    "### IPW weighting\n",
    " - Unsure which method is correct, I try my own-ish, based on the first ipw definiton\n",
    "```Def ipw (y_true, y_pred):\n",
    "\t\"\"\"\n",
    "\tinverse probability weighting, for binary target.\n",
    "\tThe weight is 1/P for treated units and 1/(1-P) for untreated units.\n",
    "\t\"\"\"\n",
    "\tIf y_true:\n",
    "\t\tReturn 1/y_pred\n",
    "\tElse:\n",
    "Return 1/(1-y_pred)\n",
    "```\n",
    "```\n",
    "## ALT, faster code for a variable:\n",
    "## https://gist.github.com/BioSciEconomist/826556977841cbf966f03542b2be0c55#file-iptw-demo-py-L85\n",
    "# calculate propensity score weights for ATT (Austin, 2011)\t\n",
    "\tdf['ipw_wt'] = np.where(df['treat']==1, 1,df.ps/(1-df.ps))\n",
    "```\n",
    "https://radiant-brushlands-42789.herokuapp.com/medium.com/swlh/demystifying-the-inverse-probability-weighting-method-b5056ba3a72d\n",
    "IPW can be implemented in two steps:\n",
    "1) Estimates a logit mode to estimate the probability (labelled as P) of being treated.\n",
    "2), one uses the Weighted Least Squares (WLS) to estimate the effect of W on Y. The weight is the inverse of the estimated probability. Specifically, the weight is 1/P for treated units and 1/(1-P) for untreated units.\n",
    "https://towardsdatascience.com/using-machine-learning-metrics-to-evaluate-causal-inference-models-d066f1bb2b7a"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abd29c22",
   "metadata": {},
   "source": [
    "## White (\"Answer\") Cards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "opening-pioneer",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    2128.0000\n",
       "mean        4.9619\n",
       "std         3.2156\n",
       "min         1.0000\n",
       "25%         2.0000\n",
       "50%         4.0000\n",
       "75%         7.0000\n",
       "max        23.0000\n",
       "Name: white_card_text, dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"white_card_text\"].drop_duplicates().str.split().str.len().describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "fluid-penguin",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>picks</th>\n",
       "      <th>pick_opportunities</th>\n",
       "      <th>pick_ratio</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>white_card_text</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>...it is unclear.</th>\n",
       "      <td>97</td>\n",
       "      <td>1576</td>\n",
       "      <td>0.0615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10 Incredible Facts About the Anus.</th>\n",
       "      <td>156</td>\n",
       "      <td>1512</td>\n",
       "      <td>0.1032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10 football players with erections barreling towards you at full speed.</th>\n",
       "      <td>4</td>\n",
       "      <td>34</td>\n",
       "      <td>0.1176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10 football players with erections barrelling towards you at full speed.</th>\n",
       "      <td>207</td>\n",
       "      <td>1465</td>\n",
       "      <td>0.1413</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10,000 Syrian refugees.</th>\n",
       "      <td>204</td>\n",
       "      <td>1531</td>\n",
       "      <td>0.1332</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Your mouth.</th>\n",
       "      <td>174</td>\n",
       "      <td>1600</td>\n",
       "      <td>0.1087</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Your weird brother.</th>\n",
       "      <td>191</td>\n",
       "      <td>1552</td>\n",
       "      <td>0.1231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yummy yummy yummy yummy rat pussy.</th>\n",
       "      <td>261</td>\n",
       "      <td>1548</td>\n",
       "      <td>0.1686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>[Picture of Sean Connery from Zardoz]</th>\n",
       "      <td>84</td>\n",
       "      <td>1520</td>\n",
       "      <td>0.0553</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>marvel</th>\n",
       "      <td>79</td>\n",
       "      <td>1430</td>\n",
       "      <td>0.0552</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2128 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                          picks  pick_opportunities  pick_ratio\n",
       "white_card_text                                                                                                \n",
       "...it is unclear.                                                            97                1576      0.0615\n",
       "10 Incredible Facts About the Anus.                                         156                1512      0.1032\n",
       "10 football players with erections barreling towards you at full speed.       4                  34      0.1176\n",
       "10 football players with erections barrelling towards you at full speed.    207                1465      0.1413\n",
       "10,000 Syrian refugees.                                                     204                1531      0.1332\n",
       "...                                                                         ...                 ...         ...\n",
       "Your mouth.                                                                 174                1600      0.1087\n",
       "Your weird brother.                                                         191                1552      0.1231\n",
       "Yummy yummy yummy yummy rat pussy.                                          261                1548      0.1686\n",
       "[Picture of Sean Connery from Zardoz]                                        84                1520      0.0553\n",
       "marvel                                                                       79                1430      0.0552\n",
       "\n",
       "[2128 rows x 3 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby(by=\"white_card_text\").agg(picks=(\"won\",\"sum\"), pick_opportunities=(\"won\",\"count\"),\n",
    "                                                pick_ratio=(\"won\",\"mean\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "generous-explanation",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "white cards against humanity - aggregated funniest\n",
      "           picks  pick_opportunities  pick_ratio  round_skip_ratio\n",
      "count  2128.0000           2128.0000   2128.0000         2128.0000\n",
      "mean    149.6748           1404.8637      0.1065            0.0589\n",
      "std      87.0771            408.7169      0.0525            0.0135\n",
      "min       0.0000             19.0000      0.0000            0.0000\n",
      "25%      91.7500           1490.0000      0.0673            0.0537\n",
      "50%     139.0000           1522.0000      0.0958            0.0580\n",
      "75%     203.0000           1552.0000      0.1384            0.0627\n",
      "max     459.0000           3006.0000      0.3571            0.1923\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>picks</th>\n",
       "      <th>pick_opportunities</th>\n",
       "      <th>pick_ratio</th>\n",
       "      <th>round_skip_ratio</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>white_card_text</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Syphilis.</th>\n",
       "      <td>10</td>\n",
       "      <td>28</td>\n",
       "      <td>0.3571</td>\n",
       "      <td>0.0357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Anal.</th>\n",
       "      <td>459</td>\n",
       "      <td>1572</td>\n",
       "      <td>0.2920</td>\n",
       "      <td>0.0337</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sluts, whores, and bitches.</th>\n",
       "      <td>456</td>\n",
       "      <td>1598</td>\n",
       "      <td>0.2854</td>\n",
       "      <td>0.0538</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Tentacle porn.</th>\n",
       "      <td>434</td>\n",
       "      <td>1525</td>\n",
       "      <td>0.2846</td>\n",
       "      <td>0.0656</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Incest.</th>\n",
       "      <td>435</td>\n",
       "      <td>1530</td>\n",
       "      <td>0.2843</td>\n",
       "      <td>0.0471</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A big black dick.</th>\n",
       "      <td>425</td>\n",
       "      <td>1500</td>\n",
       "      <td>0.2833</td>\n",
       "      <td>0.0513</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Cock.</th>\n",
       "      <td>429</td>\n",
       "      <td>1524</td>\n",
       "      <td>0.2815</td>\n",
       "      <td>0.0413</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>The death penalty.</th>\n",
       "      <td>10</td>\n",
       "      <td>36</td>\n",
       "      <td>0.2778</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Gay thoughts.</th>\n",
       "      <td>418</td>\n",
       "      <td>1523</td>\n",
       "      <td>0.2745</td>\n",
       "      <td>0.0571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Three dicks at the same time.</th>\n",
       "      <td>411</td>\n",
       "      <td>1507</td>\n",
       "      <td>0.2727</td>\n",
       "      <td>0.0498</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>The Asshole Destroyer 3000.</th>\n",
       "      <td>416</td>\n",
       "      <td>1543</td>\n",
       "      <td>0.2696</td>\n",
       "      <td>0.0570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dem titties.</th>\n",
       "      <td>414</td>\n",
       "      <td>1542</td>\n",
       "      <td>0.2685</td>\n",
       "      <td>0.0571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Masturbating.</th>\n",
       "      <td>434</td>\n",
       "      <td>1617</td>\n",
       "      <td>0.2684</td>\n",
       "      <td>0.0427</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Horny Christian teens.</th>\n",
       "      <td>386</td>\n",
       "      <td>1464</td>\n",
       "      <td>0.2637</td>\n",
       "      <td>0.0567</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Double penetration.</th>\n",
       "      <td>392</td>\n",
       "      <td>1488</td>\n",
       "      <td>0.2634</td>\n",
       "      <td>0.0511</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Masturbating to ISIS beheading videos.</th>\n",
       "      <td>10</td>\n",
       "      <td>38</td>\n",
       "      <td>0.2632</td>\n",
       "      <td>0.0526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Crazy anal orgasms.</th>\n",
       "      <td>410</td>\n",
       "      <td>1559</td>\n",
       "      <td>0.2630</td>\n",
       "      <td>0.0507</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>COVID-19.</th>\n",
       "      <td>411</td>\n",
       "      <td>1578</td>\n",
       "      <td>0.2605</td>\n",
       "      <td>0.0513</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Elf cum.</th>\n",
       "      <td>403</td>\n",
       "      <td>1551</td>\n",
       "      <td>0.2598</td>\n",
       "      <td>0.0548</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Joe Biden.</th>\n",
       "      <td>9</td>\n",
       "      <td>35</td>\n",
       "      <td>0.2571</td>\n",
       "      <td>0.0571</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        picks  pick_opportunities  pick_ratio  round_skip_ratio\n",
       "white_card_text                                                                                \n",
       "Syphilis.                                  10                  28      0.3571            0.0357\n",
       "Anal.                                     459                1572      0.2920            0.0337\n",
       "Sluts, whores, and bitches.               456                1598      0.2854            0.0538\n",
       "Tentacle porn.                            434                1525      0.2846            0.0656\n",
       "Incest.                                   435                1530      0.2843            0.0471\n",
       "A big black dick.                         425                1500      0.2833            0.0513\n",
       "Cock.                                     429                1524      0.2815            0.0413\n",
       "The death penalty.                         10                  36      0.2778            0.0000\n",
       "Gay thoughts.                             418                1523      0.2745            0.0571\n",
       "Three dicks at the same time.             411                1507      0.2727            0.0498\n",
       "The Asshole Destroyer 3000.               416                1543      0.2696            0.0570\n",
       "Dem titties.                              414                1542      0.2685            0.0571\n",
       "Masturbating.                             434                1617      0.2684            0.0427\n",
       "Horny Christian teens.                    386                1464      0.2637            0.0567\n",
       "Double penetration.                       392                1488      0.2634            0.0511\n",
       "Masturbating to ISIS beheading videos.     10                  38      0.2632            0.0526\n",
       "Crazy anal orgasms.                       410                1559      0.2630            0.0507\n",
       "COVID-19.                                 411                1578      0.2605            0.0513\n",
       "Elf cum.                                  403                1551      0.2598            0.0548\n",
       "Joe Biden.                                  9                  35      0.2571            0.0571"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"white cards against humanity - aggregated funniest\")\n",
    "df_white = df.groupby(by=\"white_card_text\").agg(picks=(\"won\",\"sum\"), pick_opportunities=(\"won\",\"count\"),\n",
    "                                                pick_ratio=(\"won\",\"mean\"),round_skip_ratio=(\"round_skipped\",\"mean\")).sort_values(\"pick_ratio\",ascending=False)\n",
    "\n",
    "print(df_white.describe())\n",
    "df_white.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "regional-northeast",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11 0 picks\n",
      "32 0-1\n",
      "106 2-5 picks\n",
      "          picks  pick_opportunities  pick_ratio  round_skip_ratio\n",
      "count  2128.000            2128.000    2128.000          2128.000\n",
      "mean    149.675            1404.864       0.106             0.059\n",
      "std      87.077             408.717       0.053             0.013\n",
      "min       0.000              19.000       0.000             0.000\n",
      "25%      91.750            1490.000       0.067             0.054\n",
      "50%     139.000            1522.000       0.096             0.058\n",
      "75%     203.000            1552.000       0.138             0.063\n",
      "max     459.000            3006.000       0.357             0.192\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAQf0lEQVR4nO3db4xcZ3XH8e/BCcGKKbGbZOraVtdVraoOWwKsXKRU1ZqgxmCE86KpjAJyJFd+E1RQLdF1kVrxwpKpFEQFROqKoG6VwNbij2wlosgyHaFKDSaGgOMENwZvU2PLFpAAi5DLmtMXc4Mn9oxnvDOT8T7z/UirufeZ59575mTz27t374wjM5EkleU1wy5AktR/hrskFchwl6QCGe6SVCDDXZIKdMOwCwC49dZbc2xsbNHb/+IXv+Dmm2/uX0FLlH1osA8N9uGSUntx9OjRH2Xmba2euy7CfWxsjKeeemrR29frdSYnJ/tX0BJlHxrsQ4N9uKTUXkTE/7R7zssyklQgw12SCmS4S1KBDHdJKpDhLkkFMtwlqUCGuyQVyHCXpAIZ7pJUoOviHaq6NmNTT7Qc3z2+wANtnuuXuX1bB7p/Sf3hmbskFchwl6QCGe6SVKCuwj0i5iLiWEQ8HRFPVWOrIuJQRDxfPa5smr8nIk5GxImIuGdQxUuSWruWM/fNmXlnZk5U61PA4czcAByu1omIjcB24A5gC/BwRCzrY82SpA56uSyzDZiplmeAe5vGZzPzQmaeAk4Cm3o4jiTpGkVmdp4UcQp4EUjgnzNzOiJeysxbmua8mJkrI+JTwJOZ+Wg1/gjwlcz8wmX73AXsAqjVam+dnZ1d9IuYn59nxYoVi95+qTn2w5+2HK8th3O/HOyxx9e8YbAH6INR+35oxz5cUmovNm/efLTpasordHuf+12ZeSYibgcORcT3rjI3Woxd8RMkM6eBaYCJiYns5V9JKfVfWWmn3b3su8cXeOjYYN+6MHf/5ED33w+j9v3Qjn24ZBR70dVlmcw8Uz2eB75M4zLLuYhYDVA9nq+mnwbWNW2+FjjTr4IlSZ11DPeIuDkiXv/yMvDnwDPAQWBHNW0HcKBaPghsj4ibImI9sAE40u/CJUntdfM7fA34ckS8PP9zmfnvEfFNYH9E7AReAO4DyMzjEbEfeBZYAB7MzIsDqV6S1FLHcM/MHwBvajH+Y+DuNtvsBfb2XJ0kaVF8h6okFchwl6QCGe6SVCDDXZIKZLhLUoEMd0kqkOEuSQUy3CWpQIa7JBXIcJekAhnuklQgw12SCmS4S1KBDHdJKpDhLkkFMtwlqUCGuyQVyHCXpAIZ7pJUIMNdkgpkuEtSgQx3SSqQ4S5JBTLcJalAhrskFchwl6QCGe6SVCDDXZIKZLhLUoFuGHYBWlrGpp4YynHn9m0dynGlpcozd0kqUNfhHhHLIuLbEfF4tb4qIg5FxPPV48qmuXsi4mREnIiIewZRuCSpvWs5c/8g8FzT+hRwODM3AIerdSJiI7AduAPYAjwcEcv6U64kqRtdhXtErAW2Ap9pGt4GzFTLM8C9TeOzmXkhM08BJ4FNfalWktSVbv+g+gngw8Drm8ZqmXkWIDPPRsTt1fga4MmmeaersVeIiF3ALoBarUa9Xr+mwpvNz8/3tP1Ss3t8oeV4bXn755a6a/nvO2rfD+3Yh0tGsRcdwz0i3g2cz8yjETHZxT6jxVheMZA5DUwDTExM5ORkN7turV6v08v2S80Dbe5Y2T2+wEPHyrwBau7+ya7njtr3Qzv24ZJR7EU3SXAX8J6IeBfwOuC3IuJR4FxErK7O2lcD56v5p4F1TduvBc70s2hJ0tV1vOaemXsyc21mjtH4Q+nXMvN9wEFgRzVtB3CgWj4IbI+ImyJiPbABONL3yiVJbfXyO/w+YH9E7AReAO4DyMzjEbEfeBZYAB7MzIs9VypJ6to1hXtm1oF6tfxj4O428/YCe3usTZK0SL5DVZIKZLhLUoEMd0kqkOEuSQUy3CWpQIa7JBXIcJekAhnuklQgw12SCmS4S1KBDHdJKpDhLkkFMtwlqUCGuyQVyHCXpAIZ7pJUIMNdkgpkuEtSgQx3SSqQ4S5JBTLcJalAhrskFchwl6QCGe6SVKAbhl3AUjY29cSwS5Ckljxzl6QCGe6SVCDDXZIKZLhLUoEMd0kqkOEuSQXqGO4R8bqIOBIR34mI4xHx0Wp8VUQciojnq8eVTdvsiYiTEXEiIu4Z5AuQJF2pmzP3C8DbM/NNwJ3Aloh4GzAFHM7MDcDhap2I2AhsB+4AtgAPR8SyAdQuSWqjY7hnw3y1emP1lcA2YKYanwHurZa3AbOZeSEzTwEngU39LFqSdHWRmZ0nNc68jwJ/AHw6M/82Il7KzFua5ryYmSsj4lPAk5n5aDX+CPCVzPzCZfvcBewCqNVqb52dnV30i5ifn2fFihWL3n6xjv3wp6/6Ma+mthzO/XLYVQzG+Jo3dD13WN8P1xv7cEmpvdi8efPRzJxo9VxXHz+QmReBOyPiFuDLEfHGq0yPVrtosc9pYBpgYmIiJycnuymlpXq9Ti/bL9YD19nHD+weX+ChY2V+osTc/ZNdzx3W98P1xj5cMoq9uKa7ZTLzJaBO41r6uYhYDVA9nq+mnQbWNW22FjjTa6GSpO51PM2LiNuAX2XmSxGxHHgH8DHgILAD2Fc9Hqg2OQh8LiI+DvwusAE4MoDaNUKu5UPado8v9PW3qrl9W/u2L+nV0s3v8KuBmeq6+2uA/Zn5eET8F7A/InYCLwD3AWTm8YjYDzwLLAAPVpd1JEmvko7hnpnfBd7cYvzHwN1tttkL7O25OknSovgOVUkqkOEuSQUy3CWpQIa7JBXIcJekAhnuklQgw12SCmS4S1KBDHdJKpDhLkkFMtwlqUCGuyQVyHCXpAIZ7pJUIMNdkgpkuEtSgQx3SSqQ4S5JBTLcJalAhrskFchwl6QCGe6SVCDDXZIKZLhLUoEMd0kqkOEuSQUy3CWpQIa7JBXIcJekAhnuklQgw12SCnRDpwkRsQ74V+B3gF8D05n5TxGxCvg3YAyYA/4yM1+sttkD7AQuAn+dmV8dSPWVYz/8KQ9MPTHIQ0jSktLNmfsCsDsz/wh4G/BgRGwEpoDDmbkBOFytUz23HbgD2AI8HBHLBlG8JKm1juGemWcz81vV8s+B54A1wDZgppo2A9xbLW8DZjPzQmaeAk4Cm/pctyTpKjpelmkWEWPAm4FvALXMPAuNHwARcXs1bQ3wZNNmp6uxy/e1C9gFUKvVqNfr11r7b9SWw+7xhUVvXwr70NDvPvTyvTlM8/PzS7b2fhvFXnQd7hGxAvgi8KHM/FlEtJ3aYiyvGMicBqYBJiYmcnJysttSrvDJxw7w0LFr+jlVpN3jC/aB/vdh7v7Jvu3r1VSv1+nl/6uSjGIvurpbJiJupBHsj2Xml6rhcxGxunp+NXC+Gj8NrGvafC1wpj/lSpK60THco3GK/gjwXGZ+vOmpg8COankHcKBpfHtE3BQR64ENwJH+lSxJ6qSb313vAt4PHIuIp6uxvwP2AfsjYifwAnAfQGYej4j9wLM07rR5MDMv9rtwSVJ7HcM9M/+T1tfRAe5us81eYG8PdUmSeuA7VCWpQIa7JBXIcJekAnlTtNTB2JA+t2hu39ahHFdl8MxdkgpkuEtSgQx3SSqQ4S5JBTLcJalAhrskFchwl6QCGe6SVCDDXZIKZLhLUoEMd0kqkOEuSQUy3CWpQIa7JBXIcJekAhnuklQgw12SCmS4S1KBDHdJKpDhLkkFMtwlqUCGuyQVyHCXpALdMOwCJLU2NvVET9vvHl/ggUXuY27f1p6OreHzzF2SCmS4S1KBDHdJKlDHcI+Iz0bE+Yh4pmlsVUQciojnq8eVTc/tiYiTEXEiIu4ZVOGSpPa6OXP/F2DLZWNTwOHM3AAcrtaJiI3AduCOapuHI2JZ36qVJHWlY7hn5teBn1w2vA2YqZZngHubxmcz80JmngJOApv6U6okqVuLvRWylplnATLzbETcXo2vAZ5smne6GrtCROwCdgHUajXq9foiS4Ha8sZtX6POPjTYh4Ze+tDL/4/Xo/n5+eJeUyf9vs89Woxlq4mZOQ1MA0xMTOTk5OSiD/rJxw7w0DFv2d89vmAfsA8v66UPc/dP9reYIavX6/SSMUvRYu+WORcRqwGqx/PV+GlgXdO8tcCZxZcnSVqMxYb7QWBHtbwDONA0vj0iboqI9cAG4EhvJUqSrlXH39ki4vPAJHBrRJwG/gHYB+yPiJ3AC8B9AJl5PCL2A88CC8CDmXlxQLVLktroGO6Z+d42T93dZv5eYG8vRUmSeuM7VCWpQIa7JBXIcJekAhnuklQgw12SCmS4S1KBDHdJKpDhLkkFMtwlqUCGuyQVyHCXpAIZ7pJUIMNdkgpkuEtSgQx3SSqQ4S5JBTLcJalA/hPxkq4wNvXEUI47t2/rUI5bIs/cJalAhrskFchwl6QCGe6SVCDDXZIK5N0ykq4bg7pLZ/f4Ag9cZd8l3qXjmbskFchwl6QCGe6SVCDDXZIKZLhLUoEMd0kqkLdCShp5w/qgNBjcbZgDO3OPiC0RcSIiTkbE1KCOI0m60kDCPSKWAZ8G3glsBN4bERsHcSxJ0pUGdea+CTiZmT/IzP8DZoFtAzqWJOkykZn932nEXwBbMvOvqvX3A3+SmR9omrML2FWt/iFwoodD3gr8qIftS2EfGuxDg324pNRe/F5m3tbqiUH9QTVajL3ip0hmTgPTfTlYxFOZOdGPfS1l9qHBPjTYh0tGsReDuixzGljXtL4WODOgY0mSLjOocP8msCEi1kfEa4HtwMEBHUuSdJmBXJbJzIWI+ADwVWAZ8NnMPD6IY1X6cnmnAPahwT402IdLRq4XA/mDqiRpuPz4AUkqkOEuSQVa0uE+Sh9xEBGfjYjzEfFM09iqiDgUEc9XjyubnttT9eVERNwznKr7LyLWRcR/RMRzEXE8Ij5YjY9iL14XEUci4jtVLz5ajY9cL6DxzviI+HZEPF6tj2QffiMzl+QXjT/Ufh/4feC1wHeAjcOua4Cv98+AtwDPNI39IzBVLU8BH6uWN1b9uAlYX/Vp2bBfQ5/6sBp4S7X8euC/q9c7ir0IYEW1fCPwDeBto9iL6vX9DfA54PFqfST78PLXUj5zH6mPOMjMrwM/uWx4GzBTLc8A9zaNz2bmhcw8BZyk0a8lLzPPZua3quWfA88BaxjNXmRmzlerN1ZfyQj2IiLWAluBzzQNj1wfmi3lcF8D/G/T+ulqbJTUMvMsNEIPuL0aH4neRMQY8GYaZ6wj2YvqUsTTwHngUGaOai8+AXwY+HXT2Cj24TeWcrh3/IiDEVZ8byJiBfBF4EOZ+bOrTW0xVkwvMvNiZt5J413gmyLijVeZXmQvIuLdwPnMPNrtJi3GlnwfLreUw92POIBzEbEaoHo8X40X3ZuIuJFGsD+WmV+qhkeyFy/LzJeAOrCF0evFXcB7ImKOxuXZt0fEo4xeH15hKYe7H3HQeL07quUdwIGm8e0RcVNErAc2AEeGUF/fRUQAjwDPZebHm54axV7cFhG3VMvLgXcA32PEepGZezJzbWaO0ciBr2Xm+xixPlxh2H/R7eULeBeNuyW+D3xk2PUM+LV+HjgL/IrGmcdO4LeBw8Dz1eOqpvkfqfpyAnjnsOvvYx/+lMav0N8Fnq6+3jWivfhj4NtVL54B/r4aH7leNL2+SS7dLTOyfchMP35Akkq0lC/LSJLaMNwlqUCGuyQVyHCXpAIZ7pJUIMNdkgpkuEtSgf4fmlZHn7MnDcEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(df_white.loc[df_white[\"picks\"]==0].shape[0], \"0 picks\")\n",
    "print(df_white.loc[df_white[\"picks\"]<2].shape[0], \"0-1\")\n",
    "print(df_white.loc[df_white[\"picks\"].between(2,5)].shape[0], \"2-5 picks\")\n",
    "\n",
    "print(df_white.describe().round(3))\n",
    "\n",
    "df_white[\"picks\"].hist();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3912a8ce",
   "metadata": {},
   "source": [
    "Just using global \"picks\" as a feature is a leak, and will have 100% precision for a few of the cards that were never picked.\n",
    "* We could correct this better, using a proper encoder, e.g. target encoding + nested CV : https://contrib.scikit-learn.org/category_encoders/targetencoder.html\n",
    "    * https://brendanhasz.github.io/2019/03/04/target-encoding\n",
    "    * https://contrib.scikit-learn.org/category_encoders/woe.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "looking-senior",
   "metadata": {},
   "source": [
    "## Black (\"Question\") Cards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "burning-there",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BLACK cards against humanity - aggregated funniest\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>picks</th>\n",
       "      <th>pick_opportunities</th>\n",
       "      <th>pick_ratio</th>\n",
       "      <th>round_skip_ratio</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>black_card_text</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Problem: _____. Solution: _____. (PICK 2)</th>\n",
       "      <td>14</td>\n",
       "      <td>70</td>\n",
       "      <td>0.2000</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>You: _____.   Me, an intellectual: _____. (PICK 2)</th>\n",
       "      <td>10</td>\n",
       "      <td>50</td>\n",
       "      <td>0.2000</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>In Jordan Peele's new thriller, a young family discovers that _____ had really been _____ all along.</th>\n",
       "      <td>206</td>\n",
       "      <td>1070</td>\n",
       "      <td>0.1925</td>\n",
       "      <td>0.0374</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>_____ is a slippery slope that leads to _____.</th>\n",
       "      <td>1300</td>\n",
       "      <td>6790</td>\n",
       "      <td>0.1915</td>\n",
       "      <td>0.0427</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>_____ would be woefully incomplete without _____.</th>\n",
       "      <td>1334</td>\n",
       "      <td>6970</td>\n",
       "      <td>0.1914</td>\n",
       "      <td>0.0430</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Instead of playing a card this round, everyone must stare at the Card Czar while making a sound you'd make after tasting something delicious.</th>\n",
       "      <td>462</td>\n",
       "      <td>6550</td>\n",
       "      <td>0.0705</td>\n",
       "      <td>0.2947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>I am so, so sorry everyone. I take full responsibility for _____.</th>\n",
       "      <td>2</td>\n",
       "      <td>30</td>\n",
       "      <td>0.0667</td>\n",
       "      <td>0.3333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>I hate to be that guy, but I just don't see the point of _____.</th>\n",
       "      <td>2</td>\n",
       "      <td>30</td>\n",
       "      <td>0.0667</td>\n",
       "      <td>0.3333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Up on the shore they work all day Out in the sun they slave away While we devotin' Full time to _____ Under the sea!</th>\n",
       "      <td>2</td>\n",
       "      <td>30</td>\n",
       "      <td>0.0667</td>\n",
       "      <td>0.3333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>As Gregor Samsa awoke one morning from uneasy dreams, he found that he was _____.</th>\n",
       "      <td>3</td>\n",
       "      <td>50</td>\n",
       "      <td>0.0600</td>\n",
       "      <td>0.4000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>650 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                               picks  pick_opportunities  pick_ratio  round_skip_ratio\n",
       "black_card_text                                                                                                                                                                                       \n",
       "Problem: _____. Solution: _____. (PICK 2)                                                                                                         14                  70      0.2000            0.0000\n",
       "You: _____.   Me, an intellectual: _____. (PICK 2)                                                                                                10                  50      0.2000            0.0000\n",
       "In Jordan Peele's new thriller, a young family discovers that _____ had really been _____ all along.                                             206                1070      0.1925            0.0374\n",
       "_____ is a slippery slope that leads to _____.                                                                                                  1300                6790      0.1915            0.0427\n",
       "_____ would be woefully incomplete without _____.                                                                                               1334                6970      0.1914            0.0430\n",
       "...                                                                                                                                              ...                 ...         ...               ...\n",
       "Instead of playing a card this round, everyone must stare at the Card Czar while making a sound you'd make after tasting something delicious.    462                6550      0.0705            0.2947\n",
       "I am so, so sorry everyone. I take full responsibility for _____.                                                                                  2                  30      0.0667            0.3333\n",
       "I hate to be that guy, but I just don't see the point of _____.                                                                                    2                  30      0.0667            0.3333\n",
       "Up on the shore they work all day Out in the sun they slave away While we devotin' Full time to _____ Under the sea!                               2                  30      0.0667            0.3333\n",
       "As Gregor Samsa awoke one morning from uneasy dreams, he found that he was _____.                                                                  3                  50      0.0600            0.4000\n",
       "\n",
       "[650 rows x 4 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"BLACK cards against humanity - aggregated funniest\")\n",
    "## pick doesn't necessarily make sense here\n",
    "df_black = df.groupby(by=\"black_card_text\").agg(picks=(\"won\",\"sum\"), pick_opportunities=(\"won\",\"count\"),\n",
    "                                                pick_ratio=(\"won\",\"mean\"),round_skip_ratio=(\"round_skipped\",\"mean\"))\n",
    "df_black.sort_values([\"pick_ratio\",\"round_skip_ratio\"],ascending=False,inplace=True)\n",
    "\n",
    "df_black"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "vertical-geology",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>picks</th>\n",
       "      <th>pick_opportunities</th>\n",
       "      <th>pick_ratio</th>\n",
       "      <th>round_skip_ratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>650.00</td>\n",
       "      <td>650.00</td>\n",
       "      <td>650.00</td>\n",
       "      <td>650.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>490.01</td>\n",
       "      <td>4599.31</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>356.67</td>\n",
       "      <td>2850.48</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.00</td>\n",
       "      <td>10.00</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>86.25</td>\n",
       "      <td>932.50</td>\n",
       "      <td>0.09</td>\n",
       "      <td>0.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>592.00</td>\n",
       "      <td>6275.00</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>624.00</td>\n",
       "      <td>6550.00</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1392.00</td>\n",
       "      <td>7430.00</td>\n",
       "      <td>0.20</td>\n",
       "      <td>0.40</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         picks  pick_opportunities  pick_ratio  round_skip_ratio\n",
       "count   650.00              650.00      650.00            650.00\n",
       "mean    490.01             4599.31        0.10              0.06\n",
       "std     356.67             2850.48        0.03              0.05\n",
       "min       1.00               10.00        0.06              0.00\n",
       "25%      86.25              932.50        0.09              0.04\n",
       "50%     592.00             6275.00        0.10              0.05\n",
       "75%     624.00             6550.00        0.10              0.07\n",
       "max    1392.00             7430.00        0.20              0.40"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_black.describe().round(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "serious-natural",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    650.0000\n",
       "mean      11.2954\n",
       "std        5.2317\n",
       "min        2.0000\n",
       "25%        7.0000\n",
       "50%       11.0000\n",
       "75%       14.0000\n",
       "max       41.0000\n",
       "Name: black_card_text, dtype: float64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"black_card_text\"].drop_duplicates().str.split().str.len().describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6796e852",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ### without pick 2 cards:  All variance in (black cards) pick_ratio is solely due to round skipping\n",
    "# df.loc[(df[\"black_card_pick_num\"]<2) & (~df[\"round_skipped\"])].groupby(by=\"black_card_text\").agg(picks=(\"won\",\"sum\"), pick_opportunities=(\"won\",\"count\"),\n",
    "#                                                 pick_ratio=(\"won\",\"mean\"),round_skip_ratio=(\"round_skipped\",\"mean\")).describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d2cc33b",
   "metadata": {},
   "source": [
    "### Filter data\n",
    "\n",
    "* drop pick 2  cards\n",
    "* Drop rounds where no card picked (ALT: could be alternate target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "355e9a19",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2446360\n",
      "fake_round_id               244636\n",
      "round_completion_seconds      1665\n",
      "round_skipped                    1\n",
      "black_card_text                583\n",
      "black_card_pick_num              1\n",
      "white_card_text               2128\n",
      "won                              2\n",
      "ID_index                        10\n",
      "prior_white                  11398\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "if DROP_DOUBLES:\n",
    "    df = df.loc[df[\"black_card_pick_num\"]<2]\n",
    "\n",
    "    \n",
    "print(df.shape[0])\n",
    "print(df.nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae88f054",
   "metadata": {},
   "source": [
    "## prior white baseline\n",
    "* check Acc@1 if sorting by baseline/mean win level for whites"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4c84e86",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8d886248",
   "metadata": {},
   "source": [
    "### round level sample dataset\n",
    "* downsample + change class freq \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f2439805",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200000\n",
      "(120000, 11)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fake_round_id</th>\n",
       "      <th>round_completion_seconds</th>\n",
       "      <th>round_skipped</th>\n",
       "      <th>black_card_text</th>\n",
       "      <th>black_card_pick_num</th>\n",
       "      <th>white_card_text</th>\n",
       "      <th>won</th>\n",
       "      <th>ID_index</th>\n",
       "      <th>prior_white</th>\n",
       "      <th>ipw_wt</th>\n",
       "      <th>index</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>72201</th>\n",
       "      <td>287366</td>\n",
       "      <td>27</td>\n",
       "      <td>False</td>\n",
       "      <td>My fellow Americans: Before this decade is out, we will have _____ on the moon!</td>\n",
       "      <td>1</td>\n",
       "      <td>Repression.</td>\n",
       "      <td>False</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0816</td>\n",
       "      <td>1.0888</td>\n",
       "      <td>2873652.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6872</th>\n",
       "      <td>282917</td>\n",
       "      <td>13</td>\n",
       "      <td>False</td>\n",
       "      <td>Now in bookstores: \"The Audacity of _____,\" by Barack Obama.</td>\n",
       "      <td>1</td>\n",
       "      <td>Teaching a robot to love.</td>\n",
       "      <td>True</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0623</td>\n",
       "      <td>16.0433</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117609</th>\n",
       "      <td>298355</td>\n",
       "      <td>23</td>\n",
       "      <td>False</td>\n",
       "      <td>Hey, kids. I'm Connor's dad, but you can call me Mr. _____.</td>\n",
       "      <td>1</td>\n",
       "      <td>A mopey zoo lion.</td>\n",
       "      <td>False</td>\n",
       "      <td>7</td>\n",
       "      <td>0.0429</td>\n",
       "      <td>1.0449</td>\n",
       "      <td>2983546.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73554</th>\n",
       "      <td>287700</td>\n",
       "      <td>5</td>\n",
       "      <td>False</td>\n",
       "      <td>This Friday at the Liquid Lounge, it's _____ Night! Ladies drink free.</td>\n",
       "      <td>1</td>\n",
       "      <td>The Hellman's Mayonnaise Corporation.</td>\n",
       "      <td>False</td>\n",
       "      <td>10</td>\n",
       "      <td>0.0757</td>\n",
       "      <td>1.0819</td>\n",
       "      <td>2876999.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45688</th>\n",
       "      <td>280759</td>\n",
       "      <td>4</td>\n",
       "      <td>False</td>\n",
       "      <td>Military historians remember Alexander the Great for his brilliant use of _____ against the Persians.</td>\n",
       "      <td>1</td>\n",
       "      <td>Two contrasting monologues--one classical, one contemporary.</td>\n",
       "      <td>False</td>\n",
       "      <td>10</td>\n",
       "      <td>0.0320</td>\n",
       "      <td>1.0330</td>\n",
       "      <td>2807589.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97779</th>\n",
       "      <td>293482</td>\n",
       "      <td>7</td>\n",
       "      <td>False</td>\n",
       "      <td>Dammit, Gary. You can't just solve every problem with _____.</td>\n",
       "      <td>1</td>\n",
       "      <td>Carrying my poop around in a ziplock bag.</td>\n",
       "      <td>False</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0904</td>\n",
       "      <td>1.0994</td>\n",
       "      <td>2934811.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91335</th>\n",
       "      <td>291950</td>\n",
       "      <td>12</td>\n",
       "      <td>False</td>\n",
       "      <td>Goldfinger! He's the man, the man with _____.</td>\n",
       "      <td>1</td>\n",
       "      <td>Trevor, the world's greatest boyfriend.</td>\n",
       "      <td>False</td>\n",
       "      <td>7</td>\n",
       "      <td>0.0602</td>\n",
       "      <td>1.0641</td>\n",
       "      <td>2919496.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77848</th>\n",
       "      <td>288730</td>\n",
       "      <td>25</td>\n",
       "      <td>False</td>\n",
       "      <td>Click Here for _____!!!</td>\n",
       "      <td>1</td>\n",
       "      <td>Slender toes begging to be sucked.</td>\n",
       "      <td>False</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0830</td>\n",
       "      <td>1.0906</td>\n",
       "      <td>2887291.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95833</th>\n",
       "      <td>293024</td>\n",
       "      <td>19</td>\n",
       "      <td>False</td>\n",
       "      <td>The fault, dear Brutus, is not in our stars, but in _____.</td>\n",
       "      <td>1</td>\n",
       "      <td>Getting Mommy another beer.</td>\n",
       "      <td>False</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0751</td>\n",
       "      <td>1.0813</td>\n",
       "      <td>2930232.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119135</th>\n",
       "      <td>298740</td>\n",
       "      <td>3</td>\n",
       "      <td>False</td>\n",
       "      <td>The Five Stages of Grief: denial, anger, bargaining, _____, acceptance.</td>\n",
       "      <td>1</td>\n",
       "      <td>A man who is so cool that he rides on a motorcycle.</td>\n",
       "      <td>False</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0388</td>\n",
       "      <td>1.0404</td>\n",
       "      <td>2987391.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12163</th>\n",
       "      <td>289440</td>\n",
       "      <td>5</td>\n",
       "      <td>False</td>\n",
       "      <td>A remarkable new study shows that chimps have evolved their own primitive version of _____.</td>\n",
       "      <td>1</td>\n",
       "      <td>Emotions.</td>\n",
       "      <td>True</td>\n",
       "      <td>3</td>\n",
       "      <td>0.1412</td>\n",
       "      <td>7.0833</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        fake_round_id  round_completion_seconds  round_skipped                                                                                        black_card_text  black_card_pick_num                                               white_card_text    won  ID_index  prior_white   ipw_wt      index\n",
       "72201          287366                        27          False                        My fellow Americans: Before this decade is out, we will have _____ on the moon!                    1                                                   Repression.  False         3       0.0816   1.0888  2873652.0\n",
       "6872           282917                        13          False                                           Now in bookstores: \"The Audacity of _____,\" by Barack Obama.                    1                                     Teaching a robot to love.   True         6       0.0623  16.0433        NaN\n",
       "117609         298355                        23          False                                            Hey, kids. I'm Connor's dad, but you can call me Mr. _____.                    1                                             A mopey zoo lion.  False         7       0.0429   1.0449  2983546.0\n",
       "73554          287700                         5          False                                 This Friday at the Liquid Lounge, it's _____ Night! Ladies drink free.                    1                         The Hellman's Mayonnaise Corporation.  False        10       0.0757   1.0819  2876999.0\n",
       "45688          280759                         4          False  Military historians remember Alexander the Great for his brilliant use of _____ against the Persians.                    1  Two contrasting monologues--one classical, one contemporary.  False        10       0.0320   1.0330  2807589.0\n",
       "97779          293482                         7          False                                           Dammit, Gary. You can't just solve every problem with _____.                    1                     Carrying my poop around in a ziplock bag.  False         2       0.0904   1.0994  2934811.0\n",
       "91335          291950                        12          False                                                          Goldfinger! He's the man, the man with _____.                    1                       Trevor, the world's greatest boyfriend.  False         7       0.0602   1.0641  2919496.0\n",
       "77848          288730                        25          False                                                                                Click Here for _____!!!                    1                            Slender toes begging to be sucked.  False         2       0.0830   1.0906  2887291.0\n",
       "95833          293024                        19          False                                             The fault, dear Brutus, is not in our stars, but in _____.                    1                                   Getting Mommy another beer.  False         3       0.0751   1.0813  2930232.0\n",
       "119135         298740                         3          False                                The Five Stages of Grief: denial, anger, bargaining, _____, acceptance.                    1           A man who is so cool that he rides on a motorcycle.  False         2       0.0388   1.0404  2987391.0\n",
       "12163          289440                         5          False            A remarkable new study shows that chimps have evolved their own primitive version of _____.                    1                                                     Emotions.   True         3       0.1412   7.0833        NaN"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sample = df.sort_values(\"fake_round_id\")\n",
    "\n",
    "df_sample = df_sample.loc[df[\"black_card_pick_num\"]<2]\n",
    "df_sample = df_sample.loc[~df[\"round_skipped\"]]\n",
    "\n",
    "df_sample['ipw_wt'] = np.where(df_sample['won']>0, 1/df_sample[\"prior_white\"],1/(1-df_sample[\"prior_white\"]))\n",
    "\n",
    "# df_sample = df_sample.iloc[0:300000]\n",
    "df_sample = df_sample.tail(200000)\n",
    "## filter groups without whole game\n",
    "df_sample = df_sample[df_sample['fake_round_id'].groupby(df_sample['fake_round_id']).transform('size')==10]\n",
    "print(df_sample.shape[0])\n",
    "# print(df_sample.tail(11)[\"fake_round_id\"])\n",
    "\n",
    "## downsample data\n",
    "df1 = df_sample.loc[df_sample[\"won\"]==True]\n",
    "df2 = df_sample.loc[df_sample[\"won\"]==False].groupby(\"fake_round_id\").sample(n=5).reset_index()\n",
    "df_sample = pd.concat([df1,df2],ignore_index=True).sample(frac=1).drop_duplicates()\n",
    "print(df_sample.shape)\n",
    "df_sample.tail(11)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "fc219c82",
   "metadata": {},
   "outputs": [],
   "source": [
    "# df_sample[['fake_round_id', 'round_completion_seconds',  'black_card_text', 'white_card_text', 'won', 'ID_index', 'prior_white', 'ipw_wt']]\\\n",
    "# .to_csv(\"cah_rounds_180k_downsampled_v1.csv.gz\",index=False,compression=\"gzip\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "golden-drilling",
   "metadata": {},
   "source": [
    "### Analyze pairs\n",
    "* Note cases with few occurrences imbalance the data\n",
    "* This is pairs that actually were shown to users (i.e #picks >= 1) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "aggressive-promise",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Winning pair combinations - aggregated funniest\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>picks</th>\n",
       "      <th>pick_opportunities</th>\n",
       "      <th>pick_ratio</th>\n",
       "      <th>round_skip_ratio</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>white_card_text</th>\n",
       "      <th>black_card_text</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Problematic depictions of Asian characters.</th>\n",
       "      <th>In Ireland, people express their grief through the traditional practice of _____.</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Silence.</th>\n",
       "      <th>I have solved politics. My solution is _____.</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Explaining how vaginas work.</th>\n",
       "      <th>Excuse me, straight man, but _____ isn't for you, STRAIGHT MAN.</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A toothless woman gumming on a peach.</th>\n",
       "      <th>Get ready for the movie of the summer! One cop plays by the book. The other's only interested in one thing: _____.</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A bong rip so massive it restores justice to the kingdom.</th>\n",
       "      <th>And would you like those buffalo wings mild, hot, or _____?</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">Freaky, pan-dimensional sex with a demigod.</th>\n",
       "      <th>According to the world's oldest woman, the secret to a long life is _____.</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>After I'm done with _____, it's over for you hoes.</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>After four platinum albums and three Grammys, it's time to get back to my roots, to what inspired me to make music in the first place: _____.</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ain't it nifty? Barb and Bob hit 50! So get off your ass and raise a glass to 50 years of _____.</th>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>marvel</th>\n",
       "      <th>✮✩✩✩✩ Do NOT go here! Found _____ in my fettuccine alfredo!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>786569 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                         picks  pick_opportunities  pick_ratio  round_skip_ratio\n",
       "white_card_text                                           black_card_text                                                                                                                                                                                       \n",
       "Problematic depictions of Asian characters.               In Ireland, people express their grief through the traditional practice of _____.                                                                  1                   1         1.0               0.0\n",
       "Silence.                                                  I have solved politics. My solution is _____.                                                                                                      1                   1         1.0               0.0\n",
       "Explaining how vaginas work.                              Excuse me, straight man, but _____ isn't for you, STRAIGHT MAN.                                                                                    1                   1         1.0               0.0\n",
       "A toothless woman gumming on a peach.                     Get ready for the movie of the summer! One cop plays by the book. The other's only interested in one thing: _____.                                 1                   1         1.0               0.0\n",
       "A bong rip so massive it restores justice to the kingdom. And would you like those buffalo wings mild, hot, or _____?                                                                                        1                   1         1.0               0.0\n",
       "...                                                                                                                                                                                                        ...                 ...         ...               ...\n",
       "Freaky, pan-dimensional sex with a demigod.               According to the world's oldest woman, the secret to a long life is _____.                                                                         0                   1         0.0               0.0\n",
       "                                                          After I'm done with _____, it's over for you hoes.                                                                                                 0                   1         0.0               0.0\n",
       "                                                          After four platinum albums and three Grammys, it's time to get back to my roots, to what inspired me to make music in the first place: _____.      0                   3         0.0               0.0\n",
       "                                                          Ain't it nifty? Barb and Bob hit 50! So get off your ass and raise a glass to 50 years of _____.                                                   0                   6         0.0               0.0\n",
       "marvel                                                    ✮✩✩✩✩ Do NOT go here! Found _____ in my fettuccine alfredo!                                                                                        0                   1         0.0               0.0\n",
       "\n",
       "[786569 rows x 4 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Winning pair combinations - aggregated funniest\")\n",
    "df_pairs = df.groupby(by=[\"white_card_text\",\"black_card_text\"]).agg(picks=(\"won\",\"sum\"), pick_opportunities=(\"won\",\"count\"),\n",
    "                                                pick_ratio=(\"won\",\"mean\"),round_skip_ratio=(\"round_skipped\",\"mean\")).sort_values(\"pick_ratio\",ascending=False)\n",
    "df_pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "0d841f56",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>white_card_text</th>\n",
       "      <th>black_card_text</th>\n",
       "      <th>picks</th>\n",
       "      <th>pick_opportunities</th>\n",
       "      <th>pick_ratio</th>\n",
       "      <th>round_skip_ratio</th>\n",
       "      <th>prior_white</th>\n",
       "      <th>ipw_wt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Problematic depictions of Asian characters.</td>\n",
       "      <td>In Ireland, people express their grief through the traditional practice of _____.</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0575</td>\n",
       "      <td>17.3992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Silence.</td>\n",
       "      <td>I have solved politics. My solution is _____.</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0976</td>\n",
       "      <td>10.2432</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Explaining how vaginas work.</td>\n",
       "      <td>Excuse me, straight man, but _____ isn't for you, STRAIGHT MAN.</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1321</td>\n",
       "      <td>7.5673</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A toothless woman gumming on a peach.</td>\n",
       "      <td>Get ready for the movie of the summer! One cop plays by the book. The other's only interested in one thing: _____.</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1052</td>\n",
       "      <td>9.5036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A bong rip so massive it restores justice to the kingdom.</td>\n",
       "      <td>And would you like those buffalo wings mild, hot, or _____?</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0751</td>\n",
       "      <td>13.3141</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>786564</th>\n",
       "      <td>Freaky, pan-dimensional sex with a demigod.</td>\n",
       "      <td>According to the world's oldest woman, the secret to a long life is _____.</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1552</td>\n",
       "      <td>1.1837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>786565</th>\n",
       "      <td>Freaky, pan-dimensional sex with a demigod.</td>\n",
       "      <td>After I'm done with _____, it's over for you hoes.</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1552</td>\n",
       "      <td>1.1837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>786566</th>\n",
       "      <td>Freaky, pan-dimensional sex with a demigod.</td>\n",
       "      <td>After four platinum albums and three Grammys, it's time to get back to my roots, to what inspired me to make music in the first place: _____.</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1552</td>\n",
       "      <td>1.1837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>786567</th>\n",
       "      <td>Freaky, pan-dimensional sex with a demigod.</td>\n",
       "      <td>Ain't it nifty? Barb and Bob hit 50! So get off your ass and raise a glass to 50 years of _____.</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1552</td>\n",
       "      <td>1.1837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>786568</th>\n",
       "      <td>marvel</td>\n",
       "      <td>✮✩✩✩✩ Do NOT go here! Found _____ in my fettuccine alfredo!</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0558</td>\n",
       "      <td>1.0591</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>786569 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  white_card_text                                                                                                                                black_card_text  picks  pick_opportunities  pick_ratio  round_skip_ratio  prior_white   ipw_wt\n",
       "0                     Problematic depictions of Asian characters.                                                              In Ireland, people express their grief through the traditional practice of _____.      1                   1         1.0               0.0       0.0575  17.3992\n",
       "1                                                        Silence.                                                                                                  I have solved politics. My solution is _____.      1                   1         1.0               0.0       0.0976  10.2432\n",
       "2                                    Explaining how vaginas work.                                                                                Excuse me, straight man, but _____ isn't for you, STRAIGHT MAN.      1                   1         1.0               0.0       0.1321   7.5673\n",
       "3                           A toothless woman gumming on a peach.                             Get ready for the movie of the summer! One cop plays by the book. The other's only interested in one thing: _____.      1                   1         1.0               0.0       0.1052   9.5036\n",
       "4       A bong rip so massive it restores justice to the kingdom.                                                                                    And would you like those buffalo wings mild, hot, or _____?      1                   1         1.0               0.0       0.0751  13.3141\n",
       "...                                                           ...                                                                                                                                            ...    ...                 ...         ...               ...          ...      ...\n",
       "786564                Freaky, pan-dimensional sex with a demigod.                                                                     According to the world's oldest woman, the secret to a long life is _____.      0                   1         0.0               0.0       0.1552   1.1837\n",
       "786565                Freaky, pan-dimensional sex with a demigod.                                                                                             After I'm done with _____, it's over for you hoes.      0                   1         0.0               0.0       0.1552   1.1837\n",
       "786566                Freaky, pan-dimensional sex with a demigod.  After four platinum albums and three Grammys, it's time to get back to my roots, to what inspired me to make music in the first place: _____.      0                   3         0.0               0.0       0.1552   1.1837\n",
       "786567                Freaky, pan-dimensional sex with a demigod.                                               Ain't it nifty? Barb and Bob hit 50! So get off your ass and raise a glass to 50 years of _____.      0                   6         0.0               0.0       0.1552   1.1837\n",
       "786568                                                     marvel                                                                                    ✮✩✩✩✩ Do NOT go here! Found _____ in my fettuccine alfredo!      0                   1         0.0               0.0       0.0558   1.0591\n",
       "\n",
       "[786569 rows x 8 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_pairs = df_pairs.reset_index().merge(df[[\"white_card_text\",\"prior_white\"]].drop_duplicates(\"white_card_text\"),\n",
    "               on=\"white_card_text\",how=\"left\")\n",
    "\n",
    "df_pairs['ipw_wt'] = np.where(df_pairs['picks']>0, 1/df_pairs[\"prior_white\"],1/(1-df_pairs[\"prior_white\"]))\n",
    "df_pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "gentle-terrorism",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>picks</th>\n",
       "      <th>pick_opportunities</th>\n",
       "      <th>pick_ratio</th>\n",
       "      <th>round_skip_ratio</th>\n",
       "      <th>prior_white</th>\n",
       "      <th>ipw_wt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>786569.000</td>\n",
       "      <td>786569.0000</td>\n",
       "      <td>786569.0000</td>\n",
       "      <td>786569.0</td>\n",
       "      <td>786569.0000</td>\n",
       "      <td>786569.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.311</td>\n",
       "      <td>3.1102</td>\n",
       "      <td>0.0997</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1064</td>\n",
       "      <td>3.2709</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.600</td>\n",
       "      <td>1.6911</td>\n",
       "      <td>0.2084</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0515</td>\n",
       "      <td>4.6028</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0037</td>\n",
       "      <td>1.0037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.000</td>\n",
       "      <td>2.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0673</td>\n",
       "      <td>1.0775</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.000</td>\n",
       "      <td>3.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0958</td>\n",
       "      <td>1.1256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.000</td>\n",
       "      <td>4.0000</td>\n",
       "      <td>0.1111</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1372</td>\n",
       "      <td>3.4798</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>8.000</td>\n",
       "      <td>15.0000</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3959</td>\n",
       "      <td>272.1975</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            picks  pick_opportunities   pick_ratio  round_skip_ratio  prior_white       ipw_wt\n",
       "count  786569.000         786569.0000  786569.0000          786569.0  786569.0000  786569.0000\n",
       "mean        0.311              3.1102       0.0997               0.0       0.1064       3.2709\n",
       "std         0.600              1.6911       0.2084               0.0       0.0515       4.6028\n",
       "min         0.000              1.0000       0.0000               0.0       0.0037       1.0037\n",
       "25%         0.000              2.0000       0.0000               0.0       0.0673       1.0775\n",
       "50%         0.000              3.0000       0.0000               0.0       0.0958       1.1256\n",
       "75%         1.000              4.0000       0.1111               0.0       0.1372       3.4798\n",
       "max         8.000             15.0000       1.0000               0.0       0.3959     272.1975"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_pairs.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "executed-pastor",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAD4CAYAAADy46FuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAXb0lEQVR4nO3df6zddZ3n8edrqGJlBIs/blhKtkxo3OVH/EFTmCUxd+1s6UQj/IFJjSN1wqYbwxjdJZnA/NOMhkSTdZjBDGQb6VAYRuiiBqKD2IA3s5Mgv9TZCsjSFRYqHepYRGpWtOx7/zifC4d6+rn33B+9BZ6P5OR8z/v7/XzO+3vT9nW/P85pqgpJkg7nd5a6AUnS0c2gkCR1GRSSpC6DQpLUZVBIkrqWLXUDC+3tb397rVq1as7jf/nLX3LcccctXEMLxL7GY1/jsa/xvBb7evDBB/+lqt4xcmVVvaYeZ599ds3Hd77znXmNXyz2NR77Go99jee12BfwQB3m31VPPUmSugwKSVLXrIIiyVuT3JrkR0keSfL7SU5MsjPJY+15xdD2VyTZneTRJOcP1c9OsqutuzpJWv3YJLe0+r1JVg2N2dTe47EkmxZw3yVJszDbI4q/Ar5VVf8GeDfwCHA5cFdVrQbuaq9JcjqwETgD2ABck+SYNs+1wGZgdXtsaPVLgGer6jTgKuALba4TgS3AOcBaYMtwIEmSFt+MQZHkeOD9wHUAVfXrqvo5cAGwvW22HbiwLV8A3FxVL1TV48BuYG2Sk4Djq+qeduHkhkPGTM91K7CuHW2cD+ysqv1V9Sywk5fDRZJ0BMzm9tjfA34K/E2SdwMPAp8GJqpqL0BV7U3yzrb9ycB3h8bvabXftOVD69NjnmpzHUzyHPC24fqIMS9JspnBkQoTExNMTU3NYrdGO3DgwLzGLxb7Go99jce+xvN662s2QbEMeB/wqaq6N8lf0U4zHUZG1KpTn+uYlwtVW4GtAGvWrKnJyclOe31TU1PMZ/xisa/x2Nd47Gs8r7e+ZnONYg+wp6ruba9vZRAcz7TTSbTnfUPbnzI0fiXwdKuvHFF/xZgky4ATgP2duSRJR8iMQVFV/ww8leRdrbQOeBi4HZi+C2kTcFtbvh3Y2O5kOpXBRev72mmq55Oc264/XHzImOm5LgLubtcx7gTWJ1nRLmKvbzVJ0hEy26/w+BRwU5I3Aj8G/phByOxIcgnwJPARgKp6KMkOBmFyELi0ql5s83wSuB5YDtzRHjC4UH5jkt0MjiQ2trn2J/kccH/b7rNVtX+O+zoru37yHJ+4/JuL+RYjPfH5Dx7x95Sk2ZhVUFTVD4A1I1atO8z2VwJXjqg/AJw5ov4rWtCMWLcN2DabPiVJC89PZkuSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUpdBIUnqMigkSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVLXrIIiyRNJdiX5QZIHWu3EJDuTPNaeVwxtf0WS3UkeTXL+UP3sNs/uJFcnSasfm+SWVr83yaqhMZvaezyWZNOC7bkkaVbGOaL491X1nqpa015fDtxVVauBu9prkpwObATOADYA1yQ5po25FtgMrG6PDa1+CfBsVZ0GXAV8oc11IrAFOAdYC2wZDiRJ0uKbz6mnC4DtbXk7cOFQ/eaqeqGqHgd2A2uTnAQcX1X3VFUBNxwyZnquW4F17WjjfGBnVe2vqmeBnbwcLpKkI2DZLLcr4NtJCvhvVbUVmKiqvQBVtTfJO9u2JwPfHRq7p9V+05YPrU+PearNdTDJc8DbhusjxrwkyWYGRypMTEwwNTU1y936bRPL4bKzDs55/FzN1POBAwfmtV+Lxb7GY1/jsa/xLFZfsw2K86rq6RYGO5P8qLNtRtSqU5/rmJcLg+DaCrBmzZqanJzstNf3pZtu44u7ZvtjWThPfGyyu35qaor57Ndisa/x2Nd47Gs8i9XXrE49VdXT7Xkf8HUG1wueaaeTaM/72uZ7gFOGhq8Enm71lSPqrxiTZBlwArC/M5ck6QiZMSiSHJfkLdPLwHrgh8DtwPRdSJuA29ry7cDGdifTqQwuWt/XTlM9n+Tcdv3h4kPGTM91EXB3u45xJ7A+yYp2EXt9q0mSjpDZnGOZAL7e7mRdBvxdVX0ryf3AjiSXAE8CHwGoqoeS7AAeBg4Cl1bVi22uTwLXA8uBO9oD4DrgxiS7GRxJbGxz7U/yOeD+tt1nq2r/PPZXkjSmGYOiqn4MvHtE/WfAusOMuRK4ckT9AeDMEfVf0YJmxLptwLaZ+pQkLQ4/mS1J6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUpdBIUnqMigkSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldsw6KJMck+X6Sb7TXJybZmeSx9rxiaNsrkuxO8miS84fqZyfZ1dZdnSStfmySW1r93iSrhsZsau/xWJJNC7LXkqRZG+eI4tPAI0OvLwfuqqrVwF3tNUlOBzYCZwAbgGuSHNPGXAtsBla3x4ZWvwR4tqpOA64CvtDmOhHYApwDrAW2DAeSJGnxzSookqwEPgh8eah8AbC9LW8HLhyq31xVL1TV48BuYG2Sk4Djq+qeqirghkPGTM91K7CuHW2cD+ysqv1V9Sywk5fDRZJ0BCyb5XZ/Cfwp8Jah2kRV7QWoqr1J3tnqJwPfHdpuT6v9pi0fWp8e81Sb62CS54C3DddHjHlJks0MjlSYmJhgampqlrv12yaWw2VnHZzz+LmaqecDBw7Ma78Wi32Nx77GY1/jWay+ZgyKJB8C9lXVg0kmZzFnRtSqU5/rmJcLVVuBrQBr1qypycnZtDnal266jS/umm1+LpwnPjbZXT81NcV89mux2Nd47Gs89jWexeprNqeezgM+nOQJ4GbgA0n+FnimnU6iPe9r2+8BThkavxJ4utVXjqi/YkySZcAJwP7OXJKkI2TGoKiqK6pqZVWtYnCR+u6q+iPgdmD6LqRNwG1t+XZgY7uT6VQGF63va6epnk9ybrv+cPEhY6bnuqi9RwF3AuuTrGgXsde3miTpCJnPOZbPAzuSXAI8CXwEoKoeSrIDeBg4CFxaVS+2MZ8ErgeWA3e0B8B1wI1JdjM4ktjY5tqf5HPA/W27z1bV/nn0LEka01hBUVVTwFRb/hmw7jDbXQlcOaL+AHDmiPqvaEEzYt02YNs4fUqSFo6fzJYkdRkUkqQug0KS1GVQSJK6DApJUpdBIUnqMigkSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqSuGYMiyZuS3Jfkn5I8lOTPW/3EJDuTPNaeVwyNuSLJ7iSPJjl/qH52kl1t3dVJ0urHJrml1e9NsmpozKb2Ho8l2bSgey9JmtFsjiheAD5QVe8G3gNsSHIucDlwV1WtBu5qr0lyOrAROAPYAFyT5Jg217XAZmB1e2xo9UuAZ6vqNOAq4AttrhOBLcA5wFpgy3AgSZIW34xBUQMH2ss3tEcBFwDbW307cGFbvgC4uapeqKrHgd3A2iQnAcdX1T1VVcANh4yZnutWYF072jgf2FlV+6vqWWAnL4eLJOkIWDabjdoRwYPAacBfV9W9SSaqai9AVe1N8s62+cnAd4eG72m137TlQ+vTY55qcx1M8hzwtuH6iDHD/W1mcKTCxMQEU1NTs9mtkSaWw2VnHZzz+LmaqecDBw7Ma78Wi32Nx77GY1/jWay+ZhUUVfUi8J4kbwW+nuTMzuYZNUWnPtcxw/1tBbYCrFmzpiYnJzvt9X3pptv44q5Z/VgW1BMfm+yun5qaYj77tVjsazz2NR77Gs9i9TXWXU9V9XNgisHpn2fa6STa87622R7glKFhK4GnW33liPorxiRZBpwA7O/MJUk6QmZz19M72pEESZYDfwD8CLgdmL4LaRNwW1u+HdjY7mQ6lcFF6/vaaarnk5zbrj9cfMiY6bkuAu5u1zHuBNYnWdEuYq9vNUnSETKbcywnAdvbdYrfAXZU1TeS3APsSHIJ8CTwEYCqeijJDuBh4CBwaTt1BfBJ4HpgOXBHewBcB9yYZDeDI4mNba79ST4H3N+2+2xV7Z/PDkuSxjNjUFTV/wTeO6L+M2DdYcZcCVw5ov4A8FvXN6rqV7SgGbFuG7Btpj4lSYvDT2ZLkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUpdBIUnqMigkSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklS14xBkeSUJN9J8kiSh5J8utVPTLIzyWPtecXQmCuS7E7yaJLzh+pnJ9nV1l2dJK1+bJJbWv3eJKuGxmxq7/FYkk0LuveSpBnN5ojiIHBZVf1b4Fzg0iSnA5cDd1XVauCu9pq2biNwBrABuCbJMW2ua4HNwOr22NDqlwDPVtVpwFXAF9pcJwJbgHOAtcCW4UCSJC2+GYOiqvZW1ffa8vPAI8DJwAXA9rbZduDCtnwBcHNVvVBVjwO7gbVJTgKOr6p7qqqAGw4ZMz3XrcC6drRxPrCzqvZX1bPATl4OF0nSEbBsnI3bKaH3AvcCE1W1FwZhkuSdbbOTge8ODdvTar9py4fWp8c81eY6mOQ54G3D9RFjhvvazOBIhYmJCaampsbZrVeYWA6XnXVwzuPnaqaeDxw4MK/9Wiz2NR77Go99jWex+pp1UCT5XeCrwGeq6hft8sLITUfUqlOf65iXC1Vbga0Aa9asqcnJycP1NqMv3XQbX9w1Vn4uiCc+NtldPzU1xXz2a7HY13jsazz2NZ7F6mtWdz0leQODkLipqr7Wys+000m0532tvgc4ZWj4SuDpVl85ov6KMUmWAScA+ztzSZKOkNnc9RTgOuCRqvqLoVW3A9N3IW0Cbhuqb2x3Mp3K4KL1fe001fNJzm1zXnzImOm5LgLubtcx7gTWJ1nRLmKvbzVJ0hEym3Ms5wEfB3Yl+UGr/RnweWBHkkuAJ4GPAFTVQ0l2AA8zuGPq0qp6sY37JHA9sBy4oz1gEEQ3JtnN4EhiY5trf5LPAfe37T5bVfvntquSpLmYMSiq6h8Zfa0AYN1hxlwJXDmi/gBw5oj6r2hBM2LdNmDbTH1KkhaHn8yWJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1HfmvSdVIqy7/Znf9ZWcd5BMzbDNXT3z+g4syr6TXBo8oJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUpdBIUnqmjEokmxLsi/JD4dqJybZmeSx9rxiaN0VSXYneTTJ+UP1s5PsauuuTpJWPzbJLa1+b5JVQ2M2tfd4LMmmBdtrSdKszeaI4npgwyG1y4G7qmo1cFd7TZLTgY3AGW3MNUmOaWOuBTYDq9tjes5LgGer6jTgKuALba4TgS3AOcBaYMtwIEmSjowZg6Kq/gHYf0j5AmB7W94OXDhUv7mqXqiqx4HdwNokJwHHV9U9VVXADYeMmZ7rVmBdO9o4H9hZVfur6llgJ78dWJKkRTbXaxQTVbUXoD2/s9VPBp4a2m5Pq53clg+tv2JMVR0EngPe1plLknQELVvg+TKiVp36XMe88k2TzQxOazExMcHU1NSMjR7OxHK47KyDcx6/WBazr/n8vA4cODCv8YvFvsZjX+N5vfU116B4JslJVbW3nVba1+p7gFOGtlsJPN3qK0fUh8fsSbIMOIHBqa49wOQhY6ZGNVNVW4GtAGvWrKnJyclRm83Kl266jS/uWuj8nL/Lzjq4aH098bHJOY+dmppiPj/vxWJf47Gv8bze+prrqafbgem7kDYBtw3VN7Y7mU5lcNH6vnZ66vkk57brDxcfMmZ6rouAu9t1jDuB9UlWtIvY61tNknQEzfgrapKvMPjN/u1J9jC4E+nzwI4klwBPAh8BqKqHkuwAHgYOApdW1Yttqk8yuINqOXBHewBcB9yYZDeDI4mNba79ST4H3N+2+2xVHXpRXZK0yGYMiqr66GFWrTvM9lcCV46oPwCcOaL+K1rQjFi3Ddg2U4+SpMXjJ7MlSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdc34f2brtW/V5d+c89jLzjrIJ+Y4/onPf3DO7yvpyPGIQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdb0qgiLJhiSPJtmd5PKl7keSXk+O+g/cJTkG+GvgPwB7gPuT3F5VDy9tZ5qv+XzQbya9DwL6QT9pPEd9UABrgd1V9WOAJDcDFwAGhV51dv3kuTl/kn0+DEfNR6pqqXvoSnIRsKGq/mN7/XHgnKr6k6FtNgOb28t3AY/O4y3fDvzLPMYvFvsaj32Nx77G81rs619X1TtGrXg1HFFkRO0V6VZVW4GtC/JmyQNVtWYh5lpI9jUe+xqPfY3n9dbXq+Fi9h7glKHXK4Gnl6gXSXrdeTUExf3A6iSnJnkjsBG4fYl7kqTXjaP+1FNVHUzyJ8CdwDHAtqp6aBHfckFOYS0C+xqPfY3HvsbzuurrqL+YLUlaWq+GU0+SpCVkUEiSugyK5mj9mpAk25LsS/LDpe5lWpJTknwnySNJHkry6aXuCSDJm5Lcl+SfWl9/vtQ9DUtyTJLvJ/nGUvcyLMkTSXYl+UGSB5a6n2lJ3prk1iQ/an/Wfv8o6Old7ec0/fhFks8cBX395/Zn/odJvpLkTQs6v9coXvqakP/F0NeEAB89Gr4mJMn7gQPADVV15lL3A5DkJOCkqvpekrcADwIXLvXPK0mA46rqQJI3AP8IfLqqvruUfU1L8l+ANcDxVfWhpe5nWpIngDVVdVR9gCzJduB/VNWX2x2Pb66qny9xWy9p/278hMEHgP/PEvZxMoM/66dX1f9NsgP4+6q6fqHewyOKgZe+JqSqfg1Mf03IkquqfwD2L3Ufw6pqb1V9ry0/DzwCnLy0XUENHGgv39AeR8VvQklWAh8EvrzUvbwaJDkeeD9wHUBV/fpoColmHfC/lzIkhiwDlidZBryZBf6smUExcDLw1NDrPRwF//C9GiRZBbwXuHeJWwFeOr3zA2AfsLOqjoq+gL8E/hT4f0vcxygFfDvJg+3rcI4Gvwf8FPibdrruy0mOW+qmDrER+MpSN1FVPwH+K/AksBd4rqq+vZDvYVAMzPg1IfptSX4X+Crwmar6xVL3A1BVL1bVexh8gn9tkiU/XZfkQ8C+qnpwqXs5jPOq6n3AHwKXttOdS20Z8D7g2qp6L/BL4Gi6dvhG4MPAfz8KelnB4AzIqcC/Ao5L8kcL+R4GxYBfEzKmdg3gq8BNVfW1pe7nUO00xRSwYWk7AeA84MPtWsDNwAeS/O3StvSyqnq6Pe8Dvs7gVOxS2wPsGToivJVBcBwt/hD4XlU9s9SNAH8APF5VP62q3wBfA/7dQr6BQTHg14SMoV00vg54pKr+Yqn7mZbkHUne2paXM/gL9KMlbQqoqiuqamVVrWLwZ+vuqlrQ3/jmKslx7YYE2qmd9cCS32FXVf8MPJXkXa20jqPrvxb4KEfBaafmSeDcJG9ufzfXMbhuuGCO+q/wOBKW4GtCZi3JV4BJ4O1J9gBbquq6pe2K84CPA7va9QCAP6uqv1+6lgA4Cdje7kb5HWBHVR1Vt6IehSaArw/+fWEZ8HdV9a2lbeklnwJuar+8/Rj44yXuB4Akb2Zwh+R/WupeAKrq3iS3At8DDgLfZ4G/ysPbYyVJXZ56kiR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXf8fqo9uTfK45HEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_pairs[\"picks\"].hist();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "instructional-marijuana",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAD4CAYAAADy46FuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAX9UlEQVR4nO3df5DcdX3H8efLxGIEwQTkGpNMj5bUCqSC3IS0TDun0SQVx+AMTM+hcIzpxGGw1c51atCZxsKkE6YiLWNJG02agNSQiTpk1IjX4I7jDAQCRY4Qaa6SwpGUqBcxZwvl8N0/9nOyOXY/u7d3t3tfeD1mdva77+/389n33l3yuu+PvVVEYGZmVssb2t2AmZnNbA4KMzPLclCYmVmWg8LMzLIcFGZmljW73Q1MtbPOOis6Ozvb3cZJfvGLX3Dqqae2u42GFanfIvUKxeq3SL1Csfqdib0+/PDDP4mIt1Vb95oLis7OTvbv39/uNk5SKpXo7u5udxsNK1K/ReoVitVvkXqFYvU7E3uV9F+11vnQk5mZZTkozMwsy0FhZmZZDgozM8tyUJiZWZaDwszMshwUZmaW5aAwM7MsB4WZmWW95t6ZPVmd67455XP2LRnl2jrzHt542ZQ/r5nZVPAehZmZZTkozMwsy0FhZmZZDgozM8tyUJiZWZaDwszMshwUZmaW5aAwM7MsB4WZmWU5KMzMLMtBYWZmWXWDQtKbJD0o6QeSDkj6m1SfJ6lf0qF0P7dizA2SBiU9KWllRf1iSQNp3W2SlOqnSLo71fdJ6qwY05ue45Ck3il99WZmVlcjexQvAu+NiHcBFwKrJC0D1gF7I2IxsDc9RtJ5QA9wPrAKuF3SrDTXJmAtsDjdVqX6GuB4RJwL3ArcnOaaB6wHLgGWAusrA8nMzKZf3aCIspH08I3pFsBqYHuqbwcuT8urgR0R8WJEPAUMAkslzQdOj4j7IyKAO8aNGZtrF7A87W2sBPojYjgijgP9vBIuZmbWAg2do5A0S9KjwDHK/3HvAzoi4ihAuj87bb4AeKZi+FCqLUjL4+snjYmIUeB54MzMXGZm1iINfR5FRLwMXCjprcDXJV2Q2VzVpsjUmx3zyhNKaykf0qKjo4NSqZRpL69vyWjTY2vpmFN/3sn0PNVGRkZmVD85ReoVitVvkXqFYvVbpF5hgh9cFBE/k1SifPjnOUnzI+JoOqx0LG02BCyqGLYQOJLqC6vUK8cMSZoNnAEMp3r3uDGlKn1tBjYDdHV1RXd39/hNGlbvA4aa0bdklFsG8l/qw1d1T/nzNqtUKjGZr2ErFalXKFa/ReoVitVvkXqFxq56elvak0DSHOB9wA+B3cDYVUi9wD1peTfQk65kOofySesH0+GpE5KWpfMP14wbMzbXFcB96TzGvcAKSXPTSewVqWZmZi3SyB7FfGB7unLpDcDOiPiGpPuBnZLWAE8DVwJExAFJO4EngFHg+nToCuA6YBswB9iTbgBbgDslDVLek+hJcw1Lugl4KG13Y0QMT+YFm5nZxNQNioh4DLioSv2nwPIaYzYAG6rU9wOvOr8RES+QgqbKuq3A1np9mpnZ9PA7s83MLMtBYWZmWQ4KMzPLclCYmVmWg8LMzLIcFGZmluWgMDOzLAeFmZllOSjMzCzLQWFmZlkOCjMzy3JQmJlZloPCzMyyHBRmZpbloDAzsywHhZmZZTkozMwsy0FhZmZZDgozM8tyUJiZWZaDwszMshwUZmaW5aAwM7OsukEhaZGk70o6KOmApE+k+mclPSvp0XT7QMWYGyQNSnpS0sqK+sWSBtK62yQp1U+RdHeq75PUWTGmV9KhdOud0ldvZmZ1zW5gm1GgLyIekfQW4GFJ/WndrRHxucqNJZ0H9ADnA28H/k3Sb0fEy8AmYC3wAPAtYBWwB1gDHI+IcyX1ADcDfyxpHrAe6AIiPffuiDg+uZdtZmaNqrtHERFHI+KRtHwCOAgsyAxZDeyIiBcj4ilgEFgqaT5wekTcHxEB3AFcXjFme1reBSxPexsrgf6IGE7h0E85XMzMrEUmdI4iHRK6CNiXSh+X9JikrZLmptoC4JmKYUOptiAtj6+fNCYiRoHngTMzc5mZWYs0cugJAEmnAV8FPhkRP5e0CbiJ8iGhm4BbgI8CqjI8MnWaHFPZ21rKh7To6OigVCplX0tO35LRpsfW0jGn/ryT6XmqjYyMzKh+corUKxSr3yL1CsXqt0i9QoNBIemNlEPiroj4GkBEPFex/ovAN9LDIWBRxfCFwJFUX1ilXjlmSNJs4AxgONW7x40pje8vIjYDmwG6urqiu7t7/CYNu3bdN5seW0vfklFuGch/qQ9f1T3lz9usUqnEZL6GrVSkXqFY/RapVyhWv0XqFRq76knAFuBgRHy+oj6/YrMPA4+n5d1AT7qS6RxgMfBgRBwFTkhalua8BrinYszYFU1XAPel8xj3AiskzU2HtlakmpmZtUgjexSXAlcDA5IeTbVPAx+RdCHlQ0GHgY8BRMQBSTuBJyhfMXV9uuIJ4DpgGzCH8tVOe1J9C3CnpEHKexI9aa5hSTcBD6XtboyI4WZeqJmZNaduUETE96l+ruBbmTEbgA1V6vuBC6rUXwCurDHXVmBrvT7NzGx6+J3ZZmaW5aAwM7MsB4WZmWU5KMzMLMtBYWZmWQ4KMzPLclCYmVmWg8LMzLIcFGZmluWgMDOzLAeFmZllOSjMzCzLQWFmZlkOCjMzy3JQmJlZloPCzMyyGvrMbHtt6xz3OeF9S0an5bPDxzu88bJpfw4zmzzvUZiZWZaDwszMshwUZmaW5aAwM7MsB4WZmWU5KMzMLKtuUEhaJOm7kg5KOiDpE6k+T1K/pEPpfm7FmBskDUp6UtLKivrFkgbSutskKdVPkXR3qu+T1Fkxpjc9xyFJvVP66s3MrK5G9ihGgb6IeCewDLhe0nnAOmBvRCwG9qbHpHU9wPnAKuB2SbPSXJuAtcDidFuV6muA4xFxLnArcHOaax6wHrgEWAqsrwwkMzObfnWDIiKORsQjafkEcBBYAKwGtqfNtgOXp+XVwI6IeDEingIGgaWS5gOnR8T9ERHAHePGjM21C1ie9jZWAv0RMRwRx4F+XgkXMzNrgQm9MzsdEroI2Ad0RMRRKIeJpLPTZguAByqGDaXaS2l5fH1szDNprlFJzwNnVtarjKnsay3lPRU6OjoolUoTeVkn6Vsy2vTYWjrm1J93Mj1P1vjeGul3KkzFax4ZGWnr126iitRvkXqFYvVbpF5hAkEh6TTgq8AnI+Ln6fRC1U2r1CJTb3bMK4WIzcBmgK6uruju7q7VW13T8acr+paMcstA/kt9+KruKX/eRo1/zY30OxWm4jWXSiUm8/1utSL1W6ReoVj9FqlXaPCqJ0lvpBwSd0XE11L5uXQ4iXR/LNWHgEUVwxcCR1J9YZX6SWMkzQbOAIYzc5mZWYs0ctWTgC3AwYj4fMWq3cDYVUi9wD0V9Z50JdM5lE9aP5gOU52QtCzNec24MWNzXQHcl85j3AuskDQ3ncRekWpmZtYijRxfuBS4GhiQ9GiqfRrYCOyUtAZ4GrgSICIOSNoJPEH5iqnrI+LlNO46YBswB9iTblAOojslDVLek+hJcw1Lugl4KG13Y0QMN/dSzcysGXWDIiK+T/VzBQDLa4zZAGyoUt8PXFCl/gIpaKqs2wpsrdenmZlND78z28zMshwUZmaW5aAwM7MsB4WZmWU5KMzMLMtBYWZmWQ4KMzPLclCYmVmWg8LMzLIcFGZmluWgMDOzrOn/0AFrSOc0fA6GmdlU8B6FmZllOSjMzCzLQWFmZlkOCjMzy3JQmJlZloPCzMyyHBRmZpbloDAzsywHhZmZZTkozMwsy0FhZmZZdYNC0lZJxyQ9XlH7rKRnJT2abh+oWHeDpEFJT0paWVG/WNJAWnebJKX6KZLuTvV9kjorxvRKOpRuvVP2qs3MrGGN7FFsA1ZVqd8aERem27cAJJ0H9ADnpzG3S5qVtt8ErAUWp9vYnGuA4xFxLnArcHOaax6wHrgEWAqslzR3wq/QzMwmpW5QRMT3gOEG51sN7IiIFyPiKWAQWCppPnB6RNwfEQHcAVxeMWZ7Wt4FLE97GyuB/ogYjojjQD/VA8vMzKbRZP7M+MclXQPsB/rSf+YLgAcqthlKtZfS8vg66f4ZgIgYlfQ8cGZlvcqYk0haS3lvhY6ODkqlUtMvqm/JaNNja+mYMz3zTpdW9TuZ79OYkZGRKZmnVYrUb5F6hWL1W6Reofmg2ATcBES6vwX4KKAq20amTpNjTi5GbAY2A3R1dUV3d3em9bxrp+FzIfqWjHLLQHE++qNV/R6+qnvSc5RKJSbz/W61IvVbpF6hWP0WqVdo8qqniHguIl6OiF8CX6R8DgHKv/Uvqth0IXAk1RdWqZ80RtJs4AzKh7pqzWVmZi3UVFCkcw5jPgyMXRG1G+hJVzKdQ/mk9YMRcRQ4IWlZOv9wDXBPxZixK5quAO5L5zHuBVZImptOYq9INTMza6G6xxckfQXoBs6SNET5SqRuSRdSPhR0GPgYQEQckLQTeAIYBa6PiJfTVNdRvoJqDrAn3QC2AHdKGqS8J9GT5hqWdBPwUNruxoho9KS6mZlNkbpBEREfqVLektl+A7ChSn0/cEGV+gvAlTXm2gpsrdejmZlNH78z28zMshwUZmaW5aAwM7MsB4WZmWU5KMzMLMtBYWZmWQ4KMzPLclCYmVmWg8LMzLIcFGZmluWgMDOzLAeFmZllOSjMzCzLQWFmZlkOCjMzy3JQmJlZloPCzMyyHBRmZpbloDAzsywHhZmZZTkozMwsy0FhZmZZDgozM8uqGxSStko6Junxito8Sf2SDqX7uRXrbpA0KOlJSSsr6hdLGkjrbpOkVD9F0t2pvk9SZ8WY3vQchyT1TtmrNjOzhjWyR7ENWDWutg7YGxGLgb3pMZLOA3qA89OY2yXNSmM2AWuBxek2Nuca4HhEnAvcCtyc5poHrAcuAZYC6ysDyczMWqNuUETE94DhceXVwPa0vB24vKK+IyJejIingEFgqaT5wOkRcX9EBHDHuDFjc+0Clqe9jZVAf0QMR8RxoJ9XB5aZmU2z2U2O64iIowARcVTS2am+AHigYruhVHspLY+vj415Js01Kul54MzKepUxJ5G0lvLeCh0dHZRKpSZfFvQtGW16bC0dc6Zn3unSqn4n830aMzIyMiXztEqR+i1Sr1CsfovUKzQfFLWoSi0y9WbHnFyM2AxsBujq6oru7u66jdZy7bpvNj22lr4lo9wyMNVf6unTqn4PX9U96TlKpRKT+X63WpH6LVKvUKx+i9QrNH/V03PpcBLp/liqDwGLKrZbCBxJ9YVV6ieNkTQbOIPyoa5ac5mZWQs1GxS7gbGrkHqBeyrqPelKpnMon7R+MB2mOiFpWTr/cM24MWNzXQHcl85j3AuskDQ3ncRekWpmZtZCdY8vSPoK0A2cJWmI8pVIG4GdktYATwNXAkTEAUk7gSeAUeD6iHg5TXUd5Suo5gB70g1gC3CnpEHKexI9aa5hSTcBD6XtboyI8SfVzcxsmtUNioj4SI1Vy2tsvwHYUKW+H7igSv0FUtBUWbcV2FqvRzMzmz5+Z7aZmWU5KMzMLMtBYWZmWQ4KMzPLclCYmVmWg8LMzLIcFGZmluWgMDOzLAeFmZllOSjMzCzLQWFmZlkOCjMzy3JQmJlZloPCzMyyHBRmZpbloDAzsywHhZmZZTkozMwsy0FhZmZZDgozM8tyUJiZWZaDwszMshwUZmaWNamgkHRY0oCkRyXtT7V5kvolHUr3cyu2v0HSoKQnJa2sqF+c5hmUdJskpfopku5O9X2SOifTr5mZTdxU7FG8JyIujIiu9HgdsDciFgN702MknQf0AOcDq4DbJc1KYzYBa4HF6bYq1dcAxyPiXOBW4OYp6NfMzCZgOg49rQa2p+XtwOUV9R0R8WJEPAUMAkslzQdOj4j7IyKAO8aNGZtrF7B8bG/DzMxaQ+X/m5scLD0FHAcC+OeI2CzpZxHx1optjkfEXElfAB6IiC+n+hZgD3AY2BgR70v1PwA+FREflPQ4sCoihtK6/wQuiYifjOtjLeU9Ejo6Oi7esWNH069p4Nnnmx5bS8cceO5/p3zaadOqfpcsOGPSc4yMjHDaaadNQTetUaR+i9QrFKvfmdjre97znocrjgydZPYk5740Io5IOhvol/TDzLbV9gQiU8+NObkQsRnYDNDV1RXd3d3ZpnOuXffNpsfW0rdklFsGJvulbp1W9Xv4qu5Jz1EqlZjM97vVitRvkXqFYvVbpF5hkoeeIuJIuj8GfB1YCjyXDieR7o+lzYeARRXDFwJHUn1hlfpJYyTNBs4AhifTs5mZTUzTQSHpVElvGVsGVgCPA7uB3rRZL3BPWt4N9KQrmc6hfNL6wYg4CpyQtCydf7hm3Jixua4A7ovJHCszM7MJm8zxhQ7g6+nc8mzgXyPi25IeAnZKWgM8DVwJEBEHJO0EngBGgesj4uU013XANmAO5fMWe1J9C3CnpEHKexI9k+jXzMya0HRQRMSPgHdVqf8UWF5jzAZgQ5X6fuCCKvUXSEFjrz2dU3A+qG/J6ITPKx3eeNmkn9fs9cTvzDYzsywHhZmZZTkozMwsy0FhZmZZDgozM8tyUJiZWZaDwszMshwUZmaW5aAwM7MsB4WZmWU5KMzMLMtBYWZmWQ4KMzPLclCYmVlWcT6f02yKTMWfN2/WtlWntu25zZrlPQozM8tyUJiZWZaDwszMshwUZmaW5aAwM7MsX/Vk1kIDzz7PtW246urwxsta/pz22uE9CjMzy3JQmJlZViGCQtIqSU9KGpS0rt39mJm9nsz4oJA0C/hH4I+A84CPSDqvvV2Zmb1+FOFk9lJgMCJ+BCBpB7AaeKKtXZkVSDN/tqRvyeiUnHj3ifTiU0S0u4csSVcAqyLiT9Pjq4FLIuLjFdusBdamh+8Anmx5o3lnAT9pdxMTUKR+i9QrFKvfIvUKxep3Jvb6GxHxtmorirBHoSq1k9ItIjYDm1vTzsRJ2h8RXe3uo1FF6rdIvUKx+i1Sr1CsfovUKxTgHAUwBCyqeLwQONKmXszMXneKEBQPAYslnSPp14AeYHebezIze92Y8YeeImJU0seBe4FZwNaIONDmtiZqxh4Wq6FI/RapVyhWv0XqFYrVb5F6nfkns83MrL2KcOjJzMzayEFhZmZZDoppJGmRpO9KOijpgKRPtLuneiTNkvTvkr7R7l7qkfRWSbsk/TB9jX+v3T3VIukv0s/A45K+IulN7e6pkqStko5JeryiNk9Sv6RD6X5uO3usVKPfv0s/C49J+rqkt7axxV+p1mvFur+UFJLOakdvjXJQTK9RoC8i3gksA64vwJ8f+QRwsN1NNOgfgG9HxO8A72KG9i1pAfDnQFdEXED5ooye9nb1KtuAVeNq64C9EbEY2JsezxTbeHW//cAFEfG7wH8AN7S6qRq28epekbQIeD/wdKsbmigHxTSKiKMR8UhaPkH5P7IF7e2qNkkLgcuAL7W7l3oknQ78IbAFICL+LyJ+1tam8mYDcyTNBt7MDHsvUER8DxgeV14NbE/L24HLW9lTTrV+I+I7ETGaHj5A+T1XbVfjawtwK/BXjHsD8UzkoGgRSZ3ARcC+NreS8/eUf3B/2eY+GvGbwI+Bf0mHyr4k6dR2N1VNRDwLfI7yb45Hgecj4jvt7aohHRFxFMq/9ABnt7mfifgosKfdTdQi6UPAsxHxg3b30ggHRQtIOg34KvDJiPh5u/upRtIHgWMR8XC7e2nQbODdwKaIuAj4BTPr0MivpGP7q4FzgLcDp0r6k/Z29dol6TOUD/ve1e5eqpH0ZuAzwF+3u5dGOSimmaQ3Ug6JuyLia+3uJ+NS4EOSDgM7gPdK+nJ7W8oaAoYiYmwPbRfl4JiJ3gc8FRE/joiXgK8Bv9/mnhrxnKT5AOn+WJv7qUtSL/BB4KqYuW8S+y3KvzT8IP17Wwg8IunX29pVhoNiGkkS5WPoByPi8+3uJyciboiIhRHRSflE630RMWN/642I/waekfSOVFrOzP3T808DyyS9Of1MLGeGnngfZzfQm5Z7gXva2EtdklYBnwI+FBH/0+5+aomIgYg4OyI607+3IeDd6Wd6RnJQTK9Lgasp/3b+aLp9oN1NvYb8GXCXpMeAC4G/bW871aW9nl3AI8AA5X93M+pPOEj6CnA/8A5JQ5LWABuB90s6RPnqnI3t7LFSjX6/ALwF6E//1v6prU0mNXotFP8JDzMzy/IehZmZZTkozMwsy0FhZmZZDgozM8tyUJiZWZaDwszMshwUZmaW9f/9Cfwq8CxWvgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_pairs[\"pick_opportunities\"].hist();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "floppy-arena",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distrib for pairs that had AT LEAST MIN_PAIR_FREQ (4)\n",
      "             picks  pick_opportunities   pick_ratio  round_skip_ratio  prior_white       ipw_wt\n",
      "count  288246.0000         288246.0000  288246.0000          288246.0  288246.0000  288246.0000\n",
      "mean        0.4963              4.9489       0.1002               0.0       0.1067       4.3101\n",
      "std         0.7579              1.1612       0.1527               0.0       0.0514       5.2487\n",
      "min         0.0000              4.0000       0.0000               0.0       0.0157       1.0159\n",
      "25%         0.0000              4.0000       0.0000               0.0       0.0677       1.0815\n",
      "50%         0.0000              5.0000       0.0000               0.0       0.0962       1.1494\n",
      "75%         1.0000              6.0000       0.2000               0.0       0.1374       6.8285\n",
      "max         8.0000             15.0000       1.0000               0.0       0.2929      63.7603\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>white_card_text</th>\n",
       "      <th>black_card_text</th>\n",
       "      <th>picks</th>\n",
       "      <th>pick_opportunities</th>\n",
       "      <th>pick_ratio</th>\n",
       "      <th>round_skip_ratio</th>\n",
       "      <th>prior_white</th>\n",
       "      <th>ipw_wt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>Existing.</td>\n",
       "      <td>You're not going to believe this, but I'm you from the future! You've got to stop _____.</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1414</td>\n",
       "      <td>7.0708</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>Balls.</td>\n",
       "      <td>My anaconda don't want none unless you got _____, hun.</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1828</td>\n",
       "      <td>5.4709</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>296</th>\n",
       "      <td>Oompa-Loompas.</td>\n",
       "      <td>We're here! We're _____! Get used to it!</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1871</td>\n",
       "      <td>5.3452</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>496</th>\n",
       "      <td>Barely legal boys.</td>\n",
       "      <td>Get ready for the movie of the summer! One cop plays by the book. The other's only interested in one thing: _____.</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1857</td>\n",
       "      <td>5.3855</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>607</th>\n",
       "      <td>Our dumb asshole president.</td>\n",
       "      <td>IF you like _____, YOU MIGHT BE A REDNECK.</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1828</td>\n",
       "      <td>5.4719</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>786545</th>\n",
       "      <td>Freaky, pan-dimensional sex with a demigod.</td>\n",
       "      <td>Blessed are you, Lord our God, creator of the universe, who has granted us _____.</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1552</td>\n",
       "      <td>1.1837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>786547</th>\n",
       "      <td>Freaky, pan-dimensional sex with a demigod.</td>\n",
       "      <td>But before I kill you, Mr. Bond, I must show you _____.</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1552</td>\n",
       "      <td>1.1837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>786551</th>\n",
       "      <td>Freaky, pan-dimensional sex with a demigod.</td>\n",
       "      <td>A recent laboratory study shows that undergraduates have 50% less sex after being exposed to _____.</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1552</td>\n",
       "      <td>1.1837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>786552</th>\n",
       "      <td>Frantically writing equations on a chalkboard.</td>\n",
       "      <td>_____. It's a trap!</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0321</td>\n",
       "      <td>1.0332</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>786567</th>\n",
       "      <td>Freaky, pan-dimensional sex with a demigod.</td>\n",
       "      <td>Ain't it nifty? Barb and Bob hit 50! So get off your ass and raise a glass to 50 years of _____.</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1552</td>\n",
       "      <td>1.1837</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>288246 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       white_card_text                                                                                                     black_card_text  picks  pick_opportunities  pick_ratio  round_skip_ratio  prior_white  ipw_wt\n",
       "68                                           Existing.                            You're not going to believe this, but I'm you from the future! You've got to stop _____.      4                   4         1.0               0.0       0.1414  7.0708\n",
       "148                                             Balls.                                                              My anaconda don't want none unless you got _____, hun.      4                   4         1.0               0.0       0.1828  5.4709\n",
       "296                                     Oompa-Loompas.                                                                            We're here! We're _____! Get used to it!      4                   4         1.0               0.0       0.1871  5.3452\n",
       "496                                 Barely legal boys.  Get ready for the movie of the summer! One cop plays by the book. The other's only interested in one thing: _____.      5                   5         1.0               0.0       0.1857  5.3855\n",
       "607                        Our dumb asshole president.                                                                          IF you like _____, YOU MIGHT BE A REDNECK.      4                   4         1.0               0.0       0.1828  5.4719\n",
       "...                                                ...                                                                                                                 ...    ...                 ...         ...               ...          ...     ...\n",
       "786545     Freaky, pan-dimensional sex with a demigod.                                   Blessed are you, Lord our God, creator of the universe, who has granted us _____.      0                   4         0.0               0.0       0.1552  1.1837\n",
       "786547     Freaky, pan-dimensional sex with a demigod.                                                             But before I kill you, Mr. Bond, I must show you _____.      0                   5         0.0               0.0       0.1552  1.1837\n",
       "786551     Freaky, pan-dimensional sex with a demigod.                 A recent laboratory study shows that undergraduates have 50% less sex after being exposed to _____.      0                   4         0.0               0.0       0.1552  1.1837\n",
       "786552  Frantically writing equations on a chalkboard.                                                                                                 _____. It's a trap!      0                   6         0.0               0.0       0.0321  1.0332\n",
       "786567     Freaky, pan-dimensional sex with a demigod.                    Ain't it nifty? Barb and Bob hit 50! So get off your ass and raise a glass to 50 years of _____.      0                   6         0.0               0.0       0.1552  1.1837\n",
       "\n",
       "[288246 rows x 8 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(f\"Distrib for pairs that had AT LEAST MIN_PAIR_FREQ ({MIN_PAIR_FREQ})\")\n",
    "print(df_pairs.loc[df_pairs[\"pick_opportunities\"]>=MIN_PAIR_FREQ].describe())\n",
    "df_pairs.loc[df_pairs[\"pick_opportunities\"]>=MIN_PAIR_FREQ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "ddc8df0f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "popular pairs / multiple times picked\n",
      "          picks  pick_opportunities  pick_ratio  round_skip_ratio  prior_white    ipw_wt\n",
      "count  38602.00            38602.00    38602.00           38602.0     38602.00  38602.00\n",
      "mean       2.22                4.74        0.52               0.0         0.15      8.12\n",
      "std        0.51                1.67        0.20               0.0         0.06      4.11\n",
      "min        2.00                2.00        0.13               0.0         0.01      3.41\n",
      "25%        2.00                4.00        0.40               0.0         0.10      5.37\n",
      "50%        2.00                5.00        0.50               0.0         0.14      6.98\n",
      "75%        2.00                6.00        0.67               0.0         0.19      9.63\n",
      "max        8.00               15.00        1.00               0.0         0.29    197.11\n",
      "38602\n",
      "nunique black_card_text     439\n",
      "white_card_text    1952\n",
      "dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>black_card_text</th>\n",
       "      <th>white_card_text</th>\n",
       "      <th>picks</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Run, run, as fast as you can! You can't catch me, I'm _____!</td>\n",
       "      <td>COVID-19.</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Hey, honey. For the bedroom, were you thinking \"Robin's Egg Blue\" or \"_____ Pink?\"</td>\n",
       "      <td>My first period.</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hey, honey. For the bedroom, were you thinking \"Robin's Egg Blue\" or \"_____ Pink?\"</td>\n",
       "      <td>Chunks of dead hitchhiker.</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Click Here for _____!!!</td>\n",
       "      <td>Bill Clinton, naked on a bearskin rug with a saxophone.</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Life for American Indians was forever changed when the White Man introduced them to _____.</td>\n",
       "      <td>The right amount of cocaine.</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Dear Abby, I'm having some trouble with _____ and would like your advice.</td>\n",
       "      <td>Fucking a corpse back to life.</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>What's there a ton of in heaven?</td>\n",
       "      <td>Sluts, whores, and bitches.</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>As the mom of five rambunctious boys, I'm no stranger to _____.</td>\n",
       "      <td>Boring vaginal sex.</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>But before I kill you, Mr. Bond, I must show you _____.</td>\n",
       "      <td>A finger up the butt.</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>What's secretly a Russian plot to destabilize the West?</td>\n",
       "      <td>This horrible thing called Twitter.</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Instead of coal, Santa now gives the bad children _____.</td>\n",
       "      <td>Dead parents.</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Goldfinger! He's the man, the man with _____.</td>\n",
       "      <td>Crippling social anxiety.</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                               black_card_text                                          white_card_text  picks\n",
       "0                                 Run, run, as fast as you can! You can't catch me, I'm _____!                                                COVID-19.      8\n",
       "1           Hey, honey. For the bedroom, were you thinking \"Robin's Egg Blue\" or \"_____ Pink?\"                                         My first period.      8\n",
       "2           Hey, honey. For the bedroom, were you thinking \"Robin's Egg Blue\" or \"_____ Pink?\"                               Chunks of dead hitchhiker.      7\n",
       "3                                                                      Click Here for _____!!!  Bill Clinton, naked on a bearskin rug with a saxophone.      7\n",
       "4   Life for American Indians was forever changed when the White Man introduced them to _____.                             The right amount of cocaine.      6\n",
       "5                    Dear Abby, I'm having some trouble with _____ and would like your advice.                           Fucking a corpse back to life.      6\n",
       "6                                                             What's there a ton of in heaven?                              Sluts, whores, and bitches.      6\n",
       "7                              As the mom of five rambunctious boys, I'm no stranger to _____.                                      Boring vaginal sex.      6\n",
       "8                                      But before I kill you, Mr. Bond, I must show you _____.                                    A finger up the butt.      6\n",
       "9                                      What's secretly a Russian plot to destabilize the West?                      This horrible thing called Twitter.      6\n",
       "10                                    Instead of coal, Santa now gives the bad children _____.                                            Dead parents.      6\n",
       "11                                               Goldfinger! He's the man, the man with _____.                                Crippling social anxiety.      6"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"popular pairs / multiple times picked\")\n",
    "df_pop = df_pairs.loc[(df_pairs[\"pick_opportunities\"]>=2) & (df_pairs[\"picks\"]>1)].sort_values(\"picks\",ascending=False)\n",
    "print(df_pop.describe().round(2))\n",
    "print(df_pop.shape[0])\n",
    "print(\"nunique\",df_pop.reset_index()[[\"black_card_text\",\"white_card_text\"]].nunique())\n",
    "df_pop.reset_index()[[\"black_card_text\",\"white_card_text\",\"picks\"]].head(12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "e8aa65f1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['FW: re: FBI WARNING!!! VIOLENT GANGS USING _____ TO RECRUIT CHILDREN!',\n",
       "       'Computer! Display _____ on screen. Enhance.',\n",
       "       \"Seriously guys, there's nothing funny about _____.\",\n",
       "       \"Hi MTV! My name is Kendra, I live in Malibu, I'm into _____, and I love to have a good time.\",\n",
       "       'The secret to a lasting marriage is communication, communication, and _____.',\n",
       "       \"After four platinum albums and three Grammys, it's time to get back to my roots, to what inspired me to make music in the first place: _____.\",\n",
       "       'The blind date was going horribly until we discovered our shared interest in _____.',\n",
       "       'Having tired of poetry and music, the immortal elves now fill their days with _____.',\n",
       "       'Why my mustache smell so bad?',\n",
       "       'What will end racism once and for all?',\n",
       "       '_____: kid-tested, mother-approved.',\n",
       "       'IF you like _____, YOU MIGHT BE A REDNECK.',\n",
       "       'What totally destroyed my asshole?',\n",
       "       'Why my mustache smell so bad?',\n",
       "       'I tell you, it was a non-stop fuckfest. When it was over, my asshole looked like _____.'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# df_pop.loc[df_pop[\"picks\"]>2].reset_index().sort_values([\"white_card_text\",\"picks\"],ascending=False)[[\"white_card_text\",\"black_card_text\"]].head(15).values\n",
    "df_pop.loc[df_pop[\"picks\"]>2].reset_index().sort_values([\"white_card_text\",\"picks\"],ascending=False)[\"black_card_text\"].head(15).values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "pressing-equilibrium",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>white_card_text</th>\n",
       "      <th>black_card_text</th>\n",
       "      <th>picks</th>\n",
       "      <th>pick_opportunities</th>\n",
       "      <th>pick_ratio</th>\n",
       "      <th>round_skip_ratio</th>\n",
       "      <th>prior_white</th>\n",
       "      <th>ipw_wt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>Existing.</td>\n",
       "      <td>You're not going to believe this, but I'm you from the future! You've got to stop _____.</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1414</td>\n",
       "      <td>7.0708</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>Balls.</td>\n",
       "      <td>My anaconda don't want none unless you got _____, hun.</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1828</td>\n",
       "      <td>5.4709</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>296</th>\n",
       "      <td>Oompa-Loompas.</td>\n",
       "      <td>We're here! We're _____! Get used to it!</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1871</td>\n",
       "      <td>5.3452</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>496</th>\n",
       "      <td>Barely legal boys.</td>\n",
       "      <td>Get ready for the movie of the summer! One cop plays by the book. The other's only interested in one thing: _____.</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1857</td>\n",
       "      <td>5.3855</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>607</th>\n",
       "      <td>Our dumb asshole president.</td>\n",
       "      <td>IF you like _____, YOU MIGHT BE A REDNECK.</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1828</td>\n",
       "      <td>5.4719</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 white_card_text                                                                                                     black_card_text  picks  pick_opportunities  pick_ratio  round_skip_ratio  prior_white  ipw_wt\n",
       "68                     Existing.                            You're not going to believe this, but I'm you from the future! You've got to stop _____.      4                   4         1.0               0.0       0.1414  7.0708\n",
       "148                       Balls.                                                              My anaconda don't want none unless you got _____, hun.      4                   4         1.0               0.0       0.1828  5.4709\n",
       "296               Oompa-Loompas.                                                                            We're here! We're _____! Get used to it!      4                   4         1.0               0.0       0.1871  5.3452\n",
       "496           Barely legal boys.  Get ready for the movie of the summer! One cop plays by the book. The other's only interested in one thing: _____.      5                   5         1.0               0.0       0.1857  5.3855\n",
       "607  Our dumb asshole president.                                                                          IF you like _____, YOU MIGHT BE A REDNECK.      4                   4         1.0               0.0       0.1828  5.4719"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_pairs.loc[df_pairs[\"pick_opportunities\"]>=4].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "a37eaa8e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pick_opportunities</th>\n",
       "      <th>picks</th>\n",
       "      <th>pick_ratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>pick_opportunities</th>\n",
       "      <td>1.0000</td>\n",
       "      <td>0.2846</td>\n",
       "      <td>0.0027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>picks</th>\n",
       "      <td>0.2846</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>0.8092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pick_ratio</th>\n",
       "      <td>0.0027</td>\n",
       "      <td>0.8092</td>\n",
       "      <td>1.0000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    pick_opportunities   picks  pick_ratio\n",
       "pick_opportunities              1.0000  0.2846      0.0027\n",
       "picks                           0.2846  1.0000      0.8092\n",
       "pick_ratio                      0.0027  0.8092      1.0000"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_pairs[[\"pick_opportunities\",\"picks\",\"pick_ratio\"]].corr() # ,\"picks_sum\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "wireless-karen",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " >= 2 636457\n",
      " >= 5 155233\n",
      " >= 7 29781\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1     150112\n",
       "2     174899\n",
       "3     173312\n",
       "4     133013\n",
       "5      82712\n",
       "6      42740\n",
       "7      18933\n",
       "8       7232\n",
       "9       2525\n",
       "10       765\n",
       "11       242\n",
       "12        61\n",
       "13        18\n",
       "14         3\n",
       "15         2\n",
       "Name: pick_opportunities, dtype: int64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\" >= 2\",df_pairs.loc[df_pairs[\"pick_opportunities\"]>=2].shape[0])\n",
    "print(\" >= 5\",df_pairs.loc[df_pairs[\"pick_opportunities\"]>=5].shape[0])\n",
    "print(\" >= 7\",df_pairs.loc[df_pairs[\"pick_opportunities\"]>=7].shape[0])\n",
    "df_pairs[\"pick_opportunities\"].value_counts().sort_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "1d7d37e5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    589032\n",
      "1    158935\n",
      "2     31566\n",
      "3      5805\n",
      "4      1031\n",
      "5       176\n",
      "6        20\n",
      "7         2\n",
      "8         2\n",
      "Name: picks, dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0    74.8862\n",
       "1    20.2061\n",
       "2     4.0131\n",
       "3     0.7380\n",
       "4     0.1311\n",
       "5     0.0224\n",
       "6     0.0025\n",
       "7     0.0003\n",
       "8     0.0003\n",
       "Name: picks, dtype: float64"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(df_pairs[\"picks\"].value_counts().sort_index())\n",
    "100*df_pairs[\"picks\"].value_counts(normalize=True).sort_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tired-charleston",
   "metadata": {},
   "source": [
    "# Initial classification target:  Was a card combination ever picked? \n",
    "* Ideally we'd use a higher threshhold, e.g. at least 3-5 instances, but that leaves us with relatively little data. \n",
    "* binary target: was it ever picked, for a given combo. (i.e clip at 1, 0/1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "26b63513",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    786569.0000\n",
       "mean          3.1102\n",
       "std           1.6911\n",
       "min           1.0000\n",
       "25%           2.0000\n",
       "50%           3.0000\n",
       "75%           4.0000\n",
       "max          15.0000\n",
       "Name: pick_opportunities, dtype: float64"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_pairs[\"pick_opportunities\"].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94d2ec93",
   "metadata": {},
   "source": [
    "* We can set the target to be picked at leat once, or picked more than a random uniform average (~10.6%)\n",
    "    * We can use the background uniform distribution, instead of the distrib for \"popularly picked\". "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "33367bf3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "count    288246.0000\n",
      "mean          0.1002\n",
      "std           0.1527\n",
      "min           0.0000\n",
      "25%           0.0000\n",
      "50%           0.0000\n",
      "75%           0.2000\n",
      "max           1.0000\n",
      "Name: pick_ratio, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "df_pairs[\"pick_ratio\"] = df_pairs[\"picks\"].div(df_pairs[\"pick_opportunities\"])\n",
    "# print(\"global pick_ratio\\n\",df_pairs[\"pick_ratio\"].describe())\n",
    "print(df_pairs.loc[df_pairs[\"pick_opportunities\"]>=MIN_PAIR_FREQ][\"pick_ratio\"].describe())\n",
    "\n",
    "MEAN_PICK_RATIO = df_pairs.loc[df_pairs[\"pick_opportunities\"]>=MIN_PAIR_FREQ][\"pick_ratio\"].mean() + 0.005 # .median() #"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "material-parker",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "786569 # rows before filter of MIN_PAIR_FREQ (4)\n",
      "0.3677414430729308 boolean target mean\n",
      "288246 # rows after filter\n",
      "pick_ratio\n",
      " count    288246.0000\n",
      "mean          0.1002\n",
      "std           0.1527\n",
      "min           0.0000\n",
      "25%           0.0000\n",
      "50%           0.0000\n",
      "75%           0.2000\n",
      "max           1.0000\n",
      "Name: pick_ratio, dtype: float64\n",
      "0.3677414430729308 target mean\n"
     ]
    }
   ],
   "source": [
    "print(df_pairs.shape[0],f\"# rows before filter of MIN_PAIR_FREQ ({MIN_PAIR_FREQ})\")\n",
    "df_data = df_pairs.loc[df_pairs[\"pick_opportunities\"]>=MIN_PAIR_FREQ][[\"white_card_text\",\"black_card_text\",\"picks\"\n",
    "                                                                       ,\"pick_opportunities\",\"ipw_wt\"]].copy()\n",
    "df_data[\"pick_ratio\"] = df_data[\"picks\"].div(df_data[\"pick_opportunities\"]) ## target % , instead of boolean\n",
    "df_data[\"picks_sum\"] = df_data[\"picks\"] # copy of raw count of times picked as funny\n",
    "## We can set the target to be picked at leat once, or picked more than a random uniform average (~10.6%)\n",
    "df_data[\"picks\"] = df_data[\"picks\"].clip(upper=1) # Bool target \n",
    "print(df_data[\"picks\"].mean(), \"boolean target mean\")\n",
    "\n",
    "# ## OPTIONAL: change to \"picked more than most\" target\n",
    "# df_data[\"picks\"] = (df_data[\"pick_ratio\"] > MEAN_PICK_RATIO).astype(int)\n",
    "\n",
    "print(df_data.shape[0],\"# rows after filter\")\n",
    "print(\"pick_ratio\\n\",df_data[\"pick_ratio\"].describe())\n",
    "print(df_data[\"picks\"].mean(), \"target mean\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "f1f5401b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prior_white_paired_reg_ratio</th>\n",
       "      <th>pick_ratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1964.0000</td>\n",
       "      <td>1964.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.0998</td>\n",
       "      <td>0.6306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.0502</td>\n",
       "      <td>0.1692</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.0118</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.0895</td>\n",
       "      <td>0.6000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>0.2929</td>\n",
       "      <td>1.0000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       prior_white_paired_reg_ratio  pick_ratio\n",
       "count                     1964.0000   1964.0000\n",
       "mean                         0.0998      0.6306\n",
       "std                          0.0502      0.1692\n",
       "min                          0.0118      0.0000\n",
       "50%                          0.0895      0.6000\n",
       "max                          0.2929      1.0000"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### add mean picks ratio, for this subset of data (regression). May not represent just white card data??? \n",
    "## for now, fit on just white cards, instead of black, white  (For black, we could just use mean_skipped as prior and multiply)\n",
    "cat_encoder.fit(X=df_data[\"white_card_text\"],y=df_data[\"pick_ratio\"])\n",
    "\n",
    "df_data[\"prior_white_paired_reg_ratio\"] = cat_encoder.transform(df_data[\"white_card_text\"]) ## catboost encoder - doesn't have min 0 (for those with 0 cases)\n",
    "df_data.drop_duplicates([\"white_card_text\"])[[\"prior_white_paired_reg_ratio\",\"pick_ratio\"]].describe(percentiles=[])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "opposed-miami",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(288246, 13)\n"
     ]
    }
   ],
   "source": [
    "## add total count of black, white cards\n",
    "df_data = df_data.merge(df_black[[\"pick_opportunities\",\"round_skip_ratio\"]].rename(columns={\"pick_opportunities\":\"black_counts\",\n",
    "                                                                                 \"round_skip_ratio\":\"black_round_skip_ratio\"}),  on=\"black_card_text\")\n",
    "\n",
    "df_data = df_data.merge(df_white[[\"pick_opportunities\",\"picks\",\"pick_ratio\"]].rename({\"pick_opportunities\":\"white_counts\",\n",
    "                                                                       \"picks\":\"white_picks\",\"pick_ratio\":\"prior_white_pick_ratio\"},axis=1),on=\"white_card_text\")\n",
    "\n",
    "# df_data.reset_index(inplace=True)\n",
    "print(df_data.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "e23bec8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_data[\"pick_ratio-sub-prior_white_pick_ratio\"] = df_data[\"pick_ratio\"].sub(df_data[\"prior_white_pick_ratio\"])\n",
    "df_data[\"pick_ratio-div-prior_white_pick_ratio\"] = df_data[\"pick_ratio\"].div(df_data[\"prior_white_pick_ratio\"])\n",
    "\n",
    "df_data[\"pick_ratio-sub-prior_white_paired_reg_ratio\"] = df_data[\"pick_ratio\"].sub(df_data[\"prior_white_paired_reg_ratio\"])\n",
    "df_data[\"pick_ratio-div-prior_white_paired_reg_ratio\"] = df_data[\"pick_ratio\"].div(df_data[\"prior_white_paired_reg_ratio\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "a55666d8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>white_card_text</th>\n",
       "      <th>black_card_text</th>\n",
       "      <th>picks</th>\n",
       "      <th>pick_opportunities</th>\n",
       "      <th>ipw_wt</th>\n",
       "      <th>pick_ratio</th>\n",
       "      <th>picks_sum</th>\n",
       "      <th>prior_white_paired_reg_ratio</th>\n",
       "      <th>black_counts</th>\n",
       "      <th>black_round_skip_ratio</th>\n",
       "      <th>white_counts</th>\n",
       "      <th>white_picks</th>\n",
       "      <th>prior_white_pick_ratio</th>\n",
       "      <th>pick_ratio-sub-prior_white_pick_ratio</th>\n",
       "      <th>pick_ratio-div-prior_white_pick_ratio</th>\n",
       "      <th>pick_ratio-sub-prior_white_paired_reg_ratio</th>\n",
       "      <th>pick_ratio-div-prior_white_paired_reg_ratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Existing.</td>\n",
       "      <td>You're not going to believe this, but I'm you from the future! You've got to stop _____.</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>7.0708</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.1429</td>\n",
       "      <td>6520</td>\n",
       "      <td>0.0414</td>\n",
       "      <td>1472</td>\n",
       "      <td>209</td>\n",
       "      <td>0.1420</td>\n",
       "      <td>0.8580</td>\n",
       "      <td>7.0431</td>\n",
       "      <td>0.8571</td>\n",
       "      <td>6.9992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27442</th>\n",
       "      <td>The economy.</td>\n",
       "      <td>Mom's to-do list: • Buy groceries • Clean up _____ • Soccer practice</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>11.5544</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.0732</td>\n",
       "      <td>6280</td>\n",
       "      <td>0.0748</td>\n",
       "      <td>1481</td>\n",
       "      <td>138</td>\n",
       "      <td>0.0932</td>\n",
       "      <td>0.9068</td>\n",
       "      <td>10.7319</td>\n",
       "      <td>0.9268</td>\n",
       "      <td>13.6621</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100146</th>\n",
       "      <td>Mexicans and Muslims.</td>\n",
       "      <td>This app is basically Tinder, but for _____.</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>6.2173</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.1490</td>\n",
       "      <td>6310</td>\n",
       "      <td>0.0523</td>\n",
       "      <td>1496</td>\n",
       "      <td>245</td>\n",
       "      <td>0.1638</td>\n",
       "      <td>0.8362</td>\n",
       "      <td>6.1061</td>\n",
       "      <td>0.8510</td>\n",
       "      <td>6.7131</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100005</th>\n",
       "      <td>Meth.</td>\n",
       "      <td>This month's Cosmo: \"Spice up your sex life by bringing _____ into the bedroom.\"</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4.0331</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>6190</td>\n",
       "      <td>0.0404</td>\n",
       "      <td>1550</td>\n",
       "      <td>374</td>\n",
       "      <td>0.2413</td>\n",
       "      <td>0.7587</td>\n",
       "      <td>4.1444</td>\n",
       "      <td>0.7570</td>\n",
       "      <td>4.1148</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100004</th>\n",
       "      <td>Meth.</td>\n",
       "      <td>To prepare for his upcoming role, Daniel Day-Lewis immersed himself in the world of _____.</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4.0331</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>6450</td>\n",
       "      <td>0.0527</td>\n",
       "      <td>1550</td>\n",
       "      <td>374</td>\n",
       "      <td>0.2413</td>\n",
       "      <td>0.7587</td>\n",
       "      <td>4.1444</td>\n",
       "      <td>0.7570</td>\n",
       "      <td>4.1148</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120288</th>\n",
       "      <td>A fart so powerful that it wakes the giants from their thousand-year slumber.</td>\n",
       "      <td>It's late at night. You're in bed with your lover, Patrick Stewart. He leans over with a twinkle in his eye and asks \"How about _____?\"</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>1.1598</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.1465</td>\n",
       "      <td>5890</td>\n",
       "      <td>0.0628</td>\n",
       "      <td>1556</td>\n",
       "      <td>219</td>\n",
       "      <td>0.1407</td>\n",
       "      <td>-0.1407</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.1465</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120290</th>\n",
       "      <td>A fart so powerful that it wakes the giants from their thousand-year slumber.</td>\n",
       "      <td>50% of all marriages end in _____.</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>1.1598</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.1465</td>\n",
       "      <td>6380</td>\n",
       "      <td>0.0251</td>\n",
       "      <td>1556</td>\n",
       "      <td>219</td>\n",
       "      <td>0.1407</td>\n",
       "      <td>-0.1407</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.1465</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120291</th>\n",
       "      <td>A fart so powerful that it wakes the giants from their thousand-year slumber.</td>\n",
       "      <td>Having the worst day EVER. #_____</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>1.1598</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.1465</td>\n",
       "      <td>6650</td>\n",
       "      <td>0.0496</td>\n",
       "      <td>1556</td>\n",
       "      <td>219</td>\n",
       "      <td>0.1407</td>\n",
       "      <td>-0.1407</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.1465</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120292</th>\n",
       "      <td>A fart so powerful that it wakes the giants from their thousand-year slumber.</td>\n",
       "      <td>Dance like there's nobody watching, love like you'll never be hurt, and live like you're _____.</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>1.1598</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.1465</td>\n",
       "      <td>6320</td>\n",
       "      <td>0.0396</td>\n",
       "      <td>1556</td>\n",
       "      <td>219</td>\n",
       "      <td>0.1407</td>\n",
       "      <td>-0.1407</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.1465</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>288245</th>\n",
       "      <td>Motherfucking tater tots.</td>\n",
       "      <td>What's my secret power?</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>1.1446</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.1002</td>\n",
       "      <td>6410</td>\n",
       "      <td>0.0499</td>\n",
       "      <td>326</td>\n",
       "      <td>41</td>\n",
       "      <td>0.1258</td>\n",
       "      <td>-0.1258</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>-0.1002</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>288246 rows × 17 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                      white_card_text                                                                                                                          black_card_text  picks  pick_opportunities   ipw_wt  pick_ratio  picks_sum  prior_white_paired_reg_ratio  black_counts  black_round_skip_ratio  white_counts  white_picks  prior_white_pick_ratio  pick_ratio-sub-prior_white_pick_ratio  pick_ratio-div-prior_white_pick_ratio  pick_ratio-sub-prior_white_paired_reg_ratio  pick_ratio-div-prior_white_paired_reg_ratio\n",
       "0                                                                           Existing.                                                 You're not going to believe this, but I'm you from the future! You've got to stop _____.      1                   4   7.0708         1.0          4                        0.1429          6520                  0.0414          1472          209                  0.1420                                 0.8580                                 7.0431                                       0.8571                                       6.9992\n",
       "27442                                                                    The economy.                                                                     Mom's to-do list: • Buy groceries • Clean up _____ • Soccer practice      1                   4  11.5544         1.0          4                        0.0732          6280                  0.0748          1481          138                  0.0932                                 0.9068                                10.7319                                       0.9268                                      13.6621\n",
       "100146                                                          Mexicans and Muslims.                                                                                             This app is basically Tinder, but for _____.      1                   4   6.2173         1.0          4                        0.1490          6310                  0.0523          1496          245                  0.1638                                 0.8362                                 6.1061                                       0.8510                                       6.7131\n",
       "100005                                                                          Meth.                                                         This month's Cosmo: \"Spice up your sex life by bringing _____ into the bedroom.\"      1                   4   4.0331         1.0          4                        0.2430          6190                  0.0404          1550          374                  0.2413                                 0.7587                                 4.1444                                       0.7570                                       4.1148\n",
       "100004                                                                          Meth.                                               To prepare for his upcoming role, Daniel Day-Lewis immersed himself in the world of _____.      1                   4   4.0331         1.0          4                        0.2430          6450                  0.0527          1550          374                  0.2413                                 0.7587                                 4.1444                                       0.7570                                       4.1148\n",
       "...                                                                               ...                                                                                                                                      ...    ...                 ...      ...         ...        ...                           ...           ...                     ...           ...          ...                     ...                                    ...                                    ...                                          ...                                          ...\n",
       "120288  A fart so powerful that it wakes the giants from their thousand-year slumber.  It's late at night. You're in bed with your lover, Patrick Stewart. He leans over with a twinkle in his eye and asks \"How about _____?\"      0                   4   1.1598         0.0          0                        0.1465          5890                  0.0628          1556          219                  0.1407                                -0.1407                                 0.0000                                      -0.1465                                       0.0000\n",
       "120290  A fart so powerful that it wakes the giants from their thousand-year slumber.                                                                                                       50% of all marriages end in _____.      0                   5   1.1598         0.0          0                        0.1465          6380                  0.0251          1556          219                  0.1407                                -0.1407                                 0.0000                                      -0.1465                                       0.0000\n",
       "120291  A fart so powerful that it wakes the giants from their thousand-year slumber.                                                                                                        Having the worst day EVER. #_____      0                   5   1.1598         0.0          0                        0.1465          6650                  0.0496          1556          219                  0.1407                                -0.1407                                 0.0000                                      -0.1465                                       0.0000\n",
       "120292  A fart so powerful that it wakes the giants from their thousand-year slumber.                                          Dance like there's nobody watching, love like you'll never be hurt, and live like you're _____.      0                   6   1.1598         0.0          0                        0.1465          6320                  0.0396          1556          219                  0.1407                                -0.1407                                 0.0000                                      -0.1465                                       0.0000\n",
       "288245                                                      Motherfucking tater tots.                                                                                                                  What's my secret power?      0                   4   1.1446         0.0          0                        0.1002          6410                  0.0499           326           41                  0.1258                                -0.1258                                 0.0000                                      -0.1002                                       0.0000\n",
       "\n",
       "[288246 rows x 17 columns]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_data.sort_values([\"pick_ratio\"],ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aware-equivalent",
   "metadata": {},
   "source": [
    "## Save data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "traditional-missile",
   "metadata": {},
   "outputs": [],
   "source": [
    "if SAVE_TRAIN_DATA:\n",
    "#     df_data[[\"black_card_text\",\"white_card_text\",\"picks\",\"pick_opportunities\",\"picks_sum\",\"ipw_wt\",\"prior_white_pick_ratio\"]].sample(frac=1).to_csv(f\"cah_min{MIN_PAIR_FREQ}_v3.csv.gz\",index=False,compression=\"gzip\")\n",
    "    df_data.drop([\"black_counts\"],axis=1,errors=\"ignore\").sample(frac=1).to_csv(f\"cah_min{MIN_PAIR_FREQ}_v4.csv.gz\",index=False,compression=\"gzip\")\n",
    "#     df_data.sample(frac=1).to_csv(f\"cah_min{MIN_PAIR_FREQ}_v3.csv.gz\",index=False,compression=\"gzip\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fc46959",
   "metadata": {},
   "source": [
    "## Sentence transformers models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "composed-adobe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>black_card_text</th>\n",
       "      <th>white_card_text</th>\n",
       "      <th>picks</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>222690</th>\n",
       "      <td>Lovin' you is easy 'cause you're _____.</td>\n",
       "      <td>Casually suggesting a threesome.</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105489</th>\n",
       "      <td>Oh my god! _____ killed Kenny!</td>\n",
       "      <td>Destroying Dick Cheney's last horcrux.</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>207015</th>\n",
       "      <td>Congratulations! You have been selected for our summer internship program. While we are unable to offer a salary, we can offer you _____.</td>\n",
       "      <td>Square dancing with other racists.</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>282045</th>\n",
       "      <td>As reparations for slavery, all African Americans will receive _____.</td>\n",
       "      <td>Strong female characters.</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98643</th>\n",
       "      <td>In his newest and most difficult stunt, David Blaine must escape from _____.</td>\n",
       "      <td>Having a heart attack.</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>263801</th>\n",
       "      <td>You know who else liked _____? Hitler.</td>\n",
       "      <td>A magnificent chrome sphere that roams the streets evaporating criminals.</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>176082</th>\n",
       "      <td>It's the 21st century. A woman shouldn't have to apologize for _____.</td>\n",
       "      <td>Giving grandma crappy things she doesn't like.</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>196553</th>\n",
       "      <td>I'm going on a cleanse this week. Nothing but kale juice and _____.</td>\n",
       "      <td>Your mouth.</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>277187</th>\n",
       "      <td>Coming to Broadway this season, _____: The Musical.</td>\n",
       "      <td>Fiery poops.</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84096</th>\n",
       "      <td>IF you like _____, YOU MIGHT BE A REDNECK.</td>\n",
       "      <td>A good sniff.</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>288246 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                  black_card_text                                                            white_card_text  picks\n",
       "222690                                                                                                    Lovin' you is easy 'cause you're _____.                                           Casually suggesting a threesome.    1.0\n",
       "105489                                                                                                             Oh my god! _____ killed Kenny!                                     Destroying Dick Cheney's last horcrux.    1.0\n",
       "207015  Congratulations! You have been selected for our summer internship program. While we are unable to offer a salary, we can offer you _____.                                         Square dancing with other racists.    0.0\n",
       "282045                                                                      As reparations for slavery, all African Americans will receive _____.                                                  Strong female characters.    1.0\n",
       "98643                                                                In his newest and most difficult stunt, David Blaine must escape from _____.                                                     Having a heart attack.    0.0\n",
       "...                                                                                                                                           ...                                                                        ...    ...\n",
       "263801                                                                                                     You know who else liked _____? Hitler.  A magnificent chrome sphere that roams the streets evaporating criminals.    0.0\n",
       "176082                                                                      It's the 21st century. A woman shouldn't have to apologize for _____.                             Giving grandma crappy things she doesn't like.    0.0\n",
       "196553                                                                        I'm going on a cleanse this week. Nothing but kale juice and _____.                                                                Your mouth.    0.0\n",
       "277187                                                                                        Coming to Broadway this season, _____: The Musical.                                                               Fiery poops.    0.0\n",
       "84096                                                                                                  IF you like _____, YOU MIGHT BE A REDNECK.                                                              A good sniff.    0.0\n",
       "\n",
       "[288246 rows x 3 columns]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df_data[[\"black_card_text\",\"white_card_text\",\"picks\"]].sample(frac=1)#.values\n",
    "df[\"picks\"] = df[\"picks\"].astype(float) ## needed for pytorch\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "compound-amino",
   "metadata": {},
   "source": [
    "## Sentence transformers model(s)\n",
    "* Sentence bert (NLI? semtantic similarity?) \n",
    "https://www.sbert.net/docs/training/overview.html\n",
    "https://www.sbert.net/docs/quickstart.html  , https://github.com/UKPLab/sentence-transformers\n",
    "* Load data from disk ? \n",
    "\n",
    "* list of pretrained sentencebert models: (All are >300 M! slow download) https://docs.google.com/spreadsheets/d/14QplCdTCDwEmTqrn1LH4yrbKvdogK4oQvYO1K1aPR5M/edit#gid=0\n",
    "\n",
    "* `stsb-distilroberta-base-v2` - 305M sized model - semantic similarity\n",
    "* `nli-distilroberta-base-v2` - NLI \n",
    "* `average_word_embeddings_glove.6B.300d`, `average_word_embeddings_glove.840B.300d` - glove/w2v embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "fdd949ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sentence_transformers import SentenceTransformer, SentencesDataset, InputExample, losses, evaluation\n",
    "import torch\n",
    "torch.cuda.is_available()\n",
    "from torch.utils.data import DataLoader\n",
    "import torch.utils.data as data_utils\n",
    "## https://stackoverflow.com/questions/50307707/convert-pandas-dataframe-to-pytorch-tensor\n",
    "from transformers import AutoTokenizer, AutoModel\n",
    "## example from : https://www.sbert.net/docs/training/overview.html\n",
    "from sentence_transformers import SentenceTransformer, SentencesDataset, InputExample, losses \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "consecutive-journalist",
   "metadata": {},
   "outputs": [],
   "source": [
    "## pretrained weights - download vis program often fails, easier to download externally from ftp\n",
    "# model = SentenceTransformer('stsb-distilroberta-base-v2') ## was distilbert-base-nli-mean-tokens\n",
    "\n",
    "# model = SentenceTransformer('average_word_embeddings_glove.6B.300d') ## was distilbert-base-nli-mean-tokens\n",
    "\n",
    "# model = SentenceTransformer(\"nli-distilroberta-base-v2\",device=\"cuda\")\n",
    "model = SentenceTransformer(\"paraphrase-MiniLM-L6-v2\") ## \"paraphrase-MiniLM-L12-v2\"\n",
    "\n",
    "# model = SentenceTransformer(\"./stsb-distilroberta-base-v2\",device=\"cuda\")\n",
    "\n",
    "# model = SentenceTransformer(\"./nli-mpnet-base-v2\",device=\"cuda\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "6026b18e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "128"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.max_seq_length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "534e4923",
   "metadata": {},
   "outputs": [],
   "source": [
    "# len(model.tokenize([\"ad and \"])[\"input_ids\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "legal-exemption",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "384"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.get_sentence_embedding_dimension() # 768"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "novel-repeat",
   "metadata": {},
   "source": [
    "#### redo above, but with real data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "ac4afa38",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>black_card_text</th>\n",
       "      <th>white_card_text</th>\n",
       "      <th>picks</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>222690</th>\n",
       "      <td>Lovin' you is easy 'cause you're _____.</td>\n",
       "      <td>Casually suggesting a threesome.</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105489</th>\n",
       "      <td>Oh my god! _____ killed Kenny!</td>\n",
       "      <td>Destroying Dick Cheney's last horcrux.</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>207015</th>\n",
       "      <td>Congratulations! You have been selected for our summer internship program. While we are unable to offer a salary, we can offer you _____.</td>\n",
       "      <td>Square dancing with other racists.</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>282045</th>\n",
       "      <td>As reparations for slavery, all African Americans will receive _____.</td>\n",
       "      <td>Strong female characters.</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98643</th>\n",
       "      <td>In his newest and most difficult stunt, David Blaine must escape from _____.</td>\n",
       "      <td>Having a heart attack.</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>263801</th>\n",
       "      <td>You know who else liked _____? Hitler.</td>\n",
       "      <td>A magnificent chrome sphere that roams the streets evaporating criminals.</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>176082</th>\n",
       "      <td>It's the 21st century. A woman shouldn't have to apologize for _____.</td>\n",
       "      <td>Giving grandma crappy things she doesn't like.</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>196553</th>\n",
       "      <td>I'm going on a cleanse this week. Nothing but kale juice and _____.</td>\n",
       "      <td>Your mouth.</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>277187</th>\n",
       "      <td>Coming to Broadway this season, _____: The Musical.</td>\n",
       "      <td>Fiery poops.</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84096</th>\n",
       "      <td>IF you like _____, YOU MIGHT BE A REDNECK.</td>\n",
       "      <td>A good sniff.</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>288246 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                  black_card_text                                                            white_card_text  picks\n",
       "222690                                                                                                    Lovin' you is easy 'cause you're _____.                                           Casually suggesting a threesome.    1.0\n",
       "105489                                                                                                             Oh my god! _____ killed Kenny!                                     Destroying Dick Cheney's last horcrux.    1.0\n",
       "207015  Congratulations! You have been selected for our summer internship program. While we are unable to offer a salary, we can offer you _____.                                         Square dancing with other racists.    0.0\n",
       "282045                                                                      As reparations for slavery, all African Americans will receive _____.                                                  Strong female characters.    1.0\n",
       "98643                                                                In his newest and most difficult stunt, David Blaine must escape from _____.                                                     Having a heart attack.    0.0\n",
       "...                                                                                                                                           ...                                                                        ...    ...\n",
       "263801                                                                                                     You know who else liked _____? Hitler.  A magnificent chrome sphere that roams the streets evaporating criminals.    0.0\n",
       "176082                                                                      It's the 21st century. A woman shouldn't have to apologize for _____.                             Giving grandma crappy things she doesn't like.    0.0\n",
       "196553                                                                        I'm going on a cleanse this week. Nothing but kale juice and _____.                                                                Your mouth.    0.0\n",
       "277187                                                                                        Coming to Broadway this season, _____: The Musical.                                                               Fiery poops.    0.0\n",
       "84096                                                                                                  IF you like _____, YOU MIGHT BE A REDNECK.                                                              A good sniff.    0.0\n",
       "\n",
       "[288246 rows x 3 columns]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "modified-journal",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "36030\n"
     ]
    }
   ],
   "source": [
    "num_epochs = 2\n",
    "TEST_SIZE = df.shape[0]//8 #4500\n",
    "print(TEST_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "f9bb652d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(288246, 3)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "711f4206",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "288246"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.drop_duplicates(['black_card_text', 'white_card_text']).shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "victorian-relevance",
   "metadata": {},
   "outputs": [],
   "source": [
    "## is list correct? error otherwise? \n",
    "train_examples = list(df.iloc[TEST_SIZE:].apply(lambda row: InputExample(texts=[row[\"white_card_text\"], row[\"black_card_text\"]],\n",
    "                                                                         label=row[\"picks\"]),axis=1)) # picks\n",
    "test_examples =  list(df.iloc[0:TEST_SIZE].apply(lambda row: InputExample(texts=[row[\"white_card_text\"], row[\"black_card_text\"]], \n",
    "                                                                          label=row[\"picks\"]),axis=1)) # picks\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "deadly-lancaster",
   "metadata": {},
   "outputs": [],
   "source": [
    "# dev_eval = evaluation.BinaryClassificationEvaluator(sentences1=df.iloc[0:2500,0],\n",
    "#     sentences2= df.iloc[0:2500,1],\n",
    "#     labels=df.iloc[0:2500,2],show_progress_bar=True,batch_size=256)\n",
    "dev_eval = evaluation.BinaryClassificationEvaluator(sentences1=list(df.iloc[0:TEST_SIZE,0].values),\n",
    "    sentences2= list(df.iloc[0:TEST_SIZE,1].values),\n",
    "    labels=list(df.iloc[0:TEST_SIZE,2].values),show_progress_bar=True,batch_size=256)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "51be1206",
   "metadata": {},
   "outputs": [],
   "source": [
    "## https://github.com/UKPLab/sentence-transformers/blob/master/examples/training/nli/training_nli.py \n",
    "\n",
    "## evaluator example \n",
    "train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=256)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "normal-bones",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "960ad0ca8e1c4a5db821377623c1db3c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7f5f0ecd26f14fbd872896f9e646a225",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1971 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "83ea2f7ad0a045a8b83364ed649a3e09",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "90d7a216edaf42c1b4a94ff9ff1e82dd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1971 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "359188603cd94383b4af6c7c8f18bacb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b3cc79a180c54ccd8350237cfceaf256",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Iteration:   0%|          | 0/1971 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "df4f64144eff41058120ab0c9bd1796c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 14min 46s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# train_loss = losses.CosineSimilarityLoss(model) # ORIG\n",
    "\n",
    "# train_loss = losses.SoftmaxLoss(model,\n",
    "#     sentence_embedding_dimension= model.get_sentence_embedding_dimension(),\n",
    "#     num_labels= 2,\n",
    "#     concatenation_sent_rep= True,\n",
    "#     concatenation_sent_difference = True,\n",
    "#     concatenation_sent_multiplication = True) # ALT - for classification https://www.sbert.net/docs/package_reference/losses.html#softmaxloss\n",
    "\n",
    "train_loss = losses.CosineSimilarityLoss(model)\n",
    "#Tune the model\n",
    "model.fit(train_objectives=[(train_dataloader, train_loss)], \n",
    "          epochs=num_epochs, \n",
    "#           warmup_steps=100, # 100\n",
    "          warmup_steps = math.ceil(len(train_dataloader) * 0.05*3), #5% of train data for warm-up\n",
    "          evaluator=dev_eval,\n",
    "          use_amp=True,\n",
    "          optimizer_params= {'lr': 3e-04}, # 2e-05\n",
    "          output_path=\"./output\",\n",
    "          save_best_model = False,\n",
    "         )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "43e0f6ae",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6da8408037344da28f6633e50e401fb6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "0.5460244073549838"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(dev_eval)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1d8eb98",
   "metadata": {},
   "source": [
    "#### Evaluate model on original data, game by game.\n",
    "* Note: should use proper test set , disjoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "7cc659d6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fake_round_id</th>\n",
       "      <th>black_card_text</th>\n",
       "      <th>white_card_text</th>\n",
       "      <th>won</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20001</th>\n",
       "      <td>274529</td>\n",
       "      <td>I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!</td>\n",
       "      <td>Swooping.</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20004</th>\n",
       "      <td>274529</td>\n",
       "      <td>I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!</td>\n",
       "      <td>The peaceful and nonthreatening rise of China.</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20003</th>\n",
       "      <td>274529</td>\n",
       "      <td>I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!</td>\n",
       "      <td>My Uber driver, Pavel.</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20002</th>\n",
       "      <td>274529</td>\n",
       "      <td>I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!</td>\n",
       "      <td>Child support payments.</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>274529</td>\n",
       "      <td>I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!</td>\n",
       "      <td>Teenage pregnancy.</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>799</th>\n",
       "      <td>275481</td>\n",
       "      <td>Instead of coal, Santa now gives the bad children _____.</td>\n",
       "      <td>Clams.</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23998</th>\n",
       "      <td>275481</td>\n",
       "      <td>Instead of coal, Santa now gives the bad children _____.</td>\n",
       "      <td>Getting caught by the police and going to jail.</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23996</th>\n",
       "      <td>275481</td>\n",
       "      <td>Instead of coal, Santa now gives the bad children _____.</td>\n",
       "      <td>Fancy Feast.®</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23995</th>\n",
       "      <td>275481</td>\n",
       "      <td>Instead of coal, Santa now gives the bad children _____.</td>\n",
       "      <td>How strange it is to be anything at all.</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23997</th>\n",
       "      <td>275481</td>\n",
       "      <td>Instead of coal, Santa now gives the bad children _____.</td>\n",
       "      <td>The power of the Dark Side.</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4800 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       fake_round_id                                                                                               black_card_text                                  white_card_text    won\n",
       "20001         274529  I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!                                        Swooping.  False\n",
       "20004         274529  I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!   The peaceful and nonthreatening rise of China.  False\n",
       "20003         274529  I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!                           My Uber driver, Pavel.  False\n",
       "20002         274529  I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!                          Child support payments.  False\n",
       "0             274529  I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!                               Teenage pregnancy.   True\n",
       "...              ...                                                                                                           ...                                              ...    ...\n",
       "799           275481                                                      Instead of coal, Santa now gives the bad children _____.                                           Clams.   True\n",
       "23998         275481                                                      Instead of coal, Santa now gives the bad children _____.  Getting caught by the police and going to jail.  False\n",
       "23996         275481                                                      Instead of coal, Santa now gives the bad children _____.                                    Fancy Feast.®  False\n",
       "23995         275481                                                      Instead of coal, Santa now gives the bad children _____.         How strange it is to be anything at all.  False\n",
       "23997         275481                                                      Instead of coal, Santa now gives the bad children _____.                      The power of the Dark Side.  False\n",
       "\n",
       "[4800 rows x 4 columns]"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# df = pd.read_parquet('cah_lab_data.parquet')\n",
    "df_test = df_sample.loc[df_sample[\"black_card_pick_num\"]==1].sort_values(\"fake_round_id\")[[\"fake_round_id\",\"black_card_text\",\"white_card_text\",\"won\"]].copy()\n",
    "\n",
    "df_test = df_test.head(4800)\n",
    "df_test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00316b2b",
   "metadata": {},
   "source": [
    "* if using just black/white, could do this for unique cards - ~1000 x less compute..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "bd91043e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 1.88 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "## https://www.sbert.net/docs/usage/semantic_textual_similarity.html\n",
    "\n",
    "sentences1 = df_test[\"black_card_text\"].values\n",
    "sentences2 = df_test[\"white_card_text\"].values\n",
    "embeddings1 = model.encode(sentences1,batch_size=256,normalize_embeddings=False, convert_to_tensor=True)\n",
    "\n",
    "embeddings2 = model.encode(sentences2,batch_size=256,normalize_embeddings=False, convert_to_tensor=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "99badd75",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "top1 accuracy: 0.08125\n",
      "top1 accuracy, reverse sort: 0.34125\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fake_round_id</th>\n",
       "      <th>black_card_text</th>\n",
       "      <th>white_card_text</th>\n",
       "      <th>won</th>\n",
       "      <th>cos_scores</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>23996</th>\n",
       "      <td>275481</td>\n",
       "      <td>Instead of coal, Santa now gives the bad children _____.</td>\n",
       "      <td>Fancy Feast.®</td>\n",
       "      <td>False</td>\n",
       "      <td>0.5201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23997</th>\n",
       "      <td>275481</td>\n",
       "      <td>Instead of coal, Santa now gives the bad children _____.</td>\n",
       "      <td>The power of the Dark Side.</td>\n",
       "      <td>False</td>\n",
       "      <td>0.4402</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>799</th>\n",
       "      <td>275481</td>\n",
       "      <td>Instead of coal, Santa now gives the bad children _____.</td>\n",
       "      <td>Clams.</td>\n",
       "      <td>True</td>\n",
       "      <td>0.3842</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23999</th>\n",
       "      <td>275481</td>\n",
       "      <td>Instead of coal, Santa now gives the bad children _____.</td>\n",
       "      <td>Marsha P. Johnson, the trans woman of color who may have thrown the first brick at Stonewall.</td>\n",
       "      <td>False</td>\n",
       "      <td>0.2346</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23995</th>\n",
       "      <td>275481</td>\n",
       "      <td>Instead of coal, Santa now gives the bad children _____.</td>\n",
       "      <td>How strange it is to be anything at all.</td>\n",
       "      <td>False</td>\n",
       "      <td>0.0967</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20002</th>\n",
       "      <td>274529</td>\n",
       "      <td>I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!</td>\n",
       "      <td>Child support payments.</td>\n",
       "      <td>False</td>\n",
       "      <td>0.4111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20003</th>\n",
       "      <td>274529</td>\n",
       "      <td>I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!</td>\n",
       "      <td>My Uber driver, Pavel.</td>\n",
       "      <td>False</td>\n",
       "      <td>0.3036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20000</th>\n",
       "      <td>274529</td>\n",
       "      <td>I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!</td>\n",
       "      <td>Schindler's List.</td>\n",
       "      <td>False</td>\n",
       "      <td>0.2734</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20004</th>\n",
       "      <td>274529</td>\n",
       "      <td>I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!</td>\n",
       "      <td>The peaceful and nonthreatening rise of China.</td>\n",
       "      <td>False</td>\n",
       "      <td>0.1944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20001</th>\n",
       "      <td>274529</td>\n",
       "      <td>I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!</td>\n",
       "      <td>Swooping.</td>\n",
       "      <td>False</td>\n",
       "      <td>0.1248</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4800 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       fake_round_id                                                                                               black_card_text                                                                                white_card_text    won  cos_scores\n",
       "23996         275481                                                      Instead of coal, Santa now gives the bad children _____.                                                                                  Fancy Feast.®  False      0.5201\n",
       "23997         275481                                                      Instead of coal, Santa now gives the bad children _____.                                                                    The power of the Dark Side.  False      0.4402\n",
       "799           275481                                                      Instead of coal, Santa now gives the bad children _____.                                                                                         Clams.   True      0.3842\n",
       "23999         275481                                                      Instead of coal, Santa now gives the bad children _____.  Marsha P. Johnson, the trans woman of color who may have thrown the first brick at Stonewall.  False      0.2346\n",
       "23995         275481                                                      Instead of coal, Santa now gives the bad children _____.                                                       How strange it is to be anything at all.  False      0.0967\n",
       "...              ...                                                                                                           ...                                                                                            ...    ...         ...\n",
       "20002         274529  I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!                                                                        Child support payments.  False      0.4111\n",
       "20003         274529  I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!                                                                         My Uber driver, Pavel.  False      0.3036\n",
       "20000         274529  I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!                                                                              Schindler's List.  False      0.2734\n",
       "20004         274529  I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!                                                 The peaceful and nonthreatening rise of China.  False      0.1944\n",
       "20001         274529  I'm Tony Robbins, and over the next sixty minutes, I'm going to teach you how to harness the power of _____!                                                                                      Swooping.  False      0.1248\n",
       "\n",
       "[4800 rows x 5 columns]"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = []\n",
    "#Compute cosine-similarits\n",
    "cosine_scores = util.pytorch_cos_sim(embeddings1, embeddings2)\n",
    "\n",
    "#Output the pairs with their score\n",
    "for i in range(len(sentences1)):\n",
    "#     print(\"{} \\t\\t {} \\t\\t Score: {:.4f}\".format(sentences1[i], sentences2[i], cosine_scores[i][i]))\n",
    "    scores.append(float(cosine_scores[i][i]))\n",
    "df_test[\"cos_scores\"] = scores\n",
    "df_test.sort_values([\"fake_round_id\",\"cos_scores\"],inplace=True,ascending=True)\n",
    "print(\"top1 accuracy:\",df_test.groupby(\"fake_round_id\").head(1)[\"won\"].mean())\n",
    "\n",
    "df_test.sort_values([\"fake_round_id\",\"cos_scores\"],inplace=True,ascending=False)\n",
    "print(\"top1 accuracy, reverse sort:\",df_test.groupby(\"fake_round_id\").head(1)[\"won\"].mean())\n",
    "df_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "a5da5263",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.08125"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "interpreted-factor",
   "metadata": {},
   "source": [
    "#### Try  other model \n",
    "* https://www.sbert.net/docs/package_reference/evaluation.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "3ef34e07",
   "metadata": {},
   "outputs": [],
   "source": [
    "# del model\n",
    "\n",
    "# torch.cuda.empty_cache()\n",
    "# with torch.cuda.device('cuda:0'):\n",
    "#     torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "1281cdba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# model2 = SentenceTransformer(\"./nli-mpnet-base-v2\",device=\"cuda\")\n",
    "# print(model2.get_sentence_embedding_dimension())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "4b94ae46",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# %%time\n",
    "# ### cuda memor allocation error? \n",
    "# train_loss = losses.SoftmaxLoss(model2,\n",
    "#     sentence_embedding_dimension= model2.get_sentence_embedding_dimension(),\n",
    "#     num_labels= 2,\n",
    "#     concatenation_sent_rep= True,\n",
    "#     concatenation_sent_difference = True,\n",
    "#     concatenation_sent_multiplication = True) # ALT - for classification https://www.sbert.net/docs/package_reference/losses.html#softmaxloss\n",
    "\n",
    "# #Tune the model\n",
    "# model2.fit(train_objectives=[(train_dataloader, train_loss)], \n",
    "#           epochs=1, \n",
    "#           warmup_steps=100, # 100\n",
    "#           evaluator=dev_eval,\n",
    "#            save_best_model = False,\n",
    "#           use_amp=True,\n",
    "#           optimizer_params= {'lr': 1e-03}, # 2e-05\n",
    "#           output_path=\"./output2\"\n",
    "#          )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b216e21f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44eb33fa",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}