examples/1_bias_bios.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import pytorch_lightning as pl\n",
    "from pytorch_lightning import seed_everything\n",
    "from pytorch_lightning.callbacks import ModelCheckpoint\n",
    "from weasel.utils import utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Bias in Bios $-$ ProfTeacher\n",
    "\n",
    "***This tutorial is a minimal example of what needs to be defined, and how to intertwine it, so that you can train Weasel.***\n",
    "<br>$\\rightarrow$ *For an extended, more feature-rich version of this notebook, check the [1_bias_bios_full script](1_bias_bios_full.py) out (including logging to wandb, callbacks, and how to retrieve your stand-alone end-model after training it with Weasel).*\n",
    "<br>$\\rightarrow$ *If you don't want to use Hydra for now or just want to see the way Weasel interfaces with PyTorch Lightning explicitly, check [this no-Hydra version of the present notebook](./1_bias_bios_no_hydra.ipynb) out - it will make your project easier if you do use it though!*\n",
    "<br>$\\rightarrow$ *In the end of this notebook you will find a (performance) comparison to [Snorkel](https://github.com/snorkel-team/snorkel)*\n",
    "\n",
    "First let's load some pre-defined dataset & model parameters and hyperparameters.\n",
    "We'll use [Hydra](https://hydra.cc/) in this notebook as config manager.\n",
    "It is very convenient, flexible and speeds the ML-pipeline process up. Totally recommended for your own problem!\n",
    "\n",
    "You will see below in the notebook output several (hyper-)parameters for the various submodules needed to train Weasel:\n",
    "- Data, of course :) - more on it in the following **Data** markdown cell below.\n",
    "- An end-model - this is your part: choose any neural net that suits your problem best!\n",
    "- Weasel's configuration itself, including the encoder network that will learn to predict based on the labeling functions (LF).\n",
    "- A standard pl.Trainer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">⚙ CONFIG                                                                                     </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">├── </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">datamodule                                                                               </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   └── </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">_target_</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">src.datamodules.ProfTeacher_datamodule.ProfTeacher_DataModule              </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">num_LFs</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">99                                                                          </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">n_classes</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">2                                                                         </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">class_balance</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">:</span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">                                                                       </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">-</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">0.5                                                                                </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">-</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">0.5                                                                                </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">batch_size</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">64                                                                       </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">val_test_split</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">:</span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">                                                                      </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">-</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">250                                                                                </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">-</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">-1                                                                                 </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">seed</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">3                                                                              </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">                                                                                     </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">├── </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">end_model                                                                                </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   └── </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">_target_</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">src.models.downstream_models.MLP.MLPNet                                    </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">dropout</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">0.3                                                                         </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">net_norm</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">none                                                                       </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">activation_func</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">ReLU                                                                </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">input_dim</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">300                                                                       </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">hidden_dims</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">:</span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">                                                                         </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">-</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">50                                                                                 </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">-</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">50                                                                                 </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">-</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">25                                                                                 </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">output_dim</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">2                                                                        </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">adjust_thresh</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">false                                                                 </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">                                                                                     </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">├── </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">Weasel                                                                                   </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   └── </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">_target_</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">src.models.weasel.Weasel                                                   </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">num_LFs</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">99                                                                          </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">n_classes</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">2                                                                         </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">class_balance</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">:</span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">                                                                       </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">-</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">0.5                                                                                </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">-</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">0.5                                                                                </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">loss_function</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">cross_entropy                                                         </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">temperature</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">2.0                                                                     </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">accuracy_scaler</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">sqrt                                                                </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">use_aux_input_for_encoder</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">true                                                      </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">class_conditional_accuracies</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">true                                                   </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">encoder</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">:</span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">                                                                             </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">  </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">_target_</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">src.models.encoder_models.encoder_MLP.MLPEncoder                         </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">  </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">dropout</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">0.3                                                                       </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">  </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">net_norm</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">batch_norm                                                               </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">  </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">activation_func</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">ReLU                                                              </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">  </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">hidden_dims</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">:</span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">                                                                       </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">  </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">-</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">70                                                                               </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">  </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">-</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\"> </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">70                                                                               </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">optim_end_model</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">:</span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">                                                                     </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">  </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">_target_</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">torch.optim.Adam                                                         </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">  </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">lr</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">0.0001                                                                         </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">  </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">weight_decay</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">7.0e-07                                                              </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">optim_encoder</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">:</span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">                                                                       </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">  </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">_target_</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">torch.optim.Adam                                                         </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">  </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">lr</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">0.0001                                                                         </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">  </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">weight_decay</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">7.0e-07                                                              </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">scheduler</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">null                                                                      </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">verbose</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">true                                                                        </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">                                                                                     </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">├── </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">trainer                                                                                  </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   └── </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">_target_</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">pytorch_lightning.Trainer                                                  </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">gpus</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">1                                                                              </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #fc92b8; text-decoration-color: #fc92b8; background-color: #272822\">max_epochs</span><span style=\"color: #fbfbf8; text-decoration-color: #fbfbf8; background-color: #272822\">: </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">100                                                                      </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│       </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">                                                                                     </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">└── </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">seed                                                                                     </span>\n",
       "<span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">    └── </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f; background-color: #272822\">3                                                                                    </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001B[2m⚙ CONFIG\u001B[0m\u001B[2m                                                                                     \u001B[0m\n",
       "\u001B[2m├── \u001B[0m\u001B[2mdatamodule\u001B[0m\u001B[2m                                                                               \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m└── \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34m_target_\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34msrc.datamodules.ProfTeacher_datamodule.ProfTeacher_DataModule\u001B[0m\u001B[2;48;2;39;40;34m              \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mnum_LFs\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m99\u001B[0m\u001B[2;48;2;39;40;34m                                                                          \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mn_classes\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m2\u001B[0m\u001B[2;48;2;39;40;34m                                                                         \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mclass_balance\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;48;2;39;40;34m                                                                       \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;48;2;39;40;34m-\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m0.5\u001B[0m\u001B[2;48;2;39;40;34m                                                                                \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;48;2;39;40;34m-\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m0.5\u001B[0m\u001B[2;48;2;39;40;34m                                                                                \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mbatch_size\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m64\u001B[0m\u001B[2;48;2;39;40;34m                                                                       \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mval_test_split\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;48;2;39;40;34m                                                                      \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;48;2;39;40;34m-\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m250\u001B[0m\u001B[2;48;2;39;40;34m                                                                                \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;48;2;39;40;34m-\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m-1\u001B[0m\u001B[2;48;2;39;40;34m                                                                                 \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mseed\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m3\u001B[0m\u001B[2;48;2;39;40;34m                                                                              \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;48;2;39;40;34m                                                                                     \u001B[0m\n",
       "\u001B[2m├── \u001B[0m\u001B[2mend_model\u001B[0m\u001B[2m                                                                                \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m└── \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34m_target_\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34msrc.models.downstream_models.MLP.MLPNet\u001B[0m\u001B[2;48;2;39;40;34m                                    \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mdropout\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m0.3\u001B[0m\u001B[2;48;2;39;40;34m                                                                         \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mnet_norm\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34mnone\u001B[0m\u001B[2;48;2;39;40;34m                                                                       \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mactivation_func\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34mReLU\u001B[0m\u001B[2;48;2;39;40;34m                                                                \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34minput_dim\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m300\u001B[0m\u001B[2;48;2;39;40;34m                                                                       \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mhidden_dims\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;48;2;39;40;34m                                                                         \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;48;2;39;40;34m-\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m50\u001B[0m\u001B[2;48;2;39;40;34m                                                                                 \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;48;2;39;40;34m-\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m50\u001B[0m\u001B[2;48;2;39;40;34m                                                                                 \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;48;2;39;40;34m-\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m25\u001B[0m\u001B[2;48;2;39;40;34m                                                                                 \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34moutput_dim\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m2\u001B[0m\u001B[2;48;2;39;40;34m                                                                        \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34madjust_thresh\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34mfalse\u001B[0m\u001B[2;48;2;39;40;34m                                                                 \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;48;2;39;40;34m                                                                                     \u001B[0m\n",
       "\u001B[2m├── \u001B[0m\u001B[2mWeasel\u001B[0m\u001B[2m                                                                                   \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m└── \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34m_target_\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34msrc.models.weasel.Weasel\u001B[0m\u001B[2;48;2;39;40;34m                                                   \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mnum_LFs\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m99\u001B[0m\u001B[2;48;2;39;40;34m                                                                          \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mn_classes\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m2\u001B[0m\u001B[2;48;2;39;40;34m                                                                         \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mclass_balance\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;48;2;39;40;34m                                                                       \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;48;2;39;40;34m-\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m0.5\u001B[0m\u001B[2;48;2;39;40;34m                                                                                \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;48;2;39;40;34m-\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m0.5\u001B[0m\u001B[2;48;2;39;40;34m                                                                                \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mloss_function\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34mcross_entropy\u001B[0m\u001B[2;48;2;39;40;34m                                                         \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mtemperature\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m2.0\u001B[0m\u001B[2;48;2;39;40;34m                                                                     \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34maccuracy_scaler\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34msqrt\u001B[0m\u001B[2;48;2;39;40;34m                                                                \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34muse_aux_input_for_encoder\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34mtrue\u001B[0m\u001B[2;48;2;39;40;34m                                                      \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mclass_conditional_accuracies\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34mtrue\u001B[0m\u001B[2;48;2;39;40;34m                                                   \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mencoder\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;48;2;39;40;34m                                                                             \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m  \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34m_target_\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34msrc.models.encoder_models.encoder_MLP.MLPEncoder\u001B[0m\u001B[2;48;2;39;40;34m                         \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m  \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mdropout\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m0.3\u001B[0m\u001B[2;48;2;39;40;34m                                                                       \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m  \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mnet_norm\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34mbatch_norm\u001B[0m\u001B[2;48;2;39;40;34m                                                               \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m  \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mactivation_func\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34mReLU\u001B[0m\u001B[2;48;2;39;40;34m                                                              \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m  \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mhidden_dims\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;48;2;39;40;34m                                                                       \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m  \u001B[0m\u001B[2;48;2;39;40;34m-\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m70\u001B[0m\u001B[2;48;2;39;40;34m                                                                               \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m  \u001B[0m\u001B[2;48;2;39;40;34m-\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m70\u001B[0m\u001B[2;48;2;39;40;34m                                                                               \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34moptim_end_model\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;48;2;39;40;34m                                                                     \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m  \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34m_target_\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34mtorch.optim.Adam\u001B[0m\u001B[2;48;2;39;40;34m                                                         \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m  \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mlr\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m0.0001\u001B[0m\u001B[2;48;2;39;40;34m                                                                         \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m  \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mweight_decay\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m7.0e-07\u001B[0m\u001B[2;48;2;39;40;34m                                                              \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34moptim_encoder\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;48;2;39;40;34m                                                                       \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m  \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34m_target_\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34mtorch.optim.Adam\u001B[0m\u001B[2;48;2;39;40;34m                                                         \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m  \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mlr\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m0.0001\u001B[0m\u001B[2;48;2;39;40;34m                                                                         \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m  \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mweight_decay\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m7.0e-07\u001B[0m\u001B[2;48;2;39;40;34m                                                              \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mscheduler\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34mnull\u001B[0m\u001B[2;48;2;39;40;34m                                                                      \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mverbose\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34mtrue\u001B[0m\u001B[2;48;2;39;40;34m                                                                        \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;48;2;39;40;34m                                                                                     \u001B[0m\n",
       "\u001B[2m├── \u001B[0m\u001B[2mtrainer\u001B[0m\u001B[2m                                                                                  \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m└── \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34m_target_\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34mpytorch_lightning.Trainer\u001B[0m\u001B[2;48;2;39;40;34m                                                  \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mgpus\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m1\u001B[0m\u001B[2;48;2;39;40;34m                                                                              \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;38;2;249;38;114;48;2;39;40;34mmax_epochs\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m:\u001B[0m\u001B[2;38;2;248;248;242;48;2;39;40;34m \u001B[0m\u001B[2;48;2;39;40;34m100\u001B[0m\u001B[2;48;2;39;40;34m                                                                      \u001B[0m\n",
       "\u001B[2m│   \u001B[0m\u001B[2m    \u001B[0m\u001B[2;48;2;39;40;34m                                                                                     \u001B[0m\n",
       "\u001B[2m└── \u001B[0m\u001B[2mseed\u001B[0m\u001B[2m                                                                                     \u001B[0m\n",
       "\u001B[2m    \u001B[0m\u001B[2m└── \u001B[0m\u001B[2;48;2;39;40;34m3\u001B[0m\u001B[2;48;2;39;40;34m                                                                                    \u001B[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Global seed set to 3\n"
     ]
    }
   ],
   "source": [
    "import hydra\n",
    "from hydra.utils import instantiate as hydra_instantiate\n",
    "\n",
    "with hydra.initialize(config_path=\"examples/configs/\"):\n",
    "    config = hydra.compose(\n",
    "        config_name=\"profTeacher_simple.yaml\",\n",
    "        overrides=[\"end_model.adjust_thresh=False\",  # Hydra makes overriding easy!\n",
    "                   \"trainer.gpus=1\"]  # train on a GPU, set to 0 for CPU only\n",
    "\n",
    "    ) \n",
    "\n",
    "utils.print_config(config) # prints only when rich library is installed\n",
    "_ = seed_everything(config.seed, workers=True)  # seed for reproducibility"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Data in WeaSEL\n",
    "\n",
    "In this example we have $C=2$ classes (teacher or professor biography within a text document).\n",
    "\n",
    "We can make use of data features, *X_train* and *X_test*, for all training and test data points.\n",
    "Eventually we hope to make predictions with our end-model based solely on these features.\n",
    "\n",
    "However, what if we don't have labels, Y, for our training examples, i.e. can't train our end-model\n",
    "in the traditional supervised way on (X, Y) examples?\n",
    "This is where multi-source weak supervision & Weasel come to the rescue! :D\n",
    "\n",
    "#### Multiple Labeling Heuristics replace ground truth training labels\n",
    "For this concrete problem, we created $m=99$ regex-based LFs, that we then applied on the $n=12.5k$ training examples, which gives us\n",
    "a label matrix $L$ of shape $n \\times m$ with values in $\\{-1, 0, .., C-1\\} = \\{-1, 0, 1\\}$.\n",
    "\n",
    "Here, $-1$ means that a LF abstained from labeling, while $0, 1$ indicate that the LF believes that the particular\n",
    "example is a teacher, professor biography, respectively.\n",
    "\n",
    "#### Optional, but recommended evaluation of the end-model\n",
    "Lastly, we will also want to evaluate the skill of our end-model on a small test set that contains\n",
    "ground truth labels *Y_test* (with the corresponding features *X_test*).\n",
    "\n",
    "### All in one place\n",
    "\n",
    "All this is conveniently encapsulated in an [abstract PyTorch Lightning DataModule](../weasel/datamodules/base_datamodule.py),\n",
    " that makes it simple for you to set up your own DataModule to easily train your end-model, provided that you already have all of the above at hand:\n",
    "\n",
    "- Training and Test features, *X_train, X_test*\n",
    "- Label matrix, *L*\n",
    "- Ground truth test labels, *Y_test*\n",
    "\n",
    "See the [code that defines ProfTeacher_DataModule](datamodules/ProfTeacher_datamodule.py)\n",
    "to see how simple it then is to get started.\n",
    "Alternatively to the approach of the ``ProfTeacher_DataModule`` that encapsulates the data loading within the DataModule itself,\n",
    " you may also create a ``base_datamodule.BasicWeaselDataModule`` by passing it in the constructor the four data components above,\n",
    " see [this notebook](0_full_pipeline.ipynb) for a simple synthetic example of it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Hydra instantiations make instantiating the DataModule, Model, Trainer, etc. a one-liner :)\n",
    "profTeacher_data_module = hydra_instantiate(config.datamodule)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End-model\n",
    "\n",
    "Having set up the data part, you'll first have to choose your favorite neural net as the end-model <br>\n",
    " (the one that you want to use as ``predictions = end-model(X)`` eventually).\n",
    "\n",
    "Here, we'll use a simple 2-layer feed-forward net/MLP, but you\n",
    "can easily replace it with *any* neural net model, see [these instructions](../weasel/models/downstream_models/README.md)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "MLP_end_model = hydra_instantiate(config.end_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Weasel: Marrying your end-model with the LFs\n",
    "We now pass this end-model to the wrapping [Weasel model](../weasel/models/weasel.py), which will take care of learning the end-model based\n",
    "on *X_train* and the LFs, i.e. the label matrix *L*.\n",
    "To do so, there is an encoder net (another MLP in this case) in Weasel's core, that will be predicting labels based on the\n",
    "LFs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "weasel_model = hydra_instantiate(config.Weasel, end_model=MLP_end_model, _recursive_=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Weasel and end-model\n",
    "\n",
    "Before fitting Weasel and the end-model, we now just need to instantiate a pl.Trainer instance\n",
    "(we will checkpoint the best model w.r.t. AUC performance on a small validation set that is split off the test set)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n"
     ]
    }
   ],
   "source": [
    "checkpoint_callback = ModelCheckpoint(monitor=\"Val/auc\", mode=\"max\")\n",
    "\n",
    "trainer = hydra_instantiate(\n",
    "        config.trainer, callbacks=checkpoint_callback, deterministic=True, max_epochs=75\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Then, with all the ease of PyTorch Lightning, we can train our model on the DataModule from above.\n",
    "\n",
    "*If any strange errors are thrown, try rerunning the cell! (e.g. \"ValueError: dictionary update sequence element #0 has length 1; 2 is required\")*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "\n",
      "  | Name          | Type       | Params\n",
      "---------------------------------------------\n",
      "0 | end_model     | MLPNet     | 18.9 K\n",
      "1 | encoder       | MLPEncoder | 47.3 K\n",
      "2 | accuracy_func | Softmax    | 0     \n",
      "---------------------------------------------\n",
      "66.2 K    Trainable params\n",
      "0         Non-trainable params\n",
      "66.2 K    Total params\n",
      "0.265     Total estimated model params size (MB)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                                      "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Global seed set to 3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0:  98%|█████████▊| 193/197 [00:03<00:00, 59.55it/s, Val/accuracy=0.508, Val/recall=1.000, Val/precision=0.508, Val/f1=0.674, Val/auc=0.593, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 0: 100%|██████████| 197/197 [00:03<00:00, 58.83it/s, Val/accuracy=0.524, Val/recall=1.000, Val/precision=0.506, Val/f1=0.672, Val/auc=0.780, decision_thresh=0.500]\n",
      "Epoch 1:  98%|█████████▊| 193/197 [00:03<00:00, 57.97it/s, Val/accuracy=0.524, Val/recall=1.000, Val/precision=0.506, Val/f1=0.672, Val/auc=0.780, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 1: 100%|██████████| 197/197 [00:03<00:00, 57.43it/s, Val/accuracy=0.648, Val/recall=1.000, Val/precision=0.581, Val/f1=0.735, Val/auc=0.866, decision_thresh=0.500]\n",
      "Epoch 2:  98%|█████████▊| 193/197 [00:03<00:00, 56.39it/s, Val/accuracy=0.648, Val/recall=1.000, Val/precision=0.581, Val/f1=0.735, Val/auc=0.866, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 2: 100%|██████████| 197/197 [00:03<00:00, 55.89it/s, Val/accuracy=0.652, Val/recall=1.000, Val/precision=0.584, Val/f1=0.737, Val/auc=0.915, decision_thresh=0.500]\n",
      "Epoch 3:  98%|█████████▊| 193/197 [00:03<00:00, 58.22it/s, Val/accuracy=0.652, Val/recall=1.000, Val/precision=0.584, Val/f1=0.737, Val/auc=0.915, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 3: 100%|██████████| 197/197 [00:03<00:00, 57.97it/s, Val/accuracy=0.740, Val/recall=0.992, Val/precision=0.654, Val/f1=0.788, Val/auc=0.933, decision_thresh=0.500]\n",
      "Epoch 4:  98%|█████████▊| 193/197 [00:03<00:00, 59.07it/s, Val/accuracy=0.740, Val/recall=0.992, Val/precision=0.654, Val/f1=0.788, Val/auc=0.933, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 4: 100%|██████████| 197/197 [00:03<00:00, 58.73it/s, Val/accuracy=0.900, Val/recall=0.926, Val/precision=0.876, Val/f1=0.900, Val/auc=0.939, decision_thresh=0.500]\n",
      "Epoch 5:  98%|█████████▊| 193/197 [00:03<00:00, 62.69it/s, Val/accuracy=0.900, Val/recall=0.926, Val/precision=0.876, Val/f1=0.900, Val/auc=0.939, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 5: 100%|██████████| 197/197 [00:03<00:00, 62.31it/s, Val/accuracy=0.896, Val/recall=0.926, Val/precision=0.869, Val/f1=0.897, Val/auc=0.940, decision_thresh=0.500]\n",
      "Epoch 6:  98%|█████████▊| 193/197 [00:03<00:00, 62.48it/s, Val/accuracy=0.896, Val/recall=0.926, Val/precision=0.869, Val/f1=0.897, Val/auc=0.940, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 6: 100%|██████████| 197/197 [00:03<00:00, 62.11it/s, Val/accuracy=0.896, Val/recall=0.926, Val/precision=0.869, Val/f1=0.897, Val/auc=0.939, decision_thresh=0.500]\n",
      "Epoch 7:  98%|█████████▊| 193/197 [00:03<00:00, 57.88it/s, Val/accuracy=0.896, Val/recall=0.926, Val/precision=0.869, Val/f1=0.897, Val/auc=0.939, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 7: 100%|██████████| 197/197 [00:03<00:00, 57.39it/s, Val/accuracy=0.904, Val/recall=0.918, Val/precision=0.889, Val/f1=0.903, Val/auc=0.939, decision_thresh=0.500]\n",
      "Epoch 8:  98%|█████████▊| 193/197 [00:03<00:00, 56.77it/s, Val/accuracy=0.904, Val/recall=0.918, Val/precision=0.889, Val/f1=0.903, Val/auc=0.939, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 8: 100%|██████████| 197/197 [00:03<00:00, 56.31it/s, Val/accuracy=0.900, Val/recall=0.918, Val/precision=0.882, Val/f1=0.900, Val/auc=0.940, decision_thresh=0.500]\n",
      "Epoch 9:  98%|█████████▊| 193/197 [00:03<00:00, 60.76it/s, Val/accuracy=0.900, Val/recall=0.918, Val/precision=0.882, Val/f1=0.900, Val/auc=0.940, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 9: 100%|██████████| 197/197 [00:03<00:00, 60.52it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.940, decision_thresh=0.500]\n",
      "Epoch 10:  98%|█████████▊| 193/197 [00:03<00:00, 60.14it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.940, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 10: 100%|██████████| 197/197 [00:03<00:00, 59.90it/s, Val/accuracy=0.900, Val/recall=0.918, Val/precision=0.882, Val/f1=0.900, Val/auc=0.939, decision_thresh=0.500]\n",
      "Epoch 11:  98%|█████████▊| 193/197 [00:03<00:00, 62.96it/s, Val/accuracy=0.900, Val/recall=0.918, Val/precision=0.882, Val/f1=0.900, Val/auc=0.939, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 11: 100%|██████████| 197/197 [00:03<00:00, 62.67it/s, Val/accuracy=0.896, Val/recall=0.910, Val/precision=0.881, Val/f1=0.895, Val/auc=0.939, decision_thresh=0.500]\n",
      "Epoch 12:  98%|█████████▊| 193/197 [00:03<00:00, 63.10it/s, Val/accuracy=0.896, Val/recall=0.910, Val/precision=0.881, Val/f1=0.895, Val/auc=0.939, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 12: 100%|██████████| 197/197 [00:03<00:00, 62.74it/s, Val/accuracy=0.896, Val/recall=0.910, Val/precision=0.881, Val/f1=0.895, Val/auc=0.939, decision_thresh=0.500]\n",
      "Epoch 13:  98%|█████████▊| 193/197 [00:03<00:00, 61.31it/s, Val/accuracy=0.896, Val/recall=0.910, Val/precision=0.881, Val/f1=0.895, Val/auc=0.939, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 13: 100%|██████████| 197/197 [00:03<00:00, 60.99it/s, Val/accuracy=0.896, Val/recall=0.910, Val/precision=0.881, Val/f1=0.895, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 14:  98%|█████████▊| 193/197 [00:02<00:00, 65.58it/s, Val/accuracy=0.896, Val/recall=0.910, Val/precision=0.881, Val/f1=0.895, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 14: 100%|██████████| 197/197 [00:03<00:00, 65.16it/s, Val/accuracy=0.896, Val/recall=0.910, Val/precision=0.881, Val/f1=0.895, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 15:  98%|█████████▊| 193/197 [00:02<00:00, 66.48it/s, Val/accuracy=0.896, Val/recall=0.910, Val/precision=0.881, Val/f1=0.895, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 15: 100%|██████████| 197/197 [00:02<00:00, 66.02it/s, Val/accuracy=0.896, Val/recall=0.902, Val/precision=0.887, Val/f1=0.894, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 16:  98%|█████████▊| 193/197 [00:02<00:00, 65.77it/s, Val/accuracy=0.896, Val/recall=0.902, Val/precision=0.887, Val/f1=0.894, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 16: 100%|██████████| 197/197 [00:03<00:00, 65.34it/s, Val/accuracy=0.896, Val/recall=0.902, Val/precision=0.887, Val/f1=0.894, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 17:  98%|█████████▊| 193/197 [00:03<00:00, 62.09it/s, Val/accuracy=0.896, Val/recall=0.902, Val/precision=0.887, Val/f1=0.894, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 17: 100%|██████████| 197/197 [00:03<00:00, 61.77it/s, Val/accuracy=0.892, Val/recall=0.893, Val/precision=0.886, Val/f1=0.890, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 18:  98%|█████████▊| 193/197 [00:03<00:00, 62.26it/s, Val/accuracy=0.892, Val/recall=0.893, Val/precision=0.886, Val/f1=0.890, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 18: 100%|██████████| 197/197 [00:03<00:00, 59.28it/s, Val/accuracy=0.900, Val/recall=0.926, Val/precision=0.876, Val/f1=0.900, Val/auc=0.936, decision_thresh=0.500]\n",
      "Epoch 19:  98%|█████████▊| 193/197 [00:03<00:00, 64.24it/s, Val/accuracy=0.900, Val/recall=0.926, Val/precision=0.876, Val/f1=0.900, Val/auc=0.936, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 19: 100%|██████████| 197/197 [00:03<00:00, 63.89it/s, Val/accuracy=0.896, Val/recall=0.893, Val/precision=0.893, Val/f1=0.893, Val/auc=0.936, decision_thresh=0.500]\n",
      "Epoch 20:  98%|█████████▊| 193/197 [00:02<00:00, 65.79it/s, Val/accuracy=0.896, Val/recall=0.893, Val/precision=0.893, Val/f1=0.893, Val/auc=0.936, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 20: 100%|██████████| 197/197 [00:03<00:00, 65.38it/s, Val/accuracy=0.900, Val/recall=0.926, Val/precision=0.876, Val/f1=0.900, Val/auc=0.936, decision_thresh=0.500]\n",
      "Epoch 21:  98%|█████████▊| 193/197 [00:02<00:00, 64.67it/s, Val/accuracy=0.900, Val/recall=0.926, Val/precision=0.876, Val/f1=0.900, Val/auc=0.936, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 21: 100%|██████████| 197/197 [00:03<00:00, 64.29it/s, Val/accuracy=0.892, Val/recall=0.910, Val/precision=0.874, Val/f1=0.892, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 22:  98%|█████████▊| 193/197 [00:02<00:00, 64.79it/s, Val/accuracy=0.892, Val/recall=0.910, Val/precision=0.874, Val/f1=0.892, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 22: 100%|██████████| 197/197 [00:03<00:00, 64.39it/s, Val/accuracy=0.884, Val/recall=0.877, Val/precision=0.884, Val/f1=0.881, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 23:  98%|█████████▊| 193/197 [00:02<00:00, 64.99it/s, Val/accuracy=0.884, Val/recall=0.877, Val/precision=0.884, Val/f1=0.881, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 23: 100%|██████████| 197/197 [00:03<00:00, 64.58it/s, Val/accuracy=0.872, Val/recall=0.836, Val/precision=0.895, Val/f1=0.864, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 24:  98%|█████████▊| 193/197 [00:02<00:00, 64.79it/s, Val/accuracy=0.872, Val/recall=0.836, Val/precision=0.895, Val/f1=0.864, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 24: 100%|██████████| 197/197 [00:03<00:00, 64.38it/s, Val/accuracy=0.896, Val/recall=0.902, Val/precision=0.887, Val/f1=0.894, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 25:  98%|█████████▊| 193/197 [00:02<00:00, 65.16it/s, Val/accuracy=0.896, Val/recall=0.902, Val/precision=0.887, Val/f1=0.894, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 25: 100%|██████████| 197/197 [00:03<00:00, 64.74it/s, Val/accuracy=0.896, Val/recall=0.910, Val/precision=0.881, Val/f1=0.895, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 26:  98%|█████████▊| 193/197 [00:03<00:00, 59.24it/s, Val/accuracy=0.896, Val/recall=0.910, Val/precision=0.881, Val/f1=0.895, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 26: 100%|██████████| 197/197 [00:03<00:00, 58.59it/s, Val/accuracy=0.892, Val/recall=0.885, Val/precision=0.893, Val/f1=0.889, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 27:  98%|█████████▊| 193/197 [00:03<00:00, 56.42it/s, Val/accuracy=0.892, Val/recall=0.885, Val/precision=0.893, Val/f1=0.889, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 27: 100%|██████████| 197/197 [00:03<00:00, 55.99it/s, Val/accuracy=0.888, Val/recall=0.877, Val/precision=0.892, Val/f1=0.884, Val/auc=0.936, decision_thresh=0.500]\n",
      "Epoch 28:  98%|█████████▊| 193/197 [00:03<00:00, 59.23it/s, Val/accuracy=0.888, Val/recall=0.877, Val/precision=0.892, Val/f1=0.884, Val/auc=0.936, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 28: 100%|██████████| 197/197 [00:03<00:00, 58.94it/s, Val/accuracy=0.892, Val/recall=0.877, Val/precision=0.899, Val/f1=0.888, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 29:  98%|█████████▊| 193/197 [00:03<00:00, 60.63it/s, Val/accuracy=0.892, Val/recall=0.877, Val/precision=0.899, Val/f1=0.888, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 29: 100%|██████████| 197/197 [00:03<00:00, 59.96it/s, Val/accuracy=0.872, Val/recall=0.828, Val/precision=0.902, Val/f1=0.863, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 30:  98%|█████████▊| 193/197 [00:03<00:00, 63.51it/s, Val/accuracy=0.872, Val/recall=0.828, Val/precision=0.902, Val/f1=0.863, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 30: 100%|██████████| 197/197 [00:03<00:00, 63.16it/s, Val/accuracy=0.892, Val/recall=0.877, Val/precision=0.899, Val/f1=0.888, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 31:  98%|█████████▊| 193/197 [00:03<00:00, 58.23it/s, Val/accuracy=0.892, Val/recall=0.877, Val/precision=0.899, Val/f1=0.888, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 31: 100%|██████████| 197/197 [00:03<00:00, 57.99it/s, Val/accuracy=0.888, Val/recall=0.869, Val/precision=0.898, Val/f1=0.883, Val/auc=0.936, decision_thresh=0.500]\n",
      "Epoch 32:  98%|█████████▊| 193/197 [00:03<00:00, 59.02it/s, Val/accuracy=0.888, Val/recall=0.869, Val/precision=0.898, Val/f1=0.883, Val/auc=0.936, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 32: 100%|██████████| 197/197 [00:03<00:00, 58.80it/s, Val/accuracy=0.888, Val/recall=0.877, Val/precision=0.892, Val/f1=0.884, Val/auc=0.936, decision_thresh=0.500]\n",
      "Epoch 33:  98%|█████████▊| 193/197 [00:03<00:00, 63.18it/s, Val/accuracy=0.888, Val/recall=0.877, Val/precision=0.892, Val/f1=0.884, Val/auc=0.936, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 33: 100%|██████████| 197/197 [00:03<00:00, 62.85it/s, Val/accuracy=0.876, Val/recall=0.836, Val/precision=0.903, Val/f1=0.868, Val/auc=0.936, decision_thresh=0.500]\n",
      "Epoch 34:  98%|█████████▊| 193/197 [00:03<00:00, 56.91it/s, Val/accuracy=0.876, Val/recall=0.836, Val/precision=0.903, Val/f1=0.868, Val/auc=0.936, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 34: 100%|██████████| 197/197 [00:03<00:00, 56.47it/s, Val/accuracy=0.876, Val/recall=0.836, Val/precision=0.903, Val/f1=0.868, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 35:  98%|█████████▊| 193/197 [00:03<00:00, 57.36it/s, Val/accuracy=0.876, Val/recall=0.836, Val/precision=0.903, Val/f1=0.868, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 35: 100%|██████████| 197/197 [00:03<00:00, 56.91it/s, Val/accuracy=0.888, Val/recall=0.877, Val/precision=0.892, Val/f1=0.884, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 36:  98%|█████████▊| 193/197 [00:03<00:00, 57.64it/s, Val/accuracy=0.888, Val/recall=0.877, Val/precision=0.892, Val/f1=0.884, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 36: 100%|██████████| 197/197 [00:03<00:00, 57.12it/s, Val/accuracy=0.884, Val/recall=0.861, Val/precision=0.897, Val/f1=0.879, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 37:  98%|█████████▊| 193/197 [00:03<00:00, 59.70it/s, Val/accuracy=0.884, Val/recall=0.861, Val/precision=0.897, Val/f1=0.879, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 37: 100%|██████████| 197/197 [00:03<00:00, 59.15it/s, Val/accuracy=0.884, Val/recall=0.869, Val/precision=0.891, Val/f1=0.880, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 38:  98%|█████████▊| 193/197 [00:03<00:00, 61.16it/s, Val/accuracy=0.884, Val/recall=0.869, Val/precision=0.891, Val/f1=0.880, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 38: 100%|██████████| 197/197 [00:03<00:00, 60.91it/s, Val/accuracy=0.888, Val/recall=0.877, Val/precision=0.892, Val/f1=0.884, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 39:  98%|█████████▊| 193/197 [00:02<00:00, 65.93it/s, Val/accuracy=0.888, Val/recall=0.877, Val/precision=0.892, Val/f1=0.884, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 39: 100%|██████████| 197/197 [00:03<00:00, 65.54it/s, Val/accuracy=0.892, Val/recall=0.885, Val/precision=0.893, Val/f1=0.889, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 40:  98%|█████████▊| 193/197 [00:02<00:00, 65.76it/s, Val/accuracy=0.892, Val/recall=0.885, Val/precision=0.893, Val/f1=0.889, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 40: 100%|██████████| 197/197 [00:03<00:00, 65.34it/s, Val/accuracy=0.892, Val/recall=0.893, Val/precision=0.886, Val/f1=0.890, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 41:  98%|█████████▊| 193/197 [00:03<00:00, 62.91it/s, Val/accuracy=0.892, Val/recall=0.893, Val/precision=0.886, Val/f1=0.890, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 41: 100%|██████████| 197/197 [00:03<00:00, 62.13it/s, Val/accuracy=0.888, Val/recall=0.869, Val/precision=0.898, Val/f1=0.883, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 42:  98%|█████████▊| 193/197 [00:02<00:00, 66.50it/s, Val/accuracy=0.888, Val/recall=0.869, Val/precision=0.898, Val/f1=0.883, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 42: 100%|██████████| 197/197 [00:02<00:00, 66.06it/s, Val/accuracy=0.888, Val/recall=0.869, Val/precision=0.898, Val/f1=0.883, Val/auc=0.938, decision_thresh=0.500]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 43:  98%|█████████▊| 193/197 [00:03<00:00, 62.35it/s, Val/accuracy=0.888, Val/recall=0.869, Val/precision=0.898, Val/f1=0.883, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 43: 100%|██████████| 197/197 [00:03<00:00, 62.05it/s, Val/accuracy=0.888, Val/recall=0.869, Val/precision=0.898, Val/f1=0.883, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 44:  98%|█████████▊| 193/197 [00:02<00:00, 65.89it/s, Val/accuracy=0.888, Val/recall=0.869, Val/precision=0.898, Val/f1=0.883, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 44: 100%|██████████| 197/197 [00:03<00:00, 65.48it/s, Val/accuracy=0.888, Val/recall=0.885, Val/precision=0.885, Val/f1=0.885, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 45:  98%|█████████▊| 193/197 [00:02<00:00, 65.74it/s, Val/accuracy=0.888, Val/recall=0.885, Val/precision=0.885, Val/f1=0.885, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 45: 100%|██████████| 197/197 [00:03<00:00, 65.29it/s, Val/accuracy=0.892, Val/recall=0.918, Val/precision=0.868, Val/f1=0.892, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 46:  98%|█████████▊| 193/197 [00:03<00:00, 63.05it/s, Val/accuracy=0.892, Val/recall=0.918, Val/precision=0.868, Val/f1=0.892, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 46: 100%|██████████| 197/197 [00:03<00:00, 62.27it/s, Val/accuracy=0.884, Val/recall=0.869, Val/precision=0.891, Val/f1=0.880, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 47:  98%|█████████▊| 193/197 [00:03<00:00, 57.42it/s, Val/accuracy=0.884, Val/recall=0.869, Val/precision=0.891, Val/f1=0.880, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 47: 100%|██████████| 197/197 [00:03<00:00, 56.83it/s, Val/accuracy=0.884, Val/recall=0.877, Val/precision=0.884, Val/f1=0.881, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 48:  98%|█████████▊| 193/197 [00:02<00:00, 64.92it/s, Val/accuracy=0.884, Val/recall=0.877, Val/precision=0.884, Val/f1=0.881, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 48: 100%|██████████| 197/197 [00:03<00:00, 64.53it/s, Val/accuracy=0.884, Val/recall=0.852, Val/precision=0.904, Val/f1=0.878, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 49:  98%|█████████▊| 193/197 [00:03<00:00, 62.33it/s, Val/accuracy=0.884, Val/recall=0.852, Val/precision=0.904, Val/f1=0.878, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 49: 100%|██████████| 197/197 [00:03<00:00, 61.56it/s, Val/accuracy=0.888, Val/recall=0.869, Val/precision=0.898, Val/f1=0.883, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 50:  98%|█████████▊| 193/197 [00:03<00:00, 61.38it/s, Val/accuracy=0.888, Val/recall=0.869, Val/precision=0.898, Val/f1=0.883, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 50: 100%|██████████| 197/197 [00:03<00:00, 61.09it/s, Val/accuracy=0.888, Val/recall=0.869, Val/precision=0.898, Val/f1=0.883, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 51:  98%|█████████▊| 193/197 [00:03<00:00, 63.37it/s, Val/accuracy=0.888, Val/recall=0.869, Val/precision=0.898, Val/f1=0.883, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 51: 100%|██████████| 197/197 [00:03<00:00, 63.01it/s, Val/accuracy=0.892, Val/recall=0.934, Val/precision=0.857, Val/f1=0.894, Val/auc=0.936, decision_thresh=0.500]\n",
      "Epoch 52:  98%|█████████▊| 193/197 [00:02<00:00, 66.07it/s, Val/accuracy=0.892, Val/recall=0.934, Val/precision=0.857, Val/f1=0.894, Val/auc=0.936, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 52: 100%|██████████| 197/197 [00:03<00:00, 65.64it/s, Val/accuracy=0.888, Val/recall=0.885, Val/precision=0.885, Val/f1=0.885, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 53:  98%|█████████▊| 193/197 [00:02<00:00, 65.52it/s, Val/accuracy=0.888, Val/recall=0.885, Val/precision=0.885, Val/f1=0.885, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 53: 100%|██████████| 197/197 [00:03<00:00, 65.10it/s, Val/accuracy=0.888, Val/recall=0.902, Val/precision=0.873, Val/f1=0.887, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 54:  98%|█████████▊| 193/197 [00:03<00:00, 59.66it/s, Val/accuracy=0.888, Val/recall=0.902, Val/precision=0.873, Val/f1=0.887, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 54: 100%|██████████| 197/197 [00:03<00:00, 59.44it/s, Val/accuracy=0.892, Val/recall=0.918, Val/precision=0.868, Val/f1=0.892, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 55:  98%|█████████▊| 193/197 [00:03<00:00, 62.81it/s, Val/accuracy=0.892, Val/recall=0.918, Val/precision=0.868, Val/f1=0.892, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 55: 100%|██████████| 197/197 [00:03<00:00, 62.50it/s, Val/accuracy=0.892, Val/recall=0.885, Val/precision=0.893, Val/f1=0.889, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 56:  98%|█████████▊| 193/197 [00:02<00:00, 65.44it/s, Val/accuracy=0.892, Val/recall=0.885, Val/precision=0.893, Val/f1=0.889, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 56: 100%|██████████| 197/197 [00:03<00:00, 65.04it/s, Val/accuracy=0.896, Val/recall=0.918, Val/precision=0.875, Val/f1=0.896, Val/auc=0.939, decision_thresh=0.500]\n",
      "Epoch 57:  98%|█████████▊| 193/197 [00:03<00:00, 61.76it/s, Val/accuracy=0.896, Val/recall=0.918, Val/precision=0.875, Val/f1=0.896, Val/auc=0.939, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 57: 100%|██████████| 197/197 [00:03<00:00, 61.42it/s, Val/accuracy=0.896, Val/recall=0.885, Val/precision=0.900, Val/f1=0.893, Val/auc=0.939, decision_thresh=0.500]\n",
      "Epoch 58:  98%|█████████▊| 193/197 [00:03<00:00, 60.14it/s, Val/accuracy=0.896, Val/recall=0.885, Val/precision=0.900, Val/f1=0.893, Val/auc=0.939, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 58: 100%|██████████| 197/197 [00:03<00:00, 59.47it/s, Val/accuracy=0.888, Val/recall=0.869, Val/precision=0.898, Val/f1=0.883, Val/auc=0.940, decision_thresh=0.500]\n",
      "Epoch 59:  98%|█████████▊| 193/197 [00:03<00:00, 56.06it/s, Val/accuracy=0.888, Val/recall=0.869, Val/precision=0.898, Val/f1=0.883, Val/auc=0.940, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 59: 100%|██████████| 197/197 [00:03<00:00, 55.64it/s, Val/accuracy=0.892, Val/recall=0.934, Val/precision=0.857, Val/f1=0.894, Val/auc=0.939, decision_thresh=0.500]\n",
      "Epoch 60:  98%|█████████▊| 193/197 [00:03<00:00, 60.47it/s, Val/accuracy=0.892, Val/recall=0.934, Val/precision=0.857, Val/f1=0.894, Val/auc=0.939, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 60: 100%|██████████| 197/197 [00:03<00:00, 60.23it/s, Val/accuracy=0.892, Val/recall=0.869, Val/precision=0.906, Val/f1=0.887, Val/auc=0.941, decision_thresh=0.500]\n",
      "Epoch 61:  98%|█████████▊| 193/197 [00:03<00:00, 61.58it/s, Val/accuracy=0.892, Val/recall=0.869, Val/precision=0.906, Val/f1=0.887, Val/auc=0.941, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 61: 100%|██████████| 197/197 [00:03<00:00, 61.28it/s, Val/accuracy=0.896, Val/recall=0.926, Val/precision=0.869, Val/f1=0.897, Val/auc=0.940, decision_thresh=0.500]\n",
      "Epoch 62:  98%|█████████▊| 193/197 [00:03<00:00, 56.28it/s, Val/accuracy=0.896, Val/recall=0.926, Val/precision=0.869, Val/f1=0.897, Val/auc=0.940, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 62: 100%|██████████| 197/197 [00:03<00:00, 56.11it/s, Val/accuracy=0.880, Val/recall=0.943, Val/precision=0.833, Val/f1=0.885, Val/auc=0.940, decision_thresh=0.500]\n",
      "Epoch 63:  98%|█████████▊| 193/197 [00:03<00:00, 59.37it/s, Val/accuracy=0.880, Val/recall=0.943, Val/precision=0.833, Val/f1=0.885, Val/auc=0.940, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 63: 100%|██████████| 197/197 [00:03<00:00, 59.13it/s, Val/accuracy=0.888, Val/recall=0.918, Val/precision=0.862, Val/f1=0.889, Val/auc=0.940, decision_thresh=0.500]\n",
      "Epoch 64:  98%|█████████▊| 193/197 [00:03<00:00, 58.78it/s, Val/accuracy=0.888, Val/recall=0.918, Val/precision=0.862, Val/f1=0.889, Val/auc=0.940, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 64: 100%|██████████| 197/197 [00:03<00:00, 58.58it/s, Val/accuracy=0.892, Val/recall=0.926, Val/precision=0.863, Val/f1=0.893, Val/auc=0.941, decision_thresh=0.500]\n",
      "Epoch 65:  98%|█████████▊| 193/197 [00:03<00:00, 58.39it/s, Val/accuracy=0.892, Val/recall=0.926, Val/precision=0.863, Val/f1=0.893, Val/auc=0.941, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 65: 100%|██████████| 197/197 [00:03<00:00, 57.39it/s, Val/accuracy=0.876, Val/recall=0.959, Val/precision=0.818, Val/f1=0.883, Val/auc=0.941, decision_thresh=0.500]\n",
      "Epoch 66:  98%|█████████▊| 193/197 [00:03<00:00, 56.72it/s, Val/accuracy=0.876, Val/recall=0.959, Val/precision=0.818, Val/f1=0.883, Val/auc=0.941, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 66: 100%|██████████| 197/197 [00:03<00:00, 56.28it/s, Val/accuracy=0.880, Val/recall=0.959, Val/precision=0.824, Val/f1=0.886, Val/auc=0.941, decision_thresh=0.500]\n",
      "Epoch 67:  98%|█████████▊| 193/197 [00:03<00:00, 57.89it/s, Val/accuracy=0.880, Val/recall=0.959, Val/precision=0.824, Val/f1=0.886, Val/auc=0.941, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 67: 100%|██████████| 197/197 [00:03<00:00, 57.71it/s, Val/accuracy=0.884, Val/recall=0.959, Val/precision=0.830, Val/f1=0.890, Val/auc=0.941, decision_thresh=0.500]\n",
      "Epoch 68:  98%|█████████▊| 193/197 [00:03<00:00, 63.00it/s, Val/accuracy=0.884, Val/recall=0.959, Val/precision=0.830, Val/f1=0.890, Val/auc=0.941, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 68: 100%|██████████| 197/197 [00:03<00:00, 62.68it/s, Val/accuracy=0.848, Val/recall=0.959, Val/precision=0.780, Val/f1=0.860, Val/auc=0.942, decision_thresh=0.500]\n",
      "Epoch 69:  98%|█████████▊| 193/197 [00:02<00:00, 65.73it/s, Val/accuracy=0.848, Val/recall=0.959, Val/precision=0.780, Val/f1=0.860, Val/auc=0.942, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 69: 100%|██████████| 197/197 [00:03<00:00, 65.33it/s, Val/accuracy=0.884, Val/recall=0.934, Val/precision=0.844, Val/f1=0.887, Val/auc=0.942, decision_thresh=0.500]\n",
      "Epoch 70:  98%|█████████▊| 193/197 [00:02<00:00, 65.92it/s, Val/accuracy=0.884, Val/recall=0.934, Val/precision=0.844, Val/f1=0.887, Val/auc=0.942, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 70: 100%|██████████| 197/197 [00:03<00:00, 65.48it/s, Val/accuracy=0.876, Val/recall=0.943, Val/precision=0.827, Val/f1=0.881, Val/auc=0.943, decision_thresh=0.500]\n",
      "Epoch 71:  98%|█████████▊| 193/197 [00:02<00:00, 65.87it/s, Val/accuracy=0.876, Val/recall=0.943, Val/precision=0.827, Val/f1=0.881, Val/auc=0.943, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 71: 100%|██████████| 197/197 [00:03<00:00, 65.40it/s, Val/accuracy=0.896, Val/recall=0.926, Val/precision=0.869, Val/f1=0.897, Val/auc=0.942, decision_thresh=0.500]\n",
      "Epoch 72:  98%|█████████▊| 193/197 [00:03<00:00, 56.94it/s, Val/accuracy=0.896, Val/recall=0.926, Val/precision=0.869, Val/f1=0.897, Val/auc=0.942, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 72: 100%|██████████| 197/197 [00:03<00:00, 56.76it/s, Val/accuracy=0.740, Val/recall=0.992, Val/precision=0.654, Val/f1=0.788, Val/auc=0.943, decision_thresh=0.500]\n",
      "Epoch 73:  98%|█████████▊| 193/197 [00:02<00:00, 64.59it/s, Val/accuracy=0.740, Val/recall=0.992, Val/precision=0.654, Val/f1=0.788, Val/auc=0.943, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 73: 100%|██████████| 197/197 [00:03<00:00, 64.21it/s, Val/accuracy=0.820, Val/recall=0.967, Val/precision=0.742, Val/f1=0.840, Val/auc=0.943, decision_thresh=0.500]\n",
      "Epoch 74:  98%|█████████▊| 193/197 [00:03<00:00, 63.32it/s, Val/accuracy=0.820, Val/recall=0.967, Val/precision=0.742, Val/f1=0.840, Val/auc=0.943, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 74: 100%|██████████| 197/197 [00:03<00:00, 62.51it/s, Val/accuracy=0.896, Val/recall=0.926, Val/precision=0.869, Val/f1=0.897, Val/auc=0.944, decision_thresh=0.500]\n",
      "Epoch 74: 100%|██████████| 197/197 [00:03<00:00, 61.08it/s, Val/accuracy=0.896, Val/recall=0.926, Val/precision=0.869, Val/f1=0.897, Val/auc=0.944, decision_thresh=0.500]\n"
     ]
    }
   ],
   "source": [
    "trainer.fit(\n",
    "    model=weasel_model,\n",
    "    datamodule=profTeacher_data_module\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "Now that Weasel has finished training, we can evaluate on the held-out test set to see how well Weasel did,\n",
    "i.e. how well the MLP end-model from above generalizes beyond the weak training signal given by the 99 LFs.\n",
    "\n",
    "That is, evaluate how good the ``predictions = end-model(X_test)`` are with respect to our gold test labels\n",
    "*Y_test*.  <br>\n",
    "Note that the LFs, *L*, and Weasel are not needed anymore after training in order to use your end-model for prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing: 100%|██████████| 189/189 [00:00<00:00, 544.59it/s]\n",
      "--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'Test/accuracy': 0.8652441049485221,\n",
      " 'Test/auc': 0.9239741370257348,\n",
      " 'Test/f1': 0.867239263803681,\n",
      " 'Test/precision': 0.8472111235416334,\n",
      " 'Test/recall': 0.8882372654155496,\n",
      " 'decision_thresh': 0.5}\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "test_stats = trainer.test(\n",
    "    datamodule=profTeacher_data_module,\n",
    "    ckpt_path='best'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# * Optional baseline *\n",
    "We can e.g. compare against [Snorkel](https://github.com/snorkel-team/snorkel)\n",
    "as a baseline. For rerunning the below, Snorkel needs to be installed (``pip install snorkel``):\n",
    "\n",
    "### Snorkel generative modeling: Predict probabilistic labels, *Y_train*, based on $L$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Eliminated noisy samples from DownstreamDataset, 2232 removed.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from weasel.datamodules.dataset_classes import BasicDownstreamDataset\n",
    "from snorkel.labeling.model.label_model import LabelModel\n",
    "\n",
    "snorkel_label_model = LabelModel(cardinality=config.datamodule.n_classes)\n",
    "label_matrix = np.array(profTeacher_data_module.ws_train_set.L)\n",
    "snorkel_label_model.fit(L_train=label_matrix)  # Snorkel only sees the label matrix while learning\n",
    "\n",
    "profTeacher_snorkel_dm = hydra_instantiate(config.datamodule)  # same as above\n",
    "# replace the (L, X)-Weasel training set in our DataModule with (X, Y_snorkel),\n",
    "#   where Y_snorkel are fixed, soft labels that Snorkel learned to predict based on L\n",
    "Y_probs_snorkel = snorkel_label_model.predict_proba(label_matrix)\n",
    "profTeacher_snorkel_dm.ws_train_set = BasicDownstreamDataset(\n",
    "    X=profTeacher_snorkel_dm.ws_train_set.X, Y=Y_probs_snorkel,\n",
    "    filter_uncertains=True  # filter_uncertains will remove those training samples where all LFs abstained,\n",
    ")                           #  which would just introduce noise if used for training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Re-training the end-model on Snorkel's *Y_train*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
      "\n",
      "  | Name    | Type       | Params | In sizes | Out sizes\n",
      "--------------------------------------------------------------\n",
      "0 | network | Sequential | 18.9 K | [1, 300] | [1, 2]   \n",
      "--------------------------------------------------------------\n",
      "18.9 K    Trainable params\n",
      "0         Non-trainable params\n",
      "18.9 K    Total params\n",
      "0.076     Total estimated model params size (MB)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data split sizes for training, validation, testing: 10062 250 12044\n",
      "                                                              "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Global seed set to 3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0:  98%|█████████▊| 158/162 [00:01<00:00, 141.02it/s, loss=0.688, Val/accuracy=0.508, Val/recall=1.000, Val/precision=0.508, Val/f1=0.674, Val/auc=0.537, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 0: 100%|██████████| 162/162 [00:01<00:00, 142.43it/s, loss=0.688, Val/accuracy=0.512, Val/recall=0.000, Val/precision=0.000, Val/f1=0.000, Val/auc=0.770, decision_thresh=0.500]\n",
      "Epoch 1:  98%|█████████▊| 158/162 [00:01<00:00, 156.76it/s, loss=0.681, Val/accuracy=0.512, Val/recall=0.000, Val/precision=0.000, Val/f1=0.000, Val/auc=0.770, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 1: 100%|██████████| 162/162 [00:01<00:00, 158.05it/s, loss=0.681, Val/accuracy=0.512, Val/recall=0.000, Val/precision=0.000, Val/f1=0.000, Val/auc=0.862, decision_thresh=0.500]\n",
      "Epoch 2:  98%|█████████▊| 158/162 [00:01<00:00, 152.31it/s, loss=0.676, Val/accuracy=0.512, Val/recall=0.000, Val/precision=0.000, Val/f1=0.000, Val/auc=0.862, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 2: 100%|██████████| 162/162 [00:01<00:00, 153.57it/s, loss=0.676, Val/accuracy=0.512, Val/recall=0.000, Val/precision=0.000, Val/f1=0.000, Val/auc=0.906, decision_thresh=0.500]\n",
      "Epoch 3:  98%|█████████▊| 158/162 [00:01<00:00, 155.60it/s, loss=0.648, Val/accuracy=0.512, Val/recall=0.000, Val/precision=0.000, Val/f1=0.000, Val/auc=0.906, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 3: 100%|██████████| 162/162 [00:01<00:00, 156.96it/s, loss=0.648, Val/accuracy=0.512, Val/recall=0.000, Val/precision=0.000, Val/f1=0.000, Val/auc=0.929, decision_thresh=0.500]\n",
      "Epoch 4:  98%|█████████▊| 158/162 [00:01<00:00, 157.02it/s, loss=0.578, Val/accuracy=0.512, Val/recall=0.000, Val/precision=0.000, Val/f1=0.000, Val/auc=0.929, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 4: 100%|██████████| 162/162 [00:01<00:00, 158.25it/s, loss=0.578, Val/accuracy=0.600, Val/recall=0.205, Val/precision=0.893, Val/f1=0.333, Val/auc=0.935, decision_thresh=0.500]\n",
      "Epoch 5:  98%|█████████▊| 158/162 [00:00<00:00, 158.42it/s, loss=0.481, Val/accuracy=0.600, Val/recall=0.205, Val/precision=0.893, Val/f1=0.333, Val/auc=0.935, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 5: 100%|██████████| 162/162 [00:01<00:00, 159.84it/s, loss=0.481, Val/accuracy=0.820, Val/recall=0.664, Val/precision=0.953, Val/f1=0.783, Val/auc=0.941, decision_thresh=0.500]\n",
      "Epoch 6:  98%|█████████▊| 158/162 [00:00<00:00, 158.56it/s, loss=0.417, Val/accuracy=0.820, Val/recall=0.664, Val/precision=0.953, Val/f1=0.783, Val/auc=0.941, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 6: 100%|██████████| 162/162 [00:01<00:00, 159.91it/s, loss=0.417, Val/accuracy=0.872, Val/recall=0.811, Val/precision=0.917, Val/f1=0.861, Val/auc=0.942, decision_thresh=0.500]\n",
      "Epoch 7:  98%|█████████▊| 158/162 [00:01<00:00, 142.06it/s, loss=0.402, Val/accuracy=0.872, Val/recall=0.811, Val/precision=0.917, Val/f1=0.861, Val/auc=0.942, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 7: 100%|██████████| 162/162 [00:01<00:00, 143.64it/s, loss=0.402, Val/accuracy=0.864, Val/recall=0.811, Val/precision=0.900, Val/f1=0.853, Val/auc=0.940, decision_thresh=0.500]\n",
      "Epoch 8:  98%|█████████▊| 158/162 [00:01<00:00, 138.32it/s, loss=0.382, Val/accuracy=0.864, Val/recall=0.811, Val/precision=0.900, Val/f1=0.853, Val/auc=0.940, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 8: 100%|██████████| 162/162 [00:01<00:00, 139.69it/s, loss=0.382, Val/accuracy=0.856, Val/recall=0.820, Val/precision=0.877, Val/f1=0.847, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 9:  98%|█████████▊| 158/162 [00:01<00:00, 137.95it/s, loss=0.376, Val/accuracy=0.856, Val/recall=0.820, Val/precision=0.877, Val/f1=0.847, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 9: 100%|██████████| 162/162 [00:01<00:00, 139.38it/s, loss=0.376, Val/accuracy=0.860, Val/recall=0.828, Val/precision=0.878, Val/f1=0.852, Val/auc=0.936, decision_thresh=0.500]\n",
      "Epoch 10:  98%|█████████▊| 158/162 [00:01<00:00, 156.24it/s, loss=0.358, Val/accuracy=0.860, Val/recall=0.828, Val/precision=0.878, Val/f1=0.852, Val/auc=0.936, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 10: 100%|██████████| 162/162 [00:01<00:00, 157.57it/s, loss=0.358, Val/accuracy=0.864, Val/recall=0.820, Val/precision=0.893, Val/f1=0.855, Val/auc=0.934, decision_thresh=0.500]\n",
      "Epoch 11:  98%|█████████▊| 158/162 [00:01<00:00, 155.71it/s, loss=0.351, Val/accuracy=0.864, Val/recall=0.820, Val/precision=0.893, Val/f1=0.855, Val/auc=0.934, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 11: 100%|██████████| 162/162 [00:01<00:00, 156.97it/s, loss=0.351, Val/accuracy=0.860, Val/recall=0.836, Val/precision=0.872, Val/f1=0.854, Val/auc=0.933, decision_thresh=0.500]\n",
      "Epoch 12:  98%|█████████▊| 158/162 [00:01<00:00, 137.28it/s, loss=0.376, Val/accuracy=0.860, Val/recall=0.836, Val/precision=0.872, Val/f1=0.854, Val/auc=0.933, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 12: 100%|██████████| 162/162 [00:01<00:00, 138.67it/s, loss=0.376, Val/accuracy=0.868, Val/recall=0.828, Val/precision=0.894, Val/f1=0.860, Val/auc=0.932, decision_thresh=0.500]\n",
      "Epoch 13:  98%|█████████▊| 158/162 [00:01<00:00, 154.25it/s, loss=0.369, Val/accuracy=0.868, Val/recall=0.828, Val/precision=0.894, Val/f1=0.860, Val/auc=0.932, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 13: 100%|██████████| 162/162 [00:01<00:00, 155.64it/s, loss=0.369, Val/accuracy=0.864, Val/recall=0.828, Val/precision=0.886, Val/f1=0.856, Val/auc=0.931, decision_thresh=0.500]\n",
      "Epoch 14:  98%|█████████▊| 158/162 [00:01<00:00, 156.90it/s, loss=0.362, Val/accuracy=0.864, Val/recall=0.828, Val/precision=0.886, Val/f1=0.856, Val/auc=0.931, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 14: 100%|██████████| 162/162 [00:01<00:00, 158.27it/s, loss=0.362, Val/accuracy=0.860, Val/recall=0.828, Val/precision=0.878, Val/f1=0.852, Val/auc=0.930, decision_thresh=0.500]\n",
      "Epoch 15:  98%|█████████▊| 158/162 [00:00<00:00, 158.59it/s, loss=0.352, Val/accuracy=0.860, Val/recall=0.828, Val/precision=0.878, Val/f1=0.852, Val/auc=0.930, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 15: 100%|██████████| 162/162 [00:01<00:00, 159.96it/s, loss=0.352, Val/accuracy=0.860, Val/recall=0.820, Val/precision=0.885, Val/f1=0.851, Val/auc=0.930, decision_thresh=0.500]\n",
      "Epoch 16:  98%|█████████▊| 158/162 [00:01<00:00, 136.46it/s, loss=0.365, Val/accuracy=0.860, Val/recall=0.820, Val/precision=0.885, Val/f1=0.851, Val/auc=0.930, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 16: 100%|██████████| 162/162 [00:01<00:00, 137.90it/s, loss=0.365, Val/accuracy=0.864, Val/recall=0.828, Val/precision=0.886, Val/f1=0.856, Val/auc=0.929, decision_thresh=0.500]\n",
      "Epoch 17:  98%|█████████▊| 158/162 [00:01<00:00, 133.95it/s, loss=0.363, Val/accuracy=0.864, Val/recall=0.828, Val/precision=0.886, Val/f1=0.856, Val/auc=0.929, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 17: 100%|██████████| 162/162 [00:01<00:00, 134.94it/s, loss=0.363, Val/accuracy=0.864, Val/recall=0.820, Val/precision=0.893, Val/f1=0.855, Val/auc=0.928, decision_thresh=0.500]\n",
      "Epoch 18:  98%|█████████▊| 158/162 [00:01<00:00, 148.16it/s, loss=0.341, Val/accuracy=0.864, Val/recall=0.820, Val/precision=0.893, Val/f1=0.855, Val/auc=0.928, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 18: 100%|██████████| 162/162 [00:01<00:00, 149.76it/s, loss=0.341, Val/accuracy=0.864, Val/recall=0.820, Val/precision=0.893, Val/f1=0.855, Val/auc=0.928, decision_thresh=0.500]\n",
      "Epoch 19:  98%|█████████▊| 158/162 [00:01<00:00, 153.56it/s, loss=0.355, Val/accuracy=0.864, Val/recall=0.820, Val/precision=0.893, Val/f1=0.855, Val/auc=0.928, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 19: 100%|██████████| 162/162 [00:01<00:00, 152.75it/s, loss=0.355, Val/accuracy=0.864, Val/recall=0.820, Val/precision=0.893, Val/f1=0.855, Val/auc=0.927, decision_thresh=0.500]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20:  98%|█████████▊| 158/162 [00:01<00:00, 135.82it/s, loss=0.352, Val/accuracy=0.864, Val/recall=0.820, Val/precision=0.893, Val/f1=0.855, Val/auc=0.927, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 20: 100%|██████████| 162/162 [00:01<00:00, 135.98it/s, loss=0.352, Val/accuracy=0.868, Val/recall=0.820, Val/precision=0.901, Val/f1=0.858, Val/auc=0.926, decision_thresh=0.500]\n",
      "Epoch 21:  98%|█████████▊| 158/162 [00:01<00:00, 132.25it/s, loss=0.336, Val/accuracy=0.868, Val/recall=0.820, Val/precision=0.901, Val/f1=0.858, Val/auc=0.926, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 21: 100%|██████████| 162/162 [00:01<00:00, 133.27it/s, loss=0.336, Val/accuracy=0.864, Val/recall=0.820, Val/precision=0.893, Val/f1=0.855, Val/auc=0.926, decision_thresh=0.500]\n",
      "Epoch 22:  98%|█████████▊| 158/162 [00:01<00:00, 136.63it/s, loss=0.345, Val/accuracy=0.864, Val/recall=0.820, Val/precision=0.893, Val/f1=0.855, Val/auc=0.926, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 22: 100%|██████████| 162/162 [00:01<00:00, 137.65it/s, loss=0.345, Val/accuracy=0.864, Val/recall=0.820, Val/precision=0.893, Val/f1=0.855, Val/auc=0.925, decision_thresh=0.500]\n",
      "Epoch 23:  98%|█████████▊| 158/162 [00:01<00:00, 137.05it/s, loss=0.352, Val/accuracy=0.864, Val/recall=0.820, Val/precision=0.893, Val/f1=0.855, Val/auc=0.925, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 23: 100%|██████████| 162/162 [00:01<00:00, 138.48it/s, loss=0.352, Val/accuracy=0.860, Val/recall=0.811, Val/precision=0.892, Val/f1=0.850, Val/auc=0.924, decision_thresh=0.500]\n",
      "Epoch 24:  98%|█████████▊| 158/162 [00:00<00:00, 167.97it/s, loss=0.341, Val/accuracy=0.860, Val/recall=0.811, Val/precision=0.892, Val/f1=0.850, Val/auc=0.924, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 24: 100%|██████████| 162/162 [00:00<00:00, 169.00it/s, loss=0.341, Val/accuracy=0.860, Val/recall=0.811, Val/precision=0.892, Val/f1=0.850, Val/auc=0.924, decision_thresh=0.500]\n",
      "Epoch 25:  98%|█████████▊| 158/162 [00:01<00:00, 156.88it/s, loss=0.333, Val/accuracy=0.860, Val/recall=0.811, Val/precision=0.892, Val/f1=0.850, Val/auc=0.924, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 25: 100%|██████████| 162/162 [00:01<00:00, 158.22it/s, loss=0.333, Val/accuracy=0.868, Val/recall=0.828, Val/precision=0.894, Val/f1=0.860, Val/auc=0.924, decision_thresh=0.500]\n",
      "Epoch 26:  98%|█████████▊| 158/162 [00:01<00:00, 154.91it/s, loss=0.347, Val/accuracy=0.868, Val/recall=0.828, Val/precision=0.894, Val/f1=0.860, Val/auc=0.924, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 26: 100%|██████████| 162/162 [00:01<00:00, 156.22it/s, loss=0.347, Val/accuracy=0.860, Val/recall=0.811, Val/precision=0.892, Val/f1=0.850, Val/auc=0.924, decision_thresh=0.500]\n",
      "Epoch 27:  98%|█████████▊| 158/162 [00:01<00:00, 136.78it/s, loss=0.341, Val/accuracy=0.860, Val/recall=0.811, Val/precision=0.892, Val/f1=0.850, Val/auc=0.924, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 27: 100%|██████████| 162/162 [00:01<00:00, 138.19it/s, loss=0.341, Val/accuracy=0.864, Val/recall=0.820, Val/precision=0.893, Val/f1=0.855, Val/auc=0.924, decision_thresh=0.500]\n",
      "Epoch 28:  98%|█████████▊| 158/162 [00:01<00:00, 155.54it/s, loss=0.33, Val/accuracy=0.864, Val/recall=0.820, Val/precision=0.893, Val/f1=0.855, Val/auc=0.924, decision_thresh=0.500] \n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 28: 100%|██████████| 162/162 [00:01<00:00, 156.64it/s, loss=0.33, Val/accuracy=0.860, Val/recall=0.811, Val/precision=0.892, Val/f1=0.850, Val/auc=0.922, decision_thresh=0.500]\n",
      "Epoch 29:  98%|█████████▊| 158/162 [00:01<00:00, 154.17it/s, loss=0.344, Val/accuracy=0.860, Val/recall=0.811, Val/precision=0.892, Val/f1=0.850, Val/auc=0.922, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 29: 100%|██████████| 162/162 [00:01<00:00, 155.68it/s, loss=0.344, Val/accuracy=0.856, Val/recall=0.803, Val/precision=0.891, Val/f1=0.845, Val/auc=0.921, decision_thresh=0.500]\n",
      "Epoch 30:  98%|█████████▊| 158/162 [00:01<00:00, 155.75it/s, loss=0.339, Val/accuracy=0.856, Val/recall=0.803, Val/precision=0.891, Val/f1=0.845, Val/auc=0.921, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 30: 100%|██████████| 162/162 [00:01<00:00, 157.07it/s, loss=0.339, Val/accuracy=0.864, Val/recall=0.828, Val/precision=0.886, Val/f1=0.856, Val/auc=0.921, decision_thresh=0.500]\n",
      "Epoch 31:  98%|█████████▊| 158/162 [00:01<00:00, 153.69it/s, loss=0.33, Val/accuracy=0.864, Val/recall=0.828, Val/precision=0.886, Val/f1=0.856, Val/auc=0.921, decision_thresh=0.500] \n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 31: 100%|██████████| 162/162 [00:01<00:00, 154.86it/s, loss=0.33, Val/accuracy=0.864, Val/recall=0.820, Val/precision=0.893, Val/f1=0.855, Val/auc=0.921, decision_thresh=0.500]\n",
      "Epoch 32:  98%|█████████▊| 158/162 [00:01<00:00, 147.90it/s, loss=0.338, Val/accuracy=0.864, Val/recall=0.820, Val/precision=0.893, Val/f1=0.855, Val/auc=0.921, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 32: 100%|██████████| 162/162 [00:01<00:00, 148.57it/s, loss=0.338, Val/accuracy=0.852, Val/recall=0.795, Val/precision=0.890, Val/f1=0.840, Val/auc=0.921, decision_thresh=0.500]\n",
      "Epoch 33:  98%|█████████▊| 158/162 [00:01<00:00, 136.45it/s, loss=0.331, Val/accuracy=0.852, Val/recall=0.795, Val/precision=0.890, Val/f1=0.840, Val/auc=0.921, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 33: 100%|██████████| 162/162 [00:01<00:00, 137.92it/s, loss=0.331, Val/accuracy=0.860, Val/recall=0.811, Val/precision=0.892, Val/f1=0.850, Val/auc=0.921, decision_thresh=0.500]\n",
      "Epoch 34:  98%|█████████▊| 158/162 [00:01<00:00, 155.85it/s, loss=0.338, Val/accuracy=0.860, Val/recall=0.811, Val/precision=0.892, Val/f1=0.850, Val/auc=0.921, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 34: 100%|██████████| 162/162 [00:01<00:00, 156.98it/s, loss=0.338, Val/accuracy=0.864, Val/recall=0.820, Val/precision=0.893, Val/f1=0.855, Val/auc=0.921, decision_thresh=0.500]\n",
      "Epoch 35:  98%|█████████▊| 158/162 [00:01<00:00, 137.68it/s, loss=0.355, Val/accuracy=0.864, Val/recall=0.820, Val/precision=0.893, Val/f1=0.855, Val/auc=0.921, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 35: 100%|██████████| 162/162 [00:01<00:00, 139.02it/s, loss=0.355, Val/accuracy=0.864, Val/recall=0.828, Val/precision=0.886, Val/f1=0.856, Val/auc=0.920, decision_thresh=0.500]\n",
      "Epoch 36:  98%|█████████▊| 158/162 [00:01<00:00, 142.58it/s, loss=0.352, Val/accuracy=0.864, Val/recall=0.828, Val/precision=0.886, Val/f1=0.856, Val/auc=0.920, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 36: 100%|██████████| 162/162 [00:01<00:00, 143.37it/s, loss=0.352, Val/accuracy=0.860, Val/recall=0.811, Val/precision=0.892, Val/f1=0.850, Val/auc=0.920, decision_thresh=0.500]\n",
      "Epoch 37:  98%|█████████▊| 158/162 [00:01<00:00, 140.73it/s, loss=0.319, Val/accuracy=0.860, Val/recall=0.811, Val/precision=0.892, Val/f1=0.850, Val/auc=0.920, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 37: 100%|██████████| 162/162 [00:01<00:00, 141.98it/s, loss=0.319, Val/accuracy=0.864, Val/recall=0.828, Val/precision=0.886, Val/f1=0.856, Val/auc=0.921, decision_thresh=0.500]\n",
      "Epoch 38:  98%|█████████▊| 158/162 [00:01<00:00, 153.22it/s, loss=0.325, Val/accuracy=0.864, Val/recall=0.828, Val/precision=0.886, Val/f1=0.856, Val/auc=0.921, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 38: 100%|██████████| 162/162 [00:01<00:00, 154.47it/s, loss=0.325, Val/accuracy=0.852, Val/recall=0.803, Val/precision=0.883, Val/f1=0.841, Val/auc=0.920, decision_thresh=0.500]\n",
      "Epoch 39:  98%|█████████▊| 158/162 [00:01<00:00, 153.01it/s, loss=0.32, Val/accuracy=0.852, Val/recall=0.803, Val/precision=0.883, Val/f1=0.841, Val/auc=0.920, decision_thresh=0.500] \n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 39: 100%|██████████| 162/162 [00:01<00:00, 154.43it/s, loss=0.32, Val/accuracy=0.860, Val/recall=0.820, Val/precision=0.885, Val/f1=0.851, Val/auc=0.920, decision_thresh=0.500]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 40:  98%|█████████▊| 158/162 [00:01<00:00, 152.96it/s, loss=0.32, Val/accuracy=0.860, Val/recall=0.820, Val/precision=0.885, Val/f1=0.851, Val/auc=0.920, decision_thresh=0.500] \n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 40: 100%|██████████| 162/162 [00:01<00:00, 154.33it/s, loss=0.32, Val/accuracy=0.860, Val/recall=0.820, Val/precision=0.885, Val/f1=0.851, Val/auc=0.919, decision_thresh=0.500]\n",
      "Epoch 41:  98%|█████████▊| 158/162 [00:01<00:00, 131.09it/s, loss=0.331, Val/accuracy=0.860, Val/recall=0.820, Val/precision=0.885, Val/f1=0.851, Val/auc=0.919, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 41: 100%|██████████| 162/162 [00:01<00:00, 131.95it/s, loss=0.331, Val/accuracy=0.860, Val/recall=0.820, Val/precision=0.885, Val/f1=0.851, Val/auc=0.920, decision_thresh=0.500]\n",
      "Epoch 42:  98%|█████████▊| 158/162 [00:01<00:00, 133.42it/s, loss=0.322, Val/accuracy=0.860, Val/recall=0.820, Val/precision=0.885, Val/f1=0.851, Val/auc=0.920, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 42: 100%|██████████| 162/162 [00:01<00:00, 134.81it/s, loss=0.322, Val/accuracy=0.852, Val/recall=0.803, Val/precision=0.883, Val/f1=0.841, Val/auc=0.920, decision_thresh=0.500]\n",
      "Epoch 43:  98%|█████████▊| 158/162 [00:01<00:00, 133.47it/s, loss=0.339, Val/accuracy=0.852, Val/recall=0.803, Val/precision=0.883, Val/f1=0.841, Val/auc=0.920, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 43: 100%|██████████| 162/162 [00:01<00:00, 134.82it/s, loss=0.339, Val/accuracy=0.856, Val/recall=0.811, Val/precision=0.884, Val/f1=0.846, Val/auc=0.920, decision_thresh=0.500]\n",
      "Epoch 44:  98%|█████████▊| 158/162 [00:01<00:00, 130.21it/s, loss=0.32, Val/accuracy=0.856, Val/recall=0.811, Val/precision=0.884, Val/f1=0.846, Val/auc=0.920, decision_thresh=0.500] \n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 44: 100%|██████████| 162/162 [00:01<00:00, 131.05it/s, loss=0.32, Val/accuracy=0.856, Val/recall=0.811, Val/precision=0.884, Val/f1=0.846, Val/auc=0.919, decision_thresh=0.500]\n",
      "Epoch 45:  98%|█████████▊| 158/162 [00:01<00:00, 139.89it/s, loss=0.312, Val/accuracy=0.856, Val/recall=0.811, Val/precision=0.884, Val/f1=0.846, Val/auc=0.919, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 45: 100%|██████████| 162/162 [00:01<00:00, 141.44it/s, loss=0.312, Val/accuracy=0.852, Val/recall=0.803, Val/precision=0.883, Val/f1=0.841, Val/auc=0.919, decision_thresh=0.500]\n",
      "Epoch 46:  98%|█████████▊| 158/162 [00:01<00:00, 154.76it/s, loss=0.319, Val/accuracy=0.852, Val/recall=0.803, Val/precision=0.883, Val/f1=0.841, Val/auc=0.919, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 46: 100%|██████████| 162/162 [00:01<00:00, 156.15it/s, loss=0.319, Val/accuracy=0.852, Val/recall=0.803, Val/precision=0.883, Val/f1=0.841, Val/auc=0.918, decision_thresh=0.500]\n",
      "Epoch 47:  98%|█████████▊| 158/162 [00:00<00:00, 164.27it/s, loss=0.318, Val/accuracy=0.852, Val/recall=0.803, Val/precision=0.883, Val/f1=0.841, Val/auc=0.918, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 47: 100%|██████████| 162/162 [00:00<00:00, 165.61it/s, loss=0.318, Val/accuracy=0.844, Val/recall=0.787, Val/precision=0.881, Val/f1=0.831, Val/auc=0.918, decision_thresh=0.500]\n",
      "Epoch 48:  98%|█████████▊| 158/162 [00:00<00:00, 168.87it/s, loss=0.336, Val/accuracy=0.844, Val/recall=0.787, Val/precision=0.881, Val/f1=0.831, Val/auc=0.918, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 48: 100%|██████████| 162/162 [00:00<00:00, 170.08it/s, loss=0.336, Val/accuracy=0.856, Val/recall=0.811, Val/precision=0.884, Val/f1=0.846, Val/auc=0.918, decision_thresh=0.500]\n",
      "Epoch 49:  98%|█████████▊| 158/162 [00:01<00:00, 139.72it/s, loss=0.325, Val/accuracy=0.856, Val/recall=0.811, Val/precision=0.884, Val/f1=0.846, Val/auc=0.918, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 49: 100%|██████████| 162/162 [00:01<00:00, 140.56it/s, loss=0.325, Val/accuracy=0.856, Val/recall=0.811, Val/precision=0.884, Val/f1=0.846, Val/auc=0.918, decision_thresh=0.500]\n",
      "Epoch 50:  98%|█████████▊| 158/162 [00:01<00:00, 135.55it/s, loss=0.308, Val/accuracy=0.856, Val/recall=0.811, Val/precision=0.884, Val/f1=0.846, Val/auc=0.918, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 50: 100%|██████████| 162/162 [00:01<00:00, 136.28it/s, loss=0.308, Val/accuracy=0.860, Val/recall=0.820, Val/precision=0.885, Val/f1=0.851, Val/auc=0.918, decision_thresh=0.500]\n",
      "Epoch 51:  98%|█████████▊| 158/162 [00:01<00:00, 129.90it/s, loss=0.311, Val/accuracy=0.860, Val/recall=0.820, Val/precision=0.885, Val/f1=0.851, Val/auc=0.918, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 51: 100%|██████████| 162/162 [00:01<00:00, 130.81it/s, loss=0.311, Val/accuracy=0.852, Val/recall=0.803, Val/precision=0.883, Val/f1=0.841, Val/auc=0.917, decision_thresh=0.500]\n",
      "Epoch 52:  98%|█████████▊| 158/162 [00:01<00:00, 132.98it/s, loss=0.315, Val/accuracy=0.852, Val/recall=0.803, Val/precision=0.883, Val/f1=0.841, Val/auc=0.917, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 52: 100%|██████████| 162/162 [00:01<00:00, 134.31it/s, loss=0.315, Val/accuracy=0.860, Val/recall=0.820, Val/precision=0.885, Val/f1=0.851, Val/auc=0.917, decision_thresh=0.500]\n",
      "Epoch 53:  98%|█████████▊| 158/162 [00:01<00:00, 156.30it/s, loss=0.326, Val/accuracy=0.860, Val/recall=0.820, Val/precision=0.885, Val/f1=0.851, Val/auc=0.917, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 53: 100%|██████████| 162/162 [00:01<00:00, 157.70it/s, loss=0.326, Val/accuracy=0.852, Val/recall=0.803, Val/precision=0.883, Val/f1=0.841, Val/auc=0.917, decision_thresh=0.500]\n",
      "Epoch 54:  98%|█████████▊| 158/162 [00:01<00:00, 156.15it/s, loss=0.319, Val/accuracy=0.852, Val/recall=0.803, Val/precision=0.883, Val/f1=0.841, Val/auc=0.917, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 54: 100%|██████████| 162/162 [00:01<00:00, 157.57it/s, loss=0.319, Val/accuracy=0.852, Val/recall=0.803, Val/precision=0.883, Val/f1=0.841, Val/auc=0.916, decision_thresh=0.500]\n",
      "Epoch 55:  98%|█████████▊| 158/162 [00:00<00:00, 160.46it/s, loss=0.314, Val/accuracy=0.852, Val/recall=0.803, Val/precision=0.883, Val/f1=0.841, Val/auc=0.916, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 55: 100%|██████████| 162/162 [00:01<00:00, 161.83it/s, loss=0.314, Val/accuracy=0.848, Val/recall=0.795, Val/precision=0.882, Val/f1=0.836, Val/auc=0.916, decision_thresh=0.500]\n",
      "Epoch 56:  98%|█████████▊| 158/162 [00:01<00:00, 138.60it/s, loss=0.313, Val/accuracy=0.848, Val/recall=0.795, Val/precision=0.882, Val/f1=0.836, Val/auc=0.916, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 56: 100%|██████████| 162/162 [00:01<00:00, 139.53it/s, loss=0.313, Val/accuracy=0.852, Val/recall=0.803, Val/precision=0.883, Val/f1=0.841, Val/auc=0.917, decision_thresh=0.500]\n",
      "Epoch 57:  98%|█████████▊| 158/162 [00:01<00:00, 156.90it/s, loss=0.322, Val/accuracy=0.852, Val/recall=0.803, Val/precision=0.883, Val/f1=0.841, Val/auc=0.917, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 57: 100%|██████████| 162/162 [00:01<00:00, 158.37it/s, loss=0.322, Val/accuracy=0.852, Val/recall=0.803, Val/precision=0.883, Val/f1=0.841, Val/auc=0.917, decision_thresh=0.500]\n",
      "Epoch 58:  98%|█████████▊| 158/162 [00:00<00:00, 160.16it/s, loss=0.3, Val/accuracy=0.852, Val/recall=0.803, Val/precision=0.883, Val/f1=0.841, Val/auc=0.917, decision_thresh=0.500]  \n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 58: 100%|██████████| 162/162 [00:01<00:00, 161.60it/s, loss=0.3, Val/accuracy=0.844, Val/recall=0.779, Val/precision=0.888, Val/f1=0.830, Val/auc=0.917, decision_thresh=0.500]\n",
      "Epoch 59:  98%|█████████▊| 158/162 [00:00<00:00, 160.49it/s, loss=0.324, Val/accuracy=0.844, Val/recall=0.779, Val/precision=0.888, Val/f1=0.830, Val/auc=0.917, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 59: 100%|██████████| 162/162 [00:01<00:00, 161.89it/s, loss=0.324, Val/accuracy=0.852, Val/recall=0.795, Val/precision=0.890, Val/f1=0.840, Val/auc=0.917, decision_thresh=0.500]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 60:  98%|█████████▊| 158/162 [00:01<00:00, 157.73it/s, loss=0.308, Val/accuracy=0.852, Val/recall=0.795, Val/precision=0.890, Val/f1=0.840, Val/auc=0.917, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 60: 100%|██████████| 162/162 [00:01<00:00, 159.25it/s, loss=0.308, Val/accuracy=0.848, Val/recall=0.787, Val/precision=0.889, Val/f1=0.835, Val/auc=0.917, decision_thresh=0.500]\n",
      "Epoch 61:  98%|█████████▊| 158/162 [00:01<00:00, 138.53it/s, loss=0.324, Val/accuracy=0.848, Val/recall=0.787, Val/precision=0.889, Val/f1=0.835, Val/auc=0.917, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 61: 100%|██████████| 162/162 [00:01<00:00, 139.37it/s, loss=0.324, Val/accuracy=0.848, Val/recall=0.795, Val/precision=0.882, Val/f1=0.836, Val/auc=0.917, decision_thresh=0.500]\n",
      "Epoch 62:  98%|█████████▊| 158/162 [00:01<00:00, 151.56it/s, loss=0.315, Val/accuracy=0.848, Val/recall=0.795, Val/precision=0.882, Val/f1=0.836, Val/auc=0.917, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 62: 100%|██████████| 162/162 [00:01<00:00, 153.03it/s, loss=0.315, Val/accuracy=0.840, Val/recall=0.779, Val/precision=0.880, Val/f1=0.826, Val/auc=0.917, decision_thresh=0.500]\n",
      "Epoch 63:  98%|█████████▊| 158/162 [00:01<00:00, 145.92it/s, loss=0.308, Val/accuracy=0.840, Val/recall=0.779, Val/precision=0.880, Val/f1=0.826, Val/auc=0.917, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 63: 100%|██████████| 162/162 [00:01<00:00, 146.67it/s, loss=0.308, Val/accuracy=0.840, Val/recall=0.770, Val/precision=0.887, Val/f1=0.825, Val/auc=0.916, decision_thresh=0.500]\n",
      "Epoch 64:  98%|█████████▊| 158/162 [00:01<00:00, 147.15it/s, loss=0.31, Val/accuracy=0.840, Val/recall=0.770, Val/precision=0.887, Val/f1=0.825, Val/auc=0.916, decision_thresh=0.500] \n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 64: 100%|██████████| 162/162 [00:01<00:00, 148.33it/s, loss=0.31, Val/accuracy=0.856, Val/recall=0.811, Val/precision=0.884, Val/f1=0.846, Val/auc=0.915, decision_thresh=0.500]\n",
      "Epoch 65:  98%|█████████▊| 158/162 [00:01<00:00, 144.02it/s, loss=0.315, Val/accuracy=0.856, Val/recall=0.811, Val/precision=0.884, Val/f1=0.846, Val/auc=0.915, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 65: 100%|██████████| 162/162 [00:01<00:00, 144.71it/s, loss=0.315, Val/accuracy=0.840, Val/recall=0.770, Val/precision=0.887, Val/f1=0.825, Val/auc=0.916, decision_thresh=0.500]\n",
      "Epoch 66:  98%|█████████▊| 158/162 [00:01<00:00, 147.05it/s, loss=0.305, Val/accuracy=0.840, Val/recall=0.770, Val/precision=0.887, Val/f1=0.825, Val/auc=0.916, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 66: 100%|██████████| 162/162 [00:01<00:00, 148.51it/s, loss=0.305, Val/accuracy=0.856, Val/recall=0.811, Val/precision=0.884, Val/f1=0.846, Val/auc=0.916, decision_thresh=0.500]\n",
      "Epoch 67:  98%|█████████▊| 158/162 [00:01<00:00, 153.74it/s, loss=0.314, Val/accuracy=0.856, Val/recall=0.811, Val/precision=0.884, Val/f1=0.846, Val/auc=0.916, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 67: 100%|██████████| 162/162 [00:01<00:00, 155.04it/s, loss=0.314, Val/accuracy=0.868, Val/recall=0.836, Val/precision=0.887, Val/f1=0.861, Val/auc=0.916, decision_thresh=0.500]\n",
      "Epoch 68:  98%|█████████▊| 158/162 [00:01<00:00, 155.91it/s, loss=0.311, Val/accuracy=0.868, Val/recall=0.836, Val/precision=0.887, Val/f1=0.861, Val/auc=0.916, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 68: 100%|██████████| 162/162 [00:01<00:00, 157.24it/s, loss=0.311, Val/accuracy=0.856, Val/recall=0.803, Val/precision=0.891, Val/f1=0.845, Val/auc=0.915, decision_thresh=0.500]\n",
      "Epoch 69:  98%|█████████▊| 158/162 [00:01<00:00, 136.49it/s, loss=0.305, Val/accuracy=0.856, Val/recall=0.803, Val/precision=0.891, Val/f1=0.845, Val/auc=0.915, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 69: 100%|██████████| 162/162 [00:01<00:00, 137.87it/s, loss=0.305, Val/accuracy=0.852, Val/recall=0.795, Val/precision=0.890, Val/f1=0.840, Val/auc=0.916, decision_thresh=0.500]\n",
      "Epoch 70:  98%|█████████▊| 158/162 [00:01<00:00, 146.74it/s, loss=0.318, Val/accuracy=0.852, Val/recall=0.795, Val/precision=0.890, Val/f1=0.840, Val/auc=0.916, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 70: 100%|██████████| 162/162 [00:01<00:00, 147.99it/s, loss=0.318, Val/accuracy=0.860, Val/recall=0.820, Val/precision=0.885, Val/f1=0.851, Val/auc=0.915, decision_thresh=0.500]\n",
      "Epoch 71:  98%|█████████▊| 158/162 [00:01<00:00, 135.23it/s, loss=0.306, Val/accuracy=0.860, Val/recall=0.820, Val/precision=0.885, Val/f1=0.851, Val/auc=0.915, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 71: 100%|██████████| 162/162 [00:01<00:00, 136.12it/s, loss=0.306, Val/accuracy=0.848, Val/recall=0.787, Val/precision=0.889, Val/f1=0.835, Val/auc=0.916, decision_thresh=0.500]\n",
      "Epoch 72:  98%|█████████▊| 158/162 [00:01<00:00, 143.44it/s, loss=0.304, Val/accuracy=0.848, Val/recall=0.787, Val/precision=0.889, Val/f1=0.835, Val/auc=0.916, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 72: 100%|██████████| 162/162 [00:01<00:00, 144.82it/s, loss=0.304, Val/accuracy=0.864, Val/recall=0.828, Val/precision=0.886, Val/f1=0.856, Val/auc=0.915, decision_thresh=0.500]\n",
      "Epoch 73:  98%|█████████▊| 158/162 [00:01<00:00, 156.42it/s, loss=0.302, Val/accuracy=0.864, Val/recall=0.828, Val/precision=0.886, Val/f1=0.856, Val/auc=0.915, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 73: 100%|██████████| 162/162 [00:01<00:00, 157.86it/s, loss=0.302, Val/accuracy=0.868, Val/recall=0.820, Val/precision=0.901, Val/f1=0.858, Val/auc=0.916, decision_thresh=0.500]\n",
      "Epoch 74:  98%|█████████▊| 158/162 [00:01<00:00, 155.60it/s, loss=0.319, Val/accuracy=0.868, Val/recall=0.820, Val/precision=0.901, Val/f1=0.858, Val/auc=0.916, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 74: 100%|██████████| 162/162 [00:01<00:00, 156.94it/s, loss=0.319, Val/accuracy=0.856, Val/recall=0.795, Val/precision=0.898, Val/f1=0.843, Val/auc=0.917, decision_thresh=0.500]\n",
      "Epoch 74: 100%|██████████| 162/162 [00:01<00:00, 156.25it/s, loss=0.319, Val/accuracy=0.856, Val/recall=0.795, Val/precision=0.898, Val/f1=0.843, Val/auc=0.917, decision_thresh=0.500]\n"
     ]
    }
   ],
   "source": [
    "# re-initiate the same MLP that Weasel trained + the same Trainer:\n",
    "MLP_end_model = hydra_instantiate(config.end_model)\n",
    "checkpoint_callback = ModelCheckpoint(monitor=\"Val/auc\", mode=\"max\")\n",
    "\n",
    "trainer = hydra_instantiate(\n",
    "        config.trainer, callbacks=checkpoint_callback, deterministic=True, max_epochs=75\n",
    ")\n",
    "# Besides the training set, the only difference is the model:\n",
    "trainer.fit(model=MLP_end_model, datamodule=profTeacher_snorkel_dm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Evaluating our end-model trained with Snorkel instead of Weasel\n",
    "You can see below that Weasel improved performance by about 3 F1 points over Snorkel!\n",
    "<br>We observed such strong performances of Weasel to happen pretty consistently across dataset and LF configurations (see paper).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing: 100%|██████████| 189/189 [00:00<00:00, 560.49it/s]\n",
      "--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'Test/accuracy': 0.8522915974759216,\n",
      " 'Test/auc': 0.9291996970456435,\n",
      " 'Test/f1': 0.8405485345522989,\n",
      " 'Test/precision': 0.9036423202929273,\n",
      " 'Test/recall': 0.7856903485254692,\n",
      " 'decision_thresh': 0.5}\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "snorkel_test_stats = trainer.test(datamodule=profTeacher_snorkel_dm, ckpt_path='best')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "E2E_FULL",
   "language": "python",
   "name": "e2efull"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}