examples/0_full_pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 334,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 335,
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 336,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Global seed set to 7\n"
     ]
    },
    {
     "data": {
      "text/plain": "7"
     },
     "execution_count": 336,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import pytorch_lightning as pl\n",
    "from pytorch_lightning import seed_everything\n",
    "seed_everything(seed=7)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Full pipeline for a new WeaSEL problem\n",
    "**This tutorial walks you through how to\n",
    "make Weasel run on *your own***:\n",
    " - ***Data***\n",
    " - ***Set of Labeling functions***\n",
    " - ***End-model***\n",
    "\n",
    "by completing the full pipeline on a synthetic example."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Simulating the data...\n",
    "In this tutorial, the data and LFs have no semantics attached, and are completely synthetic.\n",
    "This notebook is meant as an illustration for how you would go about using Weasel\n",
    "on a new problem.\n",
    "\n",
    "Therefore we will just create random data features. We will however assume that the data are image-like, and thus\n",
    "call for a custom end-model (a CNN).\n",
    "\n",
    "**Note:**\n",
    "***For sake of clarity we will make most important arguments/parameters that\n",
    "need/can be defined explicit. In practice though, you will likely want to make an analogous Yaml config\n",
    "(e.g. with [this template](../configs/template.yaml)) for your\n",
    "problem like [this one](configs/profTeacher_full.yaml) and use Hydra to read & modify it + instantiate\n",
    "all modules like in [this example notebook](1_bias_bios.ipynb).***"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 337,
   "outputs": [],
   "source": [
    "n, n_evaluation = 10_000, 1_000  # number of training and test samples\n",
    "n_channels = 3  #  e.g. could be RGB\n",
    "height = width = 28  # grid resolution\n",
    "\n",
    "X_train = np.random.randn(n, n_channels, height, width)\n",
    "X_test = np.random.randn(n_evaluation, n_channels, height, width)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "As this is a WeaSEL problem, we ***do not know any ground truth training labels***.\n",
    "For evaluation purposes we will usually want to have access to a small gold-labeled test set.\n",
    "To simulate this part of the pipeline we will also generate such labels,\n",
    "assuming that there are $C=3$ classes.\n",
    "Note though, that this is not needed to train Weasel."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 338,
   "outputs": [],
   "source": [
    "C = 3\n",
    "possible_labels = list(range(C))\n",
    "Y_test = np.random.choice(possible_labels, size=n_evaluation)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "As in the whole library, we assume that you have a label matrix $L \\in \\{-1, 0, .., C-1\\}^{n\\times m}$\n",
    "available. Here:\n",
    " - $n$ is the number of training samples\n",
    " - $m$ is the number of labeling functions (LF)/heuristics/rules\n",
    " - $C$ is the number of classes\n",
    " - $L_{i,j} = -1$ means that LF $j$ abstained from labeling example $i$.\n",
    "\n",
    "If your problem is not yet at this stage, e.g. no LFs have been defined or applied to the training set,\n",
    "you'll have to start with that. The [Snorkel library](https://github.com/snorkel-team/snorkel)\n",
    "is a neat library for this step of the pipeline.\n",
    "\n",
    "Now, we will create 10 synthetic LF (without semantics), assuming\n",
    "that all LFs abstain 85% of the time, while voting for one of the three classes uniformly at random.\n",
    "\n",
    "Of course, in a real setting LFs will depend on the data and most likely not be independent of each other as below."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 339,
   "outputs": [
    {
     "data": {
      "text/plain": "(10000, 10)"
     },
     "execution_count": 339,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = 10\n",
    "ABSTAIN = -1\n",
    "\n",
    "possible_LF_outputs = [ABSTAIN] + list(range(C))\n",
    "label_matrix = np.empty((n, m))\n",
    "for LF in range(m):\n",
    "    label_matrix[:, LF] = np.random.choice(\n",
    "        possible_LF_outputs, size=n, p=[0.85] + [(1 - 0.85)*1/C for _ in range(C)]\n",
    "    )\n",
    "\n",
    "label_matrix.shape"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# From data to DataModule\n",
    "Having checked off the raw data components, we now have to\n",
    "map them to a format usable by a pl.Trainer. We recommend to either subclass our\n",
    "[AbstractWeaselDataModule](../weasel/datamodules/base_datamodule.py) (a specific ``pl.LightningDataModule``\n",
    "suitable for training Weasel, see [ProfTeacher_DataModule](datamodules/ProfTeacher_datamodule.py)),\n",
    " or simply passing the raw components to BasicWeaselDataModule as below."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 340,
   "outputs": [],
   "source": [
    "from weasel.datamodules.base_datamodule import BasicWeaselDataModule\n",
    "weasel_datamodule = BasicWeaselDataModule(\n",
    "    label_matrix=label_matrix,\n",
    "    X_train=X_train,\n",
    "    X_test=X_test,\n",
    "    Y_test=Y_test,\n",
    "    batch_size=256,\n",
    "    val_test_split=(200, 800)  # 200 validation, 800 test points will be split from (X_test, Y_test)\n",
    ")\n",
    "        "
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Defining an End-model\n",
    "\n",
    "Having set up the data part, you'll now have to choose your favorite neural net as the end-model <br>\n",
    " (the one that you want to use as ``predictions = end-model(X)`` eventually).\n",
    "\n",
    "Here, since we are simulating image-like features, we will be using a toy CNN.\n",
    " To do so, just subclass the ``DownstreamBaseModel`` abstract class like you would any nn.Module (and override any methods as needed) as below.\n",
    " You can analogously define any other fancy neural net as the end-model.\n",
    "\n",
    " While you only have to override the ``__init__`` and ``forward`` methods, ``DownstreamBaseModel``\n",
    " actually is a appropriately defined LightningModule that allows you to easily [run baselines without Weasel](../weasel/models/downstream_models/README.md).\n"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 355,
   "outputs": [],
   "source": [
    "from weasel.models.downstream_models.base_model import DownstreamBaseModel\n",
    "class MyCNN(DownstreamBaseModel):\n",
    "    def __init__(self, in_channels,\n",
    "                 hidden_dim,\n",
    "                 conv_layers: int,\n",
    "                 n_classes: int,\n",
    "                 kernel_size=(3, 3),\n",
    "                 *args, **kwargs):\n",
    "        super().__init__()\n",
    "        # Good practice:\n",
    "        self.out_dim = n_classes\n",
    "        self.example_input_array = torch.randn((1, in_channels, height, width))\n",
    "\n",
    "        cnn_modules = []\n",
    "\n",
    "        in_dim = in_channels\n",
    "        for layer in range(conv_layers):\n",
    "            cnn_modules += [\n",
    "                nn.Conv2d(in_dim, hidden_dim, kernel_size),\n",
    "                nn.GELU(),\n",
    "                nn.MaxPool2d(2, 2)\n",
    "            ]\n",
    "            in_dim = hidden_dim\n",
    "\n",
    "        self.convs = nn.Sequential(*cnn_modules)\n",
    "\n",
    "        self.flattened_dim = torch.flatten(\n",
    "            self.convs(self.example_input_array), start_dim=1\n",
    "        ).shape[1]\n",
    "\n",
    "        mlp_modules = [\n",
    "            nn.Linear(self.flattened_dim, int(self.flattened_dim/2)),\n",
    "            nn.GELU()\n",
    "        ]\n",
    "        mlp_modules += [nn.Linear(int(self.flattened_dim/2), n_classes)]\n",
    "        self.readout = nn.Sequential(*mlp_modules)\n",
    "\n",
    "    def forward(self, X: torch.Tensor, readout=True):\n",
    "        conv_out = self.convs(X)\n",
    "        flattened = torch.flatten(conv_out, start_dim=1)\n",
    "        if not readout:\n",
    "            return flattened\n",
    "        logits = self.readout(flattened)\n",
    "        return logits # We predict the raw logits in forward!\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 342,
   "outputs": [],
   "source": [
    "cnn_end_model = MyCNN(in_channels=n_channels, hidden_dim=16, conv_layers=2, n_classes=C)\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Coupling end-model into Weasel\n",
    "Now that we have the data and end-model defined, we just need to pass them\n",
    "to Weasel as follows:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 343,
   "outputs": [],
   "source": [
    "from weasel.models import Weasel\n",
    "weasel = Weasel(\n",
    "    end_model=cnn_end_model,\n",
    "    num_LFs=m,\n",
    "    n_classes=C,\n",
    "    encoder={'hidden_dims': [32, 10]},\n",
    "    optim_encoder={'name': 'adam', 'lr': 1e-4},\n",
    "    optim_end_model=['_target=torch.optim.Adam', 'lr=1e-4']  # different way of getting the same optim with Hydra\n",
    ")\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Training Weasel and end-model\n",
    "\n",
    "Before fitting Weasel and the end-model, we now just need to instantiate a pl.Trainer instance\n",
    "(we will checkpoint the best model w.r.t. F1-macro performance on a small validation set that is split off the test set, although\n",
    "this of course makes little sense in this simulated example)."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 344,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: False, used: False\n",
      "TPU available: False, using: 0 TPU cores\n",
      "\n",
      "  | Name          | Type       | Params\n",
      "---------------------------------------------\n",
      "0 | end_model     | MyCNN      | 83.6 K\n",
      "1 | encoder       | MLPEncoder | 1.1 K \n",
      "2 | accuracy_func | Softmax    | 0     \n",
      "---------------------------------------------\n",
      "84.7 K    Trainable params\n",
      "0         Non-trainable params\n",
      "84.7 K    Total params\n",
      "0.339     Total estimated model params size (MB)\n",
      "Global seed set to 7\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data split sizes for training, validation, testing: 10000 200 800\n"
     ]
    },
    {
     "data": {
      "text/plain": "Validation sanity check: 0it [00:00, ?it/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "0671d92034394f99b11af5955022d393"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Training: 0it [00:00, ?it/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "6bd475f87e164566979d9af565468b86"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Validating: 0it [00:00, ?it/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "015aff122014412f96e70e1098dd4d97"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Validating: 0it [00:00, ?it/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "f8635026945c4f93bf3a2e3bc7a4553c"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Validating: 0it [00:00, ?it/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "d484f5970add461b8f5fa92bbdf6abbc"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pytorch_lightning.callbacks import ModelCheckpoint\n",
    "checkpoint_callback = ModelCheckpoint(monitor=\"Val/f1_macro\", mode=\"max\")\n",
    "\n",
    "trainer = pl.Trainer(\n",
    "    gpus=0,  # >= 1 to use GPU(s)\n",
    "    max_epochs=3,  # since just for illustratory purposes\n",
    "    logger=False,\n",
    "    deterministic=True,\n",
    "    callbacks=[checkpoint_callback]\n",
    ")\n",
    "\n",
    "trainer.fit(model=weasel, datamodule=weasel_datamodule)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Evaluation\n",
    "\n",
    "Now that Weasel has finished training, we can evaluate on the held-out test set to see how well Weasel did,\n",
    "That is, evaluate how good the ``predictions = end-model(X_test)`` are with respect to our gold test labels\n",
    "*Y_test*.\n",
    "<br>Note that the LFs, *L*, and Weasel are not needed anymore after training/for prediction.\n",
    "Indeed, *we will retrieve the best CNN end-model from the saved Weasel checkpoint*, which can now be used\n",
    "for predictions on the image-like features only."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 345,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: False, used: False\n",
      "TPU available: False, using: 0 TPU cores\n"
     ]
    },
    {
     "data": {
      "text/plain": "Testing: 0it [00:00, ?it/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "a6fce4bbe2c24e6e99d1df0ce67a9e20"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'Test/accuracy': 0.35875,\n",
      " 'Test/brier': 0.33304962510873465,\n",
      " 'Test/f1_macro': 0.24905422953533293,\n",
      " 'Test/f1_micro': 0.35875}\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# The below will give the same test results\n",
    "# test_stats = trainer.test(datamodule=weasel_datamodule, ckpt_path='best')\n",
    "\n",
    "final_cnn_model = weasel.load_from_checkpoint(\n",
    "    trainer.checkpoint_callback.best_model_path\n",
    ").end_model\n",
    "# Test the stand-alone, fully-trained CNN model (the metrics have of course no meaning in this simulated example):\n",
    "test_statd = pl.Trainer().test(model=final_cnn_model, test_dataloaders=weasel_datamodule.test_dataloader())\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "##  Using features as auxiliary input to the encoder\n",
    "Weasel can also use some form of the data features as auxiliary input beyond the LFs for the encoder model.\n",
    "In our paper we found this to consistently lead to slightly better performances.\n",
    "We'll now briefly show how to make this happen for our simulated problem and CNN.\n",
    "For that, we first need to define which features the encoder will use. The default, and currently only supported encoder\n",
    "is a MLP - when not using auxiliary features it predicts per ``MLP(L)``, while when using additional features\n",
    "they are concatenated to $L$.\n",
    "Therefore, we'll have to somehow return flattened/one-dimensional features in the ``get_encoder_features(.)``\n",
    "method. One option could be to just flatten the input image-like data across all dimensions.\n",
    "The other option is to return an intermediate representation, here we return the inputs right between\n",
    "the convolutions and the readout MLP of our CNN:"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "source": [
    "class MyCNN(MyCNN):\n",
    "    def get_encoder_features(self, X):\n",
    "        return self(X, readout=False).detach()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "execution_count": 356,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now we just have to set the ``use_aux_input_for_encoder=True`` flag and the encoder will automatically\n",
    "include the input ``get_encoder_features(.)`` for prediction."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 358,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: False, used: False\n",
      "TPU available: False, using: 0 TPU cores\n",
      "\n",
      "  | Name          | Type       | Params\n",
      "---------------------------------------------\n",
      "0 | end_model     | MyCNN      | 83.6 K\n",
      "1 | encoder       | MLPEncoder | 13.9 K\n",
      "2 | accuracy_func | Softmax    | 0     \n",
      "---------------------------------------------\n",
      "97.5 K    Trainable params\n",
      "0         Non-trainable params\n",
      "97.5 K    Total params\n",
      "0.390     Total estimated model params size (MB)\n",
      "Global seed set to 7\n"
     ]
    },
    {
     "data": {
      "text/plain": "Validation sanity check: 0it [00:00, ?it/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "b734e4c83c0e425b9a22e6393910fc3b"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Training: 0it [00:00, ?it/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "e2f1cd59a4fe471b82308de6532207aa"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Validating: 0it [00:00, ?it/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "6a6e7cbb7d9544afa7b219d0c440aff9"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Validating: 0it [00:00, ?it/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "a033fccf113844c5abd06c777ede35fb"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Validating: 0it [00:00, ?it/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "d8b9d242da1c452ca3a5b8010e98d1ec"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "cnn_end_model2 = MyCNN(in_channels=n_channels, hidden_dim=16, conv_layers=2, n_classes=C)\n",
    "\n",
    "weasel2 = Weasel(\n",
    "    end_model=cnn_end_model2,\n",
    "    num_LFs=m,\n",
    "    n_classes=C,\n",
    "    use_aux_input_for_encoder=True,\n",
    "    encoder={'hidden_dims': [32, 10]},\n",
    "    optim_encoder={'name': 'adam', 'lr': 1e-4},\n",
    "    optim_end_model={'name': 'adam', 'lr': 1e-4}\n",
    ")\n",
    "trainer = pl.Trainer(\n",
    "    gpus=0,  # >= 1 to use GPU(s)\n",
    "    max_epochs=3,  # since just for illustratory purposes\n",
    "    logger=False,\n",
    "    deterministic=True,\n",
    "    callbacks=[checkpoint_callback]\n",
    ")\n",
    "\n",
    "trainer.fit(model=weasel2, datamodule=weasel_datamodule)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Note how the number of parameters in the encoder is way higher than before, since the\n",
    "input shape now includes the whole flattened hidden CNN representation.\n",
    "<br>Funnily, it actually improves performance on this simulated example! :D\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 360,
   "outputs": [
    {
     "data": {
      "text/plain": "Testing: 0it [00:00, ?it/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "072b625f15d142899bc56366ac877cab"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------------------\n",
      "DATALOADER:0 TEST RESULTS\n",
      "{'Test/accuracy': 0.37875,\n",
      " 'Test/brier': 0.3327239181449149,\n",
      " 'Test/f1_macro': 0.25437041512531283,\n",
      " 'Test/f1_micro': 0.37875}\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "test_stats = trainer.test(datamodule=weasel_datamodule, ckpt_path='best')\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "pycharm-cddfe24d",
   "language": "python",
   "display_name": "PyCharm (Neural_Weak_Supervision)"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}