tutorials/How_to_create_an_ML_model_card.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "**Disclaimer**. This example uses the Evidently API as available in version 0.6.7 or lower. Please ensure you are using the correct version when running this notebook. For updated and new examples using the latest Evidently versions, visit our documentation. \n",
        "\n",
        "Evidently docs: https://docs.evidentlyai.com/\n",
        "\n",
        "Join our Discord: https://discord.com/invite/xZjKRaNp8b"
      ],
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        },
        "id": "6eM1BMvDwgee"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Install Evidently following the instructions for your environment: https://docs.evidentlyai.com/user-guide/install-evidently"
      ],
      "metadata": {
        "id": "BSnjNfczOxXg"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wDEp8XLyTVMl"
      },
      "outputs": [],
      "source": [
        "try:\n",
        "    import evidently\n",
        "except:\n",
        "    !pip install evidently==0.3.3"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "from sklearn import datasets, ensemble\n",
        "\n",
        "from evidently.report import Report\n",
        "\n",
        "from evidently.metrics import *"
      ],
      "metadata": {
        "id": "wAK2ki_SULhx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Prepare toy data\n",
        "\n",
        "Import a toy dataset and fit a simple model. You will get two resulting datasets: `reference` (training dataset) and `current` (test dataset)."
      ],
      "metadata": {
        "id": "EAu79kWcoiS_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Dataset for Binary Probabilistic Classifcation\n",
        "bcancer_data = datasets.load_breast_cancer(as_frame='auto')\n",
        "bcancer = bcancer_data.frame\n",
        "\n",
        "bcancer_ref = bcancer.sample(n=300, replace=False)\n",
        "bcancer_cur = bcancer.sample(n=200, replace=False)\n",
        "\n",
        "bcancer_label_ref = bcancer_ref.copy(deep=True)\n",
        "bcancer_label_cur = bcancer_cur.copy(deep=True)\n",
        "\n",
        "model = ensemble.RandomForestClassifier(random_state=1, n_estimators=10)\n",
        "model.fit(bcancer_ref[bcancer_data.feature_names.tolist()], bcancer_ref.target)\n",
        "\n",
        "bcancer_ref['prediction'] = model.predict_proba(bcancer_ref[bcancer_data.feature_names.tolist()])[:, 1]\n",
        "bcancer_cur['prediction'] = model.predict_proba(bcancer_cur[bcancer_data.feature_names.tolist()])[:, 1]"
      ],
      "metadata": {
        "id": "BqPLP5cTpBMy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Preview the two datasets."
      ],
      "metadata": {
        "id": "Fks80uicpSGT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "bcancer_ref.head()"
      ],
      "metadata": {
        "id": "hEJ5RK1JTv1G"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "bcancer_cur.head()"
      ],
      "metadata": {
        "id": "7UWNKgEhUb3m"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Design the template text fields"
      ],
      "metadata": {
        "id": "o5kM2nckq8f-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "These are the text fields that will appear in the model card."
      ],
      "metadata": {
        "id": "J-64eFXPKz5Q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model_details = \"\"\"\n",
        "  # Model Details\n",
        "\n",
        "  ## Description\n",
        "  * Model name: What is the model name?\n",
        "  * Model ID: Include model ID.\n",
        "  * Model version\n",
        "  * Model author: Who created the model?\n",
        "  * Model type: What is the model doing?\n",
        "  * Model architecture: Include any relevant information about algorithms, parameters, etc.\n",
        "  * Date\n",
        "  * License\n",
        "  * Contact details\n",
        "\n",
        "  ## Intended use\n",
        "  * Primary use case: What is the use case?\n",
        "  * Model users: Who are the expected model users?\n",
        "  * Secondary use cases: What are the secondary use cases, if any?\n",
        "  * Out of scope: What applications are out of scope?\n",
        "\"\"\""
      ],
      "metadata": {
        "id": "eq7gbQnvU0iq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "training_dataset = \"\"\"\n",
        "  # Training dataset\n",
        "\n",
        "  * Training dataset: How was the training dataset created?\n",
        "  * Training period: From which time period the training dataset comes from?\n",
        "  * Sub-groups: are there relevant categories, e.g. demographic?\n",
        "  * Limitations: Are there known limitations?\n",
        "  * Pre-processing: How was the data pre-processed?\n",
        "\"\"\""
      ],
      "metadata": {
        "id": "PZchWF3Er02H"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_evaluation = \"\"\"\n",
        "  # Model evaluation\n",
        "\n",
        "  * Evaluation process: How was the model evaluated?\n",
        "  * Evaluation dataset: How was the evaluation dataset created?\n",
        "  * Metrics: What are the key model quality metrics?\n",
        "  * Decision threshold: What is the decision threshold?\n",
        "\"\"\""
      ],
      "metadata": {
        "id": "LIcTFgW-slfP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "considerations = \"\"\"\n",
        "  # Ethical considerations\n",
        "Include relevant considerations.\n",
        "\n",
        "  # Caveats and Recommendations\n",
        "Include relevant considerations.\n",
        "\"\"\""
      ],
      "metadata": {
        "id": "w167xbf6uIjL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Run model card template"
      ],
      "metadata": {
        "id": "JcZog30puZGZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This Model Card report includes:\n",
        "* text fields implemented earlier\n",
        "* plots and visualizations on data and model quality\n",
        "\n",
        "The plots and text fields are listed in the same order as they will appear on the model card.\n",
        "\n",
        "Let's run it to see how it looks!"
      ],
      "metadata": {
        "id": "bP5PmI9vzz3F"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model_card = Report(metrics=[\n",
        "    Comment(model_details),\n",
        "    ClassificationClassBalance(),\n",
        "    Comment(training_dataset),\n",
        "    DatasetSummaryMetric(),\n",
        "    Comment(model_evaluation),\n",
        "    ClassificationQualityMetric(),\n",
        "    ClassificationConfusionMatrix(),\n",
        "    Comment(considerations),\n",
        "])\n",
        "\n",
        "model_card.run(current_data=bcancer_cur, reference_data=bcancer_ref)\n",
        "model_card"
      ],
      "metadata": {
        "id": "G5rYds0rUeCw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Populate and customize the template"
      ],
      "metadata": {
        "id": "xNEliAkE0QRA"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1. Populate text fields"
      ],
      "metadata": {
        "id": "zfZMflvL0a_k"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "First, let's populate the text fields. You can exclude components, modify the proposed structure, and use markdown for additional formatting."
      ],
      "metadata": {
        "id": "EOIb_GoA0U5a"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model_details = \"\"\"\n",
        "  # Model Details\n",
        "\n",
        "  ## Description\n",
        "  * **Model name**: Test model\n",
        "  * **Model author**: Evidently AI team\n",
        "  * **Model type**: Probabilistic classification model.\n",
        "  * **Model architecture**: Random forest.\n",
        "  * **Date**: June 2023.\n",
        "\n",
        "  ## Intended use\n",
        "  * **Primary use case**: Demonstration of how to create the ML model card.\n",
        "  * **Out of scope**: This model is not intended for production use.\n",
        "\"\"\""
      ],
      "metadata": {
        "id": "uwSbgPnUU5vb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "training_dataset = \"\"\"\n",
        "  # Training dataset\n",
        "\n",
        "  * **Training dataset**: breast cancer wisconsin dataset, 300 randomly sampled objects.\n",
        "  * **Source**: dataset from [sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html)\n",
        "  * **Limitations**: Demo dataset.\n",
        "\"\"\""
      ],
      "metadata": {
        "id": "2uybzMGW1LsR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_evaluation = \"\"\"\n",
        "  # Model evaluation\n",
        "\n",
        "  * **Evaluation dataset**: breast cancer wisconsin dataset, 300 randomly sampled objects.\n",
        "  * **Metrics**: ROC AUC, accuracy, precision, recall.\n",
        "  * **Decision threshold**: 0.5, objects with predicted probability over 0.5 belong to the target class.\n",
        "\"\"\""
      ],
      "metadata": {
        "id": "nCDH0VwH1xUh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "considerations = \"\"\"\n",
        "  # Caveats and Recommendations\n",
        "Not for production use.\n",
        "\"\"\""
      ],
      "metadata": {
        "id": "Pt0ZpTCL5IEO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's see how the model card looks now!"
      ],
      "metadata": {
        "id": "Ab__2XYz4FQs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model_card = Report(metrics=[\n",
        "    Comment(model_details),\n",
        "    ClassificationClassBalance(),\n",
        "    Comment(training_dataset),\n",
        "    DatasetSummaryMetric(),\n",
        "    Comment(model_evaluation),\n",
        "    ClassificationQualityMetric(),\n",
        "    ClassificationConfusionMatrix(),\n",
        "    Comment(considerations)\n",
        "])\n",
        "\n",
        "model_card.run(current_data=bcancer_cur, reference_data=bcancer_ref)\n",
        "model_card"
      ],
      "metadata": {
        "id": "lu5W-kBO2opW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_card.save_html(\"sample_data/file.html\")"
      ],
      "metadata": {
        "id": "5TOHHiIXZsC3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2. Change the composition of plots"
      ],
      "metadata": {
        "id": "N5zvmimG4KSk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can modify which plots appear on the model card. You can select from multiple \"Metrics\" available in the Evidently library.\n",
        "You can see the Metric list here: https://docs.evidentlyai.com/reference/all-metrics, or browse the example notebooks (if you open Colab, you can see pre-rendered plots and can select those you like: https://docs.evidentlyai.com/examples)."
      ],
      "metadata": {
        "id": "Gwg9bnkG4N82"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here are the changes we will make now:\n",
        "* Add plots on **dataset correlations**.\n",
        "* Add plots to show the **stats for the features we consider important** to highlight (\"mean radius,\" \"mean symmetry\").\n",
        "* Add a couple of new plots related to **Probabilistic Classification Quality**: distribution of predicted probabilities, ROC Curve.\n",
        "* Add a table showing the **alternative classification decision thresholds**, and comment about the ability to modify it.\n"
      ],
      "metadata": {
        "id": "1DHC6g-h4q6v"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "threshold_comment = \"\"\"\n",
        "  **Note**: The model quality metrics are generated using 0.5 decision threshold. It is important to consider the possibility of changing the decision threshold. Here are the alternative considerations:\n",
        "\"\"\""
      ],
      "metadata": {
        "id": "pOT3iAe3CNHj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_card = Report(metrics=[\n",
        "    Comment(model_details),\n",
        "    ClassificationClassBalance(),\n",
        "    Comment(training_dataset),\n",
        "    DatasetSummaryMetric(),\n",
        "    DatasetCorrelationsMetric(),\n",
        "    ColumnSummaryMetric(column_name=\"mean radius\"),\n",
        "    ColumnSummaryMetric(column_name=\"mean symmetry\"),\n",
        "    Comment(model_evaluation),\n",
        "    ClassificationQualityMetric(),\n",
        "    ClassificationConfusionMatrix(),\n",
        "    ClassificationProbDistribution(),\n",
        "    ClassificationRocCurve(),\n",
        "    Comment(threshold_comment),\n",
        "    ClassificationPRTable(),\n",
        "    Comment(considerations)\n",
        "])\n",
        "\n",
        "model_card.run(current_data=bcancer_cur, reference_data=bcancer_ref)\n",
        "model_card"
      ],
      "metadata": {
        "id": "mYnOVxVYB-iq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Support Evidently"
      ],
      "metadata": {
        "id": "RHGNuHNXAi1k"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Did you find the example useful? Star Evidently on GitHub to contribute back! This helps us continue creating free open-source tools for the community. https://github.com/evidentlyai/evidently"
      ],
      "metadata": {
        "id": "NGRzanpWAnQi"
      }
    }
  ]
}