Decision Trees.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "57a2b787",
   "metadata": {},
   "source": [
    "# Breast Cancer Detection with Decision Trees"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1febc38",
   "metadata": {},
   "source": [
    "Hi Guys 😀\n",
    "\n",
    "In this notebook, I'll cover the following topics,\n",
    "- What are decision trees?\n",
    "- Some advantages of decision trees\n",
    "- Some disadvantages of decision trees\n",
    "- Data preprocessing\n",
    "- Building the model\n",
    "- Model evaluation\n",
    "- Hyperparameter tuning with grid search <br/>\n",
    "\n",
    "You can follow us on [Tirendaz Academy](https://youtube.com/c/tirendazacademy) YouTube channel 👍\n",
    "\n",
    "Happy learning 🐱‍🏍 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c02280d4",
   "metadata": {},
   "source": [
    "# What are decision trees?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1447e59c",
   "metadata": {},
   "source": [
    "Decision trees are a non-parametric supervised learning. This technique is widely used for classification and regression tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1e52566",
   "metadata": {},
   "source": [
    "# Some advantages of decision trees\n",
    "- Decision trees are simple to understand and interpret. \n",
    "- You can easily visualize trees.\n",
    "- Decision trees require little data preprocessing. \n",
    "- You can deal with both numerical and categorical data using this technique."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfae66e2",
   "metadata": {},
   "source": [
    "# Some disadvantages of decision trees"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dddf4ccf",
   "metadata": {},
   "source": [
    "- Decision tree learners can create over-complex trees that do not generalize the data well. \n",
    "\n",
    "To overcome this problem, you can use some methods such as setting the maximum depth of the tree, setting the minimum number of samples required at a leaf node, and pruning.\n",
    "\n",
    "- Decision trees can be unstable.\n",
    "\n",
    "To avoid this problem, you can use decision trees within an ensemble."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "109f9db8",
   "metadata": {},
   "source": [
    "# Loading the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "642f6280",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>diagnosis</th>\n",
       "      <th>radius_mean</th>\n",
       "      <th>texture_mean</th>\n",
       "      <th>perimeter_mean</th>\n",
       "      <th>area_mean</th>\n",
       "      <th>smoothness_mean</th>\n",
       "      <th>compactness_mean</th>\n",
       "      <th>concavity_mean</th>\n",
       "      <th>concave points_mean</th>\n",
       "      <th>...</th>\n",
       "      <th>texture_worst</th>\n",
       "      <th>perimeter_worst</th>\n",
       "      <th>area_worst</th>\n",
       "      <th>smoothness_worst</th>\n",
       "      <th>compactness_worst</th>\n",
       "      <th>concavity_worst</th>\n",
       "      <th>concave points_worst</th>\n",
       "      <th>symmetry_worst</th>\n",
       "      <th>fractal_dimension_worst</th>\n",
       "      <th>Unnamed: 32</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>842302</td>\n",
       "      <td>M</td>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>...</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>842517</td>\n",
       "      <td>M</td>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.0869</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>...</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>84300903</td>\n",
       "      <td>M</td>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.1974</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>...</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>84348301</td>\n",
       "      <td>M</td>\n",
       "      <td>11.42</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.2414</td>\n",
       "      <td>0.10520</td>\n",
       "      <td>...</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.2098</td>\n",
       "      <td>0.8663</td>\n",
       "      <td>0.6869</td>\n",
       "      <td>0.2575</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>84358402</td>\n",
       "      <td>M</td>\n",
       "      <td>20.29</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.1980</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>...</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 33 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         id diagnosis  radius_mean  texture_mean  perimeter_mean  area_mean  \\\n",
       "0    842302         M        17.99         10.38          122.80     1001.0   \n",
       "1    842517         M        20.57         17.77          132.90     1326.0   \n",
       "2  84300903         M        19.69         21.25          130.00     1203.0   \n",
       "3  84348301         M        11.42         20.38           77.58      386.1   \n",
       "4  84358402         M        20.29         14.34          135.10     1297.0   \n",
       "\n",
       "   smoothness_mean  compactness_mean  concavity_mean  concave points_mean  \\\n",
       "0          0.11840           0.27760          0.3001              0.14710   \n",
       "1          0.08474           0.07864          0.0869              0.07017   \n",
       "2          0.10960           0.15990          0.1974              0.12790   \n",
       "3          0.14250           0.28390          0.2414              0.10520   \n",
       "4          0.10030           0.13280          0.1980              0.10430   \n",
       "\n",
       "   ...  texture_worst  perimeter_worst  area_worst  smoothness_worst  \\\n",
       "0  ...          17.33           184.60      2019.0            0.1622   \n",
       "1  ...          23.41           158.80      1956.0            0.1238   \n",
       "2  ...          25.53           152.50      1709.0            0.1444   \n",
       "3  ...          26.50            98.87       567.7            0.2098   \n",
       "4  ...          16.67           152.20      1575.0            0.1374   \n",
       "\n",
       "   compactness_worst  concavity_worst  concave points_worst  symmetry_worst  \\\n",
       "0             0.6656           0.7119                0.2654          0.4601   \n",
       "1             0.1866           0.2416                0.1860          0.2750   \n",
       "2             0.4245           0.4504                0.2430          0.3613   \n",
       "3             0.8663           0.6869                0.2575          0.6638   \n",
       "4             0.2050           0.4000                0.1625          0.2364   \n",
       "\n",
       "   fractal_dimension_worst  Unnamed: 32  \n",
       "0                  0.11890          NaN  \n",
       "1                  0.08902          NaN  \n",
       "2                  0.08758          NaN  \n",
       "3                  0.17300          NaN  \n",
       "4                  0.07678          NaN  \n",
       "\n",
       "[5 rows x 33 columns]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv(\"Breast Cancer Wisconsin.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "297b1715",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(569, 33)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6457dc02",
   "metadata": {},
   "source": [
    "# Data preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17f4a19e",
   "metadata": {},
   "source": [
    "The diagnosis is our target variable. Let's assign this column to y variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "68ffd289",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df.loc[:,\"diagnosis\"].values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8186b4e",
   "metadata": {},
   "source": [
    "When creating the feature variable. I'm going to remove both target column and unnecessary columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d784fa8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.drop([\"diagnosis\",\"id\",\"Unnamed: 32\"], axis = 1).values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1cfb3be",
   "metadata": {},
   "source": [
    "Pay attention that our target variable has two categories, M and B. Let's encode the target variable with label encoder. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "152577e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "le = LabelEncoder()\n",
    "y = le.fit_transform(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5625dbdb",
   "metadata": {},
   "source": [
    "Let's split the dataset into training and test set. The model is built with training set and is evaluated with test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2db0e483",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test=train_test_split(X,y,stratify=y,random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8d57762",
   "metadata": {},
   "source": [
    "# Building the model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ace1ccb",
   "metadata": {},
   "source": [
    "Let's train our model with training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2e128412",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(random_state=42)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "dt = DecisionTreeClassifier(random_state=42)\n",
    "dt.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3769daf",
   "metadata": {},
   "source": [
    "# Model evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a693910d",
   "metadata": {},
   "source": [
    "Let's evaluate our model with test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "cf6df678",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_pred = dt.predict(X_train)\n",
    "y_test_pred = dt.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4747095",
   "metadata": {},
   "source": [
    "To see the performance of the model, I'm going to use the accuracy score function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "bf20a52f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision tree train/test accuracies:1.000/0.951\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "tree_train = accuracy_score(y_train,y_train_pred)\n",
    "tree_test = accuracy_score(y_test, y_test_pred)\n",
    "print(f\"Decision tree train/test accuracies:{tree_train:.3f}/{tree_test:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd1d61aa",
   "metadata": {},
   "source": [
    "There is overfitting problem in the model since the decision trees model learned the training set so well. To overcome the overfitting problem, we control the complexity of a tree. First, let's specify the max_depth parameter which controls the maximum number of levels."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9eee9e9f",
   "metadata": {},
   "source": [
    "# Building the model with max_depth parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "37888efd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(max_depth=2, random_state=42)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt = DecisionTreeClassifier(max_depth=2, random_state=42)\n",
    "dt.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2df80f43",
   "metadata": {},
   "source": [
    "Let's take a look at the performance of the model on the training and test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2f292d1e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision tree train/test accuracies:0.951/0.923\n"
     ]
    }
   ],
   "source": [
    "y_train_pred = dt.predict(X_train)\n",
    "y_test_pred = dt.predict(X_test)\n",
    "tree_train = accuracy_score(y_train,y_train_pred)\n",
    "tree_test = accuracy_score(y_test, y_test_pred)\n",
    "print(f\"Decision tree train/test accuracies:{tree_train:.3f}/{tree_test:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edde20a9",
   "metadata": {},
   "source": [
    "By making it less complex, we improved the ability of our model to generalize. But, this tree has another problem. To make it better, we need to tune the model using different parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c1d3c60",
   "metadata": {},
   "source": [
    "# Hyperparameter tuning with grid search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22b52106",
   "metadata": {},
   "source": [
    "To find out the best parameter, I'm going to use grid search technique."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3d371178",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'max_depth': 3, 'min_samples_leaf': 1}\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "dt = DecisionTreeClassifier(random_state=42)\n",
    "parameters = {\"max_depth\":[1,2,3,4,5,7,10],\n",
    "              \"min_samples_leaf\":[1,3,6,10,20]}\n",
    "clf = GridSearchCV(dt,parameters, n_jobs=1)\n",
    "clf.fit(X_train,y_train)\n",
    "print(clf.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bf4ee95",
   "metadata": {},
   "source": [
    "When we execute this cell, you can see the best parameters. These are 3 for max_depth and 1 for min_samples_leaf. Now, I'm going to predict this model trained with these parameters. \n",
    "<br/>\n",
    "*Pay attention that* we don't need to train our model again. Because after the best parameters are found, the model is trained again. \n",
    "<br/>\n",
    "Let's predict the values of the training and the test values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "441f5cc2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision tree train/test accuracies:0.974/0.958\n"
     ]
    }
   ],
   "source": [
    "y_train_pred = clf.predict(X_train)\n",
    "y_test_pred = clf.predict(X_test)\n",
    "tree_train = accuracy_score(y_train,y_train_pred)\n",
    "tree_test = accuracy_score(y_test, y_test_pred)\n",
    "print(f\"Decision tree train/test accuracies:{tree_train:.3f}/{tree_test:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "288b777f",
   "metadata": {},
   "source": [
    "Notice that the score of our model on the training set is close to the score on the test set. In addition, both accuracy scores are close to 1. The performance of the model is now better. \n",
    "\n",
    "Thanks for reading.\n",
    "\n",
    "Pleas don't forget to follow us on [YouTube](http://youtube.com/tirendazacademy) | [Medium](http://tirendazacademy.medium.com) | [Twitter](http://twitter.com/tirendazacademy) | [GitHub](http://github.com/tirendazacademy) | [Linkedin](https://www.linkedin.com/in/tirendaz-academy) | [Kaggle](https://www.kaggle.com/tirendazacademy) 😎"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}