data_cleaning_general.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# General data cleaning techniques"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following examples show application of general data cleaning techniques. By general techniques we mean the current techniques applied. Missing data and outliers are excluded from these general techniques, as they are more problem-dependent.\n",
    "\n",
    "General data cleaning techniques:\n",
    "1. [Data types](#Data-types)\n",
    "1. [Data standardization](#Data-standardization)\n",
    "1. [Constant features](#Constant-features)\n",
    "1. [Duplicated rows](#Duplicated-rows)\n",
    "1. [Duplicated features](#Duplicated-features)\n",
    "1. [Values out of range](#Values-out-of-range)\n",
    "1. [Shuffle dataset](#Shuffle-dataset)\n",
    "\n",
    "For further information, see the corresponding blog [post]().\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use the Kaggle Titanic dataset as an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name     Sex Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35      1   \n",
       "4                           Allen, Mr. William Henry    male  35      0   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  \n",
       "0      0         A/5 21171   7.2500   NaN        S  \n",
       "1      0          PC 17599  71.2833   C85        C  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3      0            113803  53.1000  C123        S  \n",
       "4      0            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import data \n",
    "import pandas as pd\n",
    "df = pd.read_csv('./data/titanic.csv')  # Kaggle Titanic dataset\n",
    "\n",
    "# Corrupt dataset to make the example meaningful\n",
    "df['Age'] = df['Age'].astype('object')\n",
    "\n",
    "# Show dataset\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you need to remind the meaning of each feature, check this [link](https://www.kaggle.com/pmarcelino/data-analysis-and-feature-extraction-with-python)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp',\n",
       "       'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get list of features\n",
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we expect to see when checking features' data types:\n",
    "\n",
    "| Feature     | Data type        | Comment                                                    |\n",
    "|-------------|------------------|------------------------------------------------------------|\n",
    "| PassengerId | int64            | It's a categorical feature.                                |\n",
    "| Survived    | int64            | It's a categorical feature.                                |\n",
    "| Pclass      | int64            | It's an ordinal feature.                                   |\n",
    "| Name        | object           | It's character data (string).                              |\n",
    "| Sex         | int64 or object  | Categorical feature. It can be [0,1] or ['male','female']. |\n",
    "| Age         | int64 or float64 | float64 if age = date of birth - current date.             |\n",
    "| SibSp       | int64            | It's continuous.                                           |\n",
    "| Parch       | int64            | It's continuous.                                           |\n",
    "| Ticket      | object or int64  | It's categorical.                                          |\n",
    "| Fare        | float64          | It's continuous.                                           |\n",
    "| Cabin       | int64            | It's categorical.                                          |\n",
    "| Embarked    | int64 or object  | It's categorical.                                          |\n",
    "\n",
    "Looks cool, isn't it? If you also want to do fancy tables, use [this](https://www.tablesgenerator.com/markdown_tables#)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassengerId      int64\n",
       "Survived         int64\n",
       "Pclass           int64\n",
       "Name            object\n",
       "Sex             object\n",
       "Age             object\n",
       "SibSp            int64\n",
       "Parch            int64\n",
       "Ticket          object\n",
       "Fare           float64\n",
       "Cabin           object\n",
       "Embarked        object\n",
       "dtype: object"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get data types\n",
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only feature that it's not according to our expectation is 'Age' (what a surprise). This feature is defined as character data (object), when it should be numeric data (int64 or float64).\n",
    "\n",
    "We can use [pandas.to_numeric](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_numeric.html#pandas.to_numeric) to correct this situation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassengerId      int64\n",
       "Survived         int64\n",
       "Pclass           int64\n",
       "Name            object\n",
       "Sex             object\n",
       "Age            float64\n",
       "SibSp            int64\n",
       "Parch            int64\n",
       "Ticket          object\n",
       "Fare           float64\n",
       "Cabin           object\n",
       "Embarked        object\n",
       "dtype: object"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Correct data type\n",
    "df['Age'] = pd.to_numeric(df['Age'])\n",
    "\n",
    "# Debug\n",
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clean! This is the type of data cleaning operations related to data types that we usually do.\n",
    "\n",
    "Other common data type transformations are:\n",
    "\n",
    "* [pandas.to_datetime](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_datetime.html#pandas.to_datetime) - Converts argument to datetime.\n",
    "* [pandas.to_timedelta](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_timedelta.html#pandas.to_timedelta) - Converts argument to timedelta.\n",
    "* [pandas.DataFrame.astype('object')](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.astype.html#pandas.DataFrame.astype) - Converts argument to object. This is a generic transformation. You can use it to transform data to other data types than object.\n",
    "\n",
    "[Go back to the top](#General-data-cleaning-techniques)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Data standardization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](figures/meme_data_std_under_constr.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Constant features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, we will use data from the [Santander Customer Satisfaction](https://www.kaggle.com/c/santander-customer-satisfaction/data) competition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>var3</th>\n",
       "      <th>var15</th>\n",
       "      <th>imp_ent_var16_ult1</th>\n",
       "      <th>imp_op_var39_comer_ult1</th>\n",
       "      <th>imp_op_var39_comer_ult3</th>\n",
       "      <th>imp_op_var40_comer_ult1</th>\n",
       "      <th>imp_op_var40_comer_ult3</th>\n",
       "      <th>imp_op_var40_efect_ult1</th>\n",
       "      <th>imp_op_var40_efect_ult3</th>\n",
       "      <th>imp_op_var40_ult1</th>\n",
       "      <th>imp_op_var41_comer_ult1</th>\n",
       "      <th>imp_op_var41_comer_ult3</th>\n",
       "      <th>imp_op_var41_efect_ult1</th>\n",
       "      <th>imp_op_var41_efect_ult3</th>\n",
       "      <th>imp_op_var41_ult1</th>\n",
       "      <th>imp_op_var39_efect_ult1</th>\n",
       "      <th>imp_op_var39_efect_ult3</th>\n",
       "      <th>imp_op_var39_ult1</th>\n",
       "      <th>imp_sal_var16_ult1</th>\n",
       "      <th>ind_var1_0</th>\n",
       "      <th>ind_var1</th>\n",
       "      <th>ind_var2_0</th>\n",
       "      <th>ind_var2</th>\n",
       "      <th>ind_var5_0</th>\n",
       "      <th>ind_var5</th>\n",
       "      <th>ind_var6_0</th>\n",
       "      <th>ind_var6</th>\n",
       "      <th>ind_var8_0</th>\n",
       "      <th>ind_var8</th>\n",
       "      <th>ind_var12_0</th>\n",
       "      <th>ind_var12</th>\n",
       "      <th>ind_var13_0</th>\n",
       "      <th>ind_var13_corto_0</th>\n",
       "      <th>ind_var13_corto</th>\n",
       "      <th>ind_var13_largo_0</th>\n",
       "      <th>ind_var13_largo</th>\n",
       "      <th>ind_var13_medio_0</th>\n",
       "      <th>ind_var13_medio</th>\n",
       "      <th>ind_var13</th>\n",
       "      <th>ind_var14_0</th>\n",
       "      <th>ind_var14</th>\n",
       "      <th>ind_var17_0</th>\n",
       "      <th>ind_var17</th>\n",
       "      <th>ind_var18_0</th>\n",
       "      <th>ind_var18</th>\n",
       "      <th>ind_var19</th>\n",
       "      <th>ind_var20_0</th>\n",
       "      <th>ind_var20</th>\n",
       "      <th>ind_var24_0</th>\n",
       "      <th>ind_var24</th>\n",
       "      <th>ind_var25_cte</th>\n",
       "      <th>ind_var26_0</th>\n",
       "      <th>ind_var26_cte</th>\n",
       "      <th>ind_var26</th>\n",
       "      <th>ind_var25_0</th>\n",
       "      <th>ind_var25</th>\n",
       "      <th>ind_var27_0</th>\n",
       "      <th>ind_var28_0</th>\n",
       "      <th>ind_var28</th>\n",
       "      <th>ind_var27</th>\n",
       "      <th>ind_var29_0</th>\n",
       "      <th>ind_var29</th>\n",
       "      <th>ind_var30_0</th>\n",
       "      <th>ind_var30</th>\n",
       "      <th>ind_var31_0</th>\n",
       "      <th>ind_var31</th>\n",
       "      <th>ind_var32_cte</th>\n",
       "      <th>ind_var32_0</th>\n",
       "      <th>ind_var32</th>\n",
       "      <th>ind_var33_0</th>\n",
       "      <th>ind_var33</th>\n",
       "      <th>ind_var34_0</th>\n",
       "      <th>ind_var34</th>\n",
       "      <th>ind_var37_cte</th>\n",
       "      <th>ind_var37_0</th>\n",
       "      <th>ind_var37</th>\n",
       "      <th>ind_var39_0</th>\n",
       "      <th>ind_var40_0</th>\n",
       "      <th>ind_var40</th>\n",
       "      <th>ind_var41_0</th>\n",
       "      <th>ind_var41</th>\n",
       "      <th>ind_var39</th>\n",
       "      <th>ind_var44_0</th>\n",
       "      <th>ind_var44</th>\n",
       "      <th>ind_var46_0</th>\n",
       "      <th>ind_var46</th>\n",
       "      <th>num_var1_0</th>\n",
       "      <th>num_var1</th>\n",
       "      <th>num_var4</th>\n",
       "      <th>num_var5_0</th>\n",
       "      <th>num_var5</th>\n",
       "      <th>num_var6_0</th>\n",
       "      <th>num_var6</th>\n",
       "      <th>num_var8_0</th>\n",
       "      <th>num_var8</th>\n",
       "      <th>num_var12_0</th>\n",
       "      <th>num_var12</th>\n",
       "      <th>num_var13_0</th>\n",
       "      <th>num_var13_corto_0</th>\n",
       "      <th>num_var13_corto</th>\n",
       "      <th>num_var13_largo_0</th>\n",
       "      <th>num_var13_largo</th>\n",
       "      <th>num_var13_medio_0</th>\n",
       "      <th>num_var13_medio</th>\n",
       "      <th>num_var13</th>\n",
       "      <th>num_var14_0</th>\n",
       "      <th>num_var14</th>\n",
       "      <th>num_var17_0</th>\n",
       "      <th>num_var17</th>\n",
       "      <th>num_var18_0</th>\n",
       "      <th>num_var18</th>\n",
       "      <th>num_var20_0</th>\n",
       "      <th>num_var20</th>\n",
       "      <th>num_var24_0</th>\n",
       "      <th>num_var24</th>\n",
       "      <th>num_var26_0</th>\n",
       "      <th>num_var26</th>\n",
       "      <th>num_var25_0</th>\n",
       "      <th>num_var25</th>\n",
       "      <th>num_op_var40_hace2</th>\n",
       "      <th>num_op_var40_hace3</th>\n",
       "      <th>num_op_var40_ult1</th>\n",
       "      <th>num_op_var40_ult3</th>\n",
       "      <th>num_op_var41_hace2</th>\n",
       "      <th>num_op_var41_hace3</th>\n",
       "      <th>num_op_var41_ult1</th>\n",
       "      <th>num_op_var41_ult3</th>\n",
       "      <th>num_op_var39_hace2</th>\n",
       "      <th>num_op_var39_hace3</th>\n",
       "      <th>num_op_var39_ult1</th>\n",
       "      <th>num_op_var39_ult3</th>\n",
       "      <th>num_var27_0</th>\n",
       "      <th>num_var28_0</th>\n",
       "      <th>num_var28</th>\n",
       "      <th>num_var27</th>\n",
       "      <th>num_var29_0</th>\n",
       "      <th>num_var29</th>\n",
       "      <th>num_var30_0</th>\n",
       "      <th>num_var30</th>\n",
       "      <th>num_var31_0</th>\n",
       "      <th>num_var31</th>\n",
       "      <th>num_var32_0</th>\n",
       "      <th>num_var32</th>\n",
       "      <th>num_var33_0</th>\n",
       "      <th>num_var33</th>\n",
       "      <th>num_var34_0</th>\n",
       "      <th>num_var34</th>\n",
       "      <th>num_var35</th>\n",
       "      <th>num_var37_med_ult2</th>\n",
       "      <th>num_var37_0</th>\n",
       "      <th>num_var37</th>\n",
       "      <th>num_var39_0</th>\n",
       "      <th>num_var40_0</th>\n",
       "      <th>num_var40</th>\n",
       "      <th>num_var41_0</th>\n",
       "      <th>num_var41</th>\n",
       "      <th>num_var39</th>\n",
       "      <th>num_var42_0</th>\n",
       "      <th>num_var42</th>\n",
       "      <th>num_var44_0</th>\n",
       "      <th>num_var44</th>\n",
       "      <th>num_var46_0</th>\n",
       "      <th>num_var46</th>\n",
       "      <th>saldo_var1</th>\n",
       "      <th>saldo_var5</th>\n",
       "      <th>saldo_var6</th>\n",
       "      <th>saldo_var8</th>\n",
       "      <th>saldo_var12</th>\n",
       "      <th>saldo_var13_corto</th>\n",
       "      <th>saldo_var13_largo</th>\n",
       "      <th>saldo_var13_medio</th>\n",
       "      <th>saldo_var13</th>\n",
       "      <th>saldo_var14</th>\n",
       "      <th>saldo_var17</th>\n",
       "      <th>saldo_var18</th>\n",
       "      <th>saldo_var20</th>\n",
       "      <th>saldo_var24</th>\n",
       "      <th>saldo_var26</th>\n",
       "      <th>saldo_var25</th>\n",
       "      <th>saldo_var28</th>\n",
       "      <th>saldo_var27</th>\n",
       "      <th>saldo_var29</th>\n",
       "      <th>saldo_var30</th>\n",
       "      <th>saldo_var31</th>\n",
       "      <th>saldo_var32</th>\n",
       "      <th>saldo_var33</th>\n",
       "      <th>saldo_var34</th>\n",
       "      <th>saldo_var37</th>\n",
       "      <th>saldo_var40</th>\n",
       "      <th>saldo_var41</th>\n",
       "      <th>saldo_var42</th>\n",
       "      <th>saldo_var44</th>\n",
       "      <th>saldo_var46</th>\n",
       "      <th>var36</th>\n",
       "      <th>delta_imp_amort_var18_1y3</th>\n",
       "      <th>delta_imp_amort_var34_1y3</th>\n",
       "      <th>delta_imp_aport_var13_1y3</th>\n",
       "      <th>delta_imp_aport_var17_1y3</th>\n",
       "      <th>delta_imp_aport_var33_1y3</th>\n",
       "      <th>delta_imp_compra_var44_1y3</th>\n",
       "      <th>delta_imp_reemb_var13_1y3</th>\n",
       "      <th>delta_imp_reemb_var17_1y3</th>\n",
       "      <th>delta_imp_reemb_var33_1y3</th>\n",
       "      <th>delta_imp_trasp_var17_in_1y3</th>\n",
       "      <th>delta_imp_trasp_var17_out_1y3</th>\n",
       "      <th>delta_imp_trasp_var33_in_1y3</th>\n",
       "      <th>delta_imp_trasp_var33_out_1y3</th>\n",
       "      <th>delta_imp_venta_var44_1y3</th>\n",
       "      <th>delta_num_aport_var13_1y3</th>\n",
       "      <th>delta_num_aport_var17_1y3</th>\n",
       "      <th>delta_num_aport_var33_1y3</th>\n",
       "      <th>delta_num_compra_var44_1y3</th>\n",
       "      <th>delta_num_reemb_var13_1y3</th>\n",
       "      <th>delta_num_reemb_var17_1y3</th>\n",
       "      <th>delta_num_reemb_var33_1y3</th>\n",
       "      <th>delta_num_trasp_var17_in_1y3</th>\n",
       "      <th>delta_num_trasp_var17_out_1y3</th>\n",
       "      <th>delta_num_trasp_var33_in_1y3</th>\n",
       "      <th>delta_num_trasp_var33_out_1y3</th>\n",
       "      <th>delta_num_venta_var44_1y3</th>\n",
       "      <th>imp_amort_var18_hace3</th>\n",
       "      <th>imp_amort_var18_ult1</th>\n",
       "      <th>imp_amort_var34_hace3</th>\n",
       "      <th>imp_amort_var34_ult1</th>\n",
       "      <th>imp_aport_var13_hace3</th>\n",
       "      <th>imp_aport_var13_ult1</th>\n",
       "      <th>imp_aport_var17_hace3</th>\n",
       "      <th>imp_aport_var17_ult1</th>\n",
       "      <th>imp_aport_var33_hace3</th>\n",
       "      <th>imp_aport_var33_ult1</th>\n",
       "      <th>imp_var7_emit_ult1</th>\n",
       "      <th>imp_var7_recib_ult1</th>\n",
       "      <th>imp_compra_var44_hace3</th>\n",
       "      <th>imp_compra_var44_ult1</th>\n",
       "      <th>imp_reemb_var13_hace3</th>\n",
       "      <th>imp_reemb_var13_ult1</th>\n",
       "      <th>imp_reemb_var17_hace3</th>\n",
       "      <th>imp_reemb_var17_ult1</th>\n",
       "      <th>imp_reemb_var33_hace3</th>\n",
       "      <th>imp_reemb_var33_ult1</th>\n",
       "      <th>imp_var43_emit_ult1</th>\n",
       "      <th>imp_trans_var37_ult1</th>\n",
       "      <th>imp_trasp_var17_in_hace3</th>\n",
       "      <th>imp_trasp_var17_in_ult1</th>\n",
       "      <th>imp_trasp_var17_out_hace3</th>\n",
       "      <th>imp_trasp_var17_out_ult1</th>\n",
       "      <th>imp_trasp_var33_in_hace3</th>\n",
       "      <th>imp_trasp_var33_in_ult1</th>\n",
       "      <th>imp_trasp_var33_out_hace3</th>\n",
       "      <th>imp_trasp_var33_out_ult1</th>\n",
       "      <th>imp_venta_var44_hace3</th>\n",
       "      <th>imp_venta_var44_ult1</th>\n",
       "      <th>ind_var7_emit_ult1</th>\n",
       "      <th>ind_var7_recib_ult1</th>\n",
       "      <th>ind_var10_ult1</th>\n",
       "      <th>ind_var10cte_ult1</th>\n",
       "      <th>ind_var9_cte_ult1</th>\n",
       "      <th>ind_var9_ult1</th>\n",
       "      <th>ind_var43_emit_ult1</th>\n",
       "      <th>ind_var43_recib_ult1</th>\n",
       "      <th>var21</th>\n",
       "      <th>num_var2_0_ult1</th>\n",
       "      <th>num_var2_ult1</th>\n",
       "      <th>num_aport_var13_hace3</th>\n",
       "      <th>num_aport_var13_ult1</th>\n",
       "      <th>num_aport_var17_hace3</th>\n",
       "      <th>num_aport_var17_ult1</th>\n",
       "      <th>num_aport_var33_hace3</th>\n",
       "      <th>num_aport_var33_ult1</th>\n",
       "      <th>num_var7_emit_ult1</th>\n",
       "      <th>num_var7_recib_ult1</th>\n",
       "      <th>num_compra_var44_hace3</th>\n",
       "      <th>num_compra_var44_ult1</th>\n",
       "      <th>num_ent_var16_ult1</th>\n",
       "      <th>num_var22_hace2</th>\n",
       "      <th>num_var22_hace3</th>\n",
       "      <th>num_var22_ult1</th>\n",
       "      <th>num_var22_ult3</th>\n",
       "      <th>num_med_var22_ult3</th>\n",
       "      <th>num_med_var45_ult3</th>\n",
       "      <th>num_meses_var5_ult3</th>\n",
       "      <th>num_meses_var8_ult3</th>\n",
       "      <th>num_meses_var12_ult3</th>\n",
       "      <th>num_meses_var13_corto_ult3</th>\n",
       "      <th>num_meses_var13_largo_ult3</th>\n",
       "      <th>num_meses_var13_medio_ult3</th>\n",
       "      <th>num_meses_var17_ult3</th>\n",
       "      <th>num_meses_var29_ult3</th>\n",
       "      <th>num_meses_var33_ult3</th>\n",
       "      <th>num_meses_var39_vig_ult3</th>\n",
       "      <th>num_meses_var44_ult3</th>\n",
       "      <th>num_op_var39_comer_ult1</th>\n",
       "      <th>num_op_var39_comer_ult3</th>\n",
       "      <th>num_op_var40_comer_ult1</th>\n",
       "      <th>num_op_var40_comer_ult3</th>\n",
       "      <th>num_op_var40_efect_ult1</th>\n",
       "      <th>num_op_var40_efect_ult3</th>\n",
       "      <th>num_op_var41_comer_ult1</th>\n",
       "      <th>num_op_var41_comer_ult3</th>\n",
       "      <th>num_op_var41_efect_ult1</th>\n",
       "      <th>num_op_var41_efect_ult3</th>\n",
       "      <th>num_op_var39_efect_ult1</th>\n",
       "      <th>num_op_var39_efect_ult3</th>\n",
       "      <th>num_reemb_var13_hace3</th>\n",
       "      <th>num_reemb_var13_ult1</th>\n",
       "      <th>num_reemb_var17_hace3</th>\n",
       "      <th>num_reemb_var17_ult1</th>\n",
       "      <th>num_reemb_var33_hace3</th>\n",
       "      <th>num_reemb_var33_ult1</th>\n",
       "      <th>num_sal_var16_ult1</th>\n",
       "      <th>num_var43_emit_ult1</th>\n",
       "      <th>num_var43_recib_ult1</th>\n",
       "      <th>num_trasp_var11_ult1</th>\n",
       "      <th>num_trasp_var17_in_hace3</th>\n",
       "      <th>num_trasp_var17_in_ult1</th>\n",
       "      <th>num_trasp_var17_out_hace3</th>\n",
       "      <th>num_trasp_var17_out_ult1</th>\n",
       "      <th>num_trasp_var33_in_hace3</th>\n",
       "      <th>num_trasp_var33_in_ult1</th>\n",
       "      <th>num_trasp_var33_out_hace3</th>\n",
       "      <th>num_trasp_var33_out_ult1</th>\n",
       "      <th>num_venta_var44_hace3</th>\n",
       "      <th>num_venta_var44_ult1</th>\n",
       "      <th>num_var45_hace2</th>\n",
       "      <th>num_var45_hace3</th>\n",
       "      <th>num_var45_ult1</th>\n",
       "      <th>num_var45_ult3</th>\n",
       "      <th>saldo_var2_ult1</th>\n",
       "      <th>saldo_medio_var5_hace2</th>\n",
       "      <th>saldo_medio_var5_hace3</th>\n",
       "      <th>saldo_medio_var5_ult1</th>\n",
       "      <th>saldo_medio_var5_ult3</th>\n",
       "      <th>saldo_medio_var8_hace2</th>\n",
       "      <th>saldo_medio_var8_hace3</th>\n",
       "      <th>saldo_medio_var8_ult1</th>\n",
       "      <th>saldo_medio_var8_ult3</th>\n",
       "      <th>saldo_medio_var12_hace2</th>\n",
       "      <th>saldo_medio_var12_hace3</th>\n",
       "      <th>saldo_medio_var12_ult1</th>\n",
       "      <th>saldo_medio_var12_ult3</th>\n",
       "      <th>saldo_medio_var13_corto_hace2</th>\n",
       "      <th>saldo_medio_var13_corto_hace3</th>\n",
       "      <th>saldo_medio_var13_corto_ult1</th>\n",
       "      <th>saldo_medio_var13_corto_ult3</th>\n",
       "      <th>saldo_medio_var13_largo_hace2</th>\n",
       "      <th>saldo_medio_var13_largo_hace3</th>\n",
       "      <th>saldo_medio_var13_largo_ult1</th>\n",
       "      <th>saldo_medio_var13_largo_ult3</th>\n",
       "      <th>saldo_medio_var13_medio_hace2</th>\n",
       "      <th>saldo_medio_var13_medio_hace3</th>\n",
       "      <th>saldo_medio_var13_medio_ult1</th>\n",
       "      <th>saldo_medio_var13_medio_ult3</th>\n",
       "      <th>saldo_medio_var17_hace2</th>\n",
       "      <th>saldo_medio_var17_hace3</th>\n",
       "      <th>saldo_medio_var17_ult1</th>\n",
       "      <th>saldo_medio_var17_ult3</th>\n",
       "      <th>saldo_medio_var29_hace2</th>\n",
       "      <th>saldo_medio_var29_hace3</th>\n",
       "      <th>saldo_medio_var29_ult1</th>\n",
       "      <th>saldo_medio_var29_ult3</th>\n",
       "      <th>saldo_medio_var33_hace2</th>\n",
       "      <th>saldo_medio_var33_hace3</th>\n",
       "      <th>saldo_medio_var33_ult1</th>\n",
       "      <th>saldo_medio_var33_ult3</th>\n",
       "      <th>saldo_medio_var44_hace2</th>\n",
       "      <th>saldo_medio_var44_hace3</th>\n",
       "      <th>saldo_medio_var44_ult1</th>\n",
       "      <th>saldo_medio_var44_ult3</th>\n",
       "      <th>var38</th>\n",
       "      <th>TARGET</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>23</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>99</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>39205.170000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>34</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>300.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>300.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>300.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>300.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>88.89</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>300.0</td>\n",
       "      <td>122.22</td>\n",
       "      <td>300.0</td>\n",
       "      <td>240.75</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>49278.030000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>23</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>99</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3.00</td>\n",
       "      <td>0.18</td>\n",
       "      <td>3.00</td>\n",
       "      <td>2.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>67333.770000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>37</td>\n",
       "      <td>0.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>70.62</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>70.62</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>34.95</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>70.62</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>27</td>\n",
       "      <td>3</td>\n",
       "      <td>18</td>\n",
       "      <td>48</td>\n",
       "      <td>0</td>\n",
       "      <td>186.09</td>\n",
       "      <td>0.00</td>\n",
       "      <td>91.56</td>\n",
       "      <td>138.84</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>64007.970000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>39</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>135003.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>135003.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>135003.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>135003.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>135003.0</td>\n",
       "      <td>270003.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3.00</td>\n",
       "      <td>0.30</td>\n",
       "      <td>40501.08</td>\n",
       "      <td>13501.47</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>85501.89</td>\n",
       "      <td>85501.89</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>117310.979016</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   ID  var3  var15  imp_ent_var16_ult1  imp_op_var39_comer_ult1  \\\n",
       "0   1     2     23                 0.0                      0.0   \n",
       "1   3     2     34                 0.0                      0.0   \n",
       "2   4     2     23                 0.0                      0.0   \n",
       "3   8     2     37                 0.0                    195.0   \n",
       "4  10     2     39                 0.0                      0.0   \n",
       "\n",
       "   imp_op_var39_comer_ult3  imp_op_var40_comer_ult1  imp_op_var40_comer_ult3  \\\n",
       "0                      0.0                      0.0                      0.0   \n",
       "1                      0.0                      0.0                      0.0   \n",
       "2                      0.0                      0.0                      0.0   \n",
       "3                    195.0                      0.0                      0.0   \n",
       "4                      0.0                      0.0                      0.0   \n",
       "\n",
       "   imp_op_var40_efect_ult1  imp_op_var40_efect_ult3  imp_op_var40_ult1  \\\n",
       "0                      0.0                      0.0                0.0   \n",
       "1                      0.0                      0.0                0.0   \n",
       "2                      0.0                      0.0                0.0   \n",
       "3                      0.0                      0.0                0.0   \n",
       "4                      0.0                      0.0                0.0   \n",
       "\n",
       "   imp_op_var41_comer_ult1  imp_op_var41_comer_ult3  imp_op_var41_efect_ult1  \\\n",
       "0                      0.0                      0.0                      0.0   \n",
       "1                      0.0                      0.0                      0.0   \n",
       "2                      0.0                      0.0                      0.0   \n",
       "3                    195.0                    195.0                      0.0   \n",
       "4                      0.0                      0.0                      0.0   \n",
       "\n",
       "   imp_op_var41_efect_ult3  imp_op_var41_ult1  imp_op_var39_efect_ult1  \\\n",
       "0                      0.0                0.0                      0.0   \n",
       "1                      0.0                0.0                      0.0   \n",
       "2                      0.0                0.0                      0.0   \n",
       "3                      0.0              195.0                      0.0   \n",
       "4                      0.0                0.0                      0.0   \n",
       "\n",
       "   imp_op_var39_efect_ult3  imp_op_var39_ult1  imp_sal_var16_ult1  ind_var1_0  \\\n",
       "0                      0.0                0.0                 0.0           0   \n",
       "1                      0.0                0.0                 0.0           0   \n",
       "2                      0.0                0.0                 0.0           0   \n",
       "3                      0.0              195.0                 0.0           0   \n",
       "4                      0.0                0.0                 0.0           0   \n",
       "\n",
       "   ind_var1  ind_var2_0  ind_var2  ind_var5_0  ind_var5  ind_var6_0  ind_var6  \\\n",
       "0         0           0         0           1         0           0         0   \n",
       "1         0           0         0           1         0           0         0   \n",
       "2         0           0         0           1         1           0         0   \n",
       "3         0           0         0           1         1           0         0   \n",
       "4         0           0         0           1         0           0         0   \n",
       "\n",
       "   ind_var8_0  ind_var8  ind_var12_0  ind_var12  ind_var13_0  \\\n",
       "0           0         0            0          0            0   \n",
       "1           0         0            0          0            1   \n",
       "2           0         0            0          0            0   \n",
       "3           0         0            0          0            0   \n",
       "4           0         0            1          1            0   \n",
       "\n",
       "   ind_var13_corto_0  ind_var13_corto  ind_var13_largo_0  ind_var13_largo  \\\n",
       "0                  0                0                  0                0   \n",
       "1                  1                1                  0                0   \n",
       "2                  0                0                  0                0   \n",
       "3                  0                0                  0                0   \n",
       "4                  0                0                  0                0   \n",
       "\n",
       "   ind_var13_medio_0  ind_var13_medio  ind_var13  ind_var14_0  ind_var14  \\\n",
       "0                  0                0          0            0          0   \n",
       "1                  0                0          1            0          0   \n",
       "2                  0                0          0            0          0   \n",
       "3                  0                0          0            0          0   \n",
       "4                  0                0          0            0          0   \n",
       "\n",
       "   ind_var17_0  ind_var17  ind_var18_0  ind_var18  ind_var19  ind_var20_0  \\\n",
       "0            0          0            0          0          0            0   \n",
       "1            0          0            0          0          0            0   \n",
       "2            0          0            0          0          0            0   \n",
       "3            0          0            0          0          0            0   \n",
       "4            0          0            0          0          0            0   \n",
       "\n",
       "   ind_var20  ind_var24_0  ind_var24  ind_var25_cte  ind_var26_0  \\\n",
       "0          0            0          0              0            0   \n",
       "1          0            0          0              0            0   \n",
       "2          0            0          0              0            0   \n",
       "3          0            0          0              0            0   \n",
       "4          0            1          1              0            0   \n",
       "\n",
       "   ind_var26_cte  ind_var26  ind_var25_0  ind_var25  ind_var27_0  ind_var28_0  \\\n",
       "0              0          0            0          0            0            0   \n",
       "1              0          0            0          0            0            0   \n",
       "2              0          0            0          0            0            0   \n",
       "3              0          0            0          0            0            0   \n",
       "4              0          0            0          0            0            0   \n",
       "\n",
       "   ind_var28  ind_var27  ind_var29_0  ind_var29  ind_var30_0  ind_var30  \\\n",
       "0          0          0            0          0            1          0   \n",
       "1          0          0            0          0            1          1   \n",
       "2          0          0            0          0            1          1   \n",
       "3          0          0            0          0            1          1   \n",
       "4          0          0            0          0            1          1   \n",
       "\n",
       "   ind_var31_0  ind_var31  ind_var32_cte  ind_var32_0  ind_var32  ind_var33_0  \\\n",
       "0            0          0              0            0          0            0   \n",
       "1            0          0              0            0          0            0   \n",
       "2            0          0              0            0          0            0   \n",
       "3            0          0              0            0          0            0   \n",
       "4            0          0              0            0          0            0   \n",
       "\n",
       "   ind_var33  ind_var34_0  ind_var34  ind_var37_cte  ind_var37_0  ind_var37  \\\n",
       "0          0            0          0              0            0          0   \n",
       "1          0            0          0              0            0          0   \n",
       "2          0            0          0              0            0          0   \n",
       "3          0            0          0              1            1          1   \n",
       "4          0            0          0              0            0          0   \n",
       "\n",
       "   ind_var39_0  ind_var40_0  ind_var40  ind_var41_0  ind_var41  ind_var39  \\\n",
       "0            1            0          0            1          0          0   \n",
       "1            1            0          0            1          0          0   \n",
       "2            1            0          0            1          0          0   \n",
       "3            1            0          0            1          0          0   \n",
       "4            1            0          0            1          0          0   \n",
       "\n",
       "   ind_var44_0  ind_var44  ind_var46_0  ind_var46  num_var1_0  num_var1  \\\n",
       "0            0          0            0          0           0         0   \n",
       "1            0          0            0          0           0         0   \n",
       "2            0          0            0          0           0         0   \n",
       "3            0          0            0          0           0         0   \n",
       "4            0          0            0          0           0         0   \n",
       "\n",
       "   num_var4  num_var5_0  num_var5  num_var6_0  num_var6  num_var8_0  num_var8  \\\n",
       "0         0           3         0           0         0           0         0   \n",
       "1         1           3         0           0         0           0         0   \n",
       "2         1           3         3           0         0           0         0   \n",
       "3         3           3         3           0         0           0         0   \n",
       "4         1           3         0           0         0           0         0   \n",
       "\n",
       "   num_var12_0  num_var12  num_var13_0  num_var13_corto_0  num_var13_corto  \\\n",
       "0            0          0            0                  0                0   \n",
       "1            0          0            3                  3                3   \n",
       "2            0          0            0                  0                0   \n",
       "3            0          0            0                  0                0   \n",
       "4            3          3            0                  0                0   \n",
       "\n",
       "   num_var13_largo_0  num_var13_largo  num_var13_medio_0  num_var13_medio  \\\n",
       "0                  0                0                  0                0   \n",
       "1                  0                0                  0                0   \n",
       "2                  0                0                  0                0   \n",
       "3                  0                0                  0                0   \n",
       "4                  0                0                  0                0   \n",
       "\n",
       "   num_var13  num_var14_0  num_var14  num_var17_0  num_var17  num_var18_0  \\\n",
       "0          0            0          0            0          0            0   \n",
       "1          3            0          0            0          0            0   \n",
       "2          0            0          0            0          0            0   \n",
       "3          0            0          0            0          0            0   \n",
       "4          0            0          0            0          0            0   \n",
       "\n",
       "   num_var18  num_var20_0  num_var20  num_var24_0  num_var24  num_var26_0  \\\n",
       "0          0            0          0            0          0            0   \n",
       "1          0            0          0            0          0            0   \n",
       "2          0            0          0            0          0            0   \n",
       "3          0            0          0            0          0            0   \n",
       "4          0            0          0            3          3            0   \n",
       "\n",
       "   num_var26  num_var25_0  num_var25  num_op_var40_hace2  num_op_var40_hace3  \\\n",
       "0          0            0          0                   0                   0   \n",
       "1          0            0          0                   0                   0   \n",
       "2          0            0          0                   0                   0   \n",
       "3          0            0          0                   0                   0   \n",
       "4          0            0          0                   0                   0   \n",
       "\n",
       "   num_op_var40_ult1  num_op_var40_ult3  num_op_var41_hace2  \\\n",
       "0                  0                  0                   0   \n",
       "1                  0                  0                   0   \n",
       "2                  0                  0                   0   \n",
       "3                  0                  0                   0   \n",
       "4                  0                  0                   0   \n",
       "\n",
       "   num_op_var41_hace3  num_op_var41_ult1  num_op_var41_ult3  \\\n",
       "0                   0                  0                  0   \n",
       "1                   0                  0                  0   \n",
       "2                   0                  0                  0   \n",
       "3                   0                  9                  9   \n",
       "4                   0                  0                  0   \n",
       "\n",
       "   num_op_var39_hace2  num_op_var39_hace3  num_op_var39_ult1  \\\n",
       "0                   0                   0                  0   \n",
       "1                   0                   0                  0   \n",
       "2                   0                   0                  0   \n",
       "3                   0                   0                  9   \n",
       "4                   0                   0                  0   \n",
       "\n",
       "   num_op_var39_ult3  num_var27_0  num_var28_0  num_var28  num_var27  \\\n",
       "0                  0            0            0          0          0   \n",
       "1                  0            0            0          0          0   \n",
       "2                  0            0            0          0          0   \n",
       "3                  9            0            0          0          0   \n",
       "4                  0            0            0          0          0   \n",
       "\n",
       "   num_var29_0  num_var29  num_var30_0  num_var30  num_var31_0  num_var31  \\\n",
       "0            0          0            3          0            0          0   \n",
       "1            0          0            6          3            0          0   \n",
       "2            0          0            3          3            0          0   \n",
       "3            0          0            3          3            0          0   \n",
       "4            0          0            6          3            0          0   \n",
       "\n",
       "   num_var32_0  num_var32  num_var33_0  num_var33  num_var34_0  num_var34  \\\n",
       "0            0          0            0          0            0          0   \n",
       "1            0          0            0          0            0          0   \n",
       "2            0          0            0          0            0          0   \n",
       "3            0          0            0          0            0          0   \n",
       "4            0          0            0          0            0          0   \n",
       "\n",
       "   num_var35  num_var37_med_ult2  num_var37_0  num_var37  num_var39_0  \\\n",
       "0          0                   0            0          0            3   \n",
       "1          3                   0            0          0            3   \n",
       "2          3                   0            0          0            3   \n",
       "3          9                   6            6          6            3   \n",
       "4          3                   0            0          0            3   \n",
       "\n",
       "   num_var40_0  num_var40  num_var41_0  num_var41  num_var39  num_var42_0  \\\n",
       "0            0          0            3          0          0            3   \n",
       "1            0          0            3          0          0            3   \n",
       "2            0          0            3          0          0            3   \n",
       "3            0          0            3          0          0            3   \n",
       "4            0          0            3          0          0            6   \n",
       "\n",
       "   num_var42  num_var44_0  num_var44  num_var46_0  num_var46  saldo_var1  \\\n",
       "0          0            0          0            0          0         0.0   \n",
       "1          0            0          0            0          0         0.0   \n",
       "2          3            0          0            0          0         0.0   \n",
       "3          3            0          0            0          0         0.0   \n",
       "4          3            0          0            0          0         0.0   \n",
       "\n",
       "   saldo_var5  saldo_var6  saldo_var8  saldo_var12  saldo_var13_corto  \\\n",
       "0        0.00         0.0         0.0          0.0                0.0   \n",
       "1        0.00         0.0         0.0          0.0              300.0   \n",
       "2        3.00         0.0         0.0          0.0                0.0   \n",
       "3       70.62         0.0         0.0          0.0                0.0   \n",
       "4        0.00         0.0         0.0     135003.0                0.0   \n",
       "\n",
       "   saldo_var13_largo  saldo_var13_medio  saldo_var13  saldo_var14  \\\n",
       "0                0.0                  0          0.0          0.0   \n",
       "1                0.0                  0        300.0          0.0   \n",
       "2                0.0                  0          0.0          0.0   \n",
       "3                0.0                  0          0.0          0.0   \n",
       "4                0.0                  0          0.0          0.0   \n",
       "\n",
       "   saldo_var17  saldo_var18  saldo_var20  saldo_var24  saldo_var26  \\\n",
       "0          0.0            0          0.0          0.0          0.0   \n",
       "1          0.0            0          0.0          0.0          0.0   \n",
       "2          0.0            0          0.0          0.0          0.0   \n",
       "3          0.0            0          0.0          0.0          0.0   \n",
       "4          0.0            0          0.0     135003.0          0.0   \n",
       "\n",
       "   saldo_var25  saldo_var28  saldo_var27  saldo_var29  saldo_var30  \\\n",
       "0          0.0            0            0          0.0         0.00   \n",
       "1          0.0            0            0          0.0       300.00   \n",
       "2          0.0            0            0          0.0         3.00   \n",
       "3          0.0            0            0          0.0        70.62   \n",
       "4          0.0            0            0          0.0    135003.00   \n",
       "\n",
       "   saldo_var31  saldo_var32  saldo_var33  saldo_var34  saldo_var37  \\\n",
       "0          0.0          0.0          0.0            0         0.00   \n",
       "1          0.0          0.0          0.0            0         0.00   \n",
       "2          0.0          0.0          0.0            0         0.00   \n",
       "3          0.0          0.0          0.0            0        34.95   \n",
       "4          0.0          0.0          0.0            0         0.00   \n",
       "\n",
       "   saldo_var40  saldo_var41  saldo_var42  saldo_var44  saldo_var46  var36  \\\n",
       "0          0.0            0         0.00          0.0            0     99   \n",
       "1          0.0            0         0.00          0.0            0      3   \n",
       "2          0.0            0         3.00          0.0            0     99   \n",
       "3          0.0            0        70.62          0.0            0      2   \n",
       "4          0.0            0    135003.00          0.0            0      1   \n",
       "\n",
       "   delta_imp_amort_var18_1y3  delta_imp_amort_var34_1y3  \\\n",
       "0                          0                          0   \n",
       "1                          0                          0   \n",
       "2                          0                          0   \n",
       "3                          0                          0   \n",
       "4                          0                          0   \n",
       "\n",
       "   delta_imp_aport_var13_1y3  delta_imp_aport_var17_1y3  \\\n",
       "0                        0.0                        0.0   \n",
       "1                       -1.0                        0.0   \n",
       "2                        0.0                        0.0   \n",
       "3                        0.0                        0.0   \n",
       "4                        0.0                        0.0   \n",
       "\n",
       "   delta_imp_aport_var33_1y3  delta_imp_compra_var44_1y3  \\\n",
       "0                        0.0                         0.0   \n",
       "1                        0.0                         0.0   \n",
       "2                        0.0                         0.0   \n",
       "3                        0.0                         0.0   \n",
       "4                        0.0                         0.0   \n",
       "\n",
       "   delta_imp_reemb_var13_1y3  delta_imp_reemb_var17_1y3  \\\n",
       "0                          0                          0   \n",
       "1                          0                          0   \n",
       "2                          0                          0   \n",
       "3                          0                          0   \n",
       "4                          0                          0   \n",
       "\n",
       "   delta_imp_reemb_var33_1y3  delta_imp_trasp_var17_in_1y3  \\\n",
       "0                          0                             0   \n",
       "1                          0                             0   \n",
       "2                          0                             0   \n",
       "3                          0                             0   \n",
       "4                          0                             0   \n",
       "\n",
       "   delta_imp_trasp_var17_out_1y3  delta_imp_trasp_var33_in_1y3  \\\n",
       "0                              0                             0   \n",
       "1                              0                             0   \n",
       "2                              0                             0   \n",
       "3                              0                             0   \n",
       "4                              0                             0   \n",
       "\n",
       "   delta_imp_trasp_var33_out_1y3  delta_imp_venta_var44_1y3  \\\n",
       "0                              0                        0.0   \n",
       "1                              0                        0.0   \n",
       "2                              0                        0.0   \n",
       "3                              0                        0.0   \n",
       "4                              0                        0.0   \n",
       "\n",
       "   delta_num_aport_var13_1y3  delta_num_aport_var17_1y3  \\\n",
       "0                        0.0                        0.0   \n",
       "1                       -1.0                        0.0   \n",
       "2                        0.0                        0.0   \n",
       "3                        0.0                        0.0   \n",
       "4                        0.0                        0.0   \n",
       "\n",
       "   delta_num_aport_var33_1y3  delta_num_compra_var44_1y3  \\\n",
       "0                        0.0                         0.0   \n",
       "1                        0.0                         0.0   \n",
       "2                        0.0                         0.0   \n",
       "3                        0.0                         0.0   \n",
       "4                        0.0                         0.0   \n",
       "\n",
       "   delta_num_reemb_var13_1y3  delta_num_reemb_var17_1y3  \\\n",
       "0                          0                          0   \n",
       "1                          0                          0   \n",
       "2                          0                          0   \n",
       "3                          0                          0   \n",
       "4                          0                          0   \n",
       "\n",
       "   delta_num_reemb_var33_1y3  delta_num_trasp_var17_in_1y3  \\\n",
       "0                          0                             0   \n",
       "1                          0                             0   \n",
       "2                          0                             0   \n",
       "3                          0                             0   \n",
       "4                          0                             0   \n",
       "\n",
       "   delta_num_trasp_var17_out_1y3  delta_num_trasp_var33_in_1y3  \\\n",
       "0                              0                             0   \n",
       "1                              0                             0   \n",
       "2                              0                             0   \n",
       "3                              0                             0   \n",
       "4                              0                             0   \n",
       "\n",
       "   delta_num_trasp_var33_out_1y3  delta_num_venta_var44_1y3  \\\n",
       "0                              0                        0.0   \n",
       "1                              0                        0.0   \n",
       "2                              0                        0.0   \n",
       "3                              0                        0.0   \n",
       "4                              0                        0.0   \n",
       "\n",
       "   imp_amort_var18_hace3  imp_amort_var18_ult1  imp_amort_var34_hace3  \\\n",
       "0                      0                   0.0                      0   \n",
       "1                      0                   0.0                      0   \n",
       "2                      0                   0.0                      0   \n",
       "3                      0                   0.0                      0   \n",
       "4                      0                   0.0                      0   \n",
       "\n",
       "   imp_amort_var34_ult1  imp_aport_var13_hace3  imp_aport_var13_ult1  \\\n",
       "0                   0.0                    0.0                   0.0   \n",
       "1                   0.0                  300.0                   0.0   \n",
       "2                   0.0                    0.0                   0.0   \n",
       "3                   0.0                    0.0                   0.0   \n",
       "4                   0.0                    0.0                   0.0   \n",
       "\n",
       "   imp_aport_var17_hace3  imp_aport_var17_ult1  imp_aport_var33_hace3  \\\n",
       "0                    0.0                   0.0                      0   \n",
       "1                    0.0                   0.0                      0   \n",
       "2                    0.0                   0.0                      0   \n",
       "3                    0.0                   0.0                      0   \n",
       "4                    0.0                   0.0                      0   \n",
       "\n",
       "   imp_aport_var33_ult1  imp_var7_emit_ult1  imp_var7_recib_ult1  \\\n",
       "0                     0                 0.0                  0.0   \n",
       "1                     0                 0.0                  0.0   \n",
       "2                     0                 0.0                  0.0   \n",
       "3                     0                 0.0                  0.0   \n",
       "4                     0                 0.0                  0.0   \n",
       "\n",
       "   imp_compra_var44_hace3  imp_compra_var44_ult1  imp_reemb_var13_hace3  \\\n",
       "0                     0.0                    0.0                      0   \n",
       "1                     0.0                    0.0                      0   \n",
       "2                     0.0                    0.0                      0   \n",
       "3                     0.0                    0.0                      0   \n",
       "4                     0.0                    0.0                      0   \n",
       "\n",
       "   imp_reemb_var13_ult1  imp_reemb_var17_hace3  imp_reemb_var17_ult1  \\\n",
       "0                   0.0                    0.0                   0.0   \n",
       "1                   0.0                    0.0                   0.0   \n",
       "2                   0.0                    0.0                   0.0   \n",
       "3                   0.0                    0.0                   0.0   \n",
       "4                   0.0                    0.0                   0.0   \n",
       "\n",
       "   imp_reemb_var33_hace3  imp_reemb_var33_ult1  imp_var43_emit_ult1  \\\n",
       "0                      0                     0                  0.0   \n",
       "1                      0                     0                  0.0   \n",
       "2                      0                     0                  0.0   \n",
       "3                      0                     0                  0.0   \n",
       "4                      0                     0             135003.0   \n",
       "\n",
       "   imp_trans_var37_ult1  imp_trasp_var17_in_hace3  imp_trasp_var17_in_ult1  \\\n",
       "0                   0.0                       0.0                      0.0   \n",
       "1                   0.0                       0.0                      0.0   \n",
       "2                   0.0                       0.0                      0.0   \n",
       "3                   0.0                       0.0                      0.0   \n",
       "4              270003.0                       0.0                      0.0   \n",
       "\n",
       "   imp_trasp_var17_out_hace3  imp_trasp_var17_out_ult1  \\\n",
       "0                          0                       0.0   \n",
       "1                          0                       0.0   \n",
       "2                          0                       0.0   \n",
       "3                          0                       0.0   \n",
       "4                          0                       0.0   \n",
       "\n",
       "   imp_trasp_var33_in_hace3  imp_trasp_var33_in_ult1  \\\n",
       "0                       0.0                      0.0   \n",
       "1                       0.0                      0.0   \n",
       "2                       0.0                      0.0   \n",
       "3                       0.0                      0.0   \n",
       "4                       0.0                      0.0   \n",
       "\n",
       "   imp_trasp_var33_out_hace3  imp_trasp_var33_out_ult1  imp_venta_var44_hace3  \\\n",
       "0                          0                         0                    0.0   \n",
       "1                          0                         0                    0.0   \n",
       "2                          0                         0                    0.0   \n",
       "3                          0                         0                    0.0   \n",
       "4                          0                         0                    0.0   \n",
       "\n",
       "   imp_venta_var44_ult1  ind_var7_emit_ult1  ind_var7_recib_ult1  \\\n",
       "0                   0.0                   0                    0   \n",
       "1                   0.0                   0                    0   \n",
       "2                   0.0                   0                    0   \n",
       "3                   0.0                   0                    0   \n",
       "4                   0.0                   0                    0   \n",
       "\n",
       "   ind_var10_ult1  ind_var10cte_ult1  ind_var9_cte_ult1  ind_var9_ult1  \\\n",
       "0               0                  0                  0              0   \n",
       "1               0                  0                  0              0   \n",
       "2               0                  0                  0              0   \n",
       "3               1                  1                  1              1   \n",
       "4               0                  0                  0              0   \n",
       "\n",
       "   ind_var43_emit_ult1  ind_var43_recib_ult1  var21  num_var2_0_ult1  \\\n",
       "0                    0                     0      0                0   \n",
       "1                    0                     0      0                0   \n",
       "2                    0                     0      0                0   \n",
       "3                    0                     0      0                0   \n",
       "4                    1                     1      0                0   \n",
       "\n",
       "   num_var2_ult1  num_aport_var13_hace3  num_aport_var13_ult1  \\\n",
       "0              0                      0                     0   \n",
       "1              0                      3                     0   \n",
       "2              0                      0                     0   \n",
       "3              0                      0                     0   \n",
       "4              0                      0                     0   \n",
       "\n",
       "   num_aport_var17_hace3  num_aport_var17_ult1  num_aport_var33_hace3  \\\n",
       "0                      0                     0                      0   \n",
       "1                      0                     0                      0   \n",
       "2                      0                     0                      0   \n",
       "3                      0                     0                      0   \n",
       "4                      0                     0                      0   \n",
       "\n",
       "   num_aport_var33_ult1  num_var7_emit_ult1  num_var7_recib_ult1  \\\n",
       "0                     0                   0                    0   \n",
       "1                     0                   0                    0   \n",
       "2                     0                   0                    0   \n",
       "3                     0                   0                    0   \n",
       "4                     0                   0                    0   \n",
       "\n",
       "   num_compra_var44_hace3  num_compra_var44_ult1  num_ent_var16_ult1  \\\n",
       "0                       0                      0                   0   \n",
       "1                       0                      0                   0   \n",
       "2                       0                      0                   0   \n",
       "3                       0                      0                   0   \n",
       "4                       0                      0                   0   \n",
       "\n",
       "   num_var22_hace2  num_var22_hace3  num_var22_ult1  num_var22_ult3  \\\n",
       "0                0                0               0               0   \n",
       "1                0                0               0               0   \n",
       "2                0                0               0               0   \n",
       "3                3                0               0               3   \n",
       "4                3                0               6               9   \n",
       "\n",
       "   num_med_var22_ult3  num_med_var45_ult3  num_meses_var5_ult3  \\\n",
       "0                   0                   0                    0   \n",
       "1                   0                   0                    1   \n",
       "2                   0                   0                    3   \n",
       "3                   0                  15                    2   \n",
       "4                   3                   0                    3   \n",
       "\n",
       "   num_meses_var8_ult3  num_meses_var12_ult3  num_meses_var13_corto_ult3  \\\n",
       "0                    0                     0                           0   \n",
       "1                    0                     0                           3   \n",
       "2                    0                     0                           0   \n",
       "3                    0                     0                           0   \n",
       "4                    0                     1                           0   \n",
       "\n",
       "   num_meses_var13_largo_ult3  num_meses_var13_medio_ult3  \\\n",
       "0                           0                           0   \n",
       "1                           0                           0   \n",
       "2                           0                           0   \n",
       "3                           0                           0   \n",
       "4                           0                           0   \n",
       "\n",
       "   num_meses_var17_ult3  num_meses_var29_ult3  num_meses_var33_ult3  \\\n",
       "0                     0                     0                     0   \n",
       "1                     0                     0                     0   \n",
       "2                     0                     0                     0   \n",
       "3                     0                     0                     0   \n",
       "4                     0                     0                     0   \n",
       "\n",
       "   num_meses_var39_vig_ult3  num_meses_var44_ult3  num_op_var39_comer_ult1  \\\n",
       "0                         2                     0                        0   \n",
       "1                         2                     0                        0   \n",
       "2                         1                     0                        0   \n",
       "3                         1                     0                        9   \n",
       "4                         2                     0                        0   \n",
       "\n",
       "   num_op_var39_comer_ult3  num_op_var40_comer_ult1  num_op_var40_comer_ult3  \\\n",
       "0                        0                        0                        0   \n",
       "1                        0                        0                        0   \n",
       "2                        0                        0                        0   \n",
       "3                        9                        0                        0   \n",
       "4                        0                        0                        0   \n",
       "\n",
       "   num_op_var40_efect_ult1  num_op_var40_efect_ult3  num_op_var41_comer_ult1  \\\n",
       "0                        0                        0                        0   \n",
       "1                        0                        0                        0   \n",
       "2                        0                        0                        0   \n",
       "3                        0                        0                        9   \n",
       "4                        0                        0                        0   \n",
       "\n",
       "   num_op_var41_comer_ult3  num_op_var41_efect_ult1  num_op_var41_efect_ult3  \\\n",
       "0                        0                        0                        0   \n",
       "1                        0                        0                        0   \n",
       "2                        0                        0                        0   \n",
       "3                        9                        0                        0   \n",
       "4                        0                        0                        0   \n",
       "\n",
       "   num_op_var39_efect_ult1  num_op_var39_efect_ult3  num_reemb_var13_hace3  \\\n",
       "0                        0                        0                      0   \n",
       "1                        0                        0                      0   \n",
       "2                        0                        0                      0   \n",
       "3                        0                        0                      0   \n",
       "4                        0                        0                      0   \n",
       "\n",
       "   num_reemb_var13_ult1  num_reemb_var17_hace3  num_reemb_var17_ult1  \\\n",
       "0                     0                      0                     0   \n",
       "1                     0                      0                     0   \n",
       "2                     0                      0                     0   \n",
       "3                     0                      0                     0   \n",
       "4                     0                      0                     0   \n",
       "\n",
       "   num_reemb_var33_hace3  num_reemb_var33_ult1  num_sal_var16_ult1  \\\n",
       "0                      0                     0                   0   \n",
       "1                      0                     0                   0   \n",
       "2                      0                     0                   0   \n",
       "3                      0                     0                   0   \n",
       "4                      0                     0                   0   \n",
       "\n",
       "   num_var43_emit_ult1  num_var43_recib_ult1  num_trasp_var11_ult1  \\\n",
       "0                    0                     0                     0   \n",
       "1                    0                     0                     0   \n",
       "2                    0                     0                     0   \n",
       "3                    0                     0                     0   \n",
       "4                    3                     6                     3   \n",
       "\n",
       "   num_trasp_var17_in_hace3  num_trasp_var17_in_ult1  \\\n",
       "0                         0                        0   \n",
       "1                         0                        0   \n",
       "2                         0                        0   \n",
       "3                         0                        0   \n",
       "4                         0                        0   \n",
       "\n",
       "   num_trasp_var17_out_hace3  num_trasp_var17_out_ult1  \\\n",
       "0                          0                         0   \n",
       "1                          0                         0   \n",
       "2                          0                         0   \n",
       "3                          0                         0   \n",
       "4                          0                         0   \n",
       "\n",
       "   num_trasp_var33_in_hace3  num_trasp_var33_in_ult1  \\\n",
       "0                         0                        0   \n",
       "1                         0                        0   \n",
       "2                         0                        0   \n",
       "3                         0                        0   \n",
       "4                         0                        0   \n",
       "\n",
       "   num_trasp_var33_out_hace3  num_trasp_var33_out_ult1  num_venta_var44_hace3  \\\n",
       "0                          0                         0                      0   \n",
       "1                          0                         0                      0   \n",
       "2                          0                         0                      0   \n",
       "3                          0                         0                      0   \n",
       "4                          0                         0                      0   \n",
       "\n",
       "   num_venta_var44_ult1  num_var45_hace2  num_var45_hace3  num_var45_ult1  \\\n",
       "0                     0                0                0               0   \n",
       "1                     0                0                0               0   \n",
       "2                     0                0                0               0   \n",
       "3                     0               27                3              18   \n",
       "4                     0                0                0               0   \n",
       "\n",
       "   num_var45_ult3  saldo_var2_ult1  saldo_medio_var5_hace2  \\\n",
       "0               0                0                    0.00   \n",
       "1               0                0                    0.00   \n",
       "2               0                0                    3.00   \n",
       "3              48                0                  186.09   \n",
       "4               0                0                    3.00   \n",
       "\n",
       "   saldo_medio_var5_hace3  saldo_medio_var5_ult1  saldo_medio_var5_ult3  \\\n",
       "0                    0.00                   0.00                   0.00   \n",
       "1                   88.89                   0.00                   0.00   \n",
       "2                    0.18                   3.00                   2.07   \n",
       "3                    0.00                  91.56                 138.84   \n",
       "4                    0.30               40501.08               13501.47   \n",
       "\n",
       "   saldo_medio_var8_hace2  saldo_medio_var8_hace3  saldo_medio_var8_ult1  \\\n",
       "0                     0.0                     0.0                    0.0   \n",
       "1                     0.0                     0.0                    0.0   \n",
       "2                     0.0                     0.0                    0.0   \n",
       "3                     0.0                     0.0                    0.0   \n",
       "4                     0.0                     0.0                    0.0   \n",
       "\n",
       "   saldo_medio_var8_ult3  saldo_medio_var12_hace2  saldo_medio_var12_hace3  \\\n",
       "0                    0.0                      0.0                      0.0   \n",
       "1                    0.0                      0.0                      0.0   \n",
       "2                    0.0                      0.0                      0.0   \n",
       "3                    0.0                      0.0                      0.0   \n",
       "4                    0.0                      0.0                      0.0   \n",
       "\n",
       "   saldo_medio_var12_ult1  saldo_medio_var12_ult3  \\\n",
       "0                    0.00                    0.00   \n",
       "1                    0.00                    0.00   \n",
       "2                    0.00                    0.00   \n",
       "3                    0.00                    0.00   \n",
       "4                85501.89                85501.89   \n",
       "\n",
       "   saldo_medio_var13_corto_hace2  saldo_medio_var13_corto_hace3  \\\n",
       "0                            0.0                           0.00   \n",
       "1                          300.0                         122.22   \n",
       "2                            0.0                           0.00   \n",
       "3                            0.0                           0.00   \n",
       "4                            0.0                           0.00   \n",
       "\n",
       "   saldo_medio_var13_corto_ult1  saldo_medio_var13_corto_ult3  \\\n",
       "0                           0.0                          0.00   \n",
       "1                         300.0                        240.75   \n",
       "2                           0.0                          0.00   \n",
       "3                           0.0                          0.00   \n",
       "4                           0.0                          0.00   \n",
       "\n",
       "   saldo_medio_var13_largo_hace2  saldo_medio_var13_largo_hace3  \\\n",
       "0                            0.0                            0.0   \n",
       "1                            0.0                            0.0   \n",
       "2                            0.0                            0.0   \n",
       "3                            0.0                            0.0   \n",
       "4                            0.0                            0.0   \n",
       "\n",
       "   saldo_medio_var13_largo_ult1  saldo_medio_var13_largo_ult3  \\\n",
       "0                           0.0                           0.0   \n",
       "1                           0.0                           0.0   \n",
       "2                           0.0                           0.0   \n",
       "3                           0.0                           0.0   \n",
       "4                           0.0                           0.0   \n",
       "\n",
       "   saldo_medio_var13_medio_hace2  saldo_medio_var13_medio_hace3  \\\n",
       "0                            0.0                              0   \n",
       "1                            0.0                              0   \n",
       "2                            0.0                              0   \n",
       "3                            0.0                              0   \n",
       "4                            0.0                              0   \n",
       "\n",
       "   saldo_medio_var13_medio_ult1  saldo_medio_var13_medio_ult3  \\\n",
       "0                             0                           0.0   \n",
       "1                             0                           0.0   \n",
       "2                             0                           0.0   \n",
       "3                             0                           0.0   \n",
       "4                             0                           0.0   \n",
       "\n",
       "   saldo_medio_var17_hace2  saldo_medio_var17_hace3  saldo_medio_var17_ult1  \\\n",
       "0                      0.0                      0.0                     0.0   \n",
       "1                      0.0                      0.0                     0.0   \n",
       "2                      0.0                      0.0                     0.0   \n",
       "3                      0.0                      0.0                     0.0   \n",
       "4                      0.0                      0.0                     0.0   \n",
       "\n",
       "   saldo_medio_var17_ult3  saldo_medio_var29_hace2  saldo_medio_var29_hace3  \\\n",
       "0                     0.0                      0.0                      0.0   \n",
       "1                     0.0                      0.0                      0.0   \n",
       "2                     0.0                      0.0                      0.0   \n",
       "3                     0.0                      0.0                      0.0   \n",
       "4                     0.0                      0.0                      0.0   \n",
       "\n",
       "   saldo_medio_var29_ult1  saldo_medio_var29_ult3  saldo_medio_var33_hace2  \\\n",
       "0                     0.0                     0.0                      0.0   \n",
       "1                     0.0                     0.0                      0.0   \n",
       "2                     0.0                     0.0                      0.0   \n",
       "3                     0.0                     0.0                      0.0   \n",
       "4                     0.0                     0.0                      0.0   \n",
       "\n",
       "   saldo_medio_var33_hace3  saldo_medio_var33_ult1  saldo_medio_var33_ult3  \\\n",
       "0                      0.0                     0.0                     0.0   \n",
       "1                      0.0                     0.0                     0.0   \n",
       "2                      0.0                     0.0                     0.0   \n",
       "3                      0.0                     0.0                     0.0   \n",
       "4                      0.0                     0.0                     0.0   \n",
       "\n",
       "   saldo_medio_var44_hace2  saldo_medio_var44_hace3  saldo_medio_var44_ult1  \\\n",
       "0                      0.0                      0.0                     0.0   \n",
       "1                      0.0                      0.0                     0.0   \n",
       "2                      0.0                      0.0                     0.0   \n",
       "3                      0.0                      0.0                     0.0   \n",
       "4                      0.0                      0.0                     0.0   \n",
       "\n",
       "   saldo_medio_var44_ult3          var38  TARGET  \n",
       "0                     0.0   39205.170000       0  \n",
       "1                     0.0   49278.030000       0  \n",
       "2                     0.0   67333.770000       0  \n",
       "3                     0.0   64007.970000       0  \n",
       "4                     0.0  117310.979016       0  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import data\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('./data/santander.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Constant features are features that have always the same value. \n",
    "\n",
    "A quick way to inspect this is to look for features with null standard deviation. This method works for numeric features. Object features follow a similar approach, but there are some differences that we will discuss later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Get list of possible constant features \n",
    "s = df.describe().loc['std']  # Get standard deviation of each feature\n",
    "candidates = s.where(s==0).dropna().index  # Get list of features with null standard deviation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ind_var2_0</th>\n",
       "      <th>ind_var2</th>\n",
       "      <th>ind_var27_0</th>\n",
       "      <th>ind_var28_0</th>\n",
       "      <th>ind_var28</th>\n",
       "      <th>ind_var27</th>\n",
       "      <th>ind_var41</th>\n",
       "      <th>ind_var46_0</th>\n",
       "      <th>ind_var46</th>\n",
       "      <th>num_var27_0</th>\n",
       "      <th>num_var28_0</th>\n",
       "      <th>num_var28</th>\n",
       "      <th>num_var27</th>\n",
       "      <th>num_var41</th>\n",
       "      <th>num_var46_0</th>\n",
       "      <th>num_var46</th>\n",
       "      <th>saldo_var28</th>\n",
       "      <th>saldo_var27</th>\n",
       "      <th>saldo_var41</th>\n",
       "      <th>saldo_var46</th>\n",
       "      <th>imp_amort_var18_hace3</th>\n",
       "      <th>imp_amort_var34_hace3</th>\n",
       "      <th>imp_reemb_var13_hace3</th>\n",
       "      <th>imp_reemb_var33_hace3</th>\n",
       "      <th>imp_trasp_var17_out_hace3</th>\n",
       "      <th>imp_trasp_var33_out_hace3</th>\n",
       "      <th>num_var2_0_ult1</th>\n",
       "      <th>num_var2_ult1</th>\n",
       "      <th>num_reemb_var13_hace3</th>\n",
       "      <th>num_reemb_var33_hace3</th>\n",
       "      <th>num_trasp_var17_out_hace3</th>\n",
       "      <th>num_trasp_var33_out_hace3</th>\n",
       "      <th>saldo_var2_ult1</th>\n",
       "      <th>saldo_medio_var13_medio_hace3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "      <td>76020.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       ind_var2_0  ind_var2  ind_var27_0  ind_var28_0  ind_var28  ind_var27  \\\n",
       "count     76020.0   76020.0      76020.0      76020.0    76020.0    76020.0   \n",
       "mean          0.0       0.0          0.0          0.0        0.0        0.0   \n",
       "std           0.0       0.0          0.0          0.0        0.0        0.0   \n",
       "min           0.0       0.0          0.0          0.0        0.0        0.0   \n",
       "25%           0.0       0.0          0.0          0.0        0.0        0.0   \n",
       "50%           0.0       0.0          0.0          0.0        0.0        0.0   \n",
       "75%           0.0       0.0          0.0          0.0        0.0        0.0   \n",
       "max           0.0       0.0          0.0          0.0        0.0        0.0   \n",
       "\n",
       "       ind_var41  ind_var46_0  ind_var46  num_var27_0  num_var28_0  num_var28  \\\n",
       "count    76020.0      76020.0    76020.0      76020.0      76020.0    76020.0   \n",
       "mean         0.0          0.0        0.0          0.0          0.0        0.0   \n",
       "std          0.0          0.0        0.0          0.0          0.0        0.0   \n",
       "min          0.0          0.0        0.0          0.0          0.0        0.0   \n",
       "25%          0.0          0.0        0.0          0.0          0.0        0.0   \n",
       "50%          0.0          0.0        0.0          0.0          0.0        0.0   \n",
       "75%          0.0          0.0        0.0          0.0          0.0        0.0   \n",
       "max          0.0          0.0        0.0          0.0          0.0        0.0   \n",
       "\n",
       "       num_var27  num_var41  num_var46_0  num_var46  saldo_var28  saldo_var27  \\\n",
       "count    76020.0    76020.0      76020.0    76020.0      76020.0      76020.0   \n",
       "mean         0.0        0.0          0.0        0.0          0.0          0.0   \n",
       "std          0.0        0.0          0.0        0.0          0.0          0.0   \n",
       "min          0.0        0.0          0.0        0.0          0.0          0.0   \n",
       "25%          0.0        0.0          0.0        0.0          0.0          0.0   \n",
       "50%          0.0        0.0          0.0        0.0          0.0          0.0   \n",
       "75%          0.0        0.0          0.0        0.0          0.0          0.0   \n",
       "max          0.0        0.0          0.0        0.0          0.0          0.0   \n",
       "\n",
       "       saldo_var41  saldo_var46  imp_amort_var18_hace3  imp_amort_var34_hace3  \\\n",
       "count      76020.0      76020.0                76020.0                76020.0   \n",
       "mean           0.0          0.0                    0.0                    0.0   \n",
       "std            0.0          0.0                    0.0                    0.0   \n",
       "min            0.0          0.0                    0.0                    0.0   \n",
       "25%            0.0          0.0                    0.0                    0.0   \n",
       "50%            0.0          0.0                    0.0                    0.0   \n",
       "75%            0.0          0.0                    0.0                    0.0   \n",
       "max            0.0          0.0                    0.0                    0.0   \n",
       "\n",
       "       imp_reemb_var13_hace3  imp_reemb_var33_hace3  \\\n",
       "count                76020.0                76020.0   \n",
       "mean                     0.0                    0.0   \n",
       "std                      0.0                    0.0   \n",
       "min                      0.0                    0.0   \n",
       "25%                      0.0                    0.0   \n",
       "50%                      0.0                    0.0   \n",
       "75%                      0.0                    0.0   \n",
       "max                      0.0                    0.0   \n",
       "\n",
       "       imp_trasp_var17_out_hace3  imp_trasp_var33_out_hace3  num_var2_0_ult1  \\\n",
       "count                    76020.0                    76020.0          76020.0   \n",
       "mean                         0.0                        0.0              0.0   \n",
       "std                          0.0                        0.0              0.0   \n",
       "min                          0.0                        0.0              0.0   \n",
       "25%                          0.0                        0.0              0.0   \n",
       "50%                          0.0                        0.0              0.0   \n",
       "75%                          0.0                        0.0              0.0   \n",
       "max                          0.0                        0.0              0.0   \n",
       "\n",
       "       num_var2_ult1  num_reemb_var13_hace3  num_reemb_var33_hace3  \\\n",
       "count        76020.0                76020.0                76020.0   \n",
       "mean             0.0                    0.0                    0.0   \n",
       "std              0.0                    0.0                    0.0   \n",
       "min              0.0                    0.0                    0.0   \n",
       "25%              0.0                    0.0                    0.0   \n",
       "50%              0.0                    0.0                    0.0   \n",
       "75%              0.0                    0.0                    0.0   \n",
       "max              0.0                    0.0                    0.0   \n",
       "\n",
       "       num_trasp_var17_out_hace3  num_trasp_var33_out_hace3  saldo_var2_ult1  \\\n",
       "count                    76020.0                    76020.0          76020.0   \n",
       "mean                         0.0                        0.0              0.0   \n",
       "std                          0.0                        0.0              0.0   \n",
       "min                          0.0                        0.0              0.0   \n",
       "25%                          0.0                        0.0              0.0   \n",
       "50%                          0.0                        0.0              0.0   \n",
       "75%                          0.0                        0.0              0.0   \n",
       "max                          0.0                        0.0              0.0   \n",
       "\n",
       "       saldo_medio_var13_medio_hace3  \n",
       "count                        76020.0  \n",
       "mean                             0.0  \n",
       "std                              0.0  \n",
       "min                              0.0  \n",
       "25%                              0.0  \n",
       "50%                              0.0  \n",
       "75%                              0.0  \n",
       "max                              0.0  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inspect features\n",
    "pd.set_option('display.max_columns', 500)  # Change definitions to show all columns\n",
    "df[candidates].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All features are constant. They have always the same value, which is zero in this case. Let's delete these features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Delete features\n",
    "df.drop(candidates, axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False, False, False, False, False, False, False, False, False,\n",
       "       False, False, False, False, False, False, False, False, False,\n",
       "       False, False, False, False, False, False, False, False, False,\n",
       "       False, False, False, False, False, False, False])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test \n",
    "candidates.isin(df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the features that we identified as constant are no longer in the dataset. The dataset is now clean from numeric constant features.\n",
    "\n",
    "What about object features? Let's see."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Series([], dtype: object)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get character features\n",
    "df.dtypes.where(df.dtypes=='object').dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, we don't have object features. If we had, we would have to describe them and search for those with only one value.\n",
    "\n",
    "Let's create a fake dataset to exemplify this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>player</th>\n",
       "      <th>team</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>eusebio</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>jonas</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>magnusson</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>nene</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      player     team\n",
       "0    eusebio  benfica\n",
       "1      jonas  benfica\n",
       "2  magnusson  benfica\n",
       "3       nene  benfica"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create example \n",
    "df = pd.DataFrame({'player':['eusebio', 'jonas', 'magnusson', 'nene'],\n",
    "                  'team':['benfica', 'benfica', 'benfica', 'benfica']})\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['team'], dtype='object')"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Search for constant features\n",
    "s = df.describe().loc['unique']  # Look for the number of unique values in each feature\n",
    "candidates = s.where(s==1).dropna().index  # Get list of features with only one unique value\n",
    "candidates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the *team* feature is the only constant features. All the observations have the same *team* value (benfica)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>player</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>eusebio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>jonas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>magnusson</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>nene</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      player\n",
       "0    eusebio\n",
       "1      jonas\n",
       "2  magnusson\n",
       "3       nene"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Delete constant features\n",
    "df.drop(candidates, axis=1, inplace=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clean! Now we only have those non-constant features.\n",
    "\n",
    "**Note:** The *.describe()* structure of this dataset is different from the one we saw on the Santander dataset. That happens because now we are dealing with object features and previously we were dealing with numeric features. You can read more on how to describe datasets with different data types [here](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.describe.html).\n",
    "\n",
    "[Go back to the top](#General-data-cleaning-techniques)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Duplicated rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now for something completely different, a Benfica dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>favorite_team</th>\n",
       "      <th>goals</th>\n",
       "      <th>player</th>\n",
       "      <th>team</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>benfica</td>\n",
       "      <td>102</td>\n",
       "      <td>eusebio</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>olhanense</td>\n",
       "      <td>44</td>\n",
       "      <td>djalmir</td>\n",
       "      <td>olhanense</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>benfica</td>\n",
       "      <td>78</td>\n",
       "      <td>magnusson</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>benfica</td>\n",
       "      <td>65</td>\n",
       "      <td>nene</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>benfica</td>\n",
       "      <td>102</td>\n",
       "      <td>eusebio</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  favorite_team  goals     player       team\n",
       "0       benfica    102    eusebio    benfica\n",
       "1     olhanense     44    djalmir  olhanense\n",
       "2       benfica     78  magnusson    benfica\n",
       "3       benfica     65       nene    benfica\n",
       "4       benfica    102    eusebio    benfica"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create example \n",
    "df = pd.DataFrame({'favorite_team':['benfica', 'olhanense', 'benfica', 'benfica', 'benfica'],\n",
    "                   'player':['eusebio', 'djalmir', 'magnusson', 'nene', 'eusebio'],\n",
    "                   'team':['benfica', 'olhanense', 'benfica', 'benfica','benfica'],\n",
    "                   'goals':[102,44,78,65,102]})\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To delete the duplicated rows, we just need to apply [pandas.DataFrame.drop_duplicates](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop_duplicates.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>favorite_team</th>\n",
       "      <th>goals</th>\n",
       "      <th>player</th>\n",
       "      <th>team</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>olhanense</td>\n",
       "      <td>44</td>\n",
       "      <td>djalmir</td>\n",
       "      <td>olhanense</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>benfica</td>\n",
       "      <td>78</td>\n",
       "      <td>magnusson</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>benfica</td>\n",
       "      <td>65</td>\n",
       "      <td>nene</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>benfica</td>\n",
       "      <td>102</td>\n",
       "      <td>eusebio</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  favorite_team  goals     player       team\n",
       "1     olhanense     44    djalmir  olhanense\n",
       "2       benfica     78  magnusson    benfica\n",
       "3       benfica     65       nene    benfica\n",
       "4       benfica    102    eusebio    benfica"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Delete duplicated rows\n",
    "df.drop_duplicates(df, keep='last', inplace=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that it deleted row 0. That happened because I chose to keep the last duplicated observation. If I had chosen to keep the first duplicated observation, then row 4 would have been deleted instead of row 0.\n",
    "\n",
    "[Go back to the top](#General-data-cleaning-techniques)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Duplicated features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's extend the previous example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>favorite_team</th>\n",
       "      <th>goals</th>\n",
       "      <th>player</th>\n",
       "      <th>team</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>benfica</td>\n",
       "      <td>102</td>\n",
       "      <td>eusebio</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>olhanense</td>\n",
       "      <td>44</td>\n",
       "      <td>djalmir</td>\n",
       "      <td>olhanense</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>benfica</td>\n",
       "      <td>78</td>\n",
       "      <td>magnusson</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>benfica</td>\n",
       "      <td>65</td>\n",
       "      <td>nene</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  favorite_team  goals     player       team\n",
       "0       benfica    102    eusebio    benfica\n",
       "1     olhanense     44    djalmir  olhanense\n",
       "2       benfica     78  magnusson    benfica\n",
       "3       benfica     65       nene    benfica"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create example \n",
    "df = pd.DataFrame({'favorite_team':['benfica', 'olhanense', 'benfica', 'benfica'],\n",
    "                   'player':['eusebio', 'djalmir', 'magnusson', 'nene'],\n",
    "                   'team':['benfica', 'olhanense', 'benfica', 'benfica'],\n",
    "                   'goals':[102,44,78,65]})\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, our dataset has duplicated features (*favorite_team* and *team*). In this case, we can't apply *drop_duplicates* directly because it only works to detect duplicated rows. However, there's a smart workaround."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>favorite_team</th>\n",
       "      <th>goals</th>\n",
       "      <th>player</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>benfica</td>\n",
       "      <td>102</td>\n",
       "      <td>eusebio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>olhanense</td>\n",
       "      <td>44</td>\n",
       "      <td>djalmir</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>benfica</td>\n",
       "      <td>78</td>\n",
       "      <td>magnusson</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>benfica</td>\n",
       "      <td>65</td>\n",
       "      <td>nene</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  favorite_team goals     player\n",
       "0       benfica   102    eusebio\n",
       "1     olhanense    44    djalmir\n",
       "2       benfica    78  magnusson\n",
       "3       benfica    65       nene"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Delete duplicated features\n",
    "temp = df.T.drop_duplicates()  # Transform columns into rows\n",
    "df = temp.T\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Voilà! Now the dataset is clean."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Go back to the top](#General-data-cleaning-techniques)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Values out of range"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we only deal with those values that are clearly wrong because they are impossible to happen. For example, no one can have a negative age (not even Benjamin Button). There is a different type of values out of range, which belongs to the outliers analysis. Those values correspond to situations that are anomalous, but not necessarily impossible. We will analyse outliers in a different notebook.\n",
    "\n",
    "For now, let's check what's going on in our favorite dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>player</th>\n",
       "      <th>team</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>71</td>\n",
       "      <td>eusebio</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>55</td>\n",
       "      <td>magnusson</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-68</td>\n",
       "      <td>nene</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age     player     team\n",
       "0   71    eusebio  benfica\n",
       "1   55  magnusson  benfica\n",
       "2  -68       nene  benfica"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create example \n",
    "df = pd.DataFrame({'player':['eusebio', 'magnusson', 'nene'],\n",
    "                   'team':['benfica', 'benfica', 'benfica'],\n",
    "                   'age':[71,55,-68]})\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a real case, I'd probably detect that Nené's age is wrong through one of two processes:\n",
    "1. Impose up front that all age values must be above 0 (kind of sanity check).\n",
    "1. Check minimum and maximum values in the descriptive statistics.\n",
    "\n",
    "Let's see how to proceed in each case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sanity check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>player</th>\n",
       "      <th>team</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>71</td>\n",
       "      <td>eusebio</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>55</td>\n",
       "      <td>magnusson</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age     player     team\n",
       "0   71    eusebio  benfica\n",
       "1   55  magnusson  benfica"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Impose condition for age values\n",
    "df_sane = df[df['age']>0]\n",
    "df_sane"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Descriptive statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>3.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>19.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>76.054805</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-68.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-6.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>55.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>63.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>71.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             age\n",
       "count   3.000000\n",
       "mean   19.333333\n",
       "std    76.054805\n",
       "min   -68.000000\n",
       "25%    -6.500000\n",
       "50%    55.000000\n",
       "75%    63.000000\n",
       "max    71.000000"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Case 2\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the things that I pay attention when analysing descriptive statistics is *min* and *max* values. It's one of the best ways I know to catch those gross errors that would emabarass you in front of your boss.\n",
    "\n",
    "In this case, the value -68 would get my attention and I'd remove it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>player</th>\n",
       "      <th>team</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-68</td>\n",
       "      <td>nene</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age player     team\n",
       "2  -68   nene  benfica"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# See what's going on\n",
    "df[df['age']==df['age'].min()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>player</th>\n",
       "      <th>team</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>71</td>\n",
       "      <td>eusebio</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>55</td>\n",
       "      <td>magnusson</td>\n",
       "      <td>benfica</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age     player     team\n",
       "0   71    eusebio  benfica\n",
       "1   55  magnusson  benfica"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop row\n",
    "df = df[df['age']>df['age'].min()]\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's good practice to check descriptive statistics again. We never know what else is hidding there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>63.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>11.313708</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>55.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>59.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>63.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>67.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>71.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             age\n",
       "count   2.000000\n",
       "mean   63.000000\n",
       "std    11.313708\n",
       "min    55.000000\n",
       "25%    59.000000\n",
       "50%    63.000000\n",
       "75%    67.000000\n",
       "max    71.000000"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Descriptive statistics (take 2)\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks nice. Mission accomplished!\n",
    "\n",
    "[Go back to the top](#General-data-cleaning-techniques)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Shuffle dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Datasets should be shuffled before training machine learning models to avoid bias/patterns situations. It's easy to forget about this, but in many cases datasets are ordered in such a way that can bias your analysis.\n",
    "\n",
    "Consider the following example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>name</th>\n",
       "      <th>salary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18</td>\n",
       "      <td>pussidonio</td>\n",
       "      <td>780</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>19</td>\n",
       "      <td>benquerenca</td>\n",
       "      <td>767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20</td>\n",
       "      <td>jorinho</td>\n",
       "      <td>750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>19</td>\n",
       "      <td>balsagodes</td>\n",
       "      <td>760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>56</td>\n",
       "      <td>asdrubal</td>\n",
       "      <td>2580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>58</td>\n",
       "      <td>tamagnini</td>\n",
       "      <td>2750</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age         name salary\n",
       "0   18   pussidonio    780\n",
       "1   19  benquerenca    767\n",
       "2   20      jorinho    750\n",
       "3   19   balsagodes    760\n",
       "4   56     asdrubal   2580\n",
       "5   58    tamagnini   2750"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create example \n",
    "df = pd.DataFrame({'name':['pussidonio', 'benquerenca', 'jorinho','balsagodes','asdrubal','tamagnini'],\n",
    "                   'salary':['780', '767', '750','760','2580','2750'],\n",
    "                   'age':[18,19,20,19,56,58]})\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now imagine that you have the following train and test datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create train and test datasets\n",
    "import numpy as np\n",
    "\n",
    "train_size = 0.8\n",
    "n_train = int(np.shape(df)[0] * train_size)\n",
    "\n",
    "df_train = df.iloc[:n_train]\n",
    "df_test = df.iloc[n_train:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you create such datasets, you're in trouble. Comparing both datasets, we can see that the distribution of values is completely different. While in the train set age is between 18-20 and salaries are around 764, in the test set age is between 56-58 and salaries around 2665. Accordingly, using our train set we can't learn much about our test set.\n",
    "\n",
    "The message here is that we should shuffle our dataset to avoid this type of segregation, which bias our analysis. There are two ways to do so: \n",
    "1. Directly shuffle the dataset.\n",
    "1. Create train and test sets with sklearn-learn."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Directly shuffle the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>name</th>\n",
       "      <th>salary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18</td>\n",
       "      <td>pussidonio</td>\n",
       "      <td>780</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>19</td>\n",
       "      <td>balsagodes</td>\n",
       "      <td>760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>56</td>\n",
       "      <td>asdrubal</td>\n",
       "      <td>2580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>19</td>\n",
       "      <td>benquerenca</td>\n",
       "      <td>767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20</td>\n",
       "      <td>jorinho</td>\n",
       "      <td>750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>58</td>\n",
       "      <td>tamagnini</td>\n",
       "      <td>2750</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age         name salary\n",
       "0   18   pussidonio    780\n",
       "3   19   balsagodes    760\n",
       "4   56     asdrubal   2580\n",
       "1   19  benquerenca    767\n",
       "2   20      jorinho    750\n",
       "5   58    tamagnini   2750"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Shuffle dataset\n",
    "from sklearn.utils import shuffle\n",
    "\n",
    "df_shuffled = shuffle(df)\n",
    "df_shuffled"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the data is mixed. Note that we can still have [imbalanced datasets](https://blog.dominodatalab.com/imbalanced-datasets/), but that's a different problem. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create train and test sets with scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train:\n",
      "    age        name\n",
      "5   58   tamagnini\n",
      "3   19  balsagodes\n",
      "2   20     jorinho\n",
      "4   56    asdrubal\n",
      "X_test:\n",
      "    age         name\n",
      "1   19  benquerenca\n",
      "0   18   pussidonio\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X = df.drop('salary', axis=1)\n",
    "y = df['salary']\n",
    "\n",
    "X_train, X_test, y_train, y_test  = train_test_split(X, y, test_size=0.2)\n",
    "print('X_train:\\n', X_train)\n",
    "print('X_test:\\n', X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same comment as before: data is mixed but we can have imbalanced datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Go back to the top](#General-data-cleaning-techniques)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}