Add column accessor attribute to check for nullable logical type (#1127)

* Add WoodworkColumnAccessor.nullable method and tests

* Update docs and release notes

* Minor refactors in tests

* Remove testing of multiple collection types

Testing all three collection types means we're basically just testing
_get_valid_dtype again.  We already know that works from other tests so
we don't need to test it here.

* Update release notes

* Remove unsupported dtypes from _NULLABLE_PHYSICAL_TYPES

* Add WoodworkColumnAccessor.nullable to API docs

* Update nullability docs

* Update docstring

Co-authored-by: David Sanders <[email protected]>

Author: jeff-hernandez

docs/source/api_reference.rst

@@ -60,6 +60,7 @@ WoodworkColumnAccessor
     WoodworkColumnAccessor.loc
     WoodworkColumnAccessor.logical_type
     WoodworkColumnAccessor.metadata
+    WoodworkColumnAccessor.nullable
     WoodworkColumnAccessor.remove_semantic_tags
     WoodworkColumnAccessor.reset_semantic_tags
     WoodworkColumnAccessor.semantic_tags

docs/source/guides/logical_types_and_semantic_tags.ipynb

@@ -132,6 +132,7 @@
     "When Woodwork's type inference does not return any LogicalTypes for a column, Woodwork will set the column's logical type as the default LogicalType, `Unknown`. A logical type being inferred as `Unknown` may be a good indicator that a more specific logical type can be chosen and set by the user.\n",
     "\n",
     "- **physical type**: `string`\n",
+    "- **nullable**: yes\n",
     "\n",
     "Below is an example of a column for which no logical type is inferred, resulting in a Series with `Unknown` logical type. Looking at the contents of the Series, though, we can see that it contains country codes, so we set the logical type to `CountryCode`."
    ]
@@ -178,13 +179,15 @@
     "\n",
     "- **physical type**: `int64`\n",
     "- **standard tags**: `{'numeric'}`\n",
+    "- **nullable**: no\n",
     "\n",
     "##### AgeFractional\n",
     "\n",
     "Represents Logical Types that contain non-negative floating point numbers indicating a person’s age. May contain null values.\n",
     "\n",
     "- **physical type**: `float64`\n",
     "- **standard tags**: `{'numeric'}`\n",
+    "- **nullable**: yes\n",
     "\n",
     "\n",
     "##### AgeNullable\n",
@@ -193,26 +196,30 @@
     "\n",
     "- **physical type**: `Int64`\n",
     "- **standard tags**: `{'numeric'}`\n",
+    "- **nullable**: yes\n",
     "\n",
     "##### Double\n",
     "\n",
     "Represents Logical Types that contain positive and negative numbers, some of which include a fractional component.\n",
     "\n",
     "- **physical type**: `float64`\n",
     "- **standard tags**: `{'numeric'}`\n",
+    "- **nullable**: yes\n",
     "\n",
     "##### Integer\n",
     "\n",
     "Represents Logical Types that contain positive and negative numbers without a fractional component, including zero (0).\n",
     "\n",
     "- **physical type**: `int64`\n",
     "- **standard tags**: `{'numeric'}`\n",
+    "- **nullable**: no\n",
     "\n",
     "##### IntegerNullable \n",
     "Represents Logical Types that contain positive and negative numbers without a fractional component, including zero (0). May contain null values. \n",
     "\n",
     "- **physical type**: `Int64`\n",
     "- **standard tags**: `{'numeric'}`\n",
+    "- **nullable**: yes\n",
     "\n",
     "\n",
     "\n",
@@ -252,6 +259,7 @@
     "Represents a Logical Type with few unique values relative to the size of the data.\n",
     "\n",
     "- **physical type**: `category`\n",
+    "- **nullable**: yes\n",
     "- **inference**: Woodwork defines a threshold for percentage unique values relative to the size of the series below which a series will be considered categorical. See [setting config options guide](setting_config_options.ipynb#Categorical-Threshold) for more information on how to control this threshold.\n",
     "- **koalas note**: Koalas does not support the `category` dtype, so for Koalas DataFrames and Series, the `string` dtype will be used.\n",
     "\n",
@@ -269,6 +277,7 @@
     "Represents Logical Types that use the ISO-3166 standard country code to represent countries. ISO 3166-1 (countries) are supported. These codes should be in the Alpha-2 format.\n",
     "\n",
     "- **physical type**: `category`\n",
+    "- **nullable**: yes\n",
     "- **standard tags**: `{'category'}`\n",
     "- **koalas note**: Koalas does not support the `category` dtype, so for Koalas DataFrames and Series, the `string` dtype will be used.\n",
     "\n",
@@ -279,6 +288,7 @@
     "A Ordinal variable type can take ordered discrete values. Similar to Categorical, it is usually a limited, and fixed number of possible values. However, these discrete values have a certain order, and the ordering is important to understanding the values. Ordinal variable types can be represented as strings, or integers. \n",
     "\n",
     "- **physical type**: `category`\n",
+    "- **nullable**: yes\n",
     "- **standard tags**: `{'category'}`\n",
     "- **parameters**:\n",
     "    - `order` - the order of the ordinal values in the column from low to high\n",
@@ -298,6 +308,7 @@
     "Represents Logical Types that contain a series of postal codes for representing a group of addresses.\n",
     "\n",
     "- **physical type**: `category`\n",
+    "- **nullable**: yes\n",
     "- **standard tags**: `{'category'}`\n",
     "- **koalas note**: Koalas does not support the `category` dtype, so for Koalas DataFrames and Series, the `string` dtype will be used.\n",
     "\n",
@@ -306,6 +317,7 @@
     "Represents Logical Types that use the ISO-3166 standard sub-region code to represent a portion of a larger geographic region. ISO 3166-2 (sub-regions) codes are supported. These codes should be in the Alpha-2 format.\n",
     "\n",
     "- **physical type**: `category`\n",
+    "- **nullable**: yes\n",
     "- **standard tags**: `{'category'}`\n",
     "- **koalas note**: Koalas does not support the `category` dtype, so for Koalas DataFrames and Series, the `string` dtype will be used.\n",
     "\n",
@@ -348,16 +360,19 @@
     "Represents Logical Types that contain binary values indicating true/false.\n",
     "\n",
     "- **physical type**: `bool`\n",
+    "- **nullable**: no\n",
     "\n",
     "##### BooleanNullable\n",
     "Represents Logical Types that contain binary values indicating true/false. May also contain null values.\n",
     "\n",
     "- **physical type**: `boolean`\n",
+    "- **nullable**: yes\n",
     "\n",
     "##### Datetime\n",
     "A Datetime is a representation of a date and/or time. Datetime variable types can be represented as strings, or integers.\n",
     "\n",
     "- **physical type**: `datetime64[ns]`\n",
+    "- **nullable**: yes\n",
     "- **transformation**: Will convert valid strings or numbers to pandas datetimes, and will parse more datetime formats with the use of the `datetime_format` parameter.\n",
     "- **parameters**:\n",
     "    - `datetime_format` - the format of the datetimes in the column, ex: `'%Y-%m-%d'` vs `'%m-%d-%Y'`\n",
@@ -373,12 +388,14 @@
     "Represents Logical Types that contain email address values.\n",
     "\n",
     "- **physical type**: `string`\n",
+    "- **nullable**: yes\n",
     "- **inference**: Uses an email address regex that, if the data matches, means that the column contains email addresses. To learn more about controling the regex used, see the [setting config options guide](setting_config_options.ipynb#Email-Inference-Regex).\n",
     "\n",
     "##### LatLong\n",
     "A LatLong represents an ordered pair (Latitude, Longitude) that tells the location on Earth. The order of the tuple is important. LatLongs can be represented as tuple of floating point numbers. \n",
     "\n",
     "- **physical type**: `object`\n",
+    "- **nullable**: yes\n",
     "- **transformation**: Will convert inputs into a tuple of floats. Any null values will be stored as `np.nan`\n",
     "- **koalas note**: Koalas does not support tuples, so latlongs will be stored as a list of floats\n",
     "\n",
@@ -387,6 +404,7 @@
     "Represents Logical Types that contain values specifying a duration of time.\n",
     "\n",
     "- **physical type**: `timedelta64[ns]`\n",
+    "- **nullable**: yes\n",
     "\n",
     "\n",
     "Examples could inclue:\n",
@@ -457,6 +475,7 @@
     "Represents Logical Types that contain long-form text or characters representing natural human language\n",
     "\n",
     "- **physical type**: `string`\n",
+    "- **nullable**: yes\n",
     "\n",
     "Examples of natural language data:\n",
     "\n",
@@ -469,39 +488,45 @@
     "Represents Logical Types that contain address values.\n",
     "\n",
     "- **physical type**: `string`\n",
+    "- **nullable**: yes\n",
     "\n",
     "\n",
     "##### Filepath\n",
     "\n",
     "Represents Logical Types that specify locations of directories and files in a file system.\n",
     "\n",
     "- **physical type**: `string`\n",
+    "- **nullable**: yes\n",
     "\n",
     "\n",
     "##### PersonFullName\n",
     "\n",
     "Represents Logical Types that may contain first, middle and last names, including honorifics and suffixes.\n",
     "\n",
     "- **physical type**: `string`\n",
+    "- **nullable**: yes\n",
     "\n",
     "##### PhoneNumber\n",
     "\n",
     "Represents Logical Types that contain numeric digits and characters representing a phone number.\n",
     "\n",
     "- **physical type**: `string`\n",
+    "- **nullable**: yes\n",
     "\n",
     "\n",
     "##### URL\n",
     "\n",
     "Represents Logical Types that contain URLs, which may include protocol, hostname and file name.\n",
     "\n",
     "- **physical type**: `string`\n",
+    "- **nullable**: yes\n",
     "\n",
     "##### IPAddress\n",
     "\n",
     "Represents Logical Types that contain IP addresses, including both IPv4 and IPv6 addresses.\n",
     "\n",
-    "- **physical type**: `string`\n"
+    "- **physical type**: `string`\n",
+    "- **nullable**: yes\n"
    ]
   },
   {
@@ -676,6 +701,28 @@
     "\n",
     "In this way, a `ColumnSchema` can define a type space under which columns in a Woodwork DataFrame can fall."
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "sexual-adoption",
+   "metadata": {},
+   "source": [
+    "## Checking for nullable logical types\n",
+    "\n",
+    "Some logical types support having null values in the underlying data while others do not.  This is entirely based on whether a logical type's underlying primary dtype or backup dtype supports null values.  For example, the `EmailAddress` logical type has an underlying primary dtype of `string`.  Pandas allows series with the dtype `string` to contain null values marked by the `pandas.NA` sentinel.  Therefore, `EmailAddress` supports null values.  On the other hand, the `Integer` logical type does not support null values since its underlying primary pandas dtype is `int64`.  Pandas does not allow null values in series with the dtype `int64`.  However, pandas does allow null values in series with the dtype `Int64`.  Therefore, the `IntegerNullable` logical type supports null values since its primary dtype is `Int64`.\n",
+    "\n",
+    "You can check if a column contains a nullable logical type by using `nullable` on the column accessor.  The sections above that describe each type's characteristics include information about whether or not a logical type is nullable."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "surprised-today",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.ww['bools_nullable'].ww.nullable"
+   ]
   }
  ],
  "metadata": {
@@ -694,7 +741,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.2"
+   "version": "3.8.8"
   }
  },
  "nbformat": 4,

docs/source/release_notes.rst

@@ -7,6 +7,7 @@ Future Release
     * Enhancements
         * Add support for automatically inferring the ``URL`` and ``IPAddress`` logical types (:pr:`1122`, :pr:`1124`)
         * Add ``get_valid_mi_columns`` method to list columns that have valid logical types for mutual information calculation (:pr:`1129`)
+        * Add attribute to check if column has a nullable logical type (:pr:`1127`)
     * Fixes
     * Changes
         * Update ``get_invalid_schema_message`` to improve performance (:pr:`1132`)
@@ -15,7 +16,7 @@ Future Release
     * Testing Changes
 
     Thanks to the following people for contributing to this release:
-    :user:`ajaypallekonda`, :user:`thehomebrewnerd`
+    :user:`ajaypallekonda`, :user:`davesque`, :user:`jeff-hernandez`, :user:`thehomebrewnerd`
 
 v0.7.1 Aug 25, 2021
 ===================

woodwork/column_accessor.py

@@ -16,7 +16,7 @@
     TypingInfoMismatchWarning
 )
 from woodwork.indexers import _iLocIndexer, _locIndexer
-from woodwork.logical_types import LatLong, Ordinal
+from woodwork.logical_types import _NULLABLE_PHYSICAL_TYPES, LatLong, Ordinal
 from woodwork.statistics_utils import _get_box_plot_info_for_column
 from woodwork.table_schema import TableSchema
 from woodwork.utils import (
@@ -106,6 +106,13 @@ def _series(self):
     def schema(self):
         return copy.deepcopy(self._schema)
 
+    @property
+    @_check_column_schema
+    def nullable(self):
+        """Whether the column can contain null values."""
+        dtype = self._schema.logical_type._get_valid_dtype(type(self._series))
+        return dtype in _NULLABLE_PHYSICAL_TYPES
+
     @property
     @_check_column_schema
     def description(self):

woodwork/logical_types.py

@@ -503,3 +503,23 @@ class PostalCode(LogicalType):
     primary_dtype = 'category'
     backup_dtype = 'string'
     standard_tags = {'category'}
+
+
+_NULLABLE_PHYSICAL_TYPES = {
+    'boolean',
+    'category',
+    'datetime64[ns]',
+    'Int8',
+    'Int16',
+    'Int32',
+    'Int64',
+    'Float32',
+    'Float64',
+    'float16',
+    'float32',
+    'float64',
+    'float128',
+    'object',
+    'string',
+    'timedelta64[ns]',
+}

woodwork/tests/accessor/test_column_accessor.py

@@ -833,3 +833,33 @@ def test_series_methods_returning_frame_no_name(sample_series):
     reset_index_frame = sample_series.ww.reset_index(drop=False)
     assert _is_dataframe(reset_index_frame)
     assert reset_index_frame.ww.schema is not None
+
+
+EXPECTED_COLUMN_NULLABILITIES = {
+    'id': False,
+    'full_name': True,
+    'email': True,
+    'phone_number': True,
+    'age': True,
+    'signup_date': True,
+    'is_registered': True,
+    'double': True,
+    'double_with_nan': True,
+    'integer': False,
+    'nullable_integer': True,
+    'boolean': False,
+    'categorical': True,
+    'datetime_with_NaT': True,
+    'url': True,
+    'ip_address': True
+}
+
+
+def test_nullable_attribute(sample_df_pandas):
+    sample_df_pandas.ww.init()
+
+    for key in sample_df_pandas.ww.columns:
+        actual = sample_df_pandas.ww[key].ww.nullable
+        expected = EXPECTED_COLUMN_NULLABILITIES[key]
+
+        assert actual is expected