ci(mypy): add mypy check and adjust code for types

[d33bs]

Description

This PR adds a check through GitHub Actions for mypy in order to make sure Python types align with certain standards within pycytominer. I've also made some adjustments to the code in order to pass mypy checks.

Closes #437

What is the nature of your change?

• Bug fix (fixes an issue).
• Enhancement (adds functionality).
• Breaking change (fix or feature that would cause existing functionality to not work as expected).
• This change requires a documentation update.

Checklist

Please ensure that all boxes are checked before indicating that a pull request is ready for review.

• I have read the CONTRIBUTING.md guidelines.
• My code follows the style guidelines of this project.
• I have performed a self-review of my own code.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.
• New and existing unit tests pass locally with my changes.
• I have added tests that prove my fix is effective or that my feature works.
• I have deleted all non-relevant text in this pull request template.

---

📚 Documentation preview 📚: https://pycytominer--439.org.readthedocs.build/en/439/

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 91.66667% with 2 lines in your changes missing coverage. Please review.

Project coverage is 94.73%. Comparing base (c782728) to head (b057fb1).
Report is 36 commits behind head on main.

Files with missing lines	Patch %	Lines
pycytominer/aggregate.py	75.00%	1 Missing ⚠️
pycytominer/cyto_utils/collate.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #439      +/-   ##
==========================================
+ Coverage   94.72%   94.73%   +0.01%     
==========================================
  Files          56       57       +1     
  Lines        3147     3155       +8     
==========================================
+ Hits         2981     2989       +8     
  Misses        166      166              

Flag	Coverage Δ
unittests	94.73% <91.66%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[gwaybio]

LGTM, but I trust Ken more to give final approval

[kenibrewer]

This is a surprisingly low number of changes for enabling mypy in a repo. This might be because of the ignore_missing_imports and allow_redefinitions config options. What was the change burden like for turning those options off?

[d33bs]

Thanks @gwaybio and @kenibrewer for the reviews.

This is a surprisingly low number of changes for enabling mypy in a repo.

I'm not sure, but I wonder if this has to do with the low amount of type hinting alongside compatibility with inferred types. Could be useful to make a more focused effort towards adding type hints throughout.

What was the change burden like for turning those options off?

When I looked here I noticed we no longer needed the allow_redefinitions config option so I've turned this off. ignore_missing_imports results in the following errors / warnings. Generally I turned this on as I felt there might be many dependencies we would have to add and I wasn't certain of the benefit. Maybe this could be an iterative process where we move towards that in another change? I can also see why we might do it here.

pycytominer/__config__.py:5: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/output.py:7: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/features.py:6: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/cell_locations.py:6: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/cell_locations.py:10: error: Skipping analyzing "boto3": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/cell_locations.py:11: error: Skipping analyzing "botocore": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/cell_locations.py:13: error: Skipping analyzing "sqlalchemy": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/cell_locations_cmd.py:2: error: Skipping analyzing "fire": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/operations/transform.py:9: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/operations/transform.py:10: error: Skipping analyzing "scipy.stats": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/operations/transform.py:11: error: Skipping analyzing "sklearn.base": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/operations/transform.py:12: error: Skipping analyzing "sklearn.preprocessing": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/util.py:8: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/load.py:5: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/modz.py:2: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/collate.py:75: error: Skipping analyzing "cytominer_database.ingest": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/collate.py:75: error: Skipping analyzing "cytominer_database": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/collate.py:76: error: Skipping analyzing "cytominer_database.munge": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/write_gct.py:12: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/DeepProfiler_processing.py:7: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/DeepProfiler_processing.py:7: note: Hint: "python3 -m pip install pandas-stubs"
pycytominer/cyto_utils/DeepProfiler_processing.py:7: note: (or run "mypy --install-types" to install all missing stub packages)
pycytominer/cyto_utils/DeepProfiler_processing.py:7: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
pycytominer/aggregate.py:8: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/normalize.py:5: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/normalize.py:6: error: Skipping analyzing "sklearn.preprocessing": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/annotate.py:7: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/cells.py:4: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/cells.py:21: error: Skipping analyzing "sqlalchemy": module is installed, but missing library stubs or py.typed marker  [import-untyped]
Found 26 errors in 16 files (checked 31 source files)

[d33bs]

This is now ready for another look when there's a moment @gwaybio and @kenibrewer . I've added more changes to include type stubs where reasonable. Many packages used by pycytominer do not have their own stubs provided within the package or through external stub packages - these are now specifically ignored via an override setting within the pyproject.toml. I've added additional type stub dependencies to the dev group, as these would now be required to successfully develop and test during development.

Generally I found that it'd be healthy to explore alternative type checkers in the future, especially pyright, where Microsoft has done extensive work to provide custom stubs that likely integrate via python-type-stubs, or pylyzer which would grant speed and possibly other benefits. I didn't feel I could rationalize importing many of the relevant stubs into this project locally - these could be fairly burdensome to manage in addition to their likelihood of changing.

[gwaybio]

LGTM - let's wait on merging this until we merge #448 (after we merge 448, I am guessing you will need to resolve conflicts and rerun)

[kenibrewer]

LGTM too! Thanks for looking into the extra imports

[gwaybio]

Ready to merge @d33bs !

[d33bs]

Thanks @gwaybio and @kenibrewer for your reviews! I double checked from main and it looks like mypy tests still pass. Merging this in.