Roadmap for v1.0.0

[ConorMacBride]

I reviewed all the configuration options which are used in the code and found quite a few inconsistencies. I have listed proposed fixes for them, separated into breaking and non-breaking changes. We can add warning for the breaking changes now, and then change in v1.0.0.

Let's discuss these lists. Are there any items which should be added or removed? For the breaking changes, how long of a warning period should we have? For any of them, should we only start warning in v1.0.0 and then change in v1.1.0?

Review of configuration options

Option	kwarg	CLI	ini	default
Enable testing	N/A	--mpl	N/A	False
Enable baseline image generation, at specified directory path (relative to where pytest was run)	N/A	--mpl-generate-path	N/A	None
Enable baseline hash generation, at specified file path (relative to where pytest was run)	N/A	--mpl-generate-hash-library	N/A	None
Directory containing baseline images (relative to the test file)	baseline_dir	--mpl-baseline-path (relative to where pytest was run)		baseline/
Whether --mpl-baseline-path should also be relative to the test file	N/A	--mpl-baseline-relative	N/A	False
Filename of the baseline image	filename	N/A	N/A	test name
Whether to include the module name in the baseline image filename	?		mpl-use-full-test-name	False
File containing baseline hashes (relative to the test file)	hash_library	--mpl-hash-library		no hash comparison
RMS tolerance	tolerance			2
Whether to standardise metadata	deterministic			True (PNG: False)
kwargs to pass to savefig	savefig_kwargs	N/A	N/A	{}
Matplotlib style	style			classic
Whether to remove axis tick labels	remove_text			False
Matplotlib backend	backend			agg
Directory to write testing artifacts to (relative to where pytest was run)	N/A	--mpl-results-path	mpl-results-path	temp dir
Whether to save result images for passing tests	N/A	--mpl-results-always	mpl-results-always	False (True if generating a HTML summary)
Which test summaries to generate, if any	N/A	--mpl-generate-summary={html,json,basic-html}		None

Notes

1. N/A are options which we don't think should exist.
2. Empty boxes are options which we do think should exist.
3. ? are options which could exist but would be of limited use.
4. Due to how paths are computed by Python, the baseline directory and hash library paths can be absolute. In this case --mpl-baseline-relative and what directory the paths are interpreted to be relative to does not have any effect.
5. Path specified in kwargs should be relative to the test file, and paths specified in CLI and ini options should be relative to where pytest was run.

Non-breaking changes

• Document that backend can be specified. Migrate docs from README.rst to website #199
• Document the option to use the full test name. Migrate docs from README.rst to website #199
• Document that mpl-use-full-test-name ini overrides the filename kwarg. Migrate docs from README.rst to website #199
• Add mpl-baseline-path ini option (relative to where pytest was run). Add missing CLI and INI options for a consistent API #181
• Expand mpl-use-full-test-name ini option to CLI (and maybe kwarg). Add missing CLI and INI options for a consistent API #181
• Add mpl-hash-library ini option (relative to where pytest was run). Add missing CLI and INI options for a consistent API #181
• Add --mpl-default-tolerance CLI and mpl-default-tolerance ini options. Add missing CLI and INI options for a consistent API #181
• Add CLI and ini options for deterministic. Set deterministic to True for vector graphics and warn about change to True in future for PNG #197
• Add --mpl-default-style CLI and mpl-default-style ini options. Add missing CLI and INI options for a consistent API #181
• Add CLI and ini options for remove_text.
• Add CLI and ini options for backend. Add missing CLI and INI options for a consistent API #181
• Add generate-summary ini option for generating test summaries. Add missing CLI and INI options for a consistent API #181

Breaking changes

• --mpl-hash-library should be relative to where pytest was run (to match --mpl-baseline-path).
• Local hash_library kwarg should take precedence over global --mpl-hash-library CLI option. decorator hash_library kwarg option precedence  #154
• Deprecate the use of multiple hash libraries in the same pytest run (not necessary because files are small and include the full module path anyway). decorator hash_library kwarg option precedence  #154
• Deprecate the use of multiple testing modes in the same pytest run (e.g. warn if one test only has hash comparison while the others only have image comparison)
• Use Matplotlib's default style instead of 'classic'.
• Change the default RMS tolerance to 0.
• Enable deterministic PNG files by default.
• Deprecate remove_text kwarg and replace with better named remove_ticks_and_titles
• (???) Add a remove_text kwarg that actually removes all text.
• (???) Rename baseline_dir/--mpl-baseline-path to have consistent name.

[bjlittle]

@ConorMacBride This is brilliant! Thanks for putting this together.

I'm spinning back up on pytest-mpl. I've cleared the decks and pretty much secured some free time to dedicate to finally getting #150 across the line.

In the process of getting my head back in the space I'm going to cross-reference your table above with the codebase and also any observations that I previously made to make sure that we've captured everything up for discussion in the v1.0.0 roadmap 👍

[bjlittle]

@ConorMacBride The only additionally changes that I've noticed not captured above are:

• The baseline_dir kwarg takes precedence over the --mpl-baseline-path. The order elsewhere is CLI -> INI -> KWARG. Although, this may have changed in Add missing CLI and INI options for a consistent API #181, I've not checked yet. Was this intentional by design?
• Perceptual hash #150 will introduce a hashing tolerance, so we need to be able to identify the existing tolerance as the RMS tolerance e.g., --mpl-rms-tolerance (CLI), mpl-rms-tolerance (INI) and rms_tolerance (kwargs) or just stick with the changes from Add missing CLI and INI options for a consistent API #181 i.e., --mpl-default-tolerance (CLI), mpl-default-tolerance (INI) and tolerance ?