Using pytest-pyvista to improve tests of geovista

[tkoyama010]

📰 Custom Issue

pytest-pyvista is a plugin to test PyVista plot outputs.
We can add tests of geovista's plotting using this.

[bjlittle]

@tkoyama010 Cool.

Ultimately I'm aiming to use matplotlib/pytest-mpl#150.

This may complement the testing framework you're suggesting?

[tkoyama010]

Nice. It might be a good idea to encourage PyVista to move from pytest-pyvista to pytest-mpl based on that.

[bjlittle]

@tkoyama010 We've been using the perceptual image hash approach on SciTools/iris for a few years now, so I'm pretty satisfied that it gives the stability and control to detect image differences and detect levels of image similarity.

I'm keen to get my PR merged on pytest-mpl (it's well overdue) and there is also talk of pytest-mpl being capable to test images that are not generated by matplotlib. I'm definitely interested in helping make that happen on pytest-mpl as it will allow me to use that testing infrastructure with geovista.

I'll take a peek at pytest-pyvista to see how it works too, out of interest 👍

[tkoyama010]

there is also talk of pytest-mpl being capable to test images that are not generated by matplotlib.

Great. Could you point me to where the talk thread is? I am really interested in it.

[bjlittle]

See matplotlib/pytest-mpl#201

But I've talked to other pytest-mpl core devs about this capability. I think it would be pretty darn awesome, and open up the use of pytest-mpl elsewhere within the scientific python ecosystem.

[tkoyama010]

I would expect pytest-pyvista to be more profitable than pytest-mpl for GeoVista, but I don't think I can explain it because of my less knowledge of pytest-pyvista.
@MatthewFlamm I would like to hear your opinion if you like.

[MatthewFlamm]

I'm not super familiar to pytest-mpl but these packages look very similar. One problem with using pytest-mpl even if it becomes more generic is that it will require a lot of boilerplate and tinkering to get testing to work with pyvista.

pytest-pyvista hooks natively into PyVista, so you can do:

def test_my_plot(verify_image_cache):
    pv.Sphere.plot()

and it will do the image comparison.

In an ideal world in the scientific software field we would have a meta level comparing tool that is agnostic for how the images are generated. Then each tool provides a wrapper for generating those images.

But practically and selfishly, I think it benefits the pyvista community more if we make pytest-pyvista more robust. If we can offload parts of that pipeline to pytest-mpl, I think that would be great. It gets slightly closer to my ideal state above. pytest-pyvista, in my opinion, has a more flexible interface. It uses a fixture, so it allows you to do more custom things like for example doing multiple image comparisons in one test, turning off and then back on image capturing inside a test, etc....

[MatthewFlamm]

After glancing at your PR in pytest-mpl, I get the sense that it adds the ability to do different kinds of comparison. But it doesn't much touch the meta-level problem I highlighted above? I need to learn more though so please educate me.

I think there is a strong opportunity to replace the homegrown image comparison in pytest-pyvista with a better supported comparison tool. This will not be a tiny effort in PyVista as the are a lot of tests in pyvista that generate images with a lot of variance based on architecture, etc.

[MatthewFlamm]

Can you point to the discussion on making pytest-mpl hook better into other image generation tools?

Edit: Sorry for the noise, Somehow I missed that it is linked above.

[tkoyama010]

Thank you for your valuable input!
I have gained a lot of insight that I was not aware of and will try to create a Pull Request for GeoVista for a test using pytest-pyvista.

After glancing at your PR in pytest-mpl, I get the sense that it adds the ability to do different kinds of comparison. But it doesn't much touch the meta-level problem I highlighted above? I need to learn more though so please educate me.

@bjlittle We would appreciate it if you have any insight on this.

[bjlittle]

@MatthewFlamm Seems like you found it and chipped into the conversation on matplotlib/pytest-mpl#201 👍