Remove NaNs from any resulting fit, regardless of the source to ensur… #3175

Job	Run time
Generate test matrix	10s
Test (ubuntu-20.04, 3.11)	2m 22s
Test (ubuntu-20.04, 3.10)	2m 42s
Test (windows-latest, 3.11)	5m 57s
Test (windows-latest, 3.10)	5m 56s
Test (macos-13, 3.11)	3m 10s
Test (macos-13, 3.10)	3m 21s
Test (macos-latest, 3.11)	2m 15s
Test (macos-latest, 3.10)	2m 54s
Test (ubuntu-latest, 3.11)	2m 23s
Test (ubuntu-latest, 3.10)	2m 38s
Installer (ubuntu-20.04, 3.11)	8m 16s
Installer (windows-latest, 3.11)	38m 53s
Installer (macos-13, 3.11)	26m 20s
Installer (macos-latest, 3.11)	18m 30s
Test installer (windows-latest, 3.11)	2m 9s
Test installer (ubuntu-20.04, 3.11)	1m 30s
	2h 9m 26s

Provide feedback