Integration tests not running as of VS2017

[codecadwallader]

Environment

• Visual Studio version: 2017 Community
• CodeMaid version: 10.4
• Code language: *

Description

The integration tests stopped working as we upgraded to building from a VS2017 framework. There's some helpful information on #453 that repairs part of the issues, but it looks like some other changes (e.g. relative paths no longer accepted) will also need to be resolved.

Steps to recreate

1. Run the integration tests as explained at http://www.codemaid.net/contribute/

Current behavior

The integration tests are unable to execute.

Expected behavior

The integration tests should be able to execute.

[w5l]

@mlangkabel I'm looking into fixing the integration tests for CodeMaid in VS2017, but so far running into nothing but problems. I saw that you disabled your integration tests as well: CoatiSoftware/vs-sourcetrail@29aa80c, but maybe you any progress or ideas to share?

I got the regular unit tests running like a charm again, after cleanup and updating to MSTest V2. That however breaks compatibility with .testsettings files, and the [HostType(...)] attribute has disappeared. That leaves me at a dead for integration testing for now.

[mlangkabel]

At Sourcetrail we managed to get those tests running in an AppVeyor environment after they ceased to work for CodeMaid, but running the experimental instance of Visual Studio as a host for those tests somehow failed.
We solved that by building the extension in the AppVeyor environment and then installing it to the Visual Studio that is used by the AppVeyor environment before running the tests like this:

before_test:
  - cmd: START /WAIT "" "%PROGRAMFILES(X86)%\Microsoft Visual Studio\2017\Community\Common7\IDE\VSIXInstaller.exe" /q "C:\projects\vs-sourcetrail\SourcetrailExtension\bin\Release\vs-sourcetrail.vsix"

test_script:
  - ps: VSTest.Console.exe C:\projects\vs-sourcetrail\SourcetrailExtensionTests\bin\Release\SourcetrailExtension.Tests.dll

But all of a sudden (I think this was caused by an AppVeyor update of the used VS version) the integration tests stopped working for us. And after updating VS on my office machine the integration tests failed there, too.

I think that without the [HostType(...)] attribute it is not possible to run a VS instance as a host for any integration tests.

We also tried to get the BDDfy Integration tests running instead (https://github.com/oz-code/BDDfyVSIntegrationTestSample), but that didn't work for us either.

Regardless of our failures, I hope this information helps!

[w5l]

Thanks for taking the time to reply, it at least saves me going into some dead alleys!

[codecadwallader]

Here's some related articles/links.. but the picture I'm kind of getting is that it's not really being supported/maintained anymore. I'll keep searching, but wanted to jot down these notes:

microsoft/VisualStudio-TestHost#36
microsoft/VisualStudio-TestHost#37
http://www.visualstudioextensibility.com/2018/01/22/the-visualstudio-testhost-project-to-execute-interactive-ui-tests-in-visual-studio/

[w5l]

Could we mock the Document and TextPoint interfaces and unit test against that instead? It sort of defeats the point of real "integration tests" but it's better than nothing. When we finally manage to port to Roslyn I see no more point in integration tests anyway, we could test directly on the code model.

[codecadwallader]

We may be able to, but there might be a lot of work to mock out all the different methods for cursor locations, selecting and manipulating text, etc.

In general I agree about unit tests being preferable to integration tests. I think of an extension on another tool as being an exception case though. One great thing it used to be able to help us with is when a new Visual Studio update/version comes out, we could run the test and see if VS itself has changed in a way that we need to adjust to support. There's definitely room for both!

[codecadwallader]

FWIW I attended Visual Studio Extensibility Day and improving documentation and tooling around integration testing was the # 1 request across extension authors (it was definitely my vote!) That doesn't guarantee it will get improved, but I'm hopeful that it will.

https://devblogs.microsoft.com/visualstudio/visual-studio-extensibility-day-2019-was-a-hit/