Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fine Code Coverage shows coverage of 0% after enable a third pard library #464

Open
marcelbeekercgm opened this issue Nov 8, 2024 · 30 comments
Open

Fine Code Coverage shows coverage of 0% after enable a third pard library #464

marcelbeekercgm opened this issue Nov 8, 2024 · 30 comments

Comments

@marcelbeekercgm
Copy link

Installed product versions

  • Visual Studio: example 2022 Professional
  • This extension: example 1.1.223

Description

After invoking a method of a third party library there is 0% code coverage using coverlet.
When I generate a code coverage report using dotnet test --collect:"XPlat Code Coverage" and the report generator there is a code coverage report. So I guess the problem is in de fine code coverage extension.

Steps to recreate

  1. Enable third party method
  2. Run test

Current behavior

Coverage of 0% in all methods

Expected behavior

Coverage of 50%

Side Notes
@tonyhallett
Copy link
Collaborator

Perhaps you could provide some information so we can determine the nature of the problem.

I am going to guess that you have not set the option RunMsCodeCoverage to Yes. If you do not do this then FCC copies dlls changing paths which can result in tests that dependend on paths to fail and the result will be 0% coverage.

@marcelbeekercgm
Copy link
Author

Hi Tony,

Thanks for your quick response. The RunMsCodeCoverage is set to False. What exact information do you want?

@tonyhallett
Copy link
Collaborator

Switch to RunMsCodeCoverage Yes then if it is still not providing coverage then provide information as per the side notes in the issue template.
Ideally you provide a repo that exhibits the behaviour encountered so I can debug FCC.
There are also logs in the FCC Output Window Pane that may give information.

@marcelbeekercgm
Copy link
Author

Setting the switch on RunMsCodeCoverage to Yes is solving the problem. However I can only generate the report only once. In the coverage log is seems to stuk on Ms code coverage as showd in the attachted screenshot.
After restarting Visual Studio I be able to generate the report once again.

image

FCC output shows the following lines:

Fine Code Coverage 08/11/2024 12:29:24: Ms code coverage
Fine Code Coverage 08/11/2024 12:29:32: ReportGenerator Run Arguments [reporttype:Cobertura]
"-targetdir:\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output"
"-reports:\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output\690bba0c-7478-43b3-a40d-5d7c2f4aa0a1\marcel.beeker_N-4WJCLL3_2024-11-08.12_29_27.cobertura.xml"
"-reporttypes:Cobertura"
Fine Code Coverage 08/11/2024 12:29:32: ReportGenerator Run [reporttype:Cobertura]
2024-11-08T12:29:32: Arguments
2024-11-08T12:29:32: -targetdir:\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output
2024-11-08T12:29:32: -reports:\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output\690bba0c-7478-43b3-a40d-5d7c2f4aa0a1\marcel.beeker_N-4WJCLL3_2024-11-08.12_29_27.cobertura.xml
2024-11-08T12:29:32: -reporttypes:Cobertura
2024-11-08T12:29:32: Writing report file '\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output\Cobertura.xml'
2024-11-08T12:29:32: Report generation took 0,1 seconds
Fine Code Coverage 08/11/2024 12:29:32: ReportGenerator Run Arguments [reporttype:HtmlInline_AzurePipelines]
"-targetdir:\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output"
"-reports:\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output\Cobertura.xml"
"-plugins:c:\users\marcel.beeker\appdata\local\microsoft\visualstudio\17.0_916629b8\extensions\qewmklx5.aqc\ReportGeneratorPlugins.dll"
"-reporttypes:FccLight"
"riskHotspotsAnalysisThresholds:metricThresholdForCyclomaticComplexity=30"
"riskHotspotsAnalysisThresholds:metricThresholdForCrapScore=15"
"riskHotspotsAnalysisThresholds:metricThresholdForNPathComplexity=200"
Fine Code Coverage 08/11/2024 12:29:32: ReportGenerator Run [reporttype:HtmlInline_AzurePipelines]
2024-11-08T12:29:32: Arguments
2024-11-08T12:29:32: -targetdir:\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output
2024-11-08T12:29:32: -reports:\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output\Cobertura.xml
2024-11-08T12:29:32: -plugins:c:\users\marcel.beeker\appdata\local\microsoft\visualstudio\17.0_916629b8\extensions\qewmklx5.aqc\ReportGeneratorPlugins.dll
2024-11-08T12:29:32: -reporttypes:FccLight
2024-11-08T12:29:32: riskHotspotsAnalysisThresholds:metricThresholdForCyclomaticComplexity=30
2024-11-08T12:29:32: riskHotspotsAnalysisThresholds:metricThresholdForCrapScore=15
2024-11-08T12:29:32: riskHotspotsAnalysisThresholds:metricThresholdForNPathComplexity=200
2024-11-08T12:29:32: Writing report file '\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output\index.html'
2024-11-08T12:29:32: Report generation took 0,1 seconds
Fine Code Coverage 08/11/2024 12:29:32: Processing cobertura
Fine Code Coverage 08/11/2024 12:29:32: Processing report
Fine Code Coverage 08/11/2024 12:29:33: ================================== DONE ==================================

@tonyhallett
Copy link
Collaborator

Is the FCC output referring to the same run as the last in the image that is stuck on Generating html report ?
I ask as Processing cobertura and Processing report occurs after Generating cobertura report / duration and Generating html report /duration.

Also DONE occurs after the report has been generated successfully and the new report is shown.

Aside from what you see in the screenshot what makes you think that you can only generate the report only once ?
Are you sure that the coverage has changed and that you should be seeing a different report ?

@marcelbeekercgm
Copy link
Author

marcelbeekercgm commented Nov 8, 2024 •
edited
Loading

Yes it is the same run.
I did the same test once again. The generation of the report is done as you can see below. But no report is displayed.

Fine Code Coverage 08/11/2024 16:23:18: Ms code coverage
Fine Code Coverage 08/11/2024 16:23:26: ReportGenerator Run Arguments [reporttype:Cobertura]
"-targetdir:C:\Development\Repos\z\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output"
"-reports:C:\Development\Repos\z\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output\781c3be4-9036-4261-ae3e-4ca5106e8a1a\marcel.beeker_N-4WJCLL3_2024-11-08.16_23_22.cobertura.xml"
"-reporttypes:Cobertura"
Fine Code Coverage 08/11/2024 16:23:27: ReportGenerator Run [reporttype:Cobertura]
2024-11-08T16:23:26: Arguments
2024-11-08T16:23:26: -targetdir:C:\Development\Repos\z\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output
2024-11-08T16:23:26: -reports:C:\Development\Repos\z\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output\781c3be4-9036-4261-ae3e-4ca5106e8a1a\marcel.beeker_N-4WJCLL3_2024-11-08.16_23_22.cobertura.xml
2024-11-08T16:23:26: -reporttypes:Cobertura
2024-11-08T16:23:27: Writing report file 'C:\Development\Repos\z\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output\Cobertura.xml'
2024-11-08T16:23:27: Report generation took 0,1 seconds
Fine Code Coverage 08/11/2024 16:23:27: ReportGenerator Run Arguments [reporttype:HtmlInline_AzurePipelines]
"-targetdir:C:\Development\Repos\z\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output"
"-reports:C:\Development\Repos\z\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output\Cobertura.xml"
"-plugins:c:\users\marcel.beeker\appdata\local\microsoft\visualstudio\17.0_916629b8\extensions\qewmklx5.aqc\ReportGeneratorPlugins.dll"
"-reporttypes:FccLight"
"riskHotspotsAnalysisThresholds:metricThresholdForCyclomaticComplexity=30"
"riskHotspotsAnalysisThresholds:metricThresholdForCrapScore=15"
"riskHotspotsAnalysisThresholds:metricThresholdForNPathComplexity=200"
Fine Code Coverage 08/11/2024 16:23:27: ReportGenerator Run [reporttype:HtmlInline_AzurePipelines]
2024-11-08T16:23:27: Arguments
2024-11-08T16:23:27: -targetdir:C:\Development\Repos\z\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output
2024-11-08T16:23:27: -reports:C:\Development\Repos\z\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output\Cobertura.xml
2024-11-08T16:23:27: -plugins:c:\users\marcel.beeker\appdata\local\microsoft\visualstudio\17.0_916629b8\extensions\qewmklx5.aqc\ReportGeneratorPlugins.dll
2024-11-08T16:23:27: -reporttypes:FccLight
2024-11-08T16:23:27: riskHotspotsAnalysisThresholds:metricThresholdForCyclomaticComplexity=30
2024-11-08T16:23:27: riskHotspotsAnalysisThresholds:metricThresholdForCrapScore=15
2024-11-08T16:23:27: riskHotspotsAnalysisThresholds:metricThresholdForNPathComplexity=200
2024-11-08T16:23:27: Writing report file 'C:\Development\Repos\z\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output\index.html'
2024-11-08T16:23:27: Report generation took 0,1 seconds
Fine Code Coverage 08/11/2024 16:23:27: Processing cobertura
Fine Code Coverage 08/11/2024 16:23:27: Processing report
Fine Code Coverage 08/11/2024 16:23:27: ================================== DONE ==================================

image

After Visual Studio is restarted, Fine Code Coverage start to generate the code coverage report correctly

Screenshot 2024-11-08 162923

@tonyhallett
Copy link
Collaborator

Can you show the cobertura
image

@marcelbeekercgm
Copy link
Author

It's better to sent a picture.
image

@tonyhallett
Copy link
Collaborator

There is no report as there is no coverage.

If you had a coverage report before using MsCodeCoverage then you have changed something. What did you change ?

There is no coverage as nothing has been included.

Set the option IncludeReferencedProjects to true.

If this does not work then you need to provide a repo that I can look at.

@marcelbeekercgm
Copy link
Author

I didn't change anything in my code. The only thing I changed was, after the changing the setting RunMsCodeCoverage to Yes and Visual Studio restarted. After the restart the Code Coverage report is generated. The second time, if I execute the unit tests in Visual Studio, the previously generated report disappeared but no report is generated after that.

@tonyhallett
Copy link
Collaborator

Have you set IncludeReferencedProjects and re-run your tests ?

@marcelbeekercgm
Copy link
Author

marcelbeekercgm commented Nov 11, 2024 •
edited
Loading

I'm not sure if I understand you correctly. Do you mean this setting?
Screenshot 2024-11-11 095522

@tonyhallett
Copy link
Collaborator

Yes. You do have your SUT in a separate project ?

@marcelbeekercgm
Copy link
Author

Yes my SUT is a seperate project from the tests.

@tonyhallett
Copy link
Collaborator

If you are getting no coverage with those settings then you need to show your solution or create a repro.

@marcelbeekercgm
Copy link
Author

I will do. But I can't create a repro with the external library for you because it is confidential. But the issue about the RunMsCodeCoverage to Yes which I mentioned today I can create a repro for you. That's no problem. I will contact again when the repro is available. Thanks for you time for now.

@tonyhallett
Copy link
Collaborator

Do you get the same problem with a different solution ?

@marcelbeekercgm
Copy link
Author

Yes, it's very simple to reproduce the issue after RunMsCodeCoverage is set to true?

@tonyhallett
Copy link
Collaborator

If this behaviour is present for all solutions then it suggests that visual studio has changed something we rely on, which could be a massive issue.
Can you create the simplest of solutions, a test project with a single test of a SUT with a single class, single method. Do you get the same behaviour, cobertura with results first run and no results the second time.

What programming language are you using ?

@marcelbeekercgm
Copy link
Author

Sorry for my late reaction. I have created a public repository where you can reproduce the issue.
https://github.com/marcelbeekercgm/SampleCoverageProject.git

After set RunMsCodeCoverage to the value true, no code coverage is generated after the test running more than once.

@tonyhallett
Copy link
Collaborator

Thanks, as expected no issues with my version of Visual Studio
Microsoft Visual Studio Community 2022 (3) (64-bit) - Current
Version 17.10.4

What version of Visual Studio Professional do you have ?

@tonyhallett
Copy link
Collaborator

Updating my visual studio, will check tomorrow.

@tonyhallett
Copy link
Collaborator

Works with
Microsoft Visual Studio Community 2022 (3) (64-bit) - Current
Version 17.11.5

I do not have Visual Studio Profesional so the only way to proceed is for you to fork the extension and debug through yourself.

Are you willing to do that if I provide instructions ?

@marcelbeekercgm
Copy link
Author

Sorry, at this moment i don't have time to debug the extension for you. However this problem can also be reproduced with the Community Edition of Visual Studio

@tonyhallett
Copy link
Collaborator

However this problem can also be reproduced with the Community Edition of Visual Studio

With https://github.com/marcelbeekercgm/SampleCoverageProject.git RunMsCodeCoverage Yes and IncludeReferencedProjects cobertura present after the first run and not the second ?

Please ensure that if you have muliple installs of Visual Studio they have the same version of FCC installed.

After running the first test open ...\SampleCoverageProject-master\SampleCoverageProject-master\SampleCoverageProject.Test\bin\Debug\net8.0\fine-code-coverage\coverage-tool-output it should look like
image

One the second run the directory should be emptied and files generated again.
What do you see after the second run ?

@marcelbeekercgm
Copy link
Author

I see the same results as the results displayed in your screenshot. Hower when I opened the index.html page I see this:
image

Only the results of my testproject not the sut project.

@tonyhallett
Copy link
Collaborator

I am now getting the behaviour you see. I can see why, FCC generates .runsettings.

When it works initially

            <ModulePaths>
              <Exclude></Exclude>
              <Include>
                <ModulePath>.*\\SampleCoverageProject.dll$</ModulePath>
                <ModulePath>C:\\Users\\tonyh\\Downloads\\SampleCoverageProject-master\\SampleCoverageProject-master\\SampleCoverageProject.Test\\bin\\Debug\\net8.0\\SampleCoverageProject.Test.dll</ModulePath>
              </Include>

When it does not

            <ModulePaths>
              <Exclude></Exclude>
              <Include>
                <ModulePath>.*\\.dll$</ModulePath>
                <ModulePath>C:\\Users\\tonyh\\Downloads\\SampleCoverageProject-master\\SampleCoverageProject-master\\SampleCoverageProject.Test\\bin\\Debug\\net8.0\\SampleCoverageProject.Test.dll</ModulePath>
              </Include>

Will let you know when I have determined the cause.

@tonyhallett
Copy link
Collaborator

In the mean time for your original test of the third party library you can include it with a finecodecoverage-settings.xml

@tonyhallett
Copy link
Collaborator

The problem was that Visual Studio was intermittently ( for me, once in many test runs ) returning null for https://learn.microsoft.com/en-us/dotnet/api/vslangproj.reference.path?view=visualstudiosdk-2022

This has been fixed but the fix is only available in releases, vsix, untill @FortuneN resolves the Github Actions issue

@marcelbeekercgm
Copy link
Author

marcelbeekercgm commented Nov 19, 2024 •
edited
Loading

Sorry for my late answer. Can you give an indication whether a new version of the extension becomes available?

Thanks in advance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tonyhallett @marcelbeekercgm