Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test coverage reporting #169

Open
wants to merge 15 commits into
base: master
Choose a base branch
from

Conversation

Wuerfel21
Copy link
Contributor

Having come across the odd bit of code that seems to be old bitrotted gunk, I thought it'd be neat to hook some line-level coverage reporting to the test suite. Annoyingly, it only seems to work on GCC 8 under Linux due to weirdness with where it wants to place its files, but for running it on CI that's fine, I guess. Speaking of, I've hooked it to codecov for generation of fancy statistics, which looks a bit like this: https://codecov.io/gh/Wuerfel21/spin2cpp/src/test-coverage/spinc.c If you merge this PR, I think you need to log into codecov yourself so it can access the repository.

@codecov-commenter
Copy link

Codecov Report

❗ No coverage uploaded for pull request base (master@1da37fc). Click here to learn what that means.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##             master     #169   +/-   ##
=========================================
  Coverage          ?   53.91%           
=========================================
  Files             ?       57           
  Lines             ?    33898           
  Branches          ?        0           
=========================================
  Hits              ?    18276           
  Misses            ?    15622           
  Partials          ?        0           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1da37fc...64696fa. Read the comment docs.

@totalspectrum
Copy link
Owner

Hmmm... this sounds interesting, but I'd like to dig into a bit before merging.

It is probably time to add some more tests. I've got enough of the BASIC and C runtime working with bytecode now that we can do floating point and strings, so a pretty good selection of programs should run.

@Wuerfel21
Copy link
Contributor Author

It is probably time to add some more tests.

Yeah, that's the idea. Knowing what isn't covered yet is of good use in that, I think.

Though there's the difficulty that bytecode output doesn't really have an intermediate form that can be diffed. Listings don't really work that well for that since any size change causes the rest of the file to change. I think emitting a listing without the address column would work okay-ish though.

Relatedly, there probably need to be a couple more BASIC tests in general, basiclang.c has the lowest coverage of all the frontends at ~24%

@totalspectrum
Copy link
Owner

A lot of the BASIC testing is in the runtime tests. I should start adding more .bas files to the PASM and C++ backend tests too though, it would give us a bit more coverage. I'll try to make future tests in BASIC; I've tended to use Spin for the really simple cases and C for the fancy type specific things.

@totalspectrum
Copy link
Owner

On a related note, I wonder if Gear or spinsim or something like that could execute the runtime tests on a PC, allowing us to do coverage testing even for the runtime tests?

@Wuerfel21
Copy link
Contributor Author

Yeah, spinsim could probably be hooked up to run tests. Not sure about the speed though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants