-
-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add test coverage reporting #169
base: master
Are you sure you want to change the base?
Conversation
Codecov Report
@@ Coverage Diff @@
## master #169 +/- ##
=========================================
Coverage ? 53.91%
=========================================
Files ? 57
Lines ? 33898
Branches ? 0
=========================================
Hits ? 18276
Misses ? 15622
Partials ? 0 Continue to review full report at Codecov.
|
Hmmm... this sounds interesting, but I'd like to dig into a bit before merging. It is probably time to add some more tests. I've got enough of the BASIC and C runtime working with bytecode now that we can do floating point and strings, so a pretty good selection of programs should run. |
Yeah, that's the idea. Knowing what isn't covered yet is of good use in that, I think. Though there's the difficulty that bytecode output doesn't really have an intermediate form that can be diffed. Listings don't really work that well for that since any size change causes the rest of the file to change. I think emitting a listing without the address column would work okay-ish though. Relatedly, there probably need to be a couple more BASIC tests in general, |
A lot of the BASIC testing is in the runtime tests. I should start adding more .bas files to the PASM and C++ backend tests too though, it would give us a bit more coverage. I'll try to make future tests in BASIC; I've tended to use Spin for the really simple cases and C for the fancy type specific things. |
On a related note, I wonder if Gear or spinsim or something like that could execute the runtime tests on a PC, allowing us to do coverage testing even for the runtime tests? |
Yeah, spinsim could probably be hooked up to run tests. Not sure about the speed though. |
Having come across the odd bit of code that seems to be old bitrotted gunk, I thought it'd be neat to hook some line-level coverage reporting to the test suite. Annoyingly, it only seems to work on GCC 8 under Linux due to weirdness with where it wants to place its files, but for running it on CI that's fine, I guess. Speaking of, I've hooked it to codecov for generation of fancy statistics, which looks a bit like this: https://codecov.io/gh/Wuerfel21/spin2cpp/src/test-coverage/spinc.c If you merge this PR, I think you need to log into codecov yourself so it can access the repository.