Codecov

In software development, there is the concept of "code coverage". When your automated tests run, a coverage tool instruments the code, wrapping it with instructions that determine, for each line of your source code, whether the line was executed during the test or not. Many have written about the pros and cons of low and high code coverage. It turns out it's one of those areas where a "middle ground" is critically important.

There are hundreds of tools across every programming language that detect and report code coverage. Codecov is not one of them. Instead, it provides a GUI (Graphical User Interface) for inspecting, analyzing, and reporting on code coverage reports. Once your tests run, you upload the coverage report to Codecov, and their automated system provides a web app where you can visually see where you are covered or lacking coverage as you explore your source code tree.

Even more useful for most projects, the process of uploading and inspecting code coverage reports can itself be completely automated using GitHub Actions. What this means is that before you merge a PR (Pull Request), Codecov can say the equivalent of "Hey, don't merge this, you didn't test the new code you added!". Implementing and responding to this feedback is part of good "code hygeine".

This year, Codecov released v4 of it's GitHub integration. I believe the purpose was to unify its CLI (Command Line Interface) tool and the GitHub code, so they weren't running different codebases. The most severe change was that it was no longer possible for Codecov to use the "environment" of the GitHub repo to determine the proper project or credentials: you now had to explicitly set a token for it to work. For thousands of developers, their automated pipelines stopped working, and it was often difficult to tell what was wrong and how to fix it. Well, it wasn't particularly difficult per se, but required a lot of annoying steps and hard to find documentation.

All of this is to just document for future posterity the head-bang-on-desk moment I had when trying to upgrade to v4 in my own WP1 repo today. The Codecov automation kept reporting that I was missing the token. I, on the other hand, was thouroughly convinced that I had the token, but Codecov just wouldn't agree with me. I had read the documentation over and over, and double checked that I had done all of the steps but it still wasn't working.

The problem was that I assumed the hard part was the first part, adding the CODECOV_TOKEN to the GitHub repo settings. I was convinced that that's where the problem was. In actuality, what I had missed was that the syntax for calling v4 was slightly, but every so similarly lookingly (?), different from v3.

I kept seeing the line where you set env: but reading it as "I already have secrets.CODECOV_TOKEN in there!". Facepalm.

In all honesty though, the entire Codecov v3 to v4 upgrade has kind of been a nightmare, across multiple months and multiple repos. Thank god I'm not paying them any money!

Update: I'm still getting these freaking errors! Every now and then my CI will flake out like this:

Update the Second: I am still getting these intermittent flaky errors. I've decided to demote Codecov, and set my GitHub actions CI so that if all the tests pass and only the Codecov upload fails, the CI still passes. Sigh.