Hypothesis Tests and CI

Good CI pipelines are fast and deterministic. Testing with Hypothesis is neither. To have both, I restrict Hypothesis tests to a scheduled pipeline.

CI pipelines should be fast and reliable.
When I push a commit to a branch to be reviewed by someone else, I want an immediate sanity check: Is this okay to be viewed by another person. I don’t want to wait and make a pot of tea. I also don’t want to wonder if simply rerunning a failed pipeline might give me a green check-mark.

Hypothesis takes over creating test examples.
With conventional testing approaches, I try to come up with examples of correct input and output for a given piece of code. With Hypothesis, I think about what is generally true about the input and output of a piece of code and Hypothesis takes care of coming up with examples. And it comes up with a lot of examples and tries really really hard to break my code. I like that, I write less but get more tests.

However, ….
Hypothesis trying really hard, means it takes quite a long time going through random examples. Random and long are the two things I want my CI to not be.

The easy solution is to keep Hypothesis out of my regular CI and schedule it to run once a month for 20 minutes.

links: gitlab-doc on scheduled pipelines, pipeline-example, schedule-example