Story time:

There was a long data pipeline that produced wrong results. The wrong results were subtle but reproducible. Each run was about an hour long in dev, and there was no intermediate data set. It takes some input, runs for an hour, and produces an output.

The code was inherited and was a bit of a mess. Instead of digging through the code, I re-ran the pipeline through from about 6 months ago when we knew there was know bug. It was about 100+ commits since that time.

Mind you, the bug could’ve been anywhere in the codebase as far as I was concerned.

Took about a day of git bisect to narrow it down… to nothing. I found out that running code from the first commit from 6 months ago also produced incorrect data. Oops. That’s weird though because the code was running correctly back then.

A few days of debugging later, and I eventually found the culprit: a dependency package got bumped a couple weeks back. Some sort of esoteric parser had a bug but didn’t fail. It incorrectly parsed some data after the bump. Going back a version fixed the bug.

So yeah, git bisect killed about a day of my time.