So today I discovered that there’s a cron job that holds non-reproducible state that died, and now our system is fucked.
The cron job doesn’t live inside any source control. This morning it entered a terminal state, and because it overwrites its state there’s no way to revert it.
I’m currently waiting for the database rollback and have rewritten it in a reproducible/idempotent way.
Time to restore a whole machine backup to a VM with no network connectivity, and manually pull the command?
I was able to do that
Turns out there was a second bug which triggered this one, and a bug I found in this script that I thought was responsible was happening silently for months.
Now three bugs are squashed