So today I discovered that there’s a cron job that holds non-reproducible state that died, and now our system is fucked.

The cron job doesn’t live inside any source control. This morning it entered a terminal state, and because it overwrites its state there’s no way to revert it.

I’m currently waiting for the database rollback and have rewritten it in a reproducible/idempotent way.

    • wise_pancake@lemmy.caOP
      6·
      2 个月前

      He does charge a consulting fee to “fix” these issues

      Almost all of them are dumb shit like this, where something is built in super hacky and dumbass ways.

        • Agent641@lemmy.world
          1·
          2 个月前

          Judgement day postponed indefinitely due to “Object reference not set to an instance of an object”

          • Sherry@programming.dev
            1·
            2 个月前

            that might be a stupid question, but why would you running all services in tmux be a bad idea? a co-worker of mine is doing exactly that right now, which is why I’m asking.

            • qaz@lemmy.worldEnglish
              2·
              2 个月前
              1. They’re all gone when you restart
              2. It doesn’t properly deal with logging
              3. You can’t set up dependencies between services but that doesn’t matter due to point 1

              I recommend using systemd services and/or docker compose instead. systemd services are files that describe which program / script to run and when (like after networking is active or after a certain other service is loaded).

            • swab148@lemm.eeEnglish
              1·
              2 个月前

              It’s not horrible, like it’ll do the job just fine, it’s just probably a better idea to use systemd and like, containers and whatnot, but I couldn’t be arsed to fiddle with all that for Jellyfin, caddy reverse proxy, and two modded Minecraft servers, so shell scripts and tmux won the day. It takes a little extra time to restart everything after an update, and maybe I’ll get the motivation to do things “correctly™” one day, but today is not that day.