• 1 Post
  • 3 Comments
Joined 11 months ago
cake
Cake day: December 4th, 2023

help-circle


  • I realize there are solutions, but I wanted my own for various reasons (better fit to the peculiar way I store and backup).

    It was straightforward to write a python script to crawl a directory tree, adding files to an sqlite database. The script has a few commands:

    - “check” computes checksums on files whose modification times have changed since last check, or on any file whose checksum is older than X days (find bitrot this way).

    - “parity” Use par2 to compute parity files for all files in database. Store these in a “.par2” directory in the directory tree root so it doesn’t clutter the directory tree.

    I like this because I can compute checksums and parity files per directory tree (movies, music, photos, etc), and by disk (no raid here, just JBOD + mergerfs). Each disk corresponds exactly to a backup set kept in a pelican case.

    The sqlite database has the nice side effect that checksum / parity computation can run in the background and be interrupted at any time (it takes a loooooooong time). The commits are atomic, so machines crashes or have to shut down, it’s easy to resume from previous point.

    Surely… SURELY… someone has already written this. But it took me a couple of afternoons to roll my own. Now I have parity and the ability detect bitrot on all live disks and backup sets.


  • Never throw away physical media. If space is a problem, remove disks from cases and store on spindles (the packages that CD-R blanks come in are ideal).

    Most people run a compression pass on media rips (handbrake) to make storage feasible with today’s disks and budgets. The day is rapidly approaching when hard disks will be large enough and cheap enough to store bit exact copies of your media. You’ll want to rerip then, and having the media will make that possible.

    Physical media serves as long term stable backup. It should be part of your backup plan, just like multiple physical backup disks sets, offsite storage, cloud storage, etc.

    If space is an issue, there are easy solutions. Disks do not have to be in cases, and they’re too useful to part with.