Good morning all, in today’s episode of “What I learned during work hours”…
I was playing around with wxHexEditor and realised that if something catastrophic happened, I would really struggle with any data recovery if I lost the inode tables for any drive.
A quick duckle pointed me to e2image, which says in the man:
It is a very good idea to create image files for all file systems on a system and save the partition layout (which can be generated using the fdisk -l command) at regular intervals — at boot time, and/or every week or so.
I couldn’t find any prebuilt solutions for this online, so I wrote a systemd service and timer to do this for me. I save the fdisk to a text file, run e2image on a couple drives, and compress it all together in a dated 7z that can get uploaded via rsync or Mega or Dropbox etc.
The metadata image from a 500gb drive is 8gb, but compresses down to 40mb. Backup takes a couple minutes.
Unfortunately this does not work with my raid drives, but they are RAID1 so already resilient.
Apparently I was being a derp somehow. …Anyways,
My RAID drives are 16TB, e2image of this is 125gb, and 7z’d it comes down to just 63mb.
I’ll post the service, timer, and backup script in a comment, let me know if you can spot anywhere for improvements!
deleted by creator
Great tips, thanks!
I’m using ext4 across everything I think.
Can you enable superblocks after you’ve already formatted the drive?
Fdisk saves the offsets so keeping a record of that at least sounds like a good idea.
[This comment has been deleted by an automated system]
This is why I love having luks covering my entire system disk. If I want to upgrade the system with a new drive or move the drive to a different pc or sell it or dispose of it I just dd the first couple of gigs to obliterate the luks header.
It’s obviously essential to have a backup strategy, of course, but full disk encryption is the only way to go for me.
May I point out that all a RAID1 does is sync the blocks between two drives. It won’t protect against writing something dumb that would mess up the filesystem, it will just dutifully sync it.
You should be able to back up ext data from a filesystem on a RAID array, unless I’m confused about what e2image actually does. Are you trying to use it on the underlying drive devices by any chance? You have to point it at the RAID device on top of them, something like /dev/md1 rather than /dev/sda1.
This sounds like a good extra backup to have but don’t let it lull you into a false sense of security. It may help recover from a very specific kind of mistake but the recovery may be very specific as well. It’s not file backup.
Oh you’re right it does work… well fuck knows what I was doing wrong before.
Yeah this is a backup in case I like, mv file to /dev/sda1 or something.
Not a backup of the files, but a backup of the structure.
The script takes the drives as arguments:
$ pwd /usr/lib/systemd/system $ cat drive_backup.service [Unit] Description=backup fdisk + e2image Wants=drive_backup.timer [Service] Type=oneshot ExecStart=/usr/bin/backup_meta_data.sh /dev/sdc1 /dev/sdb1 [Install] WantedBy=multi-user.target
Set to run at 3:40am every day, but probably could be once weekly really.
$ cat drive_backup.timer [Unit] Description=timer to run drive backup Requires=drive_backup.service [Timer] Unit=drive_backup.service OnCalendar=*-*-* 03:40:00 [Install] WantedBy=timers.target
Should be fairly self-explanatory.
$ cat /usr/bin/backup_meta_data.sh #!/bin/bash working_dir=/home/st/drive_recovery/working backup_dir=/home/st/drive_recovery backup_date=$(date +%Y%m%d-%H%M) mkdir -p $working_dir sudo fdisk -x > $working_dir/$backup_date.fdisk for var in "$@" do clean=$(echo $var | sed 's;/;-;g') sudo e2image $var $working_dir/$backup_date.$clean done sudo 7z a $backup_dir/$backup_date.archive $working_dir/"$backup_date"* sudo rm $working_dir/"$backup_date"*
I’m really curious as to why go to all this trouble instead of using a proper file level backup and restore solution.
For fun and learning. It’s just another tool to go with file level backup.
And the backup for this is 40mb and really fast, but backing up files even when compressed would be hundreds of GB, maybe terabytes, and then you’re paying for that amount of storage online somewhere, uploading for hours…
Picture this: you open and edit one of your documents and save it.
The filesystem promptly allocates some blocks and updates the inodes. Maybe the inode table changed, maybe not. Repeat for some other files. Now your “inode backup” has a completely different picture of what is going on on your disk. If you try to recover the disk using it, all you will achieve is further corruption of the filesystem.