alexkarle.com

Source for alexkarle.com
git clone git://git.alexkarle.com/alexkarle.com.git
Log | Files | Refs | README | LICENSE

011-backups.txt (2561B) [raw]


      1 # 011-backups
      2 
      3 _October 12, 2021_
      4 
      5 My usual take on server backups is "don't put anything worth
      6 backing up on the server that's not stored in git elsewhere".
      7 
      8 This has treated me pretty well in the past. Source code,
      9 configuration files, and even documentation on setup are all
     10 stored in git both on the server and on my laptop, and so
     11 I can sleep at night knowing a catastrophic disk failure wouldn't
     12 mean I lost any serious work.
     13 
     14 This strategy breaks down when thinking about a tilde. First,
     15 the array of services we're providing is _much_ more complex
     16 than my normal blog server. Second, there are more people
     17 involved!
     18 
     19 I want to guarantee any tilde members that I will at least try
     20 my best to keep backups of their data in case of failure or
     21 accidental deletion.
     22 
     23 There are tons of backup tools, but a lot of them are fairly
     24 complex (with good reason I suppose.. compression, deduplication,
     25 etc). Since this tilde is about exploring OpenBSD, I took the
     26 opportunity to home-roll a simple backup solution with dump(8)
     27 and restore(8).
     28 
     29 The meat of it is in a script I'm calling "dumpster" that runs
     30 via cron with the day of the week (1-7) as the dump level
     31 and a weekly job dumping the whole system fresh:
     32 
     33 	#!/bin/sh
     34 	# dumpster -- taking out the garbash with dump(8)
     35 	
     36 	# %u is 1=mon 7=sun (unless given in ARGV)
     37 	LVL=${1:-"$(date +%u)"}
     38 	BAKDIR="/bak/$(date +%F)_$LVL"
     39 	
     40 	mkdir -p "$BAKDIR"
     41 	dump -$LVL -auf "$BAKDIR/root.dump.$LVL" /
     42 	dump -$LVL -auf "$BAKDIR/home.dump.$LVL" /home
     43 	dump -$LVL -auf "$BAKDIR/var.dump.$LVL" /var
     44 
     45 This dumps to /bak, which is a separate Linode Volume, which
     46 has better data redundancy guarantees than the VPS volume and
     47 can be detached/attached to other hosts in the event of VPS
     48 failure.
     49 
     50 As you can see, I'm only really dumping areas that have user
     51 data (/var for git, /home, and / for configs). /usr/\* can be
     52 rebuilt from /var/backups/pkglist for the most part!
     53 
     54 A note to anyone trying this: the Linode Volume was a bit hacky
     55 to get set up, since it expects to be mounting against a Linux
     56 machine. Linode's console will error on attaching, but I found
     57 that rebooting the host made the drive appear as wdN and from
     58 there I was able to format it, etc.
     59 
     60 As a bonus, I took the opportunity to set up /altroot backups,
     61 which is a brilliantly simple way to ensure you can boot into
     62 a known-good state of your host even if something goes very
     63 wrong with the main drive!
     64 
     65 Overall, I went from a backup-avoider to a backup-fan in the
     66 process :) it's so cool to watch the daily script create dump
     67 files of things that changed!