From 561801692e45afb41fc5333d77c6489ed8a7df61 Mon Sep 17 00:00:00 2001 From: Alex Karle Date: Mon, 4 Nov 2024 02:41:35 +0100 Subject: [PATCH] Check in nihdoc-formatted garbash-www Merging these repos just makes deploying and styling easier! Besides its probably some of the coolest stuff on the site :) --- Makefile | 2 +- www/blog/starting-a-tilde.txt | 8 ++++---- www/blog/wireguard-management.txt | 2 +- www/garbash/index.txt | 26 ++++++++++++++++++++++++++ www/garbash/~alex/001-domain-name.txt | 19 +++++++++++++++++++ www/garbash/~alex/002-install.txt | 43 +++++++++++++++++++++++++++++++++++++++++++ www/garbash/~alex/003-httpd.txt | 30 ++++++++++++++++++++++++++++++ www/garbash/~alex/004-mail-server.txt | 21 +++++++++++++++++++++ www/garbash/~alex/005-ssh-hardening.txt | 21 +++++++++++++++++++++ www/garbash/~alex/006-use-the-src.txt | 45 +++++++++++++++++++++++++++++++++++++++++++++ www/garbash/~alex/007-git-coding.txt | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ www/garbash/~alex/008-local-irc.txt | 32 ++++++++++++++++++++++++++++++++ www/garbash/~alex/009-wireguard.txt | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ www/garbash/~alex/010-irc-bouncer.txt | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ www/garbash/~alex/011-backups.txt | 67 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ www/garbash/~alex/012-mailing-lists.txt | 38 ++++++++++++++++++++++++++++++++++++++ www/garbash/~alex/013-wildcard-cert.txt | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ www/garbash/~alex/index.txt | 29 +++++++++++++++++++++++++++++ 18 files changed, 681 insertions(+), 6 deletions(-) create mode 100644 www/garbash/index.txt create mode 100644 www/garbash/~alex/001-domain-name.txt create mode 100644 www/garbash/~alex/002-install.txt create mode 100644 www/garbash/~alex/003-httpd.txt create mode 100644 www/garbash/~alex/004-mail-server.txt create mode 100644 www/garbash/~alex/005-ssh-hardening.txt create mode 100644 www/garbash/~alex/006-use-the-src.txt create mode 100644 www/garbash/~alex/007-git-coding.txt create mode 100644 www/garbash/~alex/008-local-irc.txt create mode 100644 www/garbash/~alex/009-wireguard.txt create mode 100644 www/garbash/~alex/010-irc-bouncer.txt create mode 100644 www/garbash/~alex/011-backups.txt create mode 100644 www/garbash/~alex/012-mailing-lists.txt create mode 100644 www/garbash/~alex/013-wildcard-cert.txt create mode 100644 www/garbash/~alex/index.txt diff --git a/Makefile b/Makefile index 1acdc65..590a72d 100644 --- a/Makefile +++ b/Makefile @@ -3,7 +3,7 @@ DESTDIR = /var/www/htdocs CFLAGS = -g -O2 -Wall -Wpedantic -Wextra # Variables used to determine what to build (and clean) -HTML != echo www/*.txt www/blog/*.txt | sed 's@\([^\.]*\)\.txt@\1.html@g' +HTML != find www -name '*.txt' | sed 's@\([^\.]*\)\.txt@\1.html@g' SETS != find www/jam-tuesday -name '[0-9][0-9][0-9][0-9]-*' BUILT = $(HTML) \ diff --git a/www/blog/starting-a-tilde.txt b/www/blog/starting-a-tilde.txt index b341c77..fa40da6 100644 --- a/www/blog/starting-a-tilde.txt +++ b/www/blog/starting-a-tilde.txt @@ -51,19 +51,19 @@ almost every hackernews comment will tell you *not* to self host it. Don't even try! You'll forget some DNS record and big senders will mark your domain as bad. You'll never be able to reach their inboxes. Well, with garbash, I was able to say "so what" and -[set up email addresses with OpenSMTPD](https://alexkarle.com/garbash/~alex/notes/004-mail-server.html) +[set up email addresses with OpenSMTPD](https://alexkarle.com/garbash/~alex/004-mail-server.html) fully featured with DKIM signing and proper SPF and DMARC records. Better yet, the act of pairing and explaining services I've set up -before like [git hosting via `stagit(1)`](https://alexkarle.com/garbash/~alex/notes/007-git-coding.html) -or [a Wireguard VPN tunnel for internal services](https://alexkarle.com/garbash/~alex/notes/009-wireguard.html) +before like [git hosting via `stagit(1)`](https://alexkarle.com/garbash/~alex/007-git-coding.html) +or [a Wireguard VPN tunnel for internal services](https://alexkarle.com/garbash/~alex/009-wireguard.html) solidified my mental model of the technologies. Writing scripts together to automate things like setting up users or Wireguard key management was both fun and a chance to share coding tips and tricks. But best of all, I came away from garbash with a stronger friendship. -We [set up an IRC server](https://alexkarle.com/garbash/~alex/notes/008-local-irc.html) +We [set up an IRC server](https://alexkarle.com/garbash/~alex/008-local-irc.html) expecting it to be the hub of the network. Instead it became our preferred way to chat just the two of us about life and tech. diff --git a/www/blog/wireguard-management.txt b/www/blog/wireguard-management.txt index 1c81dc1..659a44a 100644 --- a/www/blog/wireguard-management.txt +++ b/www/blog/wireguard-management.txt @@ -246,7 +246,7 @@ AllowedIP followed by a restart of the interface: ### Sending the Config Sending the config is easy--we already have [email on the -VM](https://alexkarle.com/garbash/~alex/notes/004-mail-server.html)! +VM](https://alexkarle.com/garbash/~alex/004-mail-server.html)! Using the `mail(1)` client to deliver internally is a oneliner: mail -s "Your wireguard info" "$USERNAME" < "/etc/wg/$USERNAME/client.conf" diff --git a/www/garbash/index.txt b/www/garbash/index.txt new file mode 100644 index 0000000..55addb6 --- /dev/null +++ b/www/garbash/index.txt @@ -0,0 +1,26 @@ +# Welcome + . + '). + /;(( + .'____`. + | g | create + | a | something + | r | worth + | bash | rewriting + `-____-' + +garbash was an invite-only [tilde](https://tildeverse.org) +community that was retired in early 2023. + +[~alex's page](/garbash/~alex) as well as the +[config](https://git.alexkarle.com/garbash-config) were kept +for historical purposes, and has some info on how to set +up a tilde, should you be interested! + +To all who participated--thanks, it was fun! + +*NOTE:* As of 2024 the garbash.com domain was decommissioned. +I've updated any URLs to alexkarle.com, but left tutorials +referencing it where appropriate. It's been snatched up by +domain squatters, so don't go visiting :( + diff --git a/www/garbash/~alex/001-domain-name.txt b/www/garbash/~alex/001-domain-name.txt new file mode 100644 index 0000000..08afffd --- /dev/null +++ b/www/garbash/~alex/001-domain-name.txt @@ -0,0 +1,19 @@ +# 001-domain-name + +_Fri Sep 17, 2021_ + +garbash the word was all ~anthony's idea. + +It came out of a PR review as a self-deprecating take on his bash +(which actually turned out to be mine). + +We laughed and I realized the domain was for sale, so I snagged it +and resolved to find its purpose later. + +I've always wanted to be part of a tilde community, but struggle +opening up to strangers on the internet. Starting a tilde with a +friend, however, felt much more promising. All the ascii and none +of the awkwardness! + +So here we are a week later, standing up this site. Hope you enjoy +your stay! diff --git a/www/garbash/~alex/002-install.txt b/www/garbash/~alex/002-install.txt new file mode 100644 index 0000000..e4f968b --- /dev/null +++ b/www/garbash/~alex/002-install.txt @@ -0,0 +1,43 @@ +# 002-install + +_Tues Sept 21, 2021_ + +I'm a huge fan of OpenBSD. The simplicity of the system, the cohesive +feel it has, the proactive stance on security... when we decided we'd +set up a tilde, I knew I wanted it to be on OpenBSD. + +The only problem? My preferred registrar (Linode) doesn't support it! + +Fortunately there's a comprehensive post on the Linode forum of how to +do it [here](https://www.linode.com/community/questions/10329/openbsd-on-linode). + +It took us ~45m, the longest OpenBSD install I've had since I first +flashed it on an old thinkpad. 40m of that was waiting for the node to +boot and reboot, etc (we kept messing up the configuration). + +1. Create a New Linode (any OS will do) +2. Once booted, shut it down +3. Under the "Storage" tab delete the ext4 partitions +4. Create two new disks, both "Raw" format: + 1. One labeled "install", 1GB (could do less) + 2. One labeled "os", the rest of the space +5. Boot in "Rescue" mode +6. In the serial console, wget the minirootXX.img + 1. Check the sha256 against the SHA256 file + 2. Check the signature using signify (on a different machine that + has signify) +7. Find the install disk with `lsblk` +8. Flash the img using: `dd if=minirootXX.img of=/dev/sdX bs=1M` +9. In the Configurations tab, create a new one: + 1. Full Virtualization + 2. Select a Kernel > Direct Disk + 3. /dev/sda - os + 4. /dev/sdb - install + 5. boot from sdb +10. Reboot into configuration, install OpenBSD from serial console +11. Halt/shutdown, and change configuration to boot from sda +12. Rejoice! + +In our case, our main problem was that we skipped the "Direct Disk" +kernel step so we were booting a Linux kernel and trying to load the +img... it panic'd every time! Took us a few boots to figure that out :) diff --git a/www/garbash/~alex/003-httpd.txt b/www/garbash/~alex/003-httpd.txt new file mode 100644 index 0000000..6e5e56b --- /dev/null +++ b/www/garbash/~alex/003-httpd.txt @@ -0,0 +1,30 @@ +# 003-httpd + +_Tues Sept 21, 2021_ + +One of the first things I do when I set up a machine is set up +httpd(8) and grab a HTTPS cert via acme-client(8). + +Here's a quick rundown (though reading the man pages is worth +the time!). + + # sed 's/example.com/garbash.com/g' \ + /etc/examples/httpd.conf > /etc/httpd.conf + # sed 's/example.com/garbash.com/g' \ + /etc/examples/acme-client.conf > /etc/acme-client.conf + +Then go in and edit the files to add aliases if needed! + +To get the certs for the first time: + + # rcctl enable httpd + # rcctl start httpd + # acme-client -v garbash.com # get certs + # rcctl reload httpd # load certs + +Finally, to keep the certs up to date, add the following to the +crontab: + + # crontab -e + ... + ~ * * * * acme-client garbash.com && rcctl reload httpd diff --git a/www/garbash/~alex/004-mail-server.txt b/www/garbash/~alex/004-mail-server.txt new file mode 100644 index 0000000..0acb7f9 --- /dev/null +++ b/www/garbash/~alex/004-mail-server.txt @@ -0,0 +1,21 @@ +# 004-mail-server + +_Tues Sept 21, 2021_ + +We threw this together late in our first pairing session to set +up the site. I think having a solid email server is an important +part of standing up a site (allows forwarding cron email to an +inbox that's read, etc). And of course giving out email accounts +is crucial to attract people to a tilde ;) + +The setup mostly followed Gilles' [excellent +post](https://poolp.org/posts/2019-09-14/setting-up-a-mail-server-with-opensmtpd-dovecot-and-rspamd/) +but I replaced the rspamd bits with opensmtpd-filter-dkimsign, which is super +simple to set up (see the README that comes with the installed pkg). + +I'll post the whole config soon once we get git hosting set up! + +EDIT: Sat Sep 25 00:20:01 EDT 2021 + +Git hosting is up and here's the +[config](https://git.alexkarle.com/garbash-config/file/etc/mail/smtpd.conf.html) diff --git a/www/garbash/~alex/005-ssh-hardening.txt b/www/garbash/~alex/005-ssh-hardening.txt new file mode 100644 index 0000000..86fccb2 --- /dev/null +++ b/www/garbash/~alex/005-ssh-hardening.txt @@ -0,0 +1,21 @@ +# 005-ssh-hardening + +_Tues Sept 21, 2021_ + +Just a quick note/reminder that one of the FIRST things you should +ALWAYS do on a new machine is make sure: + +1. root cannot log in (PermitRootLogin no) +2. passwords are not accepted (ssh-key only -- PasswordAuthentication no) + +Both these are set under `/etc/ssh/sshd_config`. Make sure to upload +your `~/.ssh/id_rsa.pub` first to `~/.ssh/authorized_keys` (so as not +to lock yourself out!) and then make the edits and reload the daemon: + + # rcctl reload sshd + +We waited literally only 4 hrs to do this and we already had script +kiddies knocking down our /var/log/authlog :( + +If my old coworker Joe was right about one thing, it's that the +internet these days is a cesspool. diff --git a/www/garbash/~alex/006-use-the-src.txt b/www/garbash/~alex/006-use-the-src.txt new file mode 100644 index 0000000..ba37ae4 --- /dev/null +++ b/www/garbash/~alex/006-use-the-src.txt @@ -0,0 +1,45 @@ +# 006-use-the-src + +_Tues Sept 21, 2021_ + +Use the source, Luke! + +One of the main reasons to use a FOSS OS is that you can see the code! +For me as a dev, it's been a lifechanging experience. Often it's faster +to just look at the code than try to decipher Stack Overflow answers, +and I always learn more that way! + +Another perk of the \*BSD's is that all of their source is in one repo. +This can of course make SCM slow, but from a curious-developer perspective +it's a dream come true. + +OpenBSD uses cvs(1) to manage their source, but they publish a read-only +git(1) mirror to GitHub, which I like to use for familiarity sake. + +Traditionally, all the source lives in /usr/src , and OpenBSD expects +you to put it there (for build purposes). + +To get it, first add yourself to the wsrc and wobj groups so you can build without doas + + # usermod -G wsrc,wobj + +Then clone a bare repo to /var/git (default /usr/src not big enough for .git) + + # mkdir /var/git + # chmod 775 /var/git + # chown root:wsrc /var/git + $ cd /var/git + $ git clone --bare https://github.com/openbsd/src + +Now check out a new worktree at /usr/src + + $ git -C /var/git/src.git worktree add /usr/src + +Finally find your favorite tool and build it + + $ cd /usr/src/bin/ed + $ make obj # for out of tree build, see make(1) OBJDIR + $ make + $ ./obj/ed + +How cool is that? diff --git a/www/garbash/~alex/007-git-coding.txt b/www/garbash/~alex/007-git-coding.txt new file mode 100644 index 0000000..23852d4 --- /dev/null +++ b/www/garbash/~alex/007-git-coding.txt @@ -0,0 +1,69 @@ +# 007-git-coding + +_Fri Sept 24, 2021_ + +git(1) is one of my favorite tools. All good tilde's should host it! +After all, tilde's are for sharing and what better way to share than +publishing your code! + +## Git Hosting + +Out of the box, git supports hosting for users with accounts via ssh. +You can clone like so: + + user@host:path/relative/to/home + +or: + + user@host:/abs/path/on/host + +For anonymous access, git-daemon(1) can be configured to serve over +the git:// protocol. On OpenBSD, enable and start it with the path +to the directories to serve: + + $ rcctl enable gitdaemon + $ rcctl set gitdaemon flags "--base-path=/var/git" + $ rcctl start gitdaemon + +The last bit of the puzzle is of course the shared git layout! For +git-daemon to work, we need all users to put their files under the +same dir (/var/git). But, we want to prevent accidental clobbering +via stray rm -rf, so we make a directory for each user and chown +it to their account so soley they can access it: + + /var/git/alex + .../www + .../config + /var/git/anthony + ... + +Then, for easy clone URLs, we ln(1) the dir into the home directory: + + ln -s /var/git/$USER /home/$USER/git + +Now they can clone via $USER@garbash.com:git/REPO + +## Web Hosting + +git hosting is one thing, but these days everyone likes to show off +their code in the browser for onlookers. Enter stagit(1). + +I tried cgit(1), one of the more popular git-frontends, but with httpd(8)'s +chroot(8)-ing, it was kind of a pain to get the more advanced features. + +stagit(1) generates static HTML for individual repos, which is a nice +balance of flexible and lightweight. + +The hardest part here was that I had to hack stagit(1) and stagit-index(1) +to support our two-tiered directory layout (by default it only supports +single directory layouts). This turned out to be not _that_ hard. See +my [fork](https://git.alexkarle.com/garbash-stagit/) for the specifics. + +These HTML files are then generated on-the-fly at push time via git-hooks, +specifically a post-receive hook. + +The whole process requires quite a bit of setup at repo-creation time +(assigning ownership, description, clone-url, and the post-receive hook), +so I rolled it all into a script globally available to our users: `newrepo`. +That too is available via the system [config files](https://git.alexkarle.com/garbash-config/). +Give it a look! diff --git a/www/garbash/~alex/008-local-irc.txt b/www/garbash/~alex/008-local-irc.txt new file mode 100644 index 0000000..21504c2 --- /dev/null +++ b/www/garbash/~alex/008-local-irc.txt @@ -0,0 +1,32 @@ +# 008-local-irc + +_Fri Sep 24 23:56:43 EDT 2021_ + +Tonight I took the first steps towards on-tilde communication. +It's far from done, but it'll give ~anthony and I something to +chat on while we set up the other services! + +The current plan is to have (for security reasons) a IRC server +ONLY listening on localhost. Then, we'll spin up a bouncer for +users to connect to so they can get chat history while offline. +That bouncer will be exposed externally (either over TLS or over +wireguard). + +The first step was to install ngircd. To be honest, I didn't +survey the scene toooo much. I did a search: + + pkg_info -Q irc + +And just picked the ircd that seemed most promising. + +Set up was a simple service start: + + rcctl enable ngircd + rcctl start ngircd + +And the config file was super well documented so even with my +very beginner knowledge of server admin-ship, I was able to get +it up in no time! + +The [config](https://git.alexkarle.com/garbash-config/file/etc/ngircd/ngircd.conf.html), +of course, is public! diff --git a/www/garbash/~alex/009-wireguard.txt b/www/garbash/~alex/009-wireguard.txt new file mode 100644 index 0000000..a939195 --- /dev/null +++ b/www/garbash/~alex/009-wireguard.txt @@ -0,0 +1,62 @@ +# 009-wireguard + +_Tues Sep 28, 2021_ + +Wireguard is probably one of the coolest technologies I've encountered +in a long time. The simplicity of public key auth (ssh-style where the +protocol doesn't care how you get the public key on the server) all +in the kernel? Sign me up! + +On our tilde, we want to set up wireguard so that we can provide vpn-only +services (for security reasons such as not allowing brute-force password +attempts). + +The very first of these services is IRC--we want people to be able to +connect from mobile devices and personal computers, but our network is +currently not password protected and has no services like NickServ, etc. + +The solution? Have it listen on a wireguard IP and distribute wg keys +to trusted tilde members :) + +I'll start with the obligatory RTFM -- wg(8) and ifconfig(8) are both +really well documented. However, there was a bit of fun hackery that went +down on our tuesday pair-admining call that's worth documenting! + +~anthony and I needed a simple tool to manage wireguard keys and IPs. +When a new device is to be given access we want to: + +1. Generate a private key, public key, and wg-quick(1) config file + to distribute to the user +2. Obtain the next numerical hostname +3. Add the peer to our wg endpoint on the server + +To do this, we used a small sh(1) script that has a catalog of names in +a flat file like so: + + host1 10.6.6.1 + host2 10.6.6.2 + ... + +And then each host has a directory: + + host1/ + private.key + public.key + client.conf + +The tool is called [wggen(1)](https://git.alexkarle.com/garbash-config/file/usr/local/bin/wggen.html), +and it ends up effectively: + +1. Creating a directory for NAME +2. Generating a wg(8) key using openssl(1): +3. Creating a temporary wg endpoint to get the public key using + the grep/cut hack in wg(8)'s EXAMPLES +4. tail(1)-ing the host file to get the next available IP +5. Using all the above to generate the client.conf +6. Adding the wgpeer line to /etc/hostname.wg0 and restarting the + prod endpoint with sh /etc/netstart + +I'll leave the exact details as an exercise for the reader to go look +at the git repo :) + +Needless to say, this was a lot of fun to write! diff --git a/www/garbash/~alex/010-irc-bouncer.txt b/www/garbash/~alex/010-irc-bouncer.txt new file mode 100644 index 0000000..66d9d1b --- /dev/null +++ b/www/garbash/~alex/010-irc-bouncer.txt @@ -0,0 +1,83 @@ +# 010-irc-bouncer + +_Tues Sept 28, 2021_ + +After ~anthony and I set up wggen(1), we could properly access IRC +outside of ssh(1) (on our laptops, phones, etc). + +The next missing piece of the IRC puzzle was setting up a bouncer. +For those less familiar with IRC (read: me 6 months ago), a bouncer +is simply a special IRC client that is always on, staying in the +channels for you, listening. When you connect, you then connect to +the bouncer, which feeds you missed messages. + +This is necessary because IRC has no concept of history or buffered +messages built in. So if you're not active on the network, there's +no way to get missed messages. + +Of course bouncers provide all sorts of other nice features--a single +login point for multiple networks (garbash, libera.chat, etc), +auto-away, logging support, etc. + +For our users on this tilde, we wanted to make sure they could have +chat history without having to set up their own bouncer. + +We picked [soju(1)](https://soju.im), since I've set it up before and I'm a general +fan of the software coming from the sourcehut team. It was relatively +painless to set up on OpenBSD: + + $ pkg_add go sqlite3 scdoc # dependencies + $ git clone https://git.sr.ht/~emersion/soju/ + $ cd soju + $ make + # make install + +Then, I added a new \_soju user using adduser(8) and created the cfg +to listen on our wireguard port in /home/\_soju/soju.cfg: + + listen irc+insecure://10.6.6.1:6677 + db sqlite3 /home/_soju/soju.db + +Finally, I used sojuctl(1) to add myself as a user: + + $ sojuctl -config /home/_soju/soju.cfg create-user alex -admin + +Add made a small /etc/rc.d script: + + #!/bin/ksh + daemon="/usr/local/bin/soju -config /home/_soju/soju.cfg" + daemon_user="_soju" + daemon_logger="daemon.info" + . /etc/rc.d/rc.subr + rc_bg=YES + rc_cmd "$1" + +And enabled and started soju: + + # rcctl enable soju + # rcctl start soju + +We're still ironing out the kinks in the user registration process, but +the current process is to connect to the soju instance first and add +the local network like so: + +In irssi: + + /network add -sasl_username -sasl_password -sasl_mechanism PLAIN garbash + /server add -auto -net garbash irc.garbash.com 6677 + /connect garbash + +Once connected, start a DM with the BouncerServ (provided by soju) + + /msg BouncerServ help + network create -name garbash -addr irc+insecure://localhost:6667 + +Finally, modify our garbash network username to run soju in "single +upstream mode" (aka it should only connect to this one network) by +changing our username to be /garbash (the network we just created): + + /network modify -sasl_username /garbash garbash + /connect garbash + /save + +And 10 commands and 2 connections later, we have a bouncer! diff --git a/www/garbash/~alex/011-backups.txt b/www/garbash/~alex/011-backups.txt new file mode 100644 index 0000000..8791adb --- /dev/null +++ b/www/garbash/~alex/011-backups.txt @@ -0,0 +1,67 @@ +# 011-backups + +_October 12, 2021_ + +My usual take on server backups is "don't put anything worth +backing up on the server that's not stored in git elsewhere". + +This has treated me pretty well in the past. Source code, +configuration files, and even documentation on setup are all +stored in git both on the server and on my laptop, and so +I can sleep at night knowing a catastrophic disk failure wouldn't +mean I lost any serious work. + +This strategy breaks down when thinking about a tilde. First, +the array of services we're providing is _much_ more complex +than my normal blog server. Second, there are more people +involved! + +I want to guarantee any tilde members that I will at least try +my best to keep backups of their data in case of failure or +accidental deletion. + +There are tons of backup tools, but a lot of them are fairly +complex (with good reason I suppose.. compression, deduplication, +etc). Since this tilde is about exploring OpenBSD, I took the +opportunity to home-roll a simple backup solution with dump(8) +and restore(8). + +The meat of it is in a script I'm calling "dumpster" that runs +via cron with the day of the week (1-7) as the dump level +and a weekly job dumping the whole system fresh: + + #!/bin/sh + # dumpster -- taking out the garbash with dump(8) + + # %u is 1=mon 7=sun (unless given in ARGV) + LVL=${1:-"$(date +%u)"} + BAKDIR="/bak/$(date +%F)_$LVL" + + mkdir -p "$BAKDIR" + dump -$LVL -auf "$BAKDIR/root.dump.$LVL" / + dump -$LVL -auf "$BAKDIR/home.dump.$LVL" /home + dump -$LVL -auf "$BAKDIR/var.dump.$LVL" /var + +This dumps to /bak, which is a separate Linode Volume, which +has better data redundancy guarantees than the VPS volume and +can be detached/attached to other hosts in the event of VPS +failure. + +As you can see, I'm only really dumping areas that have user +data (/var for git, /home, and / for configs). /usr/\* can be +rebuilt from /var/backups/pkglist for the most part! + +A note to anyone trying this: the Linode Volume was a bit hacky +to get set up, since it expects to be mounting against a Linux +machine. Linode's console will error on attaching, but I found +that rebooting the host made the drive appear as wdN and from +there I was able to format it, etc. + +As a bonus, I took the opportunity to set up /altroot backups, +which is a brilliantly simple way to ensure you can boot into +a known-good state of your host even if something goes very +wrong with the main drive! + +Overall, I went from a backup-avoider to a backup-fan in the +process :) it's so cool to watch the daily script create dump +files of things that changed! diff --git a/www/garbash/~alex/012-mailing-lists.txt b/www/garbash/~alex/012-mailing-lists.txt new file mode 100644 index 0000000..98e7779 --- /dev/null +++ b/www/garbash/~alex/012-mailing-lists.txt @@ -0,0 +1,38 @@ +# 012-mailing-lists + +_Weds April 27, 2022_ + +I surprised myself today while revisiting this site +to find that I had set up mailing lists months ago. +Or something that resembles them. + +We don't have any users other than myself and ~anthony, +so they've gone unused, but my vision for the tilde +is one that uses email for collaboration (outside of +IRC). + +I don't have any experience setting up majordomo or +anything like that, but since all accounts are local I +was able to create a "mailing list" by just updating +/etc/mail/aliases with the lists: + + announce: alex, anthony + +And rerunning `newaliases(8)`. Hacky but it works! +I wonder if they have a limit... + +In addition, I had set up primitive HTML archives using +`hypermail(1)`. A dedicated user account to manage the +archives simply runs the following every 5m via `cron(8)`: + + + #!/bin/sh + # archiveit -- regular archives + for l in announce bugs dev misc; do + hypermail -d /var/www/htdocs/lists/$l -g -m /var/lists/$l + done + + +Pop it behind the VPN and serve it up with `httpd(8)` and +we have a mailing list archive so new members can see +the old conversations! diff --git a/www/garbash/~alex/013-wildcard-cert.txt b/www/garbash/~alex/013-wildcard-cert.txt new file mode 100644 index 0000000..62955f2 --- /dev/null +++ b/www/garbash/~alex/013-wildcard-cert.txt @@ -0,0 +1,90 @@ +# 013-wildcard-cert + +_Thur Apr 28, 2022_ + +I've been running two internal services for a while: + +- lists.garbash.com -- mailing list archive +- irc.garbash.com -- [gamja](https://git.sr.ht/~emersion/gamja) IRC client + (and [soju](https://soju.im) bouncer) + +Because we're not using split DNS (hosting our own DNS server for +clients on the VPN), these are kept internal only by having the public +DNS have the internal IP addresses: + + $ host irc.garbash.com + irc.garbash.com has address 10.6.6.1 + +This works great, except it becomes harder to obtain a TLS certificate. +My favorite way to get a TLS certificate on OpenBSD is acme-client(8), +which is in base and works out of the box but does not support dns-01 +ACME challenges required for wildcard certs. As such, all sites requiring +certs need to be publicly accessible for HTTP challenges. + +It would have been OK to serve over plain HTTP (in the sense that the +services are served over a wireguard tunnel, so they're already encrypted), +but browsers only allow desktop notifications for HTTPS sites, so to get +notifications for `gamja`, I needed a wildcard cert. + +The wildcard cert turned out to be not too hard. For a couple months I +used [`uacme`](https://github.com/ndilieto/uacme) because it was in ports, +but getting the client to update our DNS in Linode wasn't supported in +the upstream project (as far I can tell). So for a couple times I actually +ran the tool with manual DNS mode--updating the TXT records by hand myself. + +This clearly isn't sustainable (mostly because it requires remembering every +couple months to redo it), so I moved to [`acme.sh`](https://github.com/acmesh-official/acme.sh), +which, despite not being in ports, was super easy to install and use. + +To get the new certs, I created a new user: + + # adduser + ... acmesh, nologin, daemon, etc ... + +Then I created a `certs` group so that all the services that need the certs +can read the certificates: + + # addgroup certs + # usermod -G _soju certs + # usermod -G acmesh certs + +I had to manually `chmod` some of the directories of `acme.sh` to allow +group-writable, and `chown` those directories to `acmesh:certs`. + +Finally, installing the cert was as simple as: + + $ export LINODE_V4_API_KEY=... + $ ./acme.sh --install -m alex@garbash.com # one time + $ ./acme.sh --issue --dns dns_linode_v4 --dnssleep 300 -d *.garbash.com + +This installed the certs to `/home/acmesh/.acme.sh`. `httpd(8)` needs the +fullchain and private key: + + + server "lists.garbash.com" { + listen on * tls port 443 + directory auto index + root "/htdocs/lists" + tls { + certificate "/home/acmesh/.acme.sh/*.garbash.com/fullchain.cer" + key "/home/acmesh/.acme.sh/*.garbash.com/*.garbash.com.key" + } + location "/.well-known/acme-challenge/*" { + root "/acme" + request strip 2 + } + } + + +The final step is to modify the crontab to restart the services +when it runs successfully! Since this is running as the `acmesh` +user, I needed to give it permission to run the `rcctl` command +passwordless by adding the following to doas.conf: + + permit nopass acmesh as root cmd /usr/sbin/rcctl args restart httpd + +Adding the following to the crontab will cause it to run on success: + + --reloadcmd '/usr/sbin/rcctl restart httpd' + +Hopefully I won't need to think about this for a while! diff --git a/www/garbash/~alex/index.txt b/www/garbash/~alex/index.txt new file mode 100644 index 0000000..9920ca1 --- /dev/null +++ b/www/garbash/~alex/index.txt @@ -0,0 +1,29 @@ +# ~alex's page + +## Sysadmin Notes + +As ~anthony and I set up the tilde, I tried to document +every step of the way so that others could learn from it +and/or follow in our footsteps! + +As the years have gone by, I'm also just glad I did it +for myself :) + +**NOTE:** As of 2024 the garbash.com domain was decommissioned. +I've updated any URLs to alexkarle.com, but left tutorials +referencing it where appropriate. It's been snatched up by +domain squatters, so don't go visiting :( + +- [Awesome domain name :)](001-domain-name.html) +- [OpenBSD install on Linode](002-install.html) +- [HTTP(S) server](003-httpd.html) +- [Email (SPF, DKIM, etc)](004-mail-server.html) +- [SSH hardening](005-ssh-hardening.html) +- [Obtained the source code for the system](006-use-the-src.html) +- [Set up git hosting via stagit(1)](007-git-coding.html) +- [Set up IRC for tilde members](008-local-irc.html) +- [Set up wireguard](009-wireguard.html) +- [Set up IRC bouncer](010-irc-bouncer.html) +- [Basic backup solution](011-backups.html) +- [Primitive Mailing Lists and Archive](012-mailing-lists.html) +- [Wildcard certificate for internal services](013-wildcard-cert.html) -- libgit2 1.8.1