commit 6db9b4f14337862980979eec1e092c40665e6b4c (patch)
parent 450381b6e46958ad86fd06b72059a057dde2ffd9
Author: Alex Karle <alex@alexkarle.com>
Date: Tue, 13 Sep 2022 00:57:06 -0400
blog: Add State of the Homelab
Diffstat:
2 files changed, 247 insertions(+), 0 deletions(-)
diff --git a/www/blog/index.txt b/www/blog/index.txt
@@ -12,6 +12,7 @@ For an up to date list of software/hardware I use, see
## 2022
+- 09/22 [State of the Homelab](/blog/state-of-the-homelab-2022-09.html)
- 08/22 [Using the Plan 9 Plumber to Turn Acme into a Git GUI](/blog/plan9-acme-git-gui.html)
- 06/22 [Automating Multi-User WireGuard setup on OpenBSD](/blog/wireguard-management.html)
- 05/22 [Starting a Tilde Community for Fun and the Learnings](/blog/starting-a-tilde.html)
diff --git a/www/blog/state-of-the-homelab-2022-09.txt b/www/blog/state-of-the-homelab-2022-09.txt
@@ -0,0 +1,246 @@
+# State of the Homelab
+
+_Published: September 12, 2022_
+
+I've been sinking some time recently into organizing my
+homelab, spurred by the recent addition of a NAS, and thought
+it might be a good time to write about it.
+
+## Birds Eye View
+
+Here's the network topology:
+
+ ┌──────────────┐ ┌───────┐
+ │ Wifi Clients │ │ Wired │
+ └──────────────┘ │Clients│
+ : └───┬───┘
+ : │
+ ┌───────▼───────┐ ┌────▼─────┐
+ Internet │ Verizon ├────► OpenBSD │
+ ─────────► Router │ │ Router │
+ │ (FiosGateway) ◄────┤ (apu2e4) │
+ └──────┬──▲─────┘ └──┬───▲───┘
+ │ │ │ │
+ ┌───▼──┴──┐ ┌───▼───┴────┐
+ │ pi.hole │ │ NAS/Git │
+ │ (Rpi4b) │ │(Odroid HC4)│
+ └─────────┘ └────────────┘
+
+
+Excluding the Fios router, that's 3 servers hosting
+the following services:
+
+- A git server for private repos (public repos hosted on
+ [git.alexkarle.com](https://git.alexkarle.com), and
+ [sourcehut](https://git.sr.ht)).
+- A NAS (network attached storage) for backups, photos, music, etc
+- A network wide ad-blocker ([pi-hole](https://pi-hole.net/))
+- An OpenBSD router/firewall (dhcp / pf) to provide extra security
+ to my wired devices.
+
+
+## Why Bother with a Homelab, Anyways?
+
+Before I dive into each component, I want to take a step back and
+ask _why_.
+
+In a world where you can pay `$HIP_COMPANY` $5/mo to
+run or host anything, it may seem like a homelab
+is a waste of time and effort. Looking at what I'm running,
+a lot of it could even be hosted for free!
+
+Despite the time cost of tending to this digital garden,
+I've found that
+running my homelab has been an incredible source of learning from
+hands-on experimentation. At this stage in my career, this type
+of experience is invaluable, especially because a lot of it
+(hardware tinkering, sysadmin tasks, linux distros, etc) doesn't
+come across my desk often.
+
+As an added bonus, I really enjoy the feeling of digital ownership
+I get from hosting my private data. It certainly comes with the
+weight of responsibility that I need to keep (and test!) backups,
+but the learning and ownership feel worthwhile for now.
+
+## The Nitty Gritty
+
+### Fios Gateway
+
+I have no special attachment to Verizon--I wouldn't go so far as
+to endorse them, but my coworkers don't complain about lag during
+video calls, so I haven't mustered the courage to switch providers.
+
+It's on my long todo list to switch to a more local ISP, but with
+both me and my fiance working from home it's the last place
+I want an outage.
+
+### Pi Hole
+
+Pi-hole is a network wide adblocker. It works by acting as the DNS
+server for your network and responding with localhost (0.0.0.0) for
+known spammy domains.
+
+As a concrete example, with pi-hole running right now, I can't access
+doubleclick.net (Google ads):
+
+ # Response from my router
+ $ host doubleclick.net 192.168.1.1
+ Using domain server:
+ Name: 192.168.1.1
+ Address: 192.168.1.1#53
+ Aliases:
+
+ doubleclick.net has address 0.0.0.0
+ doubleclick.net has IPv6 address ::
+
+
+ # Response from Google's DNS resolver
+ $ host doubleclick.net 192.168.1.1
+ Using domain server:
+ Name: 8.8.8.8
+ Address: 8.8.8.8#53
+ Aliases:
+
+ doubleclick.net has address 142.251.41.14
+ [...]
+
+The default configuration for the Gateway router is to tell clients
+to use it (192.168.1.1) as a DNS server. By updating Gateway to use
+pi-hole as _it's_ server (instead of the Verizon supplied ones), all
+clients on the network receive pi-hole's filtering.
+
+What's brilliant about this is that no clients need updating. As far
+as they're aware, they _really are_ trying to reach out to doubleclick.net.
+It's just a network failure that 0.0.0.0 isn't listening on 443!
+
+The results are most noticeable on mobile since I use uBlock Origin on
+all my browsers. It's amazing how much faster and less cluttered
+certain mobile apps are without ads (cough nytimes cough please stop
+it with the full screen banners).
+
+Finally, as the name might imply, the pi-hole was designed to run on the
+Raspberry Pi. I'm running mine on a
+[Raspberry Pi 4b](https://www.raspberrypi.com/products/raspberry-pi-4-model-b/),
+which is definitely overkill for the resources it needs.
+
+
+### Git Server
+
+Running a private git server is incredibly easy. In fact, there isn't
+really any separate git daemon that needs running. So long as you have
+ssh access to the host, you can clone/push/pull from:
+
+ $ git clone user-on-host@host:/path/to/server
+
+I like to get a bit fancy and `adduser` a `git` user on the server so
+that the repos can be stored in its home directory and ssh access can
+be managed separately from the user account I would normally use.
+
+Assuming the `git` user has the home directory `/home/git`, the following
+will clone `/home/git/repo.git`:
+
+ $ git clone git@host:repo.git
+
+Much cleaner!
+
+A lot more detail, including securing the git user by assigning it the
+`git-shell` for the login shell, can be found in the amazing
+[Pro Git](https://git-scm.com/book/en/v2/Git-on-the-Server-Setting-Up-the-Server)
+book.
+
+As far as _what_ I keep on my personal server (that I wouldn't trust
+to sourcehut, or even really my own git.alexkarle.com), I host the following
+repos:
+
+- My password repository (for [`pass(1)`](https://www.passwordstore.org/))
+- Personal notes (public notes go on [gopher://alexkarle.com/1/notes]!)
+
+In the past (mostly for fun), I've hosted [cgit](https://git.zx2c4.com/cgit/about/),
+[gitea](https://gitea.io/) and even GitLab (reverse chronologically
+and also least to most heavy). I've found that for the few private
+repos I host, I rarely want a web UI (let alone forking/user accounts/etc).
+
+### NAS
+
+The most recent upgrade to my homelab was the addition of a purpose-built
+NAS using the [Odroid HC4](https://www.hardkernel.com/shop/odroid-hc4/)
+toaster-style dual hard drive board.
+
+Previously, my backups were distributed across multiple drives and
+frequently offline (with all my operating system tinkering I have 5
+SSD's of which only 2 are in use at any time...). Things that I
+needed frequent access to were stored on a (rather fragile) Raspberry Pi 3b
+on a 64GB thumbdrive! Needless to say, the HC4 is a step up.
+
+It's only really been online for ~24hrs so I don't have a solid review
+of the hardware yet, but initial impressions are:
+
+- I was a bit bummed that the Linux they support is an old forked kernel,
+ which strikes me as probably missing security patches (didn't poke around
+ hard enough to confirm though--it may be up to date and just old!)
+- Despite being on the OpenBSD hardware list, I couldn't get it to boot.
+ Given the need to unscrew the plastic top to get the UART serial connection
+ in, I stopped trying after a few hours :(. I might email the mailing list
+ in the future to see what I overlooked.
+- I was pleased to find that the [Armbian](https://www.armbian.com/odroid-hc4/)
+ project supported it (with a newer kernel!). This is what I ended up
+ installing (with no issues so far)
+- I stumbled across [Chandler Swift's](https://chandlerswift.com/2021/05/07/odroid-hc4-nas)
+ awesome writeup about his experience with the HC4 and his golang
+ [odroidhc4-display](https://github.com/ChandlerSwift/odroidhc4-display) tool
+ is the perfect easy-to-deploy service for the little display
+
+Overall, operating system quirks aside, I'm really happy with how it turned
+out. I put in two 2TB western digital drives (whatever BestBuy had on sale
+a few weekends ago) and encrypted both of them with `cryptsetup` using the
+LUKS encryption mode as described in the
+[Arch Wiki](https://wiki.archlinux.org/title/Dm-crypt/Device_encryption).
+I intentionally did not RAID-1 the drives together because I'm more worried
+about accidentally `rm`-ing a file than I am not having access to the data
+in case of drive failure. Instead, I have only one drive always online. A
+cron job mounts the offline drive daily, `rsync`'s over the data, and
+unmounts it when done. This should give me hopefully a few hours or more
+if I realize I deleted a file. (I'll eventually also cycle in a third drive
+for offsite storage somewhere trusted like my parents' house).
+
+In the future I'd love to use this NAS as an excuse to explore fancier
+filesystems like ZFS, but I stuck to ext4 for now.
+
+
+### OpenBSD Firewall/Router
+
+*Puts tinfoil hat on.*
+
+The final piece of the topology is maybe the least functional in terms
+of hosting required services but the best learning tool: the OpenBSD router.
+
+There are arguably some security wins here by bisecting my network between
+wifi and wired clients. For one, the Verizon router itself may or may not
+be receiving security patches (it's proprietary, who knows?). By setting
+up a firewall so that the only traffic going in to wired clients is the
+traffic expected, the wired clients are a tad safer.
+
+While the security angle is certainly appealing, the much bigger reason
+to have this in my homelab has been experimenting with router technologies.
+In setting this up I had to grok `pf` (packet filtering for the firewall),
+`dhcpd` (to give clients IP addresses), and just basic networking
+(how _does_ a machine in one network talk to another?). There's no better
+way to learn networking than having my wifi laptop trying to ping my
+wired desktop and `tcpdump`ing the traffic.
+
+This is running on a PC Engines [apu2e4](https://www.pcengines.ch/apu2e4.htm),
+mostly since it seemed popular with the community and I wanted to make sure
+the device had good OpenBSD support. It's been running since April 2021
+without issues, so I'd recommend it!
+
+I would eventually love to write my own pi-hole using the DNS tools in
+base, but for now it's low on my todo list.
+
+## Conclusion
+
+If you made it this far, thanks! I hope you learned something or found
+something of interest.
+
+I'll hopefully write a similar "state of the cloud" post to cover the
+services I'm running outside home, but I think this post might just be
+long enough for now :)
