+++
title = "TLS Troubles and Embarrassing Secrets"
date = 2025-06-08
+++

Last night, this website saw hours of downtime because of a combination of silly mistakes. Let me explain.

### TLS Troubles

This site runs with the [Ferron](https://www.ferronweb.org/) webserver. When you have a website, you need to have TLS certificates so users can use HTTPS to access it. Ferron has these really great configuration options to do this automatically.

> [Nix](https://git.mtgmonkey.net/server-configuration.git/tree/services/ferron.nix), used to define Ferron's config.yaml
>
> ```nix
> -- ferron-conf.nix
> {
>   global = {
>     enableAutomaticTLS = true;
>     useAutomaticTLSHTTPChallenge = true;
>   };
> }
> ```

These automatic TLS certificates are issued by [Let's Encrypt](https://letsencrypt.org), a fantastic nonprofit you should definitely go check out. They do, however, have pretty strict [rate limits](https://letsencrypt.org/docs/rate-limits/). Luckily, they offer a [staging feature](https://letsencrypt.org/docs/staging-environment) with much higher rate limits.

I made a number of mistakes. The first was testing in production rather than on a local server.

The second was not taking advantage of Let's Encrypt's staging features. Ferron even has an option `automaticTLSLetsEncryptProduction` that, when `false`, lets one use Let's Encrypt's staging features. Even though there is a [whole page](https://www.ferronweb.org/docs/automatic-tls), and despite knowing not to experiment in a production environment, I thought I would be fine and forged ahead.

It was the seventh `systemctl restart ferron` which broke my sites. Suddenly, all I got was `ERR_SSL_PROTOCOL_ERR` from visiting *any* of my sites. There it was: Lesson learned. I will do my experiments locally going forward.

A problem still remained, however. This blog is updated daily, and is built with a [flake](https://git.mtgmonkey.net/blog.git/tree/flake.nix). This makes it fully declarative; a boon! Ferron, however, was configured to point to `${blog.packages.x86_64-linux.default}/wwwroot`, which would be outdated until I `systeml restart`ed the server. Ferron requests a new certificate from Let's Encrypt, though, meaning I would hit the rate limit if I restarted even a couple of times too many. To fix this, my Ferron https server points to a second http server, which hosts the blog.

> My much-overcomplicated ferron setup
>
> ```nix
> # ferron-conf.nix
> {
>   global = {
>     # enable automatic tls for https
>     secure = true;
>     enableAutomaticTLS = true;
>     useAutomaticTLSHTTPChallenge = true;
>     # let an https connection travel to the blog-ferron http server without error
>     disableProxyCertificateVerification = true;
>     # enable proxying to local servers (reverse proxying)
>     loadModules = ["rproxy"];
>   };
>   hosts = [
>     {
>       # route requests for the blog to port 8181
>       domain = "blog.mtgmonkey.net";
>       proxyTo = "http://localhost:8181/";
>     }
>   ];
> }
> ```
>
> ```nix
> # blog-ferron-conf.nix
> # take the blog flake as an input
> { blog, ... }: {
>   global = {
>     # expose server to port 8181, so the main ferron server will proxy to it
>     port = 8181;
>     # the default flake output includes the static site at wwwroot
>     wwwroot = "${blog.packages.x86_64-linux.default}/wwwroot";
>   };
> }
> ```

This means I only need to `systemctl restart blog-ferron`, rather than ...`ferron`, meaning I don't trigger a new Let's Encrypt cert request every time. I can update my blog *however frequently I want* and *still* ensure the reproducibility of Nix.

All code above is, clearly, just simplified snippets; the actual file are linked below
- [ferron.nix](https://git.mtgmonkey.net/server-configuration.git/tree/services/ferron.nix), where ferron-conf.nix is in the let binding
- [blog.nix](https://git.mtgmonkey.net/server-configuration.git/tree/services/blog.nix), where blog-ferron-conf.nix is in the let binding
- [flake.nix](https://git.mtgmonkey.net/server-configuration.git/tree/flake.nix),the flake to which the above modules are imported


### Embarrassing Secrets

I spent all morning trying to configure different secrets management programs, from [agenix](https://github.com/ryantm/agenix), to [spos-nix](https://github.com/Mic92/sops-nix), to even a simple `.gitignore`. After finally getting agenix configured properly, it occured to me that my only 'secrets' were my ssh *public keys*, which are harmless to share! I don't need *any* secrets management, much less something as complex as agenix.

### By Tomorrow

- [ ] Polish up [the blog](https://blog.mtgmonkey.net) a little bit
- [ ] Get a [translation api](https://github.com/LibreTranslate/LibreTranslate) set up