blog/content/daily/2025-06-08.md

+++ title = "TLS Troubles and Embarrassing Secrets" date = 2025-06-08 +++

Last night, this website saw hours of downtime because of a combination of silly mistakes. Let me explain.

TLS Troubles

This site runs with the Ferron webserver. When you have a website, you need to have TLS certificates so users can use HTTPS to access it. Ferron has these really great configuration options to do this automatically.

Nix, used to define Ferron's config.yaml

-- ferron-conf.nix
{
  global = {
    enableAutomaticTLS = true;
    useAutomaticTLSHTTPChallenge = true;
  };
}

These automatic TLS certificates are issued by Let's Encrypt, a fantastic nonprofit you should definitely go check out. They do, however, have pretty strict rate limits. Luckily, they offer a staging feature with much higher rate limits.

I made a number of mistakes. The first was testing in production rather than on a local server.

The second was not taking advantage of Let's Encrypt's staging features. Ferron even has an option automaticTLSLetsEncryptProduction that, when false, lets one use Let's Encrypt's staging features. Even though there is a whole page, and despite knowing not to experiment in a production environment, I thought I would be fine and forged ahead.

It was the seventh systemctl restart ferron which broke my sites. Suddenly, all I got was ERR_SSL_PROTOCOL_ERR from visiting any of my sites. There it was: Lesson learned. I will do my experiments locally going forward.

A problem still remained, however. This blog is updated daily, and is built with a flake. This makes it fully declarative; a boon! Ferron, however, was configured to point to ${blog.packages.x86_64-linux.default}/wwwroot, which would be outdated until I systeml restarted the server. Ferron requests a new certificate from Let's Encrypt, though, meaning I would hit the rate limit if I restarted even a couple of times too many. To fix this, my Ferron https server points to a second http server, which hosts the blog.

My much-overcomplicated ferron setup

# ferron-conf.nix
{
  global = {
    # enable automatic tls for https
    secure = true;
    enableAutomaticTLS = true;
    useAutomaticTLSHTTPChallenge = true;
    # let an https connection travel to the blog-ferron http server without error
    disableProxyCertificateVerification = true;
    # enable proxying to local servers (reverse proxying)
    loadModules = ["rproxy"];
  };
  hosts = [
    {
      # route requests for the blog to port 8181
      domain = "blog.mtgmonkey.net";
      proxyTo = "http://localhost:8181/";
    }
  ];
}

# blog-ferron-conf.nix
# take the blog flake as an input
{ blog, ... }: {
  global = {
    # expose server to port 8181, so the main ferron server will proxy to it
    port = 8181;
    # the default flake output includes the static site at wwwroot
    wwwroot = "${blog.packages.x86_64-linux.default}/wwwroot";
  };
}

This means I only need to systemctl restart blog-ferron, rather than ...ferron, meaning I don't trigger a new Let's Encrypt cert request every time. I can update my blog however frequently I want and still ensure the reproducibility of Nix.

All code above is, clearly, just simplified snippets; the actual file are linked below

• ferron.nix, where ferron-conf.nix is in the let binding
• blog.nix, where blog-ferron-conf.nix is in the let binding
• flake.nix,the flake to which the above modules are imported

Embarrassing Secrets

I spent all morning trying to configure different secrets management programs, from agenix, to spos-nix, to even a simple .gitignore. After finally getting agenix configured properly, it occured to me that my only 'secrets' were my ssh public keys, which are harmless to share! I don't need any secrets management, much less something as complex as agenix.

By Tomorrow

• Polish up the blog a little bit
• Get a translation api set up