bcachefs impermanence: what does it take?

Background

I’ve been wanting to set up a NixOS server with impermanence for a while, specifically using bcachefs. To briefly go over some key concepts:

• NixOS is declarative: you describe the system state you want in configuration.nix, then nixos-rebuild generates it.
• You’re just declaring the end state you want, not describing how to make it happen — that part is delegated to tools.
• You can boot previous system generations from your bootloader, enable automatic system upgrades, and even deploy your configuration to other machines.
• A system flake makes system builds reproducible: you define inputs like the package repository in flake.nix, and they’re pinned to specific versions in flake.lock.
• You can still enable automatic upgrades, they’ll update your system’s flake.lock too.

This leaves only a couple aspects of the system unspecified: disk partitioning/formatting, user environments, and data. We already have a lot to cover regarding the base system and data, but there are ways to declare disk setup and user environments in your system configuration too.

Opt-in state

NixOS only needs two directories to start up, /boot and /nix. The nix store contains all your packages and system generations under unique, read-only paths. This is why you can boot to a previous generation, or install multiple versions of any package. See this guide for more info.

With that said, we still have a root partition accumulating state while the system is running. Your deployment depends on some of it, like network configurations, keys, certs, databases, etc. Graham Christensen came up with a simple solution: erase your system every boot. NixOS doesn’t need root; just get rid of it. Move the files you actually need to a “persist” partition, and the impermanence module will link/bind them back onto root at startup.

Graham used ZFS datasets; many others have used btrfs subvolumes and tmpfs. Let’s give it a shot with bcachefs subvolumes.

My install media doesn’t work

I’ll just get this out of the way before we partition: the regular install media doesn’t support bcachefs. NixOS 25.05 ships with Linux 6.6, one version before bcachefs was merged. We can create our own install media with the latest kernel; we just need nix. Determinate systems’ version is easy to install on other distros, macOS, or WSL, and flakes aren’t gated behind a config option.

{
  # flake.nix is an attribute set composed of name value pairs
  description = "bcachefs install media";
  # inputs must also be flakes (directories with a flake.nix)
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05";
  # outputs attribute is a lambda function
  outputs = { self, nixpkgs }: {
    nixosConfigurations = {
      # bind bcachefsIso to a function call
      bcachefsIso = nixpkgs.lib.nixosSystem {
        # pass two attributes
        system = "x86_64-linux";
        modules = [
          # list elements are separated by whitespace, wrap function
          # in parentheses so it doesn't get split
          (
            # ellipsis allows additional attributes
            { pkgs, modulesPath, ... }: {
              imports = [
                "${modulesPath}/installer/cd-dvd/" +
                "installation-cd-minimal-new-kernel-no-zfs.nix"
              ];

              boot.supportedFilesystems = [ "bcachefs" ];
              # add your ssh key and uncomment
              #users.users.root.openssh.authorizedKeys.keys = [
              #  "contents of your ~/.ssh/id_ed25519.pub"
              #];
              # enable flake support for the installer
              nix.settings.experimental-features = [
                "nix-command" "flakes"
              ];
7 collapsed lines
            }
          )
        ];
      };
    };
  };
}

Make a new folder, copy the above into flake.nix, and if you’re installing to a remote system I recommend adding your ssh public key. You’ll want an easy way to transfer files and/or copy/paste to command line. Then to build it run:

nix build .#nixosConfigurations.bcachefsIso.config.system.build.isoImage

The installer image will be in ./result/iso/.

Disk setup

Once you’ve booted it’s time to partition and format. Assuming a UEFI system:

• 512M partition, type ef00, name ESP
• leave room for swap (-2G), type 8300, name bcachefs
• use remaining space, type 8200, name swap

If you need enough swap for hibernation you can check total memory capacity in MiB with free -m.

Next we’re ready to format.

mkfs.vfat /dev/vda1
bcachefs format /dev/vda2
mkswap /dev/vda3

Now we can mount our bcachefs filesystem and get the subvolumes ready.

mount /dev/vda2 /mnt
bcachefs subvolume create /mnt/root
bcachefs subvolume create /mnt/home
bcachefs subvolume create /mnt/nix
bcachefs subvolume create /mnt/persist
bcachefs subvolume create /mnt/log

bcachefs subvolume snapshot /mnt/root /mnt/blank_root

Instead of recreating the root subvolume each boot, we restore to a blank snapshot so old roots form a snapshot tree.

mkdir -p /mnt/{home,nix,persist,var/log,boot}

mount -o X-mount.subdir=root /dev/vda2 /mnt
mount -o X-mount.subdir=home /dev/vda2 /mnt/home
mount -o X-mount.subdir=nix /dev/vda2 /mnt/nix
mount -o X-mount.subdir=persist /dev/vda2 /mnt/persist
mount -o X-mount.subdir=log /dev/vda2 /mnt/var/log

mount -o umask=0077 /dev/vda1 /mnt/boot

System configuration

Now that the disk is set up we need to make our system configuration. Run:

nixos-generate-config --root /mnt --kernel latest --no-filesystems --flake

This will create configuration.nix, hardware-configuration.nix, and flake.nix in /mnt/etc/nixos.

Enable flake support:

{ config, lib, pkgs, ... }:

{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];

  nix.settings.experimental-features = [ "nix-command" "flakes" ];

Choose a stable release channel:

{
  inputs = {
    # Use `nix flake update` to update the flake to the latest revision of the chosen release channel.
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05";
  };

You can also take a moment to add additional users, configure ssh, change your hostname, etc. Take a look around configuration.nix as well as nixos.org’s option search.

When generating the config we omitted filesystems for two reasons, the first being the generator doesn’t understand X-mount.subdir.

My modules don’t work

The second is that the generator will identify your device by UUID, while the bcachefs NixOS module only supports paths.

Luckily there are symlinks to disks according to UUID, label, etc under /dev/disk/by-*. We’ll use the partition labels assigned earlier: copy the below into hardware-configuration.nix between the boot and network settings.

fileSystems."/" = {
  device = "/dev/disk/by-partlabel/bcachefs";
  fsType = "bcachefs";
  options = [ "X-mount.subdir=root" ];
};
39 collapsed lines

fileSystems."/home" = {
  device = "/dev/disk/by-partlabel/bcachefs";
  fsType = "bcachefs";
  options = [ "X-mount.subdir=home" ];
};

fileSystems."/nix" = {
  device = "/dev/disk/by-partlabel/bcachefs";
  fsType = "bcachefs";
  options = [ "X-mount.subdir=nix" ];
};

fileSystems."/persist" = {
  device = "/dev/disk/by-partlabel/bcachefs";
  fsType = "bcachefs";
  options = [ "X-mount.subdir=persist" ];
  neededForBoot = true;
};

fileSystems."/var/log" = {
  device = "/dev/disk/by-partlabel/bcachefs";
  fsType = "bcachefs";
  options = [ "X-mount.subdir=log" ];
  neededForBoot = true;
};

fileSystems."/boot" = {
  device = "/dev/disk/by-partlabel/ESP";
  fsType = "vfat";
  options = [
    "fmask=0022"
    "dmask=0022"
  ];
};

swapDevices = [
  { device = "/dev/disk/by-partlabel/swap"; }
];

If you install as-is the system will fail to boot with “stage 2 init script (…/init) not found”. Most Linux distributions use an initial ramdisk (initrd) to mount the root filesystem. NixOS’s uses BusyBox for the standard system utilities, including mount. Unfortunately, it doesn’t support X-mount.subdir, so we need to add util-linux’s version to the image. This is already done conditionally when required by ZFS partitions, so we patch the stage-1 NixOS module to check for X-mount.subdir too. I’ve made a PR, but in the meantime you can replace modules manually:

cd /mnt/etc/nixos
curl -O https://gurevitch.net/bcachefs-impermanence/stage-1.nix

Load the replacement module in configuration.nix:

imports =
  [ # Include the results of the hardware scan.
    ./hardware-configuration.nix
    ./stage-1.nix
  ];

Note that I’ve had to make some extra changes for the local replacement module:

--- a/stage-1.nix
+++ b/stage-1.nix
@@ -9,6 +9,7 @@
   lib,
   utils,
   pkgs,

   nixpkgs,
   ...
 }:

@@ -330,7 +331,7 @@ let
   # The init script of boot stage 1 (loading kernel modules for
   # mounting the root FS).
   bootStage1 = pkgs.replaceVarsWith {

     src = ./stage-1-init.sh;
     src = "${nixpkgs}/nixos/modules/system/boot/stage-1-init.sh";
     isExecutable = true;

     postInstall = ''
@@ -811,4 +812,5 @@ in
   imports = [
     (mkRenamedOptionModule [ "boot" "initrd" "mdadmConf" ] [ "boot" "swraid" "mdadmConf" ])
   ];

   disabledModules = [ "system/boot/stage-1.nix" ];
 }

We also need to modify the flake to pass the top level nixpkgs input to modules.

{
  inputs = {
    # Use `nix flake update` to update the flake to the latest revision of the chosen release channel.
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05";
  };
  outputs = inputs@{ self, nixpkgs, ... }: {
    # NOTE: 'nixos' is the default hostname
    nixosConfigurations.nixos = nixpkgs.lib.nixosSystem {
      specialArgs = inputs;
      modules = [ ./configuration.nix ];
    };
  };
}

If you install now your system will boot!

My packages don’t work

With util-linux v2.41 you’ll get errors in stage 1:

mount: /dev/disk/by-partlabel/bcachefs on /...
mount: /dev/vda2: Invalid argument
[ERROR src/commands/mount.rs:395] Mount failed: Invalid argument

And later they’ll crash systemd-remount-fs.

X-mount.subdir is still an experimental mount option, although it’s likely to be promoted soon. There’s already a fix backported to util-linux’s stable branch that ignores X-mount.subdir when remounting, bind mounting, or moving. I have another nixpkgs PR for this, but it triggers over 5000 package rebuilds. It’ll take awhile to go through staging, then staging-next, then unstable, then get backported to 25.05.

You can modify packages on your own using overlays:

# declare overlay functions
nixpkgs.overlays = [
  # outer function just accepts arg final and returns inner function
  (final: prev: {
    util-linux = prev.util-linux.overrideAttrs (previousAttrs: {
      # add two patches to util-linux package
      patches = previousAttrs.patches ++ [
        (pkgs.fetchpatch2 {
          # first patch removes unused code, lets second apply cleanly
          url = "https://github.com/util-linux/util-linux/commit/cfb80587da7bf3d6a8eeb9b846702d6d731aa1c6.patch";
          hash = "sha256-Rutyf91IhmjVa6ZOADa3dNZtUUdkN+XihfPUwCFxrLQ=";
        })
        (pkgs.fetchpatch2 {
          # restricts X-mount.subdir option to real mounts
          url = "https://github.com/util-linux/util-linux/commit/22b91501d30a65d25ecf48ce5169ec70848117b8.patch";
          hash = "sha256-LUFpyu0paRQ7HgOuvg18F6dq6MuI+8bymcMk2XUem6g=";
        })
      ];
    });
  })
];

They’ll even continue tracking updates from the repo.

System install

With all of that out of the way you can finally run:

nixos-install --root /mnt --flake /mnt/etc/nixos#nixos

If you set your own hostname you’ll need to put that after # instead of nixos. Make sure your system works here before moving on.

Impermanence

At this point the hard work is out of the way and setting up impermanence should be relatively straightforward. First we add the input to our system flake:

inputs = {
  nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05";
  impermanence.url = "github:nix-community/impermanence";
};

Next we add the impermanence attribute in the arg for configuration.nix, import the impermanence NixOS module, and declare the files and directories to persist.

{ config, lib, pkgs, impermanence, ... }:

{
  imports = [
    # Include the results of the hardware scan.
    ./hardware-configuration.nix
    ./stage-1.nix
    impermanence.nixosModule
  ];

  environment.persistence."/persist" = {
    hideMounts = true;
    directories = [
      # prevents uids/guids from changing between boots
      "/var/lib/nixos"
      "/var/lib/systemd/coredump"
      "/etc/nixos"
    ];
    files = [
      "/etc/machine-id"
      "/etc/ssh/ssh_host_ed25519_key"
      "/etc/ssh/ssh_host_ed25519_key.pub"
      "/etc/ssh/ssh_host_rsa_key"
      "/etc/ssh/ssh_host_rsa_key.pub"
    ];
  };

  users.users.root.hashedPasswordFile = "/persist/passwords/root";
};

Note that you do need to actually copy everything over to /persist:

mkdir -p /persist/etc/ssh

cp -r {,/persist}/etc/nixos
cp {,/persist}/etc/machine-id

cp {,/persist}/etc/ssh/ssh_host_ed25519_key
cp {,/persist}/etc/ssh/ssh_host_ed25519_key.pub
cp {,/persist}/etc/ssh/ssh_host_rsa_key
cp {,/persist}/etc/ssh/ssh_host_rsa_key.pub

mkdir /persist/passwords

To populate the root password on /persist, run:

mkpasswd -m sha-512 "password" > /mnt/persist/passwords/root

We still need to save the previous root and reset to the blank snapshot on boot. Copy the following into hardware-configuration.nix under filesystems.”/” = {…};

boot.initrd.postResumeCommands = lib.mkAfter ''
  mkdir /bcachefs_tmp
  mount /dev/disk/by-partlabel/bcachefs /bcachefs_tmp
  if [[ -e /bcachefs_tmp/root ]]; then
      mkdir -p /bcachefs_tmp/old_roots
      timestamp=$(date --date="@$(stat -c %Y /bcachefs_tmp/root)" "+%Y-%m-%-d_%H:%M:%S")
      mv /bcachefs_tmp/root "/bcachefs_tmp/old_roots/$timestamp"
  fi

  # optionally delete roots older than 30 days
  #for i in $(find /bcachefs_tmp/old_roots/ -maxdepth 1 -mtime +30); do
  #    chattr -i "$i"/var/empty
  #    bcachefs subvolume delete "$i"
  #done

  bcachefs subvolume snapshot /bcachefs_tmp/blank_root /bcachefs_tmp/root
  umount /bcachefs_tmp
'';

To apply your new system configuration run:

nixos-rebuild sync

That’s it! Most of these bumps should be smoothed out by the next stable OS release, but in the meantime they provide a good crash course on the Nix ecosystem and system patching.

Fixes submitted

• nixos/stage-1: respect X-mount.subdir option (solves nixpkgs#414390)

• nixpkgs/util-linux: restrict X-mount.subdir to real mounts (solves nixpkgs#414402)

Remaining issues

• nixpkgs/317901: (has workaround) unlock-bcachefs-*.service fails with device = “UUID=…”

• util-linux/3619: (safe to ignore) failure to update userspace mount table when moving mount onto subdir