Home impermanence results in empty directories

[Nikola-Milovic]

Hello everyone, this is 100% my error, but I am having a hard time understanding where exactly I am I wrong. The issue seems to be that home.persistance results in empty specified directories, while environment.persistance works as expected.

My dotfiles are, most relevant is the home-manager and the nixos impermanence modules. I am using snowfall hence the organization is specific, but hopefully you can navigate it.

I am using btrfs as my fs and currently this is the service I have, it works, it rollbacks the / and the /home but for some reason, the persisted home files are all empty directories. I must've messed up the order or something but have no idea what exactly

My rollback systemd service

  config = mkIf cfg.enable {
    security.sudo.extraConfig = ''
      # rollback results in sudo lectures after each reboot
      Defaults lecture = never
    '';

    programs.fuse.userAllowOther = true;

    boot.initrd.systemd.services.rollback = {
      description = "Simplified Rollback BTRFS root subvolume to a pristine state";
      wantedBy = [ "initrd.target" ];
      before = [
        "initrd-root-fs.target"
        "sysroot-var-lib-nixos.mount"
      ];
      after = [ "sysroot.mount" ];
      unitConfig.DefaultDependencies = "no";
      serviceConfig.Type = "oneshot";
script = ''
    mkdir -p /btrfs_tmp
    mount /dev/sda2 -o subvol=/ /btrfs_tmp

    # Backup and rotate the /root subvolume
    if [[ -e /btrfs_tmp/root ]]; then
        mkdir -p /btrfs_tmp/old_roots
        timestamp=$(date --date="@$(stat -c %Y /btrfs_tmp/root)" "+%Y-%m-%-d_%H:%M:%S")
        mv /btrfs_tmp/root "/btrfs_tmp/old_roots/$timestamp"
    fi

    # Function to recursively delete old subvolumes
    delete_subvolume_recursively() {
        IFS=$'\n'
        for i in $(btrfs subvolume list -o "$1" | cut -f 9- -d ' '); do
            delete_subvolume_recursively "/btrfs_tmp/$i"
        done
        btrfs subvolume delete "$1"
    }

    # Remove /root backups older than 14 days
    for backup in /btrfs_tmp/old_roots/*; do
        if [[ -d "$backup" && $(find "$backup" -maxdepth 0 -mtime +14) ]]; then
            delete_subvolume_recursively "$backup"
        fi
    done

    # Create a fresh /root subvolume
    btrfs subvolume create /btrfs_tmp/root
    echo "Created fresh /root subvolume"

    # Optionally backup and rotate the /home subvolume if enabled
    ${optionalString cfg.home ''
      if [[ -e /btrfs_tmp/home ]]; then
          mkdir -p /btrfs_tmp/old_home
          timestamp=$(date --date="@$(stat -c %Y /btrfs_tmp/home)" "+%Y-%m-%-d_%H:%M:%S")
          mv /btrfs_tmp/home "/btrfs_tmp/old_home/$timestamp"
      fi

      # Remove /home backups older than 14 days
      for backup in /btrfs_tmp/old_home/*; do
          if [[ -d "$backup" && $(find "$backup" -maxdepth 0 -mtime +14) ]]; then
              delete_subvolume_recursively "$backup"
          fi
      done

      # Create a fresh /home subvolume
      btrfs subvolume create /btrfs_tmp/home
      echo "Created fresh /home subvolume"

      # Ensure user home directory exists in /persist
      if [[ ! -e /persist/home/${config.${namespace}.user.name} ]]; then
          mkdir -p /persist/home/${config.${namespace}.user.name}
          chown ${config.${namespace}.user.name}:${config.${namespace}.user.group} /persist/home/${config.${namespace}.user.name}
      fi
    ''}

    # Unmount /btrfs_tmp after completing operations
    umount /btrfs_tmp
  '';
    };

My home/impermanence/default.nix

{
  lib,
  config,
  osConfig,
  namespace,
  ...
}:
let
  inherit (lib) mkIf types;
  inherit (lib.${namespace}) mkBoolOpt mkOpt;
  cfg = config.${namespace}.impermanence;
in
{
  options.${namespace}.impermanence = with types; {
    enable = mkBoolOpt osConfig.${namespace}.system.impermanence.home "Is home impermanence enabled";
    files = mkOpt (listOf (either str attrs)) [ ] "Additional home files to persist.";
    directories = mkOpt (listOf (either str attrs)) [ ] "Additional home directories to persist.";
  };

  config = mkIf cfg.enable {
    home.persistence."/persist/home/${config.${namespace}.user.name}" = {
      directories = [
        "Downloads"
        "Music"
        "Pictures"
        "Desktop"
        "Files"
        "Documents"
        "Videos"
        "VirtualBox VMs"
        ".gnupg"
        ".ssh"
        ".dotfiles"
        ".cache"
        ".nixops"
        ".local/share"
        # ".local/share/keyrings"
        # ".local/share/direnv"
      ] ++ cfg.directories;
      files = [ ] ++ cfg.files;
      allowOther = true;
    };
  };
}

And this is the result

// before boot

nikola in 🌐 vm in .dotfiles on  master ➜ ls
configs  flake.lock  flake.nix  homes  lib  modules  packages  README.md  shells  systems
nikola in 🌐 vm in .dotfiles on  master ➜ pwd
/home/nikola/.dotfiles
nikola in 🌐 vm in .dotfiles on  master ➜ ls ~/
Desktop  Documents  Downloads  Files  Music  Pictures test.txt  Videos  'VirtualBox VMs'

// after reboot

nikola in 🌐 vm in ~ ➜ ls
Desktop  Documents  Downloads  Files  Music  Pictures  Videos  'VirtualBox VMs'
nikola in 🌐 vm in ~ ➜ ls .dotfiles/
<nothing>
nikola in 🌐 vm in ~ ➜ sudo rm -rf .dotfiles/
[sudo] password for nikola:
rm: cannot remove '.dotfiles/': Device or resource busy
nikola in 🌐 vm in ~ took 3s ➜ mount
/dev/sda2 on / type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=294,subvol=/old_roots/2024-11-4_16:22:50,x-initrd.mount)
/dev/sda2 on /nix type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=258,subvol=/nix,x-initrd.mount)
/dev/sda2 on /persist type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=259,subvol=/persist,x-initrd.mount)
/dev/sda2 on /var/log type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=257,subvol=/log,x-initrd.mount)
/dev/sda2 on /var/lib type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=259,subvol=/persist)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=3591908k,nr_inodes=819200,mode=755)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=718384k,nr_inodes=1792740,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=3,mode=620,ptmxmode=666)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,size=7183812k)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/keys type ramfs (rw,nosuid,nodev,relatime,mode=750)
tmpfs on /run/wrappers type tmpfs (rw,nodev,relatime,size=7183812k,mode=755)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
/dev/sda2 on /nix/store type btrfs (ro,noatime,compress=zstd:3,space_cache=v2,subvolid=258,subvol=/nix)
/dev/sda2 on /etc/machine-id type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=259,subvol=/persist)
/dev/sda2 on /etc/ssh/ssh_host_ed25519_key type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=259,subvol=/persist)
/dev/sda2 on /etc/ssh/ssh_host_ed25519_key.pub type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=259,subvol=/persist)
/dev/sda2 on /etc/ssh/ssh_host_rsa_key type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=259,subvol=/persist)
/dev/sda2 on /etc/ssh/ssh_host_rsa_key.pub type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=259,subvol=/persist)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
/dev/sda2 on /.cache/nix type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=259,subvol=/persist,x-gvfs-hide)
/dev/sda2 on /etc/NetworkManager/system-connections type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=259,subvol=/persist,x-gvfs-hide)
/dev/sda2 on /var/cache type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=259,subvol=/persist,x-gvfs-hide)
/dev/sda2 on /srv type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=259,subvol=/persist,x-gvfs-hide)
/dev/sda2 on /var/db/sudo type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=259,subvol=/persist,x-gvfs-hide)
/dev/sda2 on /var/tmp type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=259,subvol=/persist,x-gvfs-hide)
/dev/sda2 on /home type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=296,subvol=/home)
/dev/sda2 on /swap type btrfs (rw,relatime,compress=zstd:3,space_cache=v2,subvolid=261,subvol=/swap)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=7183812k)
/dev/sda1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=1436760k,nr_inodes=359190,mode=700,uid=1000,gid=100)
/persist/home/nikola/Files on /home/nikola/Files type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=100,default_permissions,allow_other)
/persist/home/nikola/Desktop on /home/nikola/Desktop type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=100,default_permissions,allow_other)
/persist/home/nikola/Downloads on /home/nikola/Downloads type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=100,default_permissions,allow_other)
/persist/home/nikola/.dotfiles on /home/nikola/.dotfiles type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=100,default_permissions,allow_other)
/persist/home/nikola/Music on /home/nikola/Music type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=100,default_permissions,allow_other)
/persist/home/nikola/.local/share/nvim on /home/nikola/.local/share/nvim type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=100,default_permissions,allow_other)
/persist/home/nikola/Videos on /home/nikola/Videos type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=100,default_permissions,allow_other)
/persist/home/nikola/Pictures on /home/nikola/Pictures type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=100,default_permissions,allow_other)
/persist/home/nikola/Documents on /home/nikola/Documents type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=100,default_permissions,allow_other)
/persist/home/nikola/.cache on /home/nikola/.cache type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=100,default_permissions,allow_other)
/persist/home/nikola/.gnupg on /home/nikola/.gnupg type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=100,default_permissions,allow_other)
/persist/home/nikola/.local/share on /home/nikola/.local/share type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=100,default_permissions,allow_other)
/persist/home/nikola/.nixops on /home/nikola/.nixops type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=100,default_permissions,allow_other)
/persist/home/nikola/.ssh on /home/nikola/.ssh type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=100,default_permissions,allow_other)
/persist/home/nikola/VirtualBox VMs on /home/nikola/VirtualBox VMs type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=100,default_permissions,allow_other)
portal on /run/user/1000/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1000,group_id=100)
nikola in 🌐 vm in ~ ➜
nikola in 🌐 vm in /persist/home/nikola ➜ tree
.
├── Desktop
├── Documents
├── Downloads
├── Files
├── Music
├── Pictures
├── Videos
└── VirtualBox VMs

9 directories, 0 files

But the regular /persist/etc for example has the files and it all works properly

nikola in 🌐 vm in /persist/home/nikola ➜ ls /persist/etc/ssh/
ssh_host_ed25519_key  ssh_host_ed25519_key.pub  ssh_host_rsa_key  ssh_host_rsa_key.pub

So it must be something with the way home.persistance differs from environment.persistance, or I just made some stupid error

[zackattackz]

I'm thinking it has something to do with the fact that the /persist/home/* mounts all map to the /home subvol, which gets erased on boot. It's different than the etc mounts because they all map to /persist/*, which isn't erased.

To point it out in the mount output:

/dev/sda2 on /etc/machine-id type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=259,subvol=/persist)

is mapping /persist/etc/machine-id -> /etc/machine-id

/persist/home/nikola/.dotfiles on /home/nikola/.dotfiles type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=100,default_permissions,allow_other)

is mapping /persist/home/nikola/.dotfiles -> /home/nikola/.dotfiles

(I could very well be wrong here, it's just the explanation that makes sense to me, but please verify this)

I tried a very similar setup recently with a separate /home subvolume, and I had the same issue as you.

I couldn't figure out a solution so I decided to ditch the /home subvolume and just use the setup suggested in the readme with both the system and home modules. It worked with no issues.

If having a /home subvol is not super critical, and you can't figure out a solution, I'd suggest you just don't use one. I decided there wasn't really any benefit for me having one anyways, since I'd be using the same options as my /root subvol, and as far as backups go, I'd just backup the whole /persistent subvol.

If you do figure out something that works though, please do let me know!

[Nikola-Milovic]

@zackattackz Thanks for the comment, are you using home manager impermanence module for home backup or are you using the nixos module?

I wanted to have them separate just for the fact that it would be nice to have opt-in impermanence for home :/

[zackattackz]

I'm using both modules, I just gave up on having a /home subvol. I'll share some of my config and explain below :)

My configs are still changing a lot so I apologize in advance if any of it is confusing lol.

Also note that I've designed it around multiple machines, so it might add an extra layer of complexity you don't need if you're just concerned with one machine. But the overall idea should still work.

Here I have the nixos module. Note that the only impermanence directories/files I have set up are ones that aren't specific to anything but the barebones nixos stuff.

For any other nixos modules I'm using which I want to add it's directories to impermanence, I use my impermanence.extra* options. For example, here is for bluetooth.

The extraUser* options are used for adding impermanence dirs/files for ALL available users for a given nixos module. For example, here is steam. This is useful for nixos modules that don't have a home-manager counterpart, but need to persist home related files.

Now here is the impermanence home module. It's useful for only a specified user via the cfg.user option, so there's no need for extraUser* options like the nixos one. It still has extraDirs and extraFiles though, for instance it is used with neovim.

Everything ends up in /persistent, and I just back that up incrementally as one whole. It would be nice to have seperate subvols to backup the /persistent/home files separately from the rest, but it's not critical for me. Everything else is reproducible via the flake, so I only need to backup the /persistent anyways, so the separation between "home" stuff and "system" stuff doesn't really matter that much backup-wise.

Oh and also here is the initrd hook I use, which is I believe just the same as the readme.

I hope this is helpful, open to answer any questions.