The Weary Travelers

A blog for computer scientists


Date: 2023-05-21
Author: Suhail
Suhail.png

How to effectively snapshot and revert with Btrfs

This post will provide guidance on a few different use cases that are facilitated by Btrfs’s copy-on-write (COW) capability.

By the end of this post you should have a working knowledge of how to create snapshots on Btrfs1Alternate spellings: BTRFS and btrfs. and how to revert changes when needed. What follows are code snippets (with some commentary) as well as a summary of the relevant foundational concepts.2Btrfs subvolumes (also toplevel and default) and snapshots, and the Snapper conveniences. Those who are uninitiated and interested in the core concepts3Either out of curiosity and a desire to understand or out of a need to materially alter the code snippets. are encouraged to read the Background section before the How-Tos section.

Let’s dig in!

How-Tos

Support for reverting to filesystem snapshots varies across different operating systems (depending on installer defaults as well as OS configuration). While the Btrfs filesystem implementation is part of the Linux kernel, that by itself isn’t sufficient for some cases. The situation is summarized below.

  openSUSE Fedora Debian and derivatives
Revert Python environment ✅ If $HOME is on a btrfs volume
Revert Flatpak state ✅ If $HOME (similarly, /) is on a btrfs volume for user-level (similarly, system-level)
Revert OS state from the GRUB menu

The necessary (but not sufficient)4 We also need to ensure that all relevant files are included and that snapshots are being taken in a timely manner. prerequisites for being able to revert on Btrfs are twofold.

  1. The location in question is a Btrfs subvolume.
  2. There exists a snapper configuration for the subvolume in question.

Convert an existing directory to a subvolume

Btrfs doesn’t provide any utilities (aka porcelain) to convert an existing directory into a subvolume. However it does provide primitives (aka plumbing) that allow us to create a subvolume where there previously existed a directory. There are two different ways.

Create subvolume and copy content over

This method may create new extents and as such isn’t recommended for large directories, since it may take longer and also use greater space.

1: # Create empty subvolume in place of directory
2: mv /parent/dir /parent/dir-old
3: sudo btrfs subvol create /parent/dir
4: # Copy contents over
5: mv /parent/dir-old/* /parent/dir-old/.* /parent/dir/
6: # Delete what should now be an empty directory
7: rm -d /parent/dir-old

Create subvolume as snapshot and clean up as needed

The previous approach, which copied files across a subvolume boundary, can result in new file extents being created. The way to ensure that that doesn’t happen is to create a subvolume via the snapshot mechanism. This is a little more involved.

  1. First, we create a new subvolume as a Btrfs snapshot

    1: sudo btrfs subvolume snapshot /parent /parent/dir-new
    
  2. Then we cleanup subvolume contents by deleting unneeded items and moving needed items to the root of the subvolume

     2: # [1/3]: delete unneeded items
     3: cd /parent/dir-new
     4: for dir in ./*; do
     5:     [ "$dir" = ./"dir" ] && continue
     6:     rm -rf "$dir"
     7: done
     8: rm -rf ./.* # also the hidden files
     9: # [2/3]: move items to root of subvolume
    10: mv /parent/dir-new/dir /parent/dir-new/dir-ref-to-old
    11: mv /parent/dir-new/dir-ref-to-old/* /parent/dir-new/dir-ref-to-old/.* /parent/dir-new/
    12: # [3/3]: cleanup: delete what should now be an empty directory
    13: rm -d /parent/dir-new/dir-ref-to-old
    
  3. Replace the directory with the subvolume

    14: rm -rf /parent/dir && mv /parent/dir-new /parent/dir
    

Convert $HOME to a subvolume

OpenSUSE documentation has a section on enabling Snapper in user home directories. That guide is intended to allow individual users to manage their snapshots and rollbacks independently. It requires that the user directories are subvolumes of /home and accomplishes this via the pam_snapper_homeconvert.sh script, which is a part of the pam_snapper package on openSUSE. The pam_snapper_homeconvert.sh script can be used both to create additional users with their $HOME as a subvolume as well as to convert the $HOME of an existing user into a subvolume.5See the relevant CONVERTUSERDIR flag in pam_snapper_homeconvert.sh.

For users of openSUSE LEAP, the official documentation is probably the best source to consult. For users of openSUSE Tumbleweed, the steps in the LEAP documentation can probably be followed with minimal changes. For users of other operating systems, the code of the pam_snapper_homeconvert.sh script is worth perusing.

Now that we’ve covered some ways of creating subvolumes, let’s consider some practical situations where this knowledge can be put to use.

Revert Python environment

Only relevant if you use a user-level package manager for Python such as Conda, Mamba, or pip.

Python package installations aren’t reproducible in general, unless you’re using something like Nix. Thus, you may encounter situations6While Conda and Mamba take care to guard against the environment having conflicting dependencies, the environment may still break after an upgrade, in non-obvious ways, when the package dependencies don’t accurately reflect reality. Meanwhile, Pip will only inform you that an environment has conflicting packages after the fact. where the only reasonable way to fix a broken environment is by reverting to a previously-working state. For this to work we need to:

Have Python environments in a Btrfs subvolume

There are several ways of creating and managing Python virtual environments. Conda and Mamba are two notable options.7 Conda is a cross-platform package and environment management system. Mamba adds features on top of Conda offering “higher speed and more reliable environment solutions”.

If Conda (or Mamba) is already installed, follow the steps to convert an existing directory to a subvolume to convert the root prefix into a subvolume.

Otherwise, if neither is installed on the system, the easiest way is to:

  • Create a subvolume to host the virtual environments (assuming the current user is user).

    1: sudo btrfs subvolume create /home/user/miniconda3
    2: sudo chown user:user /home/user/miniconda3
    
  • Then install conda using it as the installation prefix.
  • Finally (optionally) install mamba in the base environment.

    4: conda install conda-forge::mamba -n base
    

Create a Snapper configuration

Assuming that snapper is installed,8Snapper binaries for different operating systems can be downloaded from the openSUSE website. the Python environment root prefix is /home/user/miniconda3, and the user is user:

  • We first create a snapper configuration.

    1: sudo snapper -c home_user_conda create-config /home/user/miniconda3
    
  • By default snapper requires root access to take and list snapshots. Assuming that the snapper configurations are stored in /etc/snapper/configs/, we take steps to grant access to non-root users.

    2: sudo sed -i -e "s/ALLOW_USERS=\"\"/ALLOW_USERS=\"user\"/g" /etc/snapper/configs/home_user_conda
    3: # allow user to browse .snapshots without 'sudo'
    4: # NOTE: it is necessary that the owner of the snapshot directory remain root so
    5: #       we alter the group to be a group that includes the user
    6: sudo chmod a+rx ~/miniconda3/.snapshots
    7: sudo chown root:user ~/miniconda3/.snapshots
    

Ensure that timely Snapper snapshots are taken

The default Snapper configuration is configured to take hourly snapshots. While this is a reasonable default for the $HOME directory, we can do better for the Python environment.

  • First we alter the configuration created in the previous step to disable hourly snapshots from being created.

    1: sudo sed -i -e "s/TIMELINE_CREATE=\"yes\"/TIMELINE_CREATE=\"no\"/g" /etc/snapper/configs/home_user_conda
    
  • Then we create wrappers9In addition to delimiting a command with a pre and post snapshot, one can also create standalone snapshots. as desired, and put these in our ~/.bashrc or equivalent. For instance,

    conda_update() {
        snapper -c home_user_conda create -p -d "${CONDA_DEFAULT_ENV}: Update" -c number \
                --command "mamba update --all"
    }
    conda_install() {
        snapper -c home_user_conda create -p -d "${CONDA_DEFAULT_ENV}: Installing [${@}]" -c number \
                --command "mamba install ""$@"
    }
    

Revert using snapper undochange

  • One can list existing snapper snapshots via

    1: snapper -c home_user_conda list
    
  • If one has been taking snapshots in pre/post pairs diligently, one can also do

    1: snapper -c home_user_conda list -t pre-post
    
  • One can see a file-level summary of changes between snapshots via

    1: # Format:  snapper -c <config> status <start>..<end>
    2: snapper -c home_user_conda status 1..2 # changes between 1 and 2
    3: snapper -c home_user_conda status 2..0 # changes since 2 (till present)
    
  • Diffs can be reviewed between snapshots via

    1: # Format: snapper -c <config> diff <start>..<end> [files]
    2: snapper -c home_user_conda diff 1..2 /home/user/miniconda3/pkgs/urls.txt
    
  • Changes can be reverted via the undochange command. For instance in order to restore snapshot version 2, we undo the changes since 2.

    1: # Format: snapper -c <config> undochange <start>..<end> [files]
    2: snapper -c home_user_conda undochange 2..0
    

Revert Flatpak state10Flatpak is a package manager which allows users to install and run applications independently of the system libraries provided by the OS distribution.

While it’s possible to install an older build of a Flatpak application, there is no simple way11Cf. snap revert. to revert to the state prior to a flatpak update. However, if our flatpak installation is in a Btrfs subvolume, we can use an associated snapper configuration to allow us to checkpoint on every update.12Similar to how we revert Python environments. Having created these checkpoints, we can then revert to them13Since Flatpak stores all its data (runtimes, applications and configuration) nested within a single directory, it is easy to see that we can meet the conditions for being able to revert on Btrfs. as and when needed.

  • First we convert /var/lib/flatpak 14For system-wide usage. For per-user usage, convert $HOME/.local/share/flatpak instead. into a subvolume using either of the approaches discussed previously.
  • Then we create a snapper configuration similarly to what we did for Python environments.

    1: # assuming /var/lib is on a Btrfs partition
    2: sudo snapper -c var_lib_flatpak create-config /var/lib/flatpak
    3: sudo sed -i -e "s/ALLOW_USERS=\"\"/ALLOW_USERS=\"user\"/g" /etc/snapper/configs/var_lib_flatpak
    4: # allow user to browse .snapshots without 'sudo'
    5: # NOTE: it is necessary that the owner of the snapshot directory remain root so
    6: #       we alter the group to be a group that includes the user
    7: sudo chmod a+rx /var/lib/flatpak/.snapshots
    8: sudo chown root:user /var/lib/flatpak/.snapshots
    9: sudo sed -i -e "s/TIMELINE_CREATE=\"yes\"/TIMELINE_CREATE=\"no\"/g" /etc/snapper/configs/var_lib_flatpak
    
  • Finally we create some wrappers as desired. Such as,

    flatpak_update() {
        snapper -c var_lib_flatpak create -p -d "flatpak/update" -c number \
                --command "sudo flatpak update"
    }
    flatpak_install() {
        snapper -c var_lib_flatpak create -p -d "flatpak/install: ${@}" -c number \
                --command "flatpak install ""$@"    
    }
    
  • And then, when needed, we can revert to prior versions using snapper undochange.

Revert OS state from the GRUB menu

In order to be able to boot (from the GRUB menu) into a read-only snapshot and revert to it, openSUSE has implemented some custom functionality:

  • Patches to GRUB so that paths15For loading the OS, are resolved wrt the default subvolume as opposed to the toplevel subvolume.16The situation is described in some detail in this mailing list post.
  • Hooks, that are invoked after updating system packages, to take snapshots via snapper and then update the default subvolume.17As such, the default subvolume is ephemeral and is subject to change. Specifically, the default subvolume changes at the next system update or system rollback.

Having done so, rolling back the system to a previous snapshot is as simple as:

  1. Booting into the read-only snapshot from GRUB.
  2. Verifying that system is working as expected.
  3. Rolling back.

    1: sudo snapper rollback
    2: systemctl reboot
    

However, in such systems, care needs to be taken when creating subvolumes at the root-level.18Because, in these systems, / happens to refer to the default subvolume, which as noted before is ephemeral, as opposed to the toplevel subvolume. Specifically, creating a subvolume located at / 19I.e., creating a subvolume directly at, say, /dirname. is to be avoided, since doing so violates the assumptions needed to boot into an older OS snapshot from GRUB and rolling back via Snapper’s rollback mechanism. These assumptions require that subvolumes located at / are robust to changes in the default subvolume.20For reasons that are too long for this margin to contain. Let’s consider an instance where this arises naturally.

Shield Nix 21Nix package manager is a user-level package manager which provides reproducible builds and the ability to roll back to previous versions. package installation when reverting OS state

This is only relevant on platforms where one has the ability to revert OS state (i.e., openSUSE).

We want to ensure that when we revert the root filesystem, that we also don’t revert the Nix store. Ensuring that the Nix store is within a separate subvolume is necessary for this purpose. Since Nix uses /nix as its store location, for reasons mentioned before, we want to ensure that /nix is robust to changes in the default subvolume.22Which is what gets mounted at /.

We achieve this by following the convention used for other subvolumes mounted under23As opposed to “mounted at”. /, and make nix a subvolume of @ 24In openSUSE, the toplevel subvolume contains a single subvolume called @. All other subvolumes are descendents of @. and mount it as /nix.

Below, code Listing 1 assumes that Nix isn’t yet installed.25If it already is installed, we’d need to follow steps to convert an existing directory to a subvolume.

 1: # Mount the top-level subvolume explicitly
 2: # mount -o <subvol> <partition> <mount point>
 3: # e.g., mount -o subvolid=0 /dev/sda2 /mnt
 4: # when using luks encryption, on openSUSE, the decrypted partition is at
 5: # /dev/system/root
 6: sudo mount -o subvolid=0 /dev/system/root /mnt
 7: # On openSUSE, the toplevel subvolume, has a single subvolume within it called
 8: # '@'. the '@' subvolume acts as the ancestor of all other subvolumes and
 9: # represents a "stable" target
10: sudo btrfs subvol create /mnt/@/nix
11: sudo umount /mnt
12: sudo sh -c 'echo "/dev/system/root  /nix  btrfs  subvol=/@/nix  0  0" >> /etc/fstab'
13: sudo mkdir /nix # make the mount target
14: sudo mount -a # mount /nix (and test the fstab config)

And now we can follow the steps to install Nix.

Background

Btrfs

Btrfs26For the curious, a somewhat dated, but still relevant, article: “A short history of btrfs”, LWN.is an Extent-based, copy-on-write (COW) filesystem27Other notable examples supporting copy-on-write are ZFS and NTFS. for Linux.28That is, not only for GNU​/Linux, but even non-GNU distributions such as Alpine Linux. Copy-on-write means that making copies or taking snapshots is extremely light on resources.29I.e., efficient.

Btrfs subvolume30 From the Btrfs documentation on Subvolumes.

A Btrfs subvolume is a part of [the] filesystem with its own independent file/directory hierarchy and inode number namespace. Subvolumes can share file extents. … A subvolume has always inode number 256.31Except in the case of the toplevel subvolume.

A subvolume looks like a normal directory, with some additional operations.… Subvolumes can be renamed or moved, nesting subvolumes is not restricted but has some implications regarding snapshotting. The numeric id (called subvolid or rootid) of the subvolume is persistent and cannot be changed.

The Btrfs documentation

Default subvolume

The default subvolume is what gets mounted via mount unless a subvolid or subvol is specified.32See Btrfs-specific mount options.

  • The default subvolume can be queried via

    1: sudo btrfs subvolume get-default /
    
  • The default subvolume can be set via the set-default command.

Toplevel subvolume30 From the Btrfs documentation on Subvolumes.

A freshly created [Btrfs] filesystem is also a subvolume, called top-level, internally has [sic] an id [of] 5.33subvolid=5, which corresponds to the Btrfs_FS_TREE_OBJECTID. This subvolume cannot be removed or replaced by another subvolume. This is also the subvolume that will be mounted by default, unless the default subvolume has been changed …

The Btrfs documentation

In addition, when mounting, subvolid​=0 is an alias for the toplevel subvolume.34Since Linux kernel version 2.6.34.

Btrfs snapshot

A snapshot is a subvolume like any other, with given initial content. By default, snapshots are created read-write. File modifications in a snapshot do not affect the files in the original subvolume.35 Btrfs subvolume and snapshot.

[…]

[S]napshotting is not recursive, so a subvolume or a snapshot is effectively a barrier and no files in the nested appear in the snapshot. Instead there’s a stub subvolume (also sometimes empty subvolume with the same name as original subvolume, with inode number 2). This can be used intentionally but could be confusing in case of nested layouts.36Btrfs Nested subvolumes.

The Btrfs documentation

Snapper37Not to be confused with snap and snapcraft.

Snapper is a utility that allows one to create and compare snapshots on Btrfs (and LVM) and revert changes. Some notable subcommands that snapper provides are:

  • create-config and delete-config
  • rollback to revert OS state 38Snapper’s rollback is only relevant for reverting changes to the default subvolume. For reverting changes to other subvolumes, use undochange.
  • undochange to revert subvolume state
  • create with --command to create a pre and post snapshot and run command in between

Snapper stores all snapshots in the .snapshots directory inside the subvolume in question.39From the Snapper documentation.

Comments

Comments can be left on twitter, mastodon, as well as below, so have at it.

To view the Giscus comment thread, enable Giscus and GitHub’s JavaScript or navigate to the specific discussion on Github.

Footnotes:

1

Alternate spellings: BTRFS and btrfs.

3

Either out of curiosity and a desire to understand or out of a need to materially alter the code snippets.

4

We also need to ensure that all relevant files are included and that snapshots are being taken in a timely manner.

5

See the relevant CONVERTUSERDIR flag in pam_snapper_homeconvert.sh.

6

While Conda and Mamba take care to guard against the environment having conflicting dependencies, the environment may still break after an upgrade, in non-obvious ways, when the package dependencies don’t accurately reflect reality. Meanwhile, Pip will only inform you that an environment has conflicting packages after the fact.

7

Conda is a cross-platform package and environment management system. Mamba adds features on top of Conda offering “higher speed and more reliable environment solutions”.

8

Snapper binaries for different operating systems can be downloaded from the openSUSE website.

9

In addition to delimiting a command with a pre and post snapshot, one can also create standalone snapshots.

10

Flatpak is a package manager which allows users to install and run applications independently of the system libraries provided by the OS distribution.

12

Similar to how we revert Python environments.

13

Since Flatpak stores all its data (runtimes, applications and configuration) nested within a single directory, it is easy to see that we can meet the conditions for being able to revert on Btrfs.

14

For system-wide usage. For per-user usage, convert $HOME/.local/share/flatpak instead.

16

The situation is described in some detail in this mailing list post.

17

As such, the default subvolume is ephemeral and is subject to change. Specifically, the default subvolume changes at the next system update or system rollback.

18

Because, in these systems, / happens to refer to the default subvolume, which as noted before is ephemeral, as opposed to the toplevel subvolume.

19

I.e., creating a subvolume directly at, say, /dirname.

20

For reasons that are too long for this margin to contain.

21

Nix package manager is a user-level package manager which provides reproducible builds and the ability to roll back to previous versions.

22

Which is what gets mounted at /.

23

As opposed to “mounted at”.

24

In openSUSE, the toplevel subvolume contains a single subvolume called @. All other subvolumes are descendents of @.

25

If it already is installed, we’d need to follow steps to convert an existing directory to a subvolume.

26

For the curious, a somewhat dated, but still relevant, article: “A short history of btrfs”, LWN.

27

Other notable examples supporting copy-on-write are ZFS and NTFS.

28

That is, not only for GNU​/Linux, but even non-GNU distributions such as Alpine Linux.

29

I.e., efficient.

31

Except in the case of the toplevel subvolume.

33

subvolid=5, which corresponds to the Btrfs_FS_TREE_OBJECTID.

37

Not to be confused with snap and snapcraft.

38

Snapper’s rollback is only relevant for reverting changes to the default subvolume. For reverting changes to other subvolumes, use undochange.