[RESOLVED] snapper automatic timeline cleanup not cleaning up

[jlittle]

I use snapper just for the automatic snapshots. Usually I just let snapper do it's thing and it's only when I need something that I pay any attention to the snapper snapshots. (On those occasions they're very good to have.) I've run this way for years on my desktop.

But today I thought to check for stray snapshots after the recent release upgrade and discovered:

$ sudo btrfs su li / | wc -l
925

Not good. The snapper timeline cleanup was not working properly, if at all. Looking at the snapper log the number of snapshots has been growing roughly since October, well before the release upgrade that I did in December. My desktop kubuntu install has two subvolumes, root and home, and both had hundreds of snapper snapshots.

$ sudo snapper -c home cleanup timeline
$ sudo snapper -c root cleanup timeline

and I'm back to 25 or so snapshots for each subvolume.

So, the cleanup run by the systemd timer has not worked. Checking the timer with systemctl shows that is has been running without errors, and /var/log/snapper.log and journalctl -u '*snapper*' show no errors.

Any ideas? I know the intersection of KFN and snapper is small.

[Snowhog]

Maybe see https://www.reddit.com/r/archlinux/c...old_snapshots/

[Schwarzer Kater]

What does sudo snapper get-config say?

Here you might find some additional tips regarding snapper (despite it is meant for openSUSE):
https://en.opensuse.org/openSUSE:Snapper_Tutorial

[jlittle]

Maybe see https://www.reddit.com/r/archlinux/c...old_snapshots/

I think that poster's problem was that the timers weren't set up and enabled. On my desktop the timers and the cleanup tasks are running. And before October they worked for years.

[jlittle]

What does sudo snapper get-config say?

Key                    | Value
-----------------------+------
ALLOW_GROUPS           | sudo
ALLOW_USERS            | john
BACKGROUND_COMPARISON  | yes
EMPTY_PRE_POST_CLEANUP | yes
EMPTY_PRE_POST_MIN_AGE | 1800
FREE_LIMIT             | 0.2
FSTYPE                 | btrfs
NUMBER_CLEANUP         | yes
NUMBER_LIMIT           | 5
NUMBER_LIMIT_IMPORTANT | 5
NUMBER_MIN_AGE         | 1800
QGROUP                 |
SPACE_LIMIT            | 0.7
SUBVOLUME              | /
SYNC_ACL               | no
TIMELINE_CLEANUP       | yes
TIMELINE_CREATE        | yes
TIMELINE_LIMIT_DAILY   | 7
TIMELINE_LIMIT_HOURLY  | 12
TIMELINE_LIMIT_MONTHLY | 4
TIMELINE_LIMIT_WEEKLY  | 0
TIMELINE_LIMIT_YEARLY  | 1
TIMELINE_MIN_AGE       | 1800

This hasn't changed for years. The home config is similar.

[Schwarzer Kater]

Hm, hm - the only thing I have found so far regarding timeline is the change from Jul 14 2023:

Fri Jul 14 14:05:56 CEST 2023 - aschnell@suse.com

- enable snapper-timeline.timer when adjusting config using
snapper (gh#openSUSE/snapper#788)

(from https://github.com/openSUSE/snapper)

I don't know if this affects your version of snapper in Kubuntu and I think it will not affect your problem at all, though…

[jlittle]

tldr; I needed to restart, not start or enable, the systemd timer.

This systemd timer stuff is confusing. I stumbled upon the systemctl list-timers command, and it shows:

$ systemctl list-timers snapper-cleanup.timer
NEXT LEFT LAST                         PASSED              UNIT                  ACTIVATES
-    -    Sat 2023-09-23 22:11:38 NZST 3 months 4 days ago snapper-cleanup.timer snapper-cleanup.service

1 timers listed.

3 months 4 days is just earlier than the "October" when I saw snapshots piling up. The APT history log shows a big update to systemd on 2023-09-17, so I conclude there was a change in behaviour with that update.

I was confused by

$ systemctl status --all snapper-cleanup.timer
● snapper-cleanup.timer - Daily Cleanup of Snapper Snapshots
Loaded: loaded (/lib/systemd/system/snapper-cleanup.timer; enabled; preset: enabled)
Active: active (elapsed) since Sat 2023-12-23 17:23:10 NZDT; 4 days ago
Trigger: n/a
Triggers: ● snapper-cleanup.service
Docs: man:snapper(8)
man:snapper-configs(5)

Dec 23 17:23:10 myriam systemd[1]: Started snapper-cleanup.timer

I saw "enabled", "active", "4 days ago" and thought it was good. But that meant the timer was loaded, not that it fired. Instead of "active (elapsed)" it should have "active (waiting)", and the "Trigger:" line should not say "n/a", rather show the timestamp when the timer goes off.

I have reverted my changes to the timer, and done sudo systemd daemon-reload, with no change.

Pouring over so many google search results, I found that someone had to restart the timer. So

$ sudo systemctl restart snapper-cleanup.timer
$ systemctl list-timers --all snapper-cleanup.timer
NEXT                         LEFT     LAST                         PASSED  UNIT                  ACTIVATES
Fri 2023-12-29 13:33:08 NZDT 23h left Thu 2023-12-28 13:33:08 NZDT 58s ago snapper-cleanup.timer snapper-cleanup.service

1 timers listed.

and now the status shows as "active (waiting)".

A couple of the threads suggested that there is a race condition in systemd when something is being updated, if a timer triggers during the update. Someone had 7 similar systems with the same systemd configuration and 2 out of 7 had the problem... but I think I'm indulging in speculation.

(expletives and curses elided) I should have resorted to a purge and install of snapper early on. And systemd could surely have logged something at boot about a timer needing a restart, and some diagnostic if one uses "start" on a timer uselessly.