Snapshot orpan files cannot be removed
Posted: 28 Nov 2023, 14:55
hello there,
U12-322, 50.TB, BTRFS, Snapshot was set on at creation of volume.
1. At August, 8 for unknown reason it stopped to create snapshot, and service was set off
No alarm, No any email
2. Tomorrow we have got a lack of space during backup. But we have found we have almost nothing to delete.
No alarm, No any email of incoming disaster. Screenshot from system. 99% full, 64Kb free from 50Tb. Everything is fine in blue color:
3. OK, let's free some room. We have found a lot of orphaned snapshot files at /Volume1/Backup/@snapshot in subdirs.
ls -l
total 0
drwxrwx---+ 1 boss boss 432 Aug 8 17:07 GMT+08-2023.08.08-17.07.00
drwxrwx---+ 1 boss boss 432 Aug 8 18:07 GMT+08-2023.08.08-18.07.00
drwxrwx---+ 1 boss boss 432 Aug 8 19:07 GMT+08-2023.08.08-19.07.00
drwxrwx---+ 1 boss boss 432 Aug 8 20:07 GMT+08-2023.08.08-20.07.00
drwxrwx---+ 1 boss boss 432 Aug 8 21:07 GMT+08-2023.08.08-21.07.00
drwxrwx---+ 1 boss boss 432 Aug 8 22:07 GMT+08-2023.08.08-22.07.00
drwxrwx---+ 1 boss boss 432 Aug 8 23:07 GMT+08-2023.08.08-23.07.00
looks nice, we just need to delete it. The problem is superuser cannot delete these files. any rm -rf <subdir> gives a long list of files belonging to different user. su to <differen_user> works, but won't help.
A long list of error messages like:
rm: cannot remove 'GMT+08-2023.08.08-17.07.00/Blade142 TNAS2023-08-01T003009.vib': Read-only file system
if superuser (e.g. root) cannot remove any file in system, this mean something goes wrong. Very wrong.
4. I found here a post, which looks closely related to our case:
viewtopic.php?f=31&t=4746&p=27421&hilit ... ots#p27421
Removing snapshot manager was succesfull, but further installallation failed, since it needs newer TOS.
Installation of TOS impossible due to lack of free space.
Now I plan to add additional HDD (14 hours of fastsync on 8TB gives 80% of completion right now), add space, refresh TOS, install snapshot manager and try to recover control of orphan files.
Word for TNAS R&D:
1. Customer needs critical alerts first, lack of space, hw problems an so on. I do not care if someone enters TNAS over SSH.
2. Superuser MUST have permissions to delete any file in system
3. You should reimplement (reinspect, reinvent) BTRFS snapshots. It works awful and has critical flaws. This is not for business.
du -chs *
7.9T GMT+08-2023.08.08-17.07.00
7.9T GMT+08-2023.08.08-18.07.00
7.9T GMT+08-2023.08.08-19.07.00
7.9T GMT+08-2023.08.08-20.07.00
7.9T GMT+08-2023.08.08-21.07.00
7.9T GMT+08-2023.08.08-22.07.00
7.9T GMT+08-2023.08.08-23.07.00
55T total
"Does anyone still believe snapshots needs a little space on disk?" (С)
U12-322, 50.TB, BTRFS, Snapshot was set on at creation of volume.
1. At August, 8 for unknown reason it stopped to create snapshot, and service was set off
No alarm, No any email
2. Tomorrow we have got a lack of space during backup. But we have found we have almost nothing to delete.
No alarm, No any email of incoming disaster. Screenshot from system. 99% full, 64Kb free from 50Tb. Everything is fine in blue color:
3. OK, let's free some room. We have found a lot of orphaned snapshot files at /Volume1/Backup/@snapshot in subdirs.
ls -l
total 0
drwxrwx---+ 1 boss boss 432 Aug 8 17:07 GMT+08-2023.08.08-17.07.00
drwxrwx---+ 1 boss boss 432 Aug 8 18:07 GMT+08-2023.08.08-18.07.00
drwxrwx---+ 1 boss boss 432 Aug 8 19:07 GMT+08-2023.08.08-19.07.00
drwxrwx---+ 1 boss boss 432 Aug 8 20:07 GMT+08-2023.08.08-20.07.00
drwxrwx---+ 1 boss boss 432 Aug 8 21:07 GMT+08-2023.08.08-21.07.00
drwxrwx---+ 1 boss boss 432 Aug 8 22:07 GMT+08-2023.08.08-22.07.00
drwxrwx---+ 1 boss boss 432 Aug 8 23:07 GMT+08-2023.08.08-23.07.00
looks nice, we just need to delete it. The problem is superuser cannot delete these files. any rm -rf <subdir> gives a long list of files belonging to different user. su to <differen_user> works, but won't help.
A long list of error messages like:
rm: cannot remove 'GMT+08-2023.08.08-17.07.00/Blade142 TNAS2023-08-01T003009.vib': Read-only file system
if superuser (e.g. root) cannot remove any file in system, this mean something goes wrong. Very wrong.
4. I found here a post, which looks closely related to our case:
viewtopic.php?f=31&t=4746&p=27421&hilit ... ots#p27421
Removing snapshot manager was succesfull, but further installallation failed, since it needs newer TOS.
Installation of TOS impossible due to lack of free space.
Now I plan to add additional HDD (14 hours of fastsync on 8TB gives 80% of completion right now), add space, refresh TOS, install snapshot manager and try to recover control of orphan files.
Word for TNAS R&D:
1. Customer needs critical alerts first, lack of space, hw problems an so on. I do not care if someone enters TNAS over SSH.
2. Superuser MUST have permissions to delete any file in system
3. You should reimplement (reinspect, reinvent) BTRFS snapshots. It works awful and has critical flaws. This is not for business.
du -chs *
7.9T GMT+08-2023.08.08-17.07.00
7.9T GMT+08-2023.08.08-18.07.00
7.9T GMT+08-2023.08.08-19.07.00
7.9T GMT+08-2023.08.08-20.07.00
7.9T GMT+08-2023.08.08-21.07.00
7.9T GMT+08-2023.08.08-22.07.00
7.9T GMT+08-2023.08.08-23.07.00
55T total
"Does anyone still believe snapshots needs a little space on disk?" (С)