Snapshot orpan files cannot be removed

[las68]

hello there,

U12-322, 50.TB, BTRFS, Snapshot was set on at creation of volume.

1. At August, 8 for unknown reason it stopped to create snapshot, and service was set off
No alarm, No any email

2. Tomorrow we have got a lack of space during backup. But we have found we have almost nothing to delete.
No alarm, No any email of incoming disaster. Screenshot from system. 99% full, 64Kb free from 50Tb. Everything is fine in blue color:


3. OK, let's free some room. We have found a lot of orphaned snapshot files at /Volume1/Backup/@snapshot in subdirs.

ls -l
total 0
drwxrwx---+ 1 boss boss 432 Aug 8 17:07 GMT+08-2023.08.08-17.07.00
drwxrwx---+ 1 boss boss 432 Aug 8 18:07 GMT+08-2023.08.08-18.07.00
drwxrwx---+ 1 boss boss 432 Aug 8 19:07 GMT+08-2023.08.08-19.07.00
drwxrwx---+ 1 boss boss 432 Aug 8 20:07 GMT+08-2023.08.08-20.07.00
drwxrwx---+ 1 boss boss 432 Aug 8 21:07 GMT+08-2023.08.08-21.07.00
drwxrwx---+ 1 boss boss 432 Aug 8 22:07 GMT+08-2023.08.08-22.07.00
drwxrwx---+ 1 boss boss 432 Aug 8 23:07 GMT+08-2023.08.08-23.07.00

looks nice, we just need to delete it. The problem is superuser cannot delete these files. any rm -rf <subdir> gives a long list of files belonging to different user. su to <differen_user> works, but won't help.

A long list of error messages like:
rm: cannot remove 'GMT+08-2023.08.08-17.07.00/Blade142 TNAS2023-08-01T003009.vib': Read-only file system

if superuser (e.g. root) cannot remove any file in system, this mean something goes wrong. Very wrong.

4. I found here a post, which looks closely related to our case:
viewtopic.php?f=31&t=4746&p=27421&hilit ... ots#p27421

Removing snapshot manager was succesfull, but further installallation failed, since it needs newer TOS.
Installation of TOS impossible due to lack of free space.

Now I plan to add additional HDD (14 hours of fastsync on 8TB gives 80% of completion right now), add space, refresh TOS, install snapshot manager and try to recover control of orphan files.

Word for TNAS R&D:
1. Customer needs critical alerts first, lack of space, hw problems an so on. I do not care if someone enters TNAS over SSH.
2. Superuser MUST have permissions to delete any file in system
3. You should reimplement (reinspect, reinvent) BTRFS snapshots. It works awful and has critical flaws. This is not for business.

du -chs *
7.9T GMT+08-2023.08.08-17.07.00
7.9T GMT+08-2023.08.08-18.07.00
7.9T GMT+08-2023.08.08-19.07.00
7.9T GMT+08-2023.08.08-20.07.00
7.9T GMT+08-2023.08.08-21.07.00
7.9T GMT+08-2023.08.08-22.07.00
7.9T GMT+08-2023.08.08-23.07.00
55T total

"Does anyone still believe snapshots needs a little space on disk?" (С)

[Gremlin]

I believe!

A snapshot that is not protected (by 'read only' file system for example) is not worth much as a system protection. However, as I learnt to my cost, if you play around with the snapshots, they will chew up space. Again, for example, if you copy a snapshot to another location (thinking to preserve the snapshot) it will recreate the data in full.Thinking about that post-event, it is a logical result.

If you can list the snapshot names (as you obviously can), it is easy enough to recreate the snapshot lists (in the GUI) and control them through the GUI functions which permit deletion.

Likewise, if you maintain a lot of snapshots and frequently create and delete files, that can cause use of space since the snapshot "remembers" the deleted file data until the snapshot is deleted (effectively - I'm sure it is a bit more complicated than that). The only way to "remember" is to retain the data space itself until no longer required.

I use my system (in part) to store cctv images. Several cameras taking frequent images all the time - soon mounts up . This has to be controlled, and follow a regular clean up routine (using scheduled tasks) deleting some images after a few days and images generated by actual 'alarms' after some months. So, I only use volume snapshots and keep the retained number low. Enough to restore system in an emergency, but not enough to use a lot of space.

[las68]

After adding of new disk I have another problem. I lost, for unknown reason, all iSCSI targets and LUNs definitions.
A question for all these techsupp guys, who ignores this post - HOW TO RESTORE IT?

[las68]

I believe!

A snapshot that is not protected (by 'read only' file system for example) is not worth much as a system protection. However, as I learnt to my cost, if you play around with the snapshots, they will chew up space. Again, for example, if you copy a snapshot to another location (thinking to preserve the snapshot) it will recreate the data in full.Thinking about that post-event, it is a logical result.

If you can list the snapshot names (as you obviously can), it is easy enough to recreate the snapshot lists (in the GUI) and control them through the GUI functions which permit deletion.

Likewise, if you maintain a lot of snapshots and frequently create and delete files, that can cause use of space since the snapshot "remembers" the deleted file data until the snapshot is deleted (effectively - I'm sure it is a bit more complicated than that). The only way to "remember" is to retain the data space itself until no longer required.

I use my system (in part) to store cctv images. Several cameras taking frequent images all the time - soon mounts up . This has to be controlled, and follow a regular clean up routine (using scheduled tasks) deleting some images after a few days and images generated by actual 'alarms' after some months. So, I only use volume snapshots and keep the retained number low. Enough to restore system in an emergency, but not enough to use a lot of space.

Thank you for brilliant idea how to manually get control of snapshots. I install snapshot manager, pointed out to previous location, and voila! - it shows 7 old snapshots. I deleted it from application.

And for unknown reason system restored iscsi target/lun descriptions.
Box lives its own life.

[las68]

Another idea for Terramaster R&D:

1. An administrator should be reported if snapshot size is over threshold, e.g. snaphot size is bigger than 5% of total RAID size, else free space may be easily lost and system performance would suffer of extra disk i/o and processor load.
2. Making snapshots should be stopped, and administrator must be notified if system has low free space.

P.S. Snapshots are useless if system stores big files, like Veeam .tib ones - 1TB and more in size.
There are should be plain explicit warning in docs.

[TMzethar]

{L_BUTTON_AT}las68

We will confirm and consider what you mentioned.