Understanding Snapshots

[bidmead]

I'm trying to. I'm hoping contributors here will be able to help me, although this section of the forum seems to be rather sparsely attended. Which is a pity, because snapshots seem (to me) to be a really important feature of TOS 5.

I have a Multimedia shared folder on a BTRFS+TRAID NAS which I've set up to be snapshotted daily. The snapshots live inside the Multimedia folder in a subfolder called @snapshot.

I was recently messing with a directory merge app which I didn't fully understand, something I came to realise on discovering many of the multimedia files within their subfolders had disappeared. I didn't want to restore the entire Multimedia directory, just the missing files. So I opened the @snapshot folder, found the snapshot immediately previous to my unfortunate merge experiment and began helping myself to replacements using copy and paste.

This worked well until my NAS suddenly completely ran out of space. I realised what was happening: a snapshot is just a pointer to where the file is to be found and takes up little space on its own. But every time you copy that snapshot and paste it elsewhere you're recreating the entire file, effectively duplicating the file so it takes up twice the space.

What I was looking for, it seemed to me, was a restore method that simply restored the directory information while still pointing to the original file. This would look like a copy and paste exercise except that it would be very much faster and wouldn't increase the amount of used space.

As far as I can make out, this is exactly what the restore inside the Terramaster Snapshot app does. Except that it does it to the entire snapshot, not allowing you to cherry-pick individual files. So, having added a new drive to the NAS to expand the storage pool and the volume within it I restored an entire snapshot and (as far as I could make out) didn't see any increase in the use of storage space.

If my interpretation was right, the restore had recreated the directory information without duplicating the files.

But now I want to cherry pick individual files and replace them in the Multimedia directory. Copying and pasting would return us to the duplication problem. The Terramaster WebUI's File Manager offers no Move option. But how about Cut and Paste? And once I've cherry-picked my restore files using cut and paste, I'll simply delete all the other unwanted files the snapshot has created, leaving the snapshot target directory clean for future reuse.

So, questions:

1. Is there a way of restoring individual files from a snapshot that doesn't duplicate the file?
2. When I use Snapshot Restore to restore an entire snapshot, am I right in assuming this takes up no extra drive space?
3. Does cutting and pasting individual files from this restored snapshot also not increase used drive space?
4. In summary, is the preferred way of restoring individual files: a) restore the entire snapshot, b) cut and paste individual files from that restoration, and c) delete the remaining files restored from the snapshot?

--
Chris

[bidmead]

Or can you perhaps do the following to restore individual files?

Ignore the Snapshot app. Instead, open up the appropriate @snapshot directory, select your snapshot within that and find the file you want to restore. Then…

DON'T copy and paste. Instead, CUT and paste.

Working Conjecture, not fully tested:

To cut a file from an @snapshot directory doesn't actually cut the file. It does the "directory info" copy, leaving the data this metadata points to in situ. Then when you paste this directory info into a new location, you've moved the file to the desired place without duplication. This "move" doesn't actually remove the file from the snapshot.

Comments would be very welcome.

--
Chris

[TMnorah]

1. The snapshot application does not support restoring individual files in the snapshot.
2. No additional space is taken up when using the snapshot application to restore the entire snapshot.
3. In the file management, the files copied and pasted from @snapshot are physical files, which will occupy real space.
4. We will consider whether to support the functional requirement of restoring a single file from a snapshot.

[bidmead]

1. The snapshot application does not support restoring individual files in the snapshot.

Thanks for that, @TMnorah. Wanted to make sure I wasn't missing something.

2. No additional space is taken up when using the snapshot application to restore the entire snapshot.

Yes, that was my understanding.

3. In the file management, the files copied and pasted from @snapshot are physical files, which will occupy real space.

…as I discovered, to my cost. Thanks for confirming this.

4. We will consider whether to support the functional requirement of restoring a single file from a snapshot.

This would certainly be a very useful option. In the meantime, I'd very much appreciate validation (or otherwise) of the workaround I'm proposing: CUTting and pasting from the individual files in the directory snapshot. Am I right in asserting that cut and paste in these circumstances is a viable way to restore the original without creating a new copy of the file?

If this is correct, we already have a functional equivalent of the single file restoration. The one significant way that building this into the Snapshot app could improve on this would be to solve the permissions problem.

What's the permissions problem? Cutting and pasting a snapshot doesn't (if I'm right) actually delete the original, as an ordinary cut and paste would do. However, the underlying Unix utility, which I take to be mv or its BTRFS equivalent, does seem to obey standard rules and will fail if the file permissions don't allow deletion.

I've been running into the situation where the snapshot is owned by a system user that has marked its files read-only for everyone else, including admins. Attempts by an admin to restore such a file using cut and paste will therefore fail and it's then necessary to resort to the space-wasting copy and paste.

One last point: We've established that copy and paste restorations of the sort I've described here leave unnecessary space consuming old snapshot images in their wake. Once the next snapshot run is activated on the restoration, the old snapshot is redundant. There are BTRFS command line utilities for cleaning these up. Do the devs plan to surface these utilities to the TOS 5 WebUI? Or do we all need to polish up our ssh skills?

--
Chris

[TMnorah]

Yes, it is reasonable to restore the snapshot of a single file directly in the snapshot folder. As for the permissions and space occupied in file management, we will verify and discuss this issue.

@snapshot is just a soft link for users to view snapshot files. The version inside only gives read-only permission, so delete or cut operations are not allowed. The files in @snapshot do not take up storage space, even if you delete the capacity, there will be no change.

If you want to clear redundant snapshots, you can select a snapshot in the snapshot application, open the snapshot list to complete the deletion operation, and you can edit the snapshot task and set the snapshot retention policy to delete earlier snapshots.

[bidmead]

Once again, @TMnorah, thanks for that swift and very helpful response.

In the light of what you say, I've revisited my previous post here that opens with "Or can you perhaps do the following to restore individual files?" which suggests cut and pasting directly from within the @snapshot directory. I assert there that it can be done, permissions allowing.

Subsequent tests suggest that this is incorrect and that, as you say, files inside the @snapshot directory are all read-only, even for the admin and original creator. I apologise for misleading the forum. However...

I initially suggested restoring individual files from inside the target directory created when using the Snapshot app to restore the entire source snapshot directory. To recap: I began by copy and pasting, but this, of course, duplicates the file and unnecessarily takes up space. Then it appeared that CUT and pasting was possible, didn't result in duplication and in fact didn't delete the source snapshot file. Testing I've just carried out this morning seems to confirm that my earlier reporting on this was correct.

So although I was wrong to suggest that files can be cut and pasted directly from the @snapshot directory, it appears to be the case that the following procedure is a valid way of restoring individual snapshot files without losing additional disk space:

1. Use the Snapshot app to restore an entire directory snapshot to a new, empty target directory. There is no duplication of files in this process.
2. CUT and paste individual files from this new directory into wherever is to be their final destination.

It seems this process restores the original file without duplication and wasted space and without losing the original snapshot file. CAVEAT: The files restored to the Snapshot app's target directory are no longer read-only by default. But they may be uncuttable due to permissions on the original file prior to snapshotting.

I've written this up with some caution ("…it appears to be the case…"), for the obvious reason that I have already misreported here and in the hope that others in the forum will test this for themselves and shoot me down in flames if necessary.

I look forward to responding to any questions arising from this. Snapshots appear to me to be a very important asset for a Terramaster NAS and well worth the effort needed to understand the process.

--
Chris

[bidmead]

It would be invaluable if we could have a Terramaster response to this question. In summary:

The recommended procedure, using the Snapshot app for restoring a snapshot to a designated empty directory (the target directory) makes all the files in that restoration accessible to the File Manager. As only the metadata for those files have been restored, no additional space is taken up by file duplication.

There is no recommended procedure for restoring single files. However, you can of course copy individual files out of the target directory and then delete the target directory.

Copying any file from the restoration to elsewhere on the file system will, however, double the total space taken up by that file. Deleting the original target directory and all its contents will still leave duplicates of any files that have been copied because deleting the target directory leaves the original snapshots intact.

My question was: what happens if, instead of copying files out of the target directory, files are moved out of there to some other place in the same file system. A Unix move in these circumstances only changes the file's metadata, leaving it pointing to the original set of data blocks known to the target directory, the set controlled by the snapshot. Experiment suggests that moving a file from the target directory doesn't delete the file; however the move completes without signalling any error.

So the question is: Does this move, rather than copy restore the original file from the snapshot without duplication? And if so, would this be a recommended procedure, absent any official implementation of single file restoration within the Snapshot app?

--
Chris