If data integrity is important, is TFM Backup enough or is rsync better?

[terrablaster]

I'm looking at backups between NAS and NAS.

Terramaster has a video on NAS to NAS with TFM backup (https://www.youtube.com/watch?v=B-FzVJZN2QM), but I wonder if it's using some SHA verification of files like rsync copying or does it just do a basic linux `cp`? I feel this is okay, but is this great for data integrity between NAS?

TL;DR TFM Backup is easy, but does Rsync backup do a better job for verification of data if data integrity is important?

Thanks

[OrionXie]

I think this comprehensive guide on backup tools might answer your questions: viewtopic.php?t=6093

[terrablaster]

Thanks OrionXie,

I did run into this a few days ago but I still don't know if it goes into detail well enough. Like underneath the hood, what is TFM backup doing as a command? Is it just copying files, is there integrity checking ... or something else? I'm just hoping to understand better.

Thanks

[Gremlin]

I use TFM as it works best for my needs. If I were 'brave' enough I would try rsync as a command line running as a custom scheduled task. I believe this would offer more options for the transfer process that are not available with the built-in backup routines.

[terrablaster]

Thanks Gremlin,

Yeah I did an initial backup with TFM backup and at least byte to byte it seems fine, but this was one time and what's really important is integrity when it matters.

According to ChatGPT, Rsync backup through TOS also doesn't do checksum validation (not sure if true, taking its word).

[RyanYang]

Thanks Gremlin,
Yeah I did an initial backup with TFM backup and at least byte to byte it seems fine, but this was one time and what's really important is integrity when it matters.
According to ChatGPT, Rsync backup through TOS also doesn't do checksum validation (not sure if true, taking its word).

Our system does perform checksum verification for backup results. During our laboratory testing procedures, we specifically verify file integrity by checking MD5 values to ensure data consistency.

[Gremlin]

-c, --checksum
This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a "quick check" that (by default) checks if each file's size and time of last modification match between the sender and receiver. This option changes this to compare a 128-bit checksum for each file that has a matching size. Generating the checksums means that both sides will expend a lot of disk I/O reading all the data in the files in the transfer (and this is prior to any reading that will be done to transfer changed files), so this can slow things down significantly.
The sending side generates its checksums while it is doing the file-system
scan that builds the list of the available files. The receiver generates its checksums when it is scanning for changed files, and will checksum any file that has the same size as the corresponding sender's file: files with either a changed size or a changed checksum are selected for transfer.
Note that rsync always verifies that each transferred file was
correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred, but that automatic after-the-transfer verification has nothing to do with this option's before-the-transfer "Does this file need to be updated?" check.
For protocol 30 and beyond (first supported in 3.0.0), the checksum used is
MD5. For older protocols, the checksum used is MD4.

However, it seems that for every post on the web expressing the truth of the above, there is a counter post decrying it. I have highlighted the Note for clarity as it seems to be the important bit (pun intended). This appears to say that during EVERY transfer an internal checksum is calculated on the data being transferred (without reference to the original file) and that the checksum is verified with the data received after the transfer. What it does not say is that the final write is verified. That appears to be left to the integrity of the filesystem. Hence, in any event, it would be necessary to perform a separate, independent verification between original and final files for absolute confirmation.

Perhaps @RyanYang could provide some clarity

[RyanYang]

I'm not entirely clear about the specific details, so I'll report it to the relevant project team.

[terrablaster]

Our system does perform checksum verification for backup results.

RyanYang, just to be clear, is this with regards to TFM Backup or were you referring to Rsync Backup?

During our laboratory testing procedures, we specifically verify file integrity by checking MD5 values to ensure data consistency.

So was this human testing, or was the system checking with copying? The way I read your message it's human's verified, but maybe I read it wrong. Just want to be clear on this.

[RyanYang]

After the holiday, I will consult with both project teams to gain a thorough understanding of their verification mechanisms.Currently, I'm aware that manual data integrity verification is performed post-backup. As for the automated system verification process, I'll need to investigate further.