[Help] BTRFS Error

[Gremlin]

Situation:
Remote mounted folder. {For TFM Backup on F2-221)
Device F2-424
2x500GB nvme Traid Pool1/Vol1
2x6TB Hdd Traid Pool2/Vol2
Remote mount is to shared volume on Vol2.
During TFM backup console error

BTRFS error (device dm-1): hdev /dev/mapper/vg1-lv1 errs: wr0, rd0, flush 0, corrupt nnn, gen 0

where nnn == 404 - 412; 11 messages}
TFM backup failed
Don't see any other obvious errors in GUI. Disks appear OK, system appears to be functioning.

{The other day I think I had a power issue through using incorrect power supply (chasing down another issue with @TMSupport)
ata1/ata2 comreset failure (or something like that)}

Any suggestions to fix the BTRFS error would be appreciated.

[Gremlin]

Further to my earlier, I have a bit more info from kernel log:

Code: Select all

Jul  6 10:57:57 NAS-T1 1 2025-07-06T10:57:57+01:00 - - TOSDaemon(4070) - [attrs topic="points_temp_1"] scraper started
Jul  6 10:57:57 NAS-T1 1 2025-07-06T10:57:57+01:00 - - TOSDaemon(4070) - [attrs topic="points_temp_7"] scraper started
Jul  6 10:57:57 NAS-T1 1 2025-07-06T10:57:57+01:00 - - TOSDaemon(4070) - [attrs topic="points_temp_30"] scraper started
Jul  6 12:59:54 NAS-T1 kernel: [ 7379.757581] BTRFS warning (device dm-1): csum failed root 344 ino 1743 off 24709693440 csum 0x75e8343f expected csum 0x988753e6 mirror 1
Jul  6 12:59:54 NAS-T1 kernel: [ 7379.757587] BTRFS error (device dm-1): bdev /dev/mapper/vg1-lv1 errs: wr 0, rd 0, flush 0, corrupt 403, gen 0
Jul  6 12:59:54 NAS-T1 kernel: [ 7379.757617] BTRFS warning (device dm-1): csum failed root 344 ino 1743 off 24709697536 csum 0xb9e9d05a expected csum 0x34aafe1d mirror 1
Jul  6 12:59:54 NAS-T1 kernel: [ 7379.757620] BTRFS error (device dm-1): bdev /dev/mapper/vg1-lv1 errs: wr 0, rd 0, flush 0, corrupt 404, gen 0
Jul  6 12:59:54 NAS-T1 kernel: [ 7379.757643] BTRFS warning (device dm-1): csum failed root 344 ino 1743 off 24709701632 csum 0x434e1137 expected csum 0x3b47459d mirror 1
Jul  6 12:59:54 NAS-T1 kernel: [ 7379.757645] BTRFS error (device dm-1): bdev /dev/mapper/vg1-lv1 errs: wr 0, rd 0, flush 0, corrupt 405, gen 0
Jul  6 12:59:54 NAS-T1 kernel: [ 7379.757670] BTRFS warning (device dm-1): csum failed root 344 ino 1743 off 24709705728 csum 0x5c262e8f expected csum 0x2eeec498 mirror 1
Jul  6 12:59:54 NAS-T1 kernel: [ 7379.757671] BTRFS error (device dm-1): bdev /dev/mapper/vg1-lv1 errs: wr 0, rd 0, flush 0, corrupt 406, gen 0
Jul  6 12:59:54 NAS-T1 kernel: [ 7379.757690] BTRFS warning (device dm-1): csum failed root 344 ino 1743 off 24709709824 csum 0x443a85bd expected csum 0x22f322c8 mirror 1

{abbreviated due to forum post constraints}

A subsequent TFM backup was partially sucessfull; bacikng up a newer file and failing on an older file.

[TMbuddy]

Based on your description, we believe the issue may be due to your previous use of an incorrect power supply, which could have caused interruptions during write operations or disk I/O anomalies, leading to partial data corruption. This may be why the backup for that portion of the data failed.

If your file system is still mountable and accessible, we strongly recommend copying and backing up your data as soon as possible while it is still readable.

[Gremlin]

Is there any mechanism for determining which HDD(s) may have been affected. Given that I am still able to read/write to the Traid I would guess that one mirror is still intact - or is that hoping too much?

During my research I have noted advice not to use BTRFS with LVM (or vice versa) but I'm not sure if that is current. Would it be better to stick with Ext4 file system and go back to plain raid rather than the (somewhat nebulous) Traid options?

[CursaYang]

You can use the SMART tool to check the hard drive. The two hard drives configured in TRAID are similar to RAID 1, and the ability to read and write indicates that the redundancy mechanism is working, which is good news.

In the past (many years ago), there were indeed reports about stability and compatibility issues with BTRFS and LVM stacking, mainly involving snapshot interactions, space management conflicts, or the risk of deadlocks in specific error handling paths.

In newer kernels (such as the 5.x and 6.x series) and stable BTRFS versions, the stability of this combination has greatly improved. Many users have successfully used the LVM + BTRFS combination in production environments, especially when the flexibility offered by LVM for volume management (such as more easily adjustable physical volumes and snapshot integration into backup processes) is required.

[Gremlin]

You can use the SMART tool to check the hard drive. The two hard drives configured in TRAID are similar to RAID 1, and the ability to read and write indicates that the redundancy mechanism is working, which is good news.

Yes, I can use the Smart Tool, but I'm not sure it is answering my question. How will this give me a clue as to which hard drive is corrupted?
FYI I have run short test on both drives and, as far as I can tell, nothing shows as a problem. The following values are all '0':
1, 5, 7, 10, 192, 198, 199
on both drives.
I am currently running bad block, but that will obviously take some time (2 x 6TB drives). My ultimate objective would be to understand where the 'defect' might lie and replace the hard drives (one at a time - starting with any "bad" drive).
Or, is the system trying to tell me that the raid itself is corrupt?
And there is no way to fix it?
Note that the "damaged" nas still continues to accept backups to the same volume/shared folder from 3rd party software without issue.

[Gremlin]

Follow up:
Zero bad blocks for both hdd.

Code: Select all

Filesystem           Size  Used Avail Use% Mounted on
/dev/md9             7.5G  2.3G  4.8G  33% /
tmpfs                 50M   28K   50M   1% /dev/shm
tmpfs                1.6G   21M  1.5G   2% /run
tmpfs                5.0M     0  5.0M   0% /run/lock
tmpfs                512M   58M  455M  12% /tmp
/dev/mapper/vg0-lv0  456G   69G  386G  16% /var/subvols/8vEbTxkKvwa
/dev/mapper/vg0-lv0  456G   69G  386G  16% /Volume1
/dev/mapper/vg1-lv1  5.5T  1.6T  3.9T  29% /var/subvols/8vEbTxkKvwb
/dev/mapper/vg1-lv1  5.5T  1.6T  3.9T  29% /Volume2
/dev/mapper/vg0-lv0  456G   69G  386G  16% /home

[CursaYang]

A Btrfs error isn't necessarily caused by a fault in the hard drive itself; as previously mentioned, unexpected power loss can also lead to file system damage.
If you determine that the hard drive is healthy, you can back up your data and then rebuild the storage pool and volume.

[Gremlin]

BTRFS with raid is widely touted as being a "self-healing file system". Well, obviously, it isn't in this case (and I seriously wonder about ANY case.) In the raid system (mirror in this case) there are only 2 copies of the data. If one is damaged, how would the system know which was the "correct" data to use. Where are the redundancies in the BTRFS system as implemented in TOS (whatever version)?

If, as I seem to recall being posted once upon a time, "it's not implemented" what is the point of pursuing "the future"? Can't help feeling I (and many others) would be better of sticking with tried and trusted tech where we know the limitations.

Back to the here and now - Is there a method for scanning btrfs to determine and identify where a corruption may exist? I am sure the designers intended those error messages to be useful in some manner. It is only by luck that I had set up my routines to read particular files where the corruption was apparent. Must be a way to find what has been missed, surely?

"Oh, it's corrupted - you'll have to backup and rebuild" does not seem a satisfactory response - but we see it all to often in these pages.

[CursaYang]

The commands in this post also apply to TOS 6: viewtopic.php?t=2575

You can first use 'btrfs check' to detect the current file system, and then use 'btrfs repair' to fix the file system.