How to repair a corrupted file system?

Hard Drive, volume, storage pool, RAID
User avatar
TMzethar
TerraMaster Team
Posts: 1127
Joined: 27 Oct 2020, 16:43

How to repair a corrupted file system?

Post by TMzethar »

Troubleshooting

Applicable models: All TNAS models

Applicable TOS version: TOS 4.2, TOS 5

In a few cases, the file system is at risk of corruption, such as abnormal shutdown, power failure, and hard disk failure. If this unfortunate thing happens, TOS may fail to load, storage pool and volume may fail to load, but your data may still have a chance to be rescued, please refer to the following methods to repair the faulty file system.

Disclaimer:
For the safety of your data, it is recommended that the repair needs to be operated by professionals. During the repair process, misoperation, power failure, and collision may cause data loss due to repair failure, your data still has a chance to be completely destroyed. If your data is exceptionally important, it is recommended that you rescue your data through a professional data recovery service. If you decide to carry out the repair operation, you need to bear all risks on your own.

Instruction
1. Log in your TNAS with SSH. How to log in your TNAS with SSH terminal?
2. Execute the following command to view the system log:

Code: Select all

dmesg
If you find an obvious red font mark or a message with the words like "group descriptors corrupted!","Ext4-fs error", as shown in the figure below, it means that there must be an abnormality in the EXT4 file system, please use the "General Repair Method of Ext4 File System".
(If you do not find similar errors, it does not necessarily mean that there are no errors at all. It may be that the error was not triggered during this startup)
Image

Image

If you find an obvious red font mark or a message with the words "Btrfs error", as shown in the figure below, it means that there must be an abnormality in the BTRFS file system, please use the "General BTRFS file system repair method".
(If you do not find similar errors, it does not necessarily mean that there are no errors at all. It may be that the error was not triggered during this startup)
Image

The general repair method of EXT4 file system
Note: For step 3-step 7, if your TNAS has used the early TOS 4.0 or even TOS 3.x. Since LVM management was not adopted at that time, there may not be a directory like "/dev/mapper/vg0-lv0". Please use "/dev/md0" instead.

1. Before performing the repair, please disable the SSD Hyper Cache, and you need to execute the following instructions to unmount the current file system:

Code: Select all

cd /
umount  /mnt/md0 
umount  /home
(At TOS 5 it should be "/Volume1" instead of "/mnt/md0")
2. If the prompt "Target is busy", Please cancel the occupancy of the volume by the service processes, disable the SSD cache (if no SSD cache, please ignore it), and then execute the command above again.

Code: Select all

fuser -mk /mnt/md0
If the root user is automatically logged out, please log in to SSH again and switch to the root user, then go back to step 1.
(At TOS 5 it should be "/Volume1" instead of "/mnt/md0")

3. Execute the repair instruction and wait for the repair to complete:

Code: Select all

e2fsck -p /dev/mapper/vg0-lv0
4. After the repair is complete, execute the following command to remount the file system:

Code: Select all

/etc/tos/scripts/mntdata
5. Refresh the TOS login page and log in again.
6. If the abnormality still exists, you can repeat steps 1 and 2 and execute follow:

Code: Select all

e2fsck -b 32768 /dev/mapper/vg0-lv0 -y
After the repair is complete, repeat steps 4 and 5.
7. If the abnormality still exists, you can repeat steps 1 and 2 and execute the mandatory repair instruction (risky! Please be cautious):

Code: Select all

e2fsck -f /dev/mapper/vg0-lv0 -y
After the repair is complete, repeat steps 4 and 5.



The conventional repair method of BTRFS file system
Note: For step 3-step 4, if your TNAS has used the early TOS 4.0 or even TOS 3.x. Since LVM management was not adopted at that time, there may not be a directory like "/dev/mapper/vg0-lv0". Please use "/dev/md0" instead.
Note: BTRFS is not perfect in 210 series model who uses an earlier linux kernel versions.
If your device model is F2-210 or F4-210, We recommend that you back up your data and then rebuild your BTRFS volumes to more stable EXT4 volumes.


1. Before performing the repair, please disable the SSD Hyper Cache, and you need to execute the following instructions to unmount the current file system:

Code: Select all

cd /
umount  /mnt/md0 
umount  /home
(At TOS 5 it should be "/Volume1" instead of "/mnt/md0")

2. If the prompt "Target is busy", Please cancel the occupancy of the volume by the service processes, disable the SSD cache (if no SSD cache, please ignore it), and then execute the command above again.

Code: Select all

fuser -mk /mnt/md0
If the root user is automatically logged out, please log in to SSH again and switch to the root user, then go back to step 1.
(At TOS 5 it should be "/Volume1" instead of "/mnt/md0")

3. Execute the following command to detect the BTRFS file system:

Code: Select all

btrfs check /dev/mapper/vg0-lv0
Execute the following command to check the BTRFS file system. If the check takes a long time or an obvious error message is found, please wait patiently for the check to complete
Image

4. After the detection is completed, execute the following instructions to restore the file system:

Code: Select all

btrfs check --repair /dev/mapper/vg0-lv0
Image

5. As shown in the figure above, if it prompts "10 9 8 7 6 5 4 3 2 1", it means that the probability of file system recovery is very high. Just wait for the end of the recovery process and execute the following command to remount the file system:

Code: Select all

/etc/tos/scripts/mntdata
6. Refresh the TOS login page and log in again.
Last edited by TMzethar on 13 Nov 2023, 16:08, edited 1 time in total.
To contact our team, please send email to following addresses, remember to replace (at) with @:
Technical team: support(at)terra-master.com (for technical support)
Service team: service(at)terra-master.com (for purchasing, return, replacement, RMA service)
User avatar
ElizabethButner
Posts: 1
Joined: 25 Oct 2021, 19:37

Re: Conventional Repair Method of Btrfs/Ext4 Volume File System

Post by ElizabethButner »

Wow you describe everything so detailed and cool!!!
Roccia7
Posts: 62
Joined: 05 Mar 2020, 05:02
Italy

Re: Conventional Repair Method of BTRFS/EXT4 Volume File System

Post by Roccia7 »

Hi, i'm trying to repair a EXT4 error in according with terramaster support. But when i'm at step 3 i recieve this error
No such file or directory while trying to open /dev/mapper/vg0-lv0
Possibly non-existent device?
Could you help me?
I have f2-220 and tos 4.2.18
User avatar
TMzethar
TerraMaster Team
Posts: 1127
Joined: 27 Oct 2020, 16:43

Re: Conventional Repair Method of BTRFS/EXT4 Volume File System

Post by TMzethar »

{L_BUTTON_AT}Roccia7
Please execute the following instruction and show us the result of execution:

Code: Select all

blkid
To contact our team, please send email to following addresses, remember to replace (at) with @:
Technical team: support(at)terra-master.com (for technical support)
Service team: service(at)terra-master.com (for purchasing, return, replacement, RMA service)
Roccia7
Posts: 62
Joined: 05 Mar 2020, 05:02
Italy

Re: Conventional Repair Method of BTRFS/EXT4 Volume File System

Post by Roccia7 »

{L_BUTTON_AT}Roccia7
TMzethar wrote:
> >
> Please execute the following instruction and show us the result of
> execution:
> [code]blkid[/code]
Here it is

/dev/md0: UUID="e9c52ac8-424c-457c-a830-1353f538b066" TYPE="ext4"
/dev/md8: UUID="1c4f0f17-3ab0-42bd-a5f2-dd20070a38d2" TYPE="swap"
/dev/md9: UUID="e244cdcf-6457-4943-b89f-1967411be28d" TYPE="ext4"
/dev/sda1: LABEL="UTOSDISK-X86-S64" UUID="193afa84-78ee-44a4-8310-a73048e0f767" TYPE="ext4" PARTUUID="c6a664ef-a5c7-411c-8006-e4cbec7804e5"
/dev/sda2: UUID="8b2d41cc-b12b-d1a5-319b-a500be790636" UUID_SUB="163213d8-86df-0cef-8151-a93124064e48" LABEL="TNAS:UTOSCORE-X86-S64" TYPE="linux_raid_member" PARTUUID="5d06d8e9-48ca-4652-91b6-48d8b2917b21"
/dev/sda3: UUID="60830f29-a249-12d5-21fb-2c5c169164b5" UUID_SUB="74db1a00-ae3d-704c-425e-1a3c317411a1" LABEL="TNAS-2882:UTOSSWAP-X86-S64" TYPE="linux_raid_member" PARTUUID="738101e1-8179-40fa-a434-8c94edd1c7d0"
/dev/sda4: UUID="491bd056-a6bb-2fc8-d74a-7af2b2525f4d" UUID_SUB="44f1c5f6-0dfe-f9fb-8460-c2ce7ecbb0ba" LABEL="TNAS-2882:UTOSUSER-X86-S64" TYPE="linux_raid_member" PARTUUID="7cff2639-1659-4593-8592-c150ff9b2db0"
User avatar
TMzethar
TerraMaster Team
Posts: 1127
Joined: 27 Oct 2020, 16:43

Re: Conventional Repair Method of BTRFS/EXT4 Volume File System

Post by TMzethar »

{L_BUTTON_AT}Roccia7

It can be seen that your volume was created in a very old TOS, and its storage configuration is different from the current TOS.
When executing instructions, please replace "/dev/mapper/vg0-lv0" with "/dev/md0".
To contact our team, please send email to following addresses, remember to replace (at) with @:
Technical team: support(at)terra-master.com (for technical support)
Service team: service(at)terra-master.com (for purchasing, return, replacement, RMA service)
Roccia7
Posts: 62
Joined: 05 Mar 2020, 05:02
Italy

Re: Conventional Repair Method of BTRFS/EXT4 Volume File System

Post by Roccia7 »

{L_BUTTON_AT}Roccia7
TMzethar wrote:
> >
> It can be seen that your volume was created in a very old TOS, and its
> storage configuration is different from the current TOS.
> When executing instructions, please replace "/dev/mapper/vg0-lv0"
> with "/dev/md0".
Thanks very much! yes, my first tos was somthing like 3.x
Roccia7
Posts: 62
Joined: 05 Mar 2020, 05:02
Italy

Re: Conventional Repair Method of BTRFS/EXT4 Volume File System

Post by Roccia7 »

Ok i tried but i had another error
/dev/md0: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
What should I do?
Roccia7
Posts: 62
Joined: 05 Mar 2020, 05:02
Italy

Re: Conventional Repair Method of BTRFS/EXT4 Volume File System

Post by Roccia7 »

Sorry for crossposting. I tried e2fsck whitout -p and fix the error, so i tried dmesg and no error are shown. When i looked if my problem was fixed i see that nothing changed. My problem still there. So i tried whit step 7 but nothing again. I described my problem here.

viewtopic.php?f=43&t=2793

Is there something else that i can do?
Cis
Posts: 19
Joined: 13 Apr 2020, 20:30
France

Re: Conventional Repair Method of BTRFS/EXT4 Volume File System

Post by Cis »

Hi,

I struggle a bit with the procedure but I am wondering if I ever need to go through it as I dont have obvious RED error notified in the output of dmesg command.
Below are the 'error' filtered output, do you think my file system is corrupted?

"[root@TNAS-002C6C ~]# dmesg |grep error
[ 1.011247] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[ 8.003476] r8169 0000:02:00.0: Direct firmware load for rtl_nic/rtl8168e-3.fw failed with error -2
[ 37.867079] ata1.00: error: { UNC }
[ 37.868913] sd 1:0:0:0: [sda] tag#16 Add. Sense: Unrecovered read error - auto reallocate failed
[ 37.868941] print_req_error: I/O error, dev sda, sector 2004696
[ 693.272816] EXT4-fs (md0): error count since last fsck: 2
[ 693.272833] EXT4-fs (md0): initial error at time 1638485307: ext4_mb_generate_buddy:756
[ 693.272842] EXT4-fs (md0): last error at time 1641223785: ext4_mb_generate_buddy:747
[ 9942.331002] ata2.00: error: { UNC }
[ 9942.332203] sd 2:0:0:0: [sdb] tag#8 Add. Sense: Unrecovered read error - auto reallocate failed
[ 9942.332230] print_req_error: I/O error, dev sdb, sector 2670232904
[10617.291862] ata2.00: error: { UNC }
[10617.293153] sd 2:0:0:0: [sdb] tag#1 Add. Sense: Unrecovered read error - auto reallocate failed
[10617.293179] print_req_error: I/O error, dev sdb, sector 3391653256
[88418.020526] EXT4-fs (md0): error count since last fsck: 2
[88418.020548] EXT4-fs (md0): initial error at time 1638485307: ext4_mb_generate_buddy:756
[88418.020562] EXT4-fs (md0): last error at time 1641223785: ext4_mb_generate_buddy:747"


It looks like it with the bottom count of 2 error on ext4, but since I struggle later on the procedure I got doubts.
Post Reply