[Help] F5-221 is doing sporadic reloads

Initialization of newly purchased TNAS or re-installation of your TNAS
User avatar
AlAleksiev
Posts: 44
Joined: 19 Feb 2024, 00:15
Canada

[Help] F5-221 is doing sporadic reloads

Post by AlAleksiev »

Hi guys,

That problem started about a month ago.

At this time the device was running old TOS 4.2.30-2203011626
Few apps were going along the normal storage services:
  • Plex 1.29.0.624
  • MegaDisk Sync
  • Clam AntiVirus
  • Docker
On the docker there were a few containers:
  • sftpgo
  • jellyfin
  • proxy.py
  • pi-hole
  • unifi-controller
All of the above was running fine for almost 3 years.

I contacted the support and was advised to update the system. So, I did it - full backup of the storage.
Updated the bootloader (on the USB), the boot system (from 2 to 8 GB partition) and finally new raid (TRAID+) for the storage with the latest and greatest TOS 6.0.794. Finally the data was recovered from the backup (USB Copy used)

Unforrrtunately the problem is still presented and it's sporadic. Sometimes the NAS can work for a day or two and after that to reload 4-5 times per day.
There are nothing fancy, so far, on the NAS - just:
  • the shares (for smb and nfs)
  • the USB Copy
  • I enabled Docker, but no containers yet
I started to run everytime this command - to collect some information jsut before the reload:

Code: Select all

dmesg -w>/Volume1/public/dmesg.txt &
When comparing the last lines in the log, I mentioned that this part is repeated always in the end:

Code: Select all

... more output before that last section ...
[   35.767972] r8169 0000:02:00.0 eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[   39.298028] scsi 6:0:0:0: Direct-Access     VendorCo ProductCode      2.00 PQ: 0 ANSI: 4
[   39.298806] sd 6:0:0:0: Attached scsi generic sg5 type 0
[   39.298934] [664]: scst: scst_register_device:1089:Attached to scsi6, channel 0, id 0, lun 0, type 0
[   39.299061] sd 6:0:0:0: [sdf] 491304 512-byte logical blocks: (252 MB/240 MiB)
[   39.299266] sd 6:0:0:0: [sdf] Write Protect is off
[   39.299277] sd 6:0:0:0: [sdf] Mode Sense: 03 00 00 00
[   39.299485] sd 6:0:0:0: [sdf] No Caching mode page found
[   39.299494] sd 6:0:0:0: [sdf] Assuming drive cache: write through
[   39.309802]  sdf: sdf1
[   39.310351] sd 6:0:0:0: [sdf] Attached SCSI removable disk
[   40.958959] BTRFS: device label TOS_VOL_20251211 devid 1 transid 311 /dev/dm-1 scanned by systemd-udevd (449)
[   41.118526] BTRFS: device label TOS_VOL_20251211 devid 1 transid 2036 /dev/dm-0 scanned by systemd-udevd (468)
[   41.233397] r8169 0000:02:00.0 eth0: Link is Down
[   41.261521] bond0: (slave eth1): link status definitely up, 1000 Mbps full duplex
[   41.261546] bond0: (slave eth1): making interface the new active one
[   41.269298] bond0: active interface up!
[   43.889753] [3554]: scst: scst_unregister_device:1164:Detached from scsi6, channel 0, id 0, lun 0, type 0
[   45.264636] r8169 0000:02:00.0 eth0: Link is Up - 1Gbps/Full - flow control off
[   45.328216] bond0: (slave eth0): link status definitely up, 1000 Mbps full duplex
[   45.873932] BTRFS info (device dm-0): first mount of filesystem 2020a685-9a51-4ed9-9647-9c41914ef10a
[   45.873966] BTRFS info (device dm-0): using crc32c (crc32c-generic) checksum algorithm
[   45.874041] BTRFS info (device dm-0): using free space tree
[   52.946605] BTRFS info (device dm-1): first mount of filesystem d347a600-eab2-4700-ad6f-f0274c2589a6
[   52.946643] BTRFS info (device dm-1): using crc32c (crc32c-generic) checksum algorithm
[   52.946690] BTRFS info (device dm-1): using free space tree
[   63.586185] /dev/sdf1: Can't open blockdev
[   67.835936] NFSD: Using nfsdcld client tracking operations.
[   67.835950] NFSD: no clients to reclaim, skipping NFSv4 grace period (net f0000000)
[   72.569349] NFSD: Using nfsdcld client tracking operations.
[   72.569364] NFSD: no clients to reclaim, skipping NFSv4 grace period (net f0000000)
... device is reloading here ...
There is no sdf (as detected in the above log) on that box at all. That's all block devices, currently:

Code: Select all

alex@TNAS-ALEX:~$ sudo lsblk
[sudo] password for alex:
NAME          MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sda             8:0    0  3.6T  0 disk
├─sda1          8:1    0  285M  0 part
├─sda2          8:2    0  7.6G  0 part
│ └─md9         9:9    0  7.6G  0 raid1 /
├─sda3          8:3    0  1.9G  0 part
│ └─md8         9:8    0  1.9G  0 raid1 [SWAP]
└─sda4          8:4    0  3.6T  0 part
  └─md0         9:0    0 10.9T  0 raid6
    ├─vg0-lv0 252:0    0    9T  0 lvm   /home
    │                                   /Volume1
    │                                   /var/subvols/8vEbTxkKvwa
    └─vg0-lv1 252:1    0  1.9T  0 lvm   /Volume2
                                        /var/subvols/8vEbTxkKvwb
sdb             8:16   0  3.6T  0 disk
├─sdb1          8:17   0  285M  0 part
├─sdb2          8:18   0  7.6G  0 part
│ └─md9         9:9    0  7.6G  0 raid1 /
├─sdb3          8:19   0  1.9G  0 part
│ └─md8         9:8    0  1.9G  0 raid1 [SWAP]
└─sdb4          8:20   0  3.6T  0 part
  └─md0         9:0    0 10.9T  0 raid6
    ├─vg0-lv0 252:0    0    9T  0 lvm   /home
    │                                   /Volume1
    │                                   /var/subvols/8vEbTxkKvwa
    └─vg0-lv1 252:1    0  1.9T  0 lvm   /Volume2
                                        /var/subvols/8vEbTxkKvwb
sdc             8:32   0  3.6T  0 disk
├─sdc1          8:33   0  285M  0 part
├─sdc2          8:34   0  7.6G  0 part
├─sdc3          8:35   0  1.9G  0 part
└─sdc4          8:36   0  3.6T  0 part
  └─md0         9:0    0 10.9T  0 raid6
    ├─vg0-lv0 252:0    0    9T  0 lvm   /home
    │                                   /Volume1
    │                                   /var/subvols/8vEbTxkKvwa
    └─vg0-lv1 252:1    0  1.9T  0 lvm   /Volume2
                                        /var/subvols/8vEbTxkKvwb
sdd             8:48   0  3.6T  0 disk
├─sdd1          8:49   0  285M  0 part
├─sdd2          8:50   0  7.6G  0 part
├─sdd3          8:51   0  1.9G  0 part
└─sdd4          8:52   0  3.6T  0 part
  └─md0         9:0    0 10.9T  0 raid6
    ├─vg0-lv0 252:0    0    9T  0 lvm   /home
    │                                   /Volume1
    │                                   /var/subvols/8vEbTxkKvwa
    └─vg0-lv1 252:1    0  1.9T  0 lvm   /Volume2
                                        /var/subvols/8vEbTxkKvwb
sde             8:64   0  3.6T  0 disk
├─sde1          8:65   0  285M  0 part
├─sde2          8:66   0  7.6G  0 part
├─sde3          8:67   0  1.9G  0 part
└─sde4          8:68   0  3.6T  0 part
  └─md0         9:0    0 10.9T  0 raid6
    ├─vg0-lv0 252:0    0    9T  0 lvm   /home
    │                                   /Volume1
    │                                   /var/subvols/8vEbTxkKvwa
    └─vg0-lv1 252:1    0  1.9T  0 lvm   /Volume2
                                        /var/subvols/8vEbTxkKvwb
nbd0           43:0    0    0B  0 disk
nbd1           43:32   0    0B  0 disk
nbd2           43:64   0    0B  0 disk
nbd3           43:96   0    0B  0 disk
nbd4           43:128  0    0B  0 disk
nbd5           43:160  0    0B  0 disk
nbd6           43:192  0    0B  0 disk
nbd7           43:224  0    0B  0 disk
nbd8           43:256  0    0B  0 disk
nbd9           43:288  0    0B  0 disk
nbd10          43:320  0    0B  0 disk
nbd11          43:352  0    0B  0 disk
nbd12          43:384  0    0B  0 disk
nbd13          43:416  0    0B  0 disk
nbd14          43:448  0    0B  0 disk
nbd15          43:480  0    0B  0 disk
alex@TNAS-ALEX:~$ sudo cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty]
md0 : active raid6 sdd4[0] sdc4[4] sdb4[3] sda4[2] sde4[1]
      11687473152 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/5] [UUUUU]

md8 : active raid1 sdb3[0] sda3[1]
      1997824 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md9 : active raid1 sdb2[2] sda2[1]
      7995392 blocks super 1.2 [2/2] [UU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>
alex@TNAS-ALEX:~$
Anybody with idea what can I do to troubleshoot and isolate the problem?
Thank you in advance,

Alex
______________________
F5-221; 10G RAM; 5 x 4TB TRAID; Riser, Power and/or ASMedia HW failure undetected by TM Support
F6-424; 32G RAM; 2 x 512GB 8Gb/s M.2; 6 x 4TB raidz2; TrueNAS 25.10.1
¯¯¯¯¯¯ Jellyfin, Pi-Hole, Unifi Controller, Zabbix
User avatar
AlAleksiev
Posts: 44
Joined: 19 Feb 2024, 00:15
Canada

Re: [Help] F5-221 is doing sporadic reloads

Post by AlAleksiev »

This morning NAS took multiple reloads and in the end the Storage Pool (raid) was completely lost.
Few screenshots and photos to descride the conditions:

This is from the console:
Image

The initial login screen:
Image

The SMART and RAID verification from the CLI:
Image

I started "/etc/tos/scripts/mntdata" script and it produced this output:

Code: Select all

******@TNAS-ALEX:~# /etc/tos/scripts/mntdata
[DEBUG /usr/sbin/mdadm -E /dev/sdc2]
2025/12/14 12:38:56 mdadm: No md superblock detected on /dev/sdc2.
: exit status 1
[DEBUG /usr/sbin/mdadm -E /dev/sdc3]
2025/12/14 12:38:56 mdadm: No md superblock detected on /dev/sdc3.
: exit status 1
[DEBUG /usr/sbin/mdadm -E /dev/sdd2]
2025/12/14 12:38:56 mdadm: No md superblock detected on /dev/sdd2.
: exit status 1
[DEBUG /usr/sbin/mdadm -E /dev/sdd3]
2025/12/14 12:38:56 mdadm: No md superblock detected on /dev/sdd3.
: exit status 1
[DEBUG /usr/sbin/mdadm -E /dev/sde2]
2025/12/14 12:38:56 mdadm: No md superblock detected on /dev/sde2.
: exit status 1
[DEBUG /usr/sbin/mdadm -E /dev/sde3]
2025/12/14 12:38:56 mdadm: No md superblock detected on /dev/sde3.
: exit status 1
2025-12-14 12:38:57.623 [INFO] /usr/sbin/mdadm --grow --bitmap=internal /dev/md9
2025-12-14 12:38:57.626 [INFO] [mntdata-initCore]set internal bitmap for core raid(active).Err:mdadm: bitmap already present on /dev/md9

[DEBUG /usr/sbin/mdadm --re-add /dev/md9 /dev/sda2]
[DEBUG /usr/sbin/mdadm --re-add /dev/md9 /dev/sdb2]
2025-12-14 12:38:57.631 [INFO] [mntdata-initCore]get Cores:[{2025-12-14 12:37:45 +0000 UTC /dev/sda2 Active device 1 false fc525fe0:66fa7966:46f74d4e:0e6f8d8d /dev/sda} {2025-12-14 12:37:45 +0000 UTC /dev/sdb2 Active device 0 false fc525fe0:66fa7966:46f74d4e:0e6f8d8d /dev/sdb}]
2025-12-14 12:38:57.631 [INFO] [mntdata-initSwap]get init SwapMember:[{2025-12-14 10:40:06 +0000 UTC /dev/sda3 Active device 1 false 46e35f5b:a72dc66d:099650ff:af576c34 /dev/sda} {2025-12-14 10:40:06 +0000 UTC /dev/sdb3 Active device 0 false 46e35f5b:a72dc66d:099650ff:af576c34 /dev/sdb}]
[DEBUG /usr/sbin/mdadm -A /dev/md8 /dev/sda3 -f]
[DEBUG mdadm: /dev/md8 has been started with 1 drive (out of 2).
]
2025-12-14 12:38:57.682 [INFO] /usr/sbin/mdadm --grow --bitmap=internal /dev/md8
2025-12-14 12:38:57.684 [INFO] [mntdata-initSwap]set internal bitmap for swap raid(active).Err:mdadm: bitmap already present on /dev/md8

[DEBUG /usr/sbin/mdadm --re-add /dev/md8 /dev/sda3]
[DEBUG /usr/sbin/mdadm --add /dev/md8 /dev/sdb3]
[DEBUG mdadm: re-added /dev/sdb3
]
[DEBUG /usr/sbin/mkswap -f /dev/md8]
[DEBUG mkswap: /dev/md8: insecure permissions 0666, fix with: chmod 0660 /dev/md8
mkswap: /dev/md8: warning: wiping old swap signature.
]
[DEBUG /usr/sbin/swapon /dev/md8]
[DEBUG swapon: /dev/md8: insecure permissions 0666, 0660 suggested.
]
2025-12-14 12:38:57.988 [INFO] getCacheRaidInfo successes::  {}
[DEBUG /usr/sbin/mdadm -A /dev/md0 /dev/sda4 /dev/sdb4 /dev/sdc4 /dev/sdd4 /dev/sde4 --force]
[DEBUG mdadm: /dev/md0 has been started with 5 drives.
]
[DEBUG /usr/sbin/mdadm --re-add /dev/md0 /dev/sda4]
[DEBUG /usr/sbin/mdadm --re-add /dev/md0 /dev/sdb4]
[DEBUG /usr/sbin/mdadm --re-add /dev/md0 /dev/sdc4]
[DEBUG /usr/sbin/mdadm --re-add /dev/md0 /dev/sdd4]
[DEBUG /usr/sbin/mdadm --re-add /dev/md0 /dev/sde4]
[DEBUG /usr/sbin/iniparse -a -f /etc/volume/volume.conf -s 00000001-2512-1123-5421-b66bec1a6981 -system]
mountPath =  /Volume1
check filesystem on /dev/mapper/vg0-lv0
2025-12-14 12:39:02.050 [INFO] changeBtrfsSubvol::/dev/mapper/vg0-lv0 btrfs subvolume @ is Exists.
/etc/tos/scripts/mntVolume /dev/mapper/vg0-lv0
[DEBUG /bin/bash -c /etc/tos/scripts/mntVolume /dev/mapper/vg0-lv0]
[DEBUG 2025/12/14 12:39:02 can't found flashCache block device!
]
[DEBUG /usr/sbin/iniparse -a -f /etc/volume/volume.conf -s 00000002-2512-1123-5617-b66bec1a6981 -system]
mountPath =  /Volume2
check filesystem on /dev/mapper/vg0-lv1
2025-12-14 12:39:04.804 [INFO] changeBtrfsSubvol::/dev/mapper/vg0-lv1 btrfs subvolume @ is Exists.
/etc/tos/scripts/mntVolume /dev/mapper/vg0-lv1
[DEBUG /bin/bash -c /etc/tos/scripts/mntVolume /dev/mapper/vg0-lv1]
[DEBUG 2025/12/14 12:39:05 can't found flashCache block device!
]
2025-12-14 12:39:06.377 [INFO] /.eraseAcl not exist
[DEBUG /bin/bash -c /etc/init.d/nas/mkbase]
[DEBUG [DEBUG /bin/bash -c btrfs subvolume create "/Volume1/@apps"]
[DEBUG /bin/bash -c btrfs subvolume create "/Volume1/@desktop"]
[DEBUG /bin/bash -c umount -nlf /home]
[DEBUG /bin/bash -c btrfs subvolume create "/Volume2/@apps"]
[DEBUG /bin/bash -c btrfs subvolume create "/Volume2/@cache"]
[DEBUG /bin/bash -c btrfs subvolume create "/Volume2/@desktop"]
]
******@TNAS-ALEX:~#
Now it seems like the information is recovered and started to self-reload again.

I really need somebody from the TerraMaster support to take a look.
Am I expecting too much?
______________________
F5-221; 10G RAM; 5 x 4TB TRAID; Riser, Power and/or ASMedia HW failure undetected by TM Support
F6-424; 32G RAM; 2 x 512GB 8Gb/s M.2; 6 x 4TB raidz2; TrueNAS 25.10.1
¯¯¯¯¯¯ Jellyfin, Pi-Hole, Unifi Controller, Zabbix
User avatar
AlAleksiev
Posts: 44
Joined: 19 Feb 2024, 00:15
Canada

Re: [Help] F5-221 is doing sporadic reloads

Post by AlAleksiev »

...System Report was sent to [email protected] ...

and that's a link to video capturing the NAS reload and the console messages on-side
https://mega.nz/file/2J1wnJ6A#VTnUJUP0l ... 0wqz64xRvQ
______________________
F5-221; 10G RAM; 5 x 4TB TRAID; Riser, Power and/or ASMedia HW failure undetected by TM Support
F6-424; 32G RAM; 2 x 512GB 8Gb/s M.2; 6 x 4TB raidz2; TrueNAS 25.10.1
¯¯¯¯¯¯ Jellyfin, Pi-Hole, Unifi Controller, Zabbix
User avatar
CursaYang
TM Support
Posts: 740
Joined: 06 Jan 2025, 16:34
China

Re: [Help] F5-221 is doing sporadic reloads

Post by CursaYang »

Hello, in addition to the system report, TOS 6 has added a debug function. Please refer to the instructions in the email to collect the debug logs and send them to us for analysis.
To contact our team, please send email to following addresses, remember to replace (at) with @

Technical team: support(at)terra-master.com(for technical support)
Service team: service(at)terra-master.com(for purchasing, return, replacement, RMA service)
User avatar
AlAleksiev
Posts: 44
Joined: 19 Feb 2024, 00:15
Canada

Re: [Help] F5-221 is doing sporadic reloads

Post by AlAleksiev »

The debug was enabled and 2 crashes happened, but the GUI didn't allowed me to download any files.
When verified the debug log folder, that was everything inside:
Image

The above is sent to you in a ZIP file.
The crashes were in 6:35 and 6:40 EST.

Please let me know if I can do something more!
Thank you!
Last edited by AlAleksiev on 16 Dec 2025, 05:02, edited 1 time in total.
______________________
F5-221; 10G RAM; 5 x 4TB TRAID; Riser, Power and/or ASMedia HW failure undetected by TM Support
F6-424; 32G RAM; 2 x 512GB 8Gb/s M.2; 6 x 4TB raidz2; TrueNAS 25.10.1
¯¯¯¯¯¯ Jellyfin, Pi-Hole, Unifi Controller, Zabbix
User avatar
AlAleksiev
Posts: 44
Joined: 19 Feb 2024, 00:15
Canada

Re: [Help] F5-221 is doing sporadic reloads

Post by AlAleksiev »

After enabling the debug the NAS started to crash every 5-15 minutes.
Image
THere are nothing new in the @system folder, so if you don't mind I'll stop it.
______________________
F5-221; 10G RAM; 5 x 4TB TRAID; Riser, Power and/or ASMedia HW failure undetected by TM Support
F6-424; 32G RAM; 2 x 512GB 8Gb/s M.2; 6 x 4TB raidz2; TrueNAS 25.10.1
¯¯¯¯¯¯ Jellyfin, Pi-Hole, Unifi Controller, Zabbix
User avatar
TMzethar
TM Support
Posts: 2827
Joined: 27 Oct 2020, 16:43

Re: [Help] F5-221 is doing sporadic reloads

Post by TMzethar »

The file you sent this time cannot be a good reference. Regarding the Debug log, you can download it from the corresponding page after restarting the NAS.
From your description, this seems to be caused by too many enabled services and containers leading to congestion.
You can use 'atop -r' to check the load situation. You can also try disabling individual or some services/containers and observe if there is any improvement.
You may also try increasing the RAM, which might help these services and containers run better.
To contact our team, please send email to following addresses, remember to replace (at) with @:
Technical team: support(at)terra-master.com (for technical support)
Service team: service(at)terra-master.com (for purchasing, return, replacement, RMA service)
User avatar
AlAleksiev
Posts: 44
Joined: 19 Feb 2024, 00:15
Canada

Re: [Help] F5-221 is doing sporadic reloads

Post by AlAleksiev »

The NAS is restarting himself pretty well and the debug folder is empty after the reload - nothing more to do when your debug is not producing any data.

From my description many services were working almost 3 years with no issue on the old TOS. I also mentioned pretty clear that the currently running TOS has only enabled docker engine. Since that moment I enabled portainer container and prepared another two that are not started.

The atop command is not working, but top is working perfectly fine.
This screenshot is made during the last crash and you can see how much from the 2 GB RAM is used in that moment - only 852 MB.
Image
The CPU is 93% idle and the only docker related processes are using together 4.6% of those 852 MB - that's 39 MB.

The memory, the CPU and the SWAP, along with electricity source and few more things were verified before to even start speaking with you guys.

Please, let's be constructive and streamlined withthe good old structured troubleshooting!
How can I arrange a real time session with some of your technician, guys?
______________________
F5-221; 10G RAM; 5 x 4TB TRAID; Riser, Power and/or ASMedia HW failure undetected by TM Support
F6-424; 32G RAM; 2 x 512GB 8Gb/s M.2; 6 x 4TB raidz2; TrueNAS 25.10.1
¯¯¯¯¯¯ Jellyfin, Pi-Hole, Unifi Controller, Zabbix
User avatar
AlAleksiev
Posts: 44
Joined: 19 Feb 2024, 00:15
Canada

Re: [Help] F5-221 is doing sporadic reloads

Post by AlAleksiev »

Few more RAM and CPU verifications as a prove for a good HW
The next tests are started on the TNAS from USB.

MemTest86 - PASSED (Ultimate Boot CD)
Image
private image link anyone with link

CPU and RAM on 100% for 11 minutes - stable work with SystemRescue
Image

CPU and RAM on 100% for 19 minutes - stable work with SystemRescue
Image

That last one is longer than the time between two crashes when the same TNAS platform is running TOS 6
______________________
F5-221; 10G RAM; 5 x 4TB TRAID; Riser, Power and/or ASMedia HW failure undetected by TM Support
F6-424; 32G RAM; 2 x 512GB 8Gb/s M.2; 6 x 4TB raidz2; TrueNAS 25.10.1
¯¯¯¯¯¯ Jellyfin, Pi-Hole, Unifi Controller, Zabbix
User avatar
AlAleksiev
Posts: 44
Joined: 19 Feb 2024, 00:15
Canada

Re: [Help] F5-221 is doing sporadic reloads

Post by AlAleksiev »

After creating manually debug/log folders in the /Volume1/@system/ the atop started to work - so, I started the atop service to collect some data ...

Code: Select all

alex@TNAS-ALEX:~$ sudo tree /Volume1/@system/
/Volume1/@system/
├── debug
│   └── log
│       └── atop_log
├── redis
│   └── dump.rdb
└── resource.db

3 directories, 3 files
alex@TNAS-ALEX:~$
I'll give another try to the debug from the TOS GUI now ... and I'll start both docker containers to create some load and make the issue more visible
Last edited by AlAleksiev on 16 Dec 2025, 21:55, edited 1 time in total.
______________________
F5-221; 10G RAM; 5 x 4TB TRAID; Riser, Power and/or ASMedia HW failure undetected by TM Support
F6-424; 32G RAM; 2 x 512GB 8Gb/s M.2; 6 x 4TB raidz2; TrueNAS 25.10.1
¯¯¯¯¯¯ Jellyfin, Pi-Hole, Unifi Controller, Zabbix
Post Reply

Return to “Initialization”