NVMe ADATA S11 Pro Failure in TOS 7: System Freeze, Critical Pool Error, and Web UI Failure on F8

[AES]

Good afternoon.
I am using a TerraMaster F8 running the latest version of TOS 7.
One of my ADATA XPG GAMMIX S11 Pro [AGAMMIXS11P-1TT-C] NVMe 1000 GB drives, purchased in November 2024, has failed.
Until recently, it was in excellent health; I was successfully saving files to it in the morning, but the other evening, it suddenly started malfunctioning. It was not a boot/system drive (it used to be a system drive a long time ago, but I later removed it from that role). It simply held an encrypted download folder for syncing files from my laptop, etc. The drive was running independently, without RAID.
After a reboot (using F8), the entire system completely froze due to this drive. Once I physically removed the SSD, the system booted normally. TOS itself is installed on two Samsung drives.
Key Issues & Anomalies:

1. False Alerts and Empty Logs:
The right-side panel in the TOS 7 web interface displayed a green error message claiming that another drive was broken, even though all remaining drives successfully passed SMART checks. Naturally, the logs application showed absolutely nothing—the logs were completely empty.
2. Web UI and Services Breakdown:
The next morning, I opened the web interface and found that only one icon remained in the top menu. Almost nothing is clickable now; it is impossible to launch either the terminal or the logs application. Video Surveillance and Transmission apps fail to open. The Docker container manager only lets me open its web UI, but none of the containers actually work.
3. Abnormal Overheating:
Furthermore, when I reboot the F8 device using the physical hardware button, the slot housing this specific drive and the exhaust air coming out of it feel remarkably hot, whereas all other components and drives remain perfectly cool.
4. SMART Test Behavior:
The SMART testing utility in TOS 7 does not display the completion time of the tests. In the web interface, the progress doesn't always update or complete automatically until you manually refresh the page. Judging by the fact that after refreshing, it might show "Test Completed" while still running other tasks, it seems the remaining drives are testing normally. However, I believe having the exact completion date/time of the last test is essential.

[TMzethar]

Hi, sorry to learn of the abnormal situation you are experiencing.

Regarding the SSD in question — have you attempted to restart the system after removing it? Based on the information provided, we suspect that the system may still be utilizing the kernel located on the boot partition of that drive.

The simultaneous occurrence of multiple anomalies suggests a potential correlation with system files, the kernel, or drive‑related irregularities. However, the current information does not allow us to identify the precise cause.

If terminal access remains available, you may schedule a remote troubleshooting session with our engineer at the following link:
https://salesiq.zohopublic.com/signatur ... 97d0be8892

Should you require a more expedient resolution, you may consider resetting TOS 7. If feasible, we kindly ask that you generate a system report and forward it to our technical support team prior to the reset — this would greatly assist us in diagnosing the underlying cause.

[AES]

Good afternoon, before rebooting, I was simply getting a failure to write new files to the drive. That's how I figured out the drive was bad: the laptop couldn't write new files to it over the network. I checked SMART and there were sector remapping exhaustion errors.
Everything seems to be working now, after removing the drive, of course.

I'm thinking of looking for an m2 NVMe to PCI adapter and trying to extract the encrypted data from a Windows PC using R-Studio 9.5. I think it should support both recovery and encryption, just like TOS 7 (although I'm not sure).
Unfortunately, TOS won't boot with the failed drive, even though it's located on two other drives, even though they're on different channels. The failed drive shouldn't have had an OS for a long time.
And it turns out that NVMe drives can't be hot-plugged after power is applied.
Interestingly, I have another identical drive; it works with video surveillance and is working fine. Something must have happened to the controller or something similar, although the drives seem to be branded and have excellent reliability.

[Felix]

Good afternoon, before rebooting, I was simply getting a failure to write new files to the drive. That's how I figured out the drive was bad: the laptop couldn't write new files to it over the network. I checked SMART and there were sector remapping exhaustion errors.
Everything seems to be working now, after removing the drive, of course.

I'm thinking of looking for an m2 NVMe to PCI adapter and trying to extract the encrypted data from a Windows PC using R-Studio 9.5. I think it should support both recovery and encryption, just like TOS 7 (although I'm not sure).
Unfortunately, TOS won't boot with the failed drive, even though it's located on two other drives, even though they're on different channels. The failed drive shouldn't have had an OS for a long time.
And it turns out that NVMe drives can't be hot-plugged after power is applied.
Interestingly, I have another identical drive; it works with video surveillance and is working fine. Something must have happened to the controller or something similar, although the drives seem to be branded and have excellent reliability.

You may schedule a remote troubleshooting session with our engineer at the following link:
https://salesiq.zohopublic.com/signatur ... 97d0be8892

[AES]

Hi everyone,
I want to share my recent, highly frustrating experience recovering data from a failed NVMe SSD (XPG GAMMIX S11 Pro) on my TerraMaster F8. While I eventually managed to rescue my files, the process exposed severe architectural flaws in how TOS handles hardware timeouts and user-space encryption (eCryptfs). I hope this detailed breakdown helps other users and provides the engineering team with clear areas for improvement.
## 1. The Core Issue: Missing Hardware Timeout During Boot
My clean TOS system was migrated to two Samsung SSDs a very long time ago and has been working flawlessly there. The old XPG drive was used strictly for data storage, but it still contained legacy hidden OS partitions (sdb1, sdb2, sdb3) from a million years ago. When the XPG drive experienced a hardware controller failure, it completely broke the NAS boot sequence.

* The Bug: Instead of timing out, dropping the unresponsive drive, and booting from the healthy Samsung array, TOS entered an infinite hardware polling loop. The entire system hard-froze, refusing to boot at all.
* The Workaround: I had to physically pull the broken drive out just to get the NAS to start, and then buy an external Ugreen USB-to-NVMe adapter to mount it while the system was running.

## 2. The Software Trap: Broken Permission Layers in eCryptfs Recovery
Once connected via USB, the drive’s pool errors blocked the native TOS UI. I had to manually assemble the hidden array via SSH using mdadm and activate the LVM group (vg0/lv0).
Although the drive was successfully mounted and my passphrase accepted, transferring the decrypted data became a complete nightmare due to how privileges are segregated:

* The Privilege Loop: Running file operations (cp, rsync) as a standard admin user resulted in immediate Permission denied or file has vanished errors because the user lacked structural read permissions on the low-level Read-Only mount.
* The Sudo Filename Breakage: Attempting to elevate permissions via sudo or dropping into a root shell immediately caused the OS kernel to drop the active user's cryptographic context. As a result, filenames instantly broke, reverting to raw encrypted masks (ECRYPTFS_FNEK_ENCRYPTED...). To make matters worse, basic utilities like sudo -i or su are completely locked down by the TOS environment, triggering Permission denied messages for the administrator.


------------------------------
## Technical Appendix: Failed Commands and Specific Bottlenecks
For the engineering team's reference, here is the exact progression of terminal interactions, along with the precise errors encountered under TOS:

1. Direct Mount and LVM Initialization Failures:
Direct block mounting failed immediately because the device contains nested LVM layers inside an md array. Standard vgscan ignored the device due to volume group signature conflicts with the active live drives (vg1).
* Command: mount -t btrfs -o ro,usebackuproot /dev/sdb4 /Volume2/recovery_disk
* Error: wrong fs type, bad option, bad superblock on /dev/sdb4
* Resolution: Forced single-drive assembly via mdadm --assemble --run /dev/md99 /dev/sdb4, which correctly exposed vg0/lv0 for a successful Read-Only mount.
2. The Native User Session Context vs. sudo Privilege Drop:
The ecryptfs-recover-private script failed natively due to missing temporary directory provisioning paths in the target TOS build framework.
* Error: mount(2) system call failed: No such file or directory. ERROR: Failed to mount private data.
* Workaround: Manual interactive mount execution directly via mount -t ecryptfs.
3. Complete sudo Transfer Lockout (The Pyramidal Permission Bug):
Once mounted manually under the active user session (Anton), any attempt to duplicate or move files via privileged accounts completely stripped the active eCryptfs Filename Encryption Key (FNEK) translating layer.
* Command: sudo cp -rp /Volume2/Downloads_Decrypted/. /Volume2/public/
* Result: cp: cannot stat '.': Permission denied
* Command: sudo bash -c 'cp -rp /Volume2/Downloads_Decrypted/* /Volume2/public/'
* Result: cp: cannot stat '/*': Permission denied
* Command: sudo rsync -av --ignore-errors /Volume2/Downloads_Decrypted/ /Volume2/public/
* Result: change_dir "/Volume2/Downloads_Decrypted" failed: Permission denied (13)
4. User-Space Extraction Loophole:
When operating directly as the unprivileged user Anton (without sudo), the filename translations were readable via ls -la, but structural access was dropped mid-read by the underlying kernel driver during local block cloning.
* Command: tar -cf /Volume2/public/backup.tar -C /Volume2/Downloads_Decrypted .
* Result: Loop generated continuous cascading warnings: tar: ./[Filename]: File removed before we read it
* Command: rsync -av --ignore-errors /Volume2/Downloads_Decrypted/ /Volume2/public/Downloads_MacBook_OK/
* Result: Cascading thread terminations: file has vanished: [Filename] (code 24)




Ultimately, standard local duplication utilities failed completely. The only way to bypass this locked loop was to mount the decrypted folder as a network share and pull the data remotely via SMB directly to a Mac, letting the user-session translate the filenames on the fly.
## Suggestions for TerraMaster Developers:

1. Implement Aggressive Boot Timeouts: The bootloader must gracefully skip any internal PCIe/NVMe device that fails to respond within a reasonable timeout window, allowing the NAS to boot in a degraded state.
2. Fix the Hidden Partition Mirroring: Since the system was moved to new drives a long time ago, TOS should provide a clean way to wipe or safely decommission the hidden boot mirrors on old drives so they don't threaten system stability later.
3. Enhance GUI Direct Mount for Recovery: The "Direct Mount" feature in the Storage Manager should be smart enough to scan nested mdadm + LVM structures on external USB devices and safely handle eCryptfs passphrases without throwing generic pool errors.
4. Decouple Administrative Privileges from Encryption Context: If a user is authorized via sudo, the kernel should inherit or retain the active cryptographic keys (FNEK), preventing translation failures during emergency recovery tasks.

Streamlining these edge cases would drastically improve TOS recovery capabilities for power users.

[AES]

UPDATE / ADDENDUM: Deep-Dive Technical Analysis of eCryptfs Kernel Dropping and Automated SMB Share Dismounts
To ensure the TerraMaster engineering team has sufficient technical data to investigate these edge cases, here is the granular breakdown of the specific system behaviors and automated routines observed during the final recovery phase:
## 1. The Low-Level Mechanics of the Local Copy Blockage
When trying to operate directly inside the local file system using the unprivileged admin account (Anton), we verified that the file names are fully translated and cached by the active user-space cryptographic session (FNEK). However, any subsequent sequential read command initiated by system utilities (such as cp, rsync, or tar) fails instantly during the file open/stat phase.
The kernel driver handles this under a Read-Only mount by abruptly dropping structural access, resulting in the following loop of cascading failures:

* The Root Cause: The OS core fails to propagate read privileges across different virtual layers when the underlying physical device is forced into a hardware-protection ro (read-only) mode.
* Resulting Kernel Messages: Standard file operations return continuous warnings such as tar: ./[Filename]: File removed before we read it or file has vanished (code 24). This confirms that while the indexing metadata is accessible, the raw file streams are dynamically isolated and cut off from local administrative tools.

## 2. Automated SMB Daemon Piercing and Resource Blackouting
To bypass the broken local copy layers, we successfully duplicated the raw, un-translated encrypted blocks directly into a healthy storage volume (/Volume2/Docker/Encrypted_Files). The raw sector copy completed without warnings.
However, the moment we executed a manual kernel mount command (mount -t ecryptfs /Volume2/Docker/Encrypted_Files /Volume2/Docker) to translate the data inside a live network share, a critical architectural conflict within TOS was triggered:

* The Routine: The TOS background daemon continuously monitors the file system hierarchy. The exact millisecond it detects a manual ecryptfs mount signature inside an existing network resource tree, it flags the path as a security or structural anomaly.
* The Behavioral Bug: Instead of adjusting access privileges or restricting configuration modifications, the system aggressively terminates or completely filters the SMB daemon pipe for that specific path. The connected macOS client instantly experiences a silent network dropout, rendering a fully active folder as completely blank (0 objects, Available: 294.59 GB) in Finder. This proves that the OS actively overrides standard Samba configurations when manual cryptographic namespaces are introduced.

## 3. The Structural Risk of Universal RAID-1 System Partition Mirroring
This entire catastrophic recovery loop is a direct consequence of a fundamental OS design flaw: universal, forced mirroring of system partitions across every single block device inserted into the F8 chassis.

* The Long-Term Trap: Years ago, the primary OS runtime was successfully migrated to clean, dedicated Samsung SSDs. However, TOS leaves the hidden sdb1, sdb2, and sdb3 RAID-1 partition mirrors permanently alive on secondary data expansion slots.
* The Boot Loop Trigger: When the expansion drive suffers a hardware controller failure, the bootloader or early OS init scripts attempt to probe and synchronize all available hidden OS mirrors globally. Because there is no software polling timeout implemented for unresponsive hardware during early device mapping, a single dying data drive completely compromises system availability, forcing an un-bootable hard-freeze state.

## Recommended Architectural Fixes:

1. Cryptographic Key Context Inheritance: Allow administrative execution shells (like sudo tasks) to safely inherit or share active FNEK signature rings during emergency block migrations.
2. Graceful Network Isolation: Replace the aggressive SMB network blocking routine with a transparent warning flag inside the Web UI, allowing authorized users to access mounted local partitions over network protocols safely.
3. Decoupled Boot Partition Management: Provide an explicit "Decommission Drive from System Array" feature in the Storage Manager, allowing users to wipe legacy hidden boot mirrors from drives dedicated purely to independent data storage pools.

[TMroy]

Thank you for the report. We'll look into each item and get back to you with our findings. Appreciate it!

[TMbuddy]

Hello AES,
Thank you for providing such a detailed description of the issue and technical analysis. This information is very helpful for us to investigate the problem.

Regarding the issues you reported, we will address them one by one:

1. Regarding the system boot failure caused by the NVMe drive failure
Under normal circumstances, after the system migration is completed, the original drive should no longer be used as the system drive and should not affect system boot.

Regarding the sdb1, sdb2, and sdb3 partitions you mentioned, we would like to clarify that TOS reserves system-related partition locations at the beginning of the drive to support switching a drive between system drive and non-system drive roles. Therefore, the partition numbers alone cannot determine whether the drive currently still contains system data or participates in the system boot process.

Based on the behavior you described, we are currently unable to confirm the exact cause. It may be related to the drive status, the system migration process, or device detection during the boot stage. We will further investigate and reproduce the issue:

• Whether the old drive still retains system-related partitions after system migration.

• Whether the failed NVMe device is incorrectly detected or accessed during the boot process.

• How the system currently handles an unresponsive PCIe/NVMe device during the boot process.

2. SMART test does not display the completion time and progress does not update automatically
SMART test start and completion information are both recorded in the disk logs.



We have recorded your suggestion of “displaying the latest SMART test completion time directly in the interface” and will consider optimizing the interface display in future updates.

Currently, the SMART test status is updated through a periodic polling mechanism. Under normal circumstances, the status will refresh automatically, but it may require a short waiting period. If the status displayed on the page is inconsistent with the actual test status, we will further verify the status synchronization mechanism.

3. TOS reports a disk abnormality while SMART status is normal
The new version of TOS does not rely only on SMART information for disk status judgment. Instead, it evaluates multiple indicators to determine the overall disk status.

It should be noted that SMART status can only reflect part of the drive health information and cannot cover all types of hardware failures.

4. Abnormal Web UI behavior and TNAS PC volume status color display
You reported that after the failure:

• Web page functions became abnormal;

• Some applications could not be opened;

• Docker containers could not run properly;

• In TNAS PC, the “Volume Damaged” status was displayed in green.

We will forward these issues to the corresponding teams for further analysis and fixes.

5. The slot containing the failed drive is very hot while other drives are normal
In the event of a drive failure, the SSD controller may experience abnormal retries or response issues, which can result in increased power consumption and higher temperatures.

This behavior is more likely related to SSD hardware abnormalities, and we will continue to monitor this situation.

6. Missing functionality for mounting storage pools and volumes
Regarding your request for “directly mounting storage pools and volumes through the graphical interface”, this function is already supported in TOS Storage Manager. When an mdadm + LVM structure is detected, Storage Manager will display it as an “Unmounted Storage Pool” and provide a mount option.
Based on your description, it appears that you manually performed the mdadm assembly, LVM activation, and mounting operations through the command line instead of using the mount function provided in Storage Manager.

If the storage pool was created on this system:

If no related configuration information is found on this system:


To further reproduce the issue you encountered, we would like to confirm whether your operation steps were as follows:

• Create a storage pool and volume using an internal SSD in the NAS, and create an encrypted shared folder on the volume;

• Due to the SSD failure, and because NVMe drives do not support hot swapping, reconnect the SSD externally;

• Manually assemble mdadm, activate LVM, and mount the storage;

• Encounter permission or file access issues when accessing the encrypted shared folder.

7. eCryptfs encryption issue and SMB network access issue
The issues you reported regarding abnormal file permissions after eCryptfs decryption and inability to properly access file names involve encryption mechanisms, permission inheritance, and recovery scenarios.

Since this scenario involves manually assembling arrays, activating LVM, and recovering encrypted data, we need to further confirm your complete operation steps and environment information before arranging the corresponding team to analyze the issue.

This information is very helpful for our analysis of advanced recovery scenarios.
Thank you for taking the time to organize and provide such detailed information. Your feedback and analysis are very valuable for improving the stability of TOS and the overall recovery experience.