[Help] Storage pool is now corrupt after removing Hypercache - F4-425Plus TOS 7.0.0403

[luda83]

I also posted this under the lastest post for 7.0.0403 since i am running beta software but figured it make sense to post it here too. Ive been testing TOS 7 on my new F4-425Plus and updated to the latest version the other day. All was still good but i needed to swap the 2x NVME drives i was using for hypercache. I originally swapped them for larger ones with no problem by first deleting the hypercache with all apps disabled, powering down the NAS and swapping them. But this time when swapping back to the smaller ones i had before and powering back up my entire storage pool is now saying its corrupt because of missing drives yet under required drives, nothing is listed and all 4 drives that are part of the pool show present. How do i fix this without deleting the entire pool and starting over?

[TMnight]

You may first download a system log and email it to support(at)terra-master.com. We need to review the array configuration details for further analysis.

Additionally, if you could provide the number of hard drives in the storage pool prior to the issue, their respective capacities, and the specific steps taken during the subsequent drive replacement, it would help us pinpoint the problem more quickly. We greatly appreciate your cooperation.

[luda83]

i never removed or added any drives to the storage pool itself. The drives you see in the screenshot are the same 4 drives that are in the pool now, nothing changed there. the only thing that was modified was the hypercache being deleted from the gui, using google gemini it assisted me in pinpointing the potential problem that started after i deleted the hypercache and why i cannot mount the pool now. Iemailed the logs to the email listed, Below is Google Gemini summary of the problem and possible solution:

Subject: CRITICAL BUG - TOS 7 Beta: LVM Metadata Corruption after HyperCache Removal (F4-425 Plus)

Issue Summary:
The TOS 7 GUI reported a "Corrupt" storage pool after I removed a HyperCache and swapped NVMe drives. Investigation via SSH shows that the LVM layer failed to scrub the Physical Volume (PV) metadata, and the OS now prevents manual repair due to RBAC/Lock restrictions.

Technical Timeline (from attached logs):

05:00:11: User initiated HyperCache 1 deletion.

05:26:01: CRITICAL ERROR: Hyper Cache 1 deletion failed. (The GUI did not provide an alarm or stop the process).

05:46:11: Hyper Cache 1 has degraded.

Post-Reboot: Storage Pool 2 marked as "Corrupt."

Root Cause Analysis:

Metadata Desync: The LVM Volume Group (vg1) still expects the missing NVMe drive (UUID AvV6WT...) which was previously /dev/md4.

Silent Failure: The GUI failed to alert me that the LVM "reduce" operation had failed.

Permission Lockdown: Even with Developer Mode enabled, the "user" account is blocked from LVM hardware locks (/run/lock/lvm/P_global:aux), and sudo -i is disabled, preventing a manual vgreduce --removemissing.

Suggested Fix for Devs:

Ensure HyperCache deletion verification includes a mandatory check of the LVM metadata state before allowing a reboot or drive swap.

Update the "Corrupt" status logic to differentiate between "Missing RAID Member" and "Stale Cache Metadata."

[rfbjr]

I think I had/have the same issue. I forgot to delete the hyper cache before shutting down the F6-424. I was swapping out the 1tb SSD for a 4tb SSD. I had the 4tb in my D1 SSD enclosure. So, when I brought it back up the hyper cache was gone, obviously but when I created it it showed up as Hyper cache 2. Whatever. It shutdown last night (scheduled) and this morning when it came back up is when I noticed the resync going on.

Note: The unit is on a UPS and the now 4tb hyper cache is set to read/write.

[CursaYang]

It seems we did not see your email in our inbox. Please include the link to the post in the body of the email.

Removing hypercache with anomalies may lead to metadata corruption, and we need to analyze the logs further.

[luda83]

It seems we did not see your email in our inbox. Please include the link to the post in the body of the email.

Removing hypercache with anomalies may lead to metadata corruption, and we need to analyze the logs further.

I am just now seeing your reply to my post. Since i never heard back by email after a few days, i wiped the system and am currently debating returning it to Amazon due to this issue that caused me to loss all my data and no reply to my email you said was missed. I am forwarding my original email to "[email protected]" which is where i sent the original and will include a link to my post

[TMzethar]

We have received your email and reviewed the information you provided. What we can currently ascertain is that it seems the vg still needs to be used as an SSD Hyper Cache pv at startup. This indicates that the previous SSD Hyper Cache was not successfully disabled for some reason.

If your previous SSD is still available, you can try placing it back with the HDDs and then starting the NAS. This might restore access to your data.

If your previous SSD is no longer available, or if you still cannot recover data access after placing the SSD with an HDD in the NAS and powering it on, we may need to arrange for engineers to assist you with troubleshooting and attempting solutions.

[luda83]

We have received your email and reviewed the information you provided. What we can currently ascertain is that it seems the vg still needs to be used as an SSD Hyper Cache pv at startup. This indicates that the previous SSD Hyper Cache was not successfully disabled for some reason.

If your previous SSD is still available, you can try placing it back with the HDDs and then starting the NAS. This might restore access to your data.

If your previous SSD is no longer available, or if you still cannot recover data access after placing the SSD with an HDD in the NAS and powering it on, we may need to arrange for engineers to assist you with troubleshooting and attempting solutions.

Like i stated in my previous reply, the system has been wiped so those HDDs and NVMEs have been formatted and repurposed. i send another email with an attachment of excerpts of my troubleshooting using google gemini before I ended up wiping the whole NAS. It had me run "cat /proc/mdstat” and “Bash vgdisplay lvdisplay” to pinpoint the issue.

[TMzethar]

The general Linux methods obtained from a general AI are likely not suitable for the specific environment of the NAS system.
We recommend that you book a time here, and we can arrange an engineer to help you troubleshoot remotely.
terra-master.com
Live Chat.
(Please include the URL of this forum post, your nickname, as well as your time zone.)