[Share] All Volumes Disappeared After TOS Upgrade

[lestatminiyo]

SOLVED: All Volumes Disappeared After TOS Upgrade - RAID Arrays Not Assembling

System Information:
- NAS Model: TerraMaster (Custom/Generic)
- TOS Version: X64 2.06.0.794
- Storage: Multiple RAID1 arrays with LVM

Problem Description:

After upgrading TOS, all my volumes completely disappeared from the system. The web interface showed no volumes, and accessing /Volume2 and /Volume3 returned "not found" errors.

PostgreSQL Race Condition Issue?

The Problem:

During system startup, I noticed this critical error in the logs:

Code: Select all

2025/12/09 01:41:24 /buildAgent/work/b4988470bd41b41d/cmd/storagedevicetool/storagedevicetool.go:354 failed to connect to `user=terramaster database=tos`: 127.0.0.1:5032 (127.0.0.1): dial error: dial tcp 127.0.0.1:5032: connect: connection refused
[0.948ms] [rows:-] SELECT * FROM "share" WHERE device IN ('@usb','@disk')
2025/12/09 01:41:24 Failed to scan share table: failed to connect to `user=terramaster database=tos`: 127.0.0.1:5032 (127.0.0.1): dial error: dial tcp 127.0.0.1:5032: connect: connection refused
2025/12/09 01:41:29 ==========Inserting storage device==========
2025/12/09 01:41:29 insert device name:  sdza
2025/12/09 01:41:29 insert device type(0 USB / 1 Hard disk):  1
[DEBUG /usr/sbin/mdadm -EY /dev/sdza4]
2025/12/09 01:41:29 ==========Inserting storage device==========
2025/12/09 01:41:29 insert device name:  sdb
2025/12/09 01:41:29 insert device type(0 USB / 1 Hard disk):  1
2025/12/09 01:41:29 ==========Inserting storage device==========
2025/12/09 01:41:29 insert device name:  sdd
2025/12/09 01:41:29 insert device type(0 USB / 1 Hard disk):  1
2025/12/09 01:41:29 ==========Inserting storage device==========
2025/12/09 01:41:29 insert device name:  sda
2025/12/09 01:41:29 insert device type(0 USB / 1 Hard disk):  1
E1209 01:41:29.285273    6757 storagedevicepool.go:141] 1. The device has created a Raid
   1).  TOSDeamon/cmd/storagedevicetool/storagedevicepool.(*StorageDevice).Insert
        /buildAgent/work/b4988470bd41b41d/cmd/storagedevicetool/storagedevicepool/storagedevicepool.go:140
   2).  main.serveHTTP.func1
        /buildAgent/work/b4988470bd41b41d/cmd/storagedevicetool/storagedevicetool.go:155
   3).  main.serveHTTP.WrapResp.func9.1
        /home/test/go/pkg/mod/gitlab.local/golibrary/[email protected]/resp.go:116
   4).  gitlab.local/golibrary/resp.TimeConsumed
        /home/test/go/pkg/mod/gitlab.local/golibrary/[email protected]/resp.go:52
   5).  main.serveHTTP.WrapResp.func9
        /home/test/go/pkg/mod/gitlab.local/golibrary/[email protected]/resp.go:115
2025/12/09 01:41:29 ==========Inserting storage device==========
2025/12/09 01:41:29 insert device name:  sdc
2025/12/09 01:41:29 insert device type(0 USB / 1 Hard disk):  1
E1209 01:41:29.312431    6757 storagedevicepool.go:141] 1. The device has created a Raid
   1).  TOSDeamon/cmd/storagedevicetool/storagedevicepool.(*StorageDevice).Insert
        /buildAgent/work/b4988470bd41b41d/cmd/storagedevicetool/storagedevicepool/storagedevicepool.go:140
   2).  main.serveHTTP.func1
        /buildAgent/work/b4988470bd41b41d/cmd/storagedevicetool/storagedevicetool.go:155
   3).  main.serveHTTP.WrapResp.func9.1
        /home/test/go/pkg/mod/gitlab.local/golibrary/[email protected]/resp.go:116
   4).  gitlab.local/golibrary/resp.TimeConsumed
        /home/test/go/pkg/mod/gitlab.local/golibrary/[email protected]/resp.go:52
   5).  main.serveHTTP.WrapResp.func9
        /home/test/go/pkg/mod/gitlab.local/golibrary/[email protected]/resp.go:115
E1209 01:41:29.405672    6757 storagedevicepool.go:141] 1. The device has created a Raid
   1).  TOSDeamon/cmd/storagedevicetool/storagedevicepool.(*StorageDevice).Insert
        /buildAgent/work/b4988470bd41b41d/cmd/storagedevicetool/storagedevicepool/storagedevicepool.go:140
   2).  main.serveHTTP.func1
        /buildAgent/work/b4988470bd41b41d/cmd/storagedevicetool/storagedevicetool.go:155
   3).  main.serveHTTP.WrapResp.func9.1
        /home/test/go/pkg/mod/gitlab.local/golibrary/[email protected]/resp.go:116
   4).  gitlab.local/golibrary/resp.TimeConsumed
        /home/test/go/pkg/mod/gitlab.local/golibrary/[email protected]/resp.go:52
   5).  main.serveHTTP.WrapResp.func9
        /home/test/go/pkg/mod/gitlab.local/golibrary/[email protected]/resp.go:115

What This Means:

This is a race condition during boot. The sequence is:

1. System boots → mdadm assembles RAID arrays → LVM activates volumes
2. TOS services start (TOSDaemon, storagedevicetool)
3. PostgreSQL starts

The problem: storagedevicetool tries to connect to PostgreSQL BEFORE the database is fully ready. This happens because:
- PostgreSQL takes time to initialize (especially with large databases)
- TOS storage services start immediately and expect the database to be available

At first I thought it couldn't be a race condition issue, since it seems to insert data after 5 seconds. The problem is that it's generating new RAID arrays.

Current State:
System is stable NOW but reboot risk exists

---

Evidence/Diagnosis:

1. Initial RAID Status:

Code: Select all

# cat /proc/mdstat
md1 : active raid1 sdza1[0]
md9 : active raid1 sdb2[2] sda2[0]

# Only md1 and md9 were active - md0 and md2 were missing!

2. Missing RAID Arrays:
The diagnostic files revealed:
- Array md0 (7.3TB): /dev/sda4 + /dev/sdb4 - UUID a33e66aa-6544-a767-8ee5-b7f56939d541 NOT ASSEMBLED
- Array md2 (5.5TB): /dev/sdc4 + /dev/sdd4 - UUID 4a5b9ffc-b954-9344-009e-8b75389cbc45 NOT ASSEMBLED

3. LVM Volume Status:

Code: Select all

# pvdisplay
PV Name: unknown        <-- Physical volume couldn't be found
VG Name: vg0
PV Size: 7.27 TiB

# lvdisplay
LV Status: NOT available    <-- Volume was offline!
LV Size: 7.27 TiB

4. Root Cause:
Running

Code: Select all

mdadm --assemble --scan --verbose

showed:

Code: Select all

mdadm: /dev/sda4 has wrong uuid.
mdadm: /dev/sdb4 has wrong uuid.
mdadm: /dev/sdc4 has wrong uuid.
mdadm: /dev/sdd4 has wrong uuid.

The file /etc/mdadm/mdadm.conf was incomplete and contained only OS and M.2 cache disk.
TOSDaemon rewrite it several times after the crash

Solution - Step by Step:

Step 1: Manually Assemble RAID Arrays

Instead of using --scan, assemble each array manually with its correct UUID:

Code: Select all

# Assemble main volume (7.3TB)
mdadm --assemble /dev/md0 --uuid=a33e66aa-6544-a767-8ee5-b7f56939d541 /dev/sda4 /dev/sdb4

# Assemble secondary volume (5.5TB)
mdadm --assemble /dev/md2 --uuid=4a5b9ffc-b954-9344-009e-8b75389cbc45 /dev/sdc4 /dev/sdd4

# Assemble swap (optional but recommended)
mdadm --assemble /dev/md3 --uuid=f5b4c951-b9e5-088d-723a-2e6d993e4d5b /dev/sda3 /dev/sdb3

Verify arrays are online:

Code: Select all

# cat /proc/mdstat
md3 : active raid1 sda3[0] sdb3[1]
md2 : active raid1 sdc4[0] sdd4[1]
      5848949636 blocks super 1.2 [2/2] [UU]
md0 : active raid1 sda4[0] sdb4[1]
      7802074636 blocks super 1.2 [2/2] [UU]

Step 2: Activate LVM Volumes

Code: Select all

# Scan for volume groups
vgscan

# Activate the volume group
vgchange -ay vg0

# Verify volumes are active
lvscan
# Should show: ACTIVE '/dev/vg0/lv1' and '/dev/vg1/lv2'

Step 3: Update mdadm.conf with Correct UUIDs

Code: Select all

# Generate correct configuration
mdadm --detail --scan > /etc/mdadm/mdadm.conf

# Verify the file
cat /etc/mdadm/mdadm.conf

Step 4: Update initramfs (CRITICAL STEP!)

This updates the boot configuration so arrays assemble automatically on next boot:

Code: Select all

update-initramfs -u

Step 5: Mount Volumes

Code: Select all

# Mount the volumes
mount /dev/vg0/lv1 /Volume2
mount /dev/vg1/lv2 /Volume3

# Verify
df -h | grep Volume
# /dev/mapper/vg0-lv1  7.3T  3.9T  3.5T  53% /Volume2
# /dev/mapper/vg1-lv2  5.5T  2.0T  3.6T  36% /Volume3

# Extraction
(Extract important data if needed)

# Umount
umount /Volume3
umount /Volume2

Step 6: Restart TOS Services

Code: Select all

# Restart TOSDaemon to recognize volumes
systemctl restart tosdaemon

# Restart Samba for network shares
systemctl restart smbd nmbd

# Verify TOSDaemon recognizes the volumes
tail -f /var/log/syslog | grep TOSDaemon

Verification:

After completing all steps:
- ✓ All RAID arrays showing [UU] status
- ✓ LVM volumes active and mounted
- ✓ TOSDaemon monitoring arrays correctly
- ✓ Web interface shows volumes
- ✓ Data fully accessible

Potential Solutions Before Rebooting:

Workaround: Add PostgreSQL Delay?
No, actually that's not possible since the service is already waiting for Postgres to start:

/etc/systemd/system/tnas-system.target.wants/TOSDaemon.service

Code: Select all

[Unit]
Description=Start TOSDeamon service
Wants=postgresql.service nginx.service network.target network-online.target redis.service tnas-mntdata.service
After=postgresql.service nginx.service network.target network-online.target redis.service tnas-mntdata.service

But there is a small gap between the PostgreSQL service is up and the PostgreSQL is accepting connections.


What Could Happen on Reboot:

Best case scenario:
- RAID arrays assemble correctly (thanks to updated mdadm.conf)
- LVM activates volumes
- Volumes mount from /etc/fstab
- PostgreSQL eventually starts
- TOS recognizes everything (might take 30-60 seconds)

Worst case scenario:
- storagedevicetool fails to load volume config from database
- TOS web interface shows "no volumes"
- Data is physically there but not accessible through TOS UI
- Would need to follow the previous steps again and restart TOSDaemon/services

[AlAleksiev]

Are you sure Step 4 is possible in TOS 6 ?
Check that

[lestatminiyo]

Are you sure Step 4 is possible in TOS 6 ?
Check that

The thing is, when I wrote the thread I retraced my steps based on my command history, but it’s true that there’s no binary to update the kernel image, so step 4 should be removed. Thank you very much.

[AlAleksiev]

No problem - you are welcome.
My concern is that without Step 4 the arrays will not be assembled automatically after reload...
Cheers