[Help] F5-221 is doing sporadic reloads

[AlAleksiev]

You have the debug data collected now - I hope it will help you to figure out what's going on with the TOS.
I send it to [email protected]
Please, let me know if you need something else!

[TMzethar]

When the Debug mode is started, exceptions are not recorded. Please let it encounter an exception when running in Debug mode so that relevant information can be collected.

[AlAleksiev]

Debug is enabled and I started all the containers (Jellyfin, Proxy.py and Pi-Hole) to have a realistic minimal load.
What is weird though is that the files in the debug folder are not growing:


I'll record the moment of the crash to faciliate your searching, but the above files are kind of concerning...

Cheers

[AlAleksiev]

The new debug data is sent to [email protected] - for the crash at 2025/12/16 in 22:56:55 EST

[AlAleksiev]

This is what I found interesting in the debug information:

• IPv6 is diabled, but TOSDaemon is still trying to use it

• msgcenter is fooled by the docker host container for a duplicated IP

• containers' healthcheck interval is too small

Code: Select all

--------------------------- syslog ---------------------------
... ...
Dec 16 22:52:27 TNAS-ALEX TOSDaemon[4907]: getIP http get failed url=https://ipv6.seeip.org error=Get "https://ipv6.seeip.org": dial tcp [2602:fed3:2:b75e:ecd:7ec4:6bcb:c6d3]:443: connect: cannot assign requested address
Dec 16 22:52:39 TNAS-ALEX systemd[1]: run-docker-runtime\x2drunc-moby-4aec740d5ce4618ce2423b8f32ecd51ed1b4a456e914ea4428e6bdb2bda1b56e-runc.jothka.mount: Deactivated successfully.
... ...
Dec 16 22:56:04 TNAS-ALEX systemd[1]: run-docker-runtime\x2drunc-moby-0fe8960743a0fe860b456aa548923d052b919d78989920041e205abfeccd4f0f-runc.gqd4il.mount: Deactivated successfully.

 <<<<<<<<<< CRASH >>>>>>>>>>>

Dec 16 22:59:15 TNAS-ALEX systemd-modules-load[318]: Module 'ecryptfs' is built in
Dec 16 22:59:15 TNAS-ALEX mdadm[320]: DeviceDisappeared event detected on md device /dev/md0
Dec 16 22:59:15 TNAS-ALEX systemd-modules-load[318]: Inserted module '8812au'
Dec 16 22:59:15 TNAS-ALEX mdadm[320]: DeviceDisappeared event detected on md device /dev/md8
... ...
--------------------------------------------------------------

From the kernel log - nothing abnormal

Code: Select all

-------------------------- kern.log --------------------------
... ...
Dec 16 22:38:35 TNAS-ALEX 1 2025-12-16T22:38:35-05:00 - - TOSDaemon(4907) - [attrs ws="rpc"] normal closed connection

 <<<<<<<<<< CRASH >>>>>>>>>>>

Dec 16 22:59:15 TNAS-ALEX kernel: [    0.000000] Linux version 6.1.120+ (root@51f02058744a) (x86_64-tnas-linux-gnu-gcc.br_real (Buildroot 2023.08.1) 13.2.0, GNU ld (GNU Binutils) 2.40) #52 SMP PREEMPT_DYNAMIC Thu Nov 27 09:25:39 CST 2025
Dec 16 22:59:15 TNAS-ALEX kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/bzImage type=raid hpet=disable security=
... ...

The ATOP log is giving a good information.

• RAM is exhausted

• TOSDaemon is consuming a lot of memory from the beginning

• No available memory for processes like Jellyfin

Code: Select all

----------------- LAST ATOP BEFORE THE CRASH -----------------
ATOP - TNAS-ALEX                            2025/12/16  22:56:15                            -----------------                            30s elapsed
PRC | sys    1.21s  | user   1.93s  | #proc    269  | #trun      3  | #tslpi   324 |  #tslpu   131 |  #zombie    0 |  clones   100 |  #exit     72 |
CPU | sys       4%  | user      7%  | irq       1%  | idle    186%  | wait      3% |  ipc     0.47 |  cycl  188MHz |  curf 2.00GHz |  curscal  79% |
cpu | sys       2%  | user      3%  | irq       0%  | idle     93%  | cpu001 w  1% |  ipc     0.45 |  cycl  188MHz |  curf 2.50GHz |  curscal 100% |
cpu | sys       2%  | user      3%  | irq       0%  | idle     93%  | cpu000 w  2% |  ipc     0.49 |  cycl  188MHz |  curf 1.50GHz |  curscal  59% |
CPL | avg1    0.32  | avg5    0.24  | avg15   0.38  |               | csw    56926 |               |  intr   56871 |               |  numcpu     2 |
MEM | tot     1.8G  | free   75.5M  | cache 749.9M  | dirty   0.6M  | buff    2.7M |  slab  108.0M |  shmem 109.0M |  shrss   0.0M |  numnode    1 |
SWP | tot     1.9G  | free    1.7G  | swcac  36.8M  |               |              |               |               |  vmcom   3.0G |  vmlim   2.8G |
PAG | scan      34  | steal     34  | stall      1  | compact    1  | numamig    0 |  migrate 2922 |  swin       2 |  swout      0 |  oomkill    0 |
LVM |      vg0-lv0  | busy      1%  | read      12  | write     70  | discrd     0 |  KiB/w     64 |  MBr/s    0.0 |  MBw/s    0.1 |  avio 4.27 ms |
MDD |          md0  | busy      0%  | read      12  | write     68  | discrd     0 |  KiB/w     65 |  MBr/s    0.0 |  MBw/s    0.1 |  avio 1.41 ms |
MDD |          md8  | busy      0%  | read       2  | write      0  | discrd     0 |  KiB/w      0 |  MBr/s    0.0 |  MBw/s    0.0 |  avio 9.00 ms |
... ... ...

    PID     TID MINFLT MAJFLT VSTEXT  VSLIBS  VDATA VSTACK LOCKSZ   VSIZE  RSIZE  PSIZE  VGROW  RGROW  SWAPSZ RUID     EUID      MEM  CMD       1/15
  29365       -     69      0  44.0K   99.3M 422.8M 132.0K   0.0K  260.8G 335.0M 324.0M -48.0K  -1.1M  500.0K ******   ******    18%  jellyfin
   4907       -      0      1  35.9M   47.3M 216.2M 132.0K   0.0K    2.3G 168.8M 165.4M     0B  -8.0K   90.9M ******   ******     9%  TOSDaemon
   5541       -      0      0  31.2M    4.0K  61.7M 132.0K   0.0K    1.2G  34.3M  35.7M     0B     0B    2.2M ******   ******     2%  portainer


----------------- FIRST ATOP AFTER THE CRASH -----------------
ATOP - TNAS-ALEX                           2025/12/16  22:59:38                           -----------------                            1m31s elapsed
PRC | sys   10.04s  | user  29.53s  | #proc    274  | #trun      6  | #tslpi   219 |  #tslpu   140 |  #zombie    0 |  clones  5038 |  #exit     11 |
CPU | sys      44%  | user    101%  | irq       1%  | idle     46%  | wait      8% |  ipc  initial |  cycl initial |  curf 2.40GHz |  curscal  95% |
cpu | sys      22%  | user     50%  | irq       0%  | idle     23%  | cpu000 w  4% |  ipc  initial |  cycl initial |  curf 2.40GHz |  curscal  95% |
cpu | sys      22%  | user     50%  | irq       0%  | idle     23%  | cpu001 w  4% |  ipc  initial |  cycl initial |  curf 2.40GHz |  curscal  95% |
CPL | avg1    6.10  | avg5    2.30  | avg15   0.83  |               | csw   363204 |               |  intr  308563 |               |  numcpu     2 |
MEM | tot     1.8G  | free  372.4M  | cache 574.9M  | dirty   1.4M  | buff   17.3M |  slab   85.6M |  shmem  31.4M |  shrss   0.0M |  numnode    1 |
SWP | tot     1.9G  | free    1.9G  | swcac   0.0M  |               |              |               |               |  vmcom   2.3G |  vmlim   2.8G |
LVM |      vg0-lv0  | busy      4%  | read     533  | write     95  | discrd     0 |  KiB/w     69 |  MBr/s    0.3 |  MBw/s    0.1 |  avio 5.62 ms |
LVM |      vg0-lv1  | busy      1%  | read     242  | write      0  | discrd     0 |  KiB/w      0 |  MBr/s    0.1 |  MBw/s    0.0 |  avio 4.77 ms |
MDD |          md9  | busy     11%  | read    7321  | write   1205  | discrd     0 |  KiB/w      4 |  MBr/s    5.8 |  MBw/s    0.1 |  avio 1.20 ms |
MDD |          md0  | busy      5%  | read     896  | write     93  | discrd     0 |  KiB/w     70 |  MBr/s    0.5 |  MBw/s    0.1 |  avio 4.65 ms |
MDD |          md8  | busy      0%  | read     102  | write      2  | discrd     0 |  KiB/w      4 |  MBr/s    0.0 |  MBw/s    0.0 |  avio 2.42 ms |
DSK |          sdb  | busy     15%  | read    4568  | write   2895  | discrd     0 |  KiB/w    116 |  MBr/s    4.0 |  MBw/s    3.6 |  avio 1.76 ms |
DSK |          sda  | busy     11%  | read    2803  | write   2653  | discrd     0 |  KiB/w      3 |  MBr/s    5.7 |  MBw/s    0.1 |  avio 1.89 ms |
DSK |          sdd  | busy      2%  | read     468  | write     75  | discrd     0 |  KiB/w     54 |  MBr/s    0.2 |  MBw/s    0.0 |  avio 3.03 ms |
DSK |          sde  | busy      2%  | read     547  | write     71  | discrd     0 |  KiB/w     54 |  MBr/s    0.2 |  MBw/s    0.0 |  avio 2.61 ms |
DSK |          sdc  | busy      2%  | read     471  | write     66  | discrd     0 |  KiB/w     53 |  MBr/s    0.2 |  MBw/s    0.0 |  avio 2.96 ms |
NET | transport     | tcpi    5945  | tcpo    5899  | udpi     170  | udpo     383 |  tcpao     39 |  tcppo     33 |  tcprs      5 |  udpie      0 |
NET | network       | ipi     6599  | ipo     6517  | ipfrw      0  | deliv   6577 |               |               |  icmpi    212 |  icmpo      0 |
NET | eth1      0%  | pcki     483  | pcko     283  | sp 1000 Mbps  | si    5 Kbps |  so    3 Kbps |  erri       0 |  erro       0 |  drpo       0 |
NET | eth0      0%  | pcki     278  | pcko     341  | sp 1000 Mbps  | si    2 Kbps |  so    2 Kbps |  erri       0 |  erro       0 |  drpo       0 |
NET | lo      ----  | pcki    6031  | pcko    6031  | sp    0 Mbps  | si  112 Kbps |  so  112 Kbps |  erri       0 |  erro       0 |  drpo       0 |
NET | bond0   ----  | pcki     685  | pcko     624  | sp    0 Mbps  | si    6 Kbps |  so    6 Kbps |  erri       0 |  erro       0 |  drpo       0 |
                                          *** System and Process Activity since Boot ***     Rawfile view
    PID     TID MINFLT MAJFLT VSTEXT  VSLIBS  VDATA VSTACK LOCKSZ   VSIZE  RSIZE  PSIZE  VGROW  RGROW  SWAPSZ RUID     EUID      MEM  CMD       1/13
   4566       -   8708     35  35.9M   46.7M 168.5M 132.0K   0.0K    2.0G 257.0M 253.5M   2.0G 257.0M      0B ******   ******    14%  TOSDaemon
    432       -   3686      0   7.3M    7.5M 133.1M 132.0K   0.0K    1.9G  43.4M  42.3M   1.9G  43.4M      0B ******   ******     2%  ter_storage_de
   4568       -   2667      9  16.3M    1.9M 122.4M 132.0K   0.0K    1.8G  35.4M  35.2M   1.8G  35.4M      0B ******   ******     2%  nasips
    425       -   5120     48   5.6M   16.4M   1.8M 132.0K   0.0K  214.6M  31.0M  10.3M 214.6M  31.0M      0B postgres postgres   2%  postgres
   4578       -   3949      4   5.0M    5.6M  96.5M 132.0K   0.0K    1.6G  30.8M  29.4M   1.6G  30.8M      0B ******   ******     2%  twm_authentica

The first conclusion seems to be related to the RAM. TOS v6 is consuming it pretty well and for the F5-211 platform default 2GiB is reaching the limit.
The updated containers are using more RAM than their old versions and that's pushing the limits as well.

I alreay ordered one SODIMM


I'll add it tomorrow and we'll see if that's going to fix the issue ...

[AlAleksiev]

The additinal memory is added and that's the result from two long-time stress tests

After more than 9 hours (overnight) running on-line SilverBench stress test from SystemRescue live Arch linux:


More than 4 hours running on-line CPU Expert stress test from Mint live installation (Debian based like TOS):


In both cases the hardware is stable and responsive.
Now starting TOS in 14:54:28 EST - debug enabled.

It survived only 15 minutes - the crash came in 15:09:48 EST.
This is in the moment of the crash - all parameters looks good: CPU, RAM, Network, Disks activity, Docker containers


It should be something with the TOS itself, right?

[AlAleksiev]

New Install - after Factory Default (and configuration backup)

eth0 mac: 6cbf.b501.5f4c assigned initial ip 192.168.0.163 (dhcp 016c.bfb5.015f.4c)
eth1 - disconnected

Note: No one of the specified in the post (viewtopic.php?t=6433) initboot images is accepted!

The recommended "Online download" way used - the console log shows: initBoot (567)

Same method was used for the Bootloader. That one was staying on 99% ready for more than 10 minutes. On the console we have that


Specified local (md5 verified) TOS image:


The TOS installation was waiting longer on 60% (maybe the existing arrays were resyncing) then jumped on 92%, 100%

On the console we have that

finally reload...

EULA confirmed, user created, security email - code passed

Security Advisor initiated: HTTP(S) ports, SPC, Isolation and Snapshot at risk
- for HTTP(S) ports changed to 7171 / 7443
Waiting to validate stability

5 minutes later - it shutdown himself with no reload this time.

Any suggestions for next step?

[AlAleksiev]

As suggested on a chat session, enabled the debug and sent the log to [email protected]

... waiting for technical opinion ...

[TMzethar]

Hi.
Many users who use this NAS model and system, like you, have not encountered similar issues. The available information does not clearly indicate the source of the problem.
Next, it may be necessary to consider checking power supply-related issues. Are you currently connected to a battery/UPS instead of mains power?
Is the power adapter original? Is its LED light bright? What is its output voltage (V)?
If you have an adapter with 1.5 times or even higher output power than the original one for the 12V 5.5*2.5 interface, you can try replacing it for testing.

[AlAleksiev]

Already verified that in a simple way - System Rescue live USB running more that 24 hours and the NAS is in the same conditions:
- same UPS (for any power variations ^ outlet problems)
- same ethernet bond (for any load-balancing related problems)
- same switch on the same ports (for any static electricity problems)
- same KVM

I enabled samba on the System Rescue (Arch linux actually), mounted the existing LVs and shared them on the local network.
Everything is working stable and fast.

Few photos to give you more details:




Any other suggestions?