[Help] NFS Hangs after a day or two

[mossholderm]

Hi All,
I am running 6.0.650, and am having some persistent issues with NFS. After about a day or two, mounts on client stop working. On the TNAS side, attempting to restart the nfs-server service using systemctl doesn't work, resulting in the following logs:

Code: Select all

Apr 03 06:56:08 TNAS systemd[1]: Finished NFS server and services.
Apr 04 07:26:52 TNAS systemd[1]: Stopping NFS server and services...
Apr 04 07:28:22 TNAS systemd[1]: nfs-server.service: Stopping timed out. Terminating.
Apr 04 07:29:53 TNAS systemd[1]: nfs-server.service: State 'stop-sigterm' timed out. Killing.
Apr 04 07:29:53 TNAS systemd[1]: nfs-server.service: Killing process 2086395 (rpc.nfsd) with signal
 SIGKILL.
Apr 04 07:31:23 TNAS systemd[1]: nfs-server.service: Processes still around after SIGKILL. Ignoring.
Apr 04 07:32:34 TNAS systemd[1]: nfs-server.service: Sent signal SIGTERM to control process 2092636 (exportfs) on client request.
Apr 04 07:32:34 TNAS systemd[1]: nfs-server.service: Sending signal SIGTERM to process 2086395 (rpc.nfsd) on client request.
Apr 04 07:32:53 TNAS systemd[1]: nfs-server.service: State 'stop-post' timed out. Terminating.

And `systemctl status nfs-server` shows:

Code: Select all

● nfs-server.service - NFS server and services
     Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
    Drop-In: /run/systemd/generator/nfs-server.service.d
             └─order-with-mounts.conf
     Active: deactivating (stop-post) (Result: timeout) since Fri 2025-04-04 07:26:52 EDT; 5min ago
    Process: 275342 ExecStartPre=/usr/share/nfs-kernel-server/nfs_prepare (code=exited, status=0/SUCCESS)
    Process: 275512 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
    Process: 275513 ExecStart=/usr/sbin/rpc.nfsd --nfs-version $EXECVERSION $RPCNFSDCOUNT (code=exited, status=0/SUCCESS)
   Main PID: 275513 (code=exited, status=0/SUCCESS); Control PID: 2092636 (exportfs)
      Tasks: 2 (limit: 4461)
     Memory: 552.0K
        CPU: 5ms
     CGroup: /system.slice/nfs-server.service
             ├─2086395 /usr/sbin/rpc.nfsd 0
             └─2092636 /usr/sbin/exportfs -au

It doesn't look like it is able to kill rpc.nfsd or exportfs, for whatever reason. Earlier logs are all related to the initial exports (`Assuming default behaviour ('no_subtree_check')` and related messages). No errors. Nothing in `dmesg` that appears unusual either.

Any thoughts?

[TMzethar]

Are there any relevant log entries on your device that is mounted with an NFS disk from a NAS?
When you stop working, have you noticed any other abnormalities? You can try mounting a new NFS disk; if this works, it suggests that the issue is likely related to some kind of change.

[mossholderm]

No messages on the client. It seems like NFS is working just enough to hang the client. New mount attempts just hang.

iSCSI still seems to work... some of the NFS clients are VMs running off iSCSI LUNs on the same filer , with no issues other than NFS stops working.

The only other bit of information that has cropped up is that it appears to be load based. I shut down my Frigate container (a video camera recording application), which causes a significant portion of the NFS traffic, and the NFS server has now been alive for 3 days.

[Kisuyane]

No client-side logs, meaning the issue is likely server-side or network path-level.

slope 2

[carample]

nfsd or rpcbind may have stopped working due to resource exhaustion or being killed. NFS may time out if the connection is idle for too long. If there is a NAT or firewall in between, temporary ports may expire. Are you using NFS v3 or v4?

[TMnight]

Please execute the following command in the terminal to output the startup log of the NFS service to the /Volume1/public/nfs.log file:

Code: Select all

journalctl -u nfs-server.service > /Volume1/public/nfs.log

At the same time, please follow the TOS6 operation method provided in the following link to enter Support and Help > Debug Mode to enable debugging to collect system logs: viewtopic.php?t=6446.

After completion, please send the generated log file to us by email so that we can further analyze the cause of the problem.