Marvell AQtion (atlantic) NIC in F4-424 MAX not supporting GRO

[awl29]

Hello TerraMaster Engineering,

I am just wondering whether some of the SMB speed issues with TOS 6.0 and the new devices based on the 10GBE "Marvell AQtion (aka Aquantia AQC113CS) Network Adapter" (Linux "atlantic") driver are related to the obvious inability to support Generic Receive Offload (GRO) with the current driver:

Using F4-424 MAX, latest TOS 6.0.519-00144 ("Linux f4-424max 6.1.58+ #208 SMP PREEMPT_DYNAMIC Sun Sep 29 09:49:15 CST 2024 x86_64 x86_64 x86_64 GNU/Linux") - atlantic driver version unfortunately unknown, "lspci" returns

03:00.0 "0200" "1d6a" "14c0" -r03 "1d6a" "0001"
04:00.0 "0200" "1d6a" "14c0" -r03 "1d6a" "0001"

Capabilities: [d0] Vital Product Data
Product Name: Marvell AQtion Network Adapter
Read-only fields:
[PN] Part number: 00B1A113
[V0] Vendor specific: MAC Addr: 6C:BF:B5:04:0D:79
[V1] Vendor specific: Bundle Version: 1.3.30
[V2] Vendor specific: Fw Version: 1.0.120
[RV] Reserved: checksum good, 0 byte(s) reserved
End

The current combination of adaper and kernel driver DOES NOT seem to SUPPORT GRO:

from: https://stackoverflow.com/a/47423189
"Generic Receive Offload (GRO) is a software technique in Linux to aggregate multiple incoming packets belonging to the same stream. The linked article claims that CPU utilization is reduced because, instead of each packet traversing the network stack individually, a single aggregated packet traverses the network stack. (...)
As you can see, GRO aggregation is at the very bottom of the call stack. You can also see how much work is done afterwards. Imagine how much overhead you'll have if each of these functions would operate on a single MTU."

tnasadm@f4-424max:/# dmesg | grep GRO
[ 68.137904] TCP: bond0: Driver has suspect GRO implementation, TCP performance may be compromised.

tnasadm@f4-424max:/# ethtool -k bond0 | grep gro
rx-gro-hw: off [fixed]
rx-gro-list: off
rx-udp-gro-forwarding: off

tnasadm@f4-424max:/# ethtool -k eth0 | grep gro
rx-gro-hw: off [fixed]
rx-gro-list: off
rx-udp-gro-forwarding: off

tnasadm@f4-424max:/# ethtool -k eth1 | grep gro
rx-gro-hw: off [fixed]
rx-gro-list: off
rx-udp-gro-forwarding: off

and shows PACKET LOSS (a relatively large number of packets dropped) in a balance-alb bond setup:

tnasadm@f4-424max:/# netstat -i
Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
bond0 1500 2429303 0 497542 0 1040754 0 1 0 BMmRU
docker0 1500 0 0 0 0 0 0 0 0 BMU
eth0 1500 1829523 0 0 0 539708 0 0 0 BMsRU
eth1 1500 599738 0 0 0 501001 0 0 0 BMsRU
lo 65536 1746289 0 0 0 1746289 0 0 0 LRU


Also, it looks like the firmware of the Marvell AQtion (aka Aquantia AQC113CS) NICs is ancient (from lspci, assuming it's either 1.3.30 or even 1.0.120 as reported there) - but station-drivers.com has:

Marvell/Aquantia AQC113CS (10Gbps) Firmware Version 1.5.42 (Windows-only firmware update, but seemingly appropriate for the devices in the F4-424 MAX), as can be seen from updatedata.xml:

<pciid vid="1d6a" did="04c0" sdid="0001" svid="1d6a" />

(1) Are these issues already known to TerraMaster Engineering/ Support?
(2) What is your plan to address/fix them to get GRO working?
(3) Is it supported (i.e. will it keep my warranty intact) to boot into Windows from USB NVMe and update the NIC's firmware to the version above?

Looking forward to your feedback on the whole issue, especially all 3 questions above...

Many thanks!
awl29

[TMLeeh]

Thank you for your feedback. We will address this issue to the product team.

[awl29]

It's one week tomorrow that I have created this post...

TerraMaster Support/Engineering: Please provide answers the questions (1) to (3) that I have raised here instead of simply buying some time...

Many thanks!
awl29

[awl29]

It's one week tomorrow that I have created this post...

TerraMaster Support/Engineering: Please provide answers the questions (1) to (3) that I have raised here instead of simply buying some time...

Many thanks!
awl29

Another week has passed - is anybody at TerraMaster willing/able to to reply, or do you have something to hide here wirth regards to the 10GbE NIC interfaces in your devices?

Many thanks for - finally - an answer providing the requested technical details...

BR
awl29

[bugacha]

Hello TerraMaster Engineering,

I am just wondering whether some of the SMB speed issues

I dont have same drops as you're. I'm running a single 10GB-T and it works fine

Code: Select all

# netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0      9000  4510110      0      0 0       8156517      0      0      0 BMRU
eth1      9000 18622823      0      0 0         13522      0      0      0 BMRU

Code: Select all

iperf3.exe -c 192.168.1.4                                                                                                                                                                                           Connecting to host 192.168.1.4, port 5201                                                                                                                                                                                                       [  5] local 192.168.1.16 port 51911 connected to 192.168.1.4 port 5201                                                                                                                                                                          [ ID] Interval           Transfer     Bitrate                                                                                                                                                                                                   [  5]   0.00-1.01   sec  1.16 GBytes  9.94 Gbits/sec                                                                                                                                                                                            [  5]   1.01-2.01   sec  1.16 GBytes  9.91 Gbits/sec                                                                                                                                                                                            [  5]   2.01-3.00   sec  1.14 GBytes  9.86 Gbits/sec                                                                                                                                                                                            [  5]   3.00-4.01   sec  1.17 GBytes  9.91 Gbits/sec                                                                                                                                                                                            [  5]   4.01-5.00   sec  1.14 GBytes  9.90 Gbits/sec                                                                                                                                                                                            [  5]   5.00-6.01   sec  1.16 GBytes  9.89 Gbits/sec                                                                                                                                                                                            [  5]   6.01-7.00   sec  1.15 GBytes  9.93 Gbits/sec                                                                                                                                                                                            [  5]   7.00-8.00   sec  1.15 GBytes  9.91 Gbits/sec                                                                                                                                                                                            [  5]   8.00-9.01   sec  1.16 GBytes  9.91 Gbits/sec                                                                                                                                                                                            [  5]   9.01-10.01  sec  1.15 GBytes  9.89 Gbits/sec                                                                                                                                                                                            - - - - - - - - - - - - - - - - - - - - - - - - -                                                                                                                                                                                               [ ID] Interval           Transfer     Bitrate                                                                                                                                                                                                   [  5]   0.00-10.01  sec  11.5 GBytes  9.90 Gbits/sec                  sender                                                                                                                                                                    [  5]   0.00-10.01  sec  11.5 GBytes  9.90 Gbits/sec                  receiver    

[bugacha]

corrected iperf3

Code: Select all

iperf3.exe -c 192.168.1.4
Connecting to host 192.168.1.4, port 5201
[  5] local 192.168.1.16 port 51911 connected to 192.168.1.4 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  1.16 GBytes  9.94 Gbits/sec
[  5]   1.01-2.01   sec  1.16 GBytes  9.91 Gbits/sec
[  5]   2.01-3.00   sec  1.14 GBytes  9.86 Gbits/sec
[  5]   3.00-4.01   sec  1.17 GBytes  9.91 Gbits/sec
[  5]   4.01-5.00   sec  1.14 GBytes  9.90 Gbits/sec
[  5]   5.00-6.01   sec  1.16 GBytes  9.89 Gbits/sec
[  5]   6.01-7.00   sec  1.15 GBytes  9.93 Gbits/sec
[  5]   7.00-8.00   sec  1.15 GBytes  9.91 Gbits/sec
[  5]   8.00-9.01   sec  1.16 GBytes  9.91 Gbits/sec
[  5]   9.01-10.01  sec  1.15 GBytes  9.89 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  11.5 GBytes  9.90 Gbits/sec                  sender
[  5]   0.00-10.01  sec  11.5 GBytes  9.90 Gbits/sec                  receiver    

[bugacha]

Looks like the latest firmware is 1.5.42

https://www.station-drivers.com/index.p ... ang,en-us/

https://www.station-drivers.com/index.p ... ion-1-5-42

and we are on 1.3.30

Code: Select all

ethtool -i eth0
driver: atlantic
version: 6.1.58+
firmware-version: 1.3.30

Flash requires Windows, I wonder if can be done via VM

[bugacha]

They have a bootable Windows image as well

https://www.station-drivers.com/index.p ... d-bootable

I wonder if all one need is to flash that, boot with HDMI/Keyboard and update it

[bugacha]

Hello TerraMaster Engineering,

I am just wondering whether some of the SMB speed issues with TOS 6.0 and the new devices based on the 10GBE "Marvell AQtion (aka Aquantia AQC113CS) Network Adapter" (Linux "atlantic") driver are related to the obvious inability to support Generic Receive Offload (GRO) with the current driver:

Hey mate, so I put TerraMaster Engineering hat after your post and actually tried to update firmware from links I posted above.

Long story short = no luck, F4-424 MAX card's revision is not listed in update.xml

when I run their update tool I get this ID: 0x1D6A:0x14C0:0x1D61:0x0001

I looked through several firmwares : 1.5.42, 1.5.39, 1.3.36 - I don't see this card in update.xml file and hence update tool doesn't recongnize it

There is always an option to add it manually - but I'm afraid to brick the card.

Steps I took :

1. Make USB bootable drive with WinPE and Powershell

2. Boot and open powershell

3. Run flashUpdate2 -i -v

[bugacha]

There is some discussion here with guys having same 0x0001 last 4th ID

https://www.station-drivers.com/index.p ... rt=30#2878