Error: failed to lock page in memory

The system is up and running. I can ping server from client side and ping client from server side.
qianyich@pc164:~/RecoNIC/drivers/onic-driver$ ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=64 time=0.238 ms
64 bytes from <IP_ADDRESS>: icmp_seq=2 ttl=64 time=0.092 ms
64 bytes from <IP_ADDRESS>: icmp_seq=3 ttl=64 time=0.164 ms
64 bytes from <IP_ADDRESS>: icmp_seq=4 ttl=64 time=0.166 ms
64 bytes from <IP_ADDRESS>: icmp_seq=5 ttl=64 time=0.140 ms
64 bytes from <IP_ADDRESS>: icmp_seq=6 ttl=64 time=0.104 ms
64 bytes from <IP_ADDRESS>: icmp_seq=7 ttl=64 time=0.126 ms
64 bytes from <IP_ADDRESS>: icmp_seq=8 ttl=64 time=0.115 ms
64 bytes from <IP_ADDRESS>: icmp_seq=9 ttl=64 time=0.108 ms
^C
--- <IP_ADDRESS> ping statistics ---
9 packets transmitted, 9 received, 0% packet loss, time 8193ms
rtt min/avg/max/mdev = 0.092/0.139/0.238/0.043 ms

qianyich@pc166:~/RecoNIC/drivers/onic-driver$ ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=64 time=0.167 ms
64 bytes from <IP_ADDRESS>: icmp_seq=2 ttl=64 time=0.152 ms
64 bytes from <IP_ADDRESS>: icmp_seq=3 ttl=64 time=0.154 ms
64 bytes from <IP_ADDRESS>: icmp_seq=4 ttl=64 time=0.161 ms
64 bytes from <IP_ADDRESS>: icmp_seq=5 ttl=64 time=0.154 ms
64 bytes from <IP_ADDRESS>: icmp_seq=6 ttl=64 time=0.155 ms
^C
--- <IP_ADDRESS> ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5110ms
rtt min/avg/max/mdev = 0.152/0.157/0.167/0.008 ms

When I was trying to run rdma_test read and write, I have the following error.
qianyich@pc164:~/RecoNIC/examples/rdma_test$ sudo env LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./write -r <IP_ADDRESS> -i <IP_ADDRESS> -p /sys/bus/pci/devices/0000\:3b\:00.0/resource2 -z 128 -l host_mem -d /dev/reconi
c-mm -c -u 22222 -t 11111 --dst_qp 2 -g 2>&1 | tee client_debug.log
src_ip_str = <IP_ADDRESS>
dst_ip_str = <IP_ADDRESS>
Info: mac_addr_t = 00:0a:35:fd:c0:a8
Info: PCIe resource file: /sys/bus/pci/devices/0000:3b:00.0/resource2
Info: QP allocated at: host_mem
Info: Device - /dev/reconic-mm
Info: src_ip = <IP_ADDRESS>
Info: Found network interface: enp59s0
Info: mac_addr_t = 00:0a:35:c0:8c:15
Info: Creating rn_dev
/users/qianyich/RecoNIC/lib/reconic.c:296:create_rn_dev(): Info: scr(=4)) file open successfully
create_rn_dev - testing2
Error: failed to lock page in memory

I found this in lib/reconic.c:323. Is this due to insufficient huge page? I guess I need to enable and configure the number of huge pages in Linux. How many huge pages do I need?
Currently:
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

After configure the hugepage number to 1024. I have the following error:
qianyich@pc164:~/RecoNIC/examples/rdma_test$ sudo env LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./write -r <IP_ADDRESS> -i <IP_ADDRESS> -p /sys/bus/pci/devices/0000\:3b\:00.0/resource2 -z 128 -l dev_mem -d /dev/reconic-mm -c -u 22222 -t 11111 --dst_qp 2 -g 2>&1 | tee client_debug.log
src_ip_str = <IP_ADDRESS>
dst_ip_str = <IP_ADDRESS>
Info: mac_addr_t = 00:0a:35:fd:c0:a8
Info: PCIe resource file: /sys/bus/pci/devices/0000:3b:00.0/resource2
Info: QP allocated at: dev_mem
Info: Device - /dev/reconic-mm
Info: src_ip = <IP_ADDRESS>
Info: Found network interface: enp59s0
Info: mac_addr_t = 00:0a:35:c0:8c:15
Info: Creating rn_dev
/users/qianyich/RecoNIC/lib/reconic.c:296:create_rn_dev(): Info: scr(=4)) file open successfully
create_rn_dev - testing2
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x2f54e00
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x2f54e00000
Info: pre-allocated hugepage buffer vir addr = 0x7f0b0aa00000, physical addr = 0x2f54e00000
Info: Configuring QDMA AXI bridge BDF
/users/qianyich/RecoNIC/lib/reconic.c:194:config_rn_dev_axib_bdf(): [BDF] AXIB_BDF_ADDR_TRANSLATE_ADDR_LSB=0x16420, bdf_addr_low=0x0
/users/qianyich/RecoNIC/lib/reconic.c:195:config_rn_dev_axib_bdf(): [BDF] AXIB_BDF_ADDR_TRANSLATE_ADDR_MSB=0x16424, bdf_addr_high=0x20
/users/qianyich/RecoNIC/lib/reconic.c:196:config_rn_dev_axib_bdf(): [BDF] AXIB_BDF_MAP_CONTROL_ADDR=0x16430, bdf_win_config=0xc2000000
Info: CREATE RDMA DEVICE
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x2f54e00
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x2f54e00000
/users/qianyich/RecoNIC/lib/reconic.c:227:allocate_rdma_buffer(): Info: allocated host buffer vir addr = 0x7f0b0aa00000, physical addr = 2f54e00000, rn_dev->buffer_offset = 0x200000
/users/qianyich/RecoNIC/lib/reconic.c:228:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma host buffer
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x2f54c00
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x2f54c00000
/users/qianyich/RecoNIC/lib/reconic.c:227:allocate_rdma_buffer(): Info: allocated host buffer vir addr = 0x7f0b0ac00000, physical addr = 2f54c00000, rn_dev->buffer_offset = 0x1200000
/users/qianyich/RecoNIC/lib/reconic.c:228:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma host buffer
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x2f55c00
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x2f55c00000
/users/qianyich/RecoNIC/lib/reconic.c:227:allocate_rdma_buffer(): Info: allocated host buffer vir addr = 0x7f0b0bc00000, physical addr = 2f55c00000, rn_dev->buffer_offset = 0x1202000
/users/qianyich/RecoNIC/lib/reconic.c:228:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma host buffer
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x2f55c02
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x2f55c02000
/users/qianyich/RecoNIC/lib/reconic.c:227:allocate_rdma_buffer(): Info: allocated host buffer vir addr = 0x7f0b0bc02000, physical addr = 2f55c02000, rn_dev->buffer_offset = 0x1212000
/users/qianyich/RecoNIC/lib/reconic.c:228:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma host buffer
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x2f55c12
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x2f55c12000
/users/qianyich/RecoNIC/lib/reconic.c:227:allocate_rdma_buffer(): Info: allocated host buffer vir addr = 0x7f0b0bc12000, physical addr = 2f55c12000, rn_dev->buffer_offset = 0x1222000
/users/qianyich/RecoNIC/lib/reconic.c:228:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma host buffer
Info: OPEN RDMA DEVICE
/users/qianyich/RecoNIC/lib/rdma_api.c:186:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_DATBUFBA=0x600a0, value=0x54c00000
/users/qianyich/RecoNIC/lib/rdma_api.c:188:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_DATBUFBAMSB=0x600a4, value=0xf
/users/qianyich/RecoNIC/lib/rdma_api.c:190:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_DATBUFSZ=0x600a8, value=0x10001000
/users/qianyich/RecoNIC/lib/rdma_api.c:193:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_IPKTERRQBA=0x60088, value=0x55c00000
/users/qianyich/RecoNIC/lib/rdma_api.c:195:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_IPKTERRQBAMSB=0x6008c, value=0xf
/users/qianyich/RecoNIC/lib/rdma_api.c:197:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_ERRBUFSZ=0x60090, value=0x2000
/users/qianyich/RecoNIC/lib/rdma_api.c:200:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_ERRBUFBA=0x60060, value=0x55c02000
/users/qianyich/RecoNIC/lib/rdma_api.c:202:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_ERRBUFBAMSB=0x60064, value=0xf
/users/qianyich/RecoNIC/lib/rdma_api.c:204:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_ERRBUFSZ=0x60068, value=0x1000100
/users/qianyich/RecoNIC/lib/rdma_api.c:207:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_RESPERRPKTBA=0x600b0, value=0x55c12000
/users/qianyich/RecoNIC/lib/rdma_api.c:209:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_RESPERRPKTBAMSB=0x600b4, value=0xf
/users/qianyich/RecoNIC/lib/rdma_api.c:211:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_RESPERRSZ=0x600b8, value=0x10000
/users/qianyich/RecoNIC/lib/rdma_api.c:213:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_RESPERRSZMSB=0x600bc, value=0x0
/users/qianyich/RecoNIC/lib/rdma_api.c:217:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_INTEN=0x60180, value=0xff
/users/qianyich/RecoNIC/lib/rdma_api.c:221:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_MACXADDLSB=0x60010, value=0x35c08c15
/users/qianyich/RecoNIC/lib/rdma_api.c:223:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_MACXADDMSB=0x60014, value=0xa
/users/qianyich/RecoNIC/lib/rdma_api.c:227:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_IPV4XADD=0x60070, value=0xc0643301
/users/qianyich/RecoNIC/lib/rdma_api.c:230:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_XRNICCONF=0x60000, value=0x56ce1421
/users/qianyich/RecoNIC/lib/rdma_api.c:233:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_XRNICADCONF=0x60004, value=0xa0004
Info: RDMA global control status registers are configured.
Info: rdma_dev opened
Info: ALLOCATE PD
/users/qianyich/RecoNIC/lib/rdma_api.c:254:allocate_rdma_pd(): [Register] RN_RDMA_PDT_PDPDNUM=0x40000, pd_num=0, value=0x0
Info: OPEN DEVICE FILE
Info: ALLOCATE RDMA QP
Allocating qp->sq
/users/qianyich/RecoNIC/lib/rdma_api.c:437:allocate_rdma_qp(): sq_size = 81920, cq_size = 5120, rq_size 655360, buf_location = dev_mem
/users/qianyich/RecoNIC/lib/reconic.c:251:allocate_rdma_buffer(): Info: allocated device buffer physical addr = a350000000000000, rn_dev->dev_buffer_offset = 0x14000
/users/qianyich/RecoNIC/lib/reconic.c:253:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma device buffer
Allocating qp->cq
/users/qianyich/RecoNIC/lib/reconic.c:251:allocate_rdma_buffer(): Info: allocated device buffer physical addr = a350000000014000, rn_dev->dev_buffer_offset = 0x15400
/users/qianyich/RecoNIC/lib/reconic.c:253:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma device buffer
Allocating qp->rq
/users/qianyich/RecoNIC/lib/reconic.c:251:allocate_rdma_buffer(): Info: allocated device buffer physical addr = a350000000016000, rn_dev->dev_buffer_offset = 0xb6000
/users/qianyich/RecoNIC/lib/reconic.c:253:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma device buffer
Info: queue pair setting is done! Configuring RDMA per-queu CSR registers
/users/qianyich/RecoNIC/lib/rdma_api.c:487:allocate_rdma_qp(): DEBUG: rdma_dev->rn_dev->axil_ctl = 0x7f0b2aa4c000, rdma_dev->axil_ctl = 0x7f0b2aa4c000
/users/qianyich/RecoNIC/lib/rdma_api.c:502:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_IPDESADDR1i=0x60360, qpid=2, value=0xc0643401
/users/qianyich/RecoNIC/lib/rdma_api.c:509:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_MACDESADDLSBi=0x60350, qpid=2, value=0x35fdc0a8
/users/qianyich/RecoNIC/lib/rdma_api.c:516:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_MACDESADDMSBi=0x60354, qpid=2, value=0xa
/users/qianyich/RecoNIC/lib/rdma_api.c:521:allocate_rdma_qp(): DEBUG: win_size_high = 0x1f, win_size_low = 0xffffffff
/users/qianyich/RecoNIC/lib/rdma_api.c:539:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_SQBAi=0x60310, qpid=2, value=0x0
/users/qianyich/RecoNIC/lib/rdma_api.c:546:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_SQBAMSBi=0x603c8, qpid=2, value=0xa3500000
/users/qianyich/RecoNIC/lib/rdma_api.c:550:allocate_rdma_qp(): DEBUG: qp->sq->dma_addr = 0xa350000000000000, sq_addr_msb = 0xa3500000, sq_addr_lsb = 0x0
/users/qianyich/RecoNIC/lib/rdma_api.c:568:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_CQBAi=0x60318, qpid=2, value=0x14000
/users/qianyich/RecoNIC/lib/rdma_api.c:575:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_CQBAMSBi=0x603d0, qpid=2, value=0xa3500000
/users/qianyich/RecoNIC/lib/rdma_api.c:579:allocate_rdma_qp(): DEBUG: qp->cq->dma_addr = 0xa350000000014000, cq_addr_msb = 0xa3500000, cq_addr_lsb = 0x14000
/users/qianyich/RecoNIC/lib/rdma_api.c:597:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_RQBAi=0x60308, qpid=2, value=0x16000
/users/qianyich/RecoNIC/lib/rdma_api.c:604:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_RQBAMSBi=0x603c0, qpid=2, value=0xa3500000
/users/qianyich/RecoNIC/lib/rdma_api.c:608:allocate_rdma_qp(): DEBUG: qp->rq->dma_addr = 0xa350000000016000, rq_addr_msb = 0xa3500000, rq_addr_lsb = 0x16000
/users/qianyich/RecoNIC/lib/rdma_api.c:617:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_CQDBADDi=0x60328, qpid=2, value=0x54e00000
/users/qianyich/RecoNIC/lib/rdma_api.c:624:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_CQDBADDMSBi=0x6032c, qpid=2, value=0xf
/users/qianyich/RecoNIC/lib/rdma_api.c:625:allocate_rdma_qp(): DEBUG: cq_cidb_addr = 0x2f54e00000
/users/qianyich/RecoNIC/lib/rdma_api.c:634:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_RQWPTRDBADDi=0x60320, qpid=2, value=0x54e00020
/users/qianyich/RecoNIC/lib/rdma_api.c:641:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_RQWPTRDBADDMSBi=0x60324, qpid=2, value=0xf
/users/qianyich/RecoNIC/lib/rdma_api.c:642:allocate_rdma_qp(): DEBUG: rq_cidb_addr = 0x2f54e00020
/users/qianyich/RecoNIC/lib/rdma_api.c:651:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_DESTQPCONFi=0x60348, qpid=2, value=0x2
/users/qianyich/RecoNIC/lib/rdma_api.c:660:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_QDEPTHi=0x6033c, qpid=2, value=0x400040
/users/qianyich/RecoNIC/lib/rdma_api.c:707:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_QPCONFi=0x60300, qpid=2, value=0x200043d
/users/qianyich/RecoNIC/lib/rdma_api.c:721:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_QPADVCONFi=0x60304, qpid=2, value=0x12344000
/users/qianyich/RecoNIC/lib/rdma_api.c:730:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_PDi=0x603b0, qpid=2, value=0x0
Info: allocate_rdma_qp - Successfully allocated a rdma qp
Info: CONFIGURE PSN
[Register] RN_RDMA_QCSR_LSTRQREQi=0x60344, qpid=2, value=0xa000abc
[Register] RN_RDMA_QCSR_SQPSNi=0x60340, qpid=2, value=0xabd
payload_size = 128, payload_size>>2 = 32
Info: Client is connecting to a remote server
Info: Client is connected to a remote server
Error: Can't receive remote offset of A from the remote peer

This time looks like the error is from the read application at line 319, rc = read(sockfd, &read_A_offset, sizeof(read_A_offset)); returns a value that is not over 0. And I am kind of confused with why socket is involved here? My understanding is that RDMA has nothing to do with socket.

Hi Yicheng,
Could you double check whether your TCP is working fine?
To set up RDMA, we need to first have the QP connection, which requires two peers to exchange some sort of information. The information includes such as r_key, remote addresses and psn. To exchange the information, it can be done via connection management with QP0 or TCP. We are using TCP for exchanging such information.

Hi Guanwen,
I used tcping between my two servers, and it just works fine.
8 probes transmitted on port 22 | 8 received, 0.00% packet loss
successful probes:   8
unsuccessful probes: 0
last successful probe:   2024-03-21 22:26:29
last unsuccessful probe: Never failed
total uptime:   8 seconds
total downtime: 0.0 seconds
longest consecutive uptime:   7 seconds from 2024-03-21 22:26:22 to 2024-03-21 22:26:30
retried to resolve hostname 0 times
rtt min/avg/max: 0.223/0.266/0.324 ms
--------------------------------------
TCPing started at: 2024-03-21 22:26:22
TCPing ended at:   2024-03-21 22:26:30
duration (HH:MM:SS): 00:00:08

Oh I guess you actually mean check TCP connection between <IP_ADDRESS> and <IP_ADDRESS>. So I should use tcping <IP_ADDRESS> in this case. Let me do this and will get back to you.

I saw others have the same issue. https://github.com/Xilinx/RecoNIC/issues/12#issuecomment-1986413626 Do you have any suggestions on solving this TCP handshaking issue?

Hi Yicheng,
Could you comment out all the code after exchanging the remote offset and see what's the issue? Actually, at this point, your program hasn't reached to the actual RDMA part, only a few RDMA register confgurations.

From which line to which line? Do you mean comment out all actual rdma part(wqe,etc) or this problematic part? Please bear with me, I am still trying to understand what those codes are trying to do.
When I did tcping, should I use 11111 as the port number? I saw tcp port is 11111 in the command arguments, but I did not verify if that works or not.

Hello,
Can you try with a larger payload size like 4096 or above by changing argument -z from 128 to 4096 in your command
Aditya.

Hi Aditya,
I did reprogram them every time before running the test. I will change the payload size and try today, but it seems like the problem is the TCP handshaking.
Yicheng

I think the problem is with the dma engine. I ran the dma test but it failed on reading from the device memory.
qianyich@pc167:~/RecoNIC/examples/dma_test$ sudo ./dma_test -d /dev/reconic-mm -s 65536000 -c 200
Write scenario
size=65536000 Average BW = 10.629003 GB/sec, average latency = 6165.771279 us
qianyich@pc167:~/RecoNIC/examples/dma_test$ sudo ./dma_test -d /dev/reconic-mm -s 65536000 -c 200 -r
Read scenario
/dev/reconic-mm, read off 0x0 + 0x3e80000 failed -1.
read file: Invalid argument

Then I see that the read_to_buffer always fails in read, write, systolic array test (rc == -5 returned from read_to_buffer). For the systolic array, I have the following logs:
client_debug.log
server_debug.log
client:
sudo env LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./network_systolic_mm -d /dev/reconic-mm -p /sys/bus/pci/devices/0000:3b:00.0/resource2 -r <IP_ADDRESS> -i <IP_ADDRESS> -u 22222 -t 11111 --dst_qp 2 -c 2>&1 | tee client_debug.log
server:
sudo env LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./network_systolic_mm -d /dev/reconic-mm -p /sys/bus/pci/devices/0000:3b:00.0/resource2 -r <IP_ADDRESS> -i <IP_ADDRESS> -u 22222 -t 11111 --dst_qp 2 -s 2>&1 | tee server_debug.log

@qianyich
Hi Yicheng,
Check your log below:
/users/qianyich/RecoNIC/lib/rdma_api.c:550:allocate_rdma_qp(): DEBUG: qp->sq->dma_addr = 0x2f53422000, sq_addr_msb = 0xf, sq_addr_lsb = 0x53422000
In RecoNIC's QDMA setup, we use 1TB as the AXI BAR0 (which uses PCIe slave bridge to read/write host data) and it gets further splited into 8 windows equally. This means that the BDF mask is for 128GB. I guess you're using 256GB system, right? Your dma_address is not matched with {sq_addr_msb, sq_addr_lsb}, which means that the hardware can't read any WQE from the SQ. That's why you can make it work.
To enable 256GB mask, you need to configure the BDF table's 0x2430 register with 0xC400_0000. You can check the information here.
Regarding the DMA engine issue, probably your hardware configuration or connection has some problems when porting to U280. We never encounter this issue. What you could do is to check the built-in example along with the QDMA 4.0 on U280, see any special configuration. To build the example, please check this.

Hi Guanwen,
I took a look at this https://support.xilinx.com/s/article/Demystifying-BDF-Table-programming-for-Slave-Bridge-Address-Translations-for-AXI-address?language=en_US. And it does help me understand this better. I understand the address above goes over the range of 128 GB, and the S_AXI_BRIDGE is set to be 1TB, but I am not sure if I am using 256 GB, how can I check this and how can I write 0x2430 with C4000000 (I do understand it is from 256 * 1024 * 1024 / 4)?
Will check the configuration of the example QDMA project later and keep you posted.

@qianyich I mean your server probably has 256GB host memory, that's why you will have addresses above 128GB range when you allocate memory.
To write 0x2430, please check this line. "AXIB_BDF_MAP_CONTROL_ADDR" is 0x2430 register.

@zhguanw-amd
Hi Guanwen,
I generated the example QDMA design, and tested it with dma_ip_drivers. My script tests the functionality of QDMA.
qdma3b000       0000:3b:00.0    max QP: 32, 0~31
=============Hardware Version============

RTL Version         : RTL Base
Vivado ReleaseID    : vivado 2020.2
QDMA Device Type    : Soft IP
QDMA IP Type    : EQDMA4.0 Soft IP
============Software Version============

qdma driver version : 2<IP_ADDRESS>.

=============Hardware Capabilities============

Number of PFs supported                : 1
Total number of queues supported       : 512
MM channels                            : 1
FLR Present                            : no
ST enabled                             : yes
MM enabled                             : yes
Mailbox enabled                        : no
MM completion enabled                  : no
Debug Mode enabled                     : no
Desc Engine Mode                       : Inernal only mode
qdma3b000:statistics
Total MM H2C packets processed = 17000
Total MM C2H packets processed = 17
Total ST H2C packets processed = 0
Total ST C2H packets processed = 0
Min Ping Pong Latency = 0
Max Ping Pong Latency = 0
Avg Ping Pong Latency = 0
-e
List Xilinx PCIe QDMA queues.
Zero Qs-e
Test Xilinx PCIe QDMA h2c channel.
qdma3b000-MM-0 H2C added.
Added 1 Queues.
/dev/qdma3b000-MM-0
dma-ctl: Info: Default ring size set to 2048
1 Queues started, idx 0 ~ 0.
size=4096 Average BW = 368.020416 MB/sec
size=65536 Average BW = 3.080631 GB/sec
Stopped Queues 0 -> 0.
Deleted Queues 0 -> 0.
qdma3b000:statistics
Total MM H2C packets processed = 34000
Total MM C2H packets processed = 17
Total ST H2C packets processed = 0
Total ST C2H packets processed = 0
Min Ping Pong Latency = 0
Max Ping Pong Latency = 0
Avg Ping Pong Latency = 0
-e
Test Xilinx PCIe QDMA c2h channel.
qdma3b000-MM-0 C2H added.
Added 1 Queues.
/dev/qdma3b000-MM-0
dma-ctl: Info: Default ring size set to 2048
1 Queues started, idx 0 ~ 0.
size=4096 Average BW = 336.469760 MB/sec
size=65536 Average BW = 3.472523 GB/sec
Stopped Queues 0 -> 0.
Deleted Queues 0 -> 0.
qdma3b000:statistics
Total MM H2C packets processed = 34000
Total MM C2H packets processed = 17017
Total ST H2C packets processed = 0
Total ST C2H packets processed = 0
Min Ping Pong Latency = 0
Max Ping Pong Latency = 0
Avg Ping Pong Latency = 0
-e
Test done.

Here are some differences I observed between this example QDMA project and the one in open-nic-shell:

Descriptor bypass is disabled in the built-in QDMA example, but is on in open-nic-shell
The BAR for AXI lite master in open-nic-shell is 4MB and has a value of FFFFFFFFFFC00004. It is FFFFFFFFFFFFF004 in the example with a size of 4KB. This shouldn't cause any issues.

QDMA is using advanced mode in open-nic-shell which is reasonable because we need AXI-lite CSR slave interface.

Should I disable descriptor bypass? I haven't tried running with changing BDF table entry. Maybe that will fix the issue.

Is it possible because the evaluate ERNIC IP license cracks the integrity of the whole bitstream after certain hours? And now the DMA is not working?

@qianyich

Is it possible because the evaluate ERNIC IP license cracks the integrity of the whole bitstream after certain hours? And now the DMA is not working?

The two won't affect each other.
I download the source from the repo and try it today. I also have the same issue for the DMA part. It seems we push a version, which causes the DMA read-side problem. I'll revert the version and fix it next week (this week is too busy). Thanks and sorry for the trouble! The other issue mentioned above should be related to your system setup.

Sounds good! Thank you so much Guanwen!

Any progress on this?

@qianyich You can try the latest commit, there is a fix in the driver side.

I am now having a problem loading the driver. The driver got compiled, but it failed to initialize the device.
[ 1173.819980] pci 0000:3b:00.0: [10ee:903f] type 00 class 0x058000
[ 1173.819998] pci 0000:3b:00.0: reg 0x10: [mem 0xab400000-0xab43ffff 64bit]
[ 1173.820005] pci 0000:3b:00.0: reg 0x18: [mem 0xab000000-0xab3fffff 64bit]
[ 1173.820023] pci 0000:3b:00.0: enabling Extended Tags
[ 1173.820230] pci 0000:3b:00.0: BAR 2: assigned [mem 0xab000000-0xab3fffff 64bit]
[ 1173.820236] pci 0000:3b:00.0: BAR 0: assigned [mem 0xab400000-0xab43ffff 64bit]
[ 1173.820360] onic 0000:3b:00.0 onic59s0f0 (uninitialized): Set MAC address to 0:a:35:a:3f:f7
[ 1173.820362] onic 0000:3b:00.0: device is a master PF
[ 1173.820364] onic_set_num_queue: num_msix 8, nb_queues 7, pci_msix_user_cnt 1
[ 1173.820365] onic_pci_probe mm_queues: 4
[ 1173.820417] onic:qdma_device_open: onic, 3b:00.00, pdev 0x00000000c9cf9db1, 0x10ee:0x903f.
[ 1173.820557] qdma_is_config_bar: Invalid config bar, err:-4
[ 1173.826056] qdma_hw_access_init: config bar passed is INVALID, err:-1
[ 1173.832546] onic 0000:3b:00.0: onic_qdma_setup: qdma_device_open() failed: Error Code: -22
[ 1173.840808] onic 0000:3b:00.0: onic_pci_probe: onic_qdma_setup() failed with status -22
[ 1173.848869] onic: probe of 0000:3b:00.0 failed with error -22

Ah okay. Please re-enable the part (Line 1158 - 1166 in onic_enable_cmac() located at onic_main.c) I commented out. It should work with your setup. I guess you're having two boards connected directly.

hmm, the driver can be loaded, but the dma is still problematic. Both read and write failed. BTW, my two boards are not connected directly. They are connected to network switch.
qianyich@pc164:~/RecoNIC/examples/dma_test$ sudo ./dma_test -d /dev/reconic-mm -s 65536000 -c 200
Write scenario
/dev/reconic-mm, W off 0x0, 0x3e80000 failed -1.
write file: Input/output error
qianyich@pc164:~/RecoNIC/examples/dma_test$ sudo ./dma_test -d /dev/reconic-mm -s 65536000 -c 200 -r
Read scenario
/dev/reconic-mm, read off 0x0 + 0x3e80000 failed -1.
read file: Input/output error

Hi Yicheng,
Can you put some pr_info in the onic_main.c and get (q_no, qhndl, xpriv->base_rx_q_handle, queue_id, q_handle, xpriv->pinfo->active_rx_queues, xpriv->pinfo->active_tx_queues) in onic_rx_pkt_process(), onic_rx_poll(), onic_isr_rx_tophalf(), onic_qdma_rx_queue_add(), onic_isr_tx_tophalf() and onic_qdma_tx_queue_setup()? And please share the dmesg log with me, so that I can understand what's going on at your end.
For the pr_info, do remember to put "func", so that I can know which function.
Thanks.

Regarding your rdma question, you can check ERNIC 3.1 document's Chapter 2 Register Space, there are some status registers available, which provide internal information of the RDMA engine. We are planning to upgrade it to version 4.0.
Did you solve the address issue I mentioned before by configuring more BDF table and removing address masking? If you haven't done that, the register information has no meaning, as the RDMA engine is not working properly. If you don't know how to do it, please wait for one or two weeks and I'll update it to include larger host memory up to 1TB.

The BDF table entry is now correct and the memory address space configuration is all good. However, I think it would be better to make it support a larger memory by default.
I will add those pr_info by tomorrow. Just noticed that some variables are not valid in the scope of some functions, and I will just ignore those.

Probably you can send me your modifications to support u280 (henry.zhong AT amd.com). I can take a look and check it next week.

I am not sure what is going on, but after I inserted pr_info lines, reboot the machine, reload the driver. The dma starts to work
qianyich@pc164:~/RecoNIC/examples/dma_test$ sudo ./dma_test -d /dev/reconic-mm -s 65536000 -c 200
Write scenario
size=65536000 Average BW = 4.948909 GB/sec, average latency = 13242.515735 us
qianyich@pc164:~/RecoNIC/examples/dma_test$ sudo  ./dma_test -d /dev/reconic-mm -s 65536000 -c 200 -r
Read scenario
size=65536000 Average BW = 5.257542 GB/sec, average latency = 12465.140782 us

Okay, probably your machine/FPGA was in wrong status when you loaded the driver. Can you start a fresh one by re-programing the FPGA board and loading the driver? See whether it works.
Regarding the BDF table, I realize that QDMA set the 8 windows equally, which means that each window will have up to 128GB mapping. The previous "0xC400_0000" mentioned is wrong. I'll update the correct one next week, as my bandwidth this week is full now.

The send_recv test failed as well. The receiver side just hang there. The sender side just failed to perform send operation.
Receiver Side:
sudo env LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./send_recv -r <IP_ADDRESS> -i <IP_ADDRESS> -p /sys/bus/pci/devices/0000\:3b\:00.0/resource2 -z 128 -l host_mem -d /dev/re
conic-mm -c -u 22222 --dst_qp 2 -g 2>&1 | tee client_debug.log
src_ip_str = <IP_ADDRESS>
dst_ip_str = <IP_ADDRESS>
Info: mac_addr_t = 00:0a:35:5b:e2:96
Info: PCIe resource file: /sys/bus/pci/devices/0000:3b:00.0/resource2
Info: QP allocated at: host_mem
Info: Device - /dev/reconic-mm
Info: src_ip = <IP_ADDRESS>
Info: Found network interface: enp59s0
Info: mac_addr_t = 00:0a:35:58:dd:fe
Info: CREATE RecoNIC DEVICE
/users/qianyich/RecoNIC/lib/reconic.c:297:create_rn_dev(): Info: scr(=4)) file open successfully
create_rn_dev - testing2
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x16ed000
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x16ed000000
Info: pre-allocated hugepage buffer vir addr = 0x7fb732400000, physical addr = 0x16ed000000
Info: Configuring QDMA AXI bridge BDF
/users/qianyich/RecoNIC/lib/reconic.c:195:config_rn_dev_axib_bdf(): [BDF] AXIB_BDF_ADDR_TRANSLATE_ADDR_LSB=0x16420, bdf_addr_low=0x0
/users/qianyich/RecoNIC/lib/reconic.c:196:config_rn_dev_axib_bdf(): [BDF] AXIB_BDF_ADDR_TRANSLATE_ADDR_MSB=0x16424, bdf_addr_high=0x0
/users/qianyich/RecoNIC/lib/reconic.c:197:config_rn_dev_axib_bdf(): [BDF] AXIB_BDF_MAP_CONTROL_ADDR=0x16430, bdf_win_config=0xc4000000
Info: CREATE RDMA DEVICE
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x16ed000
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x16ed000000
/users/qianyich/RecoNIC/lib/reconic.c:228:allocate_rdma_buffer(): Info: allocated host buffer vir addr = 0x7fb732400000, physical addr = 16ed000000, rn_dev->buffer_offset = 0x200000
/users/qianyich/RecoNIC/lib/reconic.c:229:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma host buffer
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x16ed200
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x16ed200000
/users/qianyich/RecoNIC/lib/reconic.c:228:allocate_rdma_buffer(): Info: allocated host buffer vir addr = 0x7fb732600000, physical addr = 16ed200000, rn_dev->buffer_offset = 0x1200000
/users/qianyich/RecoNIC/lib/reconic.c:229:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma host buffer
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x16ec200
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x16ec200000
/users/qianyich/RecoNIC/lib/reconic.c:228:allocate_rdma_buffer(): Info: allocated host buffer vir addr = 0x7fb733600000, physical addr = 16ec200000, rn_dev->buffer_offset = 0x1202000
/users/qianyich/RecoNIC/lib/reconic.c:229:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma host buffer
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x16ec202
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x16ec202000
/users/qianyich/RecoNIC/lib/reconic.c:228:allocate_rdma_buffer(): Info: allocated host buffer vir addr = 0x7fb733602000, physical addr = 16ec202000, rn_dev->buffer_offset = 0x1212000
/users/qianyich/RecoNIC/lib/reconic.c:229:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma host buffer
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x16ec212
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x16ec212000
/users/qianyich/RecoNIC/lib/reconic.c:228:allocate_rdma_buffer(): Info: allocated host buffer vir addr = 0x7fb733612000, physical addr = 16ec212000, rn_dev->buffer_offset = 0x1222000
/users/qianyich/RecoNIC/lib/reconic.c:229:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma host buffer
Info: OPEN RDMA DEVICE
/users/qianyich/RecoNIC/lib/rdma_api.c:186:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_DATBUFBA=0x600a0, value=0xed200000
/users/qianyich/RecoNIC/lib/rdma_api.c:188:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_DATBUFBAMSB=0x600a4, value=0x16
/users/qianyich/RecoNIC/lib/rdma_api.c:190:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_DATBUFSZ=0x600a8, value=0x10001000
/users/qianyich/RecoNIC/lib/rdma_api.c:193:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_IPKTERRQBA=0x60088, value=0xec200000
/users/qianyich/RecoNIC/lib/rdma_api.c:195:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_IPKTERRQBAMSB=0x6008c, value=0x16
/users/qianyich/RecoNIC/lib/rdma_api.c:197:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_ERRBUFSZ=0x60090, value=0x2000
/users/qianyich/RecoNIC/lib/rdma_api.c:200:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_ERRBUFBA=0x60060, value=0xec202000
/users/qianyich/RecoNIC/lib/rdma_api.c:202:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_ERRBUFBAMSB=0x60064, value=0x16
/users/qianyich/RecoNIC/lib/rdma_api.c:204:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_ERRBUFSZ=0x60068, value=0x1000100
/users/qianyich/RecoNIC/lib/rdma_api.c:207:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_RESPERRPKTBA=0x600b0, value=0xec212000
/users/qianyich/RecoNIC/lib/rdma_api.c:209:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_RESPERRPKTBAMSB=0x600b4, value=0x16
/users/qianyich/RecoNIC/lib/rdma_api.c:211:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_RESPERRSZ=0x600b8, value=0x10000
/users/qianyich/RecoNIC/lib/rdma_api.c:213:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_RESPERRSZMSB=0x600bc, value=0x0
/users/qianyich/RecoNIC/lib/rdma_api.c:217:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_INTEN=0x60180, value=0xff
/users/qianyich/RecoNIC/lib/rdma_api.c:221:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_MACXADDLSB=0x60010, value=0x3558ddfe
/users/qianyich/RecoNIC/lib/rdma_api.c:223:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_MACXADDMSB=0x60014, value=0xa
/users/qianyich/RecoNIC/lib/rdma_api.c:227:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_IPV4XADD=0x60070, value=0xc0643301
/users/qianyich/RecoNIC/lib/rdma_api.c:230:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_XRNICCONF=0x60000, value=0x56ce1421
/users/qianyich/RecoNIC/lib/rdma_api.c:233:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_XRNICADCONF=0x60004, value=0xa0004
Info: RDMA global control status registers are configured.
Info: rdma_dev opened
Info: ALLOCATE PD
/users/qianyich/RecoNIC/lib/rdma_api.c:254:allocate_rdma_pd(): [Register] RN_RDMA_PDT_PDPDNUM=0x40000, pd_num=0, value=0x0
Info: OPEN DEVICE FILE
Info: ALLOCATE RDMA QP
Allocating qp->sq
/users/qianyich/RecoNIC/lib/rdma_api.c:437:allocate_rdma_qp(): sq_size = 81920, cq_size = 5120, rq_size 655360, buf_location = host_mem
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x16ec222
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x16ec222000
/users/qianyich/RecoNIC/lib/reconic.c:228:allocate_rdma_buffer(): Info: allocated host buffer vir addr = 0x7fb733622000, physical addr = 16ec222000, rn_dev->buffer_offset = 0x1236000
/users/qianyich/RecoNIC/lib/reconic.c:229:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma host buffer
Allocating qp->cq
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x16ec236
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x16ec236000
/users/qianyich/RecoNIC/lib/reconic.c:228:allocate_rdma_buffer(): Info: allocated host buffer vir addr = 0x7fb733636000, physical addr = 16ec236000, rn_dev->buffer_offset = 0x1237400
/users/qianyich/RecoNIC/lib/reconic.c:229:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma host buffer
Allocating qp->rq
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x16ec238
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x16ec238000
/users/qianyich/RecoNIC/lib/reconic.c:228:allocate_rdma_buffer(): Info: allocated host buffer vir addr = 0x7fb733638000, physical addr = 16ec238000, rn_dev->buffer_offset = 0x12d8000
/users/qianyich/RecoNIC/lib/reconic.c:229:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma host buffer
Info: queue pair setting is done! Configuring RDMA per-queu CSR registers
/users/qianyich/RecoNIC/lib/rdma_api.c:487:allocate_rdma_qp(): DEBUG: rdma_dev->rn_dev->axil_ctl = 0x7fb752508000, rdma_dev->axil_ctl = 0x7fb752508000
/users/qianyich/RecoNIC/lib/rdma_api.c:502:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_IPDESADDR1i=0x60360, qpid=2, value=0xc0643401
/users/qianyich/RecoNIC/lib/rdma_api.c:509:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_MACDESADDLSBi=0x60350, qpid=2, value=0x355be296
/users/qianyich/RecoNIC/lib/rdma_api.c:516:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_MACDESADDMSBi=0x60354, qpid=2, value=0xa
/users/qianyich/RecoNIC/lib/rdma_api.c:521:allocate_rdma_qp(): DEBUG: win_size_high = 0x1f, win_size_low = 0xffffffff
/users/qianyich/RecoNIC/lib/rdma_api.c:539:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_SQBAi=0x60310, qpid=2, value=0xec222000
/users/qianyich/RecoNIC/lib/rdma_api.c:546:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_SQBAMSBi=0x603c8, qpid=2, value=0x16
/users/qianyich/RecoNIC/lib/rdma_api.c:550:allocate_rdma_qp(): DEBUG: qp->sq->dma_addr = 0x16ec222000, sq_addr_msb = 0x16, sq_addr_lsb = 0xec222000
/users/qianyich/RecoNIC/lib/rdma_api.c:568:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_CQBAi=0x60318, qpid=2, value=0xec236000
/users/qianyich/RecoNIC/lib/rdma_api.c:575:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_CQBAMSBi=0x603d0, qpid=2, value=0x16
/users/qianyich/RecoNIC/lib/rdma_api.c:579:allocate_rdma_qp(): DEBUG: qp->cq->dma_addr = 0x16ec236000, cq_addr_msb = 0x16, cq_addr_lsb = 0xec236000
/users/qianyich/RecoNIC/lib/rdma_api.c:597:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_RQBAi=0x60308, qpid=2, value=0xec238000
/users/qianyich/RecoNIC/lib/rdma_api.c:604:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_RQBAMSBi=0x603c0, qpid=2, value=0x16
/users/qianyich/RecoNIC/lib/rdma_api.c:608:allocate_rdma_qp(): DEBUG: qp->rq->dma_addr = 0x16ec238000, rq_addr_msb = 0x16, rq_addr_lsb = 0xec238000
/users/qianyich/RecoNIC/lib/rdma_api.c:617:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_CQDBADDi=0x60328, qpid=2, value=0xed000000
/users/qianyich/RecoNIC/lib/rdma_api.c:624:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_CQDBADDMSBi=0x6032c, qpid=2, value=0x16
/users/qianyich/RecoNIC/lib/rdma_api.c:625:allocate_rdma_qp(): DEBUG: cq_cidb_addr = 0x16ed000000
/users/qianyich/RecoNIC/lib/rdma_api.c:634:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_RQWPTRDBADDi=0x60320, qpid=2, value=0xed000020
/users/qianyich/RecoNIC/lib/rdma_api.c:641:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_RQWPTRDBADDMSBi=0x60324, qpid=2, value=0x16
/users/qianyich/RecoNIC/lib/rdma_api.c:642:allocate_rdma_qp(): DEBUG: rq_cidb_addr = 0x16ed000020
/users/qianyich/RecoNIC/lib/rdma_api.c:651:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_DESTQPCONFi=0x60348, qpid=2, value=0x2
/users/qianyich/RecoNIC/lib/rdma_api.c:660:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_QDEPTHi=0x6033c, qpid=2, value=0x400040
/users/qianyich/RecoNIC/lib/rdma_api.c:707:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_QPCONFi=0x60300, qpid=2, value=0x200043d
/users/qianyich/RecoNIC/lib/rdma_api.c:721:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_QPADVCONFi=0x60304, qpid=2, value=0x12344000
/users/qianyich/RecoNIC/lib/rdma_api.c:730:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_PDi=0x603b0, qpid=2, value=0x0
Info: allocate_rdma_qp - Successfully allocated a rdma qp
Info: CONFIGURE PSN
[Register] RN_RDMA_QCSR_LSTRQREQi=0x60344, qpid=2, value=0xa000abc
[Register] RN_RDMA_QCSR_SQPSNi=0x60340, qpid=2, value=0xabd
Info: APPLICATION START
payload_size = 128, payload_size>>2 = 32
Info: RDMA POST RECEIVE
/users/qianyich/RecoNIC/lib/rdma_api.c:828:poll_rq_pidb(): DEBUG: Polling on RQ PIDB. Count: 0x0

Sender Side:
sudo env LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./send_recv -r <IP_ADDRESS> -i <IP_ADDRESS> -p /sys/bus/pci/devices/0000\:3b\:00.0/resource2 -z 16384 -l dev_mem -d /dev/r
econic-mm -s -u 22222 --dst_qp 2 -g 2>&1 | tee server_debug.log
src_ip_str = <IP_ADDRESS>
dst_ip_str = <IP_ADDRESS>
Info: mac_addr_t = 00:0a:35:58:dd:fe
Info: PCIe resource file: /sys/bus/pci/devices/0000:3b:00.0/resource2
Info: QP allocated at: dev_mem
Info: Device - /dev/reconic-mm
Info: src_ip = <IP_ADDRESS>
Info: Found network interface: enp59s0
Info: mac_addr_t = 00:0a:35:5b:e2:96
Info: CREATE RecoNIC DEVICE
/users/qianyich/RecoNIC/lib/reconic.c:297:create_rn_dev(): Info: scr(=4)) file open successfully
create_rn_dev - testing2
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x2f2fa00
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x2f2fa00000
Info: pre-allocated hugepage buffer vir addr = 0x7f46ed000000, physical addr = 0x2f2fa00000
Info: Configuring QDMA AXI bridge BDF
/users/qianyich/RecoNIC/lib/reconic.c:195:config_rn_dev_axib_bdf(): [BDF] AXIB_BDF_ADDR_TRANSLATE_ADDR_LSB=0x16420, bdf_addr_low=0x0
/users/qianyich/RecoNIC/lib/reconic.c:196:config_rn_dev_axib_bdf(): [BDF] AXIB_BDF_ADDR_TRANSLATE_ADDR_MSB=0x16424, bdf_addr_high=0x20
/users/qianyich/RecoNIC/lib/reconic.c:197:config_rn_dev_axib_bdf(): [BDF] AXIB_BDF_MAP_CONTROL_ADDR=0x16430, bdf_win_config=0xc4000000
Info: CREATE RDMA DEVICE
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x2f2fa00
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x2f2fa00000
/users/qianyich/RecoNIC/lib/reconic.c:228:allocate_rdma_buffer(): Info: allocated host buffer vir addr = 0x7f46ed000000, physical addr = 2f2fa00000, rn_dev->buffer_offset = 0x200000
/users/qianyich/RecoNIC/lib/reconic.c:229:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma host buffer
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x2f2f400
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x2f2f400000
/users/qianyich/RecoNIC/lib/reconic.c:228:allocate_rdma_buffer(): Info: allocated host buffer vir addr = 0x7f46ed200000, physical addr = 2f2f400000, rn_dev->buffer_offset = 0x1200000
/users/qianyich/RecoNIC/lib/reconic.c:229:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma host buffer
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x2f2e400
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x2f2e400000
/users/qianyich/RecoNIC/lib/reconic.c:228:allocate_rdma_buffer(): Info: allocated host buffer vir addr = 0x7f46ee200000, physical addr = 2f2e400000, rn_dev->buffer_offset = 0x1202000
/users/qianyich/RecoNIC/lib/reconic.c:229:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma host buffer
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x2f2e402
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x2f2e402000
/users/qianyich/RecoNIC/lib/reconic.c:228:allocate_rdma_buffer(): Info: allocated host buffer vir addr = 0x7f46ee202000, physical addr = 2f2e402000, rn_dev->buffer_offset = 0x1212000
/users/qianyich/RecoNIC/lib/reconic.c:229:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma host buffer
/users/qianyich/RecoNIC/lib/reconic.c:145:get_buffer_paddr(): Info: get_buffer_paddr - Page frame: 0x2f2e412
/users/qianyich/RecoNIC/lib/reconic.c:150:get_buffer_paddr(): Info: get_buffer_paddr - distance from page boundary: 0x0
/users/qianyich/RecoNIC/lib/reconic.c:154:get_buffer_paddr(): Info: get_buffer_paddr - Physical address of buffer: 0x2f2e412000
/users/qianyich/RecoNIC/lib/reconic.c:228:allocate_rdma_buffer(): Info: allocated host buffer vir addr = 0x7f46ee212000, physical addr = 2f2e412000, rn_dev->buffer_offset = 0x1222000
/users/qianyich/RecoNIC/lib/reconic.c:229:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma host buffer
Info: OPEN RDMA DEVICE
/users/qianyich/RecoNIC/lib/rdma_api.c:186:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_DATBUFBA=0x600a0, value=0x2f400000
/users/qianyich/RecoNIC/lib/rdma_api.c:188:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_DATBUFBAMSB=0x600a4, value=0xf
/users/qianyich/RecoNIC/lib/rdma_api.c:190:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_DATBUFSZ=0x600a8, value=0x10001000
/users/qianyich/RecoNIC/lib/rdma_api.c:193:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_IPKTERRQBA=0x60088, value=0x2e400000
/users/qianyich/RecoNIC/lib/rdma_api.c:195:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_IPKTERRQBAMSB=0x6008c, value=0xf
/users/qianyich/RecoNIC/lib/rdma_api.c:197:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_ERRBUFSZ=0x60090, value=0x2000
/users/qianyich/RecoNIC/lib/rdma_api.c:200:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_ERRBUFBA=0x60060, value=0x2e402000
/users/qianyich/RecoNIC/lib/rdma_api.c:202:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_ERRBUFBAMSB=0x60064, value=0xf
/users/qianyich/RecoNIC/lib/rdma_api.c:204:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_ERRBUFSZ=0x60068, value=0x1000100
/users/qianyich/RecoNIC/lib/rdma_api.c:207:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_RESPERRPKTBA=0x600b0, value=0x2e412000
/users/qianyich/RecoNIC/lib/rdma_api.c:209:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_RESPERRPKTBAMSB=0x600b4, value=0xf
/users/qianyich/RecoNIC/lib/rdma_api.c:211:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_RESPERRSZ=0x600b8, value=0x10000
/users/qianyich/RecoNIC/lib/rdma_api.c:213:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_RESPERRSZMSB=0x600bc, value=0x0
/users/qianyich/RecoNIC/lib/rdma_api.c:217:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_INTEN=0x60180, value=0xff
/users/qianyich/RecoNIC/lib/rdma_api.c:221:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_MACXADDLSB=0x60010, value=0x355be296
/users/qianyich/RecoNIC/lib/rdma_api.c:223:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_MACXADDMSB=0x60014, value=0xa
/users/qianyich/RecoNIC/lib/rdma_api.c:227:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_IPV4XADD=0x60070, value=0xc0643401
/users/qianyich/RecoNIC/lib/rdma_api.c:230:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_XRNICCONF=0x60000, value=0x56ce1421
/users/qianyich/RecoNIC/lib/rdma_api.c:233:config_rdma_global_csr(): [Register] RN_RDMA_GCSR_XRNICADCONF=0x60004, value=0xa0004
Info: RDMA global control status registers are configured.
Info: rdma_dev opened
Info: ALLOCATE PD
/users/qianyich/RecoNIC/lib/rdma_api.c:254:allocate_rdma_pd(): [Register] RN_RDMA_PDT_PDPDNUM=0x40000, pd_num=0, value=0x0
Info: OPEN DEVICE FILE
Info: ALLOCATE RDMA QP
Allocating qp->sq
/users/qianyich/RecoNIC/lib/rdma_api.c:437:allocate_rdma_qp(): sq_size = 81920, cq_size = 5120, rq_size 655360, buf_location = dev_mem
/users/qianyich/RecoNIC/lib/reconic.c:252:allocate_rdma_buffer(): Info: allocated device buffer physical addr = a350000000000000, rn_dev->dev_buffer_offset = 0x14000
/users/qianyich/RecoNIC/lib/reconic.c:254:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma device buffer
Allocating qp->cq
/users/qianyich/RecoNIC/lib/reconic.c:252:allocate_rdma_buffer(): Info: allocated device buffer physical addr = a350000000014000, rn_dev->dev_buffer_offset = 0x15400
/users/qianyich/RecoNIC/lib/reconic.c:254:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma device buffer
Allocating qp->rq
/users/qianyich/RecoNIC/lib/reconic.c:252:allocate_rdma_buffer(): Info: allocated device buffer physical addr = a350000000016000, rn_dev->dev_buffer_offset = 0xb6000
/users/qianyich/RecoNIC/lib/reconic.c:254:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma device buffer
Info: queue pair setting is done! Configuring RDMA per-queu CSR registers
/users/qianyich/RecoNIC/lib/rdma_api.c:487:allocate_rdma_qp(): DEBUG: rdma_dev->rn_dev->axil_ctl = 0x7f470d197000, rdma_dev->axil_ctl = 0x7f470d197000
/users/qianyich/RecoNIC/lib/rdma_api.c:502:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_IPDESADDR1i=0x60360, qpid=2, value=0xc0643301
/users/qianyich/RecoNIC/lib/rdma_api.c:509:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_MACDESADDLSBi=0x60350, qpid=2, value=0x3558ddfe
/users/qianyich/RecoNIC/lib/rdma_api.c:516:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_MACDESADDMSBi=0x60354, qpid=2, value=0xa
/users/qianyich/RecoNIC/lib/rdma_api.c:521:allocate_rdma_qp(): DEBUG: win_size_high = 0x1f, win_size_low = 0xffffffff
/users/qianyich/RecoNIC/lib/rdma_api.c:539:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_SQBAi=0x60310, qpid=2, value=0x0
/users/qianyich/RecoNIC/lib/rdma_api.c:546:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_SQBAMSBi=0x603c8, qpid=2, value=0xa3500000
/users/qianyich/RecoNIC/lib/rdma_api.c:550:allocate_rdma_qp(): DEBUG: qp->sq->dma_addr = 0xa350000000000000, sq_addr_msb = 0xa3500000, sq_addr_lsb = 0x0
/users/qianyich/RecoNIC/lib/rdma_api.c:568:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_CQBAi=0x60318, qpid=2, value=0x14000
/users/qianyich/RecoNIC/lib/rdma_api.c:575:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_CQBAMSBi=0x603d0, qpid=2, value=0xa3500000
/users/qianyich/RecoNIC/lib/rdma_api.c:579:allocate_rdma_qp(): DEBUG: qp->cq->dma_addr = 0xa350000000014000, cq_addr_msb = 0xa3500000, cq_addr_lsb = 0x14000
/users/qianyich/RecoNIC/lib/rdma_api.c:597:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_RQBAi=0x60308, qpid=2, value=0x16000
/users/qianyich/RecoNIC/lib/rdma_api.c:604:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_RQBAMSBi=0x603c0, qpid=2, value=0xa3500000
/users/qianyich/RecoNIC/lib/rdma_api.c:608:allocate_rdma_qp(): DEBUG: qp->rq->dma_addr = 0xa350000000016000, rq_addr_msb = 0xa3500000, rq_addr_lsb = 0x16000
/users/qianyich/RecoNIC/lib/rdma_api.c:617:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_CQDBADDi=0x60328, qpid=2, value=0x2fa00000
/users/qianyich/RecoNIC/lib/rdma_api.c:624:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_CQDBADDMSBi=0x6032c, qpid=2, value=0xf
/users/qianyich/RecoNIC/lib/rdma_api.c:625:allocate_rdma_qp(): DEBUG: cq_cidb_addr = 0x2f2fa00000
/users/qianyich/RecoNIC/lib/rdma_api.c:634:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_RQWPTRDBADDi=0x60320, qpid=2, value=0x2fa00020
/users/qianyich/RecoNIC/lib/rdma_api.c:641:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_RQWPTRDBADDMSBi=0x60324, qpid=2, value=0xf
/users/qianyich/RecoNIC/lib/rdma_api.c:642:allocate_rdma_qp(): DEBUG: rq_cidb_addr = 0x2f2fa00020
/users/qianyich/RecoNIC/lib/rdma_api.c:651:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_DESTQPCONFi=0x60348, qpid=2, value=0x2
/users/qianyich/RecoNIC/lib/rdma_api.c:660:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_QDEPTHi=0x6033c, qpid=2, value=0x400040
/users/qianyich/RecoNIC/lib/rdma_api.c:707:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_QPCONFi=0x60300, qpid=2, value=0x200043d
/users/qianyich/RecoNIC/lib/rdma_api.c:721:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_QPADVCONFi=0x60304, qpid=2, value=0x12344000
/users/qianyich/RecoNIC/lib/rdma_api.c:730:allocate_rdma_qp(): [Register] RN_RDMA_QCSR_PDi=0x603b0, qpid=2, value=0x0
Info: allocate_rdma_qp - Successfully allocated a rdma qp
Info: CONFIGURE PSN
[Register] RN_RDMA_QCSR_LSTRQREQi=0x60344, qpid=2, value=0xa000abc
[Register] RN_RDMA_QCSR_SQPSNi=0x60340, qpid=2, value=0xabd
Info: APPLICATION START
payload_size = 16384, payload_size>>2 = 4096
Info: ALLOCATE PAYLOAD DATA
/users/qianyich/RecoNIC/lib/reconic.c:252:allocate_rdma_buffer(): Info: allocated device buffer physical addr = a3500000000b6000, rn_dev->dev_buffer_offset = 0xba000
/users/qianyich/RecoNIC/lib/reconic.c:254:allocate_rdma_buffer(): Info: allocate_rdma_buffer - successfully allocated rdma device buffer
Info: copy software payload to the device memory
/users/qianyich/RecoNIC/lib/rdma_api.c:769:create_a_wqe(): Info: WQE mem_buffer = 0xa3500000000b6000, masked_mem_buffer = 0xa3500000000b6000
/users/qianyich/RecoNIC/lib/rdma_api.c:796:create_a_wqe(): [WQE] wrid=0x0
/users/qianyich/RecoNIC/lib/rdma_api.c:797:create_a_wqe(): [WQE] laddr_low=0xb6000
/users/qianyich/RecoNIC/lib/rdma_api.c:798:create_a_wqe(): [WQE] laddr_high=0xa3500000
/users/qianyich/RecoNIC/lib/rdma_api.c:799:create_a_wqe(): [WQE] length=0x4000
/users/qianyich/RecoNIC/lib/rdma_api.c:800:create_a_wqe(): [WQE] opcode=0x2
/users/qianyich/RecoNIC/lib/rdma_api.c:801:create_a_wqe(): [WQE] remote_offset_low=0x0
/users/qianyich/RecoNIC/lib/rdma_api.c:802:create_a_wqe(): [WQE] remote_offset_high=0x0
/users/qianyich/RecoNIC/lib/rdma_api.c:803:create_a_wqe(): [WQE] r_key=0x8
/users/qianyich/RecoNIC/lib/rdma_api.c:804:create_a_wqe(): [WQE] send_small_payload0=0x0
/users/qianyich/RecoNIC/lib/rdma_api.c:805:create_a_wqe(): [WQE] send_small_payload1=0x0
/users/qianyich/RecoNIC/lib/rdma_api.c:806:create_a_wqe(): [WQE] send_small_payload2=0x0
/users/qianyich/RecoNIC/lib/rdma_api.c:807:create_a_wqe(): [WQE] send_small_payload3=0x0
/users/qianyich/RecoNIC/lib/rdma_api.c:808:create_a_wqe(): [WQE] immdt_data=0x0
/users/qianyich/RecoNIC/lib/rdma_api.c:811:create_a_wqe(): DEBUG: Write WQE to the device memory
/users/qianyich/RecoNIC/lib/rdma_api.c:817:create_a_wqe(): DEBUG: successfully write WQE to the device memory!
Info: Dump register values for debug purpose
Info: [RN_RDMA_GCSR_ERRBUFWPTR      = 0x6006c] = 0x0
Info: [RN_RDMA_GCSR_IPKTERRQWPTR    = 0x60094] = 0x0
Info: [RN_RDMA_GCSR_INSRRPKTCNT     = 0x60100] = 0x0
Info: [RN_RDMA_GCSR_INAMPKTCNT      = 0x60104] = 0x0
Info: [RN_RDMA_GCSR_OUTIOPKTCNT     = 0x60108] = 0x0
Info: [RN_RDMA_GCSR_OUTAMPKTCNT     = 0x6010c] = 0x1
Info: [RN_RDMA_GCSR_LSTINPKT        = 0x60110] = 0xabd020a
Info: [RN_RDMA_GCSR_LSTOUTPKT       = 0x60114] = 0x0
Info: [RN_RDMA_GCSR_ININVDUPCNT     = 0x60118] = 0x0
Info: [RN_RDMA_GCSR_INNCKPKTSTS     = 0x6011c] = 0x0
Info: [RN_RDMA_GCSR_OUTRNRPKTSTS    = 0x60120] = 0x0
Info: [RN_RDMA_GCSR_WQEPROCSTS      = 0x60124] = 0x1012040
Info: [RN_RDMA_GCSR_QPMSTS          = 0x6012c] = 0x2
Info: [RN_RDMA_GCSR_INALLDRPPKTCNT  = 0x60130] = 0x10000
Info: [RN_RDMA_GCSR_INNAKPKTCNT     = 0x60134] = 0x0
Info: [RN_RDMA_GCSR_OUTNAKPKTCNT    = 0x60138] = 0x0
Info: [RN_RDMA_GCSR_RESPHNDSTS      = 0x6013c] = 0x10000
Info: [RN_RDMA_GCSR_RETRYCNTSTS     = 0x60140] = 0x0
Info: [RN_RDMA_GCSR_INCNPPKTCNT     = 0x60174] = 0x0
Info: [RN_RDMA_GCSR_OUTCNPPKTCNT    = 0x60178] = 0x0
Info: [RN_RDMA_GCSR_OUTRDRSPPKTCNT  = 0x6017c] = 0x0
Info: [RN_RDMA_GCSR_INTSTS          = 0x60184] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS1       = 0x60190] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS2       = 0x60194] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS3       = 0x60198] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS4       = 0x6019c] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS5       = 0x601a0] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS6       = 0x601a4] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS7       = 0x601a8] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS8       = 0x601ac] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS1       = 0x601b0] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS2       = 0x601b4] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS3       = 0x601b8] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS4       = 0x601bc] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS5       = 0x601c0] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS6       = 0x601c4] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS7       = 0x601c8] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS8       = 0x601cc] = 0x0
Info: [RN_RDMA_QCSR_CQHEADi         = 0x60330] = 0x0
Info: [RN_RDMA_QCSR_STATSSNi        = 0x60380] = 0x1
Info: [RN_RDMA_QCSR_STATMSNi        = 0x60384] = 0x1
Info: [RN_RDMA_QCSR_STATQPi         = 0x60388] = 0x600
Info: [RN_RDMA_QCSR_STATCURSQPTRi   = 0x6038c] = 0x1
Info: [RN_RDMA_QCSR_STATRESPSNi     = 0x60390] = 0xabc
Info: [RN_RDMA_QCSR_STATRQBUFCAi    = 0x60394] = 0x16000
Info: [RN_RDMA_QCSR_STATWQEi        = 0x60398] = 0x0
Info: [RN_RDMA_QCSR_STATRQPIDBi     = 0x6039c] = 0x0
Info: [RN_RDMA_QCSR_STATRQBUFCAMSBi = 0x603d8] = 0xa3500000
Info: [RN_RDMA_QCSR_SQPIi           = 0x60338] = 0x1

/users/qianyich/RecoNIC/examples/rdma_test/send_recv.c:393:main(): Info: POSTING WQE of RDMA SEND
/users/qianyich/RecoNIC/lib/rdma_api.c:875:rdma_post_send(): DEBUG: Reading hardware SQPIi (0x60338) = 0x1
/users/qianyich/RecoNIC/lib/rdma_api.c:876:rdma_post_send(): DEBUG: original qp->sq_pidb = 0x0
/users/qianyich/RecoNIC/lib/rdma_api.c:882:rdma_post_send(): [Register] RN_RDMA_QCSR_SQPIi=0x60338, qpid=2, value=0x1
/users/qianyich/RecoNIC/lib/rdma_api.c:883:rdma_post_send(): DEBUG: Update hardware sq db idx from software = 1
/users/qianyich/RecoNIC/lib/rdma_api.c:884:rdma_post_send(): DEBUG: Reading hardware SQPIi (0x60338) = 0x1
/users/qianyich/RecoNIC/lib/rdma_api.c:844:poll_cq_cidb(): [Register] RN_RDMA_QCSR_CQHEADi=0x60330, qpid=2, value=0x0
/users/qianyich/RecoNIC/lib/rdma_api.c:846:poll_cq_cidb(): DEBUG: before polling: sq_cidb = 0; Polling CQ CIDB = 0
ERROR: poll_cq_cidb timeout! sq_cidb = 0; Polling CQ CIDB = 0
Info: Dump register values for debug purpose
Info: [RN_RDMA_GCSR_ERRBUFWPTR      = 0x6006c] = 0x0
Info: [RN_RDMA_GCSR_IPKTERRQWPTR    = 0x60094] = 0x0
Info: [RN_RDMA_GCSR_INSRRPKTCNT     = 0x60100] = 0x0
Info: [RN_RDMA_GCSR_INAMPKTCNT      = 0x60104] = 0x0
Info: [RN_RDMA_GCSR_OUTIOPKTCNT     = 0x60108] = 0x0
Info: [RN_RDMA_GCSR_OUTAMPKTCNT     = 0x6010c] = 0x1
Info: [RN_RDMA_GCSR_LSTINPKT        = 0x60110] = 0xabd020a
Info: [RN_RDMA_GCSR_LSTOUTPKT       = 0x60114] = 0x0
Info: [RN_RDMA_GCSR_ININVDUPCNT     = 0x60118] = 0x0
Info: [RN_RDMA_GCSR_INNCKPKTSTS     = 0x6011c] = 0x0
Info: [RN_RDMA_GCSR_OUTRNRPKTSTS    = 0x60120] = 0x0
Info: [RN_RDMA_GCSR_WQEPROCSTS      = 0x60124] = 0x1012040
Info: [RN_RDMA_GCSR_QPMSTS          = 0x6012c] = 0x2
Info: [RN_RDMA_GCSR_INALLDRPPKTCNT  = 0x60130] = 0x10000
Info: [RN_RDMA_GCSR_INNAKPKTCNT     = 0x60134] = 0x0
Info: [RN_RDMA_GCSR_OUTNAKPKTCNT    = 0x60138] = 0x0
Info: [RN_RDMA_GCSR_RESPHNDSTS      = 0x6013c] = 0x10000
Info: [RN_RDMA_GCSR_RETRYCNTSTS     = 0x60140] = 0x0
Info: [RN_RDMA_GCSR_INCNPPKTCNT     = 0x60174] = 0x0
Info: [RN_RDMA_GCSR_OUTCNPPKTCNT    = 0x60178] = 0x0
Info: [RN_RDMA_GCSR_OUTRDRSPPKTCNT  = 0x6017c] = 0x0
Info: [RN_RDMA_GCSR_INTSTS          = 0x60184] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS1       = 0x60190] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS2       = 0x60194] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS3       = 0x60198] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS4       = 0x6019c] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS5       = 0x601a0] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS6       = 0x601a4] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS7       = 0x601a8] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS8       = 0x601ac] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS1       = 0x601b0] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS2       = 0x601b4] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS3       = 0x601b8] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS4       = 0x601bc] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS5       = 0x601c0] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS6       = 0x601c4] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS7       = 0x601c8] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS8       = 0x601cc] = 0x0
Info: [RN_RDMA_QCSR_CQHEADi         = 0x60330] = 0x0
Info: [RN_RDMA_QCSR_STATSSNi        = 0x60380] = 0x1
Info: [RN_RDMA_QCSR_STATMSNi        = 0x60384] = 0x1
Info: [RN_RDMA_QCSR_STATQPi         = 0x60388] = 0x600
Info: [RN_RDMA_QCSR_STATCURSQPTRi   = 0x6038c] = 0x1
Info: [RN_RDMA_QCSR_STATRESPSNi     = 0x60390] = 0xabc
Info: [RN_RDMA_QCSR_STATRQBUFCAi    = 0x60394] = 0x16000
Info: [RN_RDMA_QCSR_STATWQEi        = 0x60398] = 0x0
Info: [RN_RDMA_QCSR_STATRQPIDBi     = 0x6039c] = 0x0
Info: [RN_RDMA_QCSR_STATRQBUFCAMSBi = 0x603d8] = 0xa3500000
Info: [RN_RDMA_QCSR_SQPIi           = 0x60338] = 0x1

Error: Failed to perform an RDMA send operation!
Warning: CQHEADi and SQPIi for QP2 are mismatched

***** QP2 FATAL RECOVERY *****
TIMEOUT: CQHEADi:0x0 and SQPIi:0x1 are different

Hi Yicheng,
I'm not surprised that it fails, as the BDF table configuration is still wrong at your end. We can't simply set the window size to 256GB, the max is 128GB per window. Please wait for a few days and I'll push one next week to cover 8 windows up to 1TB system memory mapping.

Oh yeah, I forgot about it. That makes sense. Thank you!

This could be a stupid question, but I noticed that the transport layer protocol in RoCE v2 spec is Infiniband, is the Infiniband header used in ERNIC or RecoNIC? I am asking because I did not find Infiniband in the ERNIC document.

@qianyich Please try branch "map_large_hmem". Let me know if it works or not.
The "Infiniband" you mentioned is link layer, which compared to "ethernet". RoCEv2 means RDMA protocol on top of ethernet. The RDMA protocol here is IB transport protocol in your figure, which includes base transport header (BTH), extended transport header (RETH, AETH, ImmDt, IETH, AtomicETH, AtomicAckETH, ...), etc.

@zhguanw-amd I am observing some unstable behaviors.
After the FPGAs were programmed, I ran the read, write, send_recv test immediately. They were all working fine. However, the systolic array example failed with result mismatch error. I then ran the dma_test on the client side, it showed some thing similar to the following:
/dev/reconic-mm, W off 0x0, 0x3e80000 failed -1.
write file: Input/output error

I rebooted the two servers, I ran the systolic array example, and it succeeded. But all rdma verbs test seems to be quite strange.
For example, rdma_read shows "Data read successfully" which means success but the log shows "Failed to send an RDMA read operation".
The logs for network_systolic_array after reboot are attached here:
client_debug.log
server_debug.log
The logs for rdma_read after reboot are attached here:
client_debug.log
server_debug.log
Also, I noticed that at the bottom of every log. I suspect those tests or some of the tests break the QDMA Queue Pairs somehow. It shouldn't be related to RDMA QPs though.
Warning: CQHEADi and SQPIi for QP2 are mismatched

***** QP2 FATAL RECOVERY *****
TIMEOUT: CQHEADi:0x0 and SQPIi:0x1 are different

I reprogrammed the devices, and started from the read test:
What I did in order is (1) dma test (2) read test (3) write test (4) send_recv test (5) systolic array test (6) dma test
(1) dma test passed with no error
(2) read test passed with logs attached here:
client_debug_rdma_read.log
server_debug_rdma_read.log
(3) write test passed with logs attached here:
client_debug_rdma_write.log
server_debug_rdma_write.log
(4) send_recv test passed with logs attached here:
client_debug_receiver.log
server_debug_sender.log
(5) systolic array test failed with logs attached here (for this <IP_ADDRESS> is the client who requests matrices):
server_debug_systolic.log
client_debug_systolic.log
This failed and I observed errors not only for systolic array test but also read, write, send_recv test in dmesg as shown below:
<IP_ADDRESS>:
[  159.273862] IPv6: ADDRCONF(NETDEV_UP): enp59s0: link is not ready
[  159.273888] IPv6: ADDRCONF(NETDEV_CHANGE): enp59s0: link becomes ready
[  240.930586] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[  275.085286] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[  427.603894] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[  427.603918] read[2476]: segfault at 29 ip 00007ffa0f6ed8f9 sp 00007ffc518e0d30 error 4 in libreconic.so[7ffa0f6e8000+c000]
[  632.051171] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[  632.051203] write[2496]: segfault at 29 ip 00007fd4508358f9 sp 00007ffeac2ba5f0 error 4 in libreconic.so[7fd450830000+c000]
[  795.233063] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[  795.233083] send_recv[2512]: segfault at 29 ip 00007f677f3f88f9 sp 00007ffc7484ec00 error 4 in libreconic.so[7f677f3f3000+c000]
[ 1106.850350] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[ 1106.850376] network_systoli[2555]: segfault at 29 ip 00007f0dc6bde8f9 sp 00007ffd5c084bd0 error 4 in libreconic.so[7f0dc6bd9000+c000]
[ 1222.657479] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[ 1228.868558] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.

<IP_ADDRESS>:
[   75.281512] IPv6: ADDRCONF(NETDEV_UP): enp59s0: link is not ready
[   75.281520] IPv6: ADDRCONF(NETDEV_CHANGE): enp59s0: link becomes ready
[  103.193176] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[  111.940145] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[  695.196025] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[  695.196050] read[2444]: segfault at 29 ip 00007f74f74758f9 sp 00007ffd6ae2c4d0 error 4 in libreconic.so[7f74f7470000+c000]
[  811.056709] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[  811.056739] write[2523]: segfault at 29 ip 00007f282d1b18f9 sp 00007fff51091c90 error 4 in libreconic.so[7f282d1ac000+c000]
[ 1075.081479] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[ 1075.081503] send_recv[2589]: segfault at 29 ip 00007fabcbb348f9 sp 00007ffe19910800 error 4 in libreconic.so[7fabcbb2f000+c000]
[ 1249.158570] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.

(6) dma test again. This time, dma test passed on <IP_ADDRESS>, but partially failed on <IP_ADDRESS>!
<IP_ADDRESS>:
qianyich@pc166:~/RecoNIC/examples/dma_test$ sudo ./dma_test -d /dev/reconic-mm -s 65536000 -c 200
Write scenario
size=65536000 Average BW = 4.875006 GB/sec, average latency = 13443.265110 us
qianyich@pc166:~/RecoNIC/examples/dma_test$ sudo ./dma_test -d /dev/reconic-mm -s 65536000 -c 200 -r
Read scenario
size=65536000 Average BW = 5.177333 GB/sec, average latency = 12658.255175 us

<IP_ADDRESS>:
qianyich@pc167:~/RecoNIC/examples/dma_test$ sudo ./dma_test -d /dev/reconic-mm -s 65536000 -c 200 -r
Read scenario
size=65536000 Average BW = 4.914677 GB/sec, average latency = 13334.752992 us
qianyich@pc167:~/RecoNIC/examples/dma_test$ sudo ./dma_test -d /dev/reconic-mm -s 65536000 -c 200
Write scenario
/dev/reconic-mm, W off 0x0, 0x3e80000 failed -1.
write file: Input/output error

@qianyich I'll check it tomorrow on U250 and see any issues. Porbably it's related to QMDA's queue assignment.

@zhguanw-amd Could you merge "map_large_hmem" into the main branch so that I can see if those changes work fine on my side?

Nevermind, I think you might think of merging it with other changes later. FYI, the latest commit in the main branch allows me to install the driver and ping each other. I did not test those APIs as they are not fixed yet.

Also, I am interested in adding QSPI flash into the design so that I can program the board over PCIe. https://github.com/Xilinx/open-nic-shell/blob/add_qspi_and_cms/script/generate_qspi_block_design.tcl might be the good starting point.

@qianyich please try the new version in the main branch. It should fix your issues in this version.

Also, I am interested in adding QSPI flash into the design so that I can program the board over PCIe. https://github.com/Xilinx/open-nic-shell/blob/add_qspi_and_cms/script/generate_qspi_block_design.tcl might be the starting point.

Yep, it sounds a good idea to have programmability via PCIe. To support it, apart from the hardware part, you also need to implement software to control it. You can start a new thread related to this.

@zhguanw-amd There is a new problem now.
What I did:

I ran read, write, and send_recv successfully with no issues. No error message from dmesg.
I failed to run the network systolic array example, I got the following.

<IP_ADDRESS>:
sudo env LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./network_systolic_mm -d /dev/reconic-mm -p /sys/bus/pci/devices/0000\:3b\:00.0
/resource2 -r <IP_ADDRESS> -i <IP_ADDRESS> -u 22222 -t 11111 --dst_qp 2 -c 2>&1 | tee client_debug.log
Info: Device - /dev/reconic-mm
Info: PCIe resource file: /sys/bus/pci/devices/0000:3b:00.0/resource2
src_ip_str = <IP_ADDRESS>
dst_ip_str = <IP_ADDRESS>
Info: mac_addr_t = 00:0a:35:24:44:8a
Info: src_ip = <IP_ADDRESS>
Info: Found network interface: enp59s0
Info: mac_addr_t = 00:0a:35:71:f4:5c
Info: Creating rn_dev
create_rn_dev - testing2
Info: pre-allocated hugepage buffer vir addr = 0x7fb423a00000, physical addr = 0x2f19a00000
Info: Configuring 8 windows in QDMA AXI bridge BDF, each has 128GB mapping
Info: CREATE RDMA DEVICE
Info: OPEN RDMA DEVICE
Info: RDMA global control status registers are configured.
Info: rdma_dev opened
Info: ALLOCATE PD
Info: OPEN DEVICE FILE
Info: ALLOCATE RDMA QP
Allocating qp->sq
Allocating qp->cq
Allocating qp->rq
Info: queue pair setting is done! Configuring RDMA per-queu CSR registers
Info: allocate_rdma_qp - Successfully allocated a rdma qp
Info: CONFIGURE PSN
[Register] RN_RDMA_QCSR_LSTRQREQi=0x60344, qpid=2, value=0xa000abc
[Register] RN_RDMA_QCSR_SQPSNi=0x60340, qpid=2, value=0xabd
Info: generating matrix data, matrix_size = 256
Info: Software Matrix Data generated
Info: Client is connecting to a remote server
Info: Client is connected to a remote server
Info: client received remote offset of A = 0xa350000000000000
Info: client received remote offset of B = 0xa350000000000400
Info: creating an RDMA read WQE for getting Array A
Successfully sent an RDMA read operation for Array A!
Info: Dump register values for debug purpose
Info: [RN_RDMA_GCSR_ERRBUFWPTR      = 0x6006c] = 0x0
Info: [RN_RDMA_GCSR_IPKTERRQWPTR    = 0x60094] = 0x0
Info: [RN_RDMA_GCSR_INSRRPKTCNT     = 0x60100] = 0x1
Info: [RN_RDMA_GCSR_INAMPKTCNT      = 0x60104] = 0x1
Info: [RN_RDMA_GCSR_OUTIOPKTCNT     = 0x60108] = 0x10004
Info: [RN_RDMA_GCSR_OUTAMPKTCNT     = 0x6010c] = 0x1
Info: [RN_RDMA_GCSR_LSTINPKT        = 0x60110] = 0xabd0210
Info: [RN_RDMA_GCSR_LSTOUTPKT       = 0x60114] = 0x157a204
Info: [RN_RDMA_GCSR_ININVDUPCNT     = 0x60118] = 0x0
Info: [RN_RDMA_GCSR_INNCKPKTSTS     = 0x6011c] = 0x0
Info: [RN_RDMA_GCSR_OUTRNRPKTSTS    = 0x60120] = 0x0
Info: [RN_RDMA_GCSR_WQEPROCSTS      = 0x60124] = 0xc0c2040
Info: [RN_RDMA_GCSR_QPMSTS          = 0x6012c] = 0x20002
Info: [RN_RDMA_GCSR_INALLDRPPKTCNT  = 0x60130] = 0x40000
Info: [RN_RDMA_GCSR_INNAKPKTCNT     = 0x60134] = 0x0
Info: [RN_RDMA_GCSR_OUTNAKPKTCNT    = 0x60138] = 0x0
Info: [RN_RDMA_GCSR_RESPHNDSTS      = 0x6013c] = 0x10f02
Info: [RN_RDMA_GCSR_RETRYCNTSTS     = 0x60140] = 0x0
Info: [RN_RDMA_GCSR_INCNPPKTCNT     = 0x60174] = 0x0
Info: [RN_RDMA_GCSR_OUTCNPPKTCNT    = 0x60178] = 0x0
Info: [RN_RDMA_GCSR_OUTRDRSPPKTCNT  = 0x6017c] = 0x1
Info: [RN_RDMA_GCSR_INTSTS          = 0x60184] = 0x10
Info: [RN_RDMA_GCSR_RQINTSTS1       = 0x60190] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS2       = 0x60194] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS3       = 0x60198] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS4       = 0x6019c] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS5       = 0x601a0] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS6       = 0x601a4] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS7       = 0x601a8] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS8       = 0x601ac] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS1       = 0x601b0] = 0x4
Info: [RN_RDMA_GCSR_CQINTSTS2       = 0x601b4] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS3       = 0x601b8] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS4       = 0x601bc] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS5       = 0x601c0] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS6       = 0x601c4] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS7       = 0x601c8] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS8       = 0x601cc] = 0x0
Info: [RN_RDMA_QCSR_CQHEADi         = 0x60330] = 0x1
Info: [RN_RDMA_QCSR_STATSSNi        = 0x60380] = 0x2
Info: [RN_RDMA_QCSR_STATMSNi        = 0x60384] = 0x0
Info: [RN_RDMA_QCSR_STATQPi         = 0x60388] = 0x1f0600
Info: [RN_RDMA_QCSR_STATCURSQPTRi   = 0x6038c] = 0x1
Info: [RN_RDMA_QCSR_STATRESPSNi     = 0x60390] = 0xabd
Info: [RN_RDMA_QCSR_STATRQBUFCAi    = 0x60394] = 0x1a82b000
Info: [RN_RDMA_QCSR_STATWQEi        = 0x60398] = 0x0
Info: [RN_RDMA_QCSR_STATRQPIDBi     = 0x6039c] = 0x0
Info: [RN_RDMA_QCSR_STATRQBUFCAMSBi = 0x603d8] = 0x2f
Info: [RN_RDMA_QCSR_SQPIi           = 0x60338] = 0x1

ERROR: poll_cq_cidb timeout! sq_cidb = 1; Polling CQ CIDB = 1
Info: Dump register values for debug purpose
Info: [RN_RDMA_GCSR_ERRBUFWPTR      = 0x6006c] = 0x0
Info: [RN_RDMA_GCSR_IPKTERRQWPTR    = 0x60094] = 0x0
Info: [RN_RDMA_GCSR_INSRRPKTCNT     = 0x60100] = 0x1
Info: [RN_RDMA_GCSR_INAMPKTCNT      = 0x60104] = 0x1
Info: [RN_RDMA_GCSR_OUTIOPKTCNT     = 0x60108] = 0x20004
Info: [RN_RDMA_GCSR_OUTAMPKTCNT     = 0x6010c] = 0x1
Info: [RN_RDMA_GCSR_LSTINPKT        = 0x60110] = 0xabe0210
Info: [RN_RDMA_GCSR_LSTOUTPKT       = 0x60114] = 0x157c204
Info: [RN_RDMA_GCSR_ININVDUPCNT     = 0x60118] = 0x1
Info: [RN_RDMA_GCSR_INNCKPKTSTS     = 0x6011c] = 0x0
Info: [RN_RDMA_GCSR_OUTRNRPKTSTS    = 0x60120] = 0x0
Info: [RN_RDMA_GCSR_WQEPROCSTS      = 0x60124] = 0xd0d2040
Info: [RN_RDMA_GCSR_QPMSTS          = 0x6012c] = 0x30002
Info: [RN_RDMA_GCSR_INALLDRPPKTCNT  = 0x60130] = 0x50000
Info: [RN_RDMA_GCSR_INNAKPKTCNT     = 0x60134] = 0x0
Info: [RN_RDMA_GCSR_OUTNAKPKTCNT    = 0x60138] = 0x0
Info: [RN_RDMA_GCSR_RESPHNDSTS      = 0x6013c] = 0x30f02
Info: [RN_RDMA_GCSR_RETRYCNTSTS     = 0x60140] = 0x0
Info: [RN_RDMA_GCSR_INCNPPKTCNT     = 0x60174] = 0x0
Info: [RN_RDMA_GCSR_OUTCNPPKTCNT    = 0x60178] = 0x0
Info: [RN_RDMA_GCSR_OUTRDRSPPKTCNT  = 0x6017c] = 0x1
Info: [RN_RDMA_GCSR_INTSTS          = 0x60184] = 0x11
Info: [RN_RDMA_GCSR_RQINTSTS1       = 0x60190] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS2       = 0x60194] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS3       = 0x60198] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS4       = 0x6019c] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS5       = 0x601a0] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS6       = 0x601a4] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS7       = 0x601a8] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS8       = 0x601ac] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS1       = 0x601b0] = 0x4
Info: [RN_RDMA_GCSR_CQINTSTS2       = 0x601b4] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS3       = 0x601b8] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS4       = 0x601bc] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS5       = 0x601c0] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS6       = 0x601c4] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS7       = 0x601c8] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS8       = 0x601cc] = 0x0
Info: [RN_RDMA_QCSR_CQHEADi         = 0x60330] = 0x1
Info: [RN_RDMA_QCSR_STATSSNi        = 0x60380] = 0x3
Info: [RN_RDMA_QCSR_STATMSNi        = 0x60384] = 0x0
Info: [RN_RDMA_QCSR_STATQPi         = 0x60388] = 0x1f0200
Info: [RN_RDMA_QCSR_STATCURSQPTRi   = 0x6038c] = 0x2
Info: [RN_RDMA_QCSR_STATRESPSNi     = 0x60390] = 0xabe
Info: [RN_RDMA_QCSR_STATRQBUFCAi    = 0x60394] = 0x1a82b000
Info: [RN_RDMA_QCSR_STATWQEi        = 0x60398] = 0x0
Info: [RN_RDMA_QCSR_STATRQPIDBi     = 0x6039c] = 0x0
Info: [RN_RDMA_QCSR_STATRQBUFCAMSBi = 0x603d8] = 0x2f
Info: [RN_RDMA_QCSR_SQPIi           = 0x60338] = 0x2

Failed to send an RDMA read operation for Array B!
Info: Dump register values for debug purpose
Info: [RN_RDMA_GCSR_ERRBUFWPTR      = 0x6006c] = 0x0
Info: [RN_RDMA_GCSR_IPKTERRQWPTR    = 0x60094] = 0x0
Info: [RN_RDMA_GCSR_INSRRPKTCNT     = 0x60100] = 0x1
Info: [RN_RDMA_GCSR_INAMPKTCNT      = 0x60104] = 0x1
Info: [RN_RDMA_GCSR_OUTIOPKTCNT     = 0x60108] = 0x20004
Info: [RN_RDMA_GCSR_OUTAMPKTCNT     = 0x6010c] = 0x1
Info: [RN_RDMA_GCSR_LSTINPKT        = 0x60110] = 0xabe0210
Info: [RN_RDMA_GCSR_LSTOUTPKT       = 0x60114] = 0x157c204
Info: [RN_RDMA_GCSR_ININVDUPCNT     = 0x60118] = 0x1
Info: [RN_RDMA_GCSR_INNCKPKTSTS     = 0x6011c] = 0x0
Info: [RN_RDMA_GCSR_OUTRNRPKTSTS    = 0x60120] = 0x0
Info: [RN_RDMA_GCSR_WQEPROCSTS      = 0x60124] = 0xd0d2040
Info: [RN_RDMA_GCSR_QPMSTS          = 0x6012c] = 0x30002
Info: [RN_RDMA_GCSR_INALLDRPPKTCNT  = 0x60130] = 0x50000
Info: [RN_RDMA_GCSR_INNAKPKTCNT     = 0x60134] = 0x0
Info: [RN_RDMA_GCSR_OUTNAKPKTCNT    = 0x60138] = 0x0
Info: [RN_RDMA_GCSR_RESPHNDSTS      = 0x6013c] = 0x30f02
Info: [RN_RDMA_GCSR_RETRYCNTSTS     = 0x60140] = 0x0
Info: [RN_RDMA_GCSR_INCNPPKTCNT     = 0x60174] = 0x0
Info: [RN_RDMA_GCSR_OUTCNPPKTCNT    = 0x60178] = 0x0
Info: [RN_RDMA_GCSR_OUTRDRSPPKTCNT  = 0x6017c] = 0x1
Info: [RN_RDMA_GCSR_INTSTS          = 0x60184] = 0x11
Info: [RN_RDMA_GCSR_RQINTSTS1       = 0x60190] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS2       = 0x60194] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS3       = 0x60198] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS4       = 0x6019c] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS5       = 0x601a0] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS6       = 0x601a4] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS7       = 0x601a8] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS8       = 0x601ac] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS1       = 0x601b0] = 0x4
Info: [RN_RDMA_GCSR_CQINTSTS2       = 0x601b4] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS3       = 0x601b8] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS4       = 0x601bc] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS5       = 0x601c0] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS6       = 0x601c4] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS7       = 0x601c8] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS8       = 0x601cc] = 0x0
Info: [RN_RDMA_QCSR_CQHEADi         = 0x60330] = 0x1
Info: [RN_RDMA_QCSR_STATSSNi        = 0x60380] = 0x3
Info: [RN_RDMA_QCSR_STATMSNi        = 0x60384] = 0x0
Info: [RN_RDMA_QCSR_STATQPi         = 0x60388] = 0x1f0200
Info: [RN_RDMA_QCSR_STATCURSQPTRi   = 0x6038c] = 0x2
Info: [RN_RDMA_QCSR_STATRESPSNi     = 0x60390] = 0xabe
Info: [RN_RDMA_QCSR_STATRQBUFCAi    = 0x60394] = 0x1a82b000
Info: [RN_RDMA_QCSR_STATWQEi        = 0x60398] = 0x0
Info: [RN_RDMA_QCSR_STATRQPIDBi     = 0x6039c] = 0x0
Info: [RN_RDMA_QCSR_STATRQBUFCAMSBi = 0x603d8] = 0x2f
Info: [RN_RDMA_QCSR_SQPIi           = 0x60338] = 0x2

Info: Is Computation finished, compute_done = 1
Info: The value of rc is 1024
** Avg time device /dev/reconic-mm, total time 0.001005 sec, size = 16
hw_work_id = 0xdd
Error: Result mismatch
i = 0,  CPU result = 250
Hardware result = 0
Test failed!

<IP_ADDRESS>:
 sudo env LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./network_systolic_mm -d /dev/reconic-mm -p /sys/bus/pci/devices/0000\:3b\:00.0
/resource2 -r <IP_ADDRESS> -i <IP_ADDRESS> -u 22222 -t 11111 --dst_qp 2 -s 2>&1 | tee server_debug.log
Info: Device - /dev/reconic-mm
Info: PCIe resource file: /sys/bus/pci/devices/0000:3b:00.0/resource2
src_ip_str = <IP_ADDRESS>
dst_ip_str = <IP_ADDRESS>
Info: mac_addr_t = 00:0a:35:71:f4:5c
Info: src_ip = <IP_ADDRESS>
Info: Found network interface: enp59s0
Info: mac_addr_t = 00:0a:35:24:44:8a
Info: Creating rn_dev
create_rn_dev - testing2
Info: pre-allocated hugepage buffer vir addr = 0x7fecec000000, physical addr = 0x172a400000
Info: Configuring 8 windows in QDMA AXI bridge BDF, each has 128GB mapping
Info: CREATE RDMA DEVICE
Info: OPEN RDMA DEVICE
Info: RDMA global control status registers are configured.
Info: rdma_dev opened
Info: ALLOCATE PD
Info: OPEN DEVICE FILE
Info: ALLOCATE RDMA QP
Allocating qp->sq
Allocating qp->cq
Allocating qp->rq
Info: queue pair setting is done! Configuring RDMA per-queu CSR registers
Info: allocate_rdma_qp - Successfully allocated a rdma qp
Info: CONFIGURE PSN
[Register] RN_RDMA_QCSR_LSTRQREQi=0x60344, qpid=2, value=0xa000abc
[Register] RN_RDMA_QCSR_SQPSNi=0x60340, qpid=2, value=0xabd
Info: generating matrix data, matrix_size = 256
Info: Software Matrix Data generated
Info: Server is listening to a remote peer
Info: Server is connected to a remote peer
Info: rdma_register_memory_region - registering memory region
Info: memory region for the 0-th PD is registered
Info: allocating buffer for array A
Info: mr_bufferA->buffer = 0xa350000000000000, mr_bufferA->dma_addr = 0xa350000000000000
Info: allocating buffer for array B
Info: mr_bufferB->buffer = 0xa350000000000400, mr_bufferB->dma_addr = 0xa350000000000400
Info: copy matrix data to the device memory
Info: Host buffer vir address used for RDMA read operation is mr_bufferA = 0xa350000000000000, mr_bufferB = 0xa350000000000400
Sending read_offsetA (a350000000000000) to the remote client
Sending read_offsetB (a350000000000400) to the remote client
Does the client finish its RDMA read operation? If yes, please press any key

Info: Dump register values for debug purpose
Info: [RN_RDMA_GCSR_ERRBUFWPTR      = 0x6006c] = 0x0
Info: [RN_RDMA_GCSR_IPKTERRQWPTR    = 0x60094] = 0x0
Info: [RN_RDMA_GCSR_INSRRPKTCNT     = 0x60100] = 0x10004
Info: [RN_RDMA_GCSR_INAMPKTCNT      = 0x60104] = 0x1
Info: [RN_RDMA_GCSR_OUTIOPKTCNT     = 0x60108] = 0x20000
Info: [RN_RDMA_GCSR_OUTAMPKTCNT     = 0x6010c] = 0x1
Info: [RN_RDMA_GCSR_LSTINPKT        = 0x60110] = 0xabe020c
Info: [RN_RDMA_GCSR_LSTOUTPKT       = 0x60114] = 0x157a200
Info: [RN_RDMA_GCSR_ININVDUPCNT     = 0x60118] = 0x0
Info: [RN_RDMA_GCSR_INNCKPKTSTS     = 0x6011c] = 0x0
Info: [RN_RDMA_GCSR_OUTRNRPKTSTS    = 0x60120] = 0x0
Info: [RN_RDMA_GCSR_WQEPROCSTS      = 0x60124] = 0x8082000
Info: [RN_RDMA_GCSR_QPMSTS          = 0x6012c] = 0x20002
Info: [RN_RDMA_GCSR_INALLDRPPKTCNT  = 0x60130] = 0x80000
Info: [RN_RDMA_GCSR_INNAKPKTCNT     = 0x60134] = 0x0
Info: [RN_RDMA_GCSR_OUTNAKPKTCNT    = 0x60138] = 0x0
Info: [RN_RDMA_GCSR_RESPHNDSTS      = 0x6013c] = 0x10002
Info: [RN_RDMA_GCSR_RETRYCNTSTS     = 0x60140] = 0x0
Info: [RN_RDMA_GCSR_INCNPPKTCNT     = 0x60174] = 0x0
Info: [RN_RDMA_GCSR_OUTCNPPKTCNT    = 0x60178] = 0x0
Info: [RN_RDMA_GCSR_OUTRDRSPPKTCNT  = 0x6017c] = 0x2
Info: [RN_RDMA_GCSR_INTSTS          = 0x60184] = 0x10
Info: [RN_RDMA_GCSR_RQINTSTS1       = 0x60190] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS2       = 0x60194] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS3       = 0x60198] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS4       = 0x6019c] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS5       = 0x601a0] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS6       = 0x601a4] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS7       = 0x601a8] = 0x0
Info: [RN_RDMA_GCSR_RQINTSTS8       = 0x601ac] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS1       = 0x601b0] = 0x4
Info: [RN_RDMA_GCSR_CQINTSTS2       = 0x601b4] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS3       = 0x601b8] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS4       = 0x601bc] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS5       = 0x601c0] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS6       = 0x601c4] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS7       = 0x601c8] = 0x0
Info: [RN_RDMA_GCSR_CQINTSTS8       = 0x601cc] = 0x0
Info: [RN_RDMA_QCSR_CQHEADi         = 0x60330] = 0x0
Info: [RN_RDMA_QCSR_STATSSNi        = 0x60380] = 0x2
Info: [RN_RDMA_QCSR_STATMSNi        = 0x60384] = 0x2
Info: [RN_RDMA_QCSR_STATQPi         = 0x60388] = 0x1f0600
Info: [RN_RDMA_QCSR_STATCURSQPTRi   = 0x6038c] = 0x0
Info: [RN_RDMA_QCSR_STATRESPSNi     = 0x60390] = 0xabc
Info: [RN_RDMA_QCSR_STATRQBUFCAi    = 0x60394] = 0x2ba2b000
Info: [RN_RDMA_QCSR_STATWQEi        = 0x60398] = 0x0
Info: [RN_RDMA_QCSR_STATRQPIDBi     = 0x6039c] = 0x0
Info: [RN_RDMA_QCSR_STATRQBUFCAMSBi = 0x603d8] = 0x17

no error message from dmesg on <IP_ADDRESS>.
error message from dmesg on <IP_ADDRESS>:
[  700.598882] onic:error_intr_handler: Error IRQ fired on Funtion#0: index=7, vector=272
[  700.598889] eqdma_hw_error_process: Global Err Reg(0x248) = 0x4
[  700.598903] addr = 0x00000254 val = 0x00100000
[  700.598908] GLBL_DSC_ERR_STS                         0x254     0x100000   1048576
[  700.598909] GLBL_DSC_ERR_STS_RSVD_1                  [31,26]   0x0
[  700.598909] GLBL_DSC_ERR_STS_PORT_ID                 [   25]   0x0
[  700.598910] GLBL_DSC_ERR_STS_SBE                     [   24]   0x0
[  700.598911] GLBL_DSC_ERR_STS_DBE                     [   23]   0x0
[  700.598911] GLBL_DSC_ERR_STS_RQ_CANCEL               [   22]   0x0
[  700.598912] GLBL_DSC_ERR_STS_DSC                     [   21]   0x0
[  700.598912] GLBL_DSC_ERR_STS_DMA                     [   20]   0x1
[  700.598913] GLBL_DSC_ERR_STS_FLR_CANCEL              [   19]   0x0
[  700.598914] GLBL_DSC_ERR_STS_RSVD_2                  [18,17]   0x0
[  700.598914] GLBL_DSC_ERR_STS_DAT_POISON              [   16]   0x0
[  700.598915] GLBL_DSC_ERR_STS_TIMEOUT                 [    9]   0x0
[  700.598915] GLBL_DSC_ERR_STS_FLR                     [    8]   0x0
[  700.598916] GLBL_DSC_ERR_STS_TAG                     [    6]   0x0
[  700.598916] GLBL_DSC_ERR_STS_ADDR                    [    5]   0x0
[  700.598916] GLBL_DSC_ERR_STS_PARAM                   [    4]   0x0
[  700.598917] GLBL_DSC_ERR_STS_BCNT                    [    3]   0x0
[  700.598917] GLBL_DSC_ERR_STS_UR_CA                   [    2]   0x0
[  700.598918] GLBL_DSC_ERR_STS_POISON                  [    1]   0x0
[  700.598922] GLBL_DSC_ERR_LOG0                        0x25c     0xc0000041 -1073741759
[  700.598922] GLBL_DSC_ERR_LOG0_VALID                  [   31]   0x1
[  700.598923] GLBL_DSC_ERR_LOG0_SEL                    [   30]   0x1
[  700.598923] GLBL_DSC_ERR_LOG0_RSVD_1                 [29,13]   0x0
[  700.598924] GLBL_DSC_ERR_LOG0_QID                    [12, 0]   0x41
[  700.598928] GLBL_DSC_ERR_LOG1                        0x260     0x1014     4116
[  700.598928] GLBL_DSC_ERR_LOG1_RSVD_1                 [31,28]   0x0
[  700.598929] GLBL_DSC_ERR_LOG1_CIDX                   [27,12]   0x1
[  700.598929] GLBL_DSC_ERR_LOG1_RSVD_2                 [11, 9]   0x0
[  700.598930] GLBL_DSC_ERR_LOG1_SUB_TYPE               [ 8, 5]   0x0
[  700.598930] GLBL_DSC_ERR_LOG1_ERR_TYPE               [ 4, 0]   0x14
[  700.598934] GLBL_DSC_DBG_DAT0                        0x270     0x0        0
[  700.598934] GLBL_DSC_DAT0_RSVD_1                     [31,30]   0x0
[  700.598935] GLBL_DSC_DAT0_CTXT_ARB_DIR               [   29]   0x0
[  700.598935] GLBL_DSC_DAT0_CTXT_ARB_QID               [28,17]   0x0
[  700.598936] GLBL_DSC_DAT0_CTXT_ARB_REQ               [16,12]   0x0
[  700.598936] GLBL_DSC_DAT0_IRQ_FIFO_FL                [   11]   0x0
[  700.598937] GLBL_DSC_DAT0_TMSTALL                    [   10]   0x0
[  700.598937] GLBL_DSC_DAT0_RRQ_STALL                  [ 9, 8]   0x0
[  700.598938] GLBL_DSC_DAT0_RCP_FIFO_SPC_STALL         [ 7, 6]   0x0
[  700.598938] GLBL_DSC_DAT0_RRQ_FIFO_SPC_STALL         [ 5, 4]   0x0
[  700.598939] GLBL_DSC_DAT0_FAB_MRKR_RSP_STALL         [ 3, 2]   0x0
[  700.598939] GLBL_DSC_DAT0_DSC_OUT_STALL              [ 1, 0]   0x0
[  700.598943] GLBL_DSC_DBG_DAT1                        0x274     0x0        0
[  700.598944] GLBL_DSC_DAT1_RSVD_1                     [31,28]   0x0
[  700.598944] GLBL_DSC_DAT1_EVT_SPC_C2H                [27,22]   0x0
[  700.598945] GLBL_DSC_DAT1_EVT_SP_H2C                 [21,16]   0x0
[  700.598945] GLBL_DSC_DAT1_DSC_SPC_C2H                [15, 8]   0x0
[  700.598945] GLBL_DSC_DAT1_DSC_SPC_H2C                [ 7, 0]   0x0
[  700.598949] GLBL_DSC_ERR_LOG2                        0x27c     0x10001    65537
[  700.598950] GLBL_DSC_ERR_LOG2_OLD_PIDX               [31,16]   0x1
[  700.598950] GLBL_DSC_ERR_LOG2_NEW_PIDX               [15, 0]   0x1
[  700.598951] eqdma_hw_error_process detected DMA engine error

Then I run the dma test on <IP_ADDRESS>:
qianyich@pc167:~/RecoNIC/examples/dma_test$ sudo ./dma_test -d /dev/reconic-mm -s 65536000 -c 200
Write scenario
/dev/reconic-mm, W off 0x0, 0x3e80000 failed -1.
write file: Input/output error
qianyich@pc167:~/RecoNIC/examples/dma_test$ sudo ./dma_test -d /dev/reconic-mm -s 65536000 -c 200 -r
Read scenario
/dev/reconic-mm, read off 0x0 + 0x3e80000 failed -1.
read file: Input/output error

dmesg:
[  795.624964] onic:qdma_request_wait_for_cmpl: qdma3b000-MM-67: req 0x000000002c539dbf, W,128,0/128,0x0, done 0, err 0, tm 10000.
[  795.636427] onic:qdma_descq_dump: qdma3b000-MM-67: 0x43/0x43, desc sz 1024/1022, pidx 1, cidx 0
[  795.636443] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.

@zhguanw-amd Another problem is after I ran read, write, and send_recv test successfully, I went ahead to test dma on both servers.
dma failed to work on <IP_ADDRESS>:
qianyich@pc166:~/RecoNIC/examples/dma_test$ sudo ./dma_test -d /dev/reconic-mm -s 65536000 -c 200
Write scenario
size=65536000 Average BW = 4.884794 GB/sec, average latency = 13416.328467 us
qianyich@pc166:~/RecoNIC/examples/dma_test$ sudo ./dma_test -d /dev/reconic-mm -s 65536000 -c 200 -r
Read scenario
/dev/reconic-mm, read off 0x0 + 0x3e80000 failed -1.
read file: Input/output error

dmesg:
[  492.857842] onic:error_intr_handler: Error IRQ fired on Funtion#0: index=7, vector=272
[  492.857849] eqdma_hw_error_process: Global Err Reg(0x248) = 0x4
[  492.857852] addr = 0x00000254 val = 0x00100000
[  492.857858] GLBL_DSC_ERR_STS                         0x254     0x100000   1048576
[  492.857861] GLBL_DSC_ERR_STS_RSVD_1                  [31,26]   0x0
[  492.857862] GLBL_DSC_ERR_STS_PORT_ID                 [   25]   0x0
[  492.857864] GLBL_DSC_ERR_STS_SBE                     [   24]   0x0
[  492.857865] GLBL_DSC_ERR_STS_DBE                     [   23]   0x0
[  492.857867] GLBL_DSC_ERR_STS_RQ_CANCEL               [   22]   0x0
[  492.857868] GLBL_DSC_ERR_STS_DSC                     [   21]   0x0
[  492.857869] GLBL_DSC_ERR_STS_DMA                     [   20]   0x1
[  492.857870] GLBL_DSC_ERR_STS_FLR_CANCEL              [   19]   0x0
[  492.857872] GLBL_DSC_ERR_STS_RSVD_2                  [18,17]   0x0
[  492.857873] GLBL_DSC_ERR_STS_DAT_POISON              [   16]   0x0
[  492.857875] GLBL_DSC_ERR_STS_TIMEOUT                 [    9]   0x0
[  492.857876] GLBL_DSC_ERR_STS_FLR                     [    8]   0x0
[  492.857877] GLBL_DSC_ERR_STS_TAG                     [    6]   0x0
[  492.857879] GLBL_DSC_ERR_STS_ADDR                    [    5]   0x0
[  492.857880] GLBL_DSC_ERR_STS_PARAM                   [    4]   0x0
[  492.857881] GLBL_DSC_ERR_STS_BCNT                    [    3]   0x0
[  492.857882] GLBL_DSC_ERR_STS_UR_CA                   [    2]   0x0
[  492.857884] GLBL_DSC_ERR_STS_POISON                  [    1]   0x0
[  492.857889] GLBL_DSC_ERR_LOG0                        0x25c     0xc0000041 -1073741759
[  492.857890] GLBL_DSC_ERR_LOG0_VALID                  [   31]   0x1
[  492.857892] GLBL_DSC_ERR_LOG0_SEL                    [   30]   0x1
[  492.857893] GLBL_DSC_ERR_LOG0_RSVD_1                 [29,13]   0x0
[  492.857895] GLBL_DSC_ERR_LOG0_QID                    [12, 0]   0x41
[  492.857900] GLBL_DSC_ERR_LOG1                        0x260     0x1014     4116
[  492.857902] GLBL_DSC_ERR_LOG1_RSVD_1                 [31,28]   0x0
[  492.857903] GLBL_DSC_ERR_LOG1_CIDX                   [27,12]   0x1
[  492.857905] GLBL_DSC_ERR_LOG1_RSVD_2                 [11, 9]   0x0
[  492.857906] GLBL_DSC_ERR_LOG1_SUB_TYPE               [ 8, 5]   0x0
[  492.857908] GLBL_DSC_ERR_LOG1_ERR_TYPE               [ 4, 0]   0x14
[  492.857912] GLBL_DSC_DBG_DAT0                        0x270     0x0        0
[  492.857914] GLBL_DSC_DAT0_RSVD_1                     [31,30]   0x0
[  492.857915] GLBL_DSC_DAT0_CTXT_ARB_DIR               [   29]   0x0
[  492.857917] GLBL_DSC_DAT0_CTXT_ARB_QID               [28,17]   0x0
[  492.857918] GLBL_DSC_DAT0_CTXT_ARB_REQ               [16,12]   0x0
[  492.857920] GLBL_DSC_DAT0_IRQ_FIFO_FL                [   11]   0x0
[  492.857921] GLBL_DSC_DAT0_TMSTALL                    [   10]   0x0
[  492.857922] GLBL_DSC_DAT0_RRQ_STALL                  [ 9, 8]   0x0
[  492.857924] GLBL_DSC_DAT0_RCP_FIFO_SPC_STALL         [ 7, 6]   0x0
[  492.857926] GLBL_DSC_DAT0_RRQ_FIFO_SPC_STALL         [ 5, 4]   0x0
[  492.857927] GLBL_DSC_DAT0_FAB_MRKR_RSP_STALL         [ 3, 2]   0x0
[  492.857928] GLBL_DSC_DAT0_DSC_OUT_STALL              [ 1, 0]   0x0
[  492.857933] GLBL_DSC_DBG_DAT1                        0x274     0x0        0
[  492.857934] GLBL_DSC_DAT1_RSVD_1                     [31,28]   0x0
[  492.857936] GLBL_DSC_DAT1_EVT_SPC_C2H                [27,22]   0x0
[  492.857937] GLBL_DSC_DAT1_EVT_SP_H2C                 [21,16]   0x0
[  492.857939] GLBL_DSC_DAT1_DSC_SPC_C2H                [15, 8]   0x0
[  492.857940] GLBL_DSC_DAT1_DSC_SPC_H2C                [ 7, 0]   0x0
[  492.857945] GLBL_DSC_ERR_LOG2                        0x27c     0x3ff03ff  67044351
[  492.857946] GLBL_DSC_ERR_LOG2_OLD_PIDX               [31,16]   0x3ff
[  492.857948] GLBL_DSC_ERR_LOG2_NEW_PIDX               [15, 0]   0x3ff
[  492.857949] eqdma_hw_error_process detected DMA engine error
[  503.005208] onic:qdma_request_wait_for_cmpl: qdma3b000-MM-65: req 0x00000000bf355690, R,4194304,0/65536000,0x0, done 0, err 0, tm 10000.
[  503.017464] onic:qdma_descq_dump: qdma3b000-MM-65: 0x41/0x41, desc sz 1024/0, pidx 0, cidx 1
[  503.018028] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[  512.944103] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.

@zhguanw-amd  Also sometimes when I insert the onic.ko module, I have the following errors from dmesg. This happens sometimes and a reboot helps resolve this issue, but not sure why this happens randomly. I just report this, and it could be related to the aforementioned QDMA error.
[  905.919063] qdma_is_config_bar: Invalid config bar, err:-4
[  905.924560] qdma_hw_access_init: config bar passed is INVALID, err:-1
[  905.931044] onic 0000:3b:00.0: onic_qdma_setup: qdma_device_open() failed: Error Code: -22
[  905.939304] onic 0000:3b:00.0: onic_pci_probe: onic_qdma_setup() failed with status -22
[  905.947330] onic: probe of 0000:3b:00.0 failed with error -22

@qianyich did you apply the new onic.patch to RecoNIC/drivers/onic-driver? It seems you're probably working on an old driver. I can run network_systolic_mm without issue. But there is seg fault issue when freeing some buffers and it has nothing to do with qdma and rdma. I will push a fix for this seg fault.
Can you take a screenshot of Line 439 - 453 in onic_main.c?

@zhguanw-amd The new onic.patch has been applied.

The problem is not directly related to network_systolic_mm. After I ran the read, write, and send_recv tests with no issues, the DMA test failed on <IP_ADDRESS>. I will run those tests again tomorrow.

I will reprogram the board and run network_systolic_mm first tomorrow to see if that works, and then run those verb tests, and finally the dma test.

Yes, please reprogram the board and make sure it has the fresh environment.

@zhguanw-amd The new onic.patch has been applied.

The problem is not directly related to network_systolic_mm. After I ran the read, write, and send_recv tests with no issues, the DMA test failed on <IP_ADDRESS> (this is what I had after I rebooted both machines and reinserted the kernel module, I did not run the network systolic array example that time). I will run those tests again tomorrow.

Could you comment out Line 12 and 14 in your screenshot above and only leave Line 13, and have a try?

@zhguanw-amd I commented out the lines you mentioned, and I ran network_systolic_mm first, it showed Test Passed. And then I ran read, write, and send_recv tests, and they all worked with no issues.  After these tests, I tried to run network_systolic_mm example again, and it failed.
dmesg on <IP_ADDRESS>, maybe this is the seg fault you mentioned before?:
[  952.640278] network_systoli[15528]: segfault at 122000000de ip 00007f864e53d94d sp 00007ffffa01f3a0 error 4 in libc-2.27.so[7f864e4a6000+1e7000]
[ 1054.218446] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[ 1140.048294] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[ 1268.669567] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[ 1306.199807] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[ 1306.199813] network_systoli[15626]: segfault at 122000000de ip 00007fbda6a2b94d sp 00007ffe9986c260 error 4 in libc-2.27.so[7fbda6994000+1e7000]

But this somehow has affected QDMA. The DMA tests failed as well. It is an error that is similar to the one I had yesterday.
[ 1355.762811] onic:qdma_request_wait_for_cmpl: qdma3b000-MM-67: req 0x00000000a25c3472, W,65536000,0/65536000,0x0, done 0, err 0, tm 10000.
[ 1355.775149] onic:qdma_descq_dump: qdma3b000-MM-67: 0x43/0x43, desc sz 1024/895, pidx 640, cidx 512
[ 1355.775478] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[ 1358.805712] onic:error_intr_handler: Error IRQ fired on Funtion#0: index=7, vector=272
[ 1358.805717] eqdma_hw_error_process: Global Err Reg(0x248) = 0x4
[ 1358.805719] addr = 0x00000254 val = 0x00100000
[ 1358.805723] GLBL_DSC_ERR_STS                         0x254     0x100000   1048576
[ 1358.805724] GLBL_DSC_ERR_STS_RSVD_1                  [31,26]   0x0
[ 1358.805725] GLBL_DSC_ERR_STS_PORT_ID                 [   25]   0x0
[ 1358.805726] GLBL_DSC_ERR_STS_SBE                     [   24]   0x0
[ 1358.805726] GLBL_DSC_ERR_STS_DBE                     [   23]   0x0
[ 1358.805727] GLBL_DSC_ERR_STS_RQ_CANCEL               [   22]   0x0
[ 1358.805727] GLBL_DSC_ERR_STS_DSC                     [   21]   0x0
[ 1358.805727] GLBL_DSC_ERR_STS_DMA                     [   20]   0x1
[ 1358.805728] GLBL_DSC_ERR_STS_FLR_CANCEL              [   19]   0x0
[ 1358.805729] GLBL_DSC_ERR_STS_RSVD_2                  [18,17]   0x0
[ 1358.805729] GLBL_DSC_ERR_STS_DAT_POISON              [   16]   0x0
[ 1358.805729] GLBL_DSC_ERR_STS_TIMEOUT                 [    9]   0x0
[ 1358.805730] GLBL_DSC_ERR_STS_FLR                     [    8]   0x0
[ 1358.805730] GLBL_DSC_ERR_STS_TAG                     [    6]   0x0
[ 1358.805731] GLBL_DSC_ERR_STS_ADDR                    [    5]   0x0
[ 1358.805731] GLBL_DSC_ERR_STS_PARAM                   [    4]   0x0
[ 1358.805732] GLBL_DSC_ERR_STS_BCNT                    [    3]   0x0
[ 1358.805732] GLBL_DSC_ERR_STS_UR_CA                   [    2]   0x0
[ 1358.805732] GLBL_DSC_ERR_STS_POISON                  [    1]   0x0
[ 1358.805736] GLBL_DSC_ERR_LOG0                        0x25c     0xc0000043 -1073741757
[ 1358.805737] GLBL_DSC_ERR_LOG0_VALID                  [   31]   0x1
[ 1358.805737] GLBL_DSC_ERR_LOG0_SEL                    [   30]   0x1
[ 1358.805738] GLBL_DSC_ERR_LOG0_RSVD_1                 [29,13]   0x0
[ 1358.805738] GLBL_DSC_ERR_LOG0_QID                    [12, 0]   0x43
[ 1358.805742] GLBL_DSC_ERR_LOG1                        0x260     0x280014   2621460
[ 1358.805742] GLBL_DSC_ERR_LOG1_RSVD_1                 [31,28]   0x0
[ 1358.805743] GLBL_DSC_ERR_LOG1_CIDX                   [27,12]   0x280
[ 1358.805743] GLBL_DSC_ERR_LOG1_RSVD_2                 [11, 9]   0x0
[ 1358.805744] GLBL_DSC_ERR_LOG1_SUB_TYPE               [ 8, 5]   0x0
[ 1358.805744] GLBL_DSC_ERR_LOG1_ERR_TYPE               [ 4, 0]   0x14
[ 1358.805748] GLBL_DSC_DBG_DAT0                        0x270     0x0        0
[ 1358.805748] GLBL_DSC_DAT0_RSVD_1                     [31,30]   0x0
[ 1358.805749] GLBL_DSC_DAT0_CTXT_ARB_DIR               [   29]   0x0
[ 1358.805749] GLBL_DSC_DAT0_CTXT_ARB_QID               [28,17]   0x0
[ 1358.805749] GLBL_DSC_DAT0_CTXT_ARB_REQ               [16,12]   0x0
[ 1358.805750] GLBL_DSC_DAT0_IRQ_FIFO_FL                [   11]   0x0
[ 1358.805750] GLBL_DSC_DAT0_TMSTALL                    [   10]   0x0
[ 1358.805751] GLBL_DSC_DAT0_RRQ_STALL                  [ 9, 8]   0x0
[ 1358.805752] GLBL_DSC_DAT0_RCP_FIFO_SPC_STALL         [ 7, 6]   0x0
[ 1358.805752] GLBL_DSC_DAT0_RRQ_FIFO_SPC_STALL         [ 5, 4]   0x0
[ 1358.805753] GLBL_DSC_DAT0_FAB_MRKR_RSP_STALL         [ 3, 2]   0x0
[ 1358.805753] GLBL_DSC_DAT0_DSC_OUT_STALL              [ 1, 0]   0x0
[ 1358.805757] GLBL_DSC_DBG_DAT1                        0x274     0x0        0
[ 1358.805757] GLBL_DSC_DAT1_RSVD_1                     [31,28]   0x0
[ 1358.805758] GLBL_DSC_DAT1_EVT_SPC_C2H                [27,22]   0x0
[ 1358.805759] GLBL_DSC_DAT1_EVT_SP_H2C                 [21,16]   0x0
[ 1358.805759] GLBL_DSC_DAT1_DSC_SPC_C2H                [15, 8]   0x0
[ 1358.805760] GLBL_DSC_DAT1_DSC_SPC_H2C                [ 7, 0]   0x0
[ 1358.805763] GLBL_DSC_ERR_LOG2                        0x27c     0x2800280  41943680
[ 1358.805764] GLBL_DSC_ERR_LOG2_OLD_PIDX               [31,16]   0x280
[ 1358.805764] GLBL_DSC_ERR_LOG2_NEW_PIDX               [15, 0]   0x280
[ 1358.805765] eqdma_hw_error_process detected DMA engine error
[ 1358.819350] onic:error_intr_handler: Error IRQ fired on Funtion#0: index=7, vector=272
[ 1358.819357] eqdma_hw_error_process: Global Err Reg(0x248) = 0x4
[ 1358.819359] addr = 0x00000254 val = 0x00100000
[ 1358.819366] GLBL_DSC_ERR_STS                         0x254     0x100000   1048576
[ 1358.819368] GLBL_DSC_ERR_STS_RSVD_1                  [31,26]   0x0
[ 1358.819370] GLBL_DSC_ERR_STS_PORT_ID                 [   25]   0x0
[ 1358.819371] GLBL_DSC_ERR_STS_SBE                     [   24]   0x0
[ 1358.819373] GLBL_DSC_ERR_STS_DBE                     [   23]   0x0
[ 1358.819374] GLBL_DSC_ERR_STS_RQ_CANCEL               [   22]   0x0
[ 1358.819375] GLBL_DSC_ERR_STS_DSC                     [   21]   0x0
[ 1358.819377] GLBL_DSC_ERR_STS_DMA                     [   20]   0x1
[ 1358.819378] GLBL_DSC_ERR_STS_FLR_CANCEL              [   19]   0x0
[ 1358.819379] GLBL_DSC_ERR_STS_RSVD_2                  [18,17]   0x0
[ 1358.819381] GLBL_DSC_ERR_STS_DAT_POISON              [   16]   0x0
[ 1358.819382] GLBL_DSC_ERR_STS_TIMEOUT                 [    9]   0x0
[ 1358.819383] GLBL_DSC_ERR_STS_FLR                     [    8]   0x0
[ 1358.819385] GLBL_DSC_ERR_STS_TAG                     [    6]   0x0
[ 1358.819386] GLBL_DSC_ERR_STS_ADDR                    [    5]   0x0
[ 1358.819387] GLBL_DSC_ERR_STS_PARAM                   [    4]   0x0
[ 1358.819389] GLBL_DSC_ERR_STS_BCNT                    [    3]   0x0
[ 1358.819390] GLBL_DSC_ERR_STS_UR_CA                   [    2]   0x0
[ 1358.819391] GLBL_DSC_ERR_STS_POISON                  [    1]   0x0
[ 1358.819396] GLBL_DSC_ERR_LOG0                        0x25c     0xc0000043 -1073741757
[ 1358.819398] GLBL_DSC_ERR_LOG0_VALID                  [   31]   0x1
[ 1358.819399] GLBL_DSC_ERR_LOG0_SEL                    [   30]   0x1
[ 1358.819401] GLBL_DSC_ERR_LOG0_RSVD_1                 [29,13]   0x0
[ 1358.819402] GLBL_DSC_ERR_LOG0_QID                    [12, 0]   0x43
[ 1358.819407] GLBL_DSC_ERR_LOG1                        0x260     0x280014   2621460
[ 1358.819409] GLBL_DSC_ERR_LOG1_RSVD_1                 [31,28]   0x0
[ 1358.819410] GLBL_DSC_ERR_LOG1_CIDX                   [27,12]   0x280
[ 1358.819412] GLBL_DSC_ERR_LOG1_RSVD_2                 [11, 9]   0x0
[ 1358.819413] GLBL_DSC_ERR_LOG1_SUB_TYPE               [ 8, 5]   0x0
[ 1358.819415] GLBL_DSC_ERR_LOG1_ERR_TYPE               [ 4, 0]   0x14
[ 1358.819419] GLBL_DSC_DBG_DAT0                        0x270     0x0        0
[ 1358.819421] GLBL_DSC_DAT0_RSVD_1                     [31,30]   0x0
[ 1358.819422] GLBL_DSC_DAT0_CTXT_ARB_DIR               [   29]   0x0
[ 1358.819424] GLBL_DSC_DAT0_CTXT_ARB_QID               [28,17]   0x0
[ 1358.819425] GLBL_DSC_DAT0_CTXT_ARB_REQ               [16,12]   0x0
[ 1358.819426] GLBL_DSC_DAT0_IRQ_FIFO_FL                [   11]   0x0
[ 1358.819428] GLBL_DSC_DAT0_TMSTALL                    [   10]   0x0
[ 1358.819429] GLBL_DSC_DAT0_RRQ_STALL                  [ 9, 8]   0x0
[ 1358.819431] GLBL_DSC_DAT0_RCP_FIFO_SPC_STALL         [ 7, 6]   0x0
[ 1358.819432] GLBL_DSC_DAT0_RRQ_FIFO_SPC_STALL         [ 5, 4]   0x0
[ 1358.819434] GLBL_DSC_DAT0_FAB_MRKR_RSP_STALL         [ 3, 2]   0x0
[ 1358.819435] GLBL_DSC_DAT0_DSC_OUT_STALL              [ 1, 0]   0x0
[ 1358.819440] GLBL_DSC_DBG_DAT1                        0x274     0x0        0
[ 1358.819441] GLBL_DSC_DAT1_RSVD_1                     [31,28]   0x0
[ 1358.819443] GLBL_DSC_DAT1_EVT_SPC_C2H                [27,22]   0x0
[ 1358.819444] GLBL_DSC_DAT1_EVT_SP_H2C                 [21,16]   0x0
[ 1358.819446] GLBL_DSC_DAT1_DSC_SPC_C2H                [15, 8]   0x0
[ 1358.819447] GLBL_DSC_DAT1_DSC_SPC_H2C                [ 7, 0]   0x0
[ 1358.819452] GLBL_DSC_ERR_LOG2                        0x27c     0x2800280  41943680
[ 1358.819453] GLBL_DSC_ERR_LOG2_OLD_PIDX               [31,16]   0x280
[ 1358.819455] GLBL_DSC_ERR_LOG2_NEW_PIDX               [15, 0]   0x280
[ 1358.819456] eqdma_hw_error_process detected DMA engine error
[ 1358.833053] onic:error_intr_handler: Error IRQ fired on Funtion#0: index=7, vector=272
[ 1358.833059] eqdma_hw_error_process: Global Err Reg(0x248) = 0x4
[ 1358.833062] addr = 0x00000254 val = 0x00100000
[ 1358.833068] GLBL_DSC_ERR_STS                         0x254     0x100000   1048576
[ 1358.833071] GLBL_DSC_ERR_STS_RSVD_1                  [31,26]   0x0
[ 1358.833072] GLBL_DSC_ERR_STS_PORT_ID                 [   25]   0x0
[ 1358.833074] GLBL_DSC_ERR_STS_SBE                     [   24]   0x0
[ 1358.833075] GLBL_DSC_ERR_STS_DBE                     [   23]   0x0
[ 1358.833077] GLBL_DSC_ERR_STS_RQ_CANCEL               [   22]   0x0
[ 1358.833078] GLBL_DSC_ERR_STS_DSC                     [   21]   0x0
[ 1358.833079] GLBL_DSC_ERR_STS_DMA                     [   20]   0x1
[ 1358.833080] GLBL_DSC_ERR_STS_FLR_CANCEL              [   19]   0x0
[ 1358.833082] GLBL_DSC_ERR_STS_RSVD_2                  [18,17]   0x0
[ 1358.833084] GLBL_DSC_ERR_STS_DAT_POISON              [   16]   0x0
[ 1358.833085] GLBL_DSC_ERR_STS_TIMEOUT                 [    9]   0x0
[ 1358.833086] GLBL_DSC_ERR_STS_FLR                     [    8]   0x0
[ 1358.833088] GLBL_DSC_ERR_STS_TAG                     [    6]   0x0
[ 1358.833089] GLBL_DSC_ERR_STS_ADDR                    [    5]   0x0
[ 1358.833090] GLBL_DSC_ERR_STS_PARAM                   [    4]   0x0
[ 1358.833091] GLBL_DSC_ERR_STS_BCNT                    [    3]   0x0
[ 1358.833093] GLBL_DSC_ERR_STS_UR_CA                   [    2]   0x0
[ 1358.833094] GLBL_DSC_ERR_STS_POISON                  [    1]   0x0
[ 1358.833099] GLBL_DSC_ERR_LOG0                        0x25c     0xc0000043 -1073741757
[ 1358.833101] GLBL_DSC_ERR_LOG0_VALID                  [   31]   0x1
[ 1358.833102] GLBL_DSC_ERR_LOG0_SEL                    [   30]   0x1
[ 1358.833104] GLBL_DSC_ERR_LOG0_RSVD_1                 [29,13]   0x0
[ 1358.833105] GLBL_DSC_ERR_LOG0_QID                    [12, 0]   0x43
[ 1358.833110] GLBL_DSC_ERR_LOG1                        0x260     0x280014   2621460
[ 1358.833112] GLBL_DSC_ERR_LOG1_RSVD_1                 [31,28]   0x0
[ 1358.833113] GLBL_DSC_ERR_LOG1_CIDX                   [27,12]   0x280
[ 1358.833115] GLBL_DSC_ERR_LOG1_RSVD_2                 [11, 9]   0x0
[ 1358.833116] GLBL_DSC_ERR_LOG1_SUB_TYPE               [ 8, 5]   0x0
[ 1358.833118] GLBL_DSC_ERR_LOG1_ERR_TYPE               [ 4, 0]   0x14
[ 1358.833122] GLBL_DSC_DBG_DAT0                        0x270     0x0        0
[ 1358.833124] GLBL_DSC_DAT0_RSVD_1                     [31,30]   0x0
[ 1358.833125] GLBL_DSC_DAT0_CTXT_ARB_DIR               [   29]   0x0
[ 1358.833127] GLBL_DSC_DAT0_CTXT_ARB_QID               [28,17]   0x0
[ 1358.833128] GLBL_DSC_DAT0_CTXT_ARB_REQ               [16,12]   0x0
[ 1358.833130] GLBL_DSC_DAT0_IRQ_FIFO_FL                [   11]   0x0
[ 1358.833131] GLBL_DSC_DAT0_TMSTALL                    [   10]   0x0
[ 1358.833133] GLBL_DSC_DAT0_RRQ_STALL                  [ 9, 8]   0x0
[ 1358.833134] GLBL_DSC_DAT0_RCP_FIFO_SPC_STALL         [ 7, 6]   0x0
[ 1358.833136] GLBL_DSC_DAT0_RRQ_FIFO_SPC_STALL         [ 5, 4]   0x0
[ 1358.833137] GLBL_DSC_DAT0_FAB_MRKR_RSP_STALL         [ 3, 2]   0x0
[ 1358.833139] GLBL_DSC_DAT0_DSC_OUT_STALL              [ 1, 0]   0x0
[ 1358.833143] GLBL_DSC_DBG_DAT1                        0x274     0x0        0
[ 1358.833145] GLBL_DSC_DAT1_RSVD_1                     [31,28]   0x0
[ 1358.833146] GLBL_DSC_DAT1_EVT_SPC_C2H                [27,22]   0x0
[ 1358.833148] GLBL_DSC_DAT1_EVT_SP_H2C                 [21,16]   0x0
[ 1358.833149] GLBL_DSC_DAT1_DSC_SPC_C2H                [15, 8]   0x0
[ 1358.833150] GLBL_DSC_DAT1_DSC_SPC_H2C                [ 7, 0]   0x0
[ 1358.833155] GLBL_DSC_ERR_LOG2                        0x27c     0x2800280  41943680
[ 1358.833157] GLBL_DSC_ERR_LOG2_OLD_PIDX               [31,16]   0x280
[ 1358.833158] GLBL_DSC_ERR_LOG2_NEW_PIDX               [15, 0]   0x280
[ 1358.833160] eqdma_hw_error_process detected DMA engine error
[ 1358.846799] onic:error_intr_handler: Error IRQ fired on Funtion#0: index=7, vector=272
[ 1358.846806] eqdma_hw_error_process: Global Err Reg(0x248) = 0x4
[ 1358.846809] addr = 0x00000254 val = 0x00100000
[ 1358.846815] GLBL_DSC_ERR_STS                         0x254     0x100000   1048576
[ 1358.846817] GLBL_DSC_ERR_STS_RSVD_1                  [31,26]   0x0
[ 1358.846819] GLBL_DSC_ERR_STS_PORT_ID                 [   25]   0x0
[ 1358.846821] GLBL_DSC_ERR_STS_SBE                     [   24]   0x0
[ 1358.846822] GLBL_DSC_ERR_STS_DBE                     [   23]   0x0
[ 1358.846823] GLBL_DSC_ERR_STS_RQ_CANCEL               [   22]   0x0
[ 1358.846825] GLBL_DSC_ERR_STS_DSC                     [   21]   0x0
[ 1358.846826] GLBL_DSC_ERR_STS_DMA                     [   20]   0x1
[ 1358.846827] GLBL_DSC_ERR_STS_FLR_CANCEL              [   19]   0x0
[ 1358.846829] GLBL_DSC_ERR_STS_RSVD_2                  [18,17]   0x0
[ 1358.846830] GLBL_DSC_ERR_STS_DAT_POISON              [   16]   0x0
[ 1358.846832] GLBL_DSC_ERR_STS_TIMEOUT                 [    9]   0x0
[ 1358.846833] GLBL_DSC_ERR_STS_FLR                     [    8]   0x0
[ 1358.846834] GLBL_DSC_ERR_STS_TAG                     [    6]   0x0
[ 1358.846859] GLBL_DSC_ERR_STS_ADDR                    [    5]   0x0
[ 1358.846860] GLBL_DSC_ERR_STS_PARAM                   [    4]   0x0
[ 1358.846860] GLBL_DSC_ERR_STS_BCNT                    [    3]   0x0
[ 1358.846861] GLBL_DSC_ERR_STS_UR_CA                   [    2]   0x0
[ 1358.846861] GLBL_DSC_ERR_STS_POISON                  [    1]   0x0
[ 1358.846865] GLBL_DSC_ERR_LOG0                        0x25c     0xc0000043 -1073741757
[ 1358.846866] GLBL_DSC_ERR_LOG0_VALID                  [   31]   0x1
[ 1358.846867] GLBL_DSC_ERR_LOG0_SEL                    [   30]   0x1
[ 1358.846867] GLBL_DSC_ERR_LOG0_RSVD_1                 [29,13]   0x0
[ 1358.846868] GLBL_DSC_ERR_LOG0_QID                    [12, 0]   0x43
[ 1358.846871] GLBL_DSC_ERR_LOG1                        0x260     0x280014   2621460
[ 1358.846872] GLBL_DSC_ERR_LOG1_RSVD_1                 [31,28]   0x0
[ 1358.846872] GLBL_DSC_ERR_LOG1_CIDX                   [27,12]   0x280
[ 1358.846873] GLBL_DSC_ERR_LOG1_RSVD_2                 [11, 9]   0x0
[ 1358.846873] GLBL_DSC_ERR_LOG1_SUB_TYPE               [ 8, 5]   0x0
[ 1358.846874] GLBL_DSC_ERR_LOG1_ERR_TYPE               [ 4, 0]   0x14
[ 1358.846878] GLBL_DSC_DBG_DAT0                        0x270     0x0        0
[ 1358.846878] GLBL_DSC_DAT0_RSVD_1                     [31,30]   0x0
[ 1358.846879] GLBL_DSC_DAT0_CTXT_ARB_DIR               [   29]   0x0
[ 1358.846879] GLBL_DSC_DAT0_CTXT_ARB_QID               [28,17]   0x0
[ 1358.846880] GLBL_DSC_DAT0_CTXT_ARB_REQ               [16,12]   0x0
[ 1358.846881] GLBL_DSC_DAT0_IRQ_FIFO_FL                [   11]   0x0
[ 1358.846881] GLBL_DSC_DAT0_TMSTALL                    [   10]   0x0
[ 1358.846881] GLBL_DSC_DAT0_RRQ_STALL                  [ 9, 8]   0x0
[ 1358.846882] GLBL_DSC_DAT0_RCP_FIFO_SPC_STALL         [ 7, 6]   0x0
[ 1358.846883] GLBL_DSC_DAT0_RRQ_FIFO_SPC_STALL         [ 5, 4]   0x0
[ 1358.846883] GLBL_DSC_DAT0_FAB_MRKR_RSP_STALL         [ 3, 2]   0x0
[ 1358.846884] GLBL_DSC_DAT0_DSC_OUT_STALL              [ 1, 0]   0x0
[ 1358.846887] GLBL_DSC_DBG_DAT1                        0x274     0x0        0
[ 1358.846888] GLBL_DSC_DAT1_RSVD_1                     [31,28]   0x0
[ 1358.846888] GLBL_DSC_DAT1_EVT_SPC_C2H                [27,22]   0x0
[ 1358.846889] GLBL_DSC_DAT1_EVT_SP_H2C                 [21,16]   0x0
[ 1358.846889] GLBL_DSC_DAT1_DSC_SPC_C2H                [15, 8]   0x0
[ 1358.846890] GLBL_DSC_DAT1_DSC_SPC_H2C                [ 7, 0]   0x0
[ 1358.846893] GLBL_DSC_ERR_LOG2                        0x27c     0x2800280  41943680
[ 1358.846894] GLBL_DSC_ERR_LOG2_OLD_PIDX               [31,16]   0x280
[ 1358.846894] GLBL_DSC_ERR_LOG2_NEW_PIDX               [15, 0]   0x280
[ 1358.846895] eqdma_hw_error_process detected DMA engine error
[ 1369.075104] onic:qdma_request_wait_for_cmpl: qdma3b000-MM-67: req 0x000000004a0a43c4, R,4190208,0/65536000,0x0, done 0, err 0, tm 10000.
[ 1369.087350] onic:qdma_descq_dump: qdma3b000-MM-67: 0x43/0x43, desc sz 1024/0, pidx 639, cidx 640
[ 1369.087674] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.

I push a fix for that seg fault. You can get a new version network_systolic_mm

@zhguanw-amd so the problem in network_systolic_mm has been resolved. I ran network_systolic_mm and read test back and forth. first network_systolic_mm, then read test, and again network_systolic_mm, and finally read test with no issues.
Right after, I ran a write test with no issue, it passed. And the send_recv test also passed. At this point. I think they are all good, but I ran the read test again, I had a result mismatch error, and the DMA engine broke again. There are no error messages from dmesg until I run the DMA test, it just the last read test failed with result mismatch. So the bug is either in the write test or send_recv test.
dmesg:
[  516.463259] IPv6: ADDRCONF(NETDEV_UP): enp59s0: link is not ready
[  516.463268] IPv6: ADDRCONF(NETDEV_CHANGE): enp59s0: link becomes ready
[  644.544157] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[  650.451113] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[  704.906296] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[  740.446916] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[  760.696267] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[  846.452736] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[  894.941990] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[  915.548327] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[ 1034.975745] onic:qdma_request_wait_for_cmpl: qdma3b000-MM-67: req 0x00000000eececbd4, W,65536000,0/65536000,0x0, done 0, err 0, tm 10000.
[ 1034.988096] onic:qdma_descq_dump: qdma3b000-MM-67: 0x43/0x43, desc sz 1024/1022, pidx 641, cidx 640
[ 1034.988429] onic 0000:3b:00.0: reconic-mm: Close onic_cdev.
[ 1037.802323] onic:error_intr_handler: Error IRQ fired on Funtion#0: index=7, vector=272
[ 1037.802330] eqdma_hw_error_process: Global Err Reg(0x248) = 0x4
[ 1037.802332] addr = 0x00000254 val = 0x00100000
[ 1037.802339] GLBL_DSC_ERR_STS                         0x254     0x100000   1048576
[ 1037.802341] GLBL_DSC_ERR_STS_RSVD_1                  [31,26]   0x0
[ 1037.802343] GLBL_DSC_ERR_STS_PORT_ID                 [   25]   0x0
[ 1037.802345] GLBL_DSC_ERR_STS_SBE                     [   24]   0x0
[ 1037.802346] GLBL_DSC_ERR_STS_DBE                     [   23]   0x0
[ 1037.802347] GLBL_DSC_ERR_STS_RQ_CANCEL               [   22]   0x0
[ 1037.802348] GLBL_DSC_ERR_STS_DSC                     [   21]   0x0
[ 1037.802350] GLBL_DSC_ERR_STS_DMA                     [   20]   0x1
[ 1037.802351] GLBL_DSC_ERR_STS_FLR_CANCEL              [   19]   0x0
[ 1037.802353] GLBL_DSC_ERR_STS_RSVD_2                  [18,17]   0x0
[ 1037.802354] GLBL_DSC_ERR_STS_DAT_POISON              [   16]   0x0
[ 1037.802355] GLBL_DSC_ERR_STS_TIMEOUT                 [    9]   0x0
[ 1037.802357] GLBL_DSC_ERR_STS_FLR                     [    8]   0x0
[ 1037.802358] GLBL_DSC_ERR_STS_TAG                     [    6]   0x0
[ 1037.802359] GLBL_DSC_ERR_STS_ADDR                    [    5]   0x0
[ 1037.802361] GLBL_DSC_ERR_STS_PARAM                   [    4]   0x0
[ 1037.802362] GLBL_DSC_ERR_STS_BCNT                    [    3]   0x0
[ 1037.802363] GLBL_DSC_ERR_STS_UR_CA                   [    2]   0x0
[ 1037.802364] GLBL_DSC_ERR_STS_POISON                  [    1]   0x0
[ 1037.802370] GLBL_DSC_ERR_LOG0                        0x25c     0xc0000043 -1073741757
[ 1037.802371] GLBL_DSC_ERR_LOG0_VALID                  [   31]   0x1
[ 1037.802373] GLBL_DSC_ERR_LOG0_SEL                    [   30]   0x1
[ 1037.802374] GLBL_DSC_ERR_LOG0_RSVD_1                 [29,13]   0x0
[ 1037.802376] GLBL_DSC_ERR_LOG0_QID                    [12, 0]   0x43
[ 1037.802381] GLBL_DSC_ERR_LOG1                        0x260     0x280014   2621460
[ 1037.802382] GLBL_DSC_ERR_LOG1_RSVD_1                 [31,28]   0x0
[ 1037.802384] GLBL_DSC_ERR_LOG1_CIDX                   [27,12]   0x280
[ 1037.802385] GLBL_DSC_ERR_LOG1_RSVD_2                 [11, 9]   0x0
[ 1037.802387] GLBL_DSC_ERR_LOG1_SUB_TYPE               [ 8, 5]   0x0
[ 1037.802389] GLBL_DSC_ERR_LOG1_ERR_TYPE               [ 4, 0]   0x14
[ 1037.802393] GLBL_DSC_DBG_DAT0                        0x270     0x0        0
[ 1037.802395] GLBL_DSC_DAT0_RSVD_1                     [31,30]   0x0
[ 1037.802396] GLBL_DSC_DAT0_CTXT_ARB_DIR               [   29]   0x0
[ 1037.802398] GLBL_DSC_DAT0_CTXT_ARB_QID               [28,17]   0x0
[ 1037.802399] GLBL_DSC_DAT0_CTXT_ARB_REQ               [16,12]   0x0
[ 1037.802400] GLBL_DSC_DAT0_IRQ_FIFO_FL                [   11]   0x0
[ 1037.802402] GLBL_DSC_DAT0_TMSTALL                    [   10]   0x0
[ 1037.802403] GLBL_DSC_DAT0_RRQ_STALL                  [ 9, 8]   0x0
[ 1037.802405] GLBL_DSC_DAT0_RCP_FIFO_SPC_STALL         [ 7, 6]   0x0
[ 1037.802406] GLBL_DSC_DAT0_RRQ_FIFO_SPC_STALL         [ 5, 4]   0x0
[ 1037.802408] GLBL_DSC_DAT0_FAB_MRKR_RSP_STALL         [ 3, 2]   0x0
[ 1037.802409] GLBL_DSC_DAT0_DSC_OUT_STALL              [ 1, 0]   0x0
[ 1037.802413] GLBL_DSC_DBG_DAT1                        0x274     0x0        0
[ 1037.802415] GLBL_DSC_DAT1_RSVD_1                     [31,28]   0x0
[ 1037.802416] GLBL_DSC_DAT1_EVT_SPC_C2H                [27,22]   0x0
[ 1037.802418] GLBL_DSC_DAT1_EVT_SP_H2C                 [21,16]   0x0
[ 1037.802420] GLBL_DSC_DAT1_DSC_SPC_C2H                [15, 8]   0x0
[ 1037.802421] GLBL_DSC_DAT1_DSC_SPC_H2C                [ 7, 0]   0x0
[ 1037.802426] GLBL_DSC_ERR_LOG2                        0x27c     0x2800280  41943680
[ 1037.802428] GLBL_DSC_ERR_LOG2_OLD_PIDX               [31,16]   0x280
[ 1037.802429] GLBL_DSC_ERR_LOG2_NEW_PIDX               [15, 0]   0x280
[ 1037.802431] eqdma_hw_error_process detected DMA engine error
[ 1037.815936] onic:error_intr_handler: Error IRQ fired on Funtion#0: index=7, vector=272
[ 1037.815942] eqdma_hw_error_process: Global Err Reg(0x248) = 0x4
[ 1037.815944] addr = 0x00000254 val = 0x00100000
[ 1037.815951] GLBL_DSC_ERR_STS                         0x254     0x100000   1048576
[ 1037.815953] GLBL_DSC_ERR_STS_RSVD_1                  [31,26]   0x0
[ 1037.815955] GLBL_DSC_ERR_STS_PORT_ID                 [   25]   0x0
[ 1037.815957] GLBL_DSC_ERR_STS_SBE                     [   24]   0x0
[ 1037.815958] GLBL_DSC_ERR_STS_DBE                     [   23]   0x0
[ 1037.815959] GLBL_DSC_ERR_STS_RQ_CANCEL               [   22]   0x0
[ 1037.815961] GLBL_DSC_ERR_STS_DSC                     [   21]   0x0
[ 1037.815962] GLBL_DSC_ERR_STS_DMA                     [   20]   0x1
[ 1037.815963] GLBL_DSC_ERR_STS_FLR_CANCEL              [   19]   0x0
[ 1037.815965] GLBL_DSC_ERR_STS_RSVD_2                  [18,17]   0x0
[ 1037.815966] GLBL_DSC_ERR_STS_DAT_POISON              [   16]   0x0
[ 1037.815968] GLBL_DSC_ERR_STS_TIMEOUT                 [    9]   0x0
[ 1037.815969] GLBL_DSC_ERR_STS_FLR                     [    8]   0x0
[ 1037.815970] GLBL_DSC_ERR_STS_TAG                     [    6]   0x0
[ 1037.815971] GLBL_DSC_ERR_STS_ADDR                    [    5]   0x0
[ 1037.815973] GLBL_DSC_ERR_STS_PARAM                   [    4]   0x0
[ 1037.815974] GLBL_DSC_ERR_STS_BCNT                    [    3]   0x0
[ 1037.815975] GLBL_DSC_ERR_STS_UR_CA                   [    2]   0x0
[ 1037.815976] GLBL_DSC_ERR_STS_POISON                  [    1]   0x0
[ 1037.815982] GLBL_DSC_ERR_LOG0                        0x25c     0xc0000043 -1073741757
[ 1037.815983] GLBL_DSC_ERR_LOG0_VALID                  [   31]   0x1
[ 1037.815985] GLBL_DSC_ERR_LOG0_SEL                    [   30]   0x1
[ 1037.815986] GLBL_DSC_ERR_LOG0_RSVD_1                 [29,13]   0x0
[ 1037.815988] GLBL_DSC_ERR_LOG0_QID                    [12, 0]   0x43
[ 1037.815993] GLBL_DSC_ERR_LOG1                        0x260     0x280014   2621460
[ 1037.815994] GLBL_DSC_ERR_LOG1_RSVD_1                 [31,28]   0x0
[ 1037.815996] GLBL_DSC_ERR_LOG1_CIDX                   [27,12]   0x280
[ 1037.815997] GLBL_DSC_ERR_LOG1_RSVD_2                 [11, 9]   0x0
[ 1037.815999] GLBL_DSC_ERR_LOG1_SUB_TYPE               [ 8, 5]   0x0
[ 1037.816000] GLBL_DSC_ERR_LOG1_ERR_TYPE               [ 4, 0]   0x14
[ 1037.816005] GLBL_DSC_DBG_DAT0                        0x270     0x0        0
[ 1037.816006] GLBL_DSC_DAT0_RSVD_1                     [31,30]   0x0
[ 1037.816008] GLBL_DSC_DAT0_CTXT_ARB_DIR               [   29]   0x0
[ 1037.816009] GLBL_DSC_DAT0_CTXT_ARB_QID               [28,17]   0x0
[ 1037.816022] GLBL_DSC_DAT0_CTXT_ARB_REQ               [16,12]   0x0
[ 1037.816022] GLBL_DSC_DAT0_IRQ_FIFO_FL                [   11]   0x0
[ 1037.816023] GLBL_DSC_DAT0_TMSTALL                    [   10]   0x0
[ 1037.816023] GLBL_DSC_DAT0_RRQ_STALL                  [ 9, 8]   0x0
[ 1037.816024] GLBL_DSC_DAT0_RCP_FIFO_SPC_STALL         [ 7, 6]   0x0
[ 1037.816025] GLBL_DSC_DAT0_RRQ_FIFO_SPC_STALL         [ 5, 4]   0x0
[ 1037.816025] GLBL_DSC_DAT0_FAB_MRKR_RSP_STALL         [ 3, 2]   0x0
[ 1037.816026] GLBL_DSC_DAT0_DSC_OUT_STALL              [ 1, 0]   0x0
[ 1037.816029] GLBL_DSC_DBG_DAT1                        0x274     0x0        0
[ 1037.816030] GLBL_DSC_DAT1_RSVD_1                     [31,28]   0x0
[ 1037.816030] GLBL_DSC_DAT1_EVT_SPC_C2H                [27,22]   0x0
[ 1037.816031] GLBL_DSC_DAT1_EVT_SP_H2C                 [21,16]   0x0
[ 1037.816032] GLBL_DSC_DAT1_DSC_SPC_C2H                [15, 8]   0x0
[ 1037.816032] GLBL_DSC_DAT1_DSC_SPC_H2C                [ 7, 0]   0x0
[ 1037.816036] GLBL_DSC_ERR_LOG2                        0x27c     0x2800280  41943680
[ 1037.816036] GLBL_DSC_ERR_LOG2_OLD_PIDX               [31,16]   0x280
[ 1037.816037] GLBL_DSC_ERR_LOG2_NEW_PIDX               [15, 0]   0x280
[ 1037.816037] eqdma_hw_error_process detected DMA engine error
[ 1037.829666] onic:error_intr_handler: Error IRQ fired on Funtion#0: index=7, vector=272
[ 1037.829673] eqdma_hw_error_process: Global Err Reg(0x248) = 0x4
[ 1037.829676] addr = 0x00000254 val = 0x00100000
[ 1037.829682] GLBL_DSC_ERR_STS                         0x254     0x100000   1048576
[ 1037.829684] GLBL_DSC_ERR_STS_RSVD_1                  [31,26]   0x0
[ 1037.829686] GLBL_DSC_ERR_STS_PORT_ID                 [   25]   0x0
[ 1037.829688] GLBL_DSC_ERR_STS_SBE                     [   24]   0x0
[ 1037.829689] GLBL_DSC_ERR_STS_DBE                     [   23]   0x0
[ 1037.829690] GLBL_DSC_ERR_STS_RQ_CANCEL               [   22]   0x0
[ 1037.829692] GLBL_DSC_ERR_STS_DSC                     [   21]   0x0
[ 1037.829693] GLBL_DSC_ERR_STS_DMA                     [   20]   0x1
[ 1037.829694] GLBL_DSC_ERR_STS_FLR_CANCEL              [   19]   0x0
[ 1037.829696] GLBL_DSC_ERR_STS_RSVD_2                  [18,17]   0x0
[ 1037.829697] GLBL_DSC_ERR_STS_DAT_POISON              [   16]   0x0
[ 1037.829699] GLBL_DSC_ERR_STS_TIMEOUT                 [    9]   0x0
[ 1037.829700] GLBL_DSC_ERR_STS_FLR                     [    8]   0x0
[ 1037.829701] GLBL_DSC_ERR_STS_TAG                     [    6]   0x0
[ 1037.829703] GLBL_DSC_ERR_STS_ADDR                    [    5]   0x0
[ 1037.829704] GLBL_DSC_ERR_STS_PARAM                   [    4]   0x0
[ 1037.829705] GLBL_DSC_ERR_STS_BCNT                    [    3]   0x0
[ 1037.829706] GLBL_DSC_ERR_STS_UR_CA                   [    2]   0x0
[ 1037.829708] GLBL_DSC_ERR_STS_POISON                  [    1]   0x0
[ 1037.829713] GLBL_DSC_ERR_LOG0                        0x25c     0xc0000043 -1073741757
[ 1037.829714] GLBL_DSC_ERR_LOG0_VALID                  [   31]   0x1
[ 1037.829716] GLBL_DSC_ERR_LOG0_SEL                    [   30]   0x1
[ 1037.829717] GLBL_DSC_ERR_LOG0_RSVD_1                 [29,13]   0x0
[ 1037.829719] GLBL_DSC_ERR_LOG0_QID                    [12, 0]   0x43
[ 1037.829724] GLBL_DSC_ERR_LOG1                        0x260     0x280014   2621460
[ 1037.829725] GLBL_DSC_ERR_LOG1_RSVD_1                 [31,28]   0x0
[ 1037.829727] GLBL_DSC_ERR_LOG1_CIDX                   [27,12]   0x280
[ 1037.829728] GLBL_DSC_ERR_LOG1_RSVD_2                 [11, 9]   0x0
[ 1037.829730] GLBL_DSC_ERR_LOG1_SUB_TYPE               [ 8, 5]   0x0
[ 1037.829731] GLBL_DSC_ERR_LOG1_ERR_TYPE               [ 4, 0]   0x14
[ 1037.829736] GLBL_DSC_DBG_DAT0                        0x270     0x0        0
[ 1037.829737] GLBL_DSC_DAT0_RSVD_1                     [31,30]   0x0
[ 1037.829739] GLBL_DSC_DAT0_CTXT_ARB_DIR               [   29]   0x0
[ 1037.829740] GLBL_DSC_DAT0_CTXT_ARB_QID               [28,17]   0x0
[ 1037.829742] GLBL_DSC_DAT0_CTXT_ARB_REQ               [16,12]   0x0
[ 1037.829743] GLBL_DSC_DAT0_IRQ_FIFO_FL                [   11]   0x0
[ 1037.829744] GLBL_DSC_DAT0_TMSTALL                    [   10]   0x0
[ 1037.829746] GLBL_DSC_DAT0_RRQ_STALL                  [ 9, 8]   0x0
[ 1037.829747] GLBL_DSC_DAT0_RCP_FIFO_SPC_STALL         [ 7, 6]   0x0
[ 1037.829749] GLBL_DSC_DAT0_RRQ_FIFO_SPC_STALL         [ 5, 4]   0x0
[ 1037.829750] GLBL_DSC_DAT0_FAB_MRKR_RSP_STALL         [ 3, 2]   0x0
[ 1037.829752] GLBL_DSC_DAT0_DSC_OUT_STALL              [ 1, 0]   0x0
[ 1037.829767] GLBL_DSC_DBG_DAT1                        0x274     0x0        0