In rhel9.2 I am trying to Configure the bridge, physical and virtual (vhost) ports, but running into

[sai19]

I have installed dpdk and open-Vswitch building from source code

[root@localhost ~]# ovs-vswitchd --version
ovs-vswitchd (Open vSwitch) 3.1.2
DPDK 22.11.2
[root@localhost ~]# ovs-vsctl get Open_vSwitch . dpdk_version
"DPDK 22.11.2"
[root@localhost ~]# ovs-vsctl get Open_vSwitch . dpdk_initialized
true
[root@localhost ~]#

---

Now when I try to Configure the bridge, physical and virtual (vhost) ports, running into this error

[root@localhost ~]# ovs-vsctl add-br ovs-br1 -- set bridge ovs-br1 datapath_type=netdev
[root@localhost ~]# ovs-vsctl add-port ovs-br1 dpdk-p1 -- set Interface dpdk-p1 type=dpdk options:dpdk-devargs=0000:02:07.0
ovs-vsctl: Error detected while setting up 'dpdk-p1': could not add network device dpdk-p1 to ofproto (Invalid argument). See ovs-vswitchd log for details.
ovs-vsctl: The default log directory is "/usr/local/var/log/openvswitch".
[root@localhost ~]#

---

[root@localhost ~]# ovs-vsctl show
84c6a319-b472-4d39-8dc5-ddbeb00b34e9
Bridge ovs-br1
datapath_type: netdev
Port ovs-br1
Interface ovs-br1
type: internal
Port dpdk-p1
Interface dpdk-p1
type: dpdk
options: {dpdk-devargs="0000:02:07.0"}
error: "could not add network device dpdk-p1 to ofproto (Invalid argument)"
ovs_version: "3.1.2"
[root@localhost ~]#

---

when I checked log, below is the error

2024-03-05T05:24:45.381Z|217002|dpdk|ERR|vmxnet3_dev_start(): Device activation: UNSUCCESSFUL
2024-03-05T05:24:45.381Z|217003|netdev_dpdk|ERR|Interface dpdk-p1 start error: Invalid argument
2024-03-05T05:24:45.381Z|217004|dpif_netdev|ERR|Failed to set interface dpdk-p1 new configuration

Could anyone please help me o resolve this. Thanks for the help!

[Chetan_Tiwary_]

@sai19 Please check the folloiwng links for reference - these could be helpful to you based on the "could not add  network device error :

https://mail.openvswitch.org/pipermail/ovs-discuss/2018-February/046228.html 

https://github.com/openvswitch/ovs-issues/issues/110 

https://bugs.launchpad.net/charm-ovn-chassis/+bug/1904736 

[sai19 · Accepted solution]

@Chetan_Tiwary_  Thanks for the reply. But I was able to resolve the issue, forgot to update. Below is how I got it working

Since I was trying to do this on rhel 9.2 VM in ESXi8, vmxnet3 tries to configure rss on port with one rxq and ends up in error as below,

if (VMXNET3_VERSION_GE_4(hw)) {
    /* Check for additional RSS  */
    ret = vmxnet3_v4_rss_configure(dev);
    if (ret != VMXNET3_SUCCESS) {
        PMD_INIT_LOG(ERR, "Failed to configure v4 RSS");
        return ret;
    }
}

to fix it, for ports with rxq=1 set flag as below in openVswitch code,

conf.rxmode.mq_mode = RTE_ETH_MQ_RX_NONE

[Chetan_Tiwary_]

Thanks for the info @sai19 ! Glad that your issue was resolved!

[sai19]

@Chetan_Tiwary_ , facing one more issue wrt this once I was able to create vhost ports and after editing the xml config of a VM, getting into this error as below - could you please help me with this

error: internal error: process exited while connecting to monitor: 2024-03-12T18:14:59.432803Z qemu-kvm: -chardev socket,id=charnet1,path=/usr/local/var/run/openvswitch/vhost-user1: Failed to connect to '/usr/local/var/run/openvswitch/vhost-user1': Permission denied

[Chetan_Tiwary_]

Seems a permission issue - have you checked the ownership and permission on the socket ? is selinux restricting it ?

refer this article : https://www.redhat.com/en/blog/ovs-dpdk-migrating-vhostuser-socket-mode-red-hat-openstack 

[sai19]

you are right @Chetan_Tiwary_ , selinux is restricting the access - as I see from logs below

type=AVC msg=audit(1710312140.989:9995): avc: denied { bpf } for pid=40210 comm="qemu-kvm" capability=39 scontext=system_u:system_r:svirt_t:s0:c99,c322 tcontext=system_u:system_r:svirt_t:s0:c99,c322 tclass=capability2 permissive=0
type=AVC msg=audit(1710312140.989:9995): avc: denied { sys_admin } for pid=40210 comm="qemu-kvm" capability=21 scontext=system_u:system_r:svirt_t:s0:c99,c322 tcontext=system_u:system_r:svirt_t:s0:c99,c322 tclass=capability permissive=0
type=AVC msg=audit(1710312140.992:9996): avc: denied { write } for pid=40210 comm="qemu-kvm" name="vhost-user1" dev="dm-0" ino=104034159 scontext=system_u:system_r:svirt_t:s0:c99,c322 tcontext=unconfined_u:object_r:usr_t:s0 tclass=sock_file permissive=0

--------------------------------------------

for time being I did setenforce 0 and I was able to start my VM

but could you please let me know, how to over come this, do I need to update some policies, I am quite new to this stuff. Thanks!

[Mostafa1]

Great that you fixed it but I am adding the below as a general methodology for troubleshooting such issues.

The error message "Device activation: UNSUCCESSFUL" and "Interface dpdk-p1 start error: Invalid argument" indicate that there was an issue activating the DPDK device dpdk-p1.

Here are a few steps you can take to troubleshoot and resolve the issue:

1. Check DPDK configuration: Ensure that DPDK is properly configured and initialized. Verify that the correct DPDK version is installed and that DPDK is initialized successfully. You can use DPDK tools like dpdk-devbind.py to verify the status of DPDK devices.

2. Verify device availability: Double-check that the DPDK device 0000:02:07.0 is available and correctly specified. You can use tools like lspci to list all PCI devices and confirm that the specified device exists.

3. Check kernel modules: Ensure that the necessary kernel modules for DPDK and the specific device (vmxnet3 in this case) are loaded and properly configured. Use tools like lsmod to list loaded kernel modules and verify that the required modules are present.

4. Check permissions: Ensure that the user running ovs-vsctl has sufficient permissions to access and configure DPDK devices. You may need to run the command with elevated privileges (e.g., using sudo) or adjust permissions accordingly.

5. Verify DPDK configuration in OVS: Double-check the DPDK configuration in Open vSwitch (ovs-vswitchd.conf). Ensure that the DPDK parameters are correctly set and match the configuration of the DPDK environment.

6. Review OVS logs: Continue reviewing the Open vSwitch logs (/usr/local/var/log/openvswitch) for more detailed error messages or clues about the issue. Look for any additional error messages or warnings that may provide insights into the root cause of the problem.

By following these steps and carefully reviewing the configuration and logs, you should be able to diagnose and resolve the issue with activating the DPDK device in Open vSwitch.

[ericbos]

Thank you @Mostafa1 for sharing the troubleshooting steps

Best regards, Eric

[napapornt]

The error you’re encountering while configuring the bridge with DPDK and Open vSwitch seems to be related to the network device dpdk-p1. Let’s troubleshoot this issue step by step:

Check DPDK Initialization:

Ensure that DPDK is correctly initialized. You’ve already verified that DPDK is initialized with version 22.11.2.
Confirm that the DPDK environment variables (RTE_SDK and RTE_TARGET) are set correctly.
Verify Device Binding:

Confirm that the network device dpdk-p1 is bound to DPDK using the dpdk-devbind.py script. Run:
dpdk-devbind.py -s
Ensure that dpdk-p1 is bound to the correct DPDK-compatible driver.
Check Kernel Modules:

Verify that the necessary kernel modules (such as vfio-pci) are loaded.
If not, load them using:
modprobe vfio-pci
Review OVS Configuration:

Double-check the OVS configuration for ovs-br1 and dpdk-p1. The error message indicates an issue with adding dpdk-p1 to the bridge.
Make sure the bridge is created correctly:
ovs-vsctl add-br ovs-br1 -- set bridge ovs-br1 datapath_type=netdev
Then add the port:
ovs-vsctl add-port ovs-br1 dpdk-p1 -- set Interface dpdk-p1 type=dpdk options:dpdk-devargs=0000:02:07.0
Check OVS Logs

Inspect the OVS logs for more detailed information. The default log directory is /usr/local/var/log/openvswitch.
Look for any additional error messages related to dpdk-p1.
Verify Device Compatibility:

Ensure that the network device dpdk-p1 is compatible with DPDK. Some devices may have limitations or require specific configurations.
Check if there are any known issues related to the specific device or driver version.
Update DPDK and OVS:

Consider updating DPDK and OVS to the latest versions. Sometimes newer releases address compatibility issues.
Refer to the official documentation for any specific requirements or recommendations.
Remember to consult the official documentation for DPDK, OVS, and your specific network device for detailed troubleshooting steps. If you encounter any specific error messages in the logs, share them for further analysis.