Open vSwitch over DPDK on Ubuntu

There are two approaches for using DPDK acceleration in DPDK. One is the openvswitch fork from intel, called dpdk-ovs the other is done directly in openvswitch with a different approach from intel.

VirtualBox preparations

To run openvswitch with DPDK I used a virtual machine (VirtualBox) because the NIC I had on my laptop was not supported. I created three virtual NICs for my vm, one behind NAT to use it to ssh into the vm from the host, and two in host-only mode, to be use for testing.

If you happen to go down this road too, then it’s a good advice to do a few things before starting the test application.

First I would recommend configuring two virtual processors on the virtual machine, it makes possible to use most of the DPDK test apps, like testpmd. I would also recommend reserving about 4 GB of RAM to make sure DPDK works properly.

Then, to configure a network interface in host-only mode you need to create a host-only adapter (this applies to VirtualBox). Here is a link on how to do that:

Then you need to make sure that the interface is assigned an IP address (dhcp should work and you should be able to ping the guest from the host but you may need to configure dhcp on the virtual network if virtual box doesn’t do that).

Ping the two guest interfaces in order to populate the ARP table:

Find the arp entries an make them persistent:
arp -n | grep
sudo arp -s 08:00:27:20:88:10
arp -n | grep # now you should see flags CM to your arp entry           ether   08:00:27:20:88:10   CM                    vboxnet0

arp -n | grep
sudo arp -s 08:00:27:85:40:f6
arp -n | grep # now you should see flags CM to your arp entry           ether   08:00:27:85:40:f6   CM                    vboxnet1
## Installing DPDK There is good information from intel on DPDK, quickstarts, howtos etc. But for the sake of simplicity, here is selected useful information needed when starting with DPDK for the first time. ### Getting the code Simply  git clone git://  Note: At the moment I'm writing this, openvswitch doesn't compile with latest version of dpdk so I went with version 1.7.0rc0. Unfortunately there is no tag for it, so you will have to:
git checkout 536ba2d8a867ecf4331673d4c45475e552d57e27 -b version-1.7
### Hardware requirements DPDK works only on a select range of intel devices. It seems that most of the x86 NICs should work just fine, but I couldn't get it working on my HP ProBook 6560b laptop containing an Intel 82540EM Gigabit Ethernet Controller (rev 02) To check if your NIC is supported first you need to get the device id and then look it up in the list of supported devices. lspci - lists all the pci devices; look for your network card here and get the PCI vendor and device ID in the square brackets after the type string. For example:
# lspci -nn|grep Ethernet
00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I217-LM [8086:153a] (rev 05)
The list of known and supported devices can be found in the dpdk repository, under:

#define E1000_DEV_ID_PCH_LPT_I217_LM                0x153A
**If the device is supported then there will be a subsequent line declaring it as supported, for instance:**
If you have no supported NICs then you can always try a hypervisor (VirtualBox for instance) that can emulate one of the supported NICs (like the one above for example). ### Compiling DPDK Before even trying to compile openvswitch you need the dpdk code built in a single library. See INSTALL.DPDK for details. Basically you need to open $DPDK/config/defconfig_x86_64-default-linuxapp-gcc and change
Then compile:
make install T=x86_64-default-linuxapp-gcc
### Bind interfaces to DPDK DPDK uses a specialized kernel module to allow userspace applications to control the network card.
cd $DPDK
sudo modprobe uio
sudo insmod x86_64-default-linuxapp-gcc/kmod/igb_uio.ko
Check that igb_uio is supported
./tools/ --status

You should get something like this:

Network devices using IGB_UIO driver

Network devices using kernel driver
0000:00:03.0 '82540EM Gigabit Ethernet Controller' if=eth0 drv=e1000 unused=igb_uio *Active*
0000:00:08.0 '82540EM Gigabit Ethernet Controller' if=eth1 drv=e1000 unused=igb_uio *Active*
0000:00:09.0 '82540EM Gigabit Ethernet Controller' if=eth2 drv=e1000 unused=igb_uio *Active*

Other network devices

To bind an “active” NIC (one that is being used by Linux already) you will need to force it:

sudo ./tools/ --force --bind=igb_uio eth1
Do the same for the other interface
sudo ./tools/ --force --bind=igb_uio eth2
You can bind the interface back to it's original driver with the same tool, i.e.
sudo ./tools/ --force --bind=e1000 eth1
### Quick test of DPDK using l2fwd
cd examples/l2fwd
make RTE_SDK=$DPDK RTE_TARGET=x86_64-default-linuxapp-gcc
To run it you will also need to mount hugetlbfs.
sudo sysctl -w vm.nr_hugepages=320
sudo mkdir -p /mnt/huge
sudo mount -t hugetlbfs hugetlbfs /mnt/huge

cd examples/l2fwd/build/
sudo ./l2fwd -c 0x3 -n 4 -- -p 0x3 -T 1
You can also specify the number of queues per lcore but the default value 1 is ok. The -p 0x3 is important, it selects the DPDK 'ports' aka network devices that will be used. In this case ports 1 and 2 are selected. The -T parameter is used to updates the counters once a second. Now to test that it actually works we will use Wireshark and good old ping. Start Wireshark on the host and make it capture traffic on the second host only network interface (in my case this was vboxnet1). From the shell run:
ping -i 0.2
Wireshark should now show packets coming from and with destination The counters in the l2fwd applications should also be updated once a second. Then try to close the l2fwd application and observe that wireshark doesn't get packets anymore. ### Other DPDK examples There are other examples included, I tried test-pmd just out of curiosity. Detailed information about running them can be found on: ## Installing Openvswitch with DPDK After installing and running dpdk successfully you can start working on getting openvswitch up and running with DPDK. I've used commit [b596218aa8acafd64a4c7d1c3e761f00e50c0c53](;a=commit;h=b596218aa8acafd64a4c7d1c3e761f00e50c0c53) and it worked for me. ### Compiling OVS with DPDK Official information can be found here:;a=blob_plain;f=INSTALL.DPDK;hb=HEAD I followed the exact same steps, but I added the -ldl to LIBS since my x86_64 Ubuntu didn't link with the default options. cd openvswitch ./ ./configure --with-dpdk=$DPDK LIBS=-ldl make sudo make install ### Database initialization To manually initialize the database you need to do:
sudo mkdir -p /usr/local/etc/openvswitch
sudo ovsdb-tool create /usr/local/etc/openvswitch/conf.db vswitchd/vswitch.ovsschema
### Preparing DPDK For the DPDK part you need to configure hugetables, insert the kernel modules (uio and igb_uio) and bind the network interfaces to igb_uio. These need to be done pretty much once after the computer is started.
sudo sysctl -w vm.nr_hugepages=2000
sudo mkdir -p /mnt/huge
sudo mount -t hugetlbfs hugetlbfs /mnt/huge

sudo modprobe uio.ko
cd $DPDK
sudo insmod x86_64-default-linuxapp-gcc/kmod/igb_uio.ko
sudo ./tools/ --force --bind=igb_uio eth1
sudo ./tools/ --force --bind=igb_uio eth2
### Permanent startup configuration Here is how to setup hugetables and have the kernel modules loaded at startup. I haven't looked for a solution to bind the wanted interfaces to DPDK on startup, but there should be a way to do that. To have hugetables mounted by default at startup you need to add an entry to /etc/fstab
hugetlbfs       /mnt/huge       hugetlbfs       rw,mode=0777    0       0
To have the igb_uio.ko module loaded at boot time you need to make it known to modprobe. One good option is to make a symlink to it somewhere in the linux kernel structure:
sudo ln -s $DPDK/x86_64-default-linuxapp-gcc/kmod/igb_uio.ko /lib/modules/`uname -r`/kernel/drivers/uio/igb_uio.ko
sudo depmod -a
Then you need to add both uio and igb_uio to /etc/modules so that they are loaded at boot time. My Ubuntu 13.10 installation failed to start the desktop manager properly when one of the cards didn't get IP from dhcp. If that happens to you and you want the desktop manager, just log on to console 1 (Alt + Ctrl + F1) and run: sudo service lightdm restart
sudo service lightdm restart
However disabling the X server altogether should result in better performance overall. ### Running openvswitch with DPDK To manually run openvswitch you must do these each time:
sudo ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \
                                        --remote=db:Open_vSwitch,Open_vSwitch,manager_options \
                                        --private-key=db:Open_vSwitch,SSL,private_key \
                                        --certificate=db:Open_vSwitch,SSL,certificate \
                                        --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert \
                                        --pidfile --detach

sudo ovs-vsctl --no-wait init
sudo ovs-vswitchd --dpdk -c 0x3 -n 4 -- unix:/usr/local/var/run/openvswitch/db.sock --log-file=/usr/local/var/log/openvswitch/ovs-vswitchd.log --pidfile --detach
### Bridge configuration You need to add a netdev type bridge in order to make ovs running with DPDK. This means that all datapath switching will be done in userspace. You may get an error like "ovs-vsctl: Error detected while setting up 'ovsbr0'. See ovs-vswitchd log for details." in case you don't have the openvswitch module inserted. But the bridge will be created so you ignore the warning and continue.
sudo ovs-vsctl add-br ovsbr0
sudo ovs-vsctl set bridge ovsbr0 datapath_type=netdev

sudo ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0 type=dpdk
sudo ovs-vsctl add-port ovsbr0 dpdk1 -- set Interface dpdk1 type=dpdk
### Testing with ping and tcpdump A simple test to see that packets arrive at the bridge could be the following. First add an internal port to the bridge and bring it up:
sudo ovs-vsctl add-port br0 br0p1 -- set Interface br0p1 type=internal
sudo ifconfig br0p1 up
On the guest machine then start tcpdump
sudo tcpdump -i br0p1
From the host try to ping the guest. With the virtual machine configuration described you can try to ping something inside the subnet of one of the host-only interfaces. Linux will route the requests to the host-only adapter which in turn will be relayed by VirtualBox and then broadcasted in the virtual switch, including the port br0p1.
### Testing with mirroring port
sudo ovs-vsctl -- set Bridge br0 [email protected] -- [email protected] get Port eth1 -- [email protected] get Port eth2 -- [email protected] create Mirror name=mymirror [email protected] [email protected] [email protected]
sudo ovs-vsctl -- set Bridge ovsbr0 [email protected] -- [email protected] get Port dpdk0 -- [email protected] get Port dpdk1 -- [email protected] create Mirror name=mymirror [email protected] [email protected] [email protected]



comments powered by Disqus