溫馨提示×
您好，登錄后才能下訂單哦！
點擊 登錄注冊 即表示同意《億速云用戶服務條款》
您好，登錄后才能下訂單哦！
如何用使用DPDK優化VirtIO和OVS網絡，針對這個問題，這篇文章詳細介紹了相對應的分析和解答，希望可以幫助更多想解決這個問題的小伙伴找到更簡單易行的方法。
一共有2個節點，配置基本相同。節點A用于運行虛擬機，節點B用于測試性能。
發行版版本：
$ cat /etc/redhat-release CentOS Linux release 7.4.1708 (Core)
內核版本：
$ uname -a Linux osdev-gpu 3.10.0-693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
CPU信息：
$ cat /proc/cpuinfo | tail -n26 processor : 71 vendor_id : GenuineIntel cpu family : 6 model : 85 model name : Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz stepping : 4 microcode : 0x200002c cpu MHz : 1499.941 cache size : 25344 KB physical id : 1 siblings : 36 core id : 27 cpu cores : 18 apicid : 119 initial apicid : 119 fpu : yes fpu_exception : yes cpuid level : 22 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 invpcid_single intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req bogomips : 4604.72 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management:
內存信息：
$ free -h total used free shared buff/cache available Mem: 754G 5.7G 748G 13M 425M 746G Swap: 0B 0B 0B
網卡信息：
# 節點A：
$ lspci | grep Ethernet
1a:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
1a:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
1a:00.2 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
1a:00.3 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
5f:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
5f:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
$ lspci | awk '$0~/Ethernet/{printf($1 " ");gsub(":","\\:");cmd="ls /sys/bus/pci/devices/0000\\:" $1 "/driver/module/drivers/"; system(cmd)}'
1a:00.0 pci:i40e
1a:00.1 pci:i40e
1a:00.2 pci:i40e
1a:00.3 pci:i40e
5f:00.0 pci:i40e
5f:00.1 pci:i40e
# 節點B：
$ lspci | grep Ethernet
1a:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
1a:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
1a:00.2 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
1a:00.3 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
5e:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
5e:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
60:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
60:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
$ lspci | awk '$0~/Ethernet/{printf($1 " ");gsub(":","\\:");cmd="ls /sys/bus/pci/devices/0000\\:" $1 "/driver/module/drivers/"; system(cmd)}'
1a:00.0 pci:i40e
1a:00.1 pci:i40e
1a:00.2 pci:i40e
1a:00.3 pci:i40e
5e:00.0 pci:i40e
5e:00.1 pci:i40e
60:00.0 pci:i40e
60:00.1 pci:i40e如果要使用VFIO驅動，則必須開啟IOMMU支持。進入主板的BIOS，打開處理器的Intel VT-d支持。如果需要使用單根虛擬化，則需要同時開啟PCI的SR-IOV支持。
修改Linux內核啟動參數，增加intel_iommu=on iommu=pt選項：
$ vi /etc/default/grub GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt crashkernel=auto biosdevname=0 rhgb quiet"
更新啟動Grub啟動配置，并重啟主機：
$ grub2-mkconfig -o /boot/grub2/grub.cfg $ reboot
主機重啟后，查看啟動參數是否正確添加：
$ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-693.17.1.el7.x86_64 root=UUID=1645c2f2-2308-436e-86e6-91ebdc76477e ro intel_iommu=on iommu=pt crashkernel=auto biosdevname=0 rhgb quiet
查看內核中的IOMMU是否正確初始化，如果成功則會有如下信息：
$ dmesg | grep -e DMAR -e IOMMU [ 0.000000] ACPI: DMAR 000000006ca320d8 002A0 (v01 ALASKA A M I 00000001 INTL 20091013) [ 0.000000] DMAR: IOMMU enabled [ 0.348112] DMAR: Host address width 46 [ 0.348114] DMAR: DRHD base: 0x000000d37fc000 flags: 0x0 [ 0.348123] DMAR: dmar0: reg_base_addr d37fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 0.348124] DMAR: DRHD base: 0x000000e0ffc000 flags: 0x0 [ 0.348128] DMAR: dmar1: reg_base_addr e0ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 0.348129] DMAR: DRHD base: 0x000000ee7fc000 flags: 0x0 [ 0.348133] DMAR: dmar2: reg_base_addr ee7fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 0.348134] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 [ 0.348138] DMAR: dmar3: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 0.348139] DMAR: DRHD base: 0x000000aaffc000 flags: 0x0 [ 0.348156] DMAR: dmar4: reg_base_addr aaffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [ 0.348157] DMAR: DRHD base: 0x000000b87fc000 flags: 0x0 [ 0.348162] DMAR: dmar5: reg_base_addr b87fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df ...
查看IOMMU分組是否正常，如果目錄非空則表明分組成功：
$ ls /sys/kernel/iommu_groups/ 0 11 14 17 2 22 25 28 30 33 36 39 41 44 47 5 52 55 58 60 63 66 69 71 74 77 8 82 85 1 12 15 18 20 23 26 29 31 34 37 4 42 45 48 50 53 56 59 61 64 67 7 72 75 78 80 83 86 10 13 16 19 21 24 27 3 32 35 38 40 43 46 49 51 54 57 6 62 65 68 70 73 76 79 81 84 9
新增Qmu-KVM源：
$ yum install -y epel-release $ yum install -y centos-release-qemu-ev
安裝并查看Qemu-KVM版本：
$ yum install -y qemu-kvm-ev $ /usr/libexec/qemu-kvm --version QEMU emulator version 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.14.1) Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers $ qemu-img --version qemu-img version 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.14.1) Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
安裝依賴軟件包：
$ yum install -y libvirt libguestfs-tools libguestfs-xfs genisoimage $ systemctl enable libvirtd && systemctl start libvirtd && systemctl status libvirtd
下載CentOS的Cloud鏡像：
$ export OVS_ROOT=/opt/ovs $ mkdir -pv $OVS_ROOT/images $ wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 --directory-prefix $OVS_ROOT/images
查看鏡像信息：
$ qemu-img info $OVS_ROOT/images/CentOS-7-x86_64-GenericCloud.qcow2 image: /opt/ovs/images/CentOS-7-x86_64-GenericCloud.qcow2 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 832M cluster_size: 65536 Format specific information: compat: 0.10 refcount bits: 16 $ virt-filesystems --long --parts --blkdevs -h -a $OVS_ROOT/images/CentOS-7-x86_64-GenericCloud.qcow2 Name Type MBR Size Parent /dev/sda1 partition 83 8.0G /dev/sda /dev/sda device - 8.0G - $ virt-df -h $OVS_ROOT/images/CentOS-7-x86_64-GenericCloud.qcow2 文件系統 大小 已用空間 可用空間 使用百分比% CentOS-7-x86_64-GenericCloud.qcow2:/dev/sda1 8.0G 795M 7.2G 10%
創建一個新鏡像：
$ qemu-img create -f qcow2 $OVS_ROOT/images/CentOS-7-x86_64.qcow2 20G Formatting '/opt/ovs/images/CentOS-7-x86_64.qcow2', fmt=qcow2 size=21474836480 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
擴展鏡像分區：
$ virt-resize $OVS_ROOT/images/CentOS-7-x86_64-GenericCloud.qcow2 $OVS_ROOT/images/CentOS-7-x86_64.qcow2 --expand /dev/sda1 [ 0.0] Examining /opt/ovs/images/CentOS-7-x86_64-GenericCloud.qcow2 25% ????????????????????????????????????══════════════════════════════════════════? --:-- 100% ??????????????????????????????????????????????????????????????????????????????? --:-- ********** Summary of changes: /dev/sda1: This partition will be resized from 8.0G to 20.0G. The filesystem xfs on /dev/sda1 will be expanded using the 'xfs_growfs' method. ********** [ 17.3] Setting up initial partition table on /opt/ovs/images/CentOS-7-x86_64.qcow2 [ 17.5] Copying /dev/sda1 100% ??????????????????????????????????????????????????????????????????????????????? 00:00 [ 26.8] Expanding /dev/sda1 using the 'xfs_growfs' method Resize operation completed with no errors. Before deleting the old disk, carefully check that the resized disk boots and works correctly.
查看新鏡像信息：
$ qemu-img info $OVS_ROOT/images/CentOS-7-x86_64.qcow2 image: /opt/ovs/images/CentOS-7-x86_64.qcow2 file format: qcow2 virtual size: 20G (21474836480 bytes) disk size: 834M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false $ virt-filesystems --long --parts --blkdevs -h -a $OVS_ROOT/images/CentOS-7-x86_64.qcow2 Name Type MBR Size Parent /dev/sda1 partition 83 20G /dev/sda /dev/sda device - 20G - $ virt-df -h $OVS_ROOT/images/CentOS-7-x86_64.qcow2 文件系統 大小 已用空間 可用空間 使用百分比% CentOS-7-x86_64.qcow2:/dev/sda1 20G 795M 19G 4%
創建元數據配置：
$ vi $OVS_ROOT/images/meta-data instance-id: centos7-ovs; local-hostname: centos7-ovs;
創建用戶數據配置（把ssh_authorized_keys的Key換成自己的）：
$ vi $OVS_ROOT/images/user-data
#cloud-config
user: root
password: 123456
chpasswd: { expire: False }
ssh_pwauth: True
ssh_authorized_keys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD6ijVswX40X2ercmKDZD8mVRD6GFkhdeBM/OjvGd8mSnLDdOjhZw83jhMm/rptNEDlpBW+IjxOZsDO3wm+iVcGn5LblJ3qXtdGEIHlsttFAkLsF8B3jnLBeSTRee8JXZBic5KdfYffq9JC3WrgGJl+OQz6mNW7rqquBFI98QCVlsZEvsJzw5LEm1/ej3Ka2pSkTEei+sB4PBPolnH9cUahq5T8Wwgtlw6JutNob1e5OgFWvPThTRWAtCqLaFWenedagKEA4jPseuF7dq/Eb7nEqL2jYNsWKyR1JpuUdejxAfw434guitORyvLCbj022Sgn5bwEYPIw3EVykcc6XxoZ root@osdev-gpu
final_message: "SYSTEM READY TO LOG IN"生成元數據鏡像：
$ genisoimage -output $OVS_ROOT/images/centos7-init.iso -volid cidata -joliet -rock $OVS_ROOT/images/user-data $OVS_ROOT/images/meta-data
安裝依賴軟件包：
$ yum -y install gcc autoconf automake libtool kernel kernel-devel
獲取源碼：
$ export OVS_ROOT=/opt/ovs && cd $OVS_ROOT $ git clone https://github.com/openvswitch/ovs.git $ cd ovs && git checkout -b v2.9.0/origin v2.9.0 && git checkout -b v2.9.0/devel
編譯和安裝：
$ ./boot.sh $ mkdir -pv $OVS_ROOT/build-nodpdk $OVS_ROOT/target-nodpdk && cd $OVS_ROOT/build-nodpdk $ ../ovs/configure --enable-shared --with-linux=/lib/modules/$(uname -r)/build --prefix=$OVS_ROOT/target-nodpdk CFLAGS="-g -Ofast" $ make -j16 'CFLAGS=-g -Ofast -march=native' && make install $ mkdir -pv $OVS_ROOT/target-nodpdk/modules && cp -vf $OVS_ROOT/build-nodpdk/datapath/linux/*ko $OVS_ROOT/target-nodpdk/modules/
初始化環境：
$ export OVS_ROOT=/opt/ovs && export OVS_DIR=$OVS_ROOT/target-nodpdk && export PATH=$PATH:$OVS_DIR/share/openvswitch/scripts && cd $OVS_DIR
使用腳本自動啟動：
$ ovs-ctl start
使用腳本自動停止：
$ ovs-ctl stop
使用命令手動啟動：
$ mkdir -pv $OVS_DIR/var/run/openvswitch $OVS_DIR/etc/openvswitch
# 創建ovsdb-server數據庫：
$ $OVS_DIR/bin/ovsdb-tool create $OVS_DIR/etc/openvswitch/conf.db $OVS_DIR/share/openvswitch/vswitch.ovsschema
# 啟動ovsdb-server數據庫服務：
$ $OVS_DIR/sbin/ovsdb-server --remote=punix:$OVS_DIR/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach
# 初始化數據庫（只在第一次運行時需要執行）：
$ $OVS_DIR/bin/ovs-vsctl --no-wait init
# 查看依賴的內核模塊：
$ cat /usr/lib/modules/`uname -r`/modules.dep | awk '$1~"^extra/openvswitch"{for(i=2;i<=NF;i++) {print $i}}' | xargs echo
kernel/net/ipv6/netfilter/nf_nat_ipv6.ko.xz kernel/net/ipv4/netfilter/nf_nat_ipv4.ko.xz kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko.xz kernel/net/netfilter/nf_nat.ko.xz kernel/net/netfilter/nf_conntrack.ko.xz kernel/net/ipv4/udp_tunnel.ko.xz kernel/lib/libcrc32c.ko.xz
# 加載openvswitch.ko依賴的內核模塊：
$ cat /usr/lib/modules/`uname -r`/modules.dep | awk '$1~"^extra/openvswitch"{for(i=2;i<=NF;i++) {mod=gensub(/\.ko\.xz/,"",1,gensub(/.*\//,"",1,$i));cmd="modprobe " mod;system(cmd)}}'
# 加載openvswitch.ko內核模塊：
$ insmod $OVS_DIR/modules/openvswitch.ko
# 啟動ovs-vswitchd守護進程：
$ $OVS_DIR/sbin/ovs-vswitchd --pidfile --detach --log-file使用命令手動停止：
$ pkill -9 ovs $ rm -rfv $OVS_DIR/var/run/openvswitch $OVS_DIR/etc/openvswitch/ $OVS_DIR/etc/openvswitch/conf.db
編輯環境變量腳本：
$ vi $OVS_ROOT/ovs-env-nodpdk.sh export OVS_ROOT=/opt/ovs export OVS_DIR=$OVS_ROOT/target-nodpdk export PATH=$OVS_DIR/bin/:$OVS_DIR/share/openvswitch/scripts:$PATH
編輯OVS啟動腳本：
$ vi $OVS_ROOT/start-ovs.sh
#!/bin/bash
$OVS_DIR/sbin/ovsdb-server --remote=punix:$OVS_DIR/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach
cat /usr/lib/modules/`uname -r`/modules.dep | awk '$1~"^extra/openvswitch"{for(i=2;i<=NF;i++) {mod=gensub(/\.ko\.xz/,"",1,gensub(/.*\//,"",1,$i));cmd="modprobe " mod;system(cmd)}}'
insmod $OVS_DIR/modules/openvswitch.ko
$OVS_DIR/sbin/ovs-vswitchd --pidfile --detach --log-file
$ chmod a+x $OVS_ROOT/start-ovs.sh編輯OVS停止腳本：
$ vi $OVS_ROOT/stop-ovs.sh #!/bin/bash pkill -9 ovs # rm -rfv $OVS_DIR/var/run/openvswitch $OVS_DIR/etc/openvswitch/ $OVS_DIR/etc/openvswitch/conf.db $ chmod a+x $OVS_ROOT/stop-ovs.sh
設置環境變量：
$ export OVS_ROOT=/opt/ovs && export OVS_DIR=$OVS_ROOT/target-nodpdk && export PATH=$OVS_DIR/bin/:$OVS_DIR/share/openvswitch/scripts:$PATH # Or $ . /opt/ovs/ovs-env-nodpdk.sh
開啟數據包轉發：
$ echo 1 > /proc/sys/net/ipv4/ip_forward
查看當前網卡配置：
# 節點A： $ ip a | grep -A2 "enp.*: " 2: enp26s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop portid 6c92bf74beac state DOWN qlen 1000 link/ether 6c:92:bf:74:be:ac brd ff:ff:ff:ff:ff:ff 3: enp26s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop portid 6c92bf74bead state DOWN qlen 1000 link/ether 6c:92:bf:74:be:ad brd ff:ff:ff:ff:ff:ff 4: enp26s0f2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 6c92bf74beae state UP qlen 1000 link/ether 6c:92:bf:74:be:ae brd ff:ff:ff:ff:ff:ff inet 172.29.101.166/24 brd 172.29.101.255 scope global enp26s0f2 -- 5: enp26s0f3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop portid 6c92bf74beaf state DOWN qlen 1000 link/ether 6c:92:bf:74:be:af brd ff:ff:ff:ff:ff:ff 6: enp95s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 6c92bf5cdc5e state UP qlen 1000 link/ether 6c:92:bf:5c:dc:5e brd ff:ff:ff:ff:ff:ff inet 172.29.113.28/24 brd 172.29.113.255 scope global enp95s0f0 -- 7: enp95s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop portid 6c92bf5cdc5f state DOWN qlen 1000 link/ether 6c:92:bf:5c:dc:5f brd ff:ff:ff:ff:ff:ff 8: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP # 節點B： $ ip a | grep -A2 "enp.*: " 2: enp26s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq portid 6c92bf74bfd0 state DOWN qlen 1000 link/ether 6c:92:bf:74:bf:d0 brd ff:ff:ff:ff:ff:ff 3: enp26s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq portid 6c92bf74bfd1 state DOWN qlen 1000 link/ether 6c:92:bf:74:bf:d1 brd ff:ff:ff:ff:ff:ff 4: enp26s0f2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 6c92bf74bfd2 state UP qlen 1000 link/ether 6c:92:bf:74:bf:d2 brd ff:ff:ff:ff:ff:ff 5: enp26s0f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 6c92bf74bfd3 state UP qlen 1000 link/ether 6c:92:bf:74:bf:d3 brd ff:ff:ff:ff:ff:ff inet 172.29.101.171/24 brd 172.29.101.255 scope global enp26s0f3 -- 6: enp94s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 6c92bf5bc5cd state UP qlen 1000 link/ether 6c:92:bf:5b:c5:cd brd ff:ff:ff:ff:ff:ff inet 172.29.122.43/24 brd 172.29.122.255 scope global enp94s0f0 -- 7: enp94s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 6c92bf5bc5ce state UP qlen 1000 link/ether 6c:92:bf:5b:c5:ce brd ff:ff:ff:ff:ff:ff 8: enp96s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 6c92bf5bc59d state UP qlen 1000 link/ether 6c:92:bf:5b:c5:9d brd ff:ff:ff:ff:ff:ff 9: enp96s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 6c92bf5bc59e state UP qlen 1000 link/ether 6c:92:bf:5b:c5:9e brd ff:ff:ff:ff:ff:ff 10: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
在節點A上創建的NAT網絡網段為192.168.2.0/24，網關為192.168.2.99。
新增OVS網橋br-ovs：
$ ovs-vsctl add-br br-ovs $ ip link set dev br-ovs up $ ip addr add 192.168.2.99/24 dev br-ovs $ ip addr show br-ovs 21: br-ovs: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000 link/ether a2:e8:a6:bb:55:46 brd ff:ff:ff:ff:ff:ff inet 192.168.2.99/24 scope global br-ovs valid_lft forever preferred_lft forever inet6 fe80::a0e8:a6ff:febb:5546/64 scope link valid_lft forever preferred_lft forever
增加NAT地址轉換：
$ iptables -t nat -A POSTROUTING -s 192.168.2.0/24 ! -d 192.168.2.0/24 -j MASQUERADE
配置DHCP服務：
$ yum install -y dnsmasq $ dnsmasq --strict-order --except-interface=lo --interface=br-ovs --listen-address=192.168.2.99 --bind-interfaces --dhcp-range=192.168.2.128,192.168.2.192 --conf-file="" --pid-file=/var/run/br-ovs-dhcp.pid --dhcp-leasefile=/var/run/br-ovs-dhcp.leases --dhcp-no-override
創建NAT網絡啟動腳本：
$ vi $OVS_ROOT/start-nat.sh #!/bin/bash #ovs-vsctl del-br br-ovs || return 0 #ovs-vsctl add-br br-ovs ip link set dev br-ovs up ip addr add 192.168.2.99/24 dev br-ovs iptables -t nat -A POSTROUTING -s 192.168.2.0/24 ! -d 192.168.2.0/24 -j MASQUERADE dnsmasq --strict-order --except-interface=lo --interface=br-ovs --listen-address=192.168.2.99 --bind-interfaces --dhcp-range=192.168.2.128,192.168.2.192 --conf-file="" --pid-file=/var/run/br-ovs-dhcp.pid --dhcp-leasefile=/var/run/br-ovs-dhcp.leases --dhcp-no-override $ chmod a+x $OVS_ROOT/start-nat.sh
增加測試網絡NameSpace：
$ ip netns add ovs-ns
創建測試VEth：
$ ip link add ovs-veth-br type veth peer name ovs-veth-in
把VEth一端加入br-ovs網橋：
$ ovs-vsctl add-port br-ovs ovs-veth-br $ ip link set dev ovs-veth-br up
把VEth另一端加入NameSpace，并配置IP：
$ ip link set ovs-veth-in netns ovs-ns $ ip netns exec ovs-ns ip link set dev ovs-veth-in up $ ip netns exec ovs-ns ip addr add 192.168.2.95/24 dev ovs-veth-in $ ip netns exec ovs-ns ifconfig ovs-veth-in: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.2.95 netmask 255.255.255.0 broadcast 0.0.0.0 inet6 fe80::25:e8ff:fe78:9b31 prefixlen 64 scopeid 0x20<link> ether 02:25:e8:78:9b:31 txqueuelen 1000 (Ethernet) RX packets 8 bytes 648 (648.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 648 (648.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
為NameSpace中網絡的路由：
$ ip netns exec ovs-ns route add default gw 192.168.2.99 $ ip netns exec ovs-ns route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.2.99 0.0.0.0 UG 0 0 0 ovs-veth-in 192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 ovs-veth-in
測試NAT網絡是否正常：
$ ip netns exec ovs-ns ping -c 4 8.8.8.8 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=43 time=49.5 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=43 time=49.0 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=43 time=49.0 ms 64 bytes from 8.8.8.8: icmp_seq=4 ttl=43 time=49.2 ms --- 8.8.8.8 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3006ms rtt min/avg/max/mdev = 49.022/49.197/49.504/0.193 ms $ ip netns exec ovs-ns ping -c 4 www.163.com PING 163.xdwscache.ourglb0.com (58.221.28.167) 56(84) bytes of data. 64 bytes from 58.221.28.167 (58.221.28.167): icmp_seq=1 ttl=47 time=15.1 ms 64 bytes from 58.221.28.167 (58.221.28.167): icmp_seq=2 ttl=47 time=14.5 ms 64 bytes from 58.221.28.167 (58.221.28.167): icmp_seq=3 ttl=47 time=14.6 ms 64 bytes from 58.221.28.167 (58.221.28.167): icmp_seq=4 ttl=47 time=14.8 ms --- 163.xdwscache.ourglb0.com ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3009ms rtt min/avg/max/mdev = 14.561/14.810/15.183/0.252 ms
測試DHCP服務是否正常：
$ ip netns exec ovs-ns dhclient $ tcpdump -n -i br-ovs tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on br-ovs, link-type EN10MB (Ethernet), capture size 262144 bytes 19:31:30.619593 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 02:25:e8:78:9b:31, length 300 19:31:30.619817 IP 192.168.2.99.bootps > 192.168.2.158.bootpc: BOOTP/DHCP, Reply, length 300 19:31:30.680620 ARP, Request who-has 192.168.2.158 (Broadcast) tell 0.0.0.0, length 28 19:31:31.681806 ARP, Request who-has 192.168.2.158 (Broadcast) tell 0.0.0.0, length 28 19:31:35.628085 ARP, Request who-has 192.168.2.158 tell 192.168.2.99, length 28 19:31:35.628350 ARP, Reply 192.168.2.158 is-at 02:25:e8:78:9b:31, length 28 $ ip netns exec ovs-ns ip addr 1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 24: ovs-veth-in@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000 link/ether 02:25:e8:78:9b:31 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.2.95/24 scope global ovs-veth-in valid_lft forever preferred_lft forever inet 192.168.2.158/24 brd 192.168.2.255 scope global secondary dynamic ovs-veth-in valid_lft 3377sec preferred_lft 3377sec inet6 fe80::25:e8ff:fe78:9b31/64 scope link valid_lft forever preferred_lft forever
兩個節點添加的物理網卡在一個交換機的相同VLAN上。
在節點A的br-ovs網橋上添加物理網口enp26s0f2：
$ ovs-vsctl add-port br-ovs enp26s0f2 $ ovs-vsctl show 249d4a72-bf90-4250-aafd-23e12b2a0868 Bridge br-ovs Port "tap-test1" Interface "tap-test1" Port br-ovs Interface br-ovs type: internal Port ovs-veth-br Interface ovs-veth-br Port "enp26s0f2" Interface "enp26s0f2" Port "tap-test2" Interface "tap-test2"
在節點B上創建網橋br-ovs，添加物理網卡enp26s0f2，并設置網橋IP為192.168.2.98/24：
$ ovs-vsctl add-br br-ovs $ ovs-vsctl set bridge br-ovs stp_enable=false $ ovs-vsctl add-port br-ovs enp26s0f2 $ ovs-vsctl show c50241cc-7046-4673-9f4c-c9a33ea3bb28 Bridge br-ovs Port "enp26s0f2" Interface "enp26s0f2" Port br-ovs Interface br-ovs type: internal $ ip link set dev br-ovs up $ ip addr add 192.168.2.98/24 dev br-ovs
測試是否可以正常通信：
# 節點A： $ ping -c 4 192.168.2.98 PING 192.168.2.98 (192.168.2.98) 56(84) bytes of data. 64 bytes from 192.168.2.98: icmp_seq=1 ttl=64 time=0.545 ms 64 bytes from 192.168.2.98: icmp_seq=2 ttl=64 time=0.089 ms 64 bytes from 192.168.2.98: icmp_seq=3 ttl=64 time=0.081 ms 64 bytes from 192.168.2.98: icmp_seq=4 ttl=64 time=0.082 ms --- 192.168.2.98 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3002ms rtt min/avg/max/mdev = 0.081/0.199/0.545/0.199 ms # 節點B： $ ping -c 4 192.168.2.99 PING 192.168.2.99 (192.168.2.99) 56(84) bytes of data. 64 bytes from 192.168.2.99: icmp_seq=1 ttl=64 time=0.113 ms 64 bytes from 192.168.2.99: icmp_seq=2 ttl=64 time=0.092 ms 64 bytes from 192.168.2.99: icmp_seq=3 ttl=64 time=0.085 ms 64 bytes from 192.168.2.99: icmp_seq=4 ttl=64 time=0.081 ms --- 192.168.2.99 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 2999ms rtt min/avg/max/mdev = 0.081/0.092/0.113/0.017 ms
創建網絡啟動腳本：
$ vi $OVS_ROOT/ovs-ifup
#!/bin/sh
set -ue
switch='br-ovs'
/usr/sbin/ip link set dev $1 up
$OVS_DIR/bin/ovs-vsctl add-port ${switch} $1創建網絡停止腳本：
$ vi $OVS_ROOT/ovs-ifdown
#!/bin/sh
set -ue
switch='br-ovs'
/usr/sbin/ip link set dev $1 down
$OVS_DIR/bin/ovs-vsctl del-port ${switch} $1為腳本增加可執行權限：
$ chmod a+x $OVS_ROOT/ovs-ifup && chmod a+x $OVS_ROOT/ovs-ifup
設初始化環境：
$ . /opt/ovs/ovs-env-nodpdk.sh
創建測試快照：
$ qemu-img create -f qcow2 -b $OVS_ROOT/images/CentOS-7-x86_64.qcow2 -o backing_fmt=qcow2 $OVS_ROOT/images/CentOS-7-x86_64_Snapshot1.qcow2 20G $ qemu-img create -f qcow2 -b $OVS_ROOT/images/CentOS-7-x86_64.qcow2 -o backing_fmt=qcow2 $OVS_ROOT/images/CentOS-7-x86_64_Snapshot2.qcow2 20G
啟動測試虛擬機1：
$ /usr/libexec/qemu-kvm -smp 2 -m 1024 -serial stdio -cdrom $OVS_ROOT/images/centos7-init.iso -hda $OVS_ROOT/images/CentOS-7-x86_64_Snapshot1.qcow2 -net nic,model=virtio,macaddr=fa:16:3e:4d:58:6f -net tap,ifname=tap-test1,script=$OVS_ROOT/ovs-ifup,downscript=$OVS_ROOT/ovs-ifdown $ ip addr show eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether fa:16:3e:4d:58:6f brd ff:ff:ff:ff:ff:ff inet 192.168.2.139/24 brd 192.168.2.255 scope global dynamic eth0 valid_lft 3509sec preferred_lft 3509sec inet6 fe80::f816:3eff:fe4d:586f/64 scope link valid_lft forever preferred_lft forever
啟動測試虛擬機2：
$ /usr/libexec/qemu-kvm -smp 2 -m 1024 -serial stdio -cdrom $OVS_ROOT/images/centos7-init.iso -hda $OVS_ROOT/images/CentOS-7-x86_64_Snapshot2.qcow2 -net nic,model=virtio,macaddr=fa:16:3e:4d:58:7f -net tap,ifname=tap-test2,script=$OVS_ROOT/ovs-ifup,downscript=$OVS_ROOT/ovs-ifdown $ ip addr show eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether fa:16:3e:4d:58:7f brd ff:ff:ff:ff:ff:ff inet 192.168.2.155/24 brd 192.168.2.255 scope global dynamic eth0 valid_lft 2634sec preferred_lft 2634sec inet6 fe80::f816:3eff:fe4d:587f/64 scope link valid_lft forever preferred_lft forever
安裝軟件包，并查看網卡信息：
$ yum install -y vim pciutils iperf3 $ lspci | grep Virtio 00:03.0 Ethernet controller: Red Hat, Inc Virtio network device $ ethtool -i eth0 driver: virtio_net version: 1.0.0 firmware-version: expansion-rom-version: bus-info: 0000:00:03.0 supports-statistics: no supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no
測試相同節點，虛擬機間Virt-IO網卡性能，CPU占用率分別為180%（Guest1）、200%（Guest2）和2.6%（ovs-vswitchd）：
Guest1 $ iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 192.168.2.155, port 41600 [ 5] local 192.168.2.139 port 5201 connected to 192.168.2.155 port 41602 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 401 MBytes 3.36 Gbits/sec [ 5] 1.00-2.00 sec 415 MBytes 3.48 Gbits/sec [ 5] 2.00-3.00 sec 419 MBytes 3.52 Gbits/sec [ 5] 3.00-4.00 sec 414 MBytes 3.47 Gbits/sec [ 5] 4.00-5.00 sec 320 MBytes 2.69 Gbits/sec [ 5] 5.00-6.00 sec 404 MBytes 3.39 Gbits/sec [ 5] 6.00-7.00 sec 408 MBytes 3.43 Gbits/sec [ 5] 7.00-8.00 sec 386 MBytes 3.24 Gbits/sec [ 5] 8.00-9.00 sec 373 MBytes 3.13 Gbits/sec [ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec [ 5] 10.00-10.04 sec 15.3 MBytes 3.21 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.04 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.04 sec 3.84 GBytes 3.28 Gbits/sec receiver Guest2 $ iperf3 -c 192.168.2.139 Connecting to host 192.168.2.139, port 5201 [ 4] local 192.168.2.155 port 41602 connected to 192.168.2.139 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 419 MBytes 3.51 Gbits/sec 18 1.30 MBytes [ 4] 1.00-2.00 sec 415 MBytes 3.48 Gbits/sec 1 1.09 MBytes [ 4] 2.00-3.00 sec 420 MBytes 3.52 Gbits/sec 0 1.35 MBytes [ 4] 3.00-4.00 sec 410 MBytes 3.44 Gbits/sec 1 1.14 MBytes [ 4] 4.00-5.00 sec 324 MBytes 2.72 Gbits/sec 0 1.33 MBytes [ 4] 5.00-6.00 sec 404 MBytes 3.39 Gbits/sec 1 1.12 MBytes [ 4] 6.00-7.00 sec 409 MBytes 3.43 Gbits/sec 0 1.36 MBytes [ 4] 7.00-8.00 sec 385 MBytes 3.23 Gbits/sec 2 1.12 MBytes [ 4] 8.00-9.00 sec 372 MBytes 3.12 Gbits/sec 0 1.35 MBytes [ 4] 9.00-10.00 sec 376 MBytes 3.16 Gbits/sec 4 1.09 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 3.84 GBytes 3.30 Gbits/sec 27 sender [ 4] 0.00-10.00 sec 3.84 GBytes 3.30 Gbits/sec receiver iperf Done.
測試不同節點，物理網卡與虛擬機Virt-IO網卡性能，CPU占用率分別為180%（Qemu），2.5%（iperf）和2.6%（ovs-vswitchd）：
Guest1 $ iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 192.168.2.98, port 42720 [ 5] local 192.168.2.139 port 5201 connected to 192.168.2.98 port 42722 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 108 MBytes 907 Mbits/sec [ 5] 1.00-2.00 sec 112 MBytes 941 Mbits/sec [ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec [ 5] 3.00-4.00 sec 112 MBytes 939 Mbits/sec [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec [ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec [ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec [ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec [ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec [ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec [ 5] 10.00-10.04 sec 4.19 MBytes 942 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.04 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.04 sec 1.10 GBytes 938 Mbits/sec receiver Node B $ iperf3 -c 192.168.2.139 Connecting to host 192.168.2.139, port 5201 [ 4] local 192.168.2.98 port 42722 connected to 192.168.2.139 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 114 MBytes 958 Mbits/sec 2 370 KBytes [ 4] 1.00-2.00 sec 112 MBytes 939 Mbits/sec 0 373 KBytes [ 4] 2.00-3.00 sec 112 MBytes 940 Mbits/sec 0 379 KBytes [ 4] 3.00-4.00 sec 113 MBytes 944 Mbits/sec 0 379 KBytes [ 4] 4.00-5.00 sec 112 MBytes 937 Mbits/sec 0 382 KBytes [ 4] 5.00-6.00 sec 113 MBytes 946 Mbits/sec 0 383 KBytes [ 4] 6.00-7.00 sec 112 MBytes 937 Mbits/sec 0 385 KBytes [ 4] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 0 386 KBytes [ 4] 8.00-9.00 sec 112 MBytes 940 Mbits/sec 0 427 KBytes [ 4] 9.00-10.00 sec 113 MBytes 952 Mbits/sec 0 553 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 1.10 GBytes 944 Mbits/sec 2 sender [ 4] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver
安裝依賴軟件包：
$ yum install -y gcc numactl numactl-libs numactl-devel kernel kernel-debug kernel-debug-devel kernel-devel kernel-doc kernel-headers libpcap-devel
設置環境變量：
$ export OVS_ROOT=/opt/ovs $ export DPDK_DIR=$OVS_ROOT/dpdk $ export DPDK_BUILD=$DPDK_DIR/build $ export DPDK_INSTALL=$DPDK_DIR/install $ export DPDK_TARGET=x86_64-native-linuxapp-gcc
獲取源碼：
$ cd $OVS_ROOT $ git clone http://dpdk.org/git/dpdk $ cd dpdk && git checkout -b v18.02/origin v18.02 && git checkout -b v18.02/devel
配置和編譯：
$ make config T=$DPDK_TARGET $ sed -ri 's,(PMD_PCAP=).*,\1y,' build/.config $ make -j16 && make install DESTDIR=$DPDK_INSTALL $ make -j16 -C examples RTE_SDK=$DPDK_DIR RTE_TARGET=build O=$DPDK_INSTALL/examples
僅本次生效：
$ mkdir -pv /mnt/huge $ mount -t hugetlbfs nodev /mnt/huges -o pagesize=2MB $ echo 8192 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages # Or $ sysctl -w vm.nr_hugepages=8192
重啟后生效：
$ vi /etc/fstab nodev /mnt/huges hugetlbfs pagesize=2MB 0 0 $ echo 'vm.nr_hugepages=8192' > /etc/sysctl.d/hugepages.conf # Or $ echo 'vm.nr_hugepages=8192' >> /etc/sysctl.conf $ sysctl -p
驗證配置是否生效：
$ grep HugePages /proc/meminfo AnonHugePages: 77824 kB HugePages_Total: 8192 HugePages_Free: 6656 HugePages_Rsvd: 0 HugePages_Surp: 0
查看網卡信息：
$ /opt/ovs/dpdk/usertools/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
<none>
Network devices using kernel driver
===================================
0000:1a:00.0 'Ethernet Connection X722 for 1GbE 37d1' if=enp26s0f0 drv=i40e unused=vfio-pci
0000:1a:00.1 'Ethernet Connection X722 for 1GbE 37d1' if=enp26s0f1 drv=i40e unused=vfio-pci
0000:1a:00.2 'Ethernet Connection X722 for 1GbE 37d1' if=enp26s0f2 drv=i40e unused=vfio-pci
0000:1a:00.3 'Ethernet Connection X722 for 1GbE 37d1' if=enp26s0f3 drv=i40e unused=vfio-pci *Active*
0000:5f:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=enp95s0f0 drv=i40e unused=vfio-pci *Active*
0000:5f:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=enp95s0f1 drv=i40e unused=vfio-pci
$ lspci | grep Ethernet | awk '{printf($1 " "); gsub(":","\\:"); cmd="ls /sys/bus/pci/devices/0000\\:" $1 "/driver/module/drivers/"; system(cmd)}'
1a:00.0 pci:i40e
1a:00.1 pci:i40e
1a:00.2 pci:i40e
1a:00.3 pci:i40e
5f:00.0 pci:i40e
5f:00.1 pci:i40e加載網卡UIO驅動模塊：
$ modprobe uio $ rmmod igb_uio $ insmod $DPDK_INSTALL/lib/modules/`uname -r`/extra/dpdk/igb_uio.ko
綁定enp26s0f2網卡UIO驅動：
$ $DPDK_DIR/usertools/dpdk-devbind.py -u 0000:1a:00.2 $ $DPDK_DIR/usertools/dpdk-devbind.py --bind=igb_uio 0000:1a:00.2
查看綁定狀態：
$ $DPDK_DIR/usertools/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:1a:00.2 'Ethernet Connection X722 for 1GbE 37d1' drv=igb_uio unused=i40e Network devices using kernel driver =================================== 0000:1a:00.0 'Ethernet Connection X722 for 1GbE 37d1' if=enp26s0f0 drv=i40e unused=igb_uio 0000:1a:00.1 'Ethernet Connection X722 for 1GbE 37d1' if=enp26s0f1 drv=i40e unused=igb_uio 0000:1a:00.3 'Ethernet Connection X722 for 1GbE 37d1' if=enp26s0f3 drv=i40e unused=igb_uio *Active* 0000:5f:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=enp95s0f0 drv=i40e unused=igb_uio *Active* 0000:5f:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=enp95s0f1 drv=i40e unused=igb_uio
編輯環境變量腳本：
$ vi $OVS_ROOT/ovs-env-dpdk.sh export OVS_ROOT=/opt/ovs export DPDK_DIR=$OVS_ROOT/dpdk export DPDK_BUILD=$DPDK_DIR/build export DPDK_INSTALL=$DPDK_DIR/install export DPDK_TARGET=x86_64-native-linuxapp-gcc export OVS_DIR=$OVS_ROOT/target-dpdk export PATH=$OVS_DIR/bin/:$OVS_DIR/share/openvswitch/scripts:$PATH
編譯和安裝：
$ . /opt/ovs/ovs-env-dpdk.sh $ cd $OVS_ROOT/ovs && ./boot.sh $ mkdir -pv $OVS_ROOT/build-dpdk $OVS_ROOT/target-dpdk && cd $OVS_ROOT/build-dpdk $ ../ovs/configure --with-dpdk=$DPDK_BUILD --with-linux=/lib/modules/$(uname -r)/build --prefix=$OVS_ROOT/target-dpdk CFLAGS="-g -Ofast" $ make -j16 'CFLAGS=-g -Ofast -march=native' && make install $ mkdir -pv $OVS_ROOT/target-dpdk/modules && cp -vf $OVS_ROOT/build-dpdk/datapath/linux/*ko $OVS_ROOT/target-dpdk/modules/
使用命令手動啟動：
$ . /opt/ovs/ovs-env-dpdk.sh && cd $OVS_DIR
$ mkdir -pv $OVS_DIR/var/run/openvswitch $OVS_DIR/etc/openvswitch
# 創建ovsdb-server數據庫：
$ $OVS_DIR/bin/ovsdb-tool create $OVS_DIR/etc/openvswitch/conf.db $OVS_DIR/share/openvswitch/vswitch.ovsschema
# 啟動ovsdb-server數據庫服務：
$ $OVS_DIR/sbin/ovsdb-server --remote=punix:$OVS_DIR/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach
# 初始化數據庫（只在第一次運行時需要執行）：
$ $OVS_DIR/bin/ovs-vsctl --no-wait init
# 查看依賴的內核模塊：
$ cat /usr/lib/modules/`uname -r`/modules.dep | awk '$1~"^extra/openvswitch"{for(i=2;i<=NF;i++) {print $i}}' | xargs echo
kernel/net/ipv6/netfilter/nf_nat_ipv6.ko.xz kernel/net/ipv4/netfilter/nf_nat_ipv4.ko.xz kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko.xz kernel/net/netfilter/nf_nat.ko.xz kernel/net/netfilter/nf_conntrack.ko.xz kernel/net/ipv4/udp_tunnel.ko.xz kernel/lib/libcrc32c.ko.xz
# 加載openvswitch.ko依賴的內核模塊：
$ cat /usr/lib/modules/`uname -r`/modules.dep | awk '$1~"^extra/openvswitch"{for(i=2;i<=NF;i++) {mod=gensub(/\.ko\.xz/,"",1,gensub(/.*\//,"",1,$i));cmd="modprobe " mod;system(cmd)}}'
# 加載openvswitch.ko內核模塊：
$ insmod $OVS_DIR/modules/openvswitch.ko
# 啟動ovs-vswitchd守護進程：
$ $OVS_DIR/sbin/ovs-vswitchd --pidfile --detach --log-file
# 為ovs-vswitchd增加dpdk支持：
$ ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
$ ovs-ctl --no-ovsdb-server --db-sock="$OVS_DIR/var/run/openvswitch/db.sock" start使用命令手動停止：
$ pkill -9 ovs $ rm -rfv $OVS_DIR/var/run/openvswitch $OVS_DIR/etc/openvswitch/ $OVS_DIR/etc/openvswitch/conf.db
在節點A上創建的NAT網絡網段為192.168.2.0/24，網關為192.168.2.99。
新增OVS網橋br-ovs：
$ ovs-vsctl add-br br-ovs -- set bridge br-ovs datapath_type=netdev $ ip link set dev br-ovs up $ ip addr add 192.168.2.99/24 dev br-ovs $ ip addr show br-ovs 21: br-ovs: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000 link/ether a2:e8:a6:bb:55:46 brd ff:ff:ff:ff:ff:ff inet 192.168.2.99/24 scope global br-ovs valid_lft forever preferred_lft forever inet6 fe80::a0e8:a6ff:febb:5546/64 scope link valid_lft forever preferred_lft forever
增加NAT地址轉換：
$ iptables -t nat -A POSTROUTING -s 192.168.2.0/24 ! -d 192.168.2.0/24 -j MASQUERADE
配置DHCP服務：
$ dnsmasq --strict-order --except-interface=lo --interface=br-ovs --listen-address=192.168.2.99 --bind-interfaces --dhcp-range=192.168.2.128,192.168.2.192 --conf-file="" --pid-file=/var/run/br-ovs-dhcp.pid --dhcp-leasefile=/var/run/br-ovs-dhcp.leases --dhcp-no-override
兩個節點添加的物理網卡在一個交換機的相同VLAN上。
在節點A的br-ovs網橋上天劍物理網口enp26s0f2：
$ ovs-vsctl add-port br-ovs dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:1a:00.2
$ ovs-vsctl show
429a3e72-c5c5-4330-9670-09492255e7e9
Bridge br-ovs
Port "dpdk0"
Interface "dpdk0"
type: dpdk
options: {dpdk-devargs="0000:1a:00.2"}
Port br-ovs
Interface br-ovs
type: internal在節點B上創建網橋br-ovs，添加物理網卡enp26s0f2，并設置網橋IP為192.168.2.98/24：
$ ovs-vsctl add-br br-ovs -- set bridge br-ovs datapath_type=netdev
$ ovs-vsctl set bridge br-ovs stp_enable=false
$ ovs-vsctl add-port br-ovs dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:1a:00.2
$ ovs-vsctl show
653d6824-6977-4806-8717-fc9e27d5ea8d
Bridge br-ovs
Port "dpdk0"
Interface "dpdk0"
type: dpdk
options: {dpdk-devargs="0000:1a:00.2"}
Port br-ovs
Interface br-ovs
type: internal
$ ip link set dev br-ovs up
$ ip addr add 192.168.2.98/24 dev br-ovs
$ ping -c 4 192.168.2.99
PING 192.168.2.99 (192.168.2.99) 56(84) bytes of data.
64 bytes from 192.168.2.99: icmp_seq=1 ttl=64 time=0.198 ms
64 bytes from 192.168.2.99: icmp_seq=2 ttl=64 time=0.160 ms
64 bytes from 192.168.2.99: icmp_seq=3 ttl=64 time=0.123 ms
64 bytes from 192.168.2.99: icmp_seq=4 ttl=64 time=0.119 ms
--- 192.168.2.99 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.119/0.150/0.198/0.032 ms設初始化環境：
$ . /opt/ovs/ovs-env-dpdk.sh
增加dpdkvhostuser端口：
$ ovs-vsctl add-port br-ovs vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuser
$ ovs-vsctl add-port br-ovs vhost-user2 -- set Interface vhost-user2 type=dpdkvhostuser
$ ovs-vsctl show
429a3e72-c5c5-4330-9670-09492255e7e9
Bridge br-ovs
Port "vhost-user2"
Interface "vhost-user2"
type: dpdkvhostuser
Port "dpdk0"
Interface "dpdk0"
type: dpdk
options: {dpdk-devargs="0000:1a:00.2"}
Port br-ovs
Interface br-ovs
type: internal
Port "vhost-user1"
Interface "vhost-user1"
type: dpdkvhostuser啟動測試虛擬機1：
$ /usr/libexec/qemu-kvm -smp 2 -m 2048 -serial stdio -cdrom $OVS_ROOT/images/centos7-init.iso -hda $OVS_ROOT/images/CentOS-7-x86_64_Snapshot1.qcow2 -net none -chardev socket,id=char1,path=$OVS_DIR/var/run/openvswitch/vhost-user1 -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce -device virtio-net-pci,mac=fa:16:3e:4d:58:6f,netdev=mynet1 -object memory-backend-file,id=mem,size=2048M,mem-path=/mnt/huge,share=on -numa node,memdev=mem -mem-prealloc $ ip addr show eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether fa:16:3e:4d:58:6f brd ff:ff:ff:ff:ff:ff inet 192.168.2.139/24 brd 192.168.2.255 scope global dynamic eth0 valid_lft 3540sec preferred_lft 3540sec inet6 fe80::f816:3eff:fe4d:586f/64 scope link valid_lft forever preferred_lft forever
啟動測試虛擬機2：
$ /usr/libexec/qemu-kvm -smp 2 -m 2048 -serial stdio -cdrom $OVS_ROOT/images/centos7-init.iso -hda $OVS_ROOT/images/CentOS-7-x86_64_Snapshot2.qcow2 -net none -chardev socket,id=char1,path=$OVS_DIR/var/run/openvswitch/vhost-user2 -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce -device virtio-net-pci,mac=fa:16:3e:4d:58:7f,netdev=mynet1 -object memory-backend-file,id=mem,size=2048M,mem-path=/mnt/huge,share=on -numa node,memdev=mem -mem-prealloc $ ip addr show eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether fa:16:3e:4d:58:7f brd ff:ff:ff:ff:ff:ff inet 192.168.2.155/24 brd 192.168.2.255 scope global dynamic eth0 valid_lft 3589sec preferred_lft 3589sec inet6 fe80::f816:3eff:fe4d:587f/64 scope link valid_lft forever preferred_lft forever
增加dpdkvhostuserclient端口：
$ ovs-vsctl add-port br-ovs vhost-user-client1 -- set Interface vhost-user-client1 type=dpdkvhostuserclient options:vhost-server-path=$OVS_DIR/var/run/openvswitch/vhost-user-client1
$ ovs-vsctl add-port br-ovs vhost-user-client2 -- set Interface vhost-user-client2 type=dpdkvhostuserclient options:vhost-server-path=$OVS_DIR/var/run/openvswitch/vhost-user-client2
$ ovs-vsctl show
429a3e72-c5c5-4330-9670-09492255e7e9
Bridge br-ovs
Port "dpdk0"
Interface "dpdk0"
type: dpdk
options: {dpdk-devargs="0000:1a:00.2"}
Port "vhost-user2"
Interface "vhost-user2"
type: dpdkvhostuser
Port "vhost-user-client2"
Interface "vhost-user-client2"
type: dpdkvhostuserclient
options: {vhost-server-path="/opt/ovs/target-dpdk/var/run/openvswitch/vhost-user-client2"}
Port ovs-veth-br
Interface ovs-veth-br
Port br-ovs
Interface br-ovs
type: internal
Port "vhost-user-client1"
Interface "vhost-user-client1"
type: dpdkvhostuserclient
options: {vhost-server-path="/opt/ovs/target-dpdk/var/run/openvswitch/vhost-user-client1"}
Port "vhost-user1"
Interface "vhost-user1"
type: dpdkvhostuser啟動測試虛擬機1：
$ /usr/libexec/qemu-kvm -smp 2 -m 2048 -serial stdio -cdrom $OVS_ROOT/images/centos7-init.iso -hda $OVS_ROOT/images/CentOS-7-x86_64_Snapshot1.qcow2 -net none -chardev socket,id=char1,path=$OVS_DIR/var/run/openvswitch/vhost-user-client1,server -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce -device virtio-net-pci,mac=fa:16:3e:4d:58:6f,netdev=mynet1 -object memory-backend-file,id=mem,size=2048M,mem-path=/mnt/huge,share=on -numa node,memdev=mem -mem-prealloc $ ip addr show eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether fa:16:3e:4d:58:6f brd ff:ff:ff:ff:ff:ff inet 192.168.2.139/24 brd 192.168.2.255 scope global dynamic eth0 valid_lft 3540sec preferred_lft 3540sec inet6 fe80::f816:3eff:fe4d:586f/64 scope link valid_lft forever preferred_lft forever
啟動測試虛擬機2：
$ /usr/libexec/qemu-kvm -smp 2 -m 2048 -serial stdio -cdrom $OVS_ROOT/images/centos7-init.iso -hda $OVS_ROOT/images/CentOS-7-x86_64_Snapshot2.qcow2 -net none -chardev socket,id=char1,path=$OVS_DIR/var/run/openvswitch/vhost-user-client2,server -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce -device virtio-net-pci,mac=fa:16:3e:4d:58:7f,netdev=mynet1 -object memory-backend-file,id=mem,size=2048M,mem-path=/mnt/huge,share=on -numa node,memdev=mem -mem-prealloc $ ip addr show eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether fa:16:3e:4d:58:7f brd ff:ff:ff:ff:ff:ff inet 192.168.2.155/24 brd 192.168.2.255 scope global dynamic eth0 valid_lft 3589sec preferred_lft 3589sec inet6 fe80::f816:3eff:fe4d:587f/64 scope link valid_lft forever preferred_lft forever
測試相同節點，虛擬機間Virt-IO網卡性能，CPU占用率分別為100%（Guest1）、100%（Guest2）和100%（ovs-vswitchd）：
Guest1 $ iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 192.168.2.155, port 54110 [ 5] local 192.168.2.139 port 5201 connected to 192.168.2.155 port 54112 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 514 MBytes 4.31 Gbits/sec [ 5] 1.00-2.00 sec 392 MBytes 3.29 Gbits/sec [ 5] 2.00-3.00 sec 781 MBytes 6.55 Gbits/sec [ 5] 3.00-4.00 sec 734 MBytes 6.16 Gbits/sec [ 5] 4.00-5.00 sec 569 MBytes 4.77 Gbits/sec [ 5] 5.00-6.00 sec 968 MBytes 8.12 Gbits/sec [ 5] 6.00-7.00 sec 688 MBytes 5.77 Gbits/sec [ 5] 7.00-8.00 sec 795 MBytes 6.67 Gbits/sec [ 5] 8.00-9.00 sec 693 MBytes 5.82 Gbits/sec [ 5] 9.00-10.00 sec 769 MBytes 6.45 Gbits/sec [ 5] 10.00-10.04 sec 37.6 MBytes 8.22 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.04 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.04 sec 6.78 GBytes 5.80 Gbits/sec receiver ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Guest2 $ iperf3 -c 192.168.2.139 Connecting to host 192.168.2.139, port 5201 [ 4] local 192.168.2.155 port 54112 connected to 192.168.2.139 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 529 MBytes 4.43 Gbits/sec 20 460 KBytes [ 4] 1.00-2.00 sec 403 MBytes 3.38 Gbits/sec 29 327 KBytes [ 4] 2.00-3.00 sec 794 MBytes 6.67 Gbits/sec 732 264 KBytes [ 4] 3.00-4.00 sec 728 MBytes 6.10 Gbits/sec 228 505 KBytes [ 4] 4.00-5.00 sec 576 MBytes 4.83 Gbits/sec 230 225 KBytes [ 4] 5.00-6.00 sec 930 MBytes 7.81 Gbits/sec 451 308 KBytes [ 4] 6.00-7.00 sec 721 MBytes 6.04 Gbits/sec 171 503 KBytes [ 4] 7.00-8.00 sec 796 MBytes 6.67 Gbits/sec 159 417 KBytes [ 4] 8.00-9.00 sec 698 MBytes 5.87 Gbits/sec 322 419 KBytes [ 4] 9.00-10.00 sec 768 MBytes 6.44 Gbits/sec 273 329 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 6.78 GBytes 5.82 Gbits/sec 2615 sender [ 4] 0.00-10.00 sec 6.78 GBytes 5.82 Gbits/sec receiver iperf Done.
測試不同節點，物理網卡與虛擬機Virt-IO網卡性能，CPU占用率分別為100%（Qemu），1.7%（iperf）和100%（ovs-vswitchd）：
Guest1 $ iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 192.168.2.98, port 50942 [ 5] local 192.168.2.139 port 5201 connected to 192.168.2.98 port 50944 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 62.5 MBytes 524 Mbits/sec [ 5] 1.00-2.00 sec 67.5 MBytes 566 Mbits/sec [ 5] 2.00-3.00 sec 63.0 MBytes 529 Mbits/sec [ 5] 3.00-4.00 sec 61.8 MBytes 519 Mbits/sec [ 5] 4.00-5.00 sec 61.8 MBytes 518 Mbits/sec [ 5] 5.00-6.00 sec 62.0 MBytes 520 Mbits/sec [ 5] 6.00-7.00 sec 61.9 MBytes 519 Mbits/sec [ 5] 7.00-8.00 sec 62.2 MBytes 522 Mbits/sec [ 5] 8.00-9.00 sec 63.0 MBytes 528 Mbits/sec [ 5] 9.00-10.00 sec 65.6 MBytes 550 Mbits/sec [ 5] 10.00-10.05 sec 3.62 MBytes 568 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.05 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.05 sec 635 MBytes 530 Mbits/sec receiver Node B $ iperf3 -c 192.168.2.139 Connecting to host 192.168.2.139, port 5201 [ 4] local 192.168.2.98 port 50944 connected to 192.168.2.139 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 67.7 MBytes 568 Mbits/sec 45 757 KBytes [ 4] 1.00-2.00 sec 67.5 MBytes 566 Mbits/sec 0 822 KBytes [ 4] 2.00-3.00 sec 63.8 MBytes 535 Mbits/sec 0 880 KBytes [ 4] 3.00-4.00 sec 61.2 MBytes 514 Mbits/sec 0 932 KBytes [ 4] 4.00-5.00 sec 62.5 MBytes 524 Mbits/sec 0 981 KBytes [ 4] 5.00-6.00 sec 61.2 MBytes 514 Mbits/sec 0 1.01 MBytes [ 4] 6.00-7.00 sec 62.5 MBytes 524 Mbits/sec 0 1.05 MBytes [ 4] 7.00-8.00 sec 61.2 MBytes 514 Mbits/sec 0 1.09 MBytes [ 4] 8.00-9.00 sec 62.5 MBytes 524 Mbits/sec 0 1.14 MBytes [ 4] 9.00-10.00 sec 66.2 MBytes 556 Mbits/sec 0 1.18 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 636 MBytes 534 Mbits/sec 45 sender [ 4] 0.00-10.00 sec 635 MBytes 533 Mbits/sec receiver iperf Done.
使用iperf3默認參數測試一個節點上兩個虛擬機，以及一個節點上的虛擬機與另外一個節點物理機之間（千兆網卡）的網絡性能，慢速狀態下，使用DPDK可以降低約80%的CPU占用率，同時會提升ovs-vswitchd守護進程的CPU占用率至100%（因為使用DPDK時是查詢模式）。
關于如何用使用DPDK優化VirtIO和OVS網絡問題的解答就分享到這里了，希望以上內容可以對大家有一定的幫助，如果你還有很多疑惑沒有解開，可以關注億速云行業資訊頻道了解更多相關知識。
免責聲明：本站發布的內容（圖片、視頻和文字）以原創、轉載和分享為主，文章觀點不代表本網站立場，如果涉及侵權請聯系站長郵箱：is@yisu.com進行舉報，并提供相關證據，一經查實，將立刻刪除涉嫌侵權內容。
