Symptoms

Network performance between VMs running on different nodes in the cluster is limited when using 10 Gb/s interface or more.

Private networks based on physical networks with MTU 1500 will have only 1/3 of network bandwidth (i.e. ~3.5Gb at 10Gb network). 

Cause

Nodes have MTU set to 1500 by default for a physical interface because different hardware has different maximum MTU supported.

Resolution

• Make sure that your infrastructure network switches and physical cards support jumbo frames (MTU values higher than 1500) and ensure that switch ports are already set to use higher MTU values before making changes on the nodes.

• Increase MTU on node physical interfaces up to the maximum supported by your network equipment. Note that you should apply this setting for each node in the cluster and network equipment MTU changes should be applied first. 

We recommend applying such settings during setup or before the cluster deployment due to the fact that a production cluster in most cases will already contain additional private networks deployed, which would require more steps during the MTU change. However you can apply these changes on the already running nodes using CLI by following such steps:

To increase the MTU of the bond0 interface from the default of 1500 bytes to 9000 bytes:

1. Choose the node where the MTU will be changed and put it into maintenance mode.

2. Make sure you are using a correct bond interface and its member interfaces:

# ip link show dev bond0
10: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 90:e2:ba:5e:6f:b0 brd ff:ff:ff:ff:ff:ff

# grep 'Slave Interface' /proc/net/bonding/bond0 | awk '{print $3}'
p2p1
p2p2

3. Set the desired MTU value for the node's bond interface:

# ip link set mtu 9000 dev bond0

4. Verify that new MTU values have been propagated on both bond interface and its members:

# ip link show dev bond0
10: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 90:e2:ba:5e:6f:b0 brd ff:ff:ff:ff:ff:ff

# ip link show dev p2p1
8: p2p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
link/ether 90:e2:ba:5e:6f:b0 brd ff:ff:ff:ff:ff:ff

# ip link show dev p2p2
9: p2p2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
link/ether 90:e2:ba:5e:6f:b0 brd ff:ff:ff:ff:ff:ff

5. Make changes persistent through node reboots by adding an MTU value to your bond interface configuration file and its member cfg files:

/etc/sysconfig/network-scripts/ifcfg-bond0
/etc/sysconfig/network-scripts/ifcfg-p2p1
/etc/sysconfig/network-scripts/ifcfg-p2p2

• Add an “MTU="9000"” line to these files so that they show this value as such:

# for i in {bond0,p2p1,p2p2}; do grep MTU /etc/sysconfig/network-scripts/ifcfg-$i; done
MTU="9000"
MTU="9000"
MTU="9000"

• Increase the MTU for the interface using the Web Management panel:

6. Return the node back to operations by exiting maintenance mode and move to the next node.

Repeat steps 1 to 6 for each nodes in the cluster one by one.

---

Assuming changes were applied on a running cluster with already deployed private networks - such networks (created before the MTU changes) will contain an old MTU value. In such case a private network MTU change is also required.

You should change the private network MTU to the physical interface MTU value –(minus)87
("privMTU" = "physMTU" – "87").

In order to do so:

1. Find out all your private network ID's, use the following command:

# vinfra service compute network list --long -c id -c type | awk '$(NF-1)~/virtual/ {print $2}'

All these networks` MTU should be changed.

2. Check the selected private network MTU value - choose any of such network UUID and run:

# vinfra service compute network show <UUID> -c mtu

3. Change MTU for the selected network from the main controller node:

# source /etc/kolla/admin-openrc.sh
# openstack --insecure network set --mtu 8913 <UUID>

Make sure to use the value of the physical interface MTU – 87 bytes. These 87 bytes have to be reserved for VXLAN headers (50 bytes) and encryption headers (37 bytes).

To speed up the changes you may execute this from the main controller node in cycle, e.g:

# source /etc/kolla/admin-openrc.sh
# for b in $(vinfra service compute network list --long -c id -c type | awk '$(NF-1)~/virtual/ {print $2}'); do openstack --insecure network set --mtu 8913 $b; done 

Important: do not make any changes unless you are completely sure they are necessary and you understand each and every command.