Since I have a 10 Gbit switch, I thought I would test to see what sort of network throughput I can get between VMs in two scenarios – one using standard networking and one when connected using NSX logical switches. I have previously done this and noticed that the throughput when using the VTEP vmk’s for encapsulation I see a drop in performance and I’ve been meaning to figure out why. I’m going to break it down as follows:
- Test between two VMs on the same host using a vSphere Distributed Switch (VDS)
- Test between two VMs on two different hosts using a vSphere Distributed Switch
- Repeat the above tests, however the VMs will be connected to a Logical Switch
OS wise I’m going to use Ubuntu Server (18.04.2 LTS). To test the performance I’m going to use iperf3 which is the usual go to tool for network performance benchmarking. Please note: I believe that the default commands are single core only, which is fine for 10 Gbit/s but for environments with more bandwidth available, please read into this.
First task was to build the VMs. Ubuntu builds (and updates) very quickly from the installation ISO. Spec wise I used 2 vCPU and 4 GB of RAM. I’m far from a fully fledged Linux admin, but I know my way around, I can use vi, update the OS, search log files etc (mostly learned from fault finding vCenter!) so enough for my needs. Once built, log in, install iperf3 and check for updates.
sudo apt install iperf3
sudo apt update
sudo apt upgrade
rebootDo the above one at a time as you’ll have to confirm the install of iperf3 as well as the updates (if any are available). I do love the simplicity of Linux. After ensuring that both VMs are on different hosts I started a basic test.
As you can see, without any tinkering I’m seeing near 10 Gbit/s performance. I could modify the iperf settings to gain a little more, but I feel like that would be wasted effort. Both VMs are on the same subnet, so the path between VMs is not complex.
Next step is to put the VMs on a Logical Switch. This means that the traffic is now going to go through the VTEP vmk interfaces on the host to have the encapsulation added to the IP header of the frame, so that the physical (underlay) network can pass the data without having to hairpin to a L3 device for routing.
Once on the Ubuntu VMs, I had to use netplan to give them a static IP as I’ve not set up any DHCP for the Logical Switches yet. As you can see, throughput has dropped by 50% – not good!
While 5 Gbit/s between hosts is plenty of performance for a homelab, it is not in a heavy workload production environment, plus I wanted to try and understand why this is happening.
From studying, I know that NSX is very sensitive to MTU sizes. All of my infrastructure was built with an MTU of 9000 where available. This includes the VTEP vmk, the DLR interfaces etc. Now, the only thing that hasn’t been configured is the VM itself. The majority of operating systems will default to an MTU value of 1500 when freshly built. This is because it provides very good compatibility. We’ve already seen that 1500 is capable of line rate throughput, however due to the extra overheads that encapsulation/de-encapsulation brings when processing a large number of frames, it really benefits from having the MTU increased.
A quick command on Ubuntu we can run to change the MTU is sudo ifconfig ens160 mtu 8900 where ens160 is the name of the adaptor shown in ifconfig. Please note: this will not persist through a reboot.
And we’re in! Proof that NSX is in fact extremely sensitive when it comes to MTU size. This will be far more prevalent in production environments with 25 Gbit/s and above which is becoming far more common.
I’m hoping this may help someone out that faces the same issue. Those designing NSX in production environments should also take this into account as there are areas where changing the MTU later down the line becomes difficult, such as on the VXLAN settings for example.