VM Junkie

October 13, 2010

vSphere Network I/O Control vs. HP VirtualConnect Flex-10 or FlexFabric

Filed under: bladesystem, hp, vmware, vSphere — ermac318 @ 8:48 am

As of vSphere 4.1, VMware introduced a new feature called Network I/O control. Many of the features of Network I/O Control overlap with some of the features of HP VirtualConncet Flex-10 (and subsequently, FlexFabric as well). This article provides a compare and contrast of the two systems and their pros and cons.

HP Flex-10

With HP Flex-10 onboard NICs, you can take a single 10Gb pipe and carve it up into 4 distinct FlexNICs, which appear as their own PCI function in hardware. Using VirtualConnect Server Profiles, you can then specify how much bandwidth you want each FlexNIC to have.

This allows customers in vSphere environments to partition bandwidth between different logical functions in hardware. For example, in the above diagram we could give 500MB of bandwidth for management traffic, 2Gb for vMotion, 4Gb for iSCSI traffic and 3.5Gb for Virtual Machine traffic per FlexNIC. In a FlexFabric environment, one of your four FlexNICs can assume the personality of a FlexHBA, which can act as a Fibre Channel HBA or hardware iSCSI initiator.

Pros:

  • Bandwidth shaping occurs in hardware and is stored in the VC Profile, and therefore is OS independent. For example, FlexNICs can be used by a physical Windows blade.
  • Since the physical NIC function is capped, both ingress and egress traffic is limited by the speed of the FlexNIC you set in hardware.

Cons:

  • Requires Flex-10 or FlexFabric capable blades and interconnect modules.
  • Can only dial up or dial down FlexNIC speeds while blade is powered off.
  • When bandwidth utilization on one FlexNIC is low, another FlexNIC cannot utilized its unused bandwidth.

vSphere Network I/O Control

Introduced in vSphere 4.1, Network I/O Control (or NIOC) is designed to solve many of the same problems as Flex-10. How can I make sure all types of traffic have an appropriate amount of bandwidth allocated, without letting any single network function rob the others of throughput?

By enabling Network I/O Control on a vDistributed Switch (vDS), you can specify limits and shares for particular port groups (illustrated above on the right) or host functions (illustrated above on the left). You can specify that vMotion traffic has a limit of 5Gbps and that is has a share value of 100. You can then specify that your VM traffic has a share value of 50, and your iSCSI traffic has a share value of 50. If all three functions were attempting to push maximum throughput, the vMotion traffic would push 5Gbps (since vMotion is given 100 out of 200 shares), VM and iSCSI traffic would get 2.5Gbps.

An example screenshot, taken with a 1Gb (not 10Gb) NIC.

Pros:

  • Shares allow bandwidth on a single function to utilize the entire 10Gb pipe if the link is not oversubscribed.
  • You can change the speed of a function while the vDS is online and servicing traffic.
  • No special hardware required – can be utilized on rack-mount servers with standard 1Gb or 10Gb NIC interfaces.

Cons:

  • Requires vSphere Enterprise Plus, and requires use of the vDS – NIOC is not available with traditional vSwitches.
  • NIOC can only regulate egress traffic. Ingress traffic will not be affected by NIOC settings.

Conclusions

Both options provide similar capabilities but approach the problem in different ways. While a FlexNIC cannot dial itself up dynamically based on load, it can prevent ingress traffic from overwhelming other functions, whereas NIOC cannot.

The biggest problem with NIOC is that it is only available with the vDistributed Switch, making it challenging for many customers to implement. Not only do they need to be on the most expensive version of vSphere, but they also must then implement vDS, which many customers are not doing or avoiding intentionally due to the added complexity. However, VMware will most likely be targeting only the vDS for future feature enhancements.

In HP Blade environments, it makes sense to utilize the HP VirtualConnect technology as it provides other benefits (MAC address virtualization, server profile migration, and now FlexFabric) beyond just the FlexNIC capability. However, if customers are utilizing competing Blade solutions, or traditional rack-mount servers, then NIOC provides new capabilities to them that they cannot get in hardware.

It is also possible to utilize both solutions in tandem. One could conceivably use FlexNICs to segregate certain types of traffic for security purposes (maybe if your organization doesn’t allow traffic from different security zones on the same vSwitch) and then use NIOC to do bandwidth shaping. Another use case is if you want your Management Traffic to stay on a standard vSwitch, but move all VM/vMotion/etc traffic to a vDS, you can use two FlexNICs per pipe and use NIOC on the larger of the two.

Advertisements

9 Comments

  1. Hello,

    a couple of corrections:

    1) NIOC does indeed do ingress bandwidth control ( http://www.vmware.com/files/pdf/techpaper/VMW_Netioc_BestPractices.pdf ).

    2) Conversely HP Flex-10 does not support ingress control.

    Without ingress bandwidth controls we will likely be moving away from HP’s Flexnics for bandwidth management. Rate capping, vs shares or proper QoS, is already inefficient and without ingress bandwidth controls it just isn’t good enough at the moment.

    Comment by Andrew VanSpronsen — October 14, 2010 @ 3:34 am

  2. Hi,
    It should be noted in the Cons for Flex-10 or FlexFabric can it can only control egress traffic. With Flex-10 or FlecFabric, traffic from other servers converging onto one server has no mechanism to insure one traffic type (vMotion) doesn’t saturate the link.

    Comment by Brad Hedlund — October 14, 2010 @ 3:57 am

  3. Andrew,
    I don’t believe either of these are correct.
    As you can see from the whitepaper that you linked, the settings which are enabled in order to “impose” ingress control are actually setting egress controls on other hosts. Their examples include making sure VMotion traffic doesn’t saturate a 10Gb link. They do this by enforcing egress controls on the other hosts. In fact, quoted from the whitepaper:
    “While NetIOC enables you to limit vMotion traffic initiated from a vSphere host, it fails to prevent performance loss when multiple vMotion traffic flows initiated on diferent vSphere hosts converge onto a single vSphere host and possibly overwhelm the latter. We will show how a solution based on NetIOC and Traffic Shaper can prevent such an unlikely event.”
    This implies that NetIOC cannot do ingress controls, and this is confirmed by the official VMware presentation “What’s New in vSphere 4.1 Technical Edition” that can be found on the partner portal, if you are a VMware partner.

    Can you please point me to any documentation which shows Flex10 cannot limit ingress traffic? All my documentation states otherwise.

    Comment by ermac318 — October 14, 2010 @ 9:23 am

  4. Brad,
    According to all the documentation I can find this is incorrect.
    If you isolate VMotion traffic to a particular FlexNIC, then the ingress traffic on that FlexNIC can only be up to the speed that you set on the port. If I put both VM and VMotion traffic on the same FlexNIC, then no I cannot control the differences. But that’s why I have 8 FlexNICs.
    Do you have any documentation to corroborate your statement?

    Comment by ermac318 — October 14, 2010 @ 9:27 am

  5. Hey,

    I was just trying to point out a couple of inaccuracies…not do your research. With NIOC and Traffic Shaping now available in 4.1 you can control both ingress and egress traffic flows with vDS.

    As Brad also has confirmed Virtual Connect does NOT support ingress traffic management. we have been discussing this for a long time with HP directly.

    Comment by Andrew VanSpronsen — October 14, 2010 @ 10:07 am

  6. vSphere 4.1 with the vDS can control Egress and Ingress bandwidth on a per port group bases using Traffic Shaping and provide Egress share and limits based on Traffic type using Network I/O Control.

    The main advantage is with Network I/O Control is that it only kicks in when there is contention on the uplink. If there is no contention then it will not enforce the shares on the a specific traffic but it will enforce a limit. If you really need/want to limit a specific port on a vDS then setup a port group and use traffic shaping.

    Another benefit of using the VMware solution is that you can actually get 10Gb of throughput on a given uplink. If you use the new Load Based Teaming feature then you can get close to 20Gb of throughput across two uplinks. If you use Flex10 the to control bandwidth then you only get some reduced amount of bandwidth that is spread across the 4 Flexnics. The other issue is the if the bandwidth is not being used by one Flexnic it cannot be used by another one that could use it or actually needs it.

    Remember, in 4.1 vMotion now can move 8 VMs concurrently using up to 8+Gb of bandwidth. In our tests vMotion will not use the full 8Gb if there is other traffic on the line but then again you can always use NIOC if you want to put a limit on it.

    Also note that we are dealing with uplinks between switches not a connection between a server and a switch. How much QoS do you do on switch uplinks? The typically answer I get is none unless I really have too… So why is there so much concern that 20Gb of bandwidth is not enough. Pull up ESXTop and look at your typical VM traffic and VMkernel traffic…

    Take a look at my two white papers on Best Practices on VMware and Virtual Switches. http://www.intel.com/go/ethernet
    http://www.intel.com/support/network/sb/CS-030881.htm

    I have a third paper discussing the QoS features of 4.1 in the final review cycle so I am very interested in everyone’s thoughts on this subject.

    Brian Johnson
    10Gb Ethernet and Virtualization Technologies
    LAN Access Division
    Intel Corporation

    Comment by Brian Johnson — October 14, 2010 @ 4:43 pm

  7. @ermac318,

    Nowhere in the HP Flex-10 technology brief does it say Flex-10 controls receive bandwidth. In every statement about bandwidth it only talks about transmit traffic. This is not by accident. In fact, searching for the word “receive” in the document produces zero hits.

    http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01608922/c01608922.pdf

    Cheers,
    Brad

    Comment by Brad Hedlund — October 15, 2010 @ 2:40 am

  8. Andrew,
    NetIOC can only control ingress traffic with a rate limiter. The “Shares” concept of bandwidth management is only used for traffic leaving a host (egress).

    Cisco UCS with the Cisco VIC adapter is an example of a system that is capable of applying bandwidth “Shares” (QoS) for both ingress and egress traffic because the traffic management is coordinated between the server NIC *and* the network switch port it’s connected to.

    Comment by Brad Hedlund — October 15, 2010 @ 5:34 am

  9. […] vSphere Network I/O Control vs. HP VirtualConnect Flex-10 or FlexFabric « VM Junkie (tags: nioc flex-10) AKPC_IDS += "848,";Popularity: unranked [?] […]

    Pingback by links for 2010-10-18 — mtellin.com — October 18, 2010 @ 12:07 pm


RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

Create a free website or blog at WordPress.com.

%d bloggers like this: