Distributed Resource Scheduler (DRS)
DRS is vSphere’s resource management and load balancing function. It carries out VM placement dependent on resources available. It also enforces user-defined resource allocation policies at the cluster level.
The main aim of DRS is to ensure all VMs and their applications are getting the right amount of compute resources to run efficiently.
Increased load on certain VMs can cause resource imbalances in the cluster, which DRS will aim to rectify as it monitors the cluster every 5 minutes. On detecting an imbalance, DRS checks if a VM would be better served on a different ESXi host and if so, will migrate it across using vMotion. On initial deployment, a VM goes through a DRS decision known as VM placement or initial placement based on the anticipated resource distribution change due to the VM’s inclusion (so far as there are no constraint violations caused by the VM’s inclusion). DRS will attempt to rectify load balancing issues in accordance with its schedule and algorithm.
Any imbalanced loads lead to a VM migration as mentioned previously. DRS uses the demand of every VM to determine this.
CPU demand is based on current VM CPU usage. Memory demand is based on the following formula.
VM memory demand = Function(Active memory used, swapped/shared) + 25% (idle consumed memory).
In summary, DRS looks mostly at a VM’s active memory usage and a small amount of its idle memory usage to budget for anticipated memory utilisation.
Automation Levels
DRS has the following automation levels based on who or what configures the 1) initial placement and 2) load balancing settings.
• Manual: both settings done by user
• Partially Automated: DRS applies initial placement, user applies load balancing
• Fully Automated: both settings done by DRS
This can be configured by going to the cluster’s Edit settings and selecting the vSphere DRS option.
In these options, vSphere will make recommendations where it doesn’t actually carry out an automation action e.g. in the manual and partially automated settings.
Aggression Levels
DRS has 5 different aggression levels or migration thresholds, with 1 as do not load balance VMs, only make recommendations and 5 being automate load balancing.
VM Overrides
VMs can be set to override DRS automation and migration threshold changes that are typically applied at the cluster level. For instance, a VM could be designated not need to be moved if its movement would or could adversely affect network or service operations based on its function.
This can be done by going to Cluster > Manage > Settings > VM Overrides and setting the automation or migration threshold for the VM to a non-cluster level setting if not disable them fully.
VM/Host Rules
There are instances where VMs need to be kept together or on a particular host. In these instances, VM/Host rules can be set in accordance with the following requirements.
• Keep Together VM-VM: always runs these VMs on the same host
• Keep Separate VM-VM: always run these VMs on different hosts
• VM-Host: for groups of one or more VMs on one or more hosts (configure this at Cluster > Manage > Settings > VM/Host Groups)
VM/Host rules can be configured with the following settings.
• Should: preferred but DRS can drop if cluster imbalance is very high
• Must: mandatory at all times
VM-VM rules are always set to must. HA can ignore these rules during a failover, but DRS will then fix this during its first invocation.
PowerCLI
PowerCLI is the VMware implementation of PowerShell commands for managing its infrastructure and components PowerCLI is a key tool in the automation of VMware workloads. It lends heavily from its PowerShell roots and extends the functionality of VMware by giving developers and administrators toolkits for the consistent, idempotent deployment of resources with the benefits that go with Infrastructure as Code (IaC) deployments.
Installing PowerShell on Linux
The following sections use PowerCLI commands to achieve their objectives. You will have previously set up PowerShell on your device and installed the required VMWare PowerCLI modules to carry out its commands. These setup instructions are shown below for Linux and were correct at the time of writing.
Set up the PowerShell repository for use on the local device using the curl command.
curl https://packages.microsoft.com/config/rhel/7/prod.repo | sudo tee /etc/yum.repos.d/microsoft.repo
Install PowerShell using Yum.
sudo yum install -y powershell
Run the pwsh command to launch PowerShell.
pwsh
You will now be presented with a PowerShell CLI.
Run the following command to install the required VMWare PowerCLI modules.
Install-Module -Name VMWare.PowerCLI -Scope CurrentUser
Then check the modules have been installed successfully using the Get-Module command.
Get-Module -ListAvailable VMware*
Ensure that the list generated shows these modules as successfully installed. This list should be like the following.
Exit PowerShell to return to the Linux shell prompt. exit
That’s it. You will now be able to launch PowerShell using the pwsh command and from there run any relevant PowerCLI commands.
Adding a Software Depot (Extensivity Rule Setup)
As described earlier, an extensivity rule is a means of ensuring that only the specifically required VIBs in an image profile are used. Carry out the following steps to implement this using PowerCLI
Connect to the vCenter server that AutoDeploy is registered with.
Connect-VIServer <ipaddress>
Find the public software depot of define a customer image profile using the ESXi Image Builder.
Run Add-EsxSoftwareDepot to add the software depot containing the relevant image profile to the current PowerCLI session.
Add-EsxSoftwareDepot <depoturl or zipfilelocation>
Find the image using the Get-EsxImageProfile command.
Get-EsxImageProfile
Prepare the server for bash connectivity
Enable the AutoDeploy service and get the ESXi boot image using TFTP
Set up the AutoDeploy rules using PowerCLI
This entails checking if the remote or local depot is used
• Confirm that a windows path is ready
• Create a download directory and download the ESXi software depot to it
• Create and run a Powershell script to apply the AutoDeploy rules
• Clean up the local copy of the AutoDeploy rules script
Set up AutoDeploy Assign Image Rule
Now you can run the following commands to AutoDeploy assign an Image Rule.
Run the Connect-VIServer cmdlet to connect to the vCenter Server
Connect-VIServer {{ vsphere_ip }}
Run Add-EsxSoftwareDepot to add the software depot that contains the image profile
Add-EsxSoftwareDepot "{{ remote_path }}"
OR Download the ZIP file to a local file path e.g C:\local_depot
Add-EsxSoftwareDepot {{ local_path }}
Find the image profile required
Get-EsxImageProfile {{ cmdlet }}
By default, the ESXi depot includes one base image profile that includes VMware tools and has the string standard in its name, and one base image profile that does not include VMware tools. Define a rule in which hosts with certain attributes, for example a range of IP addresses, are assigned to the image profile.
Create a rule, assign an image profile for a vendor in a specific ip range
New-DeployRule -Name "{{ image_rule }}" -Item "{{ image_profile }}" -Pattern "vendor={{ vendor }}", "ipv4={{ image_ip_range }}"
Add the rule to the rule set.
Add-DeployRule {{ image_rule }}
Set up AutoDeploy Assign Host Rule
Run the Connect-VIServer cmdlet to connect to the vCenter Server.
Connect-VIServer {{ vsphere_ip }}
Create a rule, assign an image profile for a vendor in a specific ip range
New-DeployRule -Name "{{ host_rule }}" -Item "{{ host_profile }}" -Pattern "vendor={{ vendor }}", "ipv4={{ host_ip_range }}"
Add the rule to the rule set
Add-DeployRule {{ host_rule }}
Add the Script Bundle and check that is appears in the installed bundle list.
Add-ScriptBundle {{ script_bundle_path }}
Get-ScriptBundle
Add the Deploy Rule.
New-DeployRule -Name "{{ host_rule }}_script" -Item "{{ host_profile }}_script" -Pattern "vendor={{ vendor }}", "ipv4={{ host_ip_range }}"
Add-DeployRule {{ host_rule }}_script
These are also shown below.
Connect-VIServer ipv4_or_ipv6_address
New-DeployRule -Name "testrule2" -Item my_host_profile -Pattern "vendor=Acme,Zven", "ipv4=192.XXX.1.10-192.XXX.1.20"
Add-DeployRule testrule2
Write a Rule Assigning a Host to a Cluster or Folder
This adds a Host to a cluster or folder using a deploy rule.
Connect-VIServer 1.1.1.1
New-DeployRule -Name "gridrule" -Item "GridScript" -Pattern "vendor=Gridlockaz,Gridiron", "ipv4=192.168.1.10-192.168.1.20"
Add-DeployRule gridrule
Configure Stateless System Using a Script
This adds a script bundle using the Add-ScriptBundle PowerCLI command before applying the deploy rule.
Connect-VIServer 1.1.1.1
Add-ScriptBundle c:/temp/MyScriptBundle.tgz
New-DeployRule -Name "gridrule" -Item "GridScript" -Pattern "vendor=Gridlockaz,Gridiron", "ipv4=192.168.1.10-192.168.1.20"
Add-DeployRule gridrule
Test and Repair (Needed Before a Rule Can Be Applied)
Connect-VIServer 1.1.1.1
Get-DeployRule
Copy-DeployRule -DeployRule testrule -ReplaceItem GridProfile
Get-VMHost -Name GridHost
$tr = Test-DeployRuleSetCompliance GridHost
$tr.itemlist
The output from these commands would be similar to the following.
CurrentItem ExpectedItem
Grid Profile 20 GridProfile
Repair-DeployRuleSetCompliance $tr
Register a Cache Proxy with AutoDeploy
Connect-VIServer ipv4_or_ipv6_address
Add-ProxyServer -Address 'https://proxy_server_ip_address:port_number'
PowerCLI SSH Setup Command Examples
The following commands summarise the setup of SSH connectivity to an ESXi host that you can use. It also shows a few good examples of PowerCLI’s use in a real environment.
Connect to an ESXi host
connect-viserver -Server 192.168.11.63 -User root -Password Password123
Start SSH
get-vmhostservice | where {$_.Key -eq "TSM-SSH"} | Start-VMHostService
Connect to vCenter
Connect-VIServer -Server 192.168.11.87 -user <your account> -password <your password>
List the managed hosts
Get-VMHost
List and start SSH on all found hosts
Get-VMHost | foreach { get-vmhostservice -VMHost $_.name | where {$_.Key -eq "TSMSSH"} | Start-VMHostService}
List and stop SSH on all found hosts
Get-VMHost | foreach { get-vmhostservice -VMHost $_.name | where {$_.Key -eq "TSMSSH"} | Stop-VMHostService}
PowerCLI Commands Summary
The following is a summarised list of PowerCLI commands that you will find useful in your deployments.
Command Description
• Get-DeployCommand Returns a list of vSphere Auto Deploy cmdlets.
• New-DeployRule Creates a new rule with the specified items and patterns.
• Set-DeployRule Updates an existing rule with the specified items and patterns. You cannot update a rule that is part of a rule set.
• Get-DeployRule Retrieves the rules with the specified names.
• Copy-DeployRule Clones and updates an existing rule.
• Add-DeployRule Adds one or more rules to the working rule set and, by default, also to the active rule set. Use the NoActivate parameter to add a rule only to the working rule set.
• Remove-DeployRule Removes one or more rules from the working rule set and from the active rule set. Run this command with the -Delete parameter to completely delete the rule.
• Set-DeployRuleset Explicitly sets the list of rules in the working rule set.
• Get-DeployRuleset Retrieves the current working rule set or the current active rule set.
• Switch-ActiveDeployRuleset Activates a rule set so that any new requests are evaluated through the rule set.
• Get-VMHostMatchingRules Retrieves rules matching a pattern. For example, you can retrieve all rules that apply to a host or hosts. Use this cmdlet primarily for debugging.
• Test-DeployRulesetCompliance Checks whether the items associated with a specified host are in compliance with the active rule set.
• Repair-DeployRulesetCompliance Given the output of TestDeployRulesetCompliance, this cmdlet updates the image profile, host profile, and location for each host in the vCenter Server inventory. The cmdlet might apply image profiles, apply host profiles, or move hosts to prespecified folders or clusters on the vCenter Server system.
• Apply-EsxImageProfile Associates the specified image profile with the specified host.
• Get-VMHostImageProfile Retrieves the image profile in use by a specified host. This cmdlet differs from the Get-EsxImageProfile cmdlet in vSphere ESXi Image Builder.
• Repair-DeployImageCache Use this cmdlet only if the vSphere Auto Deploy image cache is accidentally deleted.
• Get-VMHostAttributes Retrieves the attributes for a host that are used when the vSphere Auto Deploy server evaluates the rules.
• Get-DeployMachineIdentity Returns a string value that vSphere Auto Deploy uses to logically link an ESXi host in vCenter Server to a physical machine.
• Set-DeployMachineIdentity Logically links a host object in the vCenter Server database to a physical machine. Use this cmdlet to add hosts without specifying rules.
• Get-DeployOption Retrieves the vSphere Auto Deploy global configuration options. This cmdlet currently supports the vlan-id option, which specifies the default VLAN ID for the ESXi Management Network of a host provisioned with vSphere Auto Deploy. vSphere Auto Deploy uses the value only if the host boots without a host profile.
• Set-DeployOption Sets the value of a global configuration option. Currently supports the vlan-id option for setting the default VLAN ID for the ESXi Management Network.
• Add-ProxyServer Adds a proxy server to the vSphere Auto Deploy database. Run the command with the -Address parameter to specify the IPv4 or IPv6 address. The address can include a port number.
• List-ProxyServer Lists the proxy servers that are currently registered with vSphere Auto Deploy.
• Delete-ProxyServer Deletes one or more proxy servers from the list of proxy servers that are registered with vSphere Auto Deploy. You can run the command with the -id parameter from the list of proxy servers or with the-Address parameter by specifying the IPv4 or IPv6 address of the proxy server you want to delete.
• Add-ScriptBundle Adds one or more script bundles to the vSphere Auto Deploy server.
• Get-ScriptBundle Retrieves the list of script bundles available on the vSphere Auto Deploy server and the scripts they contain.
• Remove-ScriptBundle Removes a script bundle from vSphere Auto Deploy. Applicable for vSphere version 6.7 and later.
ESXCLI, Logging and Troubleshooting
The ESXCLI utility runs commands against the various ESXi 'namespaces'.
Below are a few useful commands.
esxcli system version get
esxcli system hostname get
esxcli system stats installtime get
esxcli system account list
esxcli system account add -d="Altaro Guest" -i="altaro" -p="dsfewfewf*3!4404"c="dsfewfewf*3!4404"
esxcli system maintenanceMode set –enable true
esxcli system shutdown reboot -d 10 -r "Patch Updates"
esxcli network firewall get
esxcli network firewall set –enabled true | false
esxcli network firewall ruleset list | awk '$2=="true"'
esxcli network ip interface ipv4 get
esxcli software vib list
esxcli software vib update -d "/tmp/update.zip"
esxcli vm process list
esxcli vm process kill -w 69237 -t soft
esxcli storage vmfs extent list
esxcli storage filesystem list
esxcli iscsi software set –enabled true && esxcli iscsi software get
esxcli iscsi adapter param get -A vmhba65
esxcli esxcli command list
Upgrading a Host with a VIB Using ESXCLI
If there are just a few hosts to upgrade, you can use the CLI instead of using the VUM. This can be done as follows.
Copy the VIB patch to the host using your preferred SCP tool.
Put host in maintenance mode.
vim-cmd hostsvc/maintenance_mode_enter
Update/install the patch e.g. if the patch was saved as /vmfs/volumes/55fbd499-7588730ff5a1-005056b87047/ESXi550-201601001.zip.
esxcli software vib update -d “/vmfs/volumes/55fbd499-7588730f-f5a1005056b87047/ESXi550-201601001.zip”
Note that the vib option can use either of the following options.
- update: does not overwrite a patch if currently present
- install: overwrites a patch if currently present, this is the same as upgrading the host
Take host out of maintenance mode.
vim-cmd hostsvc/maintenance_mode_exit
Reboot the host.
reboot That's it.
ESXCLI Commands by Namespace
The following list shows the various esxcli command options by their namespaces.
Command Description
• esxcli device Lists descriptions of device commands.
• esxcli elxnet Lists descriptions for commands that manage Emulex elxnet drivers.
• esxcli esxcli Lists descriptions of esxcli commands.
• esxcli fcoe FCOE (Fibre Channel over Ethernet) commands
• esxcli graphics Graphics commands
• esxcli hardware Hardware namespace. Used primarily for extracting information about the current system setup.
• esxcli iscsi iSCSI namespace for monitoring and managing hardware and software iSCSI.
• esxcli network Network namespace for managing virtual networking including virtual switches and VMkernel network interfaces.
• esxcli nvme Commands for managing NVMe devices.
• esxcli rdma Commands for monitoring RDMA devices.
• esxcli sched Manage the shared system-wide swap space.
• esxcli software Software namespace. Includes commands for managing and installing image profiles and VIBs.
• esxcli storage Includes core storage commands and other storage management commands.
• esxcli system System monitoring and management command.
• esxcli vm Namespace for listing virtual machines and shutting them down forcefully.
• esxcli vsan Namespace for Virtual SAN management commands. See the vSphere Storage publication for details.
HA Issues
Check the following ESXi folder location for issues such as HA issues.
/var/log/fdm.log
FDM is the HA agent and issues can occur when it is being installed.
Update/Upgrade Issues
Check the following location for such issues.
/var/log/esxupdate.log
For instance, the following tail command will show the latest log output.
tail -f /var/log/esxupdate.log
Root Partition Issues
Check and if necessary, change /root partition settings using the following CLI command. vdf
Cluster/vMotion Issues
Check the following items.
• VMKernel ports
• VM reservation
• Network time
• IP address
• VLAN
The esxtop CLI tool can also be used to view CPU and memory output on the ESXi host. It is like the Linux top CLI tool.
Run the following command to view its options.
esxtop -h
The output would be like the following.
Running the command on its own shows quite a lot of screen output, so handle with care especially if on a low speed connection.
Network Issues
The esxcli CLI tool can be used to gather network information. A few examples are shown below and you can run the command on its own to view its help menu.
The following are the options for the network subcommand from the above output i.e. when the following command is launched.
esxcli network
The list of network interface card subcommand options possible with thie following command is shown below.
esxcli network nic
The following command lists the network cards on the ESXi host.
esxcli network nic list
VM Issues
For example, the following command lists the VMs and their process-related information.
esxcli vm process list
Not that it also shows other key VM information such as its .vmx config file and
UUID.
The esxcli and esxcfg-* tools can be used to gather network information. A few examples are shown below, and you can run the relevant command with its -h option to view its help menu for more possible ways of getting information.
esxcfg-vmknic -l
Also, check the contents of the /var/log/vmware.log file as follows.
Storage, Memory CPU and Other Hardware Issues
You can also troubleshoot various storage issues using the esxcli storage subcommand. Below is a list of its options shown by running the following command.
esxcli storage
View the hardware namespace options as follows.
esxcli hardware
View the CPU namespace options as follows.
esxcli hardware cpu
Then you can list all of the ESXi’s CPUs with the following command. esxcli hardware cpu list
Viewing Output From /var/log
It is also worth reviewing relevant logs saved in the /var/log folder.
The following output shows a typical list of logs contained in this folder.
For instance, the dhclient.log file shows the DHCP process activity carried out by the ESXi host as shown below.
The syslog.log file output would be similar to that shown below.
VMware Integrated OpenStack (VIO)
The DefCore compliant OpenStack distribution is an OVA file and is fully supported by vSphere. It has the core functionality of Nova, Neutron, Cinder and Glance fully compatible with their equivalent vMWare services of Compute, Network, Block Storage and Image Storage respectively.
As the name itself implies, VIO is an integrated direction in terms of its working with VMware. This is not a custom fit but a fully-fledged OpenStack implementation that combines the best of both it and the VMware platform. It avoids latent artifacts and onpremises snowflake servers, that require hight levels of non-standardised configuration and maintenance.
This standard version of Openstack itself implemented on top of the VMWare SDDC architecture. It is taking the current VMware infrastructure, including the VMware community drivers, reference architecture and adding reporting and operations tools such as vRealize to enhance its management, reporting and insights.
The Software-Defined Data Center (SDDC) will consist of the following components.
• vCenter Nova
• NSX Controller Neutron
• ESXi
• VSAN
• VDS
• Datastores cinder Glance
• LDAP Keystone
• vRB Swift (object store)
VIO vs OpenStack
VIO is the same as OpenStack, but instead of running on bare metal it is running on VMware. It uses the exact same APIs and services as available in any other Openstack distribution. The control plane is Ubuntu also the same, but VIO uses vCenter drivers as well as VMDK drivers for storage as well as NSX driver. It is also DefCore validated. Check the www.openstack.org marketplace site to confirm that VIO has been verified and certified by OpenStack.
Required Networks
VIO requires a specific network implementation and so the following networks are required.
• API access – developer and user access
• Management network control plane communication, LB connecting them, over SSL by default
• Transport network used by NSX
• External network – floating IPs for instances are here
Clusters
VIO requires the following clusters or logical roles.
• Management (minimum 3 hosts)
• Edge (minimum 2 hosts, not applicable in VDS only implementations)
• Compute
A cluster can contain up to 64 hosts, but this is not recommended in VIO for performance reasons such as scheduling and boot time.
10 - 12 hosts per cluster is recommended in production.
Create new clusters as and when compute is needed to support concurrent operations.
Networking
This is based on the following platforms.
• VDS
• NSX-V
• NSX-T
These are described further below.
Networking models
These are as follows.
• Single VDS: across all three cluster types of management, Edge and workload
• Shared VDS: between management & Edge with a dedicated workload VDS
• Dedicated VDS: for each cluster role
VLANs
These are as follows.
• Management: VIO, ESXi, NSX access
• API: Openstack user access for provisioning & monitoring
• External network: routing to and from the internal network, external IP addressing
• VXLAN transport: tenant networking
• vMotion: migrating VMs between clusters, non-VIO dependent
• Storage Access: accessing disk storage, non-VIO dependent
Models
These logical roles are mapped to the physical infrastructure underpinning the implementation. This mapping is as follows.
• Integrated model: one cluster for management, edge and compute
• Consolidated model: one cluster for management and edge, others for compute
• Dedicated model: separate cluster for each logical role
The dedicated model is recommended as it brings network and management capacity planning simplicity, fault domain isolation, simplified hardware LCM and better hardware management of items such as CPU, memory and network card resource as well as simplified upgrade and migration processes.
The odd number of management hosts allows for different services to run on each of the 3 hosts and the loss of a single host would not affect the control plane availability.
Only the dedicated model has standalone edge hosts and a minimum of 2 is recommended for resilience. Both the integrated and consolidated models place the edge host with the management host, of which there must be a minimum of 3 as discussed previously.
A recommended architecture is to have the following clusters.
• Management Cluster
• Compute Clusters => 1
VIO VM Types
The main VIO VM types are as follows.
• VMware Essential VM
o NSX Manager
o NSX Controller
o vCenter
o vRealize Operations Manager vROps
o vRealize Log Insight
• VIO Core Services VM: for VIO in full application HA deployment
VIO Licencing
There are two main VIO licence types that can be obtained as follows.
• DataCenter Edition (DC)
• Carrier Edition (CE)
CE is a super set of DC and can only be purchase as part of the vCloud NFV for OpenStack bundle.
Implementation Considerations
Merantis, RDO, Canonical can be used to implement OpenStack but VIO provides a clean path for VMWare customers. VIO must be provisioned directly to clusters and not ESXi hosts. This is so the estate can use DRS and vMotion natively. Further benefits include the maintenance mode feature and other resilience redundancy features.
The VM template is cloned to make all the control plane components.
The VMs used are as follows.
• 1 Management
• 2 VM Template
• 2 Load Balancer
• 2 Controller
• 2 RabbitMQ
• 2 Memcached
• 3 MariaDB using Galera cluster hence 3 nodes minimum
• Compute driver = number of clusters
• Object Storage
Most of these will have at least 2 VMs created automatically. 4 CPUs advised as minimum per control plane VM. VMWare NSX is not mandatory, but if you need security groups then NSX must be installed. The management server, like the compute driver VM, is protected by clustering and HA. It’s not necessary for it to be up and running 24x7 hence VMWare deciding that a single instance is suffice.
The management server is distributed as an OVA and once deployed it shows a plugin I the VMWare Web Client. The Horizon Heat, Nova, Cinder, Glance and Neutron modules will communicate with the SDDC components for instance.
Setup Pre-requisites
Carry out the following instructions to get ready for the deployment of a VIO virtual appliance on VMWare.
This appliance includes the OpenStack Management Server used to deploy and manage it. This will require the following information for the installed VIO vApp.
• Folder
• Datacentre
• Cluster
• Template
• Licence agreement
• Provisioning format
• Storage policy
• Destination network
• Network properties
• System credentials
• Log management
• Time management
Ensure that you have downloaded the VIO installation OVA from the vSphere website previously
The relevant download link is shown below at the time of writing this book.
https://customerconnect.vmware.com/en/downloads/info/slug/infrastructure_operations_man agement/vmware_integrated_openstack/7_2
Confirm DRS has been configured previously on the management cluster that will be hosting VIO.
VIO Cluster Preparation
A few things need to be done in order to prepare your management cluster for VIO installation. This includes setting the cluster DRS automation level to Manual as shown in the following screen print. Do this by right clicking on the relevant cluster icon and then clicking on the Settings menu option. The following window will appear and select Manual from the Automation Level dropdown list and then click the OK button to confirm the changes.
The following window will appear confirming DRS is turned on and set the Manual DRS Automation.
Also, download the relevant VIO OVA from the VMWare website. The list at the time of writing is shown below.
VIO Installation
Now to carry out the actual installation of VIO. Again, right click on the management cluster you want to install VIO on but this time select the Deploy OVF template option.
Select the folder to use and then click on Next.
Now select the compute resource to use and click Next.
This will pause for a few seconds similar to the following screen print.
Then it will show the currently configured details for review. The top half of the page is as follows.
This shows the lower half of the page. Confirm these are correct then click Next. If not, click the Back button, update the settings as required then click Next.
Select the tick box to confirm acceptance of the settings after reading them. Then click Next.
Select the storage to use and click Next.
Select the network to use and click Next.
Now you will be presented with a customisation template. You can add items such as static IP addressing or leave this blank if you want to use DHCP.
Scroll down to set the admin password and if this should have SSH access (by selecting from the dropdown box with True/False settings).
Complete the settings as required, noting the bottom half of this page below and configuring accordingly.
Also, type in a suitable NTP server URL to use in the box shown below and then click Next to confirm these customisations.
Now review the settings before clicking the Finish button.
This is the bottom half of the page. If all is correct click Finish to start installing the VIO.
This will now show the VIO vApp being built in the window to the left as shown below.
Select it to see its detailed information as shown below.
Expand it and note the vio-controller-image and vio-manager icons listed below it.
Select each of these to view their information.
The vio-controller-image is below.
The vio-manager is below.
Select the VIO again. In the window that appears, click on the Actions dropdown towards the right.
Then navigate its menu system by going to Power and then click Power On to start VIO.
The vio-manager will start to power up.
Wait for its status to change to Completed as shown below.
You can click on Remote Console or the screen shown to view the current VM instantiation’s progress.
View the VM in the Dashboard’s main window.
Wait for this to run to completion and that will confirm the setup. A production ready control plane can be implemented in under an hour.
A few post-installation notes are shown below.
Look at the controller list carefully and ensure that each server type e.g. with a 0 and a 1 is on a different physical server.
Ensure controller 0 and 1 not on same hypervisor turn on anti-affinity on the controllers to ensure the same virtual components are on different physical servers.
In the background this runs several Ansible playbooks in the background to implement the cluster.
The users will have the same experience as they would on normal Openstack. VIO is accepted and verified by DefCore ensuring that the experience and technology being used is the same. Standard Horizon will always show security groups, but these will not be used if NSX is not installed i.e if you are using VDS.
Nova Compute
This is the VIO component that is responsible for managing the VMs deployed in the compute cluster. It uses the following components.
• nova-api: end-user interface
• nova-compute: create/destroy VMs
• nova-scheduler: determines where compute runs
• nova-conductor: database proxy co-ordination
All the hosts are aggregated within each cluster and presented as a single hypervisor by the VMWare VCD Driver to the nova-scheduler.
The nova-scheduler assigns hypervisor compute by scheduler, then vCenter determines the ESXi host within the cluster using DRS.
The underlying infrastructure is abstracted into nova-compute, making VIO the only OpenStack distribution at the time of writing that implements full control and data plane separation. This means that only a single hypervisor needs to be upgraded to change OpenStack agents such as Cinder, Glance, Ceilometer, Nova and so on. This leads to a very low upgrade time which could be as brief as e0 minutes.
Using VIO
An example would be downloading a HEAT template and using it to implement network services.
You can use the VMWare Vagrant plugin to run a Vagrant file natively.
On the CLI
vagrant up –provider=openstack
This will now start provisioning the workload.
You can also use HEAT templates.
Run the following command.
heat stack-create multi-tier.yaml multi-tier
Further information can be found at the following links.
vmware.com/go/openstacklab
vmware.com/go/openstackgraining
Nova views each compute cluster as a single node. It then selects the suitable cluster to place the VM and DRS determines how best to place the VM in the cluster.
Cinder creates block volumes and carries out other actions using the VMDK environment. vCenter creates the volume and it initially belongs to a shadow VM. vCenter changes its parent for the volume from shadow VM to the actual VM when it is attached to a running VM.
Glance is a dedicated image storage feature for various image formats including the following.
•
•
In VIO, the images are automatically converted to VMDK or OVA formats during the import process. This VMDK image is copied from the OpenStack Image Service to the vSphere database when the relevant VM is booted. From this point on, the VM booting from that datastore will use the database-cached version of this image.
Openstack uses the NSX Neutron plugin to communicate with NSX or in the case of VDS deployments with vCenter. This plugin is open source and as such can be used with any OpenStack implementation.
Day 2 Operations in VIO
Of key importance is planning for Day 2 operations, such as providing more compute and storage. Maintenance is also another consideration.
It is worth considering storage tiering, so that storage is assigned based on priority and requirement.
Adding Compute
Go to the Manage tab > Nova Compute > Click on the green plus sign to show the Add Nova cluster window
Clusters and not individual hypervisors are utilized so the feature such as DRS and vMotion can be used immediately.
Follow the onscreen prompts and select the data storage that you want to use.
Restarting the Nova services will pop up with a message citing temporary interruption to the storage.
Updates
Clicking on the Updates tab is how you can add patches. Click on the green plus sign
Select the package from the dialogue to upload it
Click on the Apply button.
Another disruptive message will appear.
Click OK.
• ISO
OVA
QCOW2
•
•
Raw
VHD
• VMDK
This will now show the patch progress onscreen.
If necessary, there is a Revert link on screen that can be deleted
Run ‘viopatch list’ from the command line to verify the patch installation as well if necessary.
Maintenance Mode
This puts hosts in a non-active role so that machines can be migrated to other hosts. Browse to the required cluster, right click and select Maintenance Mode The instances will be moved automatically by vMotion to another server transparently to the users. This can be useful for upgrade operations i.e. patch updates for security or for migrating hosts to higher versions. VUM can also be used to dynamically move workloads to and from hosts whilst it updates the hosts in the background.
Storage Tiering
This is the process of tying certain actions to certain storage e.g. putting higher performance process on virtual SAN with SSD, middle on SAN/NAS and low on NFS servers. The admin can tag each of the type using Cinder and its VMDK driver to tag these e.g. as Gold, Silver, Bronze and any VMs found with these tags will be linked to the relevant storage. Developers can use the storage policies to create specific volume types and tie these to the underlying storage policy such as Gold, Silver or Bronze. From then on, each VM requires a volume type setting so it can be associated with the correct policy.
VMWare NFVI
VMWare has its own VNFI offering corresponding to the ETSI MANO framework. This is summarised as follows.
• vSphere Director
• vCenter Server Appliance
• NSX Manager
VIM Director or the key component at this layer and controls the vCenter Server Appliance and the NSX Manager
VMWare Ready for NFV
This is VMWare’s NFV accreditation program giving vendors the ability to submit VNFs for addition to the permitted list of virtual network functions in VSX (VMWare Solution Exchange Marketplace).COver 2500 applications have been added to VSX so far using its self-service functionality. Virtual appliances, vApps and specific content extensions to VMWare management products can be added.
Technology Alliance Partner (TAP)
This is a hardware or software vendor that is in partnership with VMWare for the provision of programs and solutions that can work with its own products. This could be across its SDN, End User
or Hybrid Cloud solutions and includes the testing, integration of packaging of infrastructure, software or hardware products with those of VMWare.
vRealize
This is the next stage in the journey and vRealize give an out of the box insight into health, risk and efficiency of Openstack object. The admin can track workloads from the OpenStack layer down to the infrastructure and hardware layers. It has tenant dashboard and object relationship graphs. It gives out of the box performance and capacity reports. Openstack shows a lot of logs for errant events e.g. 10 – 15 logs entries could be generated for a VM not spinning up correctly or not getting an IP address.
It consists of a suite of management applications covering various areas as shown below.
• vRealize Automation
• vRealize Operations
• vRealize Log Insights
• vRealize Business for Cloud
• vRealize Network Insights
These are described further below.
vRealize Automation
This is a virtual cloud product for the personalisation and automatic configuration of infrastructure, application, and container management.
vRealize Operations
This carries out operational and management actions across physical, virtual and cloud environments. Such as Hyper V or AWS as well as vSphere. vRealize Operations Manager is useful for the proactive remediation of capacity and performance issues. It covers the monitoring of various services such as databases, RabbitMQ, HAProxy and more.
vRealize Log Insight
This provides logging functionality and daily troubleshooting management functions. vRealize Log Insight is a monitoring and logging tool that can analysed both structured and unstructured time series data and configuration information. It can ingest any type of log data and the automatically categorise and identify data structures and relationships. It creates indexes out of this information capable of operating in high performance scenarios for fast data retrieval. There is no need for traditional ETL actions and it can ingest huge volumes of information, terabytes of information in a daily basis. It is intrinsically tied to Openstack and can be put to immediate use on installation, with standard, out of the box features ready to go.
vRealize Business for Cloud
This covers the analysis and tracking of resource and workload cost consumption. The suite itself has the following Editions.
• Standard: Operations for SDN DC and hybrid cloud management, Log Insight and standard Business for Cloud version
• Advanced: Standard, Upgraded Business for Cloud and Automation for infrastructure
• Enterprise: Automation for applications and Operations for application monitoring
vRealize Network Insight
vRealize Network Insight carries out network analysis using NSX and is capable of analysing port and common services information. It can automatically carry out security group to enable micro-segmentation at the application level.
Applying vRealize
vRealize can be used for chargebacks by showing information such as per-tenant costs. This such as CAPEX and OPEX costs, allocations and consumption patterns can be determined. You could predictively use it to determine costs on both on-premises and cloud workloads. This will show a colour coded breakdown of each aspect such as licencing maintenance storage and so on. The calculations are done base don best practice analysis of costs.
You can delve down to cost per component being use by and instance and can compare this to private cloud vs public cloud usage. This is useful for comparing cost in line with SLA so if a business wants to meet a specific SLA then they will know what cost is associated with that.
With vRealize you can look at the underlying storage of a cluster and its infrastructure. The NSX components such as router and switch can also be reviewed as well as the compute clusters as well. It has heat zone as well to indicate where there are issues using colour coded status icons. It works well with VMWare Log Insight which could then be reviewed to look at the detailed logs, like an ELK stack view were detail can be looked at for the specific log item.
Hovering over icons in vRealize shows each of the MV status suc has datastore. Clusters can be viewed and the hypervisors that make up that cluster can be viewed. The type of OS, management services, IP address and so on can be viewed in this view.
Cloud Assembly
This is where the management of the vRealize Automation is organised. Cloud templates can be created here, infrastructure, users and other configuration can be created here. Cloud zones can also be added so that Azure, AWS and Google Cloud workloads can be built directly from it. Workflow and action-based extensibility can be managed in this section as well.
Configure > Cloud Zones > (view the clouds and carry out relevant tasks on them)
At the top are the following tabs.
• Deployments
• Design
• Infrastrcuture
• Extensibility
Catalog items can be built using the following menu.
Design > Cloud Templates > Blank Canvas
This will show the blank canvas window. Add a name description, project and share with project or not, service broker for appliva permissions.
There are lots of objects on the left that can be dragged to the canvas. These can create items as varied as VMs, Kubernetes clusters, IAM roles, traffic management, Tower and Puppet integrations, Microsoft DNS amongst many other items. The canvas can be added with objects directly or YAML can be added on the right (which has Code, Properties and Inputs tabs.
Note that initially the object might not allow configuration as it needs some constraints configured for it. This can be done in the panel on the right-hand side of the screen.
This can be used to limit the settings of the resource being added. Note that the relevant constraints are added in the YAML view, such as the vSphere network and other items.
The main view looks like this after the addition of the constraints.
The resource can be tested before deployed or it can be immediately deployed. Take note of the DEPLOY, TEST, VERSION and CLOSE buttons at the bottom left of the screen. These can be used in carrying out various actions on the catalog item. Click on the version button and the following window will be shown. Add the relevant information and then the CREATE button. This will return to the previous screen.
Toggle the DEV mode switch.
In the object’s Deployments tab, click on the ACTIONS link to review additional actions that can be taken on the object.
In this case we will clean up the deployment as this is not needed anymore.
View the version history of the deployment by clicking on the VERSION link.
Click on the DELETE option then in the below dialogue click on the SUBMIT button to remove this deployment.
It is possible to view a differential between deployments as shown below.
Select the version on the left that needs checking. The using the diff against dropdown select the version it will be checked for differences with onscreen.
In this particular example, click on the relevant version it will diff against such as 1 and the difference between it and version 2 will be highlighted as shown below.
There is also a DIFF VISUALLY / DIFF CODE toggle button on the right-hand side of the screen. Click on it and view output like the following window.
vSphere with Tanzu
vSphere with Tanzu is a suite of products that has evolved out of several VMware acquisitions. At the time of writing this suite of products can be summarised as follows.
Apply cloud native application development frameworks.
• Sprint Runtime
Assemble and containerise applications.
• Tanzu Application Catalogue
• Tanzu Build Service
Deliver software to secured platforms.
• Tanzu Application Service
Operate Kubernetes across cloud environments
• Tanzu Kubernetes Grid: run across all clouds and edge
• vSphere7 with Kubernetes: run Kubernetes in vSphere
Unify and optimise multi-cluster operations
• Tanzu Mission Control: centralised control of clusters anywhere
Observe enterprise applications and infrastructure.
• Tanzu Observability by Wavefront
VMware Cloud Foundation Licencing
This is the required licencing for combined VMware vSphere, VMware NSX-T datacenter and workload platform and Tanzu Kubernetes Grid (TKG). This deploys 3 VMs by default and deploys management components on them. This also deploys a spherelet enable the running of vSphere pods on hosts. In vSphere client there is a workload management menu showing how to deploy a supervisor or management cluster using vSphere with Kubernetes. This basically converts a vSphere cluster into a Kubernetes cluster. Then Kubernetes type declarative YAML files can be used to manage its components You can then use vSphere credentials to issue commands to the cluster as shown below.
kubectl vsphere login –server supervistor.gridlockaz.local –vsphere-username admin@gridserver.local
This command is using the vSphere plugin to authenticate an SSO user that can then carry out kubectl commands on the Kubernetes cluster.
The following command can then be used, for example, to build the cluster declarations that would be set declaratively in the cluster.yml file used in the example.
kubectl apply -f cluster.yml
TKG Standalone Components
TKG has the following components as part of its service.
• Calico is the container network interface CNI
• Ingress via Contour and fluentbit observability
• VMC is the infra platform
• Harbor (TKG’s inbuilt Docker registry)
• Sonobuoy (conformance testing)
• Velero (backup and recovery)
• Dex/Gangway (authentication)
• Choice of observability tool
TKG Deployment Methods
These can be summarised as follows.
• TKG + VMware vSphere
• TKG + AWS deployed on EC2 instances NOT EKS
• TKG Plus + VMware Cloud on AWS
Deploying TKG Standalone
The deployment requires the installation of the TKG package on a local machine that is referred to as the bootstrap machine. On installation the TKG package installs a small temporary Kubernetes in Docker (kind) management cluster that is then used to install TKG on the target platform that is set during one of the installation’s setup steps. At the time of writing, the latest version of the TKG installer can be downloaded from the following URL.
https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.1/vmware-tanzu-kubernetesgrid-11/GUID-install-tkg-set-up-tkg.html
Note that on completion, this deployment doesn’t need NSX-T thus making for a faster deployment. The installation steps are as follows.
Before downloading the TKG binary, ensure that the following components are installed on the bootstrap machine. The installation of these components is outside of this book’s scope, but there are lots of resources describing their installation online.
• kubectl - https://v1-17.docs.kubernetes.io/docs/tasks/tools/install-kubectl/
• Docker - https://docs.docker.com/install/
• Docker Desktop - https://www.docker.com/products/docker-desktop
Also, ensure that the system time is synchronised to an online NTP server like 0.pool.ntp.org.
If you have previously installed Cluster API (https://cluster-api.sigs.k8s.io/) on the bootstrap device, its configuration must be deleted. Do this by running the following command.
rm ~/.cluster-api
Download and unpack the tarball for your bootstrap machine’s OS (either Windows, Linux or MacOS) from the aforementioned URL. Do this by logging in to the following URL.
https://www.vmware.com/go/get-tkg
Then go to Downloads
Find the VMware Tanzu Kubernetes Grid 1.1.3 CLI options.
Click Download Now for the relevant OS.
When the download completes, unzip the tarball.
An example of unzipping this tarball using gunzip on a Mac is shown below.
gunzip tkg-darwin-amd64-v1.1.3-vmware.1.gz
Note that the downloaded executable for Mac in the above example would be as follows ./tkg-linux-amd64-v1.1.3-vmware.1.
Other operating systems would have similar names with the only difference being the OS reference in the file name.
From the same folder, copy the untarred binary to the /usr/local/bin folder and rename it to tkg.
cp ./tkg-darwin-amd64-v1.1.3-vmware.1 /usr/local/bin/tkg
Give the file execute permissions.
chmod +x /usr/local/bin/tkg
Confirm the tkg version as follows.
tkg version
Mac devices might require a security exception is made for the executable. Do this by going to System Permissions and setting them accordingly.
Alternatively, search for the application in Finder, control-click the application and then select Open.
Confirm that the required vSphere Account permissions are configured.
Import the Base OS image template and the API Server Load Balancer (these are both OVA files) that will be used by the Kubernetes nodes into vSphere.
The following are the latest images for these at the time of writing.
• OS: Photon v3 Kubernetes 1.18.6 OVA
• Load Balancer: Photon v3 capv haproxy v1.2.4 OVA
Carry out the following steps for both images.
• Download the image by logging into https://www.vmware.com/go/get-tkg with your VM credentials, then locate and download the file.
• Then, log into vSphere Client, right click on the required vCenter Server and select Deploy OVF template.
• Select Local file, follow the dialogue to upload the OVA file to the server, following the installer prompts to deploy a VM from an OVA image (this entails selecting the appliance name, destination datacenter and/or folder, host, cluster, resource, EULA, disk format, datastore and network).
• Click Finish to deploy the VM.
• Then, right click on the created VM (if powered on power it off), select Template > Convert to Template.
• Right click on the new template in VMs and Templates, select Add Permission and assign the tkg-user to the template with the TKG role.
You are now ready to deploy the TKG management cluster to vSphere.
This can be done a number of ways as follows.
• Installer Interface Deployment
• CLI Deployment
• Internet Restricted Deployment
• Amazon EC2 Deployment
For brevity we will only look at the Installer Interface and CLI deployments in this book.
vSphere 7.0 with Kubernetes and TKG
Note that an existing vSphere 7.0 with Kubernetes installation will not allow a TKG management cluster to be installed and therefore the below installations will quit. If so, connect the TKG CLI to the vSphere 7.0 installation with Kubernetes Supervisor Cluster.
You can do this with the following command.
kubectl vsphere login vsphere-username tkg-user@vsphere.local server=control_Plane_VIP insecure-skip-tls-verify=true
Type in the vSphere Administrator password when prompted.
From the list of contexts that appear onscreen, select the Supervisor Cluster context name and add it in the following command that should now be launched.
kubectl config use-context <Supervisor_Cluster_context>
Run the following command.
tkg get management-cluster
A list like the following will appear.
Set the TKG CLI context to the Supervisor Cluster’s IP address as follows.
tkg set management-cluster <Supervisor_Cluster_IP>
Show the storage class information.
kubectl get storageclasses
Set variables defining the storage and VM classes and the service domain to which the cluster has been created. This can be done as follows.
export CONTROL_PLANE_STORAGE_CLASS=<storage_class_name>
export WORKER_STORAGE_CLASS=<storage_class_name>
DEFAULT_STORAGE_CLASS=<storage_class_name>
Note that the default storage class can be set to nothing as shown below.
DEFAULT_STORAGE_CLASS=""
Get list of Kubernetes versions available in the Supervisor Cluster.
kubectl get virtualmachineimages
Get list of Supervisor Cluster namespaces.
kubectl get namespaces
Create the Kubernetes cluster.
tkg create cluster my-vsphere7-cluster plan=dev namespace=<namespace> kubernetesversion=v1.16.8+vmware.1-tkg.3.60d2ffd
That completes the TKG to Supervisor Cluster setup.
If vSphere with Kubernetes is not installed on vSphere 7.0, the installation will state that TKG management cluster installation is not recommended (as the preference is installing vSphere with Kubernetes with its inbuild Supervisor Cluster).
Installer Interface Deployment
Start the TKG tool to start the Kubernetes management cluster setup.
tkg init –ui
This shows a web UI asking for landing zone information such as is the destination platform for the installation e.g.vSphere, AWS, vCencter information and so on.
On completing this information, the TKG tool reaches out to the target platform and builds out the resources as the TKG management cluster with the required number of VMs that will act as cluster nodes. At this stage though, the production cluster has not been built.
CLI Deployment
Alternatively, carry out the following steps.
Run the following command to start the cluster creation using the CLI.
tkg get management-cluster
this will create the ~/.tkg folder which will contain the following files.
~/.tkg/config.yaml ~/.tkg/bom/bom-*.yaml files
Edit the .tkg/config.yaml file, which contains the following fields.
VSPHERE_SERVER:
VSPHERE_USERNAME:
VSPHERE_PASSWORD:
VSPHERE_DATACENTER:
VSPHERE_DATASTORE:
VSPHERE_NETWORK:
VSPHERE_RESOURCE_POOL:
VSPHERE_FOLDER:
VSPHERE_TEMPLATE:
VSPHERE_HAPROXY_TEMPLATE:
VSPHERE_SSH_AUTHORIZED_KEY:
SERVICE_CIDR:
CLUSTER_CIDR:
VSPHERE_WORKER_DISK_GIB:
VSPHERE_WORKER_NUM_CPUS:
VSPHERE_WORKER_MEM_MIB:
VSPHERE_CONTROL_PLANE_DISK_GIB:
VSPHERE_CONTROL_PLANE_NUM_CPUS:
VSPHERE_CONTROL_PLANE_MEM_MIB:
VSPHERE_HA_PROXY_DISK_GIB:
VSPHERE_HA_PROXY_NUM_CPUS:
VSPHERE_HA_PROXY_MEM_MIB:
Note that any conflicting environment variable settings will override these configuration file settings.
If necessary, visit the following links for an explanation of each keyword and its options.
https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.1/vmware-tanzu-kubernetesgrid-11/GUID-install-tkg-vsphere-cli.html
Save the configuration file.
Run the following command
tkg init infrastructure vsphere
This completes the setup.
The tkg init command can be launched with a variety of options, but the above command suffices for the example’s requirements of installing a basic management cluster.
tkg get credentials tkg-mgmt-cluster
You can then create the cluster with the following command
tkg create cluster gridcluster –plan=dev
Installing Kubernetes CLI Tools
Check the vSphere site for the tools download page or from vSphere Cluster > Namespaces > Summary tab Status area > Link to CLI Tools, copy the link.
Browse to the link, select your bootstrap machine’s OS.
Download the vsphere-plugin.zip from the link.
Extract its contents to a working folder. Copy its kubectl and kubectl-vsphere files to a final location if not the current folder.
Add this location to the system’s PATH variable.
Confirm both tools work by running the following commands.
kubectl
kubectl-vsphere
TKG Tool Command Examples
A few useful commands and options are shown below.
Using the CLI option config to use a specific cluster config would be as follows.
tkg init ui config /path/my-cluster-config.yaml
Using the CLI option help with any command would be as follows.
tkg create cluster –help
Using the CLI option kubeconfig to use a specific Kubernetes managerment cluster config would be as follows.
tkg init ui kubeconfig /path/my-cluster-kubeconfig.yaml
Using the CLI option log_file log command output to a file would be as follows.
tkg init –ui –log_file=/var/currentlog.txt
Using the CLI option quiet don’t show command output would be as follows.
tkg init –ui –quiet
Using the CLI option v show verbose logging would be as follows.
tkg init –ui v
Tanzu Mission Control
This is a policy management and security management tool that can be used across all of your cluster infrastructure
Can provision TKG on AWS and soon on vSphere 7. This means it can give full LCM with scaling and upgrading from TMC as well. This will enable full policy, security and LCM in TMC.
VM and Container Hybrid Setup
VMs and containers can be combined into the same names spaces in vSphere with Tanzu. The focus is more on the application so the perception of what the application resides on is agnostic. TKG consolidates the infrastructure. This also gives consistent operations for Admin teams, as they have a single pane of glass with an interface they are familiar with already.
Policies can them the set up for the following. • Backup
•
• Network
• Storage
• Security
• Compliance
This now gives programmers and developers access to namespaces without needing administrator permissions. This gives granular control to them with finer grained controls and self-service. Admins can then maintain visibility, control, governance costs for developers. But developers have speed and flexibility using either vCenter, APIs or tools and interfaces they are already familiar. These controls can go as far as determining which IP addresses and endpoints can be reached by the developers, package management, backup and so on. Configurations can be done using the GUI, CLI, API or YAML saved configurations.
DR
•
RBAC
VMware Hybrid Cloud Extension (HCX)
HCX is a vSphere-hosted application with the main function of providing VM mobility, the migration of applications, rebalancing workloads and implementation of DR functionality across data centers and cloud environments. A multi-site service mesh can be configured using HCX to provide optimal resilience across these sites. Network extension of segments with IP and MAC retention on VMs that have been migrated to new sites is also possible.
HCX also has asynchronous replication and VM recovery. It also integrates with VMware’s Site Recovery Manager and its various tools.
HCX bridges on-premises and cloud environments. For instance, it enables workloads to be moved between on-premises and cloud vSphere infrastructure such as VMware Cloud on AWS (VMC). It can be used for disaster recovery and data center evacuation. HCX has a migration type know as Cloud Motion with vSphere Replication.
vSphere compatibility between the on-premises and cloud vSphere implementations is key. A few other considerations are shown below.
• Workload dependencies
• vSphere SSO domains independent of enhanced linked mode
• Hardware age and compatibility
• Compute and storage availability
• SLAs and downtime required
• Mobility options in line with SLAs i.e. the amount of downtime acceptable if any
• Maintenance of IP and MAC information per VM, UUIDs, certificates and so on
There could be more considerations depending on your environment.
HCX takes mobility and VM migration way further than standard vMotion. For instance, HCX vMotion does not require direct ESXi HOST connectivity in either direction unlike standard vMotion. All HCX vMotion traffic is managed by the HCX vMotion proxy present at each location. This Proxy is like an ESXi host in the vCenter server inventory and is deployed at the datacenter level by default with no interaction required.
HCX supports vSphere 5.0 +. On-premises to on-premises migrations require an NSX Hybrid Connect licence for each HCX site pairing. On-premises to cloud migration does not require this licence, showing a clear indication of VMware’s to support cloud-based vSphere implementations. HCX is a default addition for VMC and GCVE (Google Cloud VMware Engine) deployments and this addition is fully automated.
HCX Features include WAN optimisation, deduplication and compression for traffic transfer efficiency. HCX requires a minimum of 100Mbps connectivity. It can be used across Internet, Interconnect or Direct Connect. It uses suite B encryption. Migrated on-premises workloads must be on a Virtual Distributed Switch (VDS). L2 stretch can be used to extend on-premises networks using a single click. This can be to other on-premises networks, VMC or GCVE. On the completion of the migration, there is also an option to migrate the extended network.
Both network and IP space can be extended. HCX is delivered as a service using a small OVA plugin that can be installed directly into vSphere. On-premises and cloud networks can be combined into a single organisational network for the efficient management of workloads in line with any security or other relevant requirements.
Other built-in functions include a native scheduler, per-VM EVC, MAC address retention, snapshot removal, force unmount of ISO images, bi-directional migration support, VM Tools upgrade and hardware compatibility.
HCX has 2 installation types as follows.
• HCX Connector
• HCX Cloud
These are described further below.
HCX Connector
This is used on the source vCenter site i.e. containing the VMs to be migrated. As this is onpremises, obtain a download link from a deployed HCX Cloud solution or use the Download Link API to get a download link for the latest HCX Connector build. It does not require NSX or NSX-T.
HCX Cloud
This is used on the destination vCenter site i.e. the target of the site pairing requires, network extensions and VM migrations. It can also be the source of a site-pair in cloud-to-cloud migrations. In public cloud deployments, HCX Cloud is automatically installed when the service is enabled. In private cloud installations, get the installer from the following link.
https://downloads.vmware.com
Alternatively, use the Download Link API to get a download link for the latest HCX Cloud build.
Unlike HCX Connector, HCX Cloud needs NSXv or NSX-T. Other differences are that it can be an endpoint for Site Pairing, requires a generally available version of vSphere and does not support legacy vSphere
Typically, the following should be applied.
• HCX Connector is on the source environment
• HCX Cloud is on the destination environment
• HCX Connector cannot be the HCX Site Pairing target
• HCX Connector cannot pair with another HCX Connector
The migration types can be summarised as follows.
HCX Connector > HCX Cloud
HCX Cloud > HCX Cloud
HCX Hybridity
This refers to a network state in which both on-premises and cloud services are not just connected but operating in tandem. It has 2 primary component services as follows
• HCX Interconnect Service
• Network Extension Service
HCX Interconnect Service
This uses Google’s Internet connective to provide Internet-based resilience. It provides strong encryption, traffic engineering and datacenter extension services. Secure pairing of both site and management of HCX components can be achieved with this.
Network Extension Service
This is a high throughput service with integrated Proximity Routing that provides mobility between sites as well as straightforward DR plans.
These solutions give teams the ability to test out new architectures seamlessly as well as managed mixed workloads of VMs. It also takes advantage of Zero-Downtime migrations as well as removing the need for Application Dependency Mapping (ADM). Typically, ADM is needed for datacenter consolidations as production apps would be in the legacy datacenters being shut down. This stops being an issue with HCX. Additionally, strategic DR solutions can be implemented between on-premises and cloud environments across Wide Area Networks (WANs).
VMware Cloud on AWS
This was the only VMware cloud offering that is directly supported by VMware when the current version of this book was being written. Items such as upgrade paths between various components are looked after by VMware. This has a 99.9% single cluster of 3 nodes, 99.99% for stretch cluster of 6 or more nodes. Typical use cases for it are as follows.
• Cloud migrations: avoids need for re-architecture
• Data centre extension: expand as a low cost
• Disaster recovery: combine VMware and AWS DR functionality
• Application modernisation: use AWS services with current platforms
Hybrid link mode connects both on-premises and cloud environment. The vCenter server can be used as a single pane of glass for both environments. Also, a high throughput, low latency private access is made from the cloud NSX to AWS services, so you can benefit directly from using Amazon EC2, S3, RDS, Redshift, Load Balancer, FSx and other services.
A VMware account and an AWS account must be provided. The SDDC is operated on the VMware Cloud account. The customer won’t have access to this account as it is managed on their behalf by VMware. Each customer has a dedicated VPC created and managed by VMware. This account is owned, operated and paid for by VMware to AWS.
The customer owned AWS account is owned, operated and paid for by the customer. There is private connectivity to VMware Cloud SDDC and this has full access to AWS services.
Note that the VMware VPC is a separate entity from the customer VPC.
The VMware Cloud on AWS can be access using the following methods
• VMware Cloud on AWS portal
• vSphere Client
• AWS Management Console
These are described further below.
VMware Cloud on AWS portal
This consists of the following items.
• ESXi host, console user and role management
• Firewall and NAT administration
• VPC connectivity and logical network configuration
• AWS Direct Connect management and configuration
vSphere Client
This provides the following functionality
• Hybrid Lined Mode (HLM)
• VM administration and storage policies
AWS Management Console
• Amazon VPC configuration
• Network and security configuration
SDDC Components
These can be summarised as follows.
• vCenter server
• VMC on AWS SDDC containing ESXi
• vSphere / vSAN /NSX
ESXi Component
The current standard EC2 VM used is the i3.metal flavour which has the following footprint.
• 36 cores
• 512 RAM
• 25GB
• 2-16 host cluster
• Privileged account access not SSH or root
• No VIB needed for installation
ROBO, DIR Pilot Light and VDI
Limitations
The AWS VMware offering has the following limitations.
• Cannot scale from 3 to 2 host, but can go from 2 hosts to higher
• Only Default storage EDRS policy
Storage Component
This is as follows for the VM flavours used.
i3.metal
This is as follows.
• Directly attached instance store-based VSAN encryption
• NVMe flash-based (cached and capacity) encryption
• You cannot access the management VM datastore.
i3en.metal
This is as follows.
• This has NVMe drives with more storage capacity
Other Options
If you don’t want to use either of the above storage options, other storage options are as follows.
• Certified MSP storage: Rackspace and Factor provide NFS datastores
• Storage application, virtual appliance on EC2
• AWS Native FSx, S3 and Efs options
Stretched Clusters
A stretched cluster can be used to implement the following features.
• Deploy a cloud across availability zones (AZs)
• Protect SDDC from AZ issues
• Span applications across multiple AZs
This ensures that workload resource can be stretched across AZ infrastructure for resilience.
Tools
The following are tools useful in AWS VMware planning and deployment. For brevity, only brief introductions are shown below but visit the URLs shown for further details on these.
RVTools
This is a windows tool that uses vSphere Management SDK and CIS REST API calls to display virtual environment information. Further information can be found at this link.
https://www.robware.net/rvtools
This shows point in time information and requires access to vCenter and a Windows PC. It can have limited performance data and view of environment. It has a number of tabs such as vInfo, vCPU, vMemorv, Disk, vPartition, vNetwork, vFloppy, vCD, vSnapshot and vTools listing information relevant to each of these areas in the vSphere environment.
Live Optics
This is a free Dell EMC workload observation application that can be used to automate the collection of data from a vSphere environment.
https://www.liveoptics.com
This shows performance information over a duration in the vSphere environment with no need for an agent. Areas covered are shown below.
• Performance Data
• vCenter and other data sources
• Requirements
• Windows or Linux
• Connectivity and Credentials
• Constraints
• Data shared with third parties
VMware Cloud Sizer
This is an online form-based tool that can be used to determine AWS VMware Cloud sizing depending on configuration requirements. It can be found at the following link.
https://vmc.vmware.com/sizer
Datacenter Migrations
In this section, we discuss various concepts relevant to datacenter migration that could be relevant to AWS and other types of cloud or on-premises implementations. These migrations must be carefully planned in accordance with the business and technical requirements of the organisation and the applications it needs to run. Each migration is carried out to meet several aims in accordance with different types of migrations. These aims and types are listed in the following sections. Note that a datacenter migration might be required to meet a number of these aims in tandem and possibly need to carry out a mixture of the migration types shown to achieve them.
Migration Aims
These are as follows.
• Consolidation: the combination of current resources into a homogenous environment
• Replacement: the updating of workloads to a new infrastructure whilst decommissioning the old setup
• Extension: this is the addition of resources capable of dealing with additional workloads to the current infrastructure
Migration Types
This describes the way each of the migrations is carried out and can be categorised as shown below.
• Cold
• Warm
• Zero-Downtime
These are described further below.
Cold
In this type of migration, the VMs to move are powered off in the source datacenter, moved then powered on again in the destination datacenter.
Warm
This requires that a snapshot of the VMs is carried out to new location while they are powered on, so the same VM exists in both locations. Both VMs are kept in sync. Then the new VM is powered on in its last known good state when the source site is down.
Zero-Downtime
This is the migration of VMs from source to destination datacenter while powered on. Data loss is not acceptable and for Mission Critical Applications.
vCloud Director
vCloud Director is a means of administering different vSphere Clusters and servers in a single pane of glass. It enables the harmonisation and uniform administration of resource pools at a higher level that can be assigned to various tenants for numerous workloads. Users can navigate to vCloud Director and assign various tenants or customers with access to the underlying resources, without them needing to be aware of the physical or virtual assignments of these resources. Clusters/resource pools can be assigned to a vCloud Director. The vCloud Director can then assign these resources to VDCs. It can go into vSphere and build the resource pools. Further information can be found at this URL.
https://docs.vmware.com/en/VMware-Cloud-Director/10.2/vcd_102_install.pdf
vCloud Director is vSphere’s automation platform for carrying out the implementation of workloads on physical, virtual, containerised, and other types of infrastructure. It provides a simple to use GUI from which deployment engineers can create blueprints that can be used by others for deployments, implement those deployments using established or previously stored blueprints, as well as carry out other types of administrative functions. It can connect to one or more vCenter servers as well as NSX-T environments. This means it will be able to manage and administer ESXi virtual machines and their corresponding workloads and features.
You can install a vCloud Director server group on Linux and configure it to use an external database. vCloud Director database high availability can be implemented by deploying 2 instances of vCloud Director as standby cells in the same server group. These cells are created during the installation and configuration of vCloud Director and are connected to the shared database and transfer service storage. The System Administrator account is also created and this can be used to connect to the vCenter servers, the ESXi hosts, and the NSX Manager or NSX-T Manager
Each vCloud Director needs two different SSL endpoints. These are for the HTTP service and the console proxy service. The same IP address can be used with two different ports or two IP addresses can be used. Do not use the Linux IP address add command to create the second address. By default, vCloud Director uses the eth0 IP address with the custom port 8443 for the console proxy service. Do not put the console proxy IP address behind an SSL terminating load balancer or reverse proxy. All requests to it must be related directly to its console proxy IP address.
vCloud Director servers must use a time service such as NTP. The maximum drift between these servers must be 2 seconds or less. All vCloud Director servers must be in the same time zone. All vCloud Director must be resolvable by DNS using forward and reverse to look up of their fully qualified domain names or of their unqualified hostnames. These hostnames must be resolved table using tools such as nslookup. They must also have IP address is resolved below using reverse DNS lookup.
It is strongly advisable that vCloud Director is not directly reachable from the Internet. Only allow the following ports.
• 22 SSH • 443 HTTPS
Installation Prep
The following must be in place before installing vCloud Director
• PostgreSQL database
• Transfer service storage
• RabbitMQ broker
• VMWare public key
• NSX data centre for vSphere for vCloud Director
• NSX-T datacenter for vCloud Director
PostgreSQL Database
Create a dedicated PostgreSQL database on your platform of choice. PostgreSQL can be connected to from vCloud Director using SSL.
The following specifications are recommended for this database.
• 4 CPUs
• UTF 8 server encoding
• 100 gigabytes of storage
• 16 gigabytes of memory
Run the following commands.
initdb locale=EN_US.utf-8
Create the database user with the following command
create user vcloud;
Create the database instance and give it an owner.
create database vcloud owner vcloud;
Give the database owner account a password.
alter user vcloud password newpass
Give the database owner permissions to log into the database.
alter roll vcloud with login;
Setting Up the Transfer Service Storage
An NFS or other shared storage volume must be created for temporary upload, download, and catalogue items storage that can be published or subscribed externally.
A vCloud Director in the vCloud Director server group must mountain uses NFS based transfer service storage. User and group permissions must be specified for each cell to mount the NFS based location for use as the transfer service storage. Note that space on this volume is consumed as follows
• Upload and download usage during transfers: These updates and downloads are removed from the storage when the transfer finishes. Any transfers that have not progressed after 60 minutes are marked as expired and cleaned up by the system. It is recommended that a few 100 gigabytes are reserved for this use, as transferred images can be large.
• Externally published catalogue items: This storage is not occupied by items from catalogues published externally but that do not enable caching. If organisations in the cloud are enabled to create catalogues published externally, this could consume thousands of catalogue items requiring space on this volume. Each catalogue item can be the size of a virtual machine in its compressed OVF form.
NFS Server Configuration Requirements
For vCloud Director to be able to write files to an NFS based transfer service storage location and read files from it, certain requirements must be met. These are as follows.
Each vCloud Director server group member must have access to the shared location identified in the NFS export list. This is so the cloud user can write and read files to and from this shared location. The NFS server must give the root system accounts on each server in the vCloud Director server group read write access to this shared location. This is so that logs can be collected from the cells in the same bundle using the VMware VCD support script. This script has multi cell options and the requirement can be met by using the know_root_squash in the NFS export configuration for the shared location. Let’s assume there is a folder set aside as transfer space. This folder must have ownership permissions for the root user & group with chmod settings of 750. Then add the following lines to the /etc/exports file.
/nfs/vCDspace vCD_Cell1_IP_Address(rw,sync,no_subtree_check,no_root_squash)
/nfs/vCDspace vCD_Cell2_IP_Address(rw,sync,no_subtree_check,no_root_squash)
/nfs/vCDspace vCD_Cell3_IP_Address(rw,sync,no_subtree_check,no_root_squash)
Do not leave any spaces between the cell IP address and the left parenthesis in the export line. Each vCloud Director server group member must have a corresponding entry in the NFS server /etc/exports file so they can all mount this NFS share. Run the following commands after changing the /etc/exports file so that all NFS shares are exported again.
exportfs -a
vCloud Director Installation
The vCloud Director can be installed in several ways. Note that it has its own embedded PostgreSQL database. This database includes the replication manager tool suites which provides high availability functionality to the PostgreSQL cluster. A vCloud Director appliance can be deployed as a primary cell, standby cell, or a vCloud Director application cell.
A HA server group can be deployed by creating one primary and two standby vCloud Director instances. These can be horizontally scaled by adding application cells to the current cluster. Each appliance i.e. both the primary and standby instances of the vCloud Director appliance will have its own API and user interface, replication manager and PostgreSQL standby instance or primary instance depending on the application cell type. This PostgreSQL instance will have its information synchronised between the application cells.
The following steps summarised the installation.
• Deploy the primary vCloud Director appliance, which will be the first instance of the server group. It will have an embedded database as the vCloud Director database and this database name is the cloud and the database user is also called the cloud.
• Confirm that the primary cell is running, which can be done by browsing to its URL using its primary eith 0 IP address as the URL end point.
• Confirm the PostgreSQL database health by logging by logging in as routes to the appliance management interface at the URL with a port of 5480 using https. Now that both the primary cell and the PostgreSQL database have been confirmed, deploy 2 instances of the vCloud Director standby cells. These will have embedded databases configured in replication mode with the primary database.
• Confirmed that all cells in the HA cluster are now running.
• As an optional step, deploy one or more instances of vCloud Director as application cells. Note that application cells do not have embedded databases. The application cell will connect to the primary database.
So, the key difference between the standby and application cells is that the application cells have PostgreSQL turned off and do not have any replication between these components and the primary PostgreSQL database. The standby cells do you have PostgreSQL databases which are synchronised with the primary cells PostgreSQL database.
It is possible to deploy vCloud Director with one primary cell and no standby or application cells but this is not recommended in a production environment. This is known as a single cell deployment. They do not receive supports for performance or stability related issues. These steps are summarised below.
• Deploy the vCloud Director primary cell.
• This will have an end embedded database with the name of the clouds and a user of the cloud.
• Confirm that the primary cell is running by browsing to its eth0 primary IP address as its URL using https.
• Confirmed that its PostgreSQL database is also running by logging into its URL at the HTTPS ports of 5480 with the root account.
• As an optional step, deploy an additional vCloud Director application cell.
VMware Telco Cloud Automation (TCA)
This is VMware’s orchestration and automation platform focused on mobile industry telco cloud deployments. These are deployments focused on the delivery of network functions (NF) and network service (NS) capabilities on telecommunications networks instead of these being rolled out using traditional monolithic technologies.
The following diagram shows TCA Worker Node customisation information.
VM Config Operator talks directly to VMs and modifies them accordingly
NodeConfig is launched in each worker node and modifies the node accordingly.
Infrastructure requirements and worker node capabilities are specified in the VNFD.
Workers are customers cased on NF requirements
The following can deploy most configurations.
• VMConfig Operator
• NodeConfig Operator
These are described further below.
VM/hypervisor level customisations are done via VMConfig Operator and run the management Kubernetes cluster. it communicates with the hypervisor i.e. vCenter/vSphere. Examples of this are CPU pinning for worker VMs, VMX settings modification and SRIOV VFs attachment to the worker node.
Worker node level customisations are done via the NodeConfig Operator and run in workload clusters. This modifies the worker VM guest OS settings. Examples of this are hugepages, kernel and package installation.
VM Lifecycle Management Operations
There are several operations that need to be carried out on and for VMs during their normal lifecycle. These include VM creation, update, upgrade and deletion activities. These actions can range from installing a VM from an image or from a pre-existing VM template, upgrading a VM’s software, changing its virtual hardware settings such as CPU, memory or storage allocation all the way up to actual decommissioning and deletion.
Cluster Template
A cluster is a group of nodes that pulls its resources together and it distributes work across its nodes. It consists of master and worker nodes.
Clusters are managed centrally and can be deployed using templates.
So different settings can be modified at runtime e.g. update of OS settings.
• Configure settings and details
• Configure master and worker nodes
• Review of current nodes and other settings before saving
The master node has the control plane and the worker node controls the network function
TCA supports upstream Kubernetes and it can use the following.
• Kalico networking
• vSphere storage
CNF Onboarding
A CNF is a Cloud-native Network Function. This is containerised network function and it can be deployed using TCA as follows.
Browse to the TCA home page and log in.
The following dashboard will be shown.
Take note of the following links on screen.
Network Functions Catalogue and Network Services Catalogue links
Dashboard, Clouds, Infrastructure and Network Functions menus
Click on Infrastructure > CaaS Infrastructure
This will show the list of Cluster Instances and Cluster Templates that can be deployed on this Infrastructure.
NFs can be deployed within the instances.
Note Partner Systems option, which shows additional resources such as Harbor that are used in the NF deployment process. For instance, Harbor is used as a Docker registry.
In the above example, clicking on the arrow to expand on its settings shows the following information.
to the NF deployment click on the Network Functions menu option, then Catalog.
the Onboard Network Function button
Harbor is out of scope but for more information on Harbor review the following link. https://goharbor.io Back
Click
This uses the SOL001 and SOL004 compliant NF descriptors or VNFDs and cloud service archive or CSAR packages.
These can be uploaded or designed in the GUI. This example will design the NF in the GUI. Click on this option as shown below.
Type in a package name, then optionally select a key and value. Select either a Virtual Network Function (VNF) or Cloud Native Network Function (CNF).
This example will select a Cloud Native Network Function.
In this case select the CNF option. The DESIGN button will now be available, so click on it to start the deployment.
This will now show the following screen listing the NF Catalog’s properties.
These fields can be edited to suit any NF specifications such as product name, company and application version.
Take note of the various settings configured as part of this NF design.
Drag the Helm icon to the main window.
The chart name and version must be the same as the Helm chart stored in the relevant registry.
Further information can be configured in the Infrastructure Requirements tab. Click on this tab to update the infrastructure relevant information.
Items such as network adapter, PCI, kernel and NUMA settings can be updated in this section.
Click on the Infra Requirements toggle button to start configuring the infra settings.
Scroll down and expand on the Files section as shown below. Click on its Add button.
This will show a list of files in the Artifacts section. The Artifacts will be discussed later on, but for now know that this file needs to be included there.
Click on the UPLOAD button to upload these settings to the current application.
The following window will be shown confirming that the changes have been saved.
Click on the name to view the following screen, which shows that this application can now be instantiated.
If necessary, review its settings in the dropdown listed items or click on INSTANTIATE to start its deployment.
This window will be shown.
Type in the NF instance name, description and then click on Select Cloud. The following windows will be shown.
Select the relevant cloud for the Select Node Pool window below (which selects the workers that will be used for this application).
Select the required node pool and then click on NEXT.
Click OK so the original window is back on screen.
The cloud will now be selected as shown below.
Complete its name then select or specify a repository. In this current example a repository was selected from the previously discussed Harbor installation.
This Harbor link contains the assets needed for this network function implementation.
Click on the Helm Charts tab for the following window.
Go to the Network Functions Properties tab and click on the icon shown to review any settings if necessary. If not click on NEXT.
Click on the Inputs tab to add any configuration inputs for this deployment.
In this case these were stored in a YAML file on the local device, which was uploaded by clicking on the BROWSE button then following the onscreen steps.
This is shown as follows. Click
NEXT.
Click INSTANTIATE so the deployment will start.
The first step is reconfiguration of the infrastructure as shown in the following screen.
This infrastructure reconfiguration can take a few minutes.
After this the network function instantiation will start if the previous item was completed successfully.
This stage does need to grant a request for the validation of the dependencies, after which the following window’s network instantiation will start.
On completion the screen will look like this.
It also has a Source tab, click on it and view the configuration in YAML format. This YAML format can be updated, saved as a draft, or uploaded from a file using onscreen buttons. These buttons are shown at the bottom right of the screen.
TOSCA Metadata, Definitions and Artifacts can be viewed by expanding each of the sections.
Click on the Resources tab to view items used by the deployment.
Further information can be uploaded to the Artifacts as shown in the following screen.
In the screen below a README.txt file was uploaded by clicking on Choose Files and following the onscreen dialogue steps.
Conclusion
That’s it for now folks. Hopefully this has been a useful ‘reminder’ of some key concepts and will continue to be useful to you as a serious networking professional. We are constantly striving to enhance the Big Little Book series so let us know if there are any topics you would like to see in future editions of this book. That’s it for now, let us know if there’s anything you would like added to the next edition of this book by sending an email to info@gridlockaz.com.
Thanks for reading and wishing you all the best in your career pursuits.
Take care.
U V Omos
