Over the last few months I’ve done a couple of VMware Cloud Foundation (VCF) 3.x deployments and the question I almost always get is how to perform a startup or shutdown of the VCF environment. Since the documentation for VCF 3.0 and 3.5 doesn’t give you any guidance on how to do this, I set out to create one so I don’t have to do it each time doing a deployment. This post covers the startup/shutdown procedure for VCF 3.x.

Since the VCF architecture is based upon the VMware Validated Design (VVD) architecture, the startup/shutdown procedure for the VVD is a good starting point. But there are some differences in the two architectures. For example, the VVD uses a load-balanced Platform Services Controller configuration where as VCF does not.

Also, the documentation for VCF 2.x does provide a startup/shutdown procedure. Since the biggest change in VCF 3.x compared to VCF 2.x is the introduction of the Bring-Your-Own-Network construct, the startup/shutdown procedure for the virtual infrastructure hasn’t changed that much.

In the end I combined both the VVD 4.3 and VCF 2.x documentation to provide a startup/shutdown procedures for VCF 3.x.

Note; VCF 3.0 is based on VVD 4.3 and VCF 3.5 is based on VVD 5.0. Since the overall architecture hasn’t changed, I used the VVD 4.3 documentation.

Note; The startup/shutdown procedures are almost the same when using VCF 3.5 with a NSX-v or NSX-T workload domain. The procedures below describe the steps for NSX-v and NSX-T workload domains.

Startup a VMware Cloud Foundation Environment

Start up the virtual machines of the management stack by following a strict order to guarantee the faultless operation of and the integration between the components.

Prerequisites

Verify that you have direct console access to the ESXi hosts in the system.

Verify that you have the host names and IP addresses of the ESXi hosts that are members of the management domain.

Verify that you have the host names and IP addresses of the ESXi hosts that are members of each workload domain.

Verify that external dependencies for the environment, such as, Active Directory, DNS, and NTP are available.

Refer to VMware Knowledge Base article 2142676 for information on starting up hosts and exiting maintenance mode.

Procedure

Power on each ESXi host in the management domain, and exit maintenance mode. Power on each ESXi host in the first workload domain, and exit maintenance mode. Repeat this step for each workload domain. Power on the infrastructure management VMs in the management domain. Wait until each VM is powered on and all it services started before powering on the next VM. Power on the VMs in the following order: Platform Services Controller virtual appliances

vCenter Server for the management domain

vCenter Server for each workload domain.

SDDC Manager

NSX Manager virtual appliance for the management domain

NSX Manager virtual appliance for each workload domain

NSX Controller cluster virtual appliances for the management domain

Optional: NSX Load Balancer virtual appliances for the management domain (only when vRealize Operations and/or vRealize Automation is deployed) Power on the vRealize Automation virtual appliance and IaaS components in the management domain. Wait until each VM is powered on and all it services started before powering on the next VM. Power on the VMs in the following order: The vRealize Automation IaaS SQL Server VM(s)

All vRealize Automation virtual appliances

All vRealize Automation IaaS Web Server VMs (Power on the primary IaaS Web Server VM first)

All vRealize Automation IaaS Manager Services (Power on the primary IaaS Manager Server VM first)

All vRealize Automation IaaS proxy agents

All vRealize Automation IaaS Distributed Execution Management (DEM) hosts Power on the vRealize Operations Manager virtual appliances in the management domain. Wait until each VM is powered on and all it services started before powering on the next VM. Power on the VMs in the following order: The vRealize Operations Manager master node

The vRealize Operations Manager master replica node

All vRealize Operations Manager data nodes Power on the vRealize Log Insight virtual appliances in the management domain. Wait until each VM is powered on and all it services started before powering on the next VM. Power on the VMs in the following order: The vRealize Log Insight master node

All vRealize Log Insight worker nodes Power on the vRealize Suite Lifecycle Manager virtual appliance in the management domain. Power on the VMs in the first VI workload domain.. Each workload domain includes a three-node NSX Controller cluster. Power on these VMs first. Repeat this procedure on each workload domain (NSX-v). Power on the NSX Edge Services Gateways and Logical Router VMs (NSX-v). Repeat this procedure on each workload domain. Power on the NSX Edge Cluster Nodes (NSX-T) Power on the workload domain VMs. Repeat this procedure on each workload domain. Using SSH, log in as root to the SDDC Manager VM and run the ./sos –health-check command to verify that everything works correctly.

Shutdown a VMware Cloud Foundation Environment

Shut down the virtual machines of the management stack by following a strict order to avoid data loss and faults in the components.

Prerequisites

Coordinate the shutdown in advance with business stakeholders to minimize any impact.

Verify that you have saved the account passwords to a location external from the Cloud Foundation system you are shutting down.

Verify that valid backups of all management and tenant VMs are available and saved to a location external from the Cloud Foundation system you are shutting down.

Verify that you have direct console access to the ESXi hosts in the system.

Verify that you have the host names and IP addresses of the ESXi hosts that are members of the management domain.

Verify that you have the host names and IP addresses of the ESXi hosts that are members of each workload domain.

Refer to VMware Knowledge Base article 2142676 for information on shutting down hosts and entering maintenance mode.

Procedure

Shut down the workload VMs in each workload domain. Power off the workload domains VMs. Repeat this procedure on each workload domain. Power off the NSX Edge Services Gateways and Logical Router VMs (NSX-v)or the NSX Edge Cluster Nodes (NSX-T). Repeat this procedure on each workload domain. Each workload domain includes a three-node NSX Controller cluster. Shut down these VMs last. Repeat this procedure on each workload domain (NSX-v). Shut down the vRealize Suite Lifecycle Manager virtual appliance in the management domain. Shut down the vRealize Log Insight virtual appliances in the management domain. Wait until each VM is powered off and all it services stopped before powering off the next VM. Shut down the VMs in the following order: All vRealize Log Insight Worker nodes

The vRealize Log Insight Master node Shut down the vRealize Operations virtual appliances in the management domain. Wait until each VM is powered off and all it services stopped before powering off the next VM. Shut down the VMs in the following order: All vRealize Operations Manager data nodes

The vRealize Operations Manager master replica node

The vRealize Operations Manager master node Shut down the vRealize Automation virtual appliances in the management domain. Wait until each VM is powered off and all it services stopped before powering off the next VM. Shut down the VMs in the following order: All vRealize Automation IaaS Distributed Execution Management (DEM) VMs

All vRealize Automation IaaS Proxy Agent VMs

All vRealize Automation IaaS Manager Server VMs Shut down the secondary IaaS Manager Server VM first; shut down the primary IaaS Manager Server VM second

All vRealize Automation IaaS Web Server VMs Shut down the secondary IaaS Web Server VM first; shut down the primary IaaS Web Server VM second

All vRealize Automation virtual appliances

The vRealize Automation IaaS SQL Server VM Shut down the infrastructure management VMs in the management domain. Wait until each VM is powered off and all it services stopped before powering off the next VM. Shut down the VMs in the following order: Optional: NSX Load Balancer virtual appliances for the management domain (only when vRealize Operations and/or vRealize Automation is deployed)

All NSX Controller cluster virtual appliances for the management domain.

The NSX Manager virtual appliances for the workload domains

The NSX Manager virtual appliance for the management domain

SDDC Manager VM

The vCenter Server virtual appliance for the workload domains

The vCenter Server virtual appliance for the management domain

The Platform Services Controller virtual appliances Place the hosts for each workload domain in maintenance mode, and power off. Repeat this procedure on each workload domain. Place the hosts for the management domain in maintenance mode, and power off. Shut down the unassigned ESXi hosts in the Cloud Foundation system, if any.

Startup/Shutdown Order of the Management VMs

The following table provides an overview of the startup and shutdown order of all the components in the environment.

Component Startup Order Shutdown Order ESXi Hosts 1 7 ESXi Hosts Management Domain 1 3 ESXi Hosts Workload Domain # 2 2 Infrastructure Management 2 6 Platform Services Controller #1 1 6 Platform Services Controller #2 1 6 vCenter Server Management Domain 2 5 vCenter Server Workload Domain # 2 5 SDDC Manager 3 4 NSX Manager Management Domain 4 3 NSX Manager Workload Domain # 4 3 NSX Controller Cluster Management Domain (#1, #2, #3) 5 2 Optional: NSX Load Balancer Management Domain (#0, #1) 6 1 vRealize Automation 3 5 vRealize Automation IaaS SQL Server VM(s) 1 7 vRealize Automation Appliances 2 7 vRealize Automation IaaS Web Servers #1 3 6 vRealize Automation IaaS Web Servers #2 4 5 vRealize Automation IaaS Manager Servers (#1) 5 4 vRealize Automation IaaS Manager Servers (#1) 6 3 vRealize Automation IaaS Proxy Agents (#1, #2) 7 2 vRealize Automation IaaS DEM Hosts (#1, #2) 7 1 vRealize Operations 4 4 vRealize Operations Manager Master Node 1 3 vRealize Operations Manager Replica Node 2 2 vRealize Operations Manager Data Node(s) 3 1 vRealize Log Insight 5 3 vRealize Log Insight Master Node 1 2 vRealize Log Insight Worker Nodes (#1, #2) 2 1 vRealize Suite Lifecycle Manager 6 2 vRealize Suite Lifecycle Manager 1 1 Workload Domain 7 1 NSX Controller Cluster Workload Domain # (#1, #2, #3) 1 3 NSX Edge Service Gateways and Logical Router VMs (NSX-v) 2 2 NSX Edge Nodes (NSX-T) 2 2 Workload Domain # VMs 3 1