Challenges are one of the reasons that keeps my interest in IT. I’ve supported many products in my career. Every product I’ve supported have one thing in common…..UPGRADES.. ( Typed that in upper case on purpose).

There are couple strategies to upgrades.

Blue / green deploys, a recent cloudy term to stand up new, deploy, test and cutover. Before this was a term, I prefer this method. If there is an issue, it’s easy to revert to original environment.

In-place upgrade of existing environment. This is NOT my favorite method, but necessary sometimes.

This post shares experiences on a recent upgrade from vRealize Automation 7.1 to 7.2. These are my notes, they include links to formal documentation, which in my experiences with any product, there are a few tips / tricks between the lines docs don’t cover. I’ve included my raw notes below. I hope this helps in your adventure.

I was fortunate enough to have a separate environment to test before doing the real environment. Since this was an in-place upgrade, going through the experience helped prepare what to expect, as well as to revert back to original environment.

With the magic of cloning Linux appliances, snapshots on IaaS windows machines, and a SQL Server backup. It helped simulate the ability to go back to original infrastructure. It’s doesn’t make me as comfortable standing up new, but was better than no backout plan at all.

Read the upgrade docs, search forums for others reporting issues

Test the upgrade multiple times. Also, think about if something goes wrong, how to revert to original version. Do this multiple times until you are sick of it, that means you’ll be prepared

Have backups, clones and use snapshots on windows machines

Disable Backups before starting. Contention on machines could cause issues

If you have support with VMware, open a proactive case and have them review your environment.

Clean-up any un-submitted requests

Clean-up any In-Progress requests that are orphaned

Give yourself 4 to 8 hours. Communicate to your end users. (Under promise, over deliver)

Coordinate with your users to test use cases after upgrade

Remember external systems like vRealize Business, Log Insight that access vRA.

Make sure end users disable automated build requests.

Here are my raw notes including links including steps I followed. These were reminders to help the overall process, the order of operations. I’ve found having these types of notes help to refer to.

Prep work, review documents and blog posts.

Items I did as prep work.

Verify all services, nodes are healthy before beginning

Check disk space on psc, vcenter, vRA / vRO appliances (df -h)

Check alarms in vCenter

Make sure nightly backups of PostgreSQL are on NFS share. I run a dump on databases for extra backups about the same time as SQL Server.

Run vRealize performance test = run a config test

https://blogs.vmware.com/management/2015/04/new-health-monitor-tool-vrealize-suite-vrealize-production-tool.html

Launch config using the command: java.exe -jar <vRPT Jar file> config

Download updated management agent. KB article with updated management agent, Workaround issue on upgrade on IaaS components

http://kb.vmware.com/kb/2147926

Load-balancing

The network guys will wonder, what?! Mine did, just tell them the vendor requires the change, double and triple check this step.

Disable secondary nodes, health checks on VIPs (Same setup as installation)

https://pubs.vmware.com/vrealize-automation-72/topic/com.vmware.ICbase/PDF/vrealize-automation-72-load-balancing.pdf

Cloning each machine vRA to upgrade

Turn off vRA appliances – Clone each machine (good to have backups of original appliances)

Turn off IaaS machines, snapshot.

Turn on all machines, verify services both IaaS and vRA are healthy (both appliances) I learned by testing if you clone windows, I see extra machines in the VAMI you’ll see “several called clone”

https://<vra-url>:5480/#cafe-services (make sure all services are started)

Close any opened files in the pgdata directory and remove any files with a .swp suffix. (on primary vRA appliance)

–/var/vmware/vpostgres/current/pgdata/

–/var/vmware/vpostgres/current/pgdata/ Backup SQL Server database (Get with DBA to schedule ahead of time)

Upload ISO to datastore, mount to primary and secondary appliances if you don’t have internet access for updates.

Adjust settings in VAMI on primary and secondary to use cd-rom

***- Run on a remote machine well connected to network, not laptop on wireless***

***Disable vCenter backups on the cluster so snapshots aren’t taken***

De-register vRB via vRB appliance (because we haven’t used it yet) – might not apply to you.

Run update on primary (Look for text similar listed below when completed)

Reboot primary and secondary vRA appliances after upgrade completed

Verify both appliances upgraded

Deploy updated Management agent on IaaS machines

Deploy updated java (version 1.8)

Reboot each IaaS machine

Verify services on IaaS and appliances are healthy

Create upgrade.properties on primary ( Backup file = cp -p upgrade.properties upgrade.properties.password)

run ./upgrade (cross your fingers) – I had to run three times and my install FINALLY upgrade all six windows machines.

Raw text after upgrade

Version 7.2.0.381 Build 4660246

Last Check:

Tuesday, 2017 January 31 15:46:07 UTC-5 (Using update CD found on: /dev/sr0)

VA-check: finished

Pre-install: finished

After all appliances are upgraded, ssh to the master appliance and go to /usr/lib/vcac/tools/upgrade. Populate all the required data in upgrade.properties and execute ./upgrade script

Replica nodes are upgraded successfully. Reboot master node to trigger the reboot of replica nodes

Post-install: finished

Update finished successfully.

WARNING: Immediately update any vRealize Automation IaaS nodes after reboot to avoid product version mismatches.

Last Install:

Tuesday, 2017 January 31 16:29:07 UTC-5

VA-check: finished

Pre-install: finished

After all appliances are upgraded, ssh to the master appliance and go to /usr/lib/vcac/tools/upgrade. Populate all the required data in upgrade.properties and execute ./upgrade script

Replica nodes are upgraded successfully. Reboot master node to trigger the reboot of replica nodes

Post-install: finished

Update finished successfully.

WARNING: Immediately update any vRealize Automation IaaS nodes after reboot to avoid product version mismatches.

Hope this helps,

Steve Schofield

#vExpert 2017

@steveschofield

http://vsteve.me