Rob Hiltbrand, director of IT at The Liberty Group, shared valuable lessons he learned during his company's transition to cloud services. Simply put: Things didn't always go as planned.

Speaking at last week's Interop conference, Rob Hiltbrand, director of IT at The Liberty Group, shared some valuable lessons he learned during his company's move from on-premises to colocated infrastructure. Simply put, things don't always go as planned.

The Liberty Group provides executive search and temporary staffing services across the U.S. Hiltbrand gave a blow-by-blow account of the challenges he faced during a push to modernize the firm's infrastructure with a unified communications approach that involved moving workloads to a colocated facility.

Hiltbrand said he saw great potential in transitioning the company from legacy on-prem infrastructure to a more flexible, services-based model. But at first he had trouble convincing his bosses. "If management doesn't see value in cloud services, what use is it to them?" he said. "These are people focused on profit and loss. They don't care how they get their phone or apps, but how they do their job. They're not interested in the latest shiny object."

The challenge was to show how an investment in new services would help the bottom line. Surprisingly, Hiltbrand got an assist from Mother Nature. From 2006 to 2010, the Houston-based company grew from one to four offices. But its main computer systems, including the email server, stayed on-premises in Houston. Whenever the systems at headquarters had a problem, it affected the other three offices but with no major disruptions. That is, until 2008, when Hurricane Ike shut down the company's main office for five days, effectively closing down the three branch offices as well. That meant no email for the sales staff and other personnel.

"The owner told me to find another place to plug in an email server," said Hiltbrand. After a frantic search, he identified a colocation facility in West Houston and made the switch, though it wasn't easy: "We had no lights in the main building, and we had to carefully drag legacy domain controllers that weighed 54 pounds each out of the building."

Get our report — Navigating IT Transformation: Tales from the Front Lines Transform with Hybrid IT

Fortunately, everything worked from the new location, which helped management see the benefits of a less centralized infrastructure.

A sales executive at the company, concerned that any more outages would impact sales, helped Hiltbrand convince senior executives that more of the infrastructure had to be moved off-site. Hiltbrand said he was happy at first but added there was also an element of "be careful what you wish for."

"We picked a place in South Austin, three hours away," Hiltbrand recalled. "Then you have to start thinking about a new firewall and IP addresses, relocating your email server and updating your domain controllers, doing a lot of message hygiene and re-IPing. That all takes time, and it was just me and a few consultants. I was physically racking servers along with everything else."

By 2011 the move was completed. It was the triumphant start of a new era, right? Not exactly. The day after launch everything went offline. After a lot of analysis, Hiltbrand's team determined that a legacy application was the culprit and the situation was resolved.

Another technology upgrade challenge was the phone system. Employees were given mobile phones, but it soon became clear that IT needed a mobile device management (MDM) system to monitor and address problems in the field.

At first his request was denied, but later the CFO gave Hiltbrand the OK with the caveat, "Don't spend a lot of money."

His early look at MDM options wasn't promising. "I had sticker shock when I saw the MDM vendors were pricing at $20 per month per device," he said. Eventually, Hiltbrand found a lower-cost vendor, but it was the old story of "you get what you pay for."

"The vendor was selling me on the idea that this was a full MDM suite that does everything, but actually the only two things it was able to do was let me turn on the phone's camera [remotely] and identify the location of each phone. When I saw the camera feature, I thought, 'That's creepy.'"

When Hiltbrand demoed the system to his CFO, he was surprised to hear her say the same thing: "That's creepy."

Liberty didn't get any MDM services, but over the next few years, Hiltbrand took advantage of the built-in tools his wireless provider offered to cover most of the mobile device management he needed.

Hiltbrand shared other tales of deployment glitches that were resolved as the company embraced unified communications and virtualized its phone system. "As the company expanded to Phoenix and Denver, it was clear we had to own and control the phone numbers and the provisioning," he said.

One specific piece of advice he had for the Interop audience was to make sure you have a backup payment provider for every one of your cloud services. Hiltbrand realized how important this was the hard way, when suddenly one office reported nobody could make calls. It took a few hours to figure out that the payment method they were using wasn't being accepted even though the account was current.

"The CFO tells me, 'Guess what? The bank that issued the credit card stopped payment because of a security algorithm that had an issue with all the charges to the card,'" said Hiltbrand. He never got a satisfactory answer for why the payments were stopped and why the company wasn't given a chance to resolve it, but lesson learned: He has since added a secondary payment provider as a safeguard.

Through it all, Hiltbrand said the main secret to his success was having a good boss.

"She's given me the freedom to fail, but always asks, 'What's your plan B?'" he said. And, he noted, good managers should always have one.

Related links:

HPE and Stephen Hawking’s COSMOS to Investigate the Early Universe