When you handle trillions of dollars a year in transactions and manage the largest known vault of gold in the world, security and efficiency are top priorities. Open source reusable software components are key to the New York Fed's successful operation.

Large companies have divisions and subsidiaries that make efficient organizational management a challenge. Perhaps no one recognizes that more than Colin Wynd, vice president and head of the Common Service Organization at the Federal Reserve Bank of New York. Wynd is charged with ensuring that software development practices and strategy are forward-thinking and secure, and adhere to compliance regulations.

Several years ago, Wynd and his team started to think more holistically about how their developer teams worked, he explained in a presentation at the recent Jenkins World conference in San Francisco. They needed to transition decades of legacy applications to more modern, flexible alternatives.

“We were doing a lot of custom development, building databases, and implementing security such as authentication and authorization, all with custom logins. The joke was that we actually had more login utilities than we had applications,” recalled Wynd. “This was slightly crazy.”

The New York Fed wanted to standardize its base technologies, development tooling, and architectural models. “Every user interface was a snowflake, completely different. An end user might use 15 different interfaces,” said Wynd.

Another goal was to use open source software to achieve far greater reuse of libraries and other common sets of code. Wynd took note of an evaluation from Sonatype, a company that specializes in software automation, that reported 75 to 80 percent of applications are made up of open source libraries. He saw value in open source adoption for the New York Fed. Doing so lets developers at the bank focus more on an application’s unique aspects, such as its business logic, rather than spend time building out the basic software plumbing, according to Wynd,

“You can have a team dedicated to these common components and microservices,” he explained. “We can invest in [those common essentials] and make sure they are rock solid.”

The nearly 2,000 developers across the Federal Reserve System used to have a disparate set of developer tools. Now, they benefit from a standard toolset and architecture, which also places limits on which applications the bank will consider using. “We don’t want a third-party application that isn’t compatible with our common architecture,” said Wynd. The organization turns down some vendors and developer projects as a result. “But the benefits are that we improve the underlying architecture’s stability and security,” he added.

The software supply chain

It takes time and research to identify the right software components, but Wynd said it’s essential.

There’s a lot of governmental oversight, plenty of stakeholders, and significant regulatory controls. “We have a board of directors and a board of governors who are government employees, and also the U.S. Treasury. I might be building an application for any of those,” Wynd said. It can get complicated to adhere to the rules of each. “We spend a lot of time in working groups to make sure we’re doing best practices.”

He explained, “We realized that it’s a supply chain. It’s not that different from, say, how a car company looks at all the suppliers that help them build a truck. It might get tires from one vendor, the airbags from another firm, and windshields from another supplier.”

And just as with any car company, the New York Fed wanted to monitor its supplier quality control. “What we found is that not all the parts that are open source are created equally,” said Wynd.

Again, pointing to the Sonatype report, which looked at over 120,000 Java libraries, Wynd noted the mean time it takes to repair software is more than 200 days—an unacceptable delay for getting a patch. This makes the New York Fed very discerning in choosing open source software. “There is a small set of suppliers that is reputable and very good at maintaining open source code, and then there are some that are OK,” he said. “But the vast majority of open source code is not really maintained at the level for applications that are truly mission critical.”

Another issue is the effect of open source licensing terms on how the New York Fed distributes its own applications and manages compliance. “Some open source licenses are very easy,” Wynd said. “You can print my code wherever you want. You can take it, rewrite it, modify it, bundle it into applications, and you can release these applications internally or externally.” But other open source software is far more restrictive in how it can distributed or used commercially. That raised concerns about whether an open source module was being maintained and “whether we felt comfortable distributing it, say, to other Federal Reserve banks,” said Wynd.

As a result, in evaluating open source libraries and related software, the New York Fed pays special attention to the licenses and any distribution limits. Having developed the expertise to identify which licenses pass muster lets the bank take advantage of these reusable components.

Giving developers more flexibility and training

One less obvious advantage to open source adoption is in career satisfaction and advancement. It gives developers opportunities to work on more interesting applications, said Wynd. Developers can now take on projects or switch jobs more easily across Federal Reserve banks because the New York Fed uses a lot of common open source components and a standard tool set, meaning retraining is minimal if needed at all. “That helps with our load balancing across the Federal Reserve system.”

Which is not to say developers don’t get training. Wynd said the bank created Common Service University to offer continuous training in developer best practices and instruction.

One reason for bringing training in house was consistency. “If you send someone off somewhere to take a course in Angular or Spring, they start to do things in slightly different ways and that affects the consistency of how software is developed,” Wynd said. “With Common Services University, we can be clear on, ‘This is how we want you to build applications.’ We also focus on security—how do we do authentication, clickjacking, and all that. We want to be consistent, because if these things are solved differently, it will lead to a pain point.”

Security challenges ahead

During a Q&A session, Wynd said security is an ongoing challenge. The New York Fed has a lot to protect, including the $4 trillion in transactions it handles and the $250 billion worth of gold bars it manages for international customers. “Countries still move gold back and forth, so we have an inventory management system” that requires extreme security, explained Wynd.

Security also looms large in regulatory issues. “We are very careful and prudent," Wynd said. "But our biggest headache is to prove to groups that an application is secure, because we have to defend against nation state attacks.”

Open source software supply chain: Lessons for leaders