A massive change happened in the U.S. intelligence community after the 9/11 attacks. Agencies were realigned and bureaucracies expanded. Congress created the Director of National Intelligence position and the National Counterterrorism Center amid growing fears that the nation was unprepared for the jihadist threat.

There is now a brewing debate on whether the intelligence community — a collection of 17 agencies with an annual budget of about $54 billion — may have become too narrowly focused and slow to respond to a changing world. Analysts and foreign policy experts have sounded alarms in recent years as U.S. intelligence largely failed to predict the Arab Spring, the emergence of Russia and China as military competitors to the United States, North Korea’s advances in nuclear weapons, Iran’s rise as a regional power and the surge of the Islamic State.

Intelligence officials insist they recognize the problem and believe it will persist until agencies can fix their talent issues: namely that the government workforce needs more technically skilled workers, and agencies have to change how they work with the private sector.

“These are critical decisions for us: What skills do we invest in for the next five to 10 years?” said National Security Agency Director Adm. Michael Rogers, who also heads U.S. Cyber Command.

At the NSA, “we continue to generate great insight. But the flip side is that it continues to be more difficult to do it,” said Rogers last week at the 2017 Intelligence and National Security Summit in Washington. “How do you sustain the workforce with the skills they need for the challenges that keep changing?”

Like other parts of the U.S. national security establishment, the intelligence community has been rattled by the democratization of technology and how enemies weaponize it in unpredictable ways.

The internet is perhaps the most dramatic example of how technology is being used against the United States. Another is satellite imagery. “Geospatial intelligence is going through a transition,” said Robert Cardillo, director of the National Geospatial-Intelligence Agency. Space technology to track and monitor objects and people around the world used to require “government access, government security, government control and ownership,” said Cardillo. “Now the barriers have been reduced.”

That creates a serious dilemma for the NGA: “How do I advance our value proposition in a flatter, more disruptive world?” Cardillo asked.

The answer is “human capital,” he said. The bulk of the technological innovation is not coming from the government but from the industry, “so our challenge is to create the right agility to work with you, to de-clutter the acquisition process to enable your innovation to get to my desktop sooner.”

How the government makes this transition is still an open question, Cardillo continued. “How do we lift the skill set of a workforce that needs to be more data-savvy and more computationally knowledgeable? If we hold on to how we did it in the past, we put ourselves at risk as an agency.”

Skills and talent also are hot topics at the Defense Intelligence Agency, said Deputy Director Melissa Drisko. The DIA in recent years has had to quickly beef up its expertise on China, Russia, North Korea and other potential adversaries.

Knowing how to exploit data is of the essence, she said. “We have to be much more data-centric, more savvy in how we handle data and the veracity of data.” The artificial intelligence revolution is giving analysts unprecedented automation and speed. By the same token, added Drisko, “we are having to reexamine what’s the role of the humans.”

It is “kind of scary for analysts,” she said. They have to try to predict where technology is going and “where does the human being add value?” That thinking is “driving decisions on what kind of people we hire.”

Government pay is just one piece of the talent puzzle. According to the Congressional Budget Office, federal workers with a professional degree or doctorate earn 24 percent less, on average, than their private-sector counterparts. The wage gap is even wider for cybersecurity and intelligence analysts.

The intelligence community, however, draws skilled professionals for reasons other than pay. The work is viewed as exciting and a resume builder. Drisko said agencies still could do more to hold on to their top talent.

“A lot of people we bring in now are very smart, very inquisitive, tech-savvy.” That is good news for the agency, but the culture has to change for the incoming workforce to thrive, she added. “We need to give them a realistic expectation of the environment they will likely have to operate in.”

The government also has a tendency to stifle creativity, sometimes causing talented workers to leave. “We tell them to leave cellphones at the door, we tell them they can’t download every app they want,” said Drisko. She is looking for DIA to make people feel less restricted “and expose them to the art of the possible, to keep the creativity alive.”

Government contractors share some of the blame, too, she noted. “One of my frustrations is that someone shows me this wonderful tool, and I have to pay a gazillion dollars to teach people how to use it. That’s not really helpful. That stifles the creativity of our analysts. They’re not going to play with it.”

Private companies that compete for work in the intelligence community are frustrated as well. They have called on the government to change contracting methods and rules that date back to the 1970s so companies are rewarded for innovative ideas and skills, rather than for billable hours.

The industry for years has complained that competitions for contractor services in the intelligence agencies are all handled the same, even though some services are commodities. That incentivizes low-ball bidding instead of “best value” and “innovative technology solutions,” noted a white paper by the Intelligence and National Security Alliance, an industry advocacy group.

“Awarding contracts on the basis of the quality of the proposed solution rather than on cost provides incentives for contactors to make investments,” said the white paper. “It also avoids the ‘race to the bottom’ syndrome in which contractors propose unrealistically low prices.”

Cardillo agreed that traditional contracting is not helping the intelligence community get the most advanced technology quickly enough to stay ahead of threats. NGA is eyeing a new approach. It would allow companies to bid on “access to government data” so they can play with it, create apps and other tools that they would offer the government. “We have data sets that we believe have high value for computer learning,” said Cardillo. “We’d like to find a way to make them competitively available to industry. We’d like to see if we can exchange that access for an innovative application of that same data.”

Allowing companies to experiment with government data could help NGA and other agencies gain a better grasp of what artificial intelligence has to offer, he said. Contractors would be rewarded based on “who could apply the most innovative algorithm or application.” And giving contractors real government data would give projects more credibility.

Susan Gordon, principal deputy director of national intelligence, said agencies face an unprecedented abundance of data, “and we have to make sense of it.” That means the intelligence community will become increasingly reliant on the private sector for artificial intelligence and automation technologies. “We are simply going to be unable to make use of all the information we have if we continue to [exploit it] manually.”

In the counterterrorism business, said Cardillo, the “need to be agile is more important than ever. We have to detect and prevent. The window is getting smaller and smaller. And the enemy can call an audible.”

Sandra Erwin is a national security and defense reporter for RealClearDefense. She can be reached at serwin@realcleardefense.com. Follow Sandra on Twitter @Sandra_I_Erwin.