Realizing the Potential of Analytics

Arming the Human Mind

The collection systems worked more or less. But, there were no analysts capable of interpreting the data, since the Army had moved to maximum automation — and the automated systems were not programmed for so unexpected a contingency as a deployment to an African backwater. — Ralph Peters, The War in 2020

The Analytic Gap

The terrorist attacks of 11 September 2001 (9/11) exposed an analytical deficiency within the intelligence community.[1] However, the US Air Force has invested in expanding collection capacity, as evidenced by the 375 intelligence, surveillance, and reconnaissance (ISR) platforms added since 9/11.[2] More remarkably, the increase in ISR platforms has come at a time when the total Air Force inventory has decreased by 500 aircraft.[3] ISR platforms such as the RQ-4, MC-12, and MQ-1 directly address an important element of ISR — but at the cost of other critical elements, such as intelligence analysis. Air Force Doctrine Document 2–0, Global Integrated Intelligence, Surveillance, & Reconnaissance Operations, defines global integrated ISR as the “cross-domain synchronization and integration of the planning and operation of ISR assets; sensors; processing, exploitation and dissemination systems; and, analysis and production capabilities across the globe to enable current and future operations.”[4] However, today’s investment in collection capacity does not address the mounting limitations of intelligence analysis initially highlighted by the 9/11 Commission in 2004.[5]

Defining Automated Analysis and Analytics

Because the imbalance between collection capacity and intelligence analysis has prevented realization of true global integrated ISR operations, key Air Force leaders are promoting investment in automated analytical programs as a means of removing this disparity.[6] This article considers automated analysis the utilization of algorithms to transform a series of collected data into a usable intelligence product. Analytics can handle immense amounts of data by using preprogrammed models to produce decisive, actionable results.[7] This procedure has boosted profits in the private sector by influencing consumers’ buying decisions and conducting Wall Street transactions in microseconds, processing more information and acting more quickly than a stock analyst.[8] Consequently, the intelligence community is exploring analytics as a means of managing large quantities of data, identifying patterns, and developing products to anticipate the activities of future adversaries.[9] A report by the Center for Strategic and International Studies revealed that an analytics program drawing on data from public databases and airline reservation codes could have identified all 19 of the 9/11 hijackers for further investigation before that fateful date.[10] Such examples have driven key leaders to look at analytics as a possible solution to the present analytical deficiency. Nevertheless, the use of analytics is not without peril. We must judiciously approach investing in analytics to improve intelligence and must fully understand its advantages and limitations. Closing the gap will involve matching analytic technology with the skill set of the intelligence analyst, thus optimizing the concept of global integrated ISR operations promoted by the Air Force.

Considerations for Analytics

Evaluating the potential of analytics must start with a determination of what the technology promises — specifically, accurate and tailored intelligence products that rapidly utilize large quantities of collected data. Incorporating analytics into intelligence assessment offers an appealing solution to our analytical shortfall because it closely mimics the system and technology-intensive solutions, which successfully enhanced collection capacity.

Research conducted by the Massachusetts Institute of Technology regarding the relationship between automation and humans revealed that automation may significantly affect a user’s situational awareness. [11] Analytics can decrease the occurrence of common human biases, including confirmation bias and assimilation bias.[12] However, it may introduce a new form — automation bias, which describes the human tendency to accept an answer generated by an automated system, even in the presence of contradictory information.[13] The likelihood of such bias increases with the complexity of the task performed by the automated system, primarily because the user has difficulty determining the major factors and processes that generate the automated solution. [14] Without this understanding, the user cannot reliably and confidently determine the accuracy of that solution. Further, the effects of an adversary’s cyber attack on our automated systems could go unnoticed, directly affecting the accuracy and reliability of global integrated ISR operations as a whole. The concept of automation bias raises a critical question: will more use of analytics close the gap between collection and analysis at the cost of increased bias?

A second significant risk associated with automated analysis and analytics concerns the lack of analytical agility associated with the coded algorithms. The term analytical agility denotes the ability of analysts, human or automated, to adapt their processes, given the introduction of new evidence or a paradigm shift. Analytically agile analysts and systems can quickly adapt assumptions and processes to changing information or new environments. Conversely, those who lack such agility dismiss new evidence or label new information incorrectly to remain within preestablished concepts. For a classic example of an individual without analytical agility, we need only look to the Cold War analyst who ignored the changing paradigms of global terrorism that led up to and followed the events of 9/11.

We must consider the significance of analytical agility when we evaluate the utility of increased analytics within global integrated ISR operations. Developing the algorithms that comprise analytics can prove time consuming. Additionally, humans do the programming, influenced by their own biases and assumptions they believed accurate and reliable at the moment of creation. Historically, we can attribute many intelligence failures to a lack of imagination or analytical agility necessary to identify new major factors and prevent surprise.[15] Analytics designers should be careful not only to look for previously observed signatures but also to consider emerging signatures or a possible change in the importance of an adversary’s existing signatures. Otherwise, analytics may hinder the discovery of evidence or misinterpret the contextual relevance of evidence that could lead to realization of an alteration in enemy tactics. Assumptions will change; therefore, we must update coding quickly to keep pace with a dynamic opponent.

Even without considering the fiscal constraints of modern military budgets, simply adding more human analysts can never match the expansive collection capacity of the Air Force. Innovative technology solutions such as analytics have been profitable in the private sector and may have even greater potential value for intelligence. However, we must consider and weigh such risks as automation bias and a paucity of analytical agility if we wish to invest effectively and improve the analytical capability resident within global integrated ISR operations.

Recommended Investment Strategy

Effective investment in these operations should concentrate on closing the gap between the Air Force’s current collection and analytical capabilities. Complete reliance on analytics and automated solutions to improve intelligence, however, represents a dangerous approach to solving that problem. The impact of automation bias and insufficient analytical agility associated with analytics could sideline human analysts, relegating them to rubber-stamping an automated assessment that reflects little understanding of accuracy or implications. But launching additional analysts who have no training in leveraging analytics effectively against an ever-expanding reservoir of data is an even bleaker proposition. Instead, Air Force leadership must develop a balanced investment strategy that includes analytics and, more importantly, the training of all-source human intelligence analysts in exploiting various techniques that will allow them to understand, operate, and optimize collection sensors and automated tools.

In spite of recent efforts to incorporate different analytical techniques into the core curriculum of the technical training school for intelligence officers and enlisted professionals at Goodfellow AFB, Texas, most ISR professionals (with the possible exception of graduates of the USAF Weapons School and the Air Force Advanced Analysis Course) are not adequately conversant in basic analytical methodologies. The latter include problem restatement, red teaming, weighted ranking, computation of conditional probabilities, hypothesis testing, and utility analysis. These courses provide a comprehensive set of tools for conducting objective, thoughtful analysis. Without this training, most intelligence analysts have to rely on their intuition, whereas effective leveraging of an automated system demands an understanding of its foundational principles and methodologies. Thus, we can maximize the capabilities of analytics only by offering training in specific analytical techniques coded in the programming of analytics. Expanding such training to include the methodologies and techniques of analytics would cost much less than a new collection system and, arguably, would produce better results.

This dual-pronged strategy will yield analytically astute, independent intelligence analysts poised to optimize the potential of analytics and realize the full capability of global integrated ISR operations. Integrating these analysts with analytics recognizes that we cannot pursue either path separately as the sole solution. Even though the speed of analytics permits the processing of more data, the human mind — even with its limitations — can supply the requisite analytical agility and imagination (a formidable task for analytics alone). Thus, productive consolidation of these pursuits can occur only if one is trained and equipped to manage an effective arsenal of analytical capabilities. Training intelligence analysts to better exploit various structured, analytical methodologies and investing in analytics will enable enhanced understanding, exploitation, and targeting of the adversary; improve the management of collection capacity and sensor employment; more effectively derive meaning from the collected data; strengthen the assessment of ISR effectiveness; and facilitate informed decision making — the ultimate objective.

End Notes

1. National Commission on Terrorist Attacks upon the United States, The 9/11 Commission Report: Final Report of the National Commission on Terrorist Attacks upon the United States (New York: W. W. Norton, 2004), 411–16.

2. Air Force Document 120201–027, Air Force Priorities for a New Strategy with Constrained Budgets, February 2012, 1,http://www.af.mil/shared/media/document/AFD-120201-027.pdf.

3. Ibid.

4. Air Force Doctrine Document 2–0, Global Integrated Intelligence, Surveillance, and Reconnaissance Operations, 6 January 2012, 1, http://www.e-publishing.af.mil/shared/media/epubs/afdd2–0.pdf.

5. National Commission on Terrorist Attacks upon the United States, 9/11 Commission Report, 77, 78, 91, 119, 276, 342, 346–48, 353.

6. Debra Werner, “Automatic Intelligence: Potential for Analyst Cuts Prompts Call for Smarter Tools,” DefenseNews, 1 September 2011, 1,http://www.defensenews.com/article/20110901/C4ISR02/109010319/Automatic-intelligence.

7. Thomas H. Davenport and Jeanne G. Harris, Competing on Analytics: The New Science of Winning (Boston: Harvard Business School Press, 2007), 7.

8. Thomas H. Davenport, “Competing on Analytics,” Harvard Business Review, January 2006, 3.

9. Werner, “Automatic Intelligence,” 1.

10. Mary DeRosa, Data Mining and Data Analysis for Counterterrorism (Washington, DC: CSIS Press, Center for Strategic and International Studies, March 2004), 6–8, http://csis.org/files/media/csis/pubs/040301_data_ mining_report.pdf.

11. M. L. Cummings, “Automation Bias in Intelligent Time Critical Decision Support Systems” (paper presented at the American Institute of Aeronautics and Astronautics First Intelligent Systems Technical Conference, Chicago, 20–22 September 2004), 1, http://web.mit.edu/aeroastro/labs/halab/papers/ CummingsAIAAbias.pdf.

12. Confirmation bias describes the tendency to give unbalanced weight to evidence consistent with a hypothesis rather than seek evidence that could invalidate a hypothesis. Assimilation bias describes the tendency to interpret ambiguous evidence to support initial hypotheses. Both biases can result in incorrect intelligence assessments. See Jane Risen and Thomas Gilovich, “Informal Logical Fallacies,” in Critical Thinking in Psychology, ed. Robert J. Sternberg, Henry J. Roediger III, and Diane F. Halpern (Cambridge, UK: Cambridge University Press, 2007), 112–14.

13. Ibid., 2.

14. Ibid.

15. James Bruce, “The Missing Link: The Analyst-Collector Relationship,” in Analyzing Intelligence: Origins, Obstacles, and Innovations, ed. Roger Z. George and James B. Bruce (Washington, DC: Georgetown University Press, 2008), 204.

Disclaimer: The views and opinions expressed or implied in the Journal are those of the authors and should not be construed as carrying the official sanction of the Department of Defense, Air Force, Air Education and Training Command, Air University, or other agencies or departments of the US government.

This article was previously published in the Air & Space Power Journal. Coauthored by Robert D. Folker, Jr. and Kyle Benjamin Bressette.