Like the parable of the six blind men trying to learn what an elephant is by only touching one piece of the creature, regulators are assessing the nation’s cybersecurity efforts by looking at initiatives individually rather than holistically. As with the elephant, selecting one piece of the broader cyber information sharing system and assessing it will never reveal the complete, and vastly more interesting, picture. Piecemeal attempts to isolate, assess and address information sharing efforts could have unintended repercussions on the web of trusted communication of threats at work today. Instead, regulators should step back and take a comprehensive approach factoring in the nuances that have evolved to facilitate information sharing between the public and private sectors.

In the end of June it was reported only six non-government organizations use Automated Indicator Sharing (AIS) to exchange threat indicators with the National Cybersecurity and Communications Integration Center (NCCIC) within the Department of Homeland Security (DHS). Created by the Cybersecurity Information Act of 2015 (CISA), AIS is a key initiative at the NCCIC because it enables the transfer of indicators between DHS and industry at “machine speed” enabling the private sector to learn about threats in near real-time so they can efficiently act. In cybersecurity, response time directly impacts success.

Congress was quick to criticize DHS for what on face value appears to be low participation in AIS. However, the announcement did not name the participating organizations or tell the whole story. It was not mentioned that those six member establishments may be widely connected through their own portals and memberships. In fact, more than 240 domestic and international organizations are connected to DHS’s AIS server. Many of those connections are commercial cybersecurity vendors or information sharing and analysis centers and organizations, which redistribute the indicators to their customers and members in addition to receiving indicators from that same customer base. This means that DHS can receive indicators from its entire customer base.

If legislators looked at the bigger picture of information sharing efforts, they would realize AIS is not the only avenue for sharing information between the public and private sectors at DHS. The department also operates the Enhanced Cybersecurity Services (ECS), Cyber Information Sharing and Collaboration Program (CISCP) and is partnered with all of the Information Sharing and Analysis Centers (ISACs) and Information Sharing and Analysis Organizations (ISAOs). Additionally DHS manages the Protected Critical Infrastructure Information Program (PCII) as well as other interagency activities and working groups dedicated to improving information sharing. Information on global cyber threats also flows into the NCICC through relationships with international CERTs.

With CISA in 2015 being the last significant cybersecurity legislation to address information sharing, stakeholders across the government have taken the initiative to formulate their own systems to share critical information with partners. Officially only an act of Congress can force agencies to share data, relationships have blossomed to move information along. While some of these are formal relationships established at the leadership level, informal relationships at the operational level, where the execution happens, are more prevalent and effective. These are personal relationships built on trust and a commitment to the same mission of keeping the United States and allied nations secure.

These informal relationships are also driving forces in the success of structured sharing methods. Companies have a lot at stake when reporting a cyber attack to the federal government and frequently need more than legal protections to feel comfortable sharing information. This assurance typically comes in the form of personal relationships established over time. To report the incident for the overall good, firms need to feel confident that their government contacts will be respectful of their business risks and they will not be put at a competitive disadvantage for doing the right thing. Frequently it is as simple as trusting the government contact will ensure a company’s name is redacted when the threat is shared.

In the defense and intelligence communities, trust usually needs to be formalized through the security clearance process. For cleared contractors the National Defense Information Sharing and Analysis Center (NDISAC) and the Defense Industrial Base Cybersecurity Information Sharing Program portal are two of the official avenues for passing information on threats between government and industry.

Unfortunately, many of the people in industry who should have clearances are still waiting. It is estimated that about 700,000 people are waiting for security clearances and each clearance on average takes over 500 days to process one person. Getting information to uncleared parties can require the defense and intelligence agencies to work with DHS and other counterparts in the government, but this game of telephone can drastically impact the timeliness and action-ability of the intel. By the time information reaches the target they are already the victim, and in some cases may have already ascertained they were attacked. In the world of clearances, information is accessed on a “need to know basis” and sometimes, especially if there is no formalized trust between the two parties, it may be determined that industry is not invited to the table.

If Congress asked agencies to provide statistics using traditional reporting methods on these varying interactions it would be nearly impossible to accurately represent the real work and progress. A quantitative measure does not equal quality service delivery. Instead of attempting to measure the success of cybersecurity initiatives using statistics, like number of organizations participating in a program, regulators should consider a combined qualitative approach. Quantifiable metrics should instead be used to support the qualitative measures to accurately articulate mission success.

Generic examples of qualitative measures in any medium or higher priority cyber issue:

How quickly does a cyber operations center respond?

With what products or services did the cyber operations center respond?

From the point of the cyber operations center involvement, what was the time to issue resolution?

At the end of the cyber operations center involvement, did the resolution of the issue exceed customers’ expectation?

What action would the customer have preferred?

Using this hybrid method to identify use cases of successful cooperation across the government and with industry will reveal the complex network of communication that exists today. No agency is operating in a vacuum, but rather finding ways to work within the current constraints to share information to give all stakeholders greater context on cyber threats. Legislators will be able to identify real weaknesses and subsequent improvements will reverberate across the national cyber community.

The current government-wide information sharing efforts are far from perfect, but still effective. Cultural impediments may exist and processes may not always be streamlined, but the cyber community has come together to make it work and is constantly making improvements. As with other national security efforts, the public only hears about a cyber incident when things go wrong. For every attack that makes the news there are thousands neutralized through a range of coordination between government and industry–We just need a better way of measuring it.