Quantstamp recently unveiled their new website along with a number of new features and the publication of a range of public audit reports. 169 publicly available reports (at the time of writing) to be precise. Every audited contract that Quantstamp makes available to the public, receives a report overview:

Fig.1: Smart contract audit report

Sidenote: You might have noticed that the score on the left and the table on the right show different sorting (left starts with ASF, right with REE etc.). Personally, it really irks me, so if anyone can forward that to the team, that’d be great.

This is indeed a quick and easy to read overview for each individual contract. But I wanted to see the bigger picture and aggregated the available reports, made some colorful graphs and draw some conclusions.

First things first

Before we start with the colorful graphs, I want to introduce the vulnerability types as defined by Quantstamp:

Re-Entracy (REE): A critical flaw where one contract exploits the execution state of another contract.

Self-Destructing Contract (SDC): A flaw in how a library contract delegates its functions to smart contracts that invoke it.

Transaction-Ordering Dependence (TOD): An uncommon flaw that allows a miner to manipulate a transaction’s output by its timestamp.

Timestamp Dependency (TID): A bug that changes the result of a transaction depending on when it executes within a block.

Assertion Failure (ASF): An indication that another, potentially critical flaw occurred upstream.

Quantstamp also has a severity and risk scale for the different types, ranging from low to critical.

Give me the promised colorful graphs

The public reports speak a very clear language, figure 1 shows the total number of vulnerabilities found in the 169 contracts:

Fig.1: Total amount of vulnerabilities found in the available reports

Sidenote: To avoid writing vulnerability so much that it looses all meaning to me, from now on the abbreviation of a vulnerability type will have to suffice.

520 vulnerabilities were found for the audited 169 available, distinct reports. However, as one can clearly see that ASF dominate the findings, making up a grand 86.15% of found vulnerabilities. TOD make up 11.5% while REE and TID only constitute 1.15%. SDC were not found at all.

As we all know from our statistics classes, an average does not mean a lot if we if do not look at the spread. Figure 2 shows the number of vulnerabilities per contract:

Fig.2: Vulnerabilities / Contract

The most interesting number is probably the very first, 73 of audited contracts (43.2%) showed not vulnerability at all. On the other hand, that means that more than half of audited contracts (56.8%) showed at least 1 vulnerability.

Out of the 96 that did reveal to be vulnerable, 23 (24%) contracts showed 1–2 vulnerabilities, followed by 35 (36.5%) contracts having 3–5 and 28 (29.2%) having between 6–10 vulnerabilities. 10 (10.4%)contracts even showed more than 10 vulnerabilities.

The average is 3 and the median are 2 vulnerabilities per contract whereas the maximum count in a single contract was 20.

Keeping in mind that ASF amounted to 86.15% of all vulnerabilities, I wanted to see how many times each vulnerability type occurred alone. Figure 3 shows nicely that when ASF are found, they usually occurred without other types:

Fig.3: Percentage of distinct vulnerabilities

If your contract will only have a single type of vulnerability, it will most likely be an ASF. That is based on a total of 77 contracts that showed only a single type of vulnerability, representing 45.6% of total contracts and 80.2% of vulnerable contracts.

Auditing your smart contract is worth it

If you develop a smart contract, you are more likely than not to have written a vulnerable smart contract. There is a 56.8% chance of your contract showing at least 1 vulnerability. If your contract is vulnerable, you will probably have around 3 of the buggers and most probably there will be at least 1 assertion failure in your contract.

However, besides the raw data that is not too bad, I believe that we are on a good path regarding smart contract security. Solidity and the EVM are relatively new and not too well chartered territory. As adoption will increase, so will the knowledge-base as well as organizational security research and communication. Quantstamp and other providers represent an affordable and easy way to enhance smart contract security and thereby blockchain adoption as a whole.

PS: Sorry for making you read vulnerability so many times. If a “vulnerability swear jar” existed, I’d be a poor man.

PPS: Breakdown of contracts submitted per user as per sdmike’s suggestion: