All about software errors Kevin Burke

What we're covering today Where do errors come from?

How often do we make errors?

How costly are errors to fix?

How can we catch more errors?

What effects will increased (testing, code review, design) have on development time?

What makes me an expert?

Methodology Survey a large company (Raytheon, AT&T, IBM, Motorola, etc)

Determine defect rate (survey, interview, or automated collection)

Determine tools used to catch errors (testing, code review etc)

Methodology (cont'd) Ask many programmers to write same program, w/ different emphases

Give programmers a program w/ known errors, ask them to find/fix

Stop using the word "bug"

Examples of bugs Colony of ants infests your laptop

Bird poops on computer & shorts the motherboard

Termites chew through server cables, cause data center outage

Things that are not bugs Syntax errors

Null pointer dereference

Misunderstanding requirements

These are errors/defects

Terminology has implications Bugs are "random", acts of God

Errors can be measured

Error rates can be tracked

Errors can be reduced

You have room to improve

50% of time is spent debugging, refactoring, reworking Mills 1983, Boehm 1987, Cooper and Mullen 1993, Fishman 1996, Haley 1996

Room to improve 10x differences between pro programmers on: Size of completed program

Speed to complete program

Error rate

Error detection rate

Room to improve, cont'd Productivity variation between programmers Sackman, Erickson, Grant, "Exploratory Experimental Studies Comparing Online/Offline Programming Performance", 1968

Room to improve, cont'd Curtis, "Substantiating Programmer Variability", 1981

Room to improve, cont'd Demarco and Lister, "Programmer performance and the effects of the workplace", 1985

Where do errors come from?

What kinds of errors? 18%-36% of errors are clerical errors (Weiss 1975, Card 1987)

(Weiss 1975, Card 1987) The 3 most expensive errors of all time - $1.6 billion, $900 million, $245 million - involved changing a single character in a correct program

What kinds of errors? Most errors (~85%) can be fixed in a short period of time Endres, "An Analysis of Errors and Their Causes in System Programs", 1975 Most errors are the programmer's fault Other common error sources - changing requirements, communication breakdown, thin domain knowledge

What things are correlated with errors? Unused variables (Card, Church, Agresti, 1986)

What things are correlated with errors? High numbers of comments

What things are correlated with errors? Complex control flow (McCabe, "A Complexity Measure", 1976) See: gocyclo 3+ layers of nesting (Yourdon, "Managing the Structured Techniques: Strategies for Software Development in the 1990s", 1986)

How are errors distributed? In general, 80% of the errors come from 20% of the code. Also, 50% of errors from 5% of the code Case study at IBM: 31 of 425 classes found to be error-prone. After repair/refactoring, defects reported by customers were reduced by 90% Capers Jones, "Software Assessments, Benchmarks, and Best Practices," 2000 Capers Jones, "Software Assessments, Benchmarks, and Best Practices," 2000

How often do we make errors? Best estimates: you will find 1-25 errors per 1000 lines of code (McConnell, "Code Complete", 2003) 5-8 defects/hour during coding (Humphrey, "Introduction to the Personal Software Process", 1997)

How many can we expect to find? If you do it well: 70% If you do it poorly: 20%

Effectiveness of different bug-finding tools

What the heck is an inspection?

What the heck is an inspection? Michael Fagan (IBM), "Design and Code Inspections to Reduce Errors in Program Development", 1976 4 roles: Moderator, Author, Reviewer, Scribe

Everyone prepares, brings notes

Solutions not discussed

Management isn't present

Reviewers have checklists for points to cover

Inspections crush testing on effectiveness

Inspections crush testing Basili & Selby 1987: Code reading found 80% more faults/hour than testing

Ackerman, Buchwald & Lewski 1989: 6x as much time to find errors with testing as inspections

Kaplan 1995: 3.5 staff hours/error with inspections, 15-25 hours per error with testing

Moore 1992: Microsoft spent 3 hours/error with inspections, 12 hours with testing

Russell 1991: One hour spent on inspections avoided 33 hours of maintenance

What won't testing catch? Changing requirements/lack of communication

Hard coded variable values

Unclear error messages (or typos)

Duplicated code

Inadequate comments

Leaving verbose logging turned on

If you do test: Automate test procedure 50% of tests run manually were run incorrectly

50% of tests run manually were run incorrectly Double check test code for errors Tests as likely or more likely to contain errors than code (Weiland 1983)

Tests as likely or more likely to contain errors than code (Weiland 1983) Make errors hard to miss Easy to miss erroneous output, use log.Fatal or panic in development

Review small changes

Review small changes In one company, 55% of one line changes were incorrect After code review was implemented, 2% were incorrect Freedman and Weinberg, 1990

"Never debug standing up." Gerald Weinberg

Pull request length

Example

Requirements change Average change is 1-4% per month. (Jones 2000) Store requirements in version control so changes are visible

Keep your own records

Things you can measure: Compiler detected errors

Total number of defects

Errors per 1000 LOC

Mean time between failures

Defect severity

Class/routine that caused defect

Origin of the defect (Design confusion, syntax error, etc)

Cost to correct defect (hours, $$$)

Cost of fixing defects at various stages Requirements defect is 5-10x as expensive once you've begun writing code

once you've begun writing code Architecture defect is 10x as expensive once you've begun writing code

once you've begun writing code Code is 10-25x as expensive after it's been released

after it's been released Requirements/architecture are 10-100x as expensive to fix after deploy Fagan 1976, Dunn 1984, Shull 2002

Personal Error Tracking

/