We all have our opinions on static strong typing versus dynamic or even no real typing at all, but hard evidence of what works is thin on the ground. Now some new research gives us some facts to throw into the debate. 

How can you prove, or at least get closer to the truth about whether, a particular technique reduces bugs or not? You can't really compare code bases created with and without the technique and then count the bugs because too much else varies - the programmers, the task. the bug detection rate. and so on. 

Zheng Gao and Earl T. Barr at University College London and Christian Bird at Microsoft Research have a good idea how to do the job. As most projects use a source management system of some sort, it should be possible to go back to the code just before a bug was fixed and see if enabling type checking finds the bug in the unfixed code. 

The only problem with this is that for most languages you can't switch strong typing on and off - it is a feature of the language that you cannot disable. However, languages like JavaScript are largely type-free but there are type checking systems that can be added to them. For example Facebook's Flow will type-check a JavaScript program and so will Microsoft's TypeScript compiler. There are some complications, you have to add type annotations manually, but with a little effort it seems to be possible to create a fair test.

So the method is roughly:

• find a bug
• go back to the code base just before the bug
• add type annotations to the code
• see if the bug is flagged by the type system tool in use.

This basic idea is that if the bug is flagged by the type-checking tool then it would have been found by the programmers before it got into the code base and would never have existed.

So what was the conclusion?

Both type checking approaches, Flow and TypeScript, detected 15% of the bugs. That is, out of 400 bugs Flow detected 57 and TypeScript detected 3 that Flow missed and vice versa.

This may not seem a great deal, but as the paper points out this is a conservative estimate of the bugs that strong typing might detect. The 400 bugs had survived testing and code reviews and so were slightly harder than bugs that were removed earlier in the process. Even so the researchers quote an unnamed engineering manager at Microsoft's response:

“That’s shocking. If you could make a change to the way we do development that would reduce the number of bugs being checked in by 10% or more overnight, that’s a no-brainer. Unless it doubles development time or something, we’d do it.”

Well this is exactly the point. We do have some hard facts to add to the debate, but the issue isn't how many bugs strong typing finds it is also a question of how much it increases the effort of producing the code. There is also the question of could we put that increased effort to better use. 

There is also the point that Flow in particular is a static type checking tool that can make use of type inference to check the code base. A language doesn't have to be strongly typed to be type-checked in this sense. In addition type-checking is just one form of static code analysis and you can ask the question why just check type?

For example, the histogram of bugs that were undetectable based on type reveals types of bug that would have a chance of showing on a static code analysis:

It seems that we still need more data and the arguments can continue.

In a world where it is still respectable to claim to code in a text editor, no matter how clever, and use the compiler from the command line, it seems that there is a great deal that can be done to improve the programming environment.

We need more tools, not necessarily more restrictive languages.

More Information

To Type or Not to Type: Quantifying Detectable Bugs in JavaScript

Paper

Related Articles

Flow - A Static Type Checker For JavaScript

TypeScript Goes Light, Moves To GitHub        

TypeScript 1.0 Released

Automatic Testing - Programmers Are Still The Problem

Code Digger Finds The Values That Break Your Code

Code Hunt - New Coding Game From Microsoft Research

Debugging and the Experimental Method       

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info