Share and Enjoy

David Saff's blog of technological joys.

Thursday, December 07, 2006


New paper: The Practice of Theories

I've been enjoying my time working with Agitar to better understand how we think about testing, and how to use those tests to make our tools smarter. We've recently written a paper on an extension to the idea of unit tests and test-driven development. Share and enjoy, and please post your thoughts.

[Copyright notice: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.]

The Practice of Theories: Adding "For-all" Statements to "There-Exists" Tests (David Saff, Marat Boshernitsan) Submitted to IEEE Software, Special Issue on Test-Driven Development.

Abstract: Traditional unit tests in test-driven development compare a few concrete example executions against the developer’s definition of correct behavior. However, a developer knows more about how a program should behave than can be expressed through concrete examples. These general insights can be captured as theories, which precisely express software properties over potentially infinite sets of values. Combining tests with theories allows developers to say what they mean, and guarantee that their code is intuitively correct, with less effort. The consistent format of theories enables automatic tools to generate or discover values that violate these properties, discovering bugs that developers didn't think to test for.

David, I can't believe that you haven't had more comments about this after posting it on the junit list.

Great, thought-provoking article which covered new ground for me and neatly summarised other tangled thoughts that I've had on this subject. Is the Theory Runner available for playing with?

Btw, I do own Mark Pilgrim's book, but DiveIntoPython might be a good citation / reference rather than the published book. Just a thought.

Thanks! The Theory Runner is not yet available for public consumption, but I'm working to make it so (as a side project).

I read Dive Into Python from the website, but (a) from my academic experience, I prefer citing printed material, (b) I thought readers might like the chance to pay Mark for his excellent book.
I think you have a bug in one of your theories:

void hashCodeIsInSyncWithEquals(Object a, Object b) {
assumeEquals(a, b);
assertEquals(a.hashCode(), b.hashCode());

What if a.hashCode() equal to b.hashCode() but a isn't equal to b? How would you fix this? Write another Theory?

How would you test theories? Theories might be more complicated and error prone than other kind of tests aren't they?

I don't believe there's a bug. Giving every object the same hashcode is inefficient but meets the hashCode specification.

If a theory is incorrect because it fails when it should pass, this can be found either with user-provided data, or data generated by automatic test-input-generation tools like JUnit Factory. More difficult is if a theory passes when it should fail--this is a problem with tests, too, but the mental effort required to catch the problem is more.
Post a Comment

<< Home


February 2005   June 2005   March 2006   August 2006   December 2006   April 2007   May 2007   January 2008  

This page is powered by Blogger. Isn't yours?

Subscribe to Posts [Atom]