A Challenge for all TDD programmers, ICFP

[Dave Harris]

olczyk@interaccess.com (Thaddeus L Olczyk) wrote (abridged): There is no place in the rules where it says that "The program should be faster than competing programs".

The 3rd (2000) contest was the ray-tracer. Some quotes from the task
description:

Submissions will be evaluated on their correctness, speed
of execution, and set of implemented GML features. [...]

Judging of the contest entries will proceed in three phases. [...]
The third phase will compare the performance and implemented
features of the submissions.

See http://www.cs.cornell.edu/icfp/task.htm for the full text. It's clear
that speed is an important winning criteria, second only to correctness.

The 4th (2001) contest uses speed as a tie-breaker.

Remember that the task is different each year. We don't yet know the 6th
(2003) task. When I wrote: "Usually they are a bit woolly, like...", I was
giving typical examples from previous years. Performance usually plays a
role, partly for practical reasons (they have to get all the entries run
in a reasonable time), and partly because they want to dispell the myth
that functional languages are slow.

One of my favourite comments comes from the "CuttySark" team in 2000.
Their program took over 3 days on one of the tests.

We are proud to be one of (or the only) entries for which
computation of the images takes longer than the programming
contest itself: it was hard to code, it should be hard to run :-).

They were using Python.

Nor does it say "it's robots should beat the other robots".

The 5th (2002) contest task description says,
The winner of a game is the robot who has the highest score at
the end of the game.

I paraphrase this by saying the robot with the highest score beats the
other robots. My robot can beat your robot by getting a higher score on a
given map, even if both robots are never on the map together.

The 4th (2001) contest wasn't about robots, but it also had open-ended
criteria. It was about optimisation; the winning program had to produce
smaller output than the others.

It just another apriori excuse by an incompetent programmer for failing at something.

Actually, I wasn't making an excuse. I was doing the opposite - showing
how the winning criteria were open-ended, making the ICFP contest the kind
of project where test-driven design should excel.

-- Dave Harris, Nottingham, UK

[Dale King]

"Dave Harris" <brangdon@cix.co.uk> wrote in message
news:memo.20030624083517.5515A@brangdon.madasafish.com... It just another apriori excuse by an incompetent programmer for failing at something. Actually, I wasn't making an excuse. I was doing the opposite - showing how the winning criteria were open-ended, making the ICFP contest the kind of project where test-driven design should excel.

But the key to winning the contest is to come up with a strategy that is
better than the strategy that others come up with. In other words it
requires developing a superior AI. There is nothing about TDD that really
aids in the development of the strategy. TDD will help you reliably
translate any strategy you want to try into code and make sure that the code
actually fulfills the strategy you have in mind, but it will not necessarily
lead you to the best strategy.

Think of this simple analogy. Consider if the contest were to write the
fastest sort algorithm for a list of records. If I decide on the strategy of
a bubble sort TDD will help me write a bubble sort that works well and
reliably, but I would still lose the contest to an entrant that uses
quicksort. There is nothing about TDD that will lead me to quicksort instead
of a bubble sort. That takes knowledge and/or insight that is completely
outside of TDD.

--
Dale King

[Dave Harris]

kingd@tmicha.net (Dale King) wrote (abridged): There is nothing about TDD that really aids in the development of the strategy.

Of course not. TDD is not a silver bullet which guarantees success. No-one
claims it is. Which is part of why Thaddeus Olczyk's claim:

If you don't [win the competition], then you have proven that
TDD is so much shit.

is absurd.

-- Dave Harris, Nottingham, UK

[Dale King]

"Dave Harris" <brangdon@cix.co.uk> wrote in message
news:memo.20030625203318.60979D@brangdon.madasafish.com... kingd@tmicha.net (Dale King) wrote (abridged): There is nothing about TDD that really aids in the development of the strategy. Of course not. TDD is not a silver bullet which guarantees success. No-one claims it is. Which is part of why Thaddeus Olczyk's claim: If you don't [win the competition], then you have proven that TDD is so much shit. is absurd.


In a round about way Thaddeus is claiming that others are claiming that it
is a silver bullet. And there has been plenty of talk in this thread about
whether TDD is a factor in the contest. Basically, Thaddeus' claim is that
unless TDD is a silver bullet then it is so much sh**. And yes that is
certainly absurd.

But I was responding to your statement:
Actually, I wasn't making an excuse. I was doing the opposite - showing how the winning criteria were open-ended, making the ICFP contest the kind of project where test-driven design should excel.

TDD is wonderful, but I see nothing about the open endedness of the winning
criteria that says that a TDD will necessarily have an edge. I assume that
by "excel" you meant more likely to win the contest. I see no correlation
between TDD and winning the contest.
--
Dale King

[Uncle Bob (Robert C. Martin)]

"Dale King" <kingd@tmicha.net> might (or might not) have written this
on (or about) Wed, 25 Jun 2003 15:13:35 -0500, :
TDD is wonderful, but I see nothing about the open endedness of the winningcriteria that says that a TDD will necessarily have an edge. I assume thatby "excel" you meant more likely to win the contest. I see no correlationbetween TDD and winning the contest.

One of the goals of TDD is to narrow the histogram.

When we practice TDD we spend less time in the debugger and more time
writing tests and code. This has a stabilizing effect upon how long
it takes to develop something.

If we look at development time as a random variable with a mean and a
sigma, then TDD shrinks the sigma; causing the histogram to huddle
around the mean.

Shrinking the sigma is not a winning strategy in a contest where the
first one done wins. The first one done is likely to be a team who
depends upon luck, and a wide sigma. On the other hand, in the
business world we are more likely to desire predictability than a 10:1
chance of making a date.

In the end, the things that help you to win a contest are not
necessarily the same things that help you to win at project
development.

Robert C. Martin | "Uncle Bob"
Object Mentor Inc.| unclebob @ objectmentor . com
PO Box 5757 | Tel: (800) 338-6716
565 Lakeview Pkwy | Fax: (847) 573-1658 | www.objectmentor.com
Suite 135 | | www.XProgramming.com
Vernon Hills, IL, | Training and Mentoring | www.junit.org
60061 | OO, XP, Java, C++, Python |

[Phlip]

Uncle Bob (Robert C. Martin) wrote:
If we look at development time as a random variable with a mean and a sigma, then TDD shrinks the sigma; causing the histogram to huddle around the mean.

A quick statistics lesson: This is what happened to pro baseball over 100
years.

In the begining, well-off people dabbled in baseball, and their statistics
spread all over the place.

Over time, mechanisms arose to select the most qualified players, and train
them accurately.

Their statistics did not increase, because they competed with their own
peers. Their statistics narrowed to small central ranges.

--
Phlip
http://www.c2.com/cgi/wiki?TestFirstUserInterfaces

[Dirk Thierbach]

"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote: "Dale King" <kingd@tmicha.net> might (or might not) have written this on (or about) Wed, 25 Jun 2003 15:13:35 -0500, :
Shrinking the sigma is not a winning strategy in a contest where the first one done wins.

But in the ICFP contest, the first one done doesn't win. Your program
is judged by correctness, and then by some other criterion (like
performance, or whatever). It doesn't matter when you finish, as
long as you are done before the deadline (which is there more for
practical purposes -- people will be busy with other things during
the week).

Think of it as squeezing one (or maybe more) iteration(s) into a 2.5
days istead of the 10 working days of two weeks. This is a bit more
extreme then XP, but, well...
In the end, the things that help you to win a contest are not necessarily the same things that help you to win at project development.

Sure. Nobody is arguing that (well, nobody should :-). OTOH, it might
be interesting to watch how the things that help you win at project
development turn out to help in a somewhat more "extreme" situation.

[F'up to c.s.e-p]

- Dirk

[Dave Harris]

kingd@tmicha.net (Dale King) wrote (abridged): I assume that by "excel" you meant more likely to win the contest. I see no correlation between TDD and winning the contest.

I think teams using TDD are more likely to win the contests than teams
which don't - especially if the latter are not really using anything at
all, which seems to be the case from what I have read. In other words, I
think TDD brings enough short-term benefit to outweigh the admitted
short-term cost.

Earlier you mentioned the importance of strategy. I agree with that, but
it's also important to implement the strategy well, and with confidence
and speed.

It's harder to tell why teams lose than why they win, because the losers
are less eager to write up their results. But still, it seems to me that
some teams are either unwilling to make big changes late in the day,
because they are not confident they can do so without screwing up; or else
they /do/ go ahead, make the big change and duly screw up. If you can
avoid failing in those two ways, you can improve your chances of doing
well in the competition. And having a good test suite will help with that.

If the competition were, say, 3 hours instead of 72, or was not so well
designed, then this wouldn't be true. In that case a team which normally
used TDD design might be well-advised to abandon it, for the competition.
Mind you, that would make some people unable to compete at all, just as
the stereotypical Italian can't talk if you tie his hands behind his back.

-- Dave Harris, Nottingham, UK

[Costin Cozianu]

Dave Harris wrote: kingd@tmicha.net (Dale King) wrote (abridged):I assume that by "excel" you meant more likely to win the contest. Isee no correlation between TDD and winning the contest. I think teams using TDD are more likely to win the contests than teams which don't - especially if the latter are not really using anything at all, which seems to be the case from what I have read. In other words, I think TDD brings enough short-term benefit to outweigh the admitted short-term cost.

Is there any correlation between your thinking and facts ?

Another thing, generally many people that have performed well (even
among the winners), have an appetite towards functional programming,
mathematcis, formal methods, and program in languages where xUnit is not
yet been ported, all in all, an antithesis of TDD.

Do you think you know better ?

[Dave Harris]

c_cozianu@hotmail.com (Costin Cozianu) wrote (abridged): I think TDD brings enough short-term benefit to outweigh the admitted short-term cost. Is there any correlation between your thinking and facts ?

Development process is notoriously difficult to test experimentally. I
doubt a single TDD competition entry would be statistically significant.
In the absence of good experimental data, reasonable men may disagree :-)

Another thing, generally many people that have performed well (even among the winners), have an appetite towards functional programming, mathematcis, formal methods, and program in languages where xUnit is not yet been ported, all in all, an antithesis of TDD. Do you think you know better ?

That's an interesting issue. Certainly a good statically typed language
will offset many of the advantages of runtime testing. Having both testing
and good type checking is beneficial, even essential, for the highest
quality software, but I don't know how the balance would work out for a
competition like this.

Here are the kind of rankings I'd expect:

TDD + Smalltalk beats Smalltalk.
TDD + Smalltalk beats C++.
TDD + Smalltalk beats C++ + TDD.
TDD + C++ beats C++.

But the next levels I am not at all sure about:

TDD + Smalltalk beats Haskell ???
TDD + Smalltalk beats TDD + Haskell ???
TDD + Haskell beats Haskell ???

That would be interesting to see. I wonder how your rankings would look?

Incidently, I think people who adopt non-mainstream languages like Haskell
are in some ways similar to people who adopt non-mainstream practices like
TDD. They will be people who have at least thought about their development
process, become aware of non-mainstream ideas, and put at least the effort
into researching them that you need to adopt them. So in both cases they
will tend to be a cut above the mundane programmer. (This is one of the
effects which makes an informal statistical study harder.)

Also, in case the above is too wishy-washy and you want a bigger target to
aim at, I'll say this: if you are going to do a lot of testing anyway, it
is better to do it in advance than in arrears.

-- Dave Harris, Nottingham, UK

[Uncle Bob (Robert C. Martin)]

Dirk Thierbach <dthierbach@gmx.de> might (or might not) have written
this on (or about) Thu, 26 Jun 2003 18:27:49 +0200, :
"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote: "Dale King" <kingd@tmicha.net> might (or might not) have written this on (or about) Wed, 25 Jun 2003 15:13:35 -0500, : Shrinking the sigma is not a winning strategy in a contest where the first one done wins.But in the ICFP contest, the first one done doesn't win.

I wasn't aware of that. Those contests that I have, upon occasion,
participated in, *did* count completion time as one criterion.
Your programis judged by correctness, and then by some other criterion (likeperformance, or whatever). It doesn't matter when you finish, aslong as you are done before the deadline (which is there more forpractical purposes -- people will be busy with other things duringthe week).

Interesting! How do they break a tie?
Think of it as squeezing one (or maybe more) iteration(s) into a 2.5days istead of the 10 working days of two weeks. This is a bit moreextreme then XP, but, well...

We do one day iterations during our XP immersion courses.
In the end, the things that help you to win a contest are not necessarily the same things that help you to win at project development.Sure. Nobody is arguing that (well, nobody should :-).

But somebody did.
OTOH, it mightbe interesting to watch how the things that help you win at projectdevelopment turn out to help in a somewhat more "extreme" situation.

Certainly! It would be very interesting to see how a TDD team fared.
However, depending upon the rules, I might not bet on them.

Robert C. Martin | "Uncle Bob"
Object Mentor Inc.| unclebob @ objectmentor . com
PO Box 5757 | Tel: (800) 338-6716
565 Lakeview Pkwy | Fax: (847) 573-1658 | www.objectmentor.com
Suite 135 | | www.XProgramming.com
Vernon Hills, IL, | Training and Mentoring | www.junit.org
60061 | OO, XP, Java, C++, Python |

[Dirk Thierbach]

"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote: Dirk Thierbach <dthierbach@gmx.de> might (or might not) have written this on (or about) Thu, 26 Jun 2003 18:27:49 +0200, :
But in the ICFP contest, the first one done doesn't win.
I wasn't aware of that. Those contests that I have, upon occasion, participated in, *did* count completion time as one criterion.

Yeah, I know those contests. But the ICFP contest isn't like the ACM
write-a-backtracking-algorithm-quickly contest.

The idea behind it, as I understand it, is more something like a PR
event. FPLs have to fight against a lot of prejudices (like XP has),
one of them is "these are academic languages, you cannot use them for
real world programs". Another one is "these languages are too slow".
So they turned the challenge around in some sense, saying "well, if
your favorite language is better, come on and show us" (to exaggerate
a bit).

As it is impossible to do a "real world" project in a limited
timeframe, you have to settle for some compromise. The contest tasks
(save the last one, which I personally didn't like too much) did have
some "real world" application, but where shrunk down to a size one can
handle during a week-end.

If you are interested, Google should find the tasks and results of the
previous contests -- IIRC they are on different websites for different
years.

And of course the tasks are a bit biased, but not unreasonably so.
There is a big emphasis on correctness (which FPLs are good at; it's
easy to write an FPL program that is guaranteed not to crash),
and all tasks involve non-trivial algorithms (which FPLs are good at,
again; though this feature is not as frequent in the "real world").
Interesting! How do they break a tie?

Once you pass the correctness requirements, you use performance.
In the last one, the programs competed against each other in some
sort of "delivery game" (which is not a good idea IMHO, because it makes
the results rather random.) This year, they announce that the programs
will run on your own computer ... so performance probably does not
play a role. I am not sure how this should work, but we'll see.

There are also "special awards", and a special category of "lightning
entries" (completed in just a day) for those with only little time.
We do one day iterations during our XP immersion courses.

That doesn't surprise me. So XP techniques are certainly usable
for the contest.
Sure. Nobody is arguing that (well, nobody should :-). But somebody did.

Unfortunately.
Certainly! It would be very interesting to see how a TDD team fared. However, depending upon the rules, I might not bet on them.

Maybe some XP people got interested now.

- Dirk

[Richard MacDonald]

"Phlip" <phlipcpp@yahoo.com> wrote in
news:oICKa.368$WT7.217@newssvr31.news.prodigy.com:

Please attribute your source next time :-)

Steven Jay Gould. "On why there are no longer any .400 hitters". (or
something like that.) (Discover magazine, early 80s. Not sure if the
article is online.)

[Phlip]

> Please attribute your source next time :-) Steven Jay Gould. "On why there are no longer any .400 hitters". (or something like that.) (Discover magazine, early 80s. Not sure if the article is online.)

Oh, but I was downstream from someone who didn't attribute (or put it in a
bibliography, or something). Probably PBS.
You're in the wrong business :-)

You mean I should join the multi-billion dollar international poetry
industry?

Here's Weird Al came on the radio just now:

I met him in a swamp down in Dagoba
Where it bubbles all the time like a giant carbonated soda
S O D A, soda

I saw the little runt sitting there on a log
I asked him his name and in a raspy voice he said "Yoda"
Y O D A, Yoda
Yo-yo-yo-yo Yoda

Well, I've been around, but I ain't never seen
A guy who looks like a muppet, but he's wrinkled and green
Oh, my Yoda
Yo-yo-yo-yo Yoda

Well, I'm not dumb, but I can't understand
How he can lift me in the air just by raising his hand
Oh, my Yoda...

http://www.com-www.com/weirdal/yoda.html

--
Phlip
http://www.c2.com/cgi/wiki?TestFirstUserInterfaces