TDD without pay-off

[John Roth]

"Gerold Keefer" <gkeefer@avoca-vsm.com> wrote in message
news:3EFDFF81.769B951A@avoca-vsm.com... hello *, for an interesting experiment on TDD or test-first, have a look at http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf . you will read at the bottom: "Test-first pays off only slightly in terms of increased reliability. In fact, there were five programs developed with test-first with a reliability over 96% compared to one program in the control group. But this result is blurred by the large variance of the data-points. Concentrating on the program versions after the implementation-phase, the result just turns around. The test-first group has signifcantly less reliable programs than the control group. So far, we do not know, if this effect is caused by a false sense of security, less importance of the acceptance-test for the test-first group, or if it is quite simply a result of too little testing." the study implies that it is at least unclear whether test-first is benificial, but certainly the whole XP consulting crowd sells it - on good trust. regards, gerold

The researcher says specifically:

{begin quote]
"One of the challenges
of studying test-first is its embedding within XP. This
embedding makes it difficult to show the effects of test-first
without being blurred by other practices such as pair programming
or a simple design. A solution to this problem would be an
experiment in which XP is applied twice: with test-first and
without test-first. But this kind of experiment is too difficult
and too expensive. To solve this problem, test-first was
extracted from XP and evaluated on its own.

[end quote]

I believe this is the same researcher as the pair programming
study you brought up earlier. She seems to have a plan to
study the components in isolation, and then put them together
later.

I'm quite looking forward to how she does that, given her
disclaimer quoted above. It should be interesting.

I'm still waiting for a credible academic study of test
driven development, with all practices in place and
well practiced before starting out.

I am not, however, holding my breath.


John Roth

[Phlip]

> I am not, however, holding my breath.

How about this: Every time Gerold, searching the 'net for evidence against
XP, happens to finds evidence pro-XP, he /also/ posts a link to it here.

DHYB.

--
Phlip

[Gerold Keefer]

how about this:

not too long ago you titled me
"the guy how made comp.software.extreme-programming useless."
in a yahoo group.
my "evidence pro-XP" seems not to convince everyone and the
reason for this is that facts are immune to propaganda.

just try again phlip. you certainly make it by iteration 8593.

regards,

gerold

Phlip wrote:
I am not, however, holding my breath. How about this: Every time Gerold, searching the 'net for evidence against XP, happens to finds evidence pro-XP, he /also/ posts a link to it here. DHYB. -- Phlip

[Gerold Keefer]

John Roth wrote:
I believe this is the same researcher as the pair programming study you brought up earlier. She seems to have a plan to study the components in isolation, and then put them together later.

a research principle called "bottom-up analysis" is based upon the
idea that you start the investigation with parts of a problem.
this principle had some success rate in the past.
I'm quite looking forward to how she does that, given her disclaimer quoted above. It should be interesting. I'm still waiting for a credible academic study of test driven development, with all practices in place and well practiced before starting out.

i am still waiting for a credible project with all practices in place
and well pracitced until the successful end of a project.
I am not, however, holding my breath. John Roth

regards,

gerold

[David Lightstone]

"John Roth" <johnroth@ameritech.net> wrote in message
news:vfscu8n9f68f29@news.supernews.com... "Gerold Keefer" <gkeefer@avoca-vsm.com> wrote in message news:3EFDFF81.769B951A@avoca-vsm.com... hello *, for an interesting experiment on TDD or test-first, have a look at http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf . you will read at the bottom: "Test-first pays off only slightly in terms of increased reliability. In fact, there were five programs developed with test-first with a reliability over 96% compared to one program in the control group. But this result is blurred by the large variance of the data-points. Concentrating on the program versions after the implementation-phase, the result just turns around. The test-first group has signifcantly less reliable programs than the control group. So far, we do not know, if this effect is caused by a false sense of security, less importance of the acceptance-test for the test-first group, or if it is quite simply a result of too little testing." the study implies that it is at least unclear whether test-first is benificial, but certainly the whole XP consulting crowd sells it - on good trust. regards, gerold The researcher says specifically: {begin quote] "One of the challenges of studying test-first is its embedding within XP. This embedding makes it difficult to show the effects of test-first without being blurred by other practices such as pair programming or a simple design. A solution to this problem would be an experiment in which XP is applied twice: with test-first and without test-first. But this kind of experiment is too difficult and too expensive. To solve this problem, test-first was extracted from XP and evaluated on its own. [end quote]
I believe this is the same researcher as the pair programming study you brought up earlier. She seems to have a plan to study the components in isolation, and then put them together later. I'm quite looking forward to how she does that, given her disclaimer quoted above. It should be interesting. I'm still waiting for a credible academic study of test driven development, with all practices in place and well practiced before starting out. I am not, however, holding my breath.

I find it rather amusing, practices that alledgedly serves to create
testable software, are difficult to test


John Roth

[Michael Feathers]

"David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message
news:nCALa.1819$%a.409@newssvr17.news.prodigy.com... "John Roth" <johnroth@ameritech.net> wrote in message news:vfscu8n9f68f29@news.supernews.com... "Gerold Keefer" <gkeefer@avoca-vsm.com> wrote in message news:3EFDFF81.769B951A@avoca-vsm.com... hello *, for an interesting experiment on TDD or test-first, have a look at http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf . you will read at the bottom: "Test-first pays off only slightly in terms of increased reliability. In fact, there were five programs developed with test-first with a reliability over 96% compared to one program in the control group. But this result is blurred by the large variance of the data-points. Concentrating on the program versions after the implementation-phase, the result just turns around. The test-first group has signifcantly less reliable programs than the control group. So far, we do not know, if this effect is caused by a false sense of security, less importance of the acceptance-test for the test-first group, or if it is quite simply a result of too little testing." the study implies that it is at least unclear whether test-first is benificial, but certainly the whole XP consulting crowd sells it - on good trust. regards, gerold The researcher says specifically: {begin quote] "One of the challenges of studying test-first is its embedding within XP. This embedding makes it difficult to show the effects of test-first without being blurred by other practices such as pair programming or a simple design. A solution to this problem would be an experiment in which XP is applied twice: with test-first and without test-first. But this kind of experiment is too difficult and too expensive. To solve this problem, test-first was extracted from XP and evaluated on its own. [end quote] I believe this is the same researcher as the pair programming study you brought up earlier. She seems to have a plan to study the components in isolation, and then put them together later. I'm quite looking forward to how she does that, given her disclaimer quoted above. It should be interesting. I'm still waiting for a credible academic study of test driven development, with all practices in place and well practiced before starting out. I am not, however, holding my breath. I find it rather amusing, practices that alledgedly serves to create testable software, are difficult to test

Well, it is a step forward. Testing only the things that are easy to test
and hard to do, doesn't do anyone much good.

Michael Feathers
www.objectmentor.com

[David Lightstone]

"Michael Feathers" <mfeathers@objectmentorNOSPAM.com> wrote in message
news:bdmlrn$9s4$1@slb9.atl.mindspring.net... "David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message news:nCALa.1819$%a.409@newssvr17.news.prodigy.com... "John Roth" <johnroth@ameritech.net> wrote in message news:vfscu8n9f68f29@news.supernews.com... "Gerold Keefer" <gkeefer@avoca-vsm.com> wrote in message news:3EFDFF81.769B951A@avoca-vsm.com... > hello *, > > for an interesting experiment on TDD or test-first, have a look at > http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf . > you will read at the bottom: > > "Test-first pays off only slightly in terms of increased
reliability. > In fact, there were five programs developed with test-first with a > reliability over 96% compared to one program in the control group. > But this result is blurred by the large variance of the data-points. > Concentrating on the program versions after the
implementation-phase, > the result just turns around. The test-first group has signifcantly > less reliable programs than the control group. So far, we do not
know, > if this effect is caused by a false sense of security, less
importance > of the acceptance-test for the test-first group, or if it is quite > simply a result of too little testing." > > the study implies that it is at least unclear whether test-first is > benificial, > but certainly the whole XP consulting crowd sells it - on good
trust. > > regards, > > gerold The researcher says specifically: {begin quote] "One of the challenges of studying test-first is its embedding within XP. This embedding makes it difficult to show the effects of test-first without being blurred by other practices such as pair programming or a simple design. A solution to this problem would be an experiment in which XP is applied twice: with test-first and without test-first. But this kind of experiment is too difficult and too expensive. To solve this problem, test-first was extracted from XP and evaluated on its own. [end quote] I believe this is the same researcher as the pair programming study you brought up earlier. She seems to have a plan to study the components in isolation, and then put them together later. I'm quite looking forward to how she does that, given her disclaimer quoted above. It should be interesting. I'm still waiting for a credible academic study of test driven development, with all practices in place and well practiced before starting out. I am not, however, holding my breath. I find it rather amusing, practices that alledgedly serves to create testable software, are difficult to test Well, it is a step forward. Testing only the things that are easy to test and hard to do, doesn't do anyone much good.

A better step would be to make the practices testable
Michael Feathers www.objectmentor.com

[Michael Feathers]

"David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message
news:YvCLa.352$1F1.45858867@newssvr15.news.prodigy.com... "Michael Feathers" <mfeathers@objectmentorNOSPAM.com> wrote in message news:bdmlrn$9s4$1@slb9.atl.mindspring.net... "David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in
message news:nCALa.1819$%a.409@newssvr17.news.prodigy.com... "John Roth" <johnroth@ameritech.net> wrote in message news:vfscu8n9f68f29@news.supernews.com... > > "Gerold Keefer" <gkeefer@avoca-vsm.com> wrote in message > news:3EFDFF81.769B951A@avoca-vsm.com... > > hello *, > > > > for an interesting experiment on TDD or test-first, have a look at > > http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf . > > you will read at the bottom: > > > > "Test-first pays off only slightly in terms of increased reliability. > > In fact, there were five programs developed with test-first with a > > reliability over 96% compared to one program in the control group. > > But this result is blurred by the large variance of the
data-points. > > Concentrating on the program versions after the implementation-phase, > > the result just turns around. The test-first group has
signifcantly > > less reliable programs than the control group. So far, we do not know, > > if this effect is caused by a false sense of security, less importance > > of the acceptance-test for the test-first group, or if it is quite > > simply a result of too little testing." > > > > the study implies that it is at least unclear whether test-first
is > > benificial, > > but certainly the whole XP consulting crowd sells it - on good trust. > > > > regards, > > > > gerold > > The researcher says specifically: > > {begin quote] > "One of the challenges > of studying test-first is its embedding within XP. This > embedding makes it difficult to show the effects of test-first > without being blurred by other practices such as pair programming > or a simple design. A solution to this problem would be an > experiment in which XP is applied twice: with test-first and > without test-first. But this kind of experiment is too difficult > and too expensive. To solve this problem, test-first was > extracted from XP and evaluated on its own. > > [end quote] > > I believe this is the same researcher as the pair programming > study you brought up earlier. She seems to have a plan to > study the components in isolation, and then put them together > later. > > I'm quite looking forward to how she does that, given her > disclaimer quoted above. It should be interesting. > > I'm still waiting for a credible academic study of test > driven development, with all practices in place and > well practiced before starting out. > > I am not, however, holding my breath. I find it rather amusing, practices that alledgedly serves to create testable software, are difficult to test Well, it is a step forward. Testing only the things that are easy to
test and hard to do, doesn't do anyone much good. A better step would be to make the practices testable

A better step would be to try them and form your own conclusions.

Development methods are notoriously difficult to test. Like everything else
in an economy, there is a large opportunity cost for doing that testing, so
generally people weigh risks, apply judgement and try things.

I don't know anyone offhand who only uses practices that they've found good
experimental results for in academic journals. In fact, I suspect you might
even use some practices that don't have that basis. Maybe I am wrong.

Michael Feathers
www.objectmentor.com

[David Lightstone]

"Michael Feathers" <mfeathers@objectmentorNOSPAM.com> wrote in message
news:bdn0sb$3ob$1@slb4.atl.mindspring.net... "David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message news:YvCLa.352$1F1.45858867@newssvr15.news.prodigy.com... "Michael Feathers" <mfeathers@objectmentorNOSPAM.com> wrote in message news:bdmlrn$9s4$1@slb9.atl.mindspring.net... "David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message news:nCALa.1819$%a.409@newssvr17.news.prodigy.com... > > "John Roth" <johnroth@ameritech.net> wrote in message > news:vfscu8n9f68f29@news.supernews.com... > > > > "Gerold Keefer" <gkeefer@avoca-vsm.com> wrote in message > > news:3EFDFF81.769B951A@avoca-vsm.com... > > > hello *, > > > > > > for an interesting experiment on TDD or test-first, have a look
at > > > http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf . > > > you will read at the bottom: > > > > > > "Test-first pays off only slightly in terms of increased reliability. > > > In fact, there were five programs developed with test-first with
a > > > reliability over 96% compared to one program in the control
group. > > > But this result is blurred by the large variance of the data-points. > > > Concentrating on the program versions after the implementation-phase, > > > the result just turns around. The test-first group has signifcantly > > > less reliable programs than the control group. So far, we do not know, > > > if this effect is caused by a false sense of security, less importance > > > of the acceptance-test for the test-first group, or if it is
quite > > > simply a result of too little testing." > > > > > > the study implies that it is at least unclear whether test-first is > > > benificial, > > > but certainly the whole XP consulting crowd sells it - on good trust. > > > > > > regards, > > > > > > gerold > > > > The researcher says specifically: > > > > {begin quote] > > "One of the challenges > > of studying test-first is its embedding within XP. This > > embedding makes it difficult to show the effects of test-first > > without being blurred by other practices such as pair programming > > or a simple design. A solution to this problem would be an > > experiment in which XP is applied twice: with test-first and > > without test-first. But this kind of experiment is too difficult > > and too expensive. To solve this problem, test-first was > > extracted from XP and evaluated on its own. > > > > [end quote] > > > > > I believe this is the same researcher as the pair programming > > study you brought up earlier. She seems to have a plan to > > study the components in isolation, and then put them together > > later. > > > > I'm quite looking forward to how she does that, given her > > disclaimer quoted above. It should be interesting. > > > > I'm still waiting for a credible academic study of test > > driven development, with all practices in place and > > well practiced before starting out. > > > > I am not, however, holding my breath. > > I find it rather amusing, practices that alledgedly serves to
create > testable software, are difficult to test Well, it is a step forward. Testing only the things that are easy to test and hard to do, doesn't do anyone much good. A better step would be to make the practices testable A better step would be to try them and form your own conclusions. Development methods are notoriously difficult to test.

That is why they are called Methods, rather than Methodologies.
Like everything else in an economy, there is a large opportunity cost for doing that testing,
so generally people weigh risks, apply judgement and try things.

That is what people do. It is indeed a very individual type of thing. I am
often surprised by the frequency with this the success of an individual is
used to justify the applicability of the results to others.
I don't know anyone offhand who only uses practices that they've found
good experimental results for in academic journals. In fact, I suspect you
might even use some practices that don't have that basis. Maybe I am wrong.

I can assure you that you are correct about my using practices which lack
experimental validity evidence. When you see me advocating the universal
adoption of those practices, do let me know
Michael Feathers www.objectmentor.com

[Michael Feathers]

"David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message
news:M%DLa.1828$pl.1812@newssvr17.news.prodigy.com... "Michael Feathers" <mfeathers@objectmentorNOSPAM.com> wrote in message news:bdn0sb$3ob$1@slb4.atl.mindspring.net... "David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in
message news:YvCLa.352$1F1.45858867@newssvr15.news.prodigy.com... "Michael Feathers" <mfeathers@objectmentorNOSPAM.com> wrote in message news:bdmlrn$9s4$1@slb9.atl.mindspring.net... > > "David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message > news:nCALa.1819$%a.409@newssvr17.news.prodigy.com... > > > > "John Roth" <johnroth@ameritech.net> wrote in message > > news:vfscu8n9f68f29@news.supernews.com... > > > > > > "Gerold Keefer" <gkeefer@avoca-vsm.com> wrote in message > > > news:3EFDFF81.769B951A@avoca-vsm.com... > > > > hello *, > > > > > > > > for an interesting experiment on TDD or test-first, have a
look at > > > > http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf . > > > > you will read at the bottom: > > > > > > > > "Test-first pays off only slightly in terms of increased reliability. > > > > In fact, there were five programs developed with test-first
with a > > > > reliability over 96% compared to one program in the control group. > > > > But this result is blurred by the large variance of the data-points. > > > > Concentrating on the program versions after the implementation-phase, > > > > the result just turns around. The test-first group has signifcantly > > > > less reliable programs than the control group. So far, we do
not know, > > > > if this effect is caused by a false sense of security, less importance > > > > of the acceptance-test for the test-first group, or if it is quite > > > > simply a result of too little testing." > > > > > > > > the study implies that it is at least unclear whether
test-first is > > > > benificial, > > > > but certainly the whole XP consulting crowd sells it - on good trust. > > > > > > > > regards, > > > > > > > > gerold > > > > > > The researcher says specifically: > > > > > > {begin quote] > > > "One of the challenges > > > of studying test-first is its embedding within XP. This > > > embedding makes it difficult to show the effects of test-first > > > without being blurred by other practices such as pair
programming > > > or a simple design. A solution to this problem would be an > > > experiment in which XP is applied twice: with test-first and > > > without test-first. But this kind of experiment is too difficult > > > and too expensive. To solve this problem, test-first was > > > extracted from XP and evaluated on its own. > > > > > > [end quote] > > > > > > > > I believe this is the same researcher as the pair programming > > > study you brought up earlier. She seems to have a plan to > > > study the components in isolation, and then put them together > > > later. > > > > > > I'm quite looking forward to how she does that, given her > > > disclaimer quoted above. It should be interesting. > > > > > > I'm still waiting for a credible academic study of test > > > driven development, with all practices in place and > > > well practiced before starting out. > > > > > > I am not, however, holding my breath. > > > > I find it rather amusing, practices that alledgedly serves to create > > testable software, are difficult to test > > Well, it is a step forward. Testing only the things that are easy
to test > and hard to do, doesn't do anyone much good. A better step would be to make the practices testable A better step would be to try them and form your own conclusions. Development methods are notoriously difficult to test. That is why they are called Methods, rather than Methodologies. Like everything else in an economy, there is a large opportunity cost for doing that testing, so generally people weigh risks, apply judgement and try things. That is what people do. It is indeed a very individual type of thing. I am often surprised by the frequency with this the success of an individual is used to justify the applicability of the results to others.

I don't know of any cases off-hand. Usually, someone tries something and if
works for them, they mention it to other people and they see if it works for
them, so it is a matter of a group predicting applicability based on their
experiences.
I don't know anyone offhand who only uses practices that they've found good experimental results for in academic journals. In fact, I suspect you might even use some practices that don't have that basis. Maybe I am wrong. I can assure you that you are correct about my using practices which lack experimental validity evidence. When you see me advocating the universal adoption of those practices, do let me know

Actually, would you let me know? If you feel confident about some practice
that I haven't tried. Let me know. I'd like to try it out.

Michael Feathers
www.objectmentor.com

[John Roth]

"David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in
message news:nCALa.1819$%a.409@newssvr17.news.prodigy.com... "John Roth" <johnroth@ameritech.net> wrote in message news:vfscu8n9f68f29@news.supernews.com... "Gerold Keefer" <gkeefer@avoca-vsm.com> wrote in message news:3EFDFF81.769B951A@avoca-vsm.com... hello *, for an interesting experiment on TDD or test-first, have a look at http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf . you will read at the bottom: "Test-first pays off only slightly in terms of increased
reliability. In fact, there were five programs developed with test-first with a reliability over 96% compared to one program in the control group. But this result is blurred by the large variance of the
data-points. Concentrating on the program versions after the
implementation-phase, the result just turns around. The test-first group has
signifcantly less reliable programs than the control group. So far, we do not
know, if this effect is caused by a false sense of security, less
importance of the acceptance-test for the test-first group, or if it is quite simply a result of too little testing." the study implies that it is at least unclear whether test-first
is benificial, but certainly the whole XP consulting crowd sells it - on good
trust. regards, gerold The researcher says specifically: {begin quote] "One of the challenges of studying test-first is its embedding within XP. This embedding makes it difficult to show the effects of test-first without being blurred by other practices such as pair programming or a simple design. A solution to this problem would be an experiment in which XP is applied twice: with test-first and without test-first. But this kind of experiment is too difficult and too expensive. To solve this problem, test-first was extracted from XP and evaluated on its own. [end quote] I believe this is the same researcher as the pair programming study you brought up earlier. She seems to have a plan to study the components in isolation, and then put them together later. I'm quite looking forward to how she does that, given her disclaimer quoted above. It should be interesting. I'm still waiting for a credible academic study of test driven development, with all practices in place and well practiced before starting out. I am not, however, holding my breath. I find it rather amusing, practices that alledgedly serve to create testable software, are difficult to test

I wouldn't say difficult. I'd say expensive. In the referenced
experiments, the experimenter specifically mentioned the
expense factor for doing a complete test with all practices
in place. It's not trivial, and it's way beyond the budget
that most academic departments have availible for research.

To speculatively quantify an experiment: let's assume that
it goes for ten four-hour work sessions, with ten developers
in each group, spread over two weeks. Each developer
is required to have at least one year production
experiance with the technologies involved. At 50USD per
hour, that starts out at $40,000 just to pay the subjects.
You also have to set up the experimental facility, create
and validate the problem and acceptance tests, and
evaluate the results. I'd be surprised if such an experiment
would come in at much under 100,000USD.

Properly designed, such an experiment would be
quite convincing, which the current spate of experiments
are not.

Where's John Veris verTipton when you need him?

John Roth

John Roth

[David Lightstone]

"John Roth" <johnroth@ameritech.net> wrote in message
news:vfu82020kgg113@news.supernews.com... "David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message news:nCALa.1819$%a.409@newssvr17.news.prodigy.com... "John Roth" <johnroth@ameritech.net> wrote in message news:vfscu8n9f68f29@news.supernews.com... "Gerold Keefer" <gkeefer@avoca-vsm.com> wrote in message news:3EFDFF81.769B951A@avoca-vsm.com... > hello *, > > for an interesting experiment on TDD or test-first, have a look at > http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf . > you will read at the bottom: > > "Test-first pays off only slightly in terms of increased reliability. > In fact, there were five programs developed with test-first with a > reliability over 96% compared to one program in the control group. > But this result is blurred by the large variance of the data-points. > Concentrating on the program versions after the implementation-phase, > the result just turns around. The test-first group has signifcantly > less reliable programs than the control group. So far, we do not know, > if this effect is caused by a false sense of security, less importance > of the acceptance-test for the test-first group, or if it is quite > simply a result of too little testing." > > the study implies that it is at least unclear whether test-first is > benificial, > but certainly the whole XP consulting crowd sells it - on good trust. > > regards, > > gerold The researcher says specifically: {begin quote] "One of the challenges of studying test-first is its embedding within XP. This embedding makes it difficult to show the effects of test-first without being blurred by other practices such as pair programming or a simple design. A solution to this problem would be an experiment in which XP is applied twice: with test-first and without test-first. But this kind of experiment is too difficult and too expensive. To solve this problem, test-first was extracted from XP and evaluated on its own. [end quote] I believe this is the same researcher as the pair programming study you brought up earlier. She seems to have a plan to study the components in isolation, and then put them together later. I'm quite looking forward to how she does that, given her disclaimer quoted above. It should be interesting. I'm still waiting for a credible academic study of test driven development, with all practices in place and well practiced before starting out. I am not, however, holding my breath. I find it rather amusing, practices that alledgedly serve to create testable software, are difficult to test I wouldn't say difficult. I'd say expensive. In the referenced experiments, the experimenter specifically mentioned the expense factor for doing a complete test with all practices in place. It's not trivial, and it's way beyond the budget that most academic departments have availible for research.

You may call it expensive if you wish, I definitely ment Difficult. Clearly
the developers did not consider its formulation as an example of TDD
John Roth

[Greg Bacon]

In article <bdn0sb$3ob$1@slb4.atl.mindspring.net>,
Michael Feathers <mfeathers@objectmentorNOSPAM.com> wrote:

: "David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message
: news:YvCLa.352$1F1.45858867@newssvr15.news.prodigy.com...
:
: > A better step would be to make the practices testable
:
: A better step would be to try them and form your own conclusions.
:
: Development methods are notoriously difficult to test. Like
: everything else in an economy, there is a large opportunity cost for
: doing that testing, so generally people weigh risks, apply judgement
: and try things.

Austrian school economists reject what they call mathematization of
economics -- e.g., the construction of economic models -- because they
claim the resulting formulae are insufficient oversimplifications and,
thus, poor analytical tools. For more, see "Why Austrian Economics
Matters" by Lew Rockwell (http://www.mises.org/why_ae.asp).

In *The Ultimate Foundation of Economic Science*, Ludwig von Mises wrote

. . . no scientific method can succeed in determining
how definite external events, liable to a description
by the methods of the natural sciences, produce within
the human mind definite ideas, value judgments, and
volitions. In this sense the individual that cannot
be dissolved into components is both the starting point
and the ultimate given of all endeavors to deal with
human action.

http://www.mises.org/ufofes/ch5~5.asp

In other words, some scholars would say the effectiveness of software
methods and processes are *impossible* to test in general, which means
that any given method won't work for everyone.

: [...]

I have no problem admitting the appeal of TDD and other XP practices
comes from my subjective value judgments such as FINALLY having a
backscratcher to reach those pesky itches of developing for years with
no feedback, integration nightmares, terrible testing, schedules plucked
from the air, etc.

Greg
--
These days it's slightly different. You give the Web site your privacy
and your spam-vulnerable email address ... it gives you a cookie. What
is it good for? Don't worry about it. Good user, have a cookie.
-- Internet Oracularities 1308-01

[Greg Bacon]

In article <bdnr72$kfh$1@slb5.atl.mindspring.net>,
Michael Feathers <mfeathers@objectmentorNOSPAM.com> wrote:

: Greg, thanks for the references. Memories of my libertarian youth
: came rushing back when I read Mises' name.

It could be worse. At least your own publications don't criticize
you. :-) See, for example

http://www.gold-eagle.com/greenspan041998.html

Greg
--
Freedom of men under government is . . . a liberty to follow my own will
in all things, when the rule prescribes not, and not to be subject to the
inconstant, unknown, arbitrary will of another man.
-- John Locke

[Michael Feathers]

"Greg Bacon" <gbacon@hiwaay.net> wrote in message
news:vg0lgni3k2h258@corp.supernews.com... In article <bdnr72$kfh$1@slb5.atl.mindspring.net>, Michael Feathers <mfeathers@objectmentorNOSPAM.com> wrote: : Greg, thanks for the references. Memories of my libertarian youth : came rushing back when I read Mises' name. It could be worse. At least your own publications don't criticize you. :-) See, for example http://www.gold-eagle.com/greenspan041998.html

Haha! But he has learned. IMHO, Greenspan is the best president we've
recently had in this country :-)

Michael