Reliability and Extreme Programming

[Code Like Hell]

tadamsmar@yahoo.com (Tom Adams) wrote in message news:<ea44f5a1.0306040618.3332d4fb@posting.google.com>... I have noticed XP seems to be largely uninformed about software reliability engineering and I want change that. The advantage for XPers in understanding what I have to say is: More reliable systems with less work. "Reliability" has lots of meanings. To be very specific I am talking about Mean Time to Failure (MTTF) or Mean Time Between Failures. Take a moment to let this sink in... There is a type of testing that is used to achieve and confirm reliability. I see little evidence that XP includes this type of testing in its process model. It certainly does not emphasize it.

Reliability is a Systems Engineering concern, orthogonal to XP. You
have to do a separate analysis in that domain and feed the required
results in as Stories.

The customer could read all the briefs from the reliability and safety
dimensions and decide how important they are.

Most terrible failures of reliability have to do with things like
reusing old components in new contexts and having them blow up, or
failure to adequately model hardware interlocks in software.
Reliability and safety dimensions have their output which should be
funneled into an XP process. The software safety analyses are
helpful, but their recommendation amounts to plastering safety
warning stickers on your software which nobody will read.

[Gerold Keefer]

Code Like Hell wrote:
Reliability is a Systems Engineering concern, orthogonal to XP.

it certainly makes not too much sense to look at the reliabilty of
a system from a software side alone, though it is certainly possible
and has to be a part of an analysis.
however, reliability requirements need certainly to be reflected in
the development processes. in this context XP is hopelessly
inadequate. i explained the reason in a recent mail to this thread.

regards,

gerold

[David Lightstone]

"Code Like Hell" <dangerfinder@yahoo.com> wrote in message
news:212dcb5c.0306270817.374629c7@posting.google.com... tadamsmar@yahoo.com (Tom Adams) wrote in message
news:<ea44f5a1.0306040618.3332d4fb@posting.google.com>... I have noticed XP seems to be largely uninformed about software reliability engineering and I want change that. The advantage for XPers in understanding what I have to say is: More reliable systems with less work. "Reliability" has lots of meanings. To be very specific I am talking about Mean Time to Failure (MTTF) or Mean Time Between Failures. Take a moment to let this sink in... There is a type of testing that is used to achieve and confirm reliability. I see little evidence that XP includes this type of testing in its process model. It certainly does not emphasize it. Reliability is a Systems Engineering concern, orthogonal to XP. You have to do a separate analysis in that domain and feed the required results in as Stories.

Bull shit. You are erroneously assuming software is deterministic.

[John Roth]

"Tom Adams" <tadamsmar@yahoo.com> wrote in message
news:ea44f5a1.0306300549.7befa57b@posting.google.com... dangerfinder@yahoo.com (Code Like Hell) wrote in message
news:<212dcb5c.0306270817.374629c7@posting.google.com>... tadamsmar@yahoo.com (Tom Adams) wrote in message
news:<ea44f5a1.0306040618.3332d4fb@posting.google.com>... I have noticed XP seems to be largely uninformed about software reliability engineering and I want change that. The advantage for XPers in understanding what I have to say is:
More reliable systems with less work. "Reliability" has lots of meanings. To be very specific I am
talking about Mean Time to Failure (MTTF) or Mean Time Between Failures.
Take a moment to let this sink in... There is a type of testing that is used to achieve and confirm reliability. I see little evidence that XP includes this type of testing in its process model. It certainly does not emphasize it. Reliability is a Systems Engineering concern, orthogonal to XP. You have to do a separate analysis in that domain and feed the required results in as Stories. IMO, This means that XP is a partial methodology pretending to be a full methodology. This situation is all too common with software engineering methodologies. Note that you only have the XP and the customer. XP does not provide software reliability engineering expertise, and you really cannot rely on the customer to always provide it.

I'd object to one part of your characterization: XP has never
said, by pretense or otherwise, that it is a complete, cradle
to grave methodology. It's explicitly minimalist. Everyone
with any credibility has always said that it has to be
extended with other processes and practices when appropriate.
Also, the reliability engineering is all tied up with idea of small releases. Part of the power of small releases comes from confirming that the reliability of each release is high. So reliability engineering should be integrated into XP, unless you are willing to assume that all your clients always understand all this without any input from XP.

There are several questions to ask before going off the deep
end. One is whether the normally delivered reliability is
sufficient. When there are major projects that are reporting
field reported defect rates of one defect per month or lower,
(and there are) additional practices may not be appropriate.

The next question to ask is: how do you integrate the
various practices that compose reliability engineering
with XP without damaging either. That's not exactly
obvious, since XP does one thing that is exactly
opposite to standard software engineering practice:
what goes into the repository **is** production code,
without any further checks, testing or inspections.

Adding back end processes interfers with the rather
tight feedback loops involved. This isn't a problem
with the standard software engineering processes.

Since my rather limited understanding of reliability
engineering says that it's mostly a matter of determining
where to focus the greatest part of the testing effort,
it seems that it should be part of the acceptance test
process. In XP, automated acceptance tests should be
complete before programming starts. That's the ideal;
it's hardly universal practice, though.
I would grant that the customer may need to be involved in determining reliability requirements and needs to be involved in estimating the operational profile. And I would grant that you sometimes can and should rely on the customer for reliability engineering of the releases.

John Roth

[Michael Feathers]

"Tom Adams" <tadamsmar@yahoo.com> wrote: IMO, This means that XP is a partial methodology pretending to be a full methodology. This situation is all too common with software engineering methodologies. Note that you only have the XP and the customer. XP does not provide software reliability engineering expertise, and you really cannot rely on the customer to always provide it.

Everything is incomplete, thank goodness. Incompleteness prompts you to
exercise judgement.

Michael Feathers (completeness causes atrophy)
www.objectmentor.com

[Code Like Hell]

"David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message news:<KmALa.1816$L9.535@newssvr17.news.prodigy.com>... "Code Like Hell" <dangerfinder@yahoo.com> wrote in message news:212dcb5c.0306270817.374629c7@posting.google.com... tadamsmar@yahoo.com (Tom Adams) wrote in message news:<ea44f5a1.0306040618.3332d4fb@posting.google.com>... I have noticed XP seems to be largely uninformed about software reliability engineering and I want change that. The advantage for XPers in understanding what I have to say is: More reliable systems with less work. "Reliability" has lots of meanings. To be very specific I am talking about Mean Time to Failure (MTTF) or Mean Time Between Failures. Take a moment to let this sink in... There is a type of testing that is used to achieve and confirm reliability. I see little evidence that XP includes this type of testing in its process model. It certainly does not emphasize it. Reliability is a Systems Engineering concern, orthogonal to XP. You have to do a separate analysis in that domain and feed the required results in as Stories. Bull shit. You are erroneously assuming software is deterministic.

I'm sure the safety and reliablity experts have thought of that, and
they would have a response for you. I wouldn't be in this field if I
held the view that software was deterministic.

However, if it weren't deterministic to some degree, they still
wouldn't know the causes of failures that they have been able to
determine.

Reliability and safety analysis can examine the evidence of actual
behavior during tests, past operational failures and inspect as best
they can.

[Code Like Hell]

Gerold Keefer <gkeefer@avoca-vsm.com> wrote in message news:<3EFDB200.5F900C5D@avoca-vsm.com>... Code Like Hell wrote: Reliability is a Systems Engineering concern, orthogonal to XP. it certainly makes not too much sense to look at the reliabilty of a system from a software side alone, though it is certainly possible and has to be a part of an analysis. however, reliability requirements need certainly to be reflected in the development processes.

Why is that? Safeware practices say the same thing, they get into
defining development practices. I'm not so sure this is such a great
idea. The resulting document appears to me to be one huge
multidimensional safety warning sticker, the kind I'm not in the
habit of reading. So all they've done is create an even more
dangerous false sense of security. Put it on a sticker and forget it,
not my problem...
Regards,
CLH
in this context XP is hopelessly inadequate. i explained the reason in a recent mail to this thread. regards, gerold

[Code Like Hell]

"David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message news:<Vt%La.34$eO6.11@newssvr31.news.prodigy.com>... "Code Like Hell" <dangerfinder@yahoo.com> wrote in message news:212dcb5c.0306300951.c0f40ec@posting.google.com... "David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message news:<KmALa.1816$L9.535@newssvr17.news.prodigy.com>... "Code Like Hell" <dangerfinder@yahoo.com> wrote in message news:212dcb5c.0306270817.374629c7@posting.google.com... > tadamsmar@yahoo.com (Tom Adams) wrote in message news:<ea44f5a1.0306040618.3332d4fb@posting.google.com>... > > I have noticed XP seems to be largely uninformed about software > > reliability engineering and I want change that. > > > > The advantage for XPers in understanding what I have to say is: More > > reliable systems with less work. > > > > "Reliability" has lots of meanings. To be very specific I am talking > > about Mean Time to Failure (MTTF) or Mean Time Between Failures. Take > > a moment to let this sink in... > > > > There is a type of testing that is used to achieve and confirm > > reliability. I see little evidence that XP includes this type of > > testing in its process model. It certainly does not emphasize it. > > > > Reliability is a Systems Engineering concern, orthogonal to XP. You > have to do a separate analysis in that domain and feed the required > results in as Stories. Bull shit. You are erroneously assuming software is deterministic. I'm sure the safety and reliablity experts have thought of that, and they would have a response for you. I wouldn't be in this field if I held the view that software was deterministic. However, if it weren't deterministic to some degree, they still wouldn't know the causes of failures that they have been able to determine. Reliability and safety analysis can examine the evidence of actual behavior during tests, past operational failures and inspect as best they can. Is your response intended to shed light upon your contention, that XP is orthogonal to reliability? What about software design? Is that also orthogonal to reliability?

You're a smart guy, why don't you just honestly say what you think
about how much of reliability is outside the scope of software design.

I responded to your claim that I held the view that software was
deterministic. Say what your position is so we can start somewhere.

[David Lightstone]

"Code Like Hell" <dangerfinder@yahoo.com> wrote in message
news:212dcb5c.0307010600.7259942d@posting.google.com... "David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message
news:<Vt%La.34$eO6.11@newssvr31.news.prodigy.com>... "Code Like Hell" <dangerfinder@yahoo.com> wrote in message news:212dcb5c.0306300951.c0f40ec@posting.google.com... "David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in
message news:<KmALa.1816$L9.535@newssvr17.news.prodigy.com>... > "Code Like Hell" <dangerfinder@yahoo.com> wrote in message > news:212dcb5c.0306270817.374629c7@posting.google.com... > > tadamsmar@yahoo.com (Tom Adams) wrote in message news:<ea44f5a1.0306040618.3332d4fb@posting.google.com>... > > > I have noticed XP seems to be largely uninformed about software > > > reliability engineering and I want change that. > > > > > > The advantage for XPers in understanding what I have to say is: More > > > reliable systems with less work. > > > > > > "Reliability" has lots of meanings. To be very specific I am talking > > > about Mean Time to Failure (MTTF) or Mean Time Between Failures. Take > > > a moment to let this sink in... > > > > > > There is a type of testing that is used to achieve and confirm > > > reliability. I see little evidence that XP includes this type
of > > > testing in its process model. It certainly does not emphasize
it. > > > > > > > Reliability is a Systems Engineering concern, orthogonal to XP.
You > > have to do a separate analysis in that domain and feed the
required > > results in as Stories. > > Bull shit. You are erroneously assuming software is deterministic. I'm sure the safety and reliablity experts have thought of that, and they would have a response for you. I wouldn't be in this field if I held the view that software was deterministic. However, if it weren't deterministic to some degree, they still wouldn't know the causes of failures that they have been able to determine. Reliability and safety analysis can examine the evidence of actual behavior during tests, past operational failures and inspect as best they can. Is your response intended to shed light upon your contention, that XP is orthogonal to reliability? What about software design? Is that also orthogonal to reliability? You're a smart guy, why don't you just honestly say what you think about how much of reliability is outside the scope of software design. I responded to your claim that I held the view that software was deterministic. Say what your position is so we can start somewhere.

Start - we are finished

I think the questions pretty much state my opinion. You can not separate
issues of reliability from design when non-determinism is present. You have
contended that XP allows such a separation (because you claim XP is
orthogonal)

[David Lightstone]

"David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message
news:lTgMa.289$%A1.72@newssvr16.news.prodigy.com... "Code Like Hell" <dangerfinder@yahoo.com> wrote in message news:212dcb5c.0307010600.7259942d@posting.google.com... "David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in
message news:<Vt%La.34$eO6.11@newssvr31.news.prodigy.com>... "Code Like Hell" <dangerfinder@yahoo.com> wrote in message news:212dcb5c.0306300951.c0f40ec@posting.google.com... > "David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message news:<KmALa.1816$L9.535@newssvr17.news.prodigy.com>... > > "Code Like Hell" <dangerfinder@yahoo.com> wrote in message > > news:212dcb5c.0306270817.374629c7@posting.google.com... > > > tadamsmar@yahoo.com (Tom Adams) wrote in message news:<ea44f5a1.0306040618.3332d4fb@posting.google.com>... > > > > I have noticed XP seems to be largely uninformed about
software > > > > reliability engineering and I want change that. > > > > > > > > The advantage for XPers in understanding what I have to say
is: More > > > > reliable systems with less work. > > > > > > > > "Reliability" has lots of meanings. To be very specific I am talking > > > > about Mean Time to Failure (MTTF) or Mean Time Between
Failures. Take > > > > a moment to let this sink in... > > > > > > > > There is a type of testing that is used to achieve and confirm > > > > reliability. I see little evidence that XP includes this type of > > > > testing in its process model. It certainly does not emphasize it. > > > > > > > > > > Reliability is a Systems Engineering concern, orthogonal to XP. You > > > have to do a separate analysis in that domain and feed the required > > > results in as Stories. > > > > Bull shit. You are erroneously assuming software is deterministic. > > I'm sure the safety and reliablity experts have thought of that,
and > they would have a response for you. I wouldn't be in this field if
I > held the view that software was deterministic. > > However, if it weren't deterministic to some degree, they still > wouldn't know the causes of failures that they have been able to > determine. > > Reliability and safety analysis can examine the evidence of actual > behavior during tests, past operational failures and inspect as
best > they can. Is your response intended to shed light upon your contention, that XP
is orthogonal to reliability? What about software design? Is that also orthogonal to reliability? You're a smart guy, why don't you just honestly say what you think about how much of reliability is outside the scope of software design. I responded to your claim that I held the view that software was deterministic. Say what your position is so we can start somewhere. Start - we are finished I think the questions pretty much state my opinion. You can not separate issues of reliability from design when non-determinism is present. You
have contended that XP allows such a separation (because you claim XP is orthogonal)

There is another interpretation of your initial comment (I excluded it from
consideration based upon the thread subject). That being - XP is inadequate
with respect to issues of reliability. It assumes that such issues are
orthogonal to the tasks that is performs in the development effort.

If this is a more correct interpretation, then I see the error in my ways

[Tom Adams]

"John Roth" <johnroth@ameritech.net> wrote in message news:<vg0jmt3l94aa5@news.supernews.com>... Also, the reliability engineering is all tied up with idea of small releases. Part of the power of small releases comes from confirming that the reliability of each release is high. So reliability engineering should be integrated into XP, unless you are willing to assume that all your clients always understand all this without any input from XP. There are several questions to ask before going off the deep end. One is whether the normally delivered reliability is sufficient. When there are major projects that are reporting field reported defect rates of one defect per month or lower, (and there are) additional practices may not be appropriate. The next question to ask is: how do you integrate the various practices that compose reliability engineering with XP without damaging either. That's not exactly obvious, since XP does one thing that is exactly opposite to standard software engineering practice: what goes into the repository **is** production code, without any further checks, testing or inspections.

Do XPer's actually promote this bizarre, doctrinaire,
programmer's-eye-view of the process? I doubt that most clients
actually treat the small releases as ready for production.
Adding back end processes interfers with the rather tight feedback loops involved. This isn't a problem with the standard software engineering processes.

According to you, the repository is production code with no feedback
from anything like production usage.
Since my rather limited understanding of reliability engineering says that it's mostly a matter of determining where to focus the greatest part of the testing effort, it seems that it should be part of the acceptance test process. In XP, automated acceptance tests should be complete before programming starts. That's the ideal; it's hardly universal practice, though.

Correct, part of the acceptance testing, and/or part of the transition
process of a small release into production.

[John Roth]

"Tom Adams" <tadamsmar@yahoo.com> wrote in message
news:ea44f5a1.0307011019.51207776@posting.google.com... "John Roth" <johnroth@ameritech.net> wrote in message
news:<vg0jmt3l94aa5@news.supernews.com>... Also, the reliability engineering is all tied up with idea of
small releases. Part of the power of small releases comes from
confirming that the reliability of each release is high. So reliability engineering should be integrated into XP, unless you are willing
to assume that all your clients always understand all this without
any input from XP. There are several questions to ask before going off the deep end. One is whether the normally delivered reliability is sufficient. When there are major projects that are reporting field reported defect rates of one defect per month or lower, (and there are) additional practices may not be appropriate. The next question to ask is: how do you integrate the various practices that compose reliability engineering with XP without damaging either. That's not exactly obvious, since XP does one thing that is exactly opposite to standard software engineering practice: what goes into the repository **is** production code, without any further checks, testing or inspections. Do XPer's actually promote this bizarre, doctrinaire, programmer's-eye-view of the process? I doubt that most clients actually treat the small releases as ready for production.

You're misinterpreting what I said, and I'd hate to think
deliberately. What most processes check in is not production
*quality* code. It needs additional work (testing, debugging,
inspections, etc.) to make it so. The XP process strives
to make what you check in production quality, without
needing additional steps to find and fix errors. Adding back end processes interfers with the rather tight feedback loops involved. This isn't a problem with the standard software engineering processes. According to you, the repository is production code with no feedback from anything like production usage.

This is a bizzare statement. If I took it literally, the
only way to get production code is to ship it to the
customer and find out if it croaks. Since XP is one
of the "agile" methodologies, and since they are **all**
well known for advocating frequent releases to get
real world feedback, I fail to see your point.

Since my rather limited understanding of reliability engineering says that it's mostly a matter of determining where to focus the greatest part of the testing effort, it seems that it should be part of the acceptance test process. In XP, automated acceptance tests should be complete before programming starts. That's the ideal; it's hardly universal practice, though. Correct, part of the acceptance testing, and/or part of the transition process of a small release into production.

[Code Like Hell]

"David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message news:<6hhMa.90$r24.6516992@newssvr15.news.prodigy.com>... "David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message news:lTgMa.289$%A1.72@newssvr16.news.prodigy.com... "Code Like Hell" <dangerfinder@yahoo.com> wrote in message news:212dcb5c.0307010600.7259942d@posting.google.com... "David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message news:<Vt%La.34$eO6.11@newssvr31.news.prodigy.com>... > "Code Like Hell" <dangerfinder@yahoo.com> wrote in message > news:212dcb5c.0306300951.c0f40ec@posting.google.com... > > "David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message news:<KmALa.1816$L9.535@newssvr17.news.prodigy.com>... > > > "Code Like Hell" <dangerfinder@yahoo.com> wrote in message > > > news:212dcb5c.0306270817.374629c7@posting.google.com... > > > > tadamsmar@yahoo.com (Tom Adams) wrote in message news:<ea44f5a1.0306040618.3332d4fb@posting.google.com>... > > > > > I have noticed XP seems to be largely uninformed about software > > > > > reliability engineering and I want change that. > > > > > > > > > > The advantage for XPers in understanding what I have to say is: More > > > > > reliable systems with less work. > > > > > > > > > > "Reliability" has lots of meanings. To be very specific I am talking > > > > > about Mean Time to Failure (MTTF) or Mean Time Between Failures. Take > > > > > a moment to let this sink in... > > > > > > > > > > There is a type of testing that is used to achieve and confirm > > > > > reliability. I see little evidence that XP includes this type of > > > > > testing in its process model. It certainly does not emphasize it. > > > > > > > > > > > > > Reliability is a Systems Engineering concern, orthogonal to XP. You > > > > have to do a separate analysis in that domain and feed the required > > > > results in as Stories. > > > > > > Bull shit. You are erroneously assuming software is deterministic. > > > > I'm sure the safety and reliablity experts have thought of that, and > > they would have a response for you. I wouldn't be in this field if I > > held the view that software was deterministic. > > > > However, if it weren't deterministic to some degree, they still > > wouldn't know the causes of failures that they have been able to > > determine. > > > > Reliability and safety analysis can examine the evidence of actual > > behavior during tests, past operational failures and inspect as best > > they can. > > Is your response intended to shed light upon your contention, that XP is > orthogonal to reliability? > What about software design? Is that also orthogonal to reliability? You're a smart guy, why don't you just honestly say what you think about how much of reliability is outside the scope of software design. I responded to your claim that I held the view that software was deterministic. Say what your position is so we can start somewhere. Start - we are finished

Nah, I don't think so. I just found some danger and you want me to
quit?
I think the questions pretty much state my opinion. You can not separate issues of reliability from design when non-determinism is present. You have contended that XP allows such a separation (because you claim XP is orthogonal) There is another interpretation of your initial comment (I excluded it from consideration based upon the thread subject). That being - XP is inadequate with respect to issues of reliability. It assumes that such issues are orthogonal to the tasks that is performs in the development effort. If this is a more correct interpretation, then I see the error in my ways

Yeah. Let's start here :-)

I'm happy to say XP is inadequate with respect to issues of
reliability and safety. And I'm still an XP fan.

You're other reply was good though in that you clearly stated your
message:
You can not separate issues of reliability from design when non-determinism is present

That is interesting. Please elaborate. How do you separate
reliability from design with non-determinism is *not* present?

In practice, XP is also joyfully inadequate ("orthogonal" in other
words) with respect to Design Issues. So really we're not talking
about XP at all. We're talking about Reliablity with Respect to
Design.

[Code Like Hell]

"John Roth" <johnroth@ameritech.net> wrote in message news:<vg57kig2qdiv99@news.supernews.com>... "Tom Adams" <tadamsmar@yahoo.com> wrote in message news:ea44f5a1.0307011019.51207776@posting.google.com... "John Roth" <johnroth@ameritech.net> wrote in message news:<vg0jmt3l94aa5@news.supernews.com>... > Also, the reliability engineering is all tied up with idea of small > releases. Part of the power of small releases comes from confirming > that the reliability of each release is high. So reliability > engineering should be integrated into XP, unless you are willing to > assume that all your clients always understand all this without any > input from XP. There are several questions to ask before going off the deep end. One is whether the normally delivered reliability is sufficient. When there are major projects that are reporting field reported defect rates of one defect per month or lower, (and there are) additional practices may not be appropriate. The next question to ask is: how do you integrate the various practices that compose reliability engineering with XP without damaging either. That's not exactly obvious, since XP does one thing that is exactly opposite to standard software engineering practice: what goes into the repository **is** production code, without any further checks, testing or inspections. Do XPer's actually promote this bizarre, doctrinaire, programmer's-eye-view of the process? I doubt that most clients actually treat the small releases as ready for production. You're misinterpreting what I said, and I'd hate to think deliberately. What most processes check in is not production *quality* code. It needs additional work (testing, debugging, inspections, etc.) to make it so. The XP process strives to make what you check in production quality, without needing additional steps to find and fix errors. Adding back end processes interfers with the rather tight feedback loops involved. This isn't a problem with the standard software engineering processes. According to you, the repository is production code with no feedback from anything like production usage. This is a bizzare statement. If I took it literally, the only way to get production code is to ship it to the customer and find out if it croaks. Since XP is one of the "agile" methodologies, and since they are **all** well known for advocating frequent releases to get real world feedback, I fail to see your point.

I believe he is saying that XP will do no further deliberation about
reliability on the back end, the repository, which is what you said.

You're both saying the same thing, only with different emphasis. If
code goes to check-in as production, then there is no further
opportunity for feedback until what shows up on the front-end,
identified through parallel and decoupled analyses, as a defect.

Reliability analysis is like speech functions of the brain, slower
and decoupled, haven't you ever experienced saying something in the
moment that some other faster part of your brain recognizes that
things had changed rendering what you say a contradiction or totally
irrelevant, nevertheless it's not possible to pull out of not saying
it, so it's said and disregarded, while you've already redone your
intentions and have started taking action to do the more relevant
thing?

The code that gets checked in is production quality and is 100%
aligned with what has been defined as quality on the *front* end
through other assessments not coupled with XP.
Since my rather limited understanding of reliability engineering says that it's mostly a matter of determining where to focus the greatest part of the testing effort, it seems that it should be part of the acceptance test process. In XP, automated acceptance tests should be complete before programming starts. That's the ideal; it's hardly universal practice, though. Correct, part of the acceptance testing, and/or part of the transition process of a small release into production.

[Code Like Hell]

I'm waiting Mr Keefer...why do reliablity requirements need to be
coupled to the development processes? Do we know enough about
reliability requirements to design efficient development processes
with them hard-coded in there? Does every project need the same
reliability requirements, so you need a difference version of the
development process for every conceivable situation? Isn't this why
separation of concerns and modularity is considered efficient, and
the coupling considered monolithic?

Wait. Please don't answer any of those.

I much more interested in your questions. What are your questions
with respect to these matters?


dangerfinder@yahoo.com (Code Like Hell) wrote in message news:<212dcb5c.0306301050.584030ee@posting.google.com>... Gerold Keefer <gkeefer@avoca-vsm.com> wrote in message news:<3EFDB200.5F900C5D@avoca-vsm.com>... Code Like Hell wrote: Reliability is a Systems Engineering concern, orthogonal to XP. it certainly makes not too much sense to look at the reliabilty of a system from a software side alone, though it is certainly possible and has to be a part of an analysis. however, reliability requirements need certainly to be reflected in the development processes. Why is that? Safeware practices say the same thing, they get into defining development practices. I'm not so sure this is such a great idea. The resulting document appears to me to be one huge multidimensional safety warning sticker, the kind I'm not in the habit of reading. So all they've done is create an even more dangerous false sense of security. Put it on a sticker and forget it, not my problem... Regards, CLH in this context XP is hopelessly inadequate. i explained the reason in a recent mail to this thread. regards, gerold

[David Lightstone]

"Code Like Hell" <dangerfinder@yahoo.com> wrote in message
news:212dcb5c.0307020652.76313011@posting.google.com... "David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message
news:<6hhMa.90$r24.6516992@newssvr15.news.prodigy.com>...
There is another interpretation of your initial comment (I excluded it
from consideration based upon the thread subject). That being - XP is
inadequate with respect to issues of reliability. It assumes that such issues are orthogonal to the tasks that is performs in the development effort. If this is a more correct interpretation, then I see the error in my
ways Yeah. Let's start here :-) I'm happy to say XP is inadequate with respect to issues of reliability and safety. And I'm still an XP fan. You're other reply was good though in that you clearly stated your message: You can not separate issues of reliability from design when non-determinism is present That is interesting. Please elaborate. How do you separate reliability from design with non-determinism is *not* present?

For the present I will pass. My concern is in preventing an erroneous belief
from being accepted as correct. In this situation the belief relates to XP
creating reliable programs (without deterministic behavior being properly
established as present)

In practice, XP is also joyfully inadequate ("orthogonal" in other words) with respect to Design Issues. So really we're not talking about XP at all. We're talking about Reliablity with Respect to Design.

There are those you claim (certainly not me) that XP (and its refactoring)
allows designs to successfully evolve. Ergo, it is claimed to accomplished
design without the effort

[Uncle Bob (Robert C. Martin)]

tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this
on (or about) 2 Jul 2003 04:52:33 -0700, :
"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<pql4gvkl499ptd8deqldtj660ev6u70vc9@4ax.com>... tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this on (or about) 1 Jul 2003 11:19:21 -0700, :Do XPer's actually promote this bizarre, doctrinaire,programmer's-eye-view of the process? I doubt that most clientsactually treat the small releases as ready for production. Yes and no. Yes, in that you must be willing to ship anything you check in. No, in that we all know that the customer has his own time frame for putting things into production.This is not about customer time frames. This is about what it reallytakes to prepare code for production.

And what is that? What does it really take to prepare code for
production?
I don't see the point inpromoting a "methodology" that cannot create production code.

Neither do I.
The point is simply this. We don't check in crap. We check in code many times per day. Any code we check in passes all the tests we can think of,All tests we can think of? Please make the effort required toconstruct statements that are not obviously false after a millisecondof examination.

Don't be obtuse. I am not a fool. Make a real point.Anyway, if you believe this, then you cannot understand what I amtrying to do here.

Yes, it must be very far beyond my meager comprehension.


Robert C. Martin | "Uncle Bob"
Object Mentor Inc.| unclebob @ objectmentor . com
PO Box 5757 | Tel: (800) 338-6716
565 Lakeview Pkwy | Fax: (847) 573-1658 | www.objectmentor.com
Suite 135 | | www.XProgramming.com
Vernon Hills, IL, | Training and Mentoring | www.junit.org
60061 | OO, XP, Java, C++, Python |

[Tom Adams]

"John Roth" <johnroth@ameritech.net> wrote in message news:<vg0jmt3l94aa5@news.supernews.com>... "Tom Adams" <tadamsmar@yahoo.com> wrote in message news:ea44f5a1.0306300549.7befa57b@posting.google.com... dangerfinder@yahoo.com (Code Like Hell) wrote in message news:<212dcb5c.0306270817.374629c7@posting.google.com>... tadamsmar@yahoo.com (Tom Adams) wrote in message news:<ea44f5a1.0306040618.3332d4fb@posting.google.com>... > I have noticed XP seems to be largely uninformed about software > reliability engineering and I want change that. > > The advantage for XPers in understanding what I have to say is: More > reliable systems with less work. > > "Reliability" has lots of meanings. To be very specific I am talking > about Mean Time to Failure (MTTF) or Mean Time Between Failures. Take > a moment to let this sink in... > > There is a type of testing that is used to achieve and confirm > reliability. I see little evidence that XP includes this type of > testing in its process model. It certainly does not emphasize it. > Reliability is a Systems Engineering concern, orthogonal to XP. You have to do a separate analysis in that domain and feed the required results in as Stories. IMO, This means that XP is a partial methodology pretending to be a full methodology. This situation is all too common with software engineering methodologies. Note that you only have the XP and the customer. XP does not provide software reliability engineering expertise, and you really cannot rely on the customer to always provide it. I'd object to one part of your characterization: XP has never said, by pretense or otherwise, that it is a complete, cradle to grave methodology. It's explicitly minimalist.

It's not minimalist. There is no such thing as a minimalist software
methodology. Some methodologies use inspection and use no unit
tesing. XP does test-first unit testing and no code inspection. If
you start will all the posssible ingredients of a software methodology
and you winnow down to a minimal set, you don't get one single minimal
set. The phrase "minimalist software methodology" is gibberish.

[John Roth]

"Tom Adams" <tadamsmar@yahoo.com> wrote in message
news:ea44f5a1.0307020913.4398eb74@posting.google.com... "John Roth" <johnroth@ameritech.net> wrote in message
news:<vg0jmt3l94aa5@news.supernews.com>... "Tom Adams" <tadamsmar@yahoo.com> wrote in message news:ea44f5a1.0306300549.7befa57b@posting.google.com... dangerfinder@yahoo.com (Code Like Hell) wrote in message news:<212dcb5c.0306270817.374629c7@posting.google.com>... > tadamsmar@yahoo.com (Tom Adams) wrote in message news:<ea44f5a1.0306040618.3332d4fb@posting.google.com>... > > I have noticed XP seems to be largely uninformed about
software > > reliability engineering and I want change that. > > > > The advantage for XPers in understanding what I have to say
is: More > > reliable systems with less work. > > > > "Reliability" has lots of meanings. To be very specific I am talking > > about Mean Time to Failure (MTTF) or Mean Time Between
Failures. Take > > a moment to let this sink in... > > > > There is a type of testing that is used to achieve and confirm > > reliability. I see little evidence that XP includes this type
of > > testing in its process model. It certainly does not emphasize
it. > > > > Reliability is a Systems Engineering concern, orthogonal to XP.
You > have to do a separate analysis in that domain and feed the
required > results in as Stories. IMO, This means that XP is a partial methodology pretending to be
a full methodology. This situation is all too common with software engineering methodologies. Note that you only have the XP and the customer. XP does not provide software reliability engineering expertise, and you really cannot rely on the customer to always provide it. I'd object to one part of your characterization: XP has never said, by pretense or otherwise, that it is a complete, cradle to grave methodology. It's explicitly minimalist. It's not minimalist. There is no such thing as a minimalist software methodology. Some methodologies use inspection and use no unit tesing. XP does test-first unit testing and no code inspection. If you start will all the posssible ingredients of a software methodology and you winnow down to a minimal set, you don't get one single minimal set. The phrase "minimalist software methodology" is gibberish.

Minimalist does not, to me at least, imply that there is only one
minimum. What it implies is that it can't be further reduced
without sacrificing a large amount of effectiveness.

It also implies that it can, and in may environments, must,
be extended with additional practices.

John Roth

[Richard MacDonald]

"David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in
news:GlDMa.18$6f1.11@newssvr33.news.prodigy.com:
There are those you claim (certainly not me) that XP (and its refactoring) allows designs to successfully evolve. Ergo, it is claimed to accomplished design without the effort

Yeah right. No effort at all. You think XP is easy? Ergo, you're a putz.

[Paul Sinnett]

Tom Adams wrote: My point is that you are not doing all the test you can think of. It is not practical to do all the test you can think of. If you current with the softare testing literature circa 1980 (that is, only 23 years behind) you would never say something like "we do all the test we can think of". And, understanding the true limitations of testing, rather than spouting these XP brainwashing mantras, is the starting point for understanding software testing.

True. However, writing tests and working with tests is a good way to learn
the limitations of testing. The XP rules for unit testing limit us to
writing tests that run quickly. Therefore we only really test a small
fraction of the possible cases. Is that a problem?

[Uncle Bob (Robert C. Martin)]

"David Lightstone" <david._NoSpamlightstone@prodigy.net> might (or
might not) have written this on (or about) Wed, 02 Jul 2003 16:01:42
GMT, :
There are those you claim (certainly not me) that XP (and its refactoring)allows designs to successfully evolve.

This much is certainly true. I've done quite a bit of it.
Ergo, it is claimed to accomplished design without the effort

There is no design without effort. Anyone who claims that a process
or method provides design without effort is a friggin' idiot. (and is
probably selling something.) Design in XP takes a lot of effort,
spread all through the development lifecycle.



Robert C. Martin | "Uncle Bob"
Object Mentor Inc.| unclebob @ objectmentor . com
PO Box 5757 | Tel: (800) 338-6716
565 Lakeview Pkwy | Fax: (847) 573-1658 | www.objectmentor.com
Suite 135 | | www.XProgramming.com
Vernon Hills, IL, | Training and Mentoring | www.junit.org
60061 | OO, XP, Java, C++, Python |

[Uncle Bob (Robert C. Martin)]

tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this
on (or about) 3 Jul 2003 04:41:42 -0700, :
"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<cf26gv8huthen9okatre9ocifbfjvoigvb@4ax.com>... tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this on (or about) 2 Jul 2003 04:52:33 -0700, :"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<pql4gvkl499ptd8deqldtj660ev6u70vc9@4ax.com>...
> The point is simply this. We don't check in crap. We check in code> many times per day. Any code we check in passes all the tests we can> think of,All tests we can think of? Please make the effort required toconstruct statements that are not obviously false after a millisecondof examination. Don't be obtuse. I am not a fool. Make a real point.My point is that you are not doing all the test you can think of.


Then let me clarify. "Any code we check in passes all the tests we
can think of ***that are worth running***."
Itis not practical to do all the test you can think of. If you currentwith the softare testing literature circa 1980 (that is, only 23 yearsbehind) you would never say something like "we do all the test we canthink of".

Unless, of course, you *were* current with the testing literature and
simply expected people to understand the idiom. You know, like when
people say "I've looked everywhere and I just can't find it." Clearly
they don't mean that they have looked *everywhere*.
And, understanding the true limitations of testing, ratherthan spouting these XP brainwashing mantras, is the starting point forunderstanding software testing.

Believe me, anyone who writes as many tests and XP requires of you has
a *very* good understanding of the limitations of testing. We
experience those limitations daily.

As for the term "brainwashing mantras" I think it sounds a lot like a
brainwashing mantra.







Robert C. Martin | "Uncle Bob"
Object Mentor Inc.| unclebob @ objectmentor . com
PO Box 5757 | Tel: (800) 338-6716
565 Lakeview Pkwy | Fax: (847) 573-1658 | www.objectmentor.com
Suite 135 | | www.XProgramming.com
Vernon Hills, IL, | Training and Mentoring | www.junit.org
60061 | OO, XP, Java, C++, Python |

[David Lightstone]

"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in
message news:jdahgvs7ip12ma3c9q3tjdoegsnv13uj7o@4ax.com... "David Lightstone" <david._NoSpamlightstone@prodigy.net> might (or might not) have written this on (or about) Wed, 02 Jul 2003 16:01:42 GMT, :There are those you claim (certainly not me) that XP (and its
refactoring)allows designs to successfully evolve. This much is certainly true. I've done quite a bit of it.Ergo, it is claimed to accomplished design without the effort There is no design without effort. Anyone who claims that a process or method provides design without effort is a friggin' idiot. (and is probably selling something.) Design in XP takes a lot of effort, spread all through the development lifecycle.

More or less effort that the alternatives? Pick you strawman
Robert C. Martin | "Uncle Bob" Object Mentor Inc.| unclebob @ objectmentor . com PO Box 5757 | Tel: (800) 338-6716 565 Lakeview Pkwy | Fax: (847) 573-1658 | www.objectmentor.com Suite 135 | | www.XProgramming.com Vernon Hills, IL, | Training and Mentoring | www.junit.org 60061 | OO, XP, Java, C++, Python |

[David Lightstone]

"David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message
news:JR2Oa.2510$gH7.1512@newssvr17.news.prodigy.com... "Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:jdahgvs7ip12ma3c9q3tjdoegsnv13uj7o@4ax.com... "David Lightstone" <david._NoSpamlightstone@prodigy.net> might (or might not) have written this on (or about) Wed, 02 Jul 2003 16:01:42 GMT, :There are those you claim (certainly not me) that XP (and its refactoring)allows designs to successfully evolve. This much is certainly true. I've done quite a bit of it.Ergo, it is claimed to accomplished design without the effort There is no design without effort. Anyone who claims that a process or method provides design without effort is a friggin' idiot. (and is probably selling something.) Design in XP takes a lot of effort, spread all through the development lifecycle. More or less effort that the alternatives? Pick you strawman
Pick your strawman Robert C. Martin | "Uncle Bob" Object Mentor Inc.| unclebob @ objectmentor . com PO Box 5757 | Tel: (800) 338-6716 565 Lakeview Pkwy | Fax: (847) 573-1658 | www.objectmentor.com Suite 135 | | www.XProgramming.com Vernon Hills, IL, | Training and Mentoring | www.junit.org 60061 | OO, XP, Java, C++, Python |

[Uncle Bob (Robert C. Martin)]

"David Lightstone" <david._NoSpamlightstone@prodigy.net> might (or
might not) have written this on (or about) Mon, 07 Jul 2003 00:08:09
GMT, :
"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote inmessage news:jdahgvs7ip12ma3c9q3tjdoegsnv13uj7o@4ax.com... "David Lightstone" <david._NoSpamlightstone@prodigy.net> might (or might not) have written this on (or about) Wed, 02 Jul 2003 16:01:42 GMT, :There are those you claim (certainly not me) that XP (and itsrefactoring)allows designs to successfully evolve. This much is certainly true. I've done quite a bit of it.Ergo, it is claimed to accomplished design without the effort There is no design without effort. Anyone who claims that a process or method provides design without effort is a friggin' idiot. (and is probably selling something.) Design in XP takes a lot of effort, spread all through the development lifecycle.More or less effort that the alternatives?

Don't know. Don't care. I personally get a *much* better result with
XP.


Robert C. Martin | "Uncle Bob"
Object Mentor Inc.| unclebob @ objectmentor . com
PO Box 5757 | Tel: (800) 338-6716
565 Lakeview Pkwy | Fax: (847) 573-1658 | www.objectmentor.com
Suite 135 | | www.XProgramming.com
Vernon Hills, IL, | Training and Mentoring | www.junit.org
60061 | OO, XP, Java, C++, Python |

[David Lightstone]

"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in
message news:48uhgvoo9uqirb62ip6ru6vrfvu15isnhv@4ax.com... "David Lightstone" <david._NoSpamlightstone@prodigy.net> might (or might not) have written this on (or about) Mon, 07 Jul 2003 00:08:09 GMT, :"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote
inmessage news:jdahgvs7ip12ma3c9q3tjdoegsnv13uj7o@4ax.com... "David Lightstone" <david._NoSpamlightstone@prodigy.net> might (or might not) have written this on (or about) Wed, 02 Jul 2003 16:01:42 GMT, : >There are those you claim (certainly not me) that XP (and itsrefactoring) >allows designs to successfully evolve. This much is certainly true. I've done quite a bit of it. >Ergo, it is claimed to accomplished design without the effort There is no design without effort. Anyone who claims that a process or method provides design without effort is a friggin' idiot. (and is probably selling something.) Design in XP takes a lot of effort, spread all through the development lifecycle.More or less effort that the alternatives? Don't know. Don't care. I personally get a *much* better result with XP.

Why am I not surprised.
Robert C. Martin | "Uncle Bob" Object Mentor Inc.| unclebob @ objectmentor . com PO Box 5757 | Tel: (800) 338-6716 565 Lakeview Pkwy | Fax: (847) 573-1658 | www.objectmentor.com Suite 135 | | www.XProgramming.com Vernon Hills, IL, | Training and Mentoring | www.junit.org 60061 | OO, XP, Java, C++, Python |

[Tom Adams]

"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<5jahgvkjqqqi7akpt9t4dd17tptif4dr66@4ax.com>... tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this on (or about) 3 Jul 2003 04:41:42 -0700, :"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<cf26gv8huthen9okatre9ocifbfjvoigvb@4ax.com>... tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this on (or about) 2 Jul 2003 04:52:33 -0700, : >"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<pql4gvkl499ptd8deqldtj660ev6u70vc9@4ax.com>... >> The point is simply this. We don't check in crap. We check in code >> many times per day. Any code we check in passes all the tests we can >> think of, > >All tests we can think of? Please make the effort required to >construct statements that are not obviously false after a millisecond >of examination. Don't be obtuse. I am not a fool. Make a real point.My point is that you are not doing all the test you can think of. Then let me clarify. "Any code we check in passes all the tests we can think of ***that are worth running***."Itis not practical to do all the test you can think of. If you currentwith the softare testing literature circa 1980 (that is, only 23 yearsbehind) you would never say something like "we do all the test we canthink of". Unless, of course, you *were* current with the testing literature and simply expected people to understand the idiom. You know, like when people say "I've looked everywhere and I just can't find it." Clearly they don't mean that they have looked *everywhere*.

This idiom, that implies the perfection of XP unit testing, prevades
all the XP literature that I have read. And, you never see this idiom
in any book or discussion on testing outside of XP world. People who
are prepared for a serious discussion of software testing don't use
this idiom.

[Uncle Bob (Robert C. Martin)]

"David Lightstone" <david._NoSpamlightstone@prodigy.net> might (or
might not) have written this on (or about) Mon, 07 Jul 2003 10:26:29
GMT, :
Don't know. Don't care. I personally get a *much* better result with XP.Why am I not surprised.

Why should you be?


Robert C. Martin | "Uncle Bob"
Object Mentor Inc.| unclebob @ objectmentor . com
PO Box 5757 | Tel: (800) 338-6716
565 Lakeview Pkwy | Fax: (847) 573-1658 | www.objectmentor.com
Suite 135 | | www.XProgramming.com
Vernon Hills, IL, | Training and Mentoring | www.junit.org
60061 | OO, XP, Java, C++, Python |

[Uncle Bob (Robert C. Martin)]

tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this
on (or about) 7 Jul 2003 06:44:47 -0700, :
"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<5jahgvkjqqqi7akpt9t4dd17tptif4dr66@4ax.com>...
Unless, of course, you *were* current with the testing literature and simply expected people to understand the idiom. You know, like when people say "I've looked everywhere and I just can't find it." Clearly they don't mean that they have looked *everywhere*.This idiom, that implies the perfection of XP unit testing, prevadesall the XP literature that I have read.

There has never been, nor will there ever be, an implication of
perfection. The implication is that it's a good idea to test
everything that is worth testing. (We idiomatically say "test
anything that could possibly break.")
And, you never see this idiomin any book or discussion on testing outside of XP world.

You don't see the notion that you should test everything you can think
of that's worth testing? I think you do.
People whoare prepared for a serious discussion of software testing don't usethis idiom.

Serious discussions don't get caught up in semantics like this.
People who want to discuss something seriously get past the silly
semantics and move on to make real points.

Robert C. Martin | "Uncle Bob"
Object Mentor Inc.| unclebob @ objectmentor . com
PO Box 5757 | Tel: (800) 338-6716
565 Lakeview Pkwy | Fax: (847) 573-1658 | www.objectmentor.com
Suite 135 | | www.XProgramming.com
Vernon Hills, IL, | Training and Mentoring | www.junit.org
60061 | OO, XP, Java, C++, Python |

[Tom Adams]

"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<5jahgvkjqqqi7akpt9t4dd17tptif4dr66@4ax.com>... tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this on (or about) 3 Jul 2003 04:41:42 -0700, :"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<cf26gv8huthen9okatre9ocifbfjvoigvb@4ax.com>... tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this on (or about) 2 Jul 2003 04:52:33 -0700, : >"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<pql4gvkl499ptd8deqldtj660ev6u70vc9@4ax.com>... >> The point is simply this. We don't check in crap. We check in code >> many times per day. Any code we check in passes all the tests we can >> think of, > >All tests we can think of? Please make the effort required to >construct statements that are not obviously false after a millisecond >of examination. Don't be obtuse. I am not a fool. Make a real point.My point is that you are not doing all the test you can think of. Then let me clarify. "Any code we check in passes all the tests we can think of ***that are worth running***."

Well, I would normally be inclined to show you that that is not true
either, but you would probably accuse me of densely taking you
literally.

So far, you have recanted every statement that you have tried to make
about your software testing.

Do you want to save us some time and just recant this one?

[Kevin Cline]

tadamsmar@yahoo.com (Tom Adams) wrote in message news:<ea44f5a1.0306300549.7befa57b@posting.google.com>... Also, the reliability engineering is all tied up with idea of small releases. Part of the power of small releases comes from confirming that the reliability of each release is high. So reliability engineering should be integrated into XP, unless you are willing to assume that all your clients always understand all this without any input from XP.

Coming late to this thread, I'm unsure how reliability testing makes
sense for software. It's reasonable for a manufactured product, where
you may find that components of type X have a mean lifetime of T, so
if you need ten of them all working, the mean time to failure is
f(T,10). But when software fails, there is a bug that can be fixed.
If you run reliability testing on software and find failures, what do
you do? Write down the result, or fix the bugs and retest? Why would
reliability testing be preferable to full-coverage testing?

[Tom Adams]

kcline17@hotmail.com (Kevin Cline) wrote in message news:<ba162549.0307081037.3c8ce61a@posting.google.com>... tadamsmar@yahoo.com (Tom Adams) wrote in message news:<ea44f5a1.0306300549.7befa57b@posting.google.com>... Also, the reliability engineering is all tied up with idea of small releases. Part of the power of small releases comes from confirming that the reliability of each release is high. So reliability engineering should be integrated into XP, unless you are willing to assume that all your clients always understand all this without any input from XP. Coming late to this thread, I'm unsure how reliability testing makes sense for software. It's reasonable for a manufactured product, where you may find that components of type X have a mean lifetime of T, so if you need ten of them all working, the mean time to failure is f(T,10). But when software fails, there is a bug that can be fixed. If you run reliability testing on software and find failures, what do you do? Write down the result, or fix the bugs and retest? Why would reliability testing be preferable to full-coverage testing?

What is full-coverage testing? I have heard of line-coverage,
branch-coverage. But what is full-coverage?

Full coverage of all the places that the bugs can lurk is impractical
for most systems. This level of coverage is the equivalent of a
program correctness proof. (Reference: see the paper "Partition
testing does not inspire confidence" by Dick Hamlet)

Partial coverage (line-coverage, etc.) of all the places that bugs can
lurk leaves holes. And, this type of partial coverage does not
measure the size of the holes. That is, it does not measure the
probability that a user of the system will encounter a bug in these
holes.

Software reliability engineering involves selecting test cases
according to the usage profile of the system. This means that the
likelihood that the user will encounter a bug can be estimated during
the testing process. It is this characteristic that means that
usage-oriented testing accomplishes something important that
line-coverage, branch-coverage, etc. cannot do.

Of course, usage-oriented testing also provides only partial coverage.
But, the level of coverage can be estimated in terms of the
probability that a user will encounter a bug in a uncovered region.

[Tom Adams]

"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<ks3pgv0kvgk4l9ed51vf7e5imcmphpl2rc@4ax.com>... tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this on (or about) 8 Jul 2003 05:39:35 -0700, : Then let me clarify. "Any code we check in passes all the tests we can think of ***that are worth running***."Well, I would normally be inclined to show you that that is not trueeither, but you would probably accuse me of densely taking youliterally. Please follow your inclination and show me that any code we check in should not pass all tests we can think of that are worth running.

Let's review.

First you claim that you run all the tests you can think of.

Then, no, you claim that you run all the tests you can think of that
are worth running.

Now the question is why the code should not pass all the tests we can
think of that are worth running.

I would assume that any test that might find a bug is worth running.
Correct?

So, all the tests that are worth running would consists of a set of
tests that completely tested a system - if the system passed this set
of test then there would be no chance that it had any bugs.

Now, it is possible to design a set of tests with this characteristic.
However, it is not practical, it takes too long to design the set of
all tests that will find all possible bugs for most systems.

All this is too obvious for you, no doubt.
So far, you have recanted every statement that you have tried to makeabout your software testing. Whoo-ahh!Do you want to save us some time and just recant this one? Heck no, this is too much fun. Robert C. Martin | "Uncle Bob" Object Mentor Inc.| unclebob @ objectmentor . com PO Box 5757 | Tel: (800) 338-6716 565 Lakeview Pkwy | Fax: (847) 573-1658 | www.objectmentor.com Suite 135 | | www.XProgramming.com Vernon Hills, IL, | Training and Mentoring | www.junit.org 60061 | OO, XP, Java, C++, Python |

[Uncle Bob (Robert C. Martin)]

tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this
on (or about) 10 Jul 2003 04:43:25 -0700, :
I would assume that any test that might find a bug is worth running.Correct?

Not if the test costs too much to build or to run. By definition,
such a test would not be worth the cost.



Robert C. Martin | "Uncle Bob"
Object Mentor Inc.| unclebob @ objectmentor . com
PO Box 5757 | Tel: (800) 338-6716
565 Lakeview Pkwy | Fax: (847) 573-1658 | www.objectmentor.com
Suite 135 | | www.XProgramming.com
Vernon Hills, IL, | Training and Mentoring | www.junit.org
60061 | OO, XP, Java, C++, Python |

[Uncle Bob (Robert C. Martin)]

tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this
on (or about) 10 Jul 2003 04:43:25 -0700, :
Now, it is possible to design a set of tests with this characteristic. However, it is not practical, it takes too long to design the set ofall tests that will find all possible bugs for most systems.

In other words, they are not worth running.

Robert C. Martin | "Uncle Bob"
Object Mentor Inc.| unclebob @ objectmentor . com
PO Box 5757 | Tel: (800) 338-6716
565 Lakeview Pkwy | Fax: (847) 573-1658 | www.objectmentor.com
Suite 135 | | www.XProgramming.com
Vernon Hills, IL, | Training and Mentoring | www.junit.org
60061 | OO, XP, Java, C++, Python |

[Tom Adams]

"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<ftaugv41uuh9ddm0qj6tu3rqc89c50dq6u@4ax.com>... tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this on (or about) 10 Jul 2003 04:43:25 -0700, :I would assume that any test that might find a bug is worth running.Correct? Not if the test costs too much to build or to run. By definition, such a test would not be worth the cost.

So, you have to accept the risk that the system will contain some
faults after delivery, faults that were not detected by all the test
that were worth running. Faults that may cause system failures when
the system is being used.

Do you use the operational profile of the system when selecting which
tests to run and which tests to not bother with running?

[Uncle Bob (Robert C. Martin)]

tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this
on (or about) 12 Jul 2003 05:10:37 -0700, :
"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<ftaugv41uuh9ddm0qj6tu3rqc89c50dq6u@4ax.com>... tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this on (or about) 10 Jul 2003 04:43:25 -0700, :I would assume that any test that might find a bug is worth running.Correct? Not if the test costs too much to build or to run. By definition, such a test would not be worth the cost.So, you have to accept the risk that the system will contain somefaults after delivery, faults that were not detected by all the testthat were worth running. Faults that may cause system failures whenthe system is being used.

Yes. We accept that such faults may exist.
Do you use the operational profile of the system when selecting whichtests to run and which tests to not bother with running?

Of course. The value of a test goes up as the system becomes more
critical.



Robert C. Martin | "Uncle Bob"
Object Mentor Inc.| unclebob @ objectmentor . com
PO Box 5757 | Tel: (800) 338-6716
565 Lakeview Pkwy | Fax: (847) 573-1658 | www.objectmentor.com
Suite 135 | | www.XProgramming.com
Vernon Hills, IL, | Training and Mentoring | www.junit.org
60061 | OO, XP, Java, C++, Python |

[Gerold Keefer]

"Uncle Bob (Robert C. Martin)" wrote:
Do you use the operational profile of the system when selecting whichtests to run and which tests to not bother with running? Of course. The value of a test goes up as the system becomes more critical.

of course, he did no get the question.

[Tom Adams]

Gerold Keefer <gkeefer@avoca-vsm.com> wrote in message news:<3F11D186.A0599AF0@avoca-vsm.com>... "Uncle Bob (Robert C. Martin)" wrote:Do you use the operational profile of the system when selecting whichtests to run and which tests to not bother with running? Of course. The value of a test goes up as the system becomes more critical. of course, he did no get the question.

Yes, Uncle Bob did not get the question. He does not understand the
term "operational profile". It has nothing to do with criticality. I
has to do with the relative frequency of use of the various features
of a system. See the definition in the Encyclopedia of Software
Engineering.

[Uncle Bob (Robert C. Martin)]

tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this
on (or about) 16 Jul 2003 10:31:32 -0700, :
Gerold Keefer <gkeefer@avoca-vsm.com> wrote in message news:<3F11D186.A0599AF0@avoca-vsm.com>... "Uncle Bob (Robert C. Martin)" wrote: >Do you use the operational profile of the system when selecting which >tests to run and which tests to not bother with running? Of course. The value of a test goes up as the system becomes more critical. of course, he did no get the question.Yes, Uncle Bob did not get the question. He does not understand theterm "operational profile". It has nothing to do with criticality. Ihas to do with the relative frequency of use of the various featuresof a system. See the definition in the Encyclopedia of SoftwareEngineering.

Pardon my ignorance. Can you rephrase the question?



Robert C. Martin | "Uncle Bob"
Object Mentor Inc.| unclebob @ objectmentor . com
PO Box 5757 | Tel: (800) 338-6716
565 Lakeview Pkwy | Fax: (847) 573-1658 | www.objectmentor.com
Suite 135 | | www.XProgramming.com
Vernon Hills, IL, | Training and Mentoring | www.junit.org
60061 | OO, XP, Java, C++, Python |

[Tom Adams]

"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<2e4bhv4vbh08a0d18hhlvtgolkc695u2i0@4ax.com>... tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this on (or about) 16 Jul 2003 10:31:32 -0700, :Gerold Keefer <gkeefer@avoca-vsm.com> wrote in message news:<3F11D186.A0599AF0@avoca-vsm.com>... "Uncle Bob (Robert C. Martin)" wrote: > >Do you use the operational profile of the system when selecting which > >tests to run and which tests to not bother with running? > > Of course. The value of a test goes up as the system becomes more > critical. of course, he did no get the question.Yes, Uncle Bob did not get the question. He does not understand theterm "operational profile". It has nothing to do with criticality. Ihas to do with the relative frequency of use of the various featuresof a system. See the definition in the Encyclopedia of SoftwareEngineering. Pardon my ignorance. Can you rephrase the question?

For some systems, you can estimate the operational profile of the
system. That is, you can estimate that Feature 1 will be used 10
times more than Feature 2, etc. If this is the case, then there is an
argument for testing Feature 1 ten times more than you teach feature
2.

In formal Software Reliability Engineering, you test via simulated
usage based on you estimates of future usage patterns. But there are
various informal approaches to accomplish this via usage-oriented
testing (simulated or provisional usage). These methods will test
Feature 1 ten times more than Feature 2. If the operational profile
is accurate, then Feature 1 will get "tested" ten times more than
Feature 2 after the system is delivered to its users.

Somewhere in this thread, I gave a reference on an abstract by
Williams and Vouk about "Good Enough Reliability" for XP. They want
the client to estimate the relative usage rates of the stories and
they want the number of automated system tests for each storie to
reflect these frequencies. This is sort of like usage-oriented
testing, only usage-oriented testing has a random selection aspect
lacking from the Williams/Vouk idea. But Vouk has done a lot of
research on the idea on trying to get some of the good effects of
Software Reliability Engineering without doing the random test
selection that is implied by true usage-oriented testing
(usage-oriented testing also called "operational testing")

Anyway, the whole issue of what tests are worth running is pretty
complicated, IMO.

One universal attribute of all good soup-to-nuts software engineering
methodologies is that they include at least two defect finding
techniques. One is good at reducing the defect density, the number of
defect per lines of code. The other is good at increasing
reliability, that is reducing mean-time-to-failure.

[Code Like Hell]

Absolute understanding costs too much. Understanding software
testing...well that's pretty far down on the list. Now and then I'll
read snippets of Kaner/Bach/Pettichord, just for laughs. Like,
"business context" he he, what's that? Et Tu Brute? That's yer
business context. Bloody mess.

Anyhow, customer not paying for my understanding, unless it
increases my velocity on a story of particular importance. In other
words, you don't want to pay the electric bill on a 4000 watt bulb if
you don't need it. Absolute understanding has a fixed duration, and
it costs money to keep it lit up for that time, then it goes out.
Period. The rest is subjective illusion, within the time the
lightbulb is state subsidized. Sometimes you do need to be picked up
off your track and placed in a different location, I'll grant you
that. Now and then understanding is necessary.


tadamsmar@yahoo.com (Tom Adams) wrote in message news:<ea44f5a1.0307030341.43644af7@posting.google.com>... "Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<cf26gv8huthen9okatre9ocifbfjvoigvb@4ax.com>... tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this on (or about) 2 Jul 2003 04:52:33 -0700, :"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<pql4gvkl499ptd8deqldtj660ev6u70vc9@4ax.com>...> tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this> on (or about) 1 Jul 2003 11:19:21 -0700, :>> >Do XPer's actually promote this bizarre, doctrinaire,> >programmer's-eye-view of the process? I doubt that most clients> >actually treat the small releases as ready for production.>> Yes and no. Yes, in that you must be willing to ship anything you> check in. No, in that we all know that the customer has his own time> frame for putting things into production.This is not about customer time frames. This is about what it reallytakes to prepare code for production. And what is that? What does it really take to prepare code for production?I don't see the point inpromoting a "methodology" that cannot create production code. Neither do I.>> The point is simply this. We don't check in crap. We check in code> many times per day. Any code we check in passes all the tests we can> think of,All tests we can think of? Please make the effort required toconstruct statements that are not obviously false after a millisecondof examination. Don't be obtuse. I am not a fool. Make a real point. My point is that you are not doing all the test you can think of. It is not practical to do all the test you can think of. If you current with the softare testing literature circa 1980 (that is, only 23 years behind) you would never say something like "we do all the test we can think of". And, understanding the true limitations of testing, rather than spouting these XP brainwashing mantras, is the starting point for understanding software testing.

[Uncle Bob (Robert C. Martin)]

tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this
on (or about) 17 Jul 2003 05:28:19 -0700, :
"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<2e4bhv4vbh08a0d18hhlvtgolkc695u2i0@4ax.com>... tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this on (or about) 16 Jul 2003 10:31:32 -0700, :Gerold Keefer <gkeefer@avoca-vsm.com> wrote in message news:<3F11D186.A0599AF0@avoca-vsm.com>...> "Uncle Bob (Robert C. Martin)" wrote:>> > >Do you use the operational profile of the system when selecting which> > >tests to run and which tests to not bother with running?> >> > Of course. The value of a test goes up as the system becomes more> > critical.>> of course, he did no get the question.Yes, Uncle Bob did not get the question. He does not understand theterm "operational profile". It has nothing to do with criticality. Ihas to do with the relative frequency of use of the various featuresof a system. See the definition in the Encyclopedia of SoftwareEngineering. Pardon my ignorance. Can you rephrase the question?For some systems, you can estimate the operational profile of thesystem. That is, you can estimate that Feature 1 will be used 10times more than Feature 2, etc. If this is the case, then there is anargument for testing Feature 1 ten times more than you teach feature2.

Again, pardon my ignorance. I presume that the operational profile
includes more than just the *frequency* of a feature, but also the
criticality of each feature. For example, the software for a credit
card scanner will scan lots of cards, but will only do its upload to
the credit card company once per day. And yet a single failure in the
upload software could cost much more than a single failure in the
scanning software.
In formal Software Reliability Engineering, you test via simulatedusage based on you estimates of future usage patterns. But there arevarious informal approaches to accomplish this via usage-orientedtesting (simulated or provisional usage). These methods will testFeature 1 ten times more than Feature 2. If the operational profileis accurate, then Feature 1 will get "tested" ten times more thanFeature 2 after the system is delivered to its users.

Are we talking about ten times the number of test cases, or are we
just talking about running the same tests ten times more frequently?
Somewhere in this thread, I gave a reference on an abstract byWilliams and Vouk about "Good Enough Reliability" for XP. They wantthe client to estimate the relative usage rates of the stories andthey want the number of automated system tests for each storie toreflect these frequencies.

I presume they also want an estimate for the cost of failure.
This is sort of like usage-orientedtesting, only usage-oriented testing has a random selection aspectlacking from the Williams/Vouk idea. But Vouk has done a lot ofresearch on the idea on trying to get some of the good effects ofSoftware Reliability Engineering without doing the random testselection that is implied by true usage-oriented testing(usage-oriented testing also called "operational testing")Anyway, the whole issue of what tests are worth running is prettycomplicated, IMO.

And yet, it is an issue that must be resolved if we are to test the
system.
One universal attribute of all good soup-to-nuts software engineeringmethodologies is that they include at least two defect findingtechniques. One is good at reducing the defect density, the number ofdefect per lines of code. The other is good at increasingreliability, that is reducing mean-time-to-failure.

Would you consider the XP split of unit testing and acceptance testing
as a reasonable first approximation of this goal?

Robert C. Martin | "Uncle Bob"
Object Mentor Inc.| unclebob @ objectmentor . com
PO Box 5757 | Tel: (800) 338-6716
565 Lakeview Pkwy | Fax: (847) 573-1658 | www.objectmentor.com
Suite 135 | | www.XProgramming.com
Vernon Hills, IL, | Training and Mentoring | www.junit.org
60061 | OO, XP, Java, C++, Python |

[Ron Jeffries]

On 17 Jul 2003 05:28:19 -0700, tadamsmar@yahoo.com (Tom Adams) wrote:
For some systems, you can estimate the operational profile of thesystem. That is, you can estimate that Feature 1 will be used 10times more than Feature 2, etc. If this is the case, then there is anargument for testing Feature 1 ten times more than you teach feature2.

Wouldn't the impact of an error be important, not just the frequency of use?

We only land the plane once per flight, but we would really like it to work
right.

--
Ronald E Jeffries
http://www.XProgramming.com
http://www.objectmentor.com
I'm giving the best advice I have. You get to decide whether it's true for you.

[Ron Jeffries]

On Sun, 20 Jul 2003 11:22:51 -0400, Ron Jeffries <ronjeffries@REMOVEacm.org>
wrote:
Wouldn't the impact of an error be important, not just the frequency of use?We only land the plane once per flight, but we would really like it to workright.

Ah. I see that this was addressed in subsequent notes. That's what I get for
trying to catch up.

--
Ronald E Jeffries
http://www.XProgramming.com
http://www.objectmentor.com
I'm giving the best advice I have. You get to decide whether it's true for you.

[Tom Adams]

Ron Jeffries <ronjeffries@REMOVEacm.org> wrote in message news:<dtclhvksr2nivfvglndr3vf8fefndhul62@4ax.com>... On 17 Jul 2003 05:28:19 -0700, tadamsmar@yahoo.com (Tom Adams) wrote:For some systems, you can estimate the operational profile of thesystem. That is, you can estimate that Feature 1 will be used 10times more than Feature 2, etc. If this is the case, then there is anargument for testing Feature 1 ten times more than you teach feature2. Wouldn't the impact of an error be important, not just the frequency of use?

Yes, but not directly addressed by basic Software Reliability
Engineering. If you had impact classes, then you could come up with a
reliability measurement for each of the impact classes.
We only land the plane once per flight, but we would really like it to work right.

True. Have you heard of "Separation of concerns" as a principle in
solving complex problems? Basic Software Reliability Engineering
addresses overall software reliability. But it is true that you might
want a part of a system to have higher reliability. This can be
addressed by more operational testing I suppose, but it is typically
addressed by multiple techniques, I think. See _Software Assessment:
Reliability, Safety, Testability_ by Michael Friedman and Jeffrey
Voas.

The Space Shuttle software team must have come up with some approach
to estimate errors that cause fatalities separate from other errors,
since they estimated that a software bug would kill the crew in 1 of
every 250 flights. That was considered good enough since the risk for
hardware was 1 in 25 flights.

[Richard MacDonald]

tadamsmar@yahoo.com (Tom Adams) wrote in
news:ea44f5a1.0307180427.84af866@posting.google.com:
"Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<6jtdhvkls3ovapn72nef547g1ue5aemtq5@4ax.com>... tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this on (or about) 17 Jul 2003 05:28:19 -0700, :In formal Software Reliability Engineering, you test via simulatedusage based on you estimates of future usage patterns. But thereare various informal approaches to accomplish this viausage-oriented testing (simulated or provisional usage). Thesemethods will test Feature 1 ten times more than Feature 2. If theoperational profile is accurate, then Feature 1 will get "tested"ten times more than Feature 2 after the system is delivered to itsusers. Are we talking about ten times the number of test cases, or are we just talking about running the same tests ten times more frequently? Ten times more test cases. Randomly selected test cases from the operational profile, simulating ten times more usage.

That's pretty weak stuff. An additional ten times randomly selected test
cases is gonna provide very little additional value: "Lessee, I checked 2
+2 and it worked ok. Now I'll try 2+i for i=1 to 10 (or randomly selected
i in the domain [1.10]). Oh, that worked too. Now I feel much better."

Why not provide ten times additional brain power and look for some
additional boundary conditions or possible execution paths to examine.

And since the computer is sitting there idle all night long, why aren't
you just running random tests the entire night? Why bother with just
tenfold? Weighted according to critically*frequency, of course.

Anyway, XP says you need to act stupid and not think outside the box, so
all these ideas are expressly forbidden...naturally, since they may not
have mentioned them in the XP bible, and no XPer would be caught dead
actually reading a, gasp, book on testing.

[Tom Adams]

Richard MacDonald <macdonaldrj@worldnet.att.net> wrote in message news:<Xns93BFC5C2ECBC6macdonaldrjworldneta@204.127.36.1>... tadamsmar@yahoo.com (Tom Adams) wrote in news:ea44f5a1.0307180427.84af866@posting.google.com: "Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<6jtdhvkls3ovapn72nef547g1ue5aemtq5@4ax.com>... tadamsmar@yahoo.com (Tom Adams) might (or might not) have written this on (or about) 17 Jul 2003 05:28:19 -0700, : >In formal Software Reliability Engineering, you test via simulated >usage based on you estimates of future usage patterns. But there >are various informal approaches to accomplish this via >usage-oriented testing (simulated or provisional usage). These >methods will test Feature 1 ten times more than Feature 2. If the >operational profile is accurate, then Feature 1 will get "tested" >ten times more than Feature 2 after the system is delivered to its >users. Are we talking about ten times the number of test cases, or are we just talking about running the same tests ten times more frequently? Ten times more test cases. Randomly selected test cases from the operational profile, simulating ten times more usage. That's pretty weak stuff. An additional ten times randomly selected test cases is gonna provide very little additional value: "Lessee, I checked 2 +2 and it worked ok. Now I'll try 2+i for i=1 to 10 (or randomly selected i in the domain [1.10]). Oh, that worked too. Now I feel much better." Why not provide ten times additional brain power and look for some additional boundary conditions or possible execution paths to examine.

I know it seems counter-intuitive, but random testing has worked as
good as or better than boundary condition/execution path coverage
testing in a number of head-to-head experiments. See Dick Hamlet's
paper "Partition Testing does not Inspire Confidence".
And since the computer is sitting there idle all night long, why aren't you just running random tests the entire night? Why bother with just tenfold? Weighted according to critically*frequency, of course.

There is the "test oracle" problem. You would need an automated way
to confirm that each test ran correctly. It can be tons of work
creating a test oracle that can automatically detect all test
failures. The amount of work involved in creating the test oracle is
the show-stopper.
Anyway, XP says you need to act stupid and not think outside the box, so all these ideas are expressly forbidden...naturally, since they may not have mentioned them in the XP bible, and no XPer would be caught dead actually reading a, gasp, book on testing.

Many books on testing never even mention the word "reliability", just
check the index of a few. Most are not a good source of info on the
most effective ways to make your software more realiable.

Anyway, XP has this idea that it does not apply to lots of software
projects. There was a recent article that showed %85 of software
developers are working on projects that are outside the scope of XP,
that is the scope defined by the *advocates* of XP. I think that
XPers might not be put off by operationl testing or the use an
operational profile, and it could be a way to extend the scope of XP.

[Richard MacDonald]

tadamsmar@yahoo.com (Tom Adams) wrote in
news:ea44f5a1.0307220429.76b7acd4@posting.google.com:
Richard MacDonald <macdonaldrj@worldnet.att.net> wrote in message news:<Xns93BFC5C2ECBC6macdonaldrjworldneta@204.127.36.1>... tadamsmar@yahoo.com (Tom Adams) wrote in news:ea44f5a1.0307180427.84af866@posting.google.com: "Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> wrote in message news:<6jtdhvkls3ovapn72nef547g1ue5aemtq5@4ax.com>...> tadamsmar@yahoo.com (Tom Adams) might (or might not) have written> this on (or about) 17 Jul 2003 05:28:19 -0700, :>>> >In formal Software Reliability Engineering, you test via> >simulated usage based on you estimates of future usage patterns.> > But there are various informal approaches to accomplish this via> >usage-oriented testing (simulated or provisional usage). These> >methods will test Feature 1 ten times more than Feature 2. If> >the operational profile is accurate, then Feature 1 will get> >"tested" ten times more than Feature 2 after the system is> >delivered to its users.>> Are we talking about ten times the number of test cases, or are we> just talking about running the same tests ten times more> frequently? Ten times more test cases. Randomly selected test cases from the operational profile, simulating ten times more usage. That's pretty weak stuff. An additional ten times randomly selected test cases is gonna provide very little additional value: "Lessee, I checked 2 +2 and it worked ok. Now I'll try 2+i for i=1 to 10 (or randomly selected i in the domain [1.10]). Oh, that worked too. Now I feel much better." Why not provide ten times additional brain power and look for some additional boundary conditions or possible execution paths to examine. I know it seems counter-intuitive, but random testing has worked as good as or better than boundary condition/execution path coverage testing in a number of head-to-head experiments. See Dick Hamlet's paper "Partition Testing does not Inspire Confidence".

I'm afraid I cannot because Im not an ACM member, citeseer doesn't
archive it, and the author doesn't have it on his website. But I can
imagine. Still, you get my drift? If the buggy boundary condition is 1e-6
of the total "space" of the testing possibilities, increasing the number
of random tests by tenfold isn't likely to catch it. (e.g., how many
random tests would I have to generate to catch a numeric overflow in int
addition?)
And since the computer is sitting there idle all night long, why aren't you just running random tests the entire night? Why bother with just tenfold? Weighted according to critically*frequency, of course. There is the "test oracle" problem. You would need an automated way to confirm that each test ran correctly. It can be tons of work creating a test oracle that can automatically detect all test failures. The amount of work involved in creating the test oracle is the show-stopper.

If you can generate an additional tenfold random tests you can just as
easily generate an additional millionfold. What issues are present in the
overnight random testing vs your tenfold suggestion?
Anyway, XP says you need to act stupid and not think outside the box, so all these ideas are expressly forbidden...naturally, since they may not have mentioned them in the XP bible, and no XPer would be caught dead actually reading a, gasp, book on testing. Many books on testing never even mention the word "reliability", just check the index of a few. Most are not a good source of info on the most effective ways to make your software more realiable.

Funnily enough, I have a PhD on the subject of "reliability". In chemical
engineering though, and it means something else in that field, or at
least it leads to a different type of statistical analysis. But whatever.
My point is that most of us have a good practical intuition for the
subject, so if you treat us as simpletons, don't expect to be taken
seriously.
Anyway, XP has this idea that it does not apply to lots of software projects. There was a recent article that showed %85 of software developers are working on projects that are outside the scope of XP, that is the scope defined by the *advocates* of XP. I think that XPers might not be put off by operationl testing or the use an operational profile, and it could be a way to extend the scope of XP.
First I'd challenge that article. Second, I don't particularly care
because I guess I just work in the other 15%. Third, I think XPers would
not be put off by the idea. Fourth, they're definitely put off by *your*
presention of the idea (see a post by Phlip a while back: when the
messenger comes off like a know-it-all talking down to idiots, the
messenger is wasting his time). Fifth, I fail to see how more rigorous
testing applied to XP would increase its range of applicability...because
the rigor of testing is basically orthogonal to XP, i.e., its already
there if you desire it. Nothing in XP says you have to stop adding tests,
stop applying additional test techniques, reading a book about testing,
etc, etc.

[Tom Adams]

Richard MacDonald <macdonaldrj@worldnet.att.net> wrote in message news:<Xns93C219CE1F61macdonaldrjworldneta@204.127.36.1>... tadamsmar@yahoo.com (Tom Adams) wrote in news:ea44f5a1.0307220429.76b7acd4@posting.google.com: Richard MacDonald <macdonaldrj@worldnet.att.net> wrote in message news:<Xns93BFC5C2ECBC6macdonaldrjworldneta@204.127.36.1>... tadamsmar@yahoo.com (Tom Adams) wrote in news:ea44f5a1.0307180427.84af866@posting.google.com: > "Uncle Bob (Robert C. Martin)" <u.n.c.l.e.b.o.b@objectmentor.com> > wrote in message > news:<6jtdhvkls3ovapn72nef547g1ue5aemtq5@4ax.com>... >> tadamsmar@yahoo.com (Tom Adams) might (or might not) have written >> this on (or about) 17 Jul 2003 05:28:19 -0700, : >> >> >> >In formal Software Reliability Engineering, you test via >> >simulated usage based on you estimates of future usage patterns. >> > But there are various informal approaches to accomplish this via >> >usage-oriented testing (simulated or provisional usage). These >> >methods will test Feature 1 ten times more than Feature 2. If >> >the operational profile is accurate, then Feature 1 will get >> >"tested" ten times more than Feature 2 after the system is >> >delivered to its users. >> >> Are we talking about ten times the number of test cases, or are we >> just talking about running the same tests ten times more >> frequently? > > Ten times more test cases. Randomly selected test cases from the > operational profile, simulating ten times more usage. > That's pretty weak stuff. An additional ten times randomly selected test cases is gonna provide very little additional value: "Lessee, I checked 2 +2 and it worked ok. Now I'll try 2+i for i=1 to 10 (or randomly selected i in the domain [1.10]). Oh, that worked too. Now I feel much better." Why not provide ten times additional brain power and look for some additional boundary conditions or possible execution paths to examine. I know it seems counter-intuitive, but random testing has worked as good as or better than boundary condition/execution path coverage testing in a number of head-to-head experiments. See Dick Hamlet's paper "Partition Testing does not Inspire Confidence". I'm afraid I cannot because Im not an ACM member, citeseer doesn't archive it, and the author doesn't have it on his website. But I can imagine.

Here is a recent paper on the subject, but it is more of a survey and
does not go into much detail about the experiments:

http://www.utdallas.edu/~ntafos/issta98.doc

Ntafos did the first experiment, before Hamlet.

The latest article on the matter is by Ntafos published in the July
2003 IEEE Transaction on Software Engineering.

I think that if you read the papers there are people making arguments
similar to yours. But random testing holds up amazingly well when
compared in a controlled experiment. However, the dust has certainly
not settled on the matter.
Still, you get my drift? If the buggy boundary condition is 1e-6 of the total "space" of the testing possibilities, increasing the number of random tests by tenfold isn't likely to catch it. (e.g., how many random tests would I have to generate to catch a numeric overflow in int addition?) And since the computer is sitting there idle all night long, why aren't you just running random tests the entire night? Why bother with just tenfold? Weighted according to critically*frequency, of course. There is the "test oracle" problem. You would need an automated way to confirm that each test ran correctly. It can be tons of work creating a test oracle that can automatically detect all test failures. The amount of work involved in creating the test oracle is the show-stopper. If you can generate an additional tenfold random tests you can just as easily generate an additional millionfold. What issues are present in the overnight random testing vs your tenfold suggestion?

Well, no, unless you have a test oracle for every possible test of
your system.
Anyway, XP says you need to act stupid and not think outside the box, so all these ideas are expressly forbidden...naturally, since they may not have mentioned them in the XP bible, and no XPer would be caught dead actually reading a, gasp, book on testing. Many books on testing never even mention the word "reliability", just check the index of a few. Most are not a good source of info on the most effective ways to make your software more realiable. Funnily enough, I have a PhD on the subject of "reliability". In chemical engineering though, and it means something else in that field, or at least it leads to a different type of statistical analysis. But whatever.

Are you sure that it means something different? The situation with
testing vs reliability is very odd in the software field. It is the
only field I know of where one group of authors write books on testing
without discussing reliability and another group of authors write
books on reliability. I think this is due to a sort of implicit
belief that software can be made perfect by testing so mere
reliability is not the issue. No one will actually say they believe
that, but I think that the idea has some sort of influence on the
authors of books on software testing.
My point is that most of us have a good practical intuition for the subject, so if you treat us as simpletons, don't expect to be taken seriously. Anyway, XP has this idea that it does not apply to lots of software projects. There was a recent article that showed %85 of software developers are working on projects that are outside the scope of XP, that is the scope defined by the *advocates* of XP. I think that XPers might not be put off by operationl testing or the use an operational profile, and it could be a way to extend the scope of XP. First I'd challenge that article.

It was in a counter-point with Ken Beck in the recent IEEE Software.
Beck did not challenge it in the debate, but I guess he might not have
had the
opportunity to do so.

Remember that the statement was about %85 percent of total effort, not
total projects. XP tends to be used on smaller projects.
Second, I don't particularly care because I guess I just work in the other 15%. Third, I think XPers would not be put off by the idea. Fourth, they're definitely put off by *your* presention of the idea (see a post by Phlip a while back: when the messenger comes off like a know-it-all talking down to idiots, the messenger is wasting his time). Fifth, I fail to see how more rigorous testing applied to XP would increase its range of applicability...because the rigor of testing is basically orthogonal to XP, i.e., its already there if you desire it. Nothing in XP says you have to stop adding tests, stop applying additional test techniques, reading a book about testing, etc, etc.

[Tom Adams]

Richard MacDonald <macdonaldrj@worldnet.att.net> wrote in message news:<Xns93C219CE1F61macdonaldrjworldneta@204.127.36.1>... Fifth, I fail to see how more rigorous testing applied to XP would increase its range of applicability...because the rigor of testing is basically orthogonal to XP, i.e., its already there if you desire it. Nothing in XP says you have to stop adding tests, stop applying additional test techniques, reading a book about testing, etc, etc.

NP is a superior methodology to XP. NP, Null Programming, has no key
practices. As a matter of fact, NP has no practices whatsoever.

Unlike XP, NP has no practices to object to. And, NP is vastly more
flexible than XP.

Now, if someone objects to the absence of a particular practice, you
just say "its already there, since you are premitted to tailor NP"

[Richard MacDonald]

tadamsmar@yahoo.com (Tom Adams) wrote in
news:ea44f5a1.0307240428.6d7e1f05@posting.google.com:
Richard MacDonald <macdonaldrj@worldnet.att.net> wrote in message news:<Xns93C219CE1F61macdonaldrjworldneta@204.127.36.1>... tadamsmar@yahoo.com (Tom Adams) wrote in news:ea44f5a1.0307220429.76b7acd4@posting.google.com: I know it seems counter-intuitive, but random testing has worked as good as or better than boundary condition/execution path coverage testing in a number of head-to-head experiments. See Dick Hamlet's paper "Partition Testing does not Inspire Confidence". I'm afraid I cannot because Im not an ACM member, citeseer doesn't archive it, and the author doesn't have it on his website. But I can imagine. Here is a recent paper on the subject, but it is more of a survey and does not go into much detail about the experiments: http://www.utdallas.edu/~ntafos/issta98.doc

Ok, I did understand the issue. Its basically "Does stratified/random
testing always beat random testing?" And the article(s) will discover the
answer is "No", because its a random test comparison so you're always
playing percentages. So there is no such thing as "always".

IMHO, I can feel justified in ignoring any specific tests/studies in any
specific domain, because the underlying statistics is the same. If you
have intelligence in determining the correct partitions, and you have the
ability to proportion your number of tests properly within each
partition, then you *will* improve your odds by using partitions. Its a
Monte Carlo "shortcut" that works. No amount of test simulations is going
to alter that fundamental principle. If the results show different, then
we need to look at the other factors which are skewing the assumptions
into the statistical model.

But as all statistical principles, go, its only true on average. And it
can be a small "on average".

Of course, the other key is "intelligent insight into the 'proper'
partitions". If you're pissing into the wind, you should not bother:
Simply run the random testing alone. FWIW, in my chemical engineering
work, I always held to this tendency, because the extra effort and extra
complexity was never worth it. So, for example, if I'm integrating a
function, I'll always use Monte Carlo rather than "stratified" Monte
Carlo, even though it can be shown that the latter usually performs
better on the simpler problems. But the advantage lessens as the
dimensionality increases, the function becomes more "unusual", etc, so
why bother? You have a good-enough, lifetime-guaranteed hammer already,
so just use it on all types of nails.

I haven't spent much time on the idea, but what about code/path coverage?
Easy enough to cover the code, but it becomes much more difficult to
handle path coverage as the size of the problem increases. Probably its
the same issue.

The article is probably correct: When the effort to determine the
partitioning is high -- and its high in software -- then why not just let
the poor computer bash out an extra magnitude or two of tests rather than
trying to be smart?

But the counterpoint is also valid: If you can identify a rare but
significant partition, then perhaps you should spend the time to write a
test for it, rather than hoping the random testing will find it. Its
simple cost-benefit, albeit with a difficult-to-quantify underlying
model.

If you can generate an additional tenfold random tests you can just as easily generate an additional millionfold. What issues are present in the overnight random testing vs your tenfold suggestion? Well, no, unless you have a test oracle for every possible test of your system.

If you don't have a test oracle, you don't have random tests.
Anyway, XP has this idea that it does not apply to lots of software projects. There was a recent article that showed %85 of software developers are working on projects that are outside the scope of XP, that is the scope defined by the *advocates* of XP. I think that XPers might not be put off by operationl testing or the use an operational profile, and it could be a way to extend the scope of XP. First I'd challenge that article. It was in a counter-point with Ken Beck in the recent IEEE Software. Beck did not challenge it in the debate, but I guess he might not have had the opportunity to do so. Remember that the statement was about %85 percent of total effort, not total projects. XP tends to be used on smaller projects.

Oh, ok. Well, if the speaker has statistics to back it up, fine. If not,
lump it in with every other hand-waving "statistic" that is bandied about
and gains "truth through repetition".

[John Roth]

"Tom Adams" <tadamsmar@yahoo.com> wrote in message
news:ea44f5a1.0307280416.49d2fcd0@posting.google.com... I realize that XP formally contains no testing that does not involve automatic checking of the test results. But, I bet that customers usually or always do some non-automated operational testing before putting a release into production.

Well, yes. One of the major criteria for "doing XP" is a fully
engaged customer. I find it difficult to imagine a fully engaged
customer that isn't at least playing with the system as it's being
developed.

John Roth

[Tom Plunket]

Phlip wrote:
In this case, of course the "onsite customer" role tests manually. They are expected to do that every day, once per feature or more, over the coders' shoulders, and as the product goes out the door.

Just think about how much fun that role is in a video game
company. ;)

-tom!

[Phlip]

Tom Plunket wrote: Phlip wrote: In this case, of course the "onsite customer" role tests manually. They are expected to do that every day, once per feature or more, over the coders' shoulders, and as the product goes out the door. Just think about how much fun that role is in a video game company. ;)

Tom!, "XP in game development" is a FAQ on games fora and on the XP mailing
list.

I never get involved because I figure if you did you could apply experience
instead of sophistry. And I myself would also like to learn about it. Please
look those threads up!

--
Phlip
http://www.c2.com/cgi/wiki?TestFirstUserInterfaces

[Tom Plunket]

Phlip wrote:
Tom!, "XP in game development" is a FAQ on games fora and on the XP mailing list.

Yes, unfortunately there's only so much time in the day. ;) I
follow a few of the key newsgroups, but the XP mailing list gets
WAY too much traffic for me to even read. I may try subscribing
anyway, but...

-tom!