Scalability a Boojum

[Felger Carbon]

Full disclosure: I've always been keenly interested in latency as a
limiting factor in the performance of a cluster of standard desktop
CPUs to make a supercomputer. And I've been reading a lot about
supercomputers lately. On comp.arch, there'r lotsa interesting URLs
posted by folks like Robert Myers and David Wang.

I think I just got one of them "epiphany" things. ;-)

1. I just read on comp.arch that arithmetic is free these days;
the expense is in the communications and I/O. Followed by a
technical analysis which attempted to prove the point.

2. I just scanned a Red Storm .pdf file whose URL was provided by
Robert:

<http://www.lanl.gov/orgs/ccn/salishan2003/pdf/camp.pdf>

As he pointed out, slides 25 et seq are indeed interesting. They're
about "scalability", which is the what big clusters typically don't
have, mostly due to poor communications.

3. The Earth Simulator, the world's current leading supercomputer,
costs at least $400M, and I've seen the figure $600M. Big Mac was
put together by slave labor at Va Tech and cost $5.2M, roughly 1% of
the ES price tag. Yet Big Mac has 25% of the performance of the ES
as measured by Linpack. That means it's 25 times as cost-efficient
(on Linpack type workloads).

4. Question: is scalability so important that an infinite amount
of money should be thrown at it? Obviously not, "infinite" is a
lotta gelt. I propose something heretical: let's ignore
scalability. It's a Boojum, a phantom with no value. What _does_
have value? Two things: money has value, as does a solution to a
computing problem inside of a time limit.

5. I suggest that the "best" supercomputer for a given job is the
one that will provide a solution to a given problem in the desired
time ***for the least money***. If the scalability is lousy, so
what? If arithmetic is free, then scalability literally doesn't
matter! Honest!

If it's cheaper to provide a given level of performance by
increasing the number of arithmetic units (CPUs) while using cheaper
communications, then that's the way to go, even though by definition
this results in poorer scalability. Scalability really _is_ a
Boojum (apologies to Lewis Carrol).

If power consumption is more important than money, one winds up with
something like IBM's Blue Gene, which has also been in the news
lately. But that's a topic for another day. ;-)

[Robert Myers]

On Mon, 17 Nov 2003 21:37:38 GMT, "Felger Carbon" <fmsfnf@jfoops.net>
wrote:
If it's cheaper to provide a given level of performance byincreasing the number of arithmetic units (CPUs) while using cheapercommunications, then that's the way to go, even though by definitionthis results in poorer scalability. Scalability really _is_ aBoojum (apologies to Lewis Carrol).

If only it were so.

From http://www.top500.org/

"The best performance on the Linpack benchmark is used as performance
measure for ranking the computer systems."

In an exchange on comp.arch which is memorable for me because I picked
a fight and got set upon by a pack of dogs, John McCalpin, who is the
keeper of the STREAM memory bandwidth benchmark, corrected my use of
the term "Linpack Benchmark" and stated that Linpack uses very little
memory bandwidth:

:In article <KlcQ8.14275$EP.1273@sccrnsc03>,
:Robert Myers <rmyers1400@attbi.com> wrote:
:>There are plenty of Linpack benchmarks to be found in which performance for
:>sufficiently large matrices is purely a matter of memory throughput, and
:>those benchmarks are absolutely relevant to my needs.
:
:Ouch! Bzzt -- wrong answer....
:
:"LINPACK benchmarks" are the prototypical *low-bandwidth* benchmarks.
:The standard "LINPACK 1000" benchmark performs about 667 Million FP
:operations on 8 Million Bytes of data. If the solver is structured
:correctly (and the ones in the LINPACK and LAPACK libraries are), then
:the memory bandwidth utilization should be tiny.
:
:On the SGI Origin2000, I measured a whopping 25 MB/s memory bandwidth
:utilization while running LINPACK at 344 MFLOPS. The bandwidth usage
:could probably have been even lower, but there was no point in further
:optimizing that aspect of the code, since it was already running at
:88% of the peak capability of the processor.
:
:Smaller caches don't result in much more bandwidth demand for LINPACK,
:due to the nature of the DGEMM kernel in the solver routine. So on a
:GHz class PentiumIII, which should sustain about twice what the 1996-era
:Origin2000 managed, it looks like LINPACK might need as much as 50 MB/s
:of main memory bandwidth.
:
:--
:John D. McCalpin, Ph.D. mccalpin@austin.ibm.com
:Senior Technical Staff Member IBM POWER Microprocessor Development

If you use a LINPACK implementation that is not cache-aware and let a
naive implementation spill out of cache, performance is eventually
limited by memory bandwidth, as I stated. If you use a *blocked*
implementation of LINPACK that is cache-aware, you can achieve very
high throughput with very low memory bandwidth as John McCalpin
stated.

The same argument carries through to clusters and to interconnect
bandwidth. The Top 500 list is almost a fraud. The chosen benchmark,
LINPACK, can be done in such a way as to depend hardly on the
interconnect at all.

That's how people can so carelessly throw together boxes and get
"supercomputers."

Most problems don't partition nearly that nicely. Some problems don't
really partition at all. The former problems don't require much
sophistication in the way of interconnect; the latter probably should
be done on shared-memory architecture machines.

Many problems that people want to do on supercomputers lie somewhere
in between. The creators of RedStorm would not have gone to such
lengths to achieve low latency and high bandwidth if it didn't matter,
and the slides in the LANL presentation argue that it does matter.
For LINPACK it doesn't, but LINPACK as a yardstick for
"supercomputers" was chosen by people who wanted to make huge machines
with limited interconnect look good.

This topic doesn't get discussed much because, in my opinion, there is
a conspiracy of silence around it. Above a certain level in the
federal bureaucracy, almost no decisionmakers that matter really
understand the issue, and once congressmen got it into their head that
there was such a thing as a "scaling" problem with big machines,
that's all they would remember.

RM

[Felger Carbon]

"Robert Myers" <rmyers@rustuck.com> wrote in message
news:kliirvcs5hnnltqp7llck23srt5erk41ng@4ax.com... If you use a LINPACK implementation that is not cache-aware and let a naive implementation spill out of cache, performance is eventually limited by memory bandwidth, as I stated. If you use a *blocked* implementation of LINPACK that is cache-aware, you can achieve very high throughput with very low memory bandwidth as John McCalpin stated. The same argument carries through to clusters and to interconnect bandwidth. The Top 500 list is almost a fraud. The chosen benchmark, LINPACK, can be done in such a way as to depend hardly on the interconnect at all. That's how people can so carelessly throw together boxes and get "supercomputers." Most problems don't partition nearly that nicely. Some problems don't really partition at all. The former problems don't require much sophistication in the way of interconnect; the latter probably should be done on shared-memory architecture machines.

Robert, I agree 100% with the above. No reservations whatever. However, it
has nothing to do with the point I was making. Let me repeat:

"5. I suggest that the "best" supercomputer for a given job is the
one that will provide a solution to a given problem in the desired
time ***for the least money***. If the scalability is lousy, so
what? If arithmetic is free, then scalability literally doesn't
matter! Honest!

"If it's cheaper to provide a given level of performance by
increasing the number of arithmetic units (CPUs) while using cheaper
communications, then that's the way to go, even though by definition
this results in poorer scalability. Scalability really _is_ a
Boojum (apologies to Lewis Carrol)."

Note that I say nothing about Linpack, but only "for a given job".
Many problems that people want to do on supercomputers lie somewhere in between. The creators of RedStorm would not have gone to such lengths to achieve low latency and high bandwidth if it didn't matter, and the slides in the LANL presentation argue that it does matter. For LINPACK it doesn't, but LINPACK as a yardstick for "supercomputers" was chosen by people who wanted to make huge machines with limited interconnect look good. This topic doesn't get discussed much because, in my opinion, there is a conspiracy of silence around it. Above a certain level in the federal bureaucracy, almost no decisionmakers that matter really understand the issue, and once congressmen got it into their head that there was such a thing as a "scaling" problem with big machines, that's all they would remember. RM

[Robert Myers]

On Tue, 18 Nov 2003 00:23:11 GMT, "Felger Carbon" <fmsfnf@jfoops.net>
wrote:

<snip>Robert, I agree 100% with the above. No reservations whatever. However, ithas nothing to do with the point I was making. Let me repeat:"5. I suggest that the "best" supercomputer for a given job is theone that will provide a solution to a given problem in the desiredtime ***for the least money***. If the scalability is lousy, sowhat? If arithmetic is free, then scalability literally doesn'tmatter! Honest!"If it's cheaper to provide a given level of performance byincreasing the number of arithmetic units (CPUs) while using cheapercommunications, then that's the way to go, even though by definitionthis results in poorer scalability. Scalability really _is_ aBoojum (apologies to Lewis Carrol)."
If you need to move alot of information around, you need to move alot
of information around. If you're moving it around only one-tenth as
fast, and that's the rate-limiting step, then you have to use your
capital equipment ten times as long to do the same amount of work.
Your $2,000 cheapy G5 node with a 1 gigabit connection just turned
into a $20,000 box as opposed to one with a 10 gigabit connection.
You can buy pretty fancy interconnect for the $18,000 difference.
There are similar factors of 10 (or more) to be found in
latency-limited calculations.

If you're using capital equipment ten times as long, you're also using
up physical plant resources (real estate, etc) ten times as long.
True TCO calculations can get really complicated.

The key here is *balance*. The cost of memory is not a
design-variable (except in a cache-only system). When the Virginia
Tech system does have Infiniband, it will spend about as much on
interconnect as it is spending on compute nodes. That's a pretty
sensible balance of resources (remember, this is a usenet post, not a
whitepaper). The beef I have with the Va. Tech system, aside from the
rush to get onto the Top 500 list with only 1 gigabit ethernet in
place, is such a huge amount of memory with no ECC.

Even *with* a high-performance interconnect, you can run into problems
because of traffic congestion, so just having links with high
bandwidth and low latency might not be good enough, and I have serious
concerns about the topology of RedStorm. I feel certain, though, that
whatever problems they have in that area, we won't hear about them.

The point of the SC2003 paper I cited is that the choices that are
offered by conventional CPU and cache systems don't really allow full
exploitation of the premise, "arithmetic is free, bandwidth is
expensive." At least in theory, streaming architectures do.

Blue Gene/L makes alot of sense for a design from off-the-shelf IP,
and eventually the cost of energy is going to dominate the cost of
computation.

From all respects though: cost, bandwidth, latency, energy
consumption, in fact everything but non-recurring cost of design, you
can almost certainly do much better with a streaming architecture than
you can with a conventional architecture. Those non-recurring design
costs, though, mount up real fast, and they have so far killed off
every attempt to build and sell machines with a streaming
architecture.

I haven't done an exceptionally job of covering the territory
concisely, but I have tried. Scalability is not a boojum. We need
computers a million times or more more powerful than we have now, and
there is no way to get there just by plugging together existing boxes,
chips, and boards.

RM

[Felger Carbon]

"Robert Myers" <rmyers@rustuck.com> wrote in message
news:iqqirv8o85ngsfrtogk67iqt04bo1geuc3@4ax.com... If you need to move alot of information around, you need to move alot of information around. If you're moving it around only one-tenth as fast, and that's the rate-limiting step, then you have to use your capital equipment ten times as long to do the same amount of work. Your $2,000 cheapy G5 node with a 1 gigabit connection just turned into a $20,000 box as opposed to one with a 10 gigabit connection. You can buy pretty fancy interconnect for the $18,000 difference. There are similar factors of 10 (or more) to be found in latency-limited calculations. If you're using capital equipment ten times as long, you're also using up physical plant resources (real estate, etc) ten times as long. True TCO calculations can get really complicated.

Robert, we are having a violent agreement. Once again, I agree with the above
100%, with no reservations. Once again, the above is unrelated to my point,
that scalability is a Boojum (that is, it is irrelevant). What _is_ relevant
is optimum performance for a fixed dollar amount ***for a given problem***.

If we have, say, $10M to spend on arithmetic units and interconnect for a
computer, we cannot increase spending on the one without decreasing spending
on the other. What we want is the best performance for that $10M ***on our
problem***. If our problem is solving systems of linear equations (e.g.
Linpack), then we would spend rather less on interconnect than on arithmetic
units. For less parallelizable problems, we would spend a higher proportion
on interconnect, and less on arithmetic units. What we are optimizing for is
**performance** under the constraint of fixed costs. We ***don't care*** what
the scalability is!

The _best_ scalability will be found where almost all the $10M is spent on
interconnect, and the arithmetic unit is one 17-cent smoke-alarm CPU! Perfect
scalability! Obviously, this is not the best use of our $10M.
The point of the SC2003 paper I cited is that the choices that are offered by conventional CPU and cache systems don't really allow full exploitation of the premise, "arithmetic is free, bandwidth is expensive." At least in theory, streaming architectures do.

I know absolutely nothing about streaming architectures, except for your
assertion that such do not exist at this time. My posting was about how to
optimize clusters.

Honestly, we are in almost total agreement re clusters. I merely point out
that what we want to optimize is performance, not a meaningless parameter
which has a maximum wherever the arithmetic unit expenditure is almost zero.

[Tony Hill]

On Mon, 17 Nov 2003 20:26:13 -0500, Robert Myers <rmyers@rustuck.com>
wrote:
<snip>The key here is *balance*. The cost of memory is not adesign-variable (except in a cache-only system). When the VirginiaTech system does have Infiniband, it will spend about as much on

Total cost for Infiniband on the VT system was somewhere around $1.7
or $1.8M. This was factored into the $5.2M in up front costs for the
system.
interconnect as it is spending on compute nodes. That's a prettysensible balance of resources (remember, this is a usenet post, not awhitepaper). The beef I have with the Va. Tech system, aside from therush to get onto the Top 500 list with only 1 gigabit ethernet inplace, is such a huge amount of memory with no ECC.

The score in the Top500 list was done with Infiniband. Gigabit
ethernet is only used for the "extra" I/O stuff. In fact, even their
first attempt that resulted in ~9.5TFlops was with Infiniband, and
they were quoted as saying that the boost in performance was due to a
combination of tweaking the code itself and improving the infiniband
drivers.

The lack of ECC is a major downfall though. Quick, back of the napkin
calculations suggest to me that they will probably be getting at least
about 1 soft memory error every hour or two.

-------------
Tony Hill
hilla <underscore> 20 <at> yahoo <dot> ca

[Tony Hill]

On Mon, 17 Nov 2003 21:37:38 GMT, "Felger Carbon" <fmsfnf@jfoops.net>
wrote:Full disclosure: I've always been keenly interested in latency as alimiting factor in the performance of a cluster of standard desktopCPUs to make a supercomputer. And I've been reading a lot aboutsupercomputers lately. On comp.arch, there'r lotsa interesting URLsposted by folks like Robert Myers and David Wang.I think I just got one of them "epiphany" things. ;-)1. I just read on comp.arch that arithmetic is free these days;the expense is in the communications and I/O. Followed by atechnical analysis which attempted to prove the point.2. I just scanned a Red Storm .pdf file whose URL was provided byRobert:<http://www.lanl.gov/orgs/ccn/salishan2003/pdf/camp.pdf>As he pointed out, slides 25 et seq are indeed interesting. They'reabout "scalability", which is the what big clusters typically don'thave, mostly due to poor communications.3. The Earth Simulator, the world's current leading supercomputer,costs at least $400M, and I've seen the figure $600M. Big Mac wasput together by slave labor at Va Tech and cost $5.2M, roughly 1% ofthe ES price tag. Yet Big Mac has 25% of the performance of the ESas measured by Linpack. That means it's 25 times as cost-efficient(on Linpack type workloads).

Important point of note: the costs above are not even remotely
equivalent. The $400M-$600M figure for the Earth Simulator is, to the
best of my knowledge, the total cost of the project over 'x' number of
years. The $5.2M for the VA Tech cluster was JUST the up-front cost
of the hardware. It didn't even include the extra $2M in
infrastructure improvements, let alone the "cost" associated with
using a rather large building that just happened to be sitting empty
on their campus.
4. Question: is scalability so important that an infinite amountof money should be thrown at it? Obviously not, "infinite" is alotta gelt. I propose something heretical: let's ignorescalability. It's a Boojum, a phantom with no value. What _does_have value? Two things: money has value, as does a solution to acomputing problem inside of a time limit.5. I suggest that the "best" supercomputer for a given job is theone that will provide a solution to a given problem in the desiredtime ***for the least money***. If the scalability is lousy, sowhat? If arithmetic is free, then scalability literally doesn'tmatter! Honest!

I see where you're going with this, but I think that the fundamental
problem with the argument is that arithmetic is never *free*, even if
it cheap. More CPUs = more infrastructure, more support (more
components to fail if nothing else), and higher power costs.

By most estimates I've heard, the total cost of the project is usually
about 10 times what the up front hardware costs are. In those up
front hardware costs, the CPU is only a fairly small portion of the
cost. Hence we get the argument that the CPUs are "free". If your
project costs $100M (roughly the cost of Red Storm if I remember
correctly), then spending the difference between spending $1M and $2M
on the CPU doesn't really factor in much if all other costs are equal.
If power consumption is more important than money, one winds up with

Well, power consumptions IS money. The Big Mac cluster uses roughly
1.5MW of total power (processing + cooling). Earth simulator uses
~3.5MW. Red Storm is supposed to come in at about ~2MW.

If we figure a price of $0.10/kwh, you're looking at $1.75M a year to
power Red Storm. Still only a small piece of the $100M puzzle, but it
exists. If we look at the case above where doubling the processors
only adds $1M to the price tag, we also now have to figure in an extra
$1.75M a year in extra power costs (assuming a simple doubling of
processors = doubling of power consumption, a reasonably fair
assumption). For a 5 year project, you're now looking at that $1M
difference becoming just shy of a $10M difference. Now we're starting
to talk about real money.
something like IBM's Blue Gene, which has also been in the newslately. But that's a topic for another day. ;-)

IBM's Blue Gene may very well be pointing the way to the future of
supercomputing (or not.. as you said, that's a topic for another day)

-------------
Tony Hill
hilla <underscore> 20 <at> yahoo <dot> ca

[Robert Myers]

On Tue, 18 Nov 2003 14:43:54 GMT, "Felger Carbon" <fmsfnf@jfoops.net>
wrote:
"Robert Myers" <rmyers@rustuck.com> wrote in messagenews:iqqirv8o85ngsfrtogk67iqt04bo1geuc3@4ax.com...
<snip>
...We ***don't care*** what the scalability is!
But we do care. Soon we will be able to put together many times more
processors than we can now, and we need to be learning about how to go
about it when we do. That's scalability.

The whole supercluster craze, and especially the race to get on the
Top 500 list, is troubling to me. We're spending too much on putting
together big piles of hardware and running big simulations without
spending enough time and money understanding what we're doing.

The Va. Tech cluster will be out of date in just a few years. If
people start noticing the frequency with which we're filling up entire
buldings on campus only to haul it all away to the dump and with very
little to show for it but some nice color plots, the net effect on
science will be negative.
The _best_ scalability will be found where almost all the $10M is spent oninterconnect, and the arithmetic unit is one 17-cent smoke-alarm CPU! Perfectscalability! Obviously, this is not the best use of our $10M.

We almost have the analagous situation already in terms of spending
priorities. All the money is going into high Rmax, with very little
going to understanding what's going on. You just don't learn much by
cabling together a large number of boxes. You do create alot of photo
opportunities, but, after a while, once you've seen one building full
of racks, you've seen them all.
The point of the SC2003 paper I cited is that the choices that are offered by conventional CPU and cache systems don't really allow full exploitation of the premise, "arithmetic is free, bandwidth is expensive." At least in theory, streaming architectures do.I know absolutely nothing about streaming architectures, except for yourassertion that such do not exist at this time. My posting was about how tooptimize clusters.

And my point was about how to optimize clusters. If you're putting
together huge numbers of boxes containing the wrong microarchitecture,
you're not spending your money optimally.

I'm much happier to see money spent on the kind of work in the paper I
cited than on big piles of hardware.

It's just too easy to get a big machine, describe a big calculation,
get a big grant, and churn out alot of Ph.D.'s. It may be good for
people's careers, but it's not good science policy.

RM

[Tony Hill]

On Tue, 18 Nov 2003 14:47:36 -0500, Robert Myers <rmyers@rustuck.com>
wrote:
<snip>The Va. Tech cluster will be out of date in just a few years. Ifpeople start noticing the frequency with which we're filling up entirebuldings on campus only to haul it all away to the dump and with verylittle to show for it but some nice color plots, the net effect onscience will be negative.

Hmm, perhaps there is some method to VA Tech's madness though. Think
of it, those are PowerMac G5 desktop systems with good graphics cards
and all the trimmings of a high-end desktop. 3 years from now,
they'll still make a pretty decent PC that VA Tech could probably sell
off to students for $1000 a pop. That's something that you sure as
hell won't be able to do with Earth Simulator!

Whaddya mean the glass isn't always half full? :>

Ok, maybe simply selling off the computers won't be all that
beneficial to science, but hey, at least they aren't filling up some
landfill somewhere!

-------------
Tony Hill
hilla <underscore> 20 <at> yahoo <dot> ca

[Robert Myers]

On Tue, 18 Nov 2003 20:12:11 GMT, Tony Hill <hilla_nospam_20@yahoo.ca>
wrote:

<snip>Hmm, perhaps there is some method to VA Tech's madness though. Thinkof it, those are PowerMac G5 desktop systems with good graphics cardsand all the trimmings of a high-end desktop. 3 years from now,they'll still make a pretty decent PC that VA Tech could probably selloff to students for $1000 a pop. That's something that you sure ashell won't be able to do with Earth Simulator!Whaddya mean the glass isn't always half full? :>Ok, maybe simply selling off the computers won't be all thatbeneficial to science, but hey, at least they aren't filling up somelandfill somewhere!
Sounds like you wouldn't mind having one right now!

Whether you intended to be taken seriously or not, you actually make a
good point. From a TCO point of view, those boxes will make sense as
personal workstations much longer than they will make sense as part of
a supercluster.

RM

[Felger Carbon]

"Robert Myers" <rmyers@rustuck.com> wrote in message
news:s5rkrvk6fe1i9l4cbn7o65crsmnqkfiboc@4ax.com... On Tue, 18 Nov 2003 14:43:54 GMT, "Felger Carbon" <fmsfnf@jfoops.net> wrote:"Robert Myers" <rmyers@rustuck.com> wrote in messagenews:iqqirv8o85ngsfrtogk67iqt04bo1geuc3@4ax.com... <snip>...We ***don't care*** what the scalability is! But we do care. Soon we will be able to put together many times more processors than we can now, and we need to be learning about how to go about it when we do. That's scalability.

Once again, I agree, except for the final 2-word phrase. There may have been
a time when folks would cobble together a 32-CPU prototype and expect its
performance to scale to a 4096-CPU system using the same interconnect, but
that was a _long_ time ago. Nowadays everybody knows better.

If you add _lots_ more CPUs, you need to rethink the interconnect from
scratch, because you have a whole new computer. Otherwise you are faced with
designing interconnect for a 500,000-CPU cluster just so you can build today's
10,000-CPU cluster. Now, _that's_ scalability. And it makes no sense at all.
-----------------

On a slightly different point, at the current rate of bang/buck improvement
the normal life of a supercomputer is in fact about 3 years. As Tony pointed
out, at least the Va Tech supercomputer won't become landfill 3 years from
now.

There are _some_ scientific/technical research problems that _require_ a
supercomputer. In 1985, the DoD wasn't handing out many research grants to
universities whose profs were still using slide rules. In that sense, a
supercomputer can be very cost-effective.

And like a winning football program, a supercomputer may attract private
funding for a university that might not otherwise be available. ;-)

[Robert Myers]

On Wed, 19 Nov 2003 00:59:58 GMT, "Felger Carbon" <fmsfnf@jfoops.net>
wrote:
"Robert Myers" <rmyers@rustuck.com> wrote in messagenews:s5rkrvk6fe1i9l4cbn7o65crsmnqkfiboc@4ax.com... On Tue, 18 Nov 2003 14:43:54 GMT, "Felger Carbon" <fmsfnf@jfoops.net> wrote:
<snip>...We ***don't care*** what the scalability is! But we do care. Soon we will be able to put together many times more processors than we can now, and we need to be learning about how to go about it when we do. That's scalability.Once again, I agree, except for the final 2-word phrase. There may have beena time when folks would cobble together a 32-CPU prototype and expect itsperformance to scale to a 4096-CPU system using the same interconnect, butthat was a _long_ time ago. Nowadays everybody knows better.
I'm older and in some ways a little wiser, but I'm not sure this is
one of them.
If you add _lots_ more CPUs, you need to rethink the interconnect fromscratch, because you have a whole new computer. Otherwise you are faced withdesigning interconnect for a 500,000-CPU cluster just so you can build today's10,000-CPU cluster. Now, _that's_ scalability. And it makes no sense at all.-----------------
When you say it makes no sense _at_all_, you take your point too far.
It would make no sense not to build 10,000 CPU clusters until you were
sure you knew how to build 500,000 CPU clusters, but it makes a great
deal of sense to think about what might change and to plan
intermediate steps that would be helpful in getting from here to
there. That is so fundamental to engineering that I am astonished to
find you arguing so vigorously against it.

Integrated circuits have gone from a few transistors to hundreds of
millions. The original concept is still quite recognizable, and with
huge amounts of money so obviously at stake, people started thinking
long ago about how the technology would have to change with dramatic
changes in scale.
On a slightly different point, at the current rate of bang/buck improvementthe normal life of a supercomputer is in fact about 3 years. As Tony pointedout, at least the Va Tech supercomputer won't become landfill 3 years fromnow.
Recyclability is a benefit I had not thought of, and it does argue for
computers made out of pieces that can be taken apart and used in some
other way when they have passed their useful life as pieces of a
supercomputer.
There are _some_ scientific/technical research problems that _require_ asupercomputer. In 1985, the DoD wasn't handing out many research grants touniversities whose profs were still using slide rules. In that sense, asupercomputer can be very cost-effective.
I think it probably has occurred to you that my strong feelings on
this subject have some basis in experience.

You can throw megaflops at almost anything, but megaflops will never
be a subsitute for insight and understanding.
And like a winning football program, a supercomputer may attract privatefunding for a university that might not otherwise be available. ;-)
I fear that far too much weight has been given to that last point.

RM

[Felger Carbon]

"Robert Myers" <rmyers@rustuck.com> wrote in message
news:eltlrvkjs5k5ckgqbsrtgc90p8mq4e82mr@4ax.com... On Wed, 19 Nov 2003 00:59:58 GMT, "Felger Carbon" <fmsfnf@jfoops.net> wrote:Otherwise you are faced with designing interconnectdesigning interconnect for a 500,000-CPU cluster just so you can build today's 10,000-CPU cluster. Now, _that's_ scalability. And it makes no sense at all. When you say it makes no sense _at_all_, you take your point too far. It would make no sense not to build 10,000 CPU clusters until you were sure you knew how to build 500,000 CPU clusters, but it makes a great deal of sense to think about what might change and to plan intermediate steps that would be helpful in getting from here to there. That is so fundamental to engineering that I am astonished to find you arguing so vigorously against it.

The designers of the Honda Civic gave no consideration whatever to expansion
to an 18-wheeler. Neither should the designers of an N-CPU cluster design the
interconnect to handle a 10N or 100N cluster.
There are _some_ scientific/technical research problems that _require_ asupercomputer. In 1985, the DoD wasn't handing out many research grantsto universities whose profs were still using slide rules. In that sense, asupercomputer can be very cost-effective. You can throw megaflops at almost anything, but megaflops will never be a subsitute for insight and understanding.

I agree. But can't we have _both_ insight and a supercomputer? Do we have to
choose one or the other?
And like a winning football program, a supercomputer may attract privatefunding for a university that might not otherwise be available. ;-) I fear that far too much weight has been given to that last point.

Yep. The real world is an untidy place.

[Robert Myers]

On Wed, 19 Nov 2003 10:29:05 GMT, "Felger Carbon" <fmsfnf@jfoops.net>
wrote:
"Robert Myers" <rmyers@rustuck.com> wrote in messagenews:eltlrvkjs5k5ckgqbsrtgc90p8mq4e82mr@4ax.com... On Wed, 19 Nov 2003 00:59:58 GMT, "Felger Carbon" <fmsfnf@jfoops.net> wrote:
<snip>The designers of the Honda Civic gave no consideration whatever to expansionto an 18-wheeler. Neither should the designers of an N-CPU cluster design theinterconnect to handle a 10N or 100N cluster.
The analogy doesn't seem especially apt. There is no prospect that a
Honda Civic will transmogrify itself over time into an 18-wheeler.
There is every prospect that clusters will grow.

Fortunately, the people who are paid to think about these things in
Washington do seem to be appropriately concerned about scalability.
(I can't believe it; I'm siding with bureuacrats).
There are _some_ scientific/technical research problems that _require_ asupercomputer. In 1985, the DoD wasn't handing out many research grantsto universities whose profs were still using slide rules. In that sense, asupercomputer can be very cost-effective. You can throw megaflops at almost anything, but megaflops will never be a subsitute for insight and understanding.I agree. But can't we have _both_ insight and a supercomputer? Do we have tochoose one or the other?
No, we don't. The approach of concentrating resources at a few
carefully-chosen centers of activity for large-scale supercomputing
has always seemed right, and modest-sized clusters have been a
tremendous boon to research. It's the "we're number x" relative to a
meaningless number that bothers me.

Virginia Tech wants to make itself a player, and they have chosen a
way of going about it that I don't find particularly appealing.

In actual fact, since large-scale computation is what I'm interested
in, I should be pleased at what's going on. Virginia Tech just hired
the former head of my graduate department at UIUC, which always has
been a big player in large scale computation, as the Dean of the
College of Engineering. I should be wishing the effort godspeed, not
carping about it. I'm just being a crank.

RM

[Tony Hill]

On Tue, 18 Nov 2003 16:07:12 -0500, Robert Myers <rmyers@rustuck.com>
wrote:On Tue, 18 Nov 2003 20:12:11 GMT, Tony Hill <hilla_nospam_20@yahoo.ca>wrote:Hmm, perhaps there is some method to VA Tech's madness though. Thinkof it, those are PowerMac G5 desktop systems with good graphics cardsand all the trimmings of a high-end desktop. 3 years from now,they'll still make a pretty decent PC that VA Tech could probably selloff to students for $1000 a pop. That's something that you sure ashell won't be able to do with Earth Simulator!Whaddya mean the glass isn't always half full? :>Ok, maybe simply selling off the computers won't be all thatbeneficial to science, but hey, at least they aren't filling up somelandfill somewhere!Sounds like you wouldn't mind having one right now!

Hmm, a dual-processor 64-bit workstation with 4GB of memory, PCI-X, a
Radeon 9600 Pro video card and a DVD burner? Nahh, why would I want
one of those?!

I've never owned a Mac myself and probably won't get one in the near
future at least (they're pretty darn expensive and I'm pretty darn
broke :> ), but I do like some of the things that they're doing with
the PowerMac G5s.
Whether you intended to be taken seriously or not, you actually make agood point. From a TCO point of view, those boxes will make sense aspersonal workstations much longer than they will make sense as part ofa supercluster.

For sure! They never made much sense as a supercluster, but they
definitely have some good points as individual computers.

-------------
Tony Hill
hilla <underscore> 20 <at> yahoo <dot> ca

[Felger Carbon]

"Robert Myers" <rmyers@rustuck.com> wrote in message
news:ql3nrvkesmbubnn67k3gmmdvf5nmp92mju@4ax.com... On Wed, 19 Nov 2003 10:29:05 GMT, "Felger Carbon" <fmsfnf@jfoops.net> wrote:The designers of the Honda Civic gave no consideration whatever to expansion to an 18-wheeler. Neither should the designers of an N-CPU cluster design the interconnect to handle a 10N or 100N cluster. The analogy doesn't seem especially apt. There is no prospect that a Honda Civic will transmogrify itself over time into an 18-wheeler. There is every prospect that clusters will grow.

After searching hard for an element in which we disagree, we seem to have
settled on the above. Since neither you nor I have any say on how actual
superclusters are designed, our opinions would seem moot. I'd suggest a
duel with cream puffs at 30 feet, but I'd probably lose. ;-)
Virginia Tech wants to make itself a player, and they have chosen a way of going about it that I don't find particularly appealing. Virginia Tech just hired the former head of my graduate department at UIUC, which always has been a big player in large scale computation, as the Dean of the College of Engineering. I should be wishing the effort godspeed, not carping about it. I'm just being a crank.

Gracefully said, Robert.

[Rob Stow]

Robert Myers wrote:
On Wed, 19 Nov 2003 10:29:05 GMT, "Felger Carbon" <fmsfnf@jfoops.net> wrote:"Robert Myers" <rmyers@rustuck.com> wrote in messagenews:eltlrvkjs5k5ckgqbsrtgc90p8mq4e82mr@4ax.com...On Wed, 19 Nov 2003 00:59:58 GMT, "Felger Carbon" <fmsfnf@jfoops.net>wrote: <snip>The designers of the Honda Civic gave no consideration whatever to expansionto an 18-wheeler. Neither should the designers of an N-CPU cluster design theinterconnect to handle a 10N or 100N cluster.

Cray certainly seemed to have thought otherwise when they
designed Red Storm. Red Storm is supposed to be scalable from
240 to 30000 cpus. The 10xxx processor Red Storm being built
at Sandia is certainly the largest Red Storm so far, but I recall
reading that the typical Red Storm is expected to be in the
1000 processor neighbourhood.

Designing a system that scales well is possibly a necessity for
a company like Cray. It is probably a lot cheaper to design the
highly scalable Red Storm *once* and sell it to both the big
customers and the little ones than it is to design one system
for the guys who need 240 processors, another once for the guys
who need 1000, another for the guys who need 10000, and so on.




--
Reply to newsgroup only please. This e-mail account is real
but effectively abandoned because of excessive spamming.

[Yousuf Khan]

"Tony Hill" <hilla_nospam_20@yahoo.ca> wrote in message
news:b18592a3b25e1e6b8eeb0c36aa3ed9e4@news.1usenet.com... Hmm, perhaps there is some method to VA Tech's madness though. Think of it, those are PowerMac G5 desktop systems with good graphics cards and all the trimmings of a high-end desktop. 3 years from now, they'll still make a pretty decent PC that VA Tech could probably sell off to students for $1000 a pop. That's something that you sure as hell won't be able to do with Earth Simulator!

I'm surprised they kept them in their original cases rather than try to
remove their motherboards and put them into special racks or something.

Yousuf Khan

[Robert Myers]

On Wed, 19 Nov 2003 17:25:09 -0600, Rob Stow
<rob.stow@sk.sympatico.ca> wrote:
On Wed, 19 Nov 2003 10:29:05 GMT, "Felger Carbon" <fmsfnf@jfoops.net> wrote:"Robert Myers" <rmyers@rustuck.com> wrote in messagenews:eltlrvkjs5k5ckgqbsrtgc90p8mq4e82mr@4ax.com...>On Wed, 19 Nov 2003 00:59:58 GMT, "Felger Carbon" <fmsfnf@jfoops.net>>wrote: <snip>The designers of the Honda Civic gave no consideration whatever to expansionto an 18-wheeler. Neither should the designers of an N-CPU cluster design theinterconnect to handle a 10N or 100N cluster.Cray certainly seemed to have thought otherwise when theydesigned Red Storm. Red Storm is supposed to be scalable from240 to 30000 cpus. The 10xxx processor Red Storm being builtat Sandia is certainly the largest Red Storm so far, but I recallreading that the typical Red Storm is expected to be in the1000 processor neighbourhood.Designing a system that scales well is possibly a necessity fora company like Cray. It is probably a lot cheaper to design thehighly scalable Red Storm *once* and sell it to both the bigcustomers and the little ones than it is to design one systemfor the guys who need 240 processors, another once for the guyswho need 1000, another for the guys who need 10000, and so on.

Cray has the good fortune to have the backing of the US Treasury.
Were it not for that backing, they wouldn't be in business. Even if
they were in business, they'd be forced to think like everybody else
thinks, which is to build the computer they can sell today at the
lowest cost and highest gross margin.

As it is, the people who decided that the country could not live
without Cray place a high priority on scalability. They have the
charter of seeing far into the future regarding national security
issues, and they have (even if belatedly) come to the realization
that, left to their own devices, IBM et. al. aren't all that
interested in designing hardware that is useful only for HPC. IBM has
already announced that they're trying to sell Blue Gene for non-HPC
applications.

In short, the concern about scalability is coming from Washington, not
from Cray.

RM

[Keith R. Williams]

In article <9t9vb.3397$H8X1.40
@twister01.bloor.is.net.cable.rogers.com>,
removethisspam.bjsk90.removethispam@hotmail.com says... "Tony Hill" <hilla_nospam_20@yahoo.ca> wrote in message news:b18592a3b25e1e6b8eeb0c36aa3ed9e4@news.1usenet.com... Hmm, perhaps there is some method to VA Tech's madness though. Think of it, those are PowerMac G5 desktop systems with good graphics cards and all the trimmings of a high-end desktop. 3 years from now, they'll still make a pretty decent PC that VA Tech could probably sell off to students for $1000 a pop. That's something that you sure as hell won't be able to do with Earth Simulator! I'm surprised they kept them in their original cases rather than try to remove their motherboards and put them into special racks or something.

I'm not. There was a lot of work put into cooling these things
that would have to be completely redone. As it is the room had
to be designed to cool the boxes, which is far easier than
cooling each component in the system, times 1100.

--
Keith

[Keith R. Williams]

In article <1vmqrv4v949vse48dt2tpv3lcd3poi4acu@4ax.com>,
rmyers@rustuck.com says... On Wed, 19 Nov 2003 17:25:09 -0600, Rob Stow <rob.stow@sk.sympatico.ca> wrote: On Wed, 19 Nov 2003 10:29:05 GMT, "Felger Carbon" <fmsfnf@jfoops.net> wrote:>"Robert Myers" <rmyers@rustuck.com> wrote in message>news:eltlrvkjs5k5ckgqbsrtgc90p8mq4e82mr@4ax.com...>>>On Wed, 19 Nov 2003 00:59:58 GMT, "Felger Carbon" <fmsfnf@jfoops.net>>>wrote: <snip>>The designers of the Honda Civic gave no consideration whatever to expansion>to an 18-wheeler. Neither should the designers of an N-CPU cluster design the>interconnect to handle a 10N or 100N cluster.Cray certainly seemed to have thought otherwise when theydesigned Red Storm. Red Storm is supposed to be scalable from240 to 30000 cpus. The 10xxx processor Red Storm being builtat Sandia is certainly the largest Red Storm so far, but I recallreading that the typical Red Storm is expected to be in the1000 processor neighbourhood.Designing a system that scales well is possibly a necessity fora company like Cray. It is probably a lot cheaper to design thehighly scalable Red Storm *once* and sell it to both the bigcustomers and the little ones than it is to design one systemfor the guys who need 240 processors, another once for the guyswho need 1000, another for the guys who need 10000, and so on. Cray has the good fortune to have the backing of the US Treasury. Were it not for that backing, they wouldn't be in business. Even if they were in business, they'd be forced to think like everybody else thinks, which is to build the computer they can sell today at the lowest cost and highest gross margin.

Good for them. However, they're now "forced" to build a computer
that has no redeeming value other than Uncle Sugar wants to
subsidize Frostbite Falls, at the "highest gross margin". If the
widget were all that useful, I'd expect there wouldn't be much
press on it. ...security and all.
As it is, the people who decided that the country could not live without Cray place a high priority on scalability. They have the charter of seeing far into the future regarding national security issues, and they have (even if belatedly) come to the realization that, left to their own devices, IBM et. al. aren't all that interested in designing hardware that is useful only for HPC. IBM has already announced that they're trying to sell Blue Gene for non-HPC applications.

Gee, isn't that what I've been arguing with you for *months*
about? There isn't any money in dedicated *processors*. ...not
that this is a dedicated *processor*, merely another collection
of processors bailed up together differently. In short, the concern about scalability is coming from Washington, not from Cray.

Pork sizzles when fried.

--
Keith

[Robert Myers]

On Thu, 20 Nov 2003 23:23:04 -0500, Keith R. Williams
<krw@attglobal.net> wrote:
In article <1vmqrv4v949vse48dt2tpv3lcd3poi4acu@4ax.com>,rmyers@rustuck.com says...

<snip>
Cray has the good fortune to have the backing of the US Treasury. Were it not for that backing, they wouldn't be in business. Even if they were in business, they'd be forced to think like everybody else thinks, which is to build the computer they can sell today at the lowest cost and highest gross margin.Good for them. However, they're now "forced" to build a computerthat has no redeeming value other than Uncle Sugar wants tosubsidize Frostbite Falls, at the "highest gross margin". If thewidget were all that useful, I'd expect there wouldn't be muchpress on it. ...security and all.
I'm sure the United States would be delighted to have some adversary
run off and squander the resources that would be necessary to
duplicate the router chip that is at the heart of Red Storm. In one
of the few statements Edward Teller ever made that I agree with is
that you don't have to bother with classifying technology that no
adversary has the capability to duplicate.

If Iran or North Korea decided that it just *had* to have such a
computer and started approaching people who could even attempt it,
we'd find out soon enough to stop even the attempt.

As to publicity, the US has a long history of flexing its muscles to
show the world what it is technologically capable of so as to give
adversaries some idea of the kinds of resources we are willing to
bring to bear in the interests of national security.

Some technologies are supposed to remain completely black, unless Tom
Clancy happens to hear about them over a beer, of course.
As it is, the people who decided that the country could not live without Cray place a high priority on scalability. They have the charter of seeing far into the future regarding national security issues, and they have (even if belatedly) come to the realization that, left to their own devices, IBM et. al. aren't all that interested in designing hardware that is useful only for HPC. IBM has already announced that they're trying to sell Blue Gene for non-HPC applications.Gee, isn't that what I've been arguing with you for *months*about? There isn't any money in dedicated *processors*. ...notthat this is a dedicated *processor*, merely another collectionof processors bailed up together differently.

The Cray SV2 certainly isn't, and the router chip that makes Red Storm
work certainly isn't.

I hate to pop your balloon, but you've been shadow-boxing with
yourself on this issue. I've made the point on line more than once
that computer manufacturers aren't interested in building computers
just for HPC.
In short, the concern about scalability is coming from Washington, not from Cray.Pork sizzles when fried.

If you had ever been close to this kind of decision-making, you would
know that, while political influence never hurts, the lifers who are
at the core of the decision-making process are pretty well insulated
from it. When a congressional manate comes down that doesn't agree
with what the professional policy-makers have to say, it generally
gets noticed in a place like Aviation Week or some other public forum.

RM

[Felger Carbon]

"Robert Myers" <rmyers@rustuck.com> wrote in message
news:nu9rrvk5h873et4e27niiglgep9u39iu4h@4ax.com... I'm sure the United States would be delighted to have some adversary run off and squander the resources that would be necessary to duplicate the router chip that is at the heart of Red Storm. In one of the few statements Edward Teller ever made that I agree with is that you don't have to bother with classifying technology that no adversary has the capability to duplicate.

Lessee here. The AMD CPUs are widely available all over the world. I wonder
where that cute Black Widow "router chip" is fabbed? China? ;-)

[Robert Myers]

On Fri, 21 Nov 2003 08:57:59 GMT, "Felger Carbon" <fmsfnf@jfoops.net>
wrote:
"Robert Myers" <rmyers@rustuck.com> wrote in messagenews:nu9rrvk5h873et4e27niiglgep9u39iu4h@4ax.com... I'm sure the United States would be delighted to have some adversary run off and squander the resources that would be necessary to duplicate the router chip that is at the heart of Red Storm. In one of the few statements Edward Teller ever made that I agree with is that you don't have to bother with classifying technology that no adversary has the capability to duplicate.Lessee here. The AMD CPUs are widely available all over the world. I wonderwhere that cute Black Widow "router chip" is fabbed? China? ;-)

If the US has decided to let Intel build a state of the art fab in
China, what secrets of this kind do we have that we can keep from
them?

Apparently, they also have blueprints for our state of the art nuclear
warheads.

We can only hope that we know as much about them as they do about us.

RM