On 11 Nov 2003 00:58:48 -0500, Mannr@uwaterloo.ca wrote:Thanks for the information. I just checked. OK I guess the chips will accessmemory more or less transparently, no matter where it is.
Yup, that about covers it.
But my questions still remain:- What is the performance of accessing memory directly and going through hypertransport? I would like to see both numbers, since I expect direct memory accesses to improve due the onchip controller too. I guess it is possible that hypertransport adds very little to the access time, compared to the memory itself. But I'd like to see some numbers.
AMD had some numbers at various times. I believe their saying was
that the difference between a local and remote memory access was less
than the difference between a page hit and a page miss. My
understanding is that the latency of local memory access is roughly
115 clock cycles at 2GHz, and something on the order of 150 clock
cycles for remote memory. I'm afraid that I don't have the documents
on hand to back up those numbers though, but they're at least in the
right ball-park.
For comparison, with an external memory controller you would tend to
be looking at about 170 or 180 clock cycles latency at 2GHz.
- Will Linux (or any other OS) make sensible decisions, eg., allocating new memory on the CPU it is running on, and, if possible, keeping the thread/process on that processor in the future? If so, this architecture could scale very well.
What you are talking about is NUMA optimizations (having your OS know
about local and remote memory) and processor affinity (having
processes stick to one processor so that they don't thrash your
cache).
The Linux 2.4 kernel contains no NUMA optimizations and very little
(none?) in the way of processor affinity optimizations. The upcoming
2.6 kernel has a significantly improved scheduler that with much
better processor affinity. I believe that WinXP also has some
processor affinity.
NUMA is a bit trickier. Again, the 2.6 Linux kernel does have NUMA
support, but according to one kernel developer, it doesn't do much of
anything for the Opteron.
As for scalability, the Opteron is already showing some very promising
signs of scaling well, and it should continue to improve as the
software is better able optimized. This can most easily be seen when
comparing SPEC CPU2000 "rate" numbers. You can find identical systems
being used in 1, 2 and 4 processor mode for both an AMD Opteron system
(Einux A4800, listed under AMD's name) and an Intel XeonMP system
(Dell PowerEdge 6650, listed under Dell's name). In CINT_rate for
single processor systems, the 2.8GHz XeonMP is 5% faster than the
1.8GHz Opteron. For the dual-processor systems, the difference drops
to less than 1%, and for the 4-processor systems, the Opteron is just
over 2% faster.
Of course, that scalability is nothing as compared to what you see in
Spec CFP2000_rate. Here, the same two systems are again tested in 1,
2 and 4 processor modes. Here, the Opteron starts out 8% faster in a
1P system, then extends it's lead to 38% in a 2P system. The real
kicker though is in the 4 processor system, where the Opteron is no
less than 94% faster than the XeonMP system!
As you suggest, the Opteron does indeed scale very well, at least up
to 4 processor systems (no one sells anything bigger at this point in
time). Now, to be fair, comparing it to a Xeon doesn't say much,
because the Xeon is well known not to scale well at all. However, the
scaling that the Opteron shows is pretty darn close to that of the Sun
UltraSparcIII and the IBM Power4, which puts it in pretty good
company.
-------------
Tony Hill
hilla <underscore> 20 <at> yahoo <dot> ca