Re: Benchmarks
On Fri, 4 Aug 1995, Mike Losh wrote:
> Excuse any ignorance on my part about the SPEC benchmarks, but is a port of the
> SPECint integer benchmark possible for F21? Or is a C compiler MANDATORY for
> the benchmark to be considered a true "SPEC" benchmark? It seems to me that one
> could port the source into hand-optimized F21 machine code which performs the
> same or equivalent computations. If SPECint requires wide (64 bit?)
> computations, F21 may not be extremely efficient, having to string 20 bit ops
> together, but so what? At least then we could say "F21 provides a SPECint of XX
> out of SRAM, YY out of DRAM, at cost $$." The Great Masses could start
> thinking, "Hmm, I can get 1/3 of the SPECInt of an Alpha workstation for 1/10th
> the cost..." (or whatever the numbers are going to be). Hopefully, SPECint
> running on a cluster of 8 (or whatever) of F21s will be really impressive for
> the price.
>
> SPEC benchmarks may not be interesting to most MISC readers (who envision 20 bit
> apps), but I think most workstation buyers/users expect to see them. If F21 and
> later chips are going to be a commercial success in the scientific and
> engineering market, we must convince many of these people to take a look. If we
> can get a foot in the door with SPECint, it may be easier to convice them of the
> advantages of MISC for their applications.
I do not see why would people need to look at SPECint and SPECfp to
measure the performance of a computer. As John Gustafson and Quinn Snell
put it:
----
The SPEC benchmark is popular among workstation vendors. It is not an
independent measure; a consortium of vendors determine what is in SPEC
and how to report it. SPEC does not scale , and runs on a narrow range of
computers at any given time. It has had to be revised once, as the first
version proved too small for workstations after a few years of
technological progress. SPEC claims to be the geometric ratio of the
time reduction of various kernels and applications to the time required
by a VAX-11/780. Unfortunately, the VAX-11/780 currently gets a SPECmark
of about 3, indicating it is three times as fast as itself! SPEC survives
largely because of the lack of credible alternatives.
----
They go ahead and develop a benchmark -- HINT -- that is scalable from humans
with pen and pencil, through micros, through PCs, through workstations,
through SMPs, to vector machines, MMPs and beyond.
I do think, that HINT is a much more relevant benchmark than MIPS, FLOPS,
SPECS and WinMarks. And I think, that MISC machines will compare quite
favorably with the rest of the flock of machines the scientific community
is using.
--
Penio Penev <Penev@venezia.Rockefeller.edu> 1-212-327-7423
