Bryan,
Thanks for the eulogy link. Seems like your main concern is the lack of any pricing in SpecSFS benchmarks (past, present, and future) and certainly your other point about SpecSFS workload parameters are worthy of discussion as well. It’s hard to argue your lack of pricing as a prime objection to SpecSFS.

However, benchmarks without system pricing can supply a useful upper bound for the performance of a system. Also, benchmarks (even SpecSFS) typically require a detailing of the hardware used for the benchmark and as such, can be used to understand hardware oriented metrics, e.g., for SpecSFS NFS throughput operations per hard drive/spindle. Although seldom done, such a metric can be used to compare system performance without pricing at least from a ops/spindle perspective.

In addition, while I would agree that a working set size of 10% will cripple most caching algorithms, the question from a benchmark perspective is do you want the cache active or inactive during the benchmark. As you indicate having a variable or set-able working set size would at least handle the complete range from 100% cache miss to 100% cache hit, lacking that, I would prefer one that disables the cache making it inactive (100% miss) rather than accessing only data from cache (100% hit) masking I/O activity.

Nonetheless, what you say is important. I hope that SPEC listens and modifies the benchmark to be better next time.

The funny thing about SPECsfs 2008 today is that most of the submissions are mid-range systems. Only lately have a couple of vendors come in with higher end systems. I attribute that to the newness of the benchmark and the unwillingness of any vendor to look bad in comparison to the older SPECsfs 97 bencmark.

Ray