Benchmarking - Anandtech, please include this concept

[Grit]

I sometimes feel there is a bit of a "real world" factor that gets missed when benchmarks are reported. I understand the need to eliminate all other sources of interference to get true results. However, I do not use my video card, CPU, or SSD in a lab.

What I want to see included are results that reflect my personal computer use.

I was looking into SSDs and saw where someone reported how much faster his SSD loaded World of Warcraft. When I watched the YouTube video, the SSD loaded in about 18 seconds; the HDD in almost 60. I used a stopwatch to check my own V-Raptor system and found - hmm, about 18 seconds. No way that makes sense.

Well, it does. I only reboot about once every few weeks or so, but I play WoW about once a day. I'm certain Windows 7 has prefetched most of that into the 8GB of ram I've got on board and that's what's allowing WoW to load so fast.

So, unless I'm switching applications frequently or frequently rebooting (which I'm not), it doesn't seem an SSD is going to make nearly the profound impact for me that reviews show it does? I can't always tell because most of the reviews state something like "and we rebooted every time to avoid the impact of caching". What happens when I DON'T reboot my computer after running every application??


It would be nice to see something like this reflected in one page of the reviews. I think (lately) graphics card reviews have started to reflect this by showing benchmarks with all of the graphical features turned on or up (AA, AF, etc). I don't want to know how many FPS I'm going to get in Dragon Age with all the eye candy turned down. I want to know how many FPS I'm going to get when I turn all the eye candy UP.


Point being - I'd like to see at least some benchmarks on how hardware is going to impact performance on an everyday computer too, not just in the lab.

 

[Jd007]

I know how you feel. Real world user experience is more importantly to me than synthetic and controlled benchmarks in the lab as well.

The problem is not everybody uses their computers the same way. You may only reboot once every few weeks, but I reboot daily, and some people may reboot even more frequently. It's impossible for them to do a review for every user habit. What a fresh-boot test indicates is that even in the worst case (no cached data), the SSD is still very fast.

 

[Nothinman]

I sometimes feel there is a bit of a "real world" factor that gets missed when benchmarks are reported.

Benchmarks and reality are pretty much mutually exclusive.

 

[HendrixFan]

I know how you feel. Real world user experience is more importantly to me than synthetic and controlled benchmarks in the lab as well.

The problem is not everybody uses their computers the same way. You may only reboot once every few weeks, but I reboot daily, and some people may reboot even more frequently. It's impossible for them to do a review for every user habit. What a fresh-boot test indicates is that even in the worst case (no cached data), the SSD is still very fast.

Good point, people that load up different games throughout the day will experience different results. I personally go weeks without rebooting, usually only when Windows update wants me to reboot to complete something. During that time I may only launch one or two different games, but I do plenty of editing and fill my RAM each time.

The point of using the clean room benchmarking for SSDs is to show where the differences will lie. As is the case with most hobbyist tinkering, you want to upgrade where your bottlenecks are. If your computer usage doesn't utilize hard disks often, SSDs probably won't be for you.

 

[Grit]

I know its not realistic to make benchmarks that encompass everyone's reality. However, it would be nice if they included both sides of the fence. Say for example, the best case scenario (eg, right after a reboot), and a worst case scenario (where the data is already cached).

 

[tcsenter]

I was looking into SSDs and saw where someone reported how much faster his SSD loaded World of Warcraft. When I watched the YouTube video, the SSD loaded in about 18 seconds; the HDD in almost 60. I used a stopwatch to check my own V-Raptor system and found - hmm, about 18 seconds. No way that makes sense.

You want benchmarks to reflect reality but yet you have an enthusiast class hard drive that maybe 1% of computer users/owners are even aware exists, let alone actually have.

Well, OK, you didn't say mainstream reality, so its all good.

 

[Nothinman]

I know its not realistic to make benchmarks that encompass everyone's reality. However, it would be nice if they included both sides of the fence. Say for example, the best case scenario (eg, right after a reboot), and a worst case scenario (where the data is already cached).

You have those backwards. Right after a fresh reboot is the worst case because everything has to be paged in and the data already being cached is the best case because you don't need any I/O to accomplish the same thing.

 

[Voo]

So that would tell people who already know that their applications are in the RAM, that they won't get much of a speedup - which they would already know because otherwise they wouldn't know if the benchmarks apply or not?

I think everyone who reads AT has a good grasp on simple concepts like caching, memory hierarchies and similar stuff. Why benchmark things where you already know the results in advance?

 

[Nothinman]

I think everyone who reads AT has a good grasp on simple concepts like caching, memory hierarchies and similar stuff. Why benchmark things where you already know the results in advance?

The periodic, very incorrect posts about virtual memory would seem to indicate otherwise. Unless the majority of those knowledgeable users are reading and not posting.