Originally posted by: William Gaatjes
I just think there is a tighter integration possible if the OS writers and hardware designers would work a bit more together. I know i ask a lot. It is amazing that the standards today are even standards. Anyway, I do wish that sometimes there would be a less marketing and more pure performance attitude by the designers.
Put this as the intro to your OP and you'd have spawned a whole different attitude in the thread in my opinion, but hindsight is that way.
It's the subtleties of how you opened the OP, it comes across as "I had this stroke of genius while writing in another blog, and it was sooo genius that I just had to come here and share with you all so you could revel in my genius".
Obviously that wasn't how you meant it, am just explaining how it kind of came across in hopes that it goes some distance to help explain perhaps why the majority of responses in the thread were aimed at "squeezing the ego".
At any rate communication barriers exist even within one's own dialect, so don't get overly concerned with other's misinterpreting you unless you are really interested in figuring out how to engineer your posts so as to better avoid such misinterpretations in the future.
Gaatjes, that's Dutch, right?
On your topic, which now that you've explained it in enough different ways I think I am finely starting to comprehend the novelty of it all, really strikes at the fact that prefetchers have experienced an uneven amount of focus and implementation.
CPU's and their cpu-side of the prefetching business have enjoyed a healthy and robust development environment. But on the data-storage side of things, not so much. What we need are better prefetchers in terms of pulling data out of the hard-drive and into ram ahead of the user actually attempting to use the data that was pre-fetched.
Yeah I know this is what super-fetch and turbo-memory was supposed to accomplish, but since we are having this conversation clearly those efforts did not complete the task.
Raid cards with their cache and controllers do a fair amount more prefetching and buffering, but still they don't materially improve the reads because the pre-fetchers just aren't all that great.
The first step, as you hit on already, is the available bandwidth. Prefetchers in cpu's didn't get too aggressive until dual-channel 6GB/s memory topography's were common. We need hard-drive bandwidths that are 2-3x what the HDD itself needs for reading/writing so that an aggressive pre-fetcher doesn't starve the HDD itself of bandwidth. (just as prefetchers are disabled on 4S systems for the same reasons/issues)
But why hasn't this been done? Money. Look at the ASP for even the most expensive 2TB desktop drive. There is not money on HDD business anymore, not like a decade ago when at least your top of the line desktop HDD could fetch $800 and healthy margins.
So even if a consumer would step up and buy a much better streamlined OS and HDD package, who is going to pay for the development of such a system in advance of those first units hitting the shelves?
I own an areca 1280ML card with 2GB of cache, IIRC it cost me either $1500 or $1800. Point is that the sum total purchase price of all the drives I have hooked up to that 1280ML to date doesn't even come close to equally the amount I spent on the raid card, and I'd do it again if I had the opportunity because sticking spindle drives behind a 2GB cache really really helps mask the horrid write bandwidth and random IO capability of HDD's.