Why doesn't AT do 'real world' endurance testing for SSDs?

[Elixer]

I have seen sites that basically do tons of writes to the SSD, and while that may show how long the NAND lasts in terms of writes, that isn't very realistic, people don't leave machines on 24/7 writing data to the SSD.

With the amount of SSDs that AT gets for free, seems they should set up a few systems that do typical workloads, and include power cycles every day, to give a more accurate picture on the state of SSDs today.

Each system should have different chipsets (AMD based & intel), and have different test suites, and different OSs (win, linux, mac).

IMO, that information would be much better than doing a review that only looks at the SSD in question for a few days to say how reliable they are.

 

[vshah]

normal everyday loads wouldn't really make a dent in ssd life...it would be hard to measure and quantify, let alone compare.

 

[Anteaus]

This forum is full of people who have been doing real world endurance testing of SSDs since they hit the market. SSD reliability has always been sort of an odd topic because with SSDs it has been difficult to quantify. Some early generation SSDs had a tendancy to fail catastrophically with little notice while others of the same model lasted for years.

This is the way I look at it. For every day real world use, no user is likely to ever experience the write limit of an SSD. Chances are that either the SSD will either fail outright and ahead of schedule or get replaced by a faster, higher capacity unit down the road. Firmware plays a much larger role in SSD longeivity, and it is likely the reason why SSDs in general are more reliability today than when they first came out. My twin Samsung 830s are still running like I just pulled them out of the box.

There are exceptions though. I can imagine that if someone is using SSDs for video editing and are constant writing large files then the service life of the SSD could be reduced. In spite of that, I have never heard of anyone hitting that limit unless forced artificially through excessive write tests and purposeful usage.

As for reliability, just stick with reputable brands and models and reliability is going to be as good as it gets.

 

[dave_the_nerd]

I'm working on it. I'll let you know if a couple years.

But seriously - it wouldn't prove anything. Anandtech's reviewers could set up a dozen computers between them. That would give you a sample size of a dozen, which is worthless.

 

[KingFatty]

I would definitely be interested in reading about such a review. Like how car magazines will periodically update their "Long Term" testing of a vehicle, you could see how every 3 months they could re-visit the article and say, like, hey, 3 of this brand of drives just died, that makes 20 total of those suckers that failed, whereas this other brand has been going strong so far, blah blah..."

Anyway, my point is that there are several ways to make a good article out of this subject, and you see how there is definitely a desire to read about it (e.g., I read through those very interesting tests that guys just volunteered to do and post on a forum).

So the question is, what do sites like Anandtech do with all the old SSD drives they have? Hmm, could it be that they happen to disappear somehow? Or do the manufacturers demand their return? If you have a bunch just lying around collecting dust, I say put them to use with torture testing and let us read about it! The Game of Thrones version of SSD review, see who dies next...

 

[Hellhammer]

We are not equipped with the facilities and tools to test hundreds and hundreds of SSDs simultaneously. There is no centralized AnandTech office with dozens of systems -- we all work from home. I live in a one-room 33m^2 apartment, so three desktops is already hitting the limit in terms of space.

We try to do "real world" testing as well as we can with the resources we have. That means I send SSDs to other editors if they need one and I try to employ as many as possible in my own usage and switch the drives by time to time. If there is a failure, we don't hide it, but we don't go out and say that the SSD is garbage because one sample isn't enough for that conclusion. If the drive dies during our review testing, then we usually make it public because there is a good chance that there is something wrong with the drive if it dies within a couple of days of use. We have had drives to fail from every single manufacturer.

As for torture testing, there are a couple of reasons why it's pointless. First off, there is no one workload to use for the test. What most sites and people have been doing is 100% sequential writes but that is the best case scenario because write amplification is close to 1.00x. That is not applicable to real world as most IOs tend to be random. On the other hand, 100% random write is not ideal either because it's the worst case scenario with the highest write amplification. Sure we could always use something in the middle but it would still be just a rough estimation since ultimately the endurance depends on one's workload.

Secondly, having a sample size of one drive is not enough for scientific results. For performance testing it's enough because we are not exceeding the specs but because NAND is binned for endurance, there may be difference among drives. It's the same as with CPUs -- some dies overclock more than the others, so for detailed analysis more than one is needed.

Thirdly, I just don't see the point now that every OEM is including an endurance rating. When buying a drive, you should take the rating into account to make sure the drive is suitable to your usage. There are differences among consumer drives -- some offer more endurance than the others. If you are exceeding the rating, then you bought a wrong drive in the first place because your usage is likely far heavier than typical client workload (that means 100GBs of data written per day). You can still use the SSD if you want but personally I'm not comfortable with the idea of us recommending the usage of the drive after the endurance rating has been hit and the warranty void.

 

[Elixer]

@KingFatty, yeah, that is what I was reading at the doc's office... and thought it would be good for SSDs as well.

@Hellhammer, fair enough, I thought there was a main AT HQ that is loaded to the brim with motherboards and pretty much everything you need to make tons of machines.

As you pointed out, "We have had drives to fail from every single manufacturer." and I am trying to figure out just why a SSD dies, or more specifically, decays as time goes on, when you aren't anywhere near writing tens of GBs of data per day.
I am under the assumption that the NAND goes bad, and the controller logic doesn't detect the issue in time to swap NAND cells before windows notices and starts spamming the event log with errors/warnings about the SSD.

A secure erase does fix the problem for a time, but it always comes back, so the 'bad' NAND doesn't seem to be mapped out (nor does there seem to be a way to tell the controller about the issue. It just continues on as if nothing is wrong.) SMART always shows everything is fine.
This has happened both on a samsung 830, and a toshiba Q series with varying amount of time of use.

 

[BrightCandle]

Another reason I think its not worth really running these tests is that the level of endurance is so high that it doesn't really matter to most people using them. Its simply not a useful test (especially considering the time and cost to do it) considering that at realistic write volumes the drives will actually survive a lifetime. A lot of people worry about SSD endurance but its ridiculously overblown as its really hard to actually use all those writes. Writing the entire contents of the SSD 3000 times is hard work, its a lot of data.

 

[Revolution 11]

Elixer, I used to think the same as you. But considering that most SSDs can write 10 - 100 GB per day and last years, longevity is not that much of an issue. Having a good, portable,and persistent backup regime is more important in maintaining data reliability.

If you are worried about the technical reliability of the SSD components, NAND endurance will almost always be exhausted long after a controller fails or the firmware breaks and the whole device turns into a brick. There is far too little data on component failure rates and causes, considering how important SSDs have become. The whole field is a black box of confusion, FUD, and anecdotal hearsay. It does not help that SSDs are fundamentally more complex as a concept compared to hard drives (spinning discs of metal).

Looking at other users' experiences, unreliable power and surges are far more damaging. Even the most reliable SSDs, in terms of controller quality and NAND durability, can and will die easily if the power starts jumping and dipping too much. And we are not even talking about data loss yet.

The unreasonable worry I still have from the early days is data retention. If your drive sits for a year unpowered, does the data remain? What is the maximum amount of time you can leave a drive unpowered. I am sure hard drives have some limit but I have been able to pull and push data off a drive that was unpowered for 5 years. Pretty sure a SSD can't do that.

 

[Ayah]

do you seriously want to wait 10-30+ years for real world testing to complete?

My 80GB X-25MG2 has 16611GB of host writes, 1110 power ons and 33813 power on hours. The media wearout indicator is at 94.

 

[dac7nco]

Also here with striped 80GB Intel G2s for nearly five years and three systems being hammered as a scratch disc... These would probably be fine as OS drives until the zombies come.

 

[KingFatty]

I agree it's not interesting for testing purposes.

But it's interesting for entertainment purposes. For example, I would also like to see an article about how you 'dispose' of old hardware by lining it up, and shooting it with a high powered artillery. Maybe line up different SSDs in a row, and see with can stop the bullet in the fewest number of SSDs.

Similarly, I like the idea of killing off SSDs by writing them to death!

But another more subtle point is that many many people out there are "scared" of SSDs because they worry about the drives dying too soon. I mean we see lots of posts about people too hesitant to run a test on their drive because they would "use up" the lifetime of it or something, or worrying about putting the Windows temp file on it for using up lifetime and thereby hurting performance by moving the temp file to a spinning disk (maybe some of you guys are guilty of this!).

So by writing an article, and revisiting it for status update etc., would do a public service of help educating readers on how reliable SSDs really are, and you don't have to baby them. That itself would perhaps have a side-effect of reducing clutter on this forum because you wouldn't see those recurring arguments when people can just point to articles demonstrating that SSDs are great.

Think about it, computer tech sites like Anandtech, Toms, etc., they might be somewhat similar in that they do the "proper" tests, but maybe it's a way to really make your site stand out from others, being kind of like a Mythbusters style of doing funny or interesting things, like intentionally trying to kill SSDs for entertainment by writing them. Break the mold.

 

[BrightCandle]

There have been loads of these articles however and so far it hasn't cured the problem, people still fear the SSD running out of life. Its not something that is going to be solved, even if the average SSD will actually last longer than a HDD they still have the irrational fear of it running out of writes.

If HDD went into a read only mode when they ran out of writes and that was mostly how they ran out I would be extremely happy with that! The sad reality is that almost all SSDs/HDDs just fail instead of running out. But going read only is a great alternative to failure if and when they reach that level of reliability.

 

[Ayah]

There have been loads of these articles however and so far it hasn't cured the problem, people still fear the SSD running out of life. Its not something that is going to be solved, even if the average SSD will actually last longer than a HDD they still have the irrational fear of it running out of writes.

If HDD went into a read only mode when they ran out of writes and that was mostly how they ran out I would be extremely happy with that! The sad reality is that almost all SSDs/HDDs just fail instead of running out. But going read only is a great alternative to failure if and when they reach that level of reliability.

or we can move to something like mram. cost/capacity sucks a ton right now though.

 

[Enigmoid]

See Techreports SSD endurance testing. Those drives can take a massive beating.

600TB and all are still kicking

http://techreport.com/review/26058/the-ssd-endurance-experiment-data-retention-after-600tb/2

300 GB/day for five years equivalent. You can see the difficulty and sample size needed for comparison.

 

[Elixer]

There are a few types of endurance, one is just writing the drive to death, to see how many TBs of data it takes, and yeah, that has been done to death, enough so, that people shouldn't worry about writing data to the SSD.

The other type of endurance, the one I want to have more facts on is what normal usage patterns do.
People don't leave their machines on 24/7, and they don't access the SSD 24/7.

Yeah, it takes a long time to power cycle the system a couple times of day, and also do different test suites, but I don't know another way to test the actual life cycle of a SSD.
It was said that power problems can lead to premature failures, but, that can't be the whole story, since, in my case, a line conditioner is in use, and the PSU has rock stable rails.

That means, that something else is going wrong with the NAND, something that the controller can't detect for some unknown reason, and that is the worrying part.
I have a strong feeling it is still power related, since that is the only real difference between running the SSD 24/7 versus normal use.

I am wondering if getting a SSD that has CAPs in it would be a better choice, than the ones without CAPs in it.
That should deal with all kinds of power issues that may crop up, but, again, there are no tests to back this kind of stuff up.

 

[dave_the_nerd]

There are a few types of endurance, one is just writing the drive to death, to see how many TBs of data it takes, and yeah, that has been done to death, enough so, that people shouldn't worry about writing data to the SSD.

The other type of endurance, the one I want to have more facts on is what normal usage patterns do.
People don't leave their machines on 24/7, and they don't access the SSD 24/7.

Yeah, it takes a long time to power cycle the system a couple times of day, and also do different test suites, but I don't know another way to test the actual life cycle of a SSD.
It was said that power problems can lead to premature failures, but, that can't be the whole story, since, in my case, a line conditioner is in use, and the PSU has rock stable rails.

That means, that something else is going wrong with the NAND, something that the controller can't detect for some unknown reason, and that is the worrying part.
I have a strong feeling it is still power related, since that is the only real difference between running the SSD 24/7 versus normal use.

I am wondering if getting a SSD that has CAPs in it would be a better choice, than the ones without CAPs in it.
That should deal with all kinds of power issues that may crop up, but, again, there are no tests to back this kind of stuff up.

The only companies that have done testing like that with hard drives, with a large enough sample size to matter, are big data firms with warehouses full of disks.

So relax and give backblaze a few years. The data you want doesn't exist yet.

 

[Techhog]

I was just concerned becasue AT's review of the MX100 said 72TB, which really doesn't soiund like much with stuff like the page file at work.

 

[SSBrain]

I was just concerned becasue AT's review of the MX100 said 72TB, which really doesn't soiund like much with stuff like the page file at work.

If you tried keeping track of SSD writes over time, you would see that you don't need to be concerned with them and that there is no need to "optimize" anything. 72TB is quite a lot for most consumer workloads (with perhaps the exception of anything involving video editing/processing, although the SSD won't wear out overnight). The pagefile involves a minimal amount of writes unless you're using it heavily due to chronic lack of system RAM, which normally isn't the case.

Anyway, 72TB is a very conservative figure which Crucial keeps low for commercial purposes.

 

[Hellhammer]

I was just concerned becasue AT's review of the MX100 said 72TB, which really doesn't soiund like much with stuff like the page file at work.

72TB is ~66GB per day for three years. Very few client workloads (e.g. high-end video editing is one exception) write that much per day for seven days a week. Besides, it's not like the drive dies immediately after you hit 72TB -- the manufacturer just won't cover the failure after that point. The endurance ratings are usually based on 4KB random write workloads, which are the worst case scenarios, so for a typical client workload the write endurance can be a lot more.

 

[Revolution 11]

What I want to see instead of vanilla endurance tests where the whole drive is filled and emptied regularly is tests that use a 80/20 data mix. 80% of the drive is permanent, static data, while the other 20% is emptied and filled to test endurance. Real users don't empty entire drives only to fill them back up again. Most users fill up a large portion of their drive and then start deleting old data to make more room.

To be conservative, use 4KB random write workloads or workloads with high write amplification on the 20% portion. I have seen some drives mention that they use a testing method similar to this but not every manufacturer specifies how they test endurance.

 

[Cerb]

I was just concerned becasue AT's review of the MX100 said 72TB, which really doesn't soiund like much with stuff like the page file at work.

Even with the page file going, you're highly unlikely to use more than 5-10GB/day. You should be using much more in browser caches when browsing these forums, FI, than gets written to the PF, unless you are starved for RAM. I don't like when they have the same total amount of endurance for all sizes, but you're still unlikely to reach it, unless you already have been wearing out HDDs and SSDs regularly.

 

[Cerb]

normal everyday loads wouldn't really make a dent in ssd life...it would be hard to measure and quantify, let alone compare.

For the most part, regular use will expose data corruption bugs, power state change bugs, idle GC not working right bugs, unexpected SATA controller responses, etc., more than anything else, and not leave much useful data, given that the SSDs, by series, are replaced every 1-2 years, with the new ones having all-new firmwares.

 

[Techhog]

Even with the page file going, you're highly unlikely to use more than 5-10GB/day. You should be using much more in browser caches when browsing these forums, FI, than gets written to the PF, unless you are starved for RAM. I don't like when they have the same total amount of endurance for all sizes, but you're still unlikely to reach it, unless you already have been wearing out HDDs and SSDs regularly.

Yeah, the cache worries me too. I watch a lot of streaming video, so that's my biggest worry.

 

[Cerb]

Yeah, the cache worries me too. I watch a lot of streaming video, so that's my biggest worry.

Then get the biggest one you can, an Intel Sandforce if you are willing to pay for it, avoid the Samsung 840 Evo, and call it a day.