Samsung 840/850 PRO +more Fatal Trim Firmware bugs?

[Phynaz]

As in data destroying major.

Originally via Tech Report.

http://blog.algolia.com/when-solid-state-drives-are-not-that-solid/





Threads Merged

Anandtech Administrator
KeithTalent

 

[Elixer]

https://blog.algolia.com/when-solid-state-drives-are-not-that-solid/
Yikes.

The complete picture

At this moment we finally got a complete picture of what was going on. The system was issuing a TRIM to erase empty blocks, the command got misinterpreted by the drive and the controller erased blocks it was not supposed to. Therefore our files ended-up with 512 bytes of zeroes, files smaller than 512 bytes were completely zeroed. When we were lucky enough, the misbehaving TRIM hit the super-block of the filesystem and caused a corruption. After disabling the TRIM, the live big files were no longer corrupted but the small files that were once mapped to the memory and never changed since then had two states – correct content in the memory and corrupted one on the drive. Running a check on the files found nothing because they were never fetched again from the drive and just silently read from the memory. Massive reboot of servers came into play to restore the data consistency but after many weeks of hunting a ghost we came to the end.

As a result, we informed our server provider about the affected SSDs and they informed the manufacturer. Our new deployments were switched to different SSD drives and we don’t recommend anyone to use any SSD that is anyhow mentioned in a bad way by the Linux kernel. Also be careful, even when you don’t enable the TRIM explicitly, at least since Ubuntu 14.04 the explicit FSTRIM runs in a cron once per week on all partitions – the freeze of your storage for a couple of seconds will be your smallest problem.

Broken SSDs:

SAMSUNG MZ7WD480HCGM-00003
SAMSUNG MZ7GE480HMHP-00003
SAMSUNG MZ7GE240HMGR-00003
Samsung SSD 840 PRO Series
recently blacklisted for 8-series blacklist
Samsung SSD 850 PRO 512GB
recently blacklisted as 850 Pro and later in 8-series blacklist

Working SSDs:

Intel S3500
Intel S3700
Intel S3710

 

[VirtualLarry]

Yeah, Linux has some weirdness with TRIM. Probably has to do with drive optimizations and FS determinism and block device write barriers.

Windows 7 doesn't seem to have these problems.

 

[KingFatty]

What about the Samsung SSD 850 EVO line, are they affected just like the PRO is?

 

[Berryracer]

Another reason to never go near Samsung SSDs from now on

 

[Elixer]

What about the Samsung SSD 850 EVO line, are they affected just like the PRO is?

Don't know... you care to run some tests to verify if their firmware is faulty as well?

You know, these past few months have show that Samsung's QC seems to be lacking, they are almost at the point where OCZ was a long, long time ago.

 

[Coup27]

they are almost at the point where OCZ was a long, long time ago.

Think we are a long way from that.

 

[VirtualLarry]

Don't know... you care to run some tests to verify if their firmware is faulty as well?

You know, these past few months have show that Samsung's QC seems to be lacking, they are almost at the point where OCZ was a long, long time ago.

/me remembers their platter drives... that had a firmware bug that would cause data-corruption, by reading SMART data while writing data.

/me thinks Samsung storage has always been at the quality level of OCZ.

 

[Binky]

Linux only?

 

[Z15CAM]

Reminder this thread points to Ext 4 tables and Linux OS NOT NTFS Tables or FAT tables / Windows OS.

Granted Samsung has to do something with their SSD's running RAID Vs Trim.

Least their flaky Magician Software when their SSD's are mounted in Intel SATA Raid Mode whether or not their SSD is or is not a member of an array. Intel say if the SSD is not a Member of an Array the mode is AHCI where Samsung software doesn't recognize it.

In my opinion - Screw Samsung Software where their SSD's do not support TRIM when their SSD's are mounted as a member of an Array.

However; Windows TRIM does work on their SSD's when placed in RAID Mode as long as they are not a member of an array and in AHCI Mode - Can't say about Linux Ext4 Tables though - Been awhile - SUSE 9 or 10 - Anyway SSD's weren't available ;o)

 

[ShintaiDK]

Samsung have really fallen through the floor. TLC issues, bricked firmware issues and now corrupt data. Avoid like the plague.

 

[mindbomb]

So, what are the good ssd brands? intel and sandisk?

 

[Elixer]

Reminder this thread points to Ext 4 tables and Linux OS NOT NTFS Tables or FAT tables / Windows OS.

At this point, yeah, if I was a linux user, I wouldn't be using a Samsung SSD.
(Though, I wouldn't get it for any OS, since I really hate their RMA process)

However, since those same systems are using TRIM just fine with Intel SSDs, seems to me there is more to it...

I know I have both a Toshiba and Crucial SSDs on my linux boxes, and I use TRIM on those and had no issues.

 

[VirtualLarry]

This issue is good to know though. I just threw a 120GB 850 EVO in a laptop, and I was considering dual-booting Mint on it.

 

[Zodiark1593]

In the span of a few months, Samsung SSDs went from some of the best to virtually dirt. Fast fall.

 

[finglobes]

I was new to SSDs when I started buying last November. I saw Samsung had issues despite sexy vibe in media and in reviews. I went with Intel. Its not the fastest but its been tested to heck for enterprise market so it seemed the way to go for now.

 

[Jovec]

It is probably a Samsung firmware bug, but the issue is related to consumer hardware in an enterprise environment with a specific setup. It's not happening on Windows, or it would be much more widespread. It might not be happening solely on Linux in general either, but rather something specific to their version, apps, usage, and config.

Still sucks, I was planning to move my 840 Pros into a new Linux system, and this does give me second thoughts.

 

[coercitiv]

It might not be happening solely on Linux in general either, but rather something specific to their version, apps, usage, and config.

Still sucks, I was planning to move my 840 Pros into a new Linux system, and this does give me second thoughts.

It may be even more specific than that, since the affected systems were using RAID. One commenter from the OP link pointed out a specific "recent mdraid bug with raid0 that was introduced in various stable kernels, where TRIM would actually trim the wrong files/regions, zero-ing out random files on the file system". Warning was issued on 20 May 2015.

all users using a Software Raid 0 on SSD's with discard should disable discard, if they use any recent kernel since mid-April 2015. The bug was introduced by commit 47d68979cc968535cb87f3e5f2e6a3533ea48fbd and the fix is not yet in Linus tree.

Moreover, did anyone take a look at the blacklist mentioned in the blog post? Some interesting info right there:

/* devices that don't properly handle queued TRIM commands */
    { "Micron_M500*",        NULL,    ATA_HORKAGE_NO_NCQ_TRIM |
                        ATA_HORKAGE_ZERO_AFTER_TRIM, },
    { "Crucial_CT*M500*",        NULL,    ATA_HORKAGE_NO_NCQ_TRIM |
                        ATA_HORKAGE_ZERO_AFTER_TRIM, },
    { "Micron_M5[15]0*",        "MU01",    ATA_HORKAGE_NO_NCQ_TRIM |
                        ATA_HORKAGE_ZERO_AFTER_TRIM, },
    { "Crucial_CT*M550*",        "MU01",    ATA_HORKAGE_NO_NCQ_TRIM |
                        ATA_HORKAGE_ZERO_AFTER_TRIM, },
    { "Crucial_CT*MX100*",        "MU01",    ATA_HORKAGE_NO_NCQ_TRIM |
                        ATA_HORKAGE_ZERO_AFTER_TRIM, },
    { "Samsung SSD 8*",        NULL,    ATA_HORKAGE_NO_NCQ_TRIM |
                        ATA_HORKAGE_ZERO_AFTER_TRIM, },

/*
     * As defined, the DRAT (Deterministic Read After Trim) and RZAT
     * (Return Zero After Trim) flags in the ATA Command Set are
     * unreliable in the sense that they only define what happens if
     * the device successfully executed the DSM TRIM command. TRIM
     * is only advisory, however, and the device is free to silently
     * ignore all or parts of the request.
     *
     * Whitelist drives that are known to reliably return zeroes
     * after TRIM.
     */

    /*
     * The intel 510 drive has buggy DRAT/RZAT. Explicitly exclude
     * that model before whitelisting all other intel SSDs.
     */
    { "INTEL*SSDSC2MH*",        NULL,    0, },

    { "Micron*",            NULL,    ATA_HORKAGE_ZERO_AFTER_TRIM, },
    { "Crucial*",            NULL,    ATA_HORKAGE_ZERO_AFTER_TRIM, },
    { "INTEL*SSD*",         NULL,    ATA_HORKAGE_ZERO_AFTER_TRIM, },
    { "SSD*INTEL*",            NULL,    ATA_HORKAGE_ZERO_AFTER_TRIM, },
    { "Samsung*SSD*",        NULL,    ATA_HORKAGE_ZERO_AFTER_TRIM, },
    { "SAMSUNG*SSD*",        NULL,    ATA_HORKAGE_ZERO_AFTER_TRIM, },
    { "ST[1248][0248]0[FH]*",    NULL,    ATA_HORKAGE_ZERO_AFTER_TRIM, },

It may be safer not to use Samsung SSDs with Linux, but it may not be safe enough.

 

[Z15CAM]

However, since those same systems are using TRIM just fine with Intel SSDs, seems to me there is more to it..

Intel Sata RAID, Ext4 Tables , Patent Firmware? Something stinks regarding Intel Sata RAID Controllers / Samsung SSD TRIM comparability.

 

[daveybrat]

Major TRIM bug found in Samsung SSD's running Linux

From reading the article, the topic of this thread should really be changed. ^^

 

[Binky]

More appropriately: "Major TRIM bug found in Linux"

 

[coercitiv]

Update on the blog article mentioned in the OP.

UPDATE June 17:
We got contacted by Samsung (for the second time regarding this issue) and we provided them all the system specifications and all the information about the issue we had. We will continue to provide Samsung all the necessary information in order to resolve the issue.

 

[R0H1T]

More appropriately: "Major TRIM bug found in Linux"

Yeah, Linux should definitely be in the OP & the title, since most people won't read such a lengthy blog to make heads/tails out of it

 

[coercitiv]

Update on the blog article mentioned in the OP.

UPDATE June 18:
We just had a conference call with European branch of Samsung and the Korean HQ of Samsung. Their engineers are going to visit one of the datacenters we have servers in and in cooperation with our server provider they will inspect the mentioned SSDs in our SW and HW setup.

 

[WhoBeDaPlaya]

/me remembers their platter drives... that had a firmware bug that would cause data-corruption, by reading SMART data while writing data.

You're probably thinking of the SpinPoint F4 2TB. Dumped those POS drives for Hitachis and haven't looked back since.

Yeah, Linux should definitely be in the OP & the title, since most people won't read such a lengthy blog to make heads/tails out of it

+1 Almost shit my pants since I took the trouble of upgrading from the vanilla 840 (read bug) to 840 Pro last year, at some expense.