UPDATE July 13:
Since the last update of this blog-post, we have been in a cooperation with Samsung trying to help them find the issue, during this investigation we agreed with Samsung not to communicate until their approval. As the issue was not reproduced on our server in Singapore, the reproduction is now running under Samsung supervision in Korea, out of our environment. Although Samsung requested multiple times an access to our software and corrupted data, we could not provide it to them in order to protect the privacy and data of our customers.
Samsung asked us to inform you about this:
- Samsung tried to duplicate the failure with the latest script provided to them, but no single failure has been reproduced so far.
After unsuccessful tries to reproduce the issue with Bash scripts we have decided to help them by creating a small C++ program that simulates the writing style and pattern of our application (no files are open with O_DIRECT). We believe that if the issue is coming from a specific way we are using the standard kernel calls, it might take a couple of days and terabytes of data to be written to the drive. We have been informed by Samsung that no issue of this kind have been reported to them. Our server provider has modified their Ubuntu 14.04 images to disable the fstrim cron in order to avoid this issue. For the last couple of months after not using trim anymore we have not seen the issue again.
- Samsung will do further tests, most likely from week 29 onwards, with a much more intensive script provided by Algolia.
UPDATE July 17:
We have just finished a conference call with Samsung considering the failure analysis of this issue. Samsung engineering team has been able to successfully reproduce the issue with our latest provided binary. Samsung had a concrete conclusion that the issue is not related to Samsung SSD or Algolia software but is related to the Linux kernel. Samsung has developed a kernel patch to resolve this issue and the official statement with details will be released tomorrow, July 18 on Linux community with the Linux patch guide. Our testing code is available on GitHub.
This has been an amazing ride, thank you everyone for joining, we have arrived at the destination.
Another reason to never go near Samsung SSDs from now on
Samsung have really fallen through the floor. TLC issues, bricked firmware issues and now corrupt data. Avoid like the plague.
Does this mean that Samsung also fixed/patched those Crucial/Micron SSD's?Code:/* devices that don't properly handle queued TRIM commands */ { "Micron_M500*", NULL, ATA_HORKAGE_NO_NCQ_TRIM | ATA_HORKAGE_ZERO_AFTER_TRIM, }, { "Crucial_CT*M500*", NULL, ATA_HORKAGE_NO_NCQ_TRIM | ATA_HORKAGE_ZERO_AFTER_TRIM, }, { "Micron_M5[15]0*", "MU01", ATA_HORKAGE_NO_NCQ_TRIM | ATA_HORKAGE_ZERO_AFTER_TRIM, }, { "Crucial_CT*M550*", "MU01", ATA_HORKAGE_NO_NCQ_TRIM | ATA_HORKAGE_ZERO_AFTER_TRIM, }, { "Crucial_CT*MX100*", "MU01", ATA_HORKAGE_NO_NCQ_TRIM | ATA_HORKAGE_ZERO_AFTER_TRIM, }, { "Samsung SSD 8*", NULL, ATA_HORKAGE_NO_NCQ_TRIM | ATA_HORKAGE_ZERO_AFTER_TRIM, },It may be safer not to use Samsung SSDs with Linux, but it may not be safe enough.Code:/* * As defined, the DRAT (Deterministic Read After Trim) and RZAT * (Return Zero After Trim) flags in the ATA Command Set are * unreliable in the sense that they only define what happens if * the device successfully executed the DSM TRIM command. TRIM * is only advisory, however, and the device is free to silently * ignore all or parts of the request. * * Whitelist drives that are known to reliably return zeroes * after TRIM. */ /* * The intel 510 drive has buggy DRAT/RZAT. Explicitly exclude * that model before whitelisting all other intel SSDs. */ { "INTEL*SSDSC2MH*", NULL, 0, }, { "Micron*", NULL, ATA_HORKAGE_ZERO_AFTER_TRIM, }, { "Crucial*", NULL, ATA_HORKAGE_ZERO_AFTER_TRIM, }, { "INTEL*SSD*", NULL, ATA_HORKAGE_ZERO_AFTER_TRIM, }, { "SSD*INTEL*", NULL, ATA_HORKAGE_ZERO_AFTER_TRIM, }, { "Samsung*SSD*", NULL, ATA_HORKAGE_ZERO_AFTER_TRIM, }, { "SAMSUNG*SSD*", NULL, ATA_HORKAGE_ZERO_AFTER_TRIM, }, { "ST[1248][0248]0[FH]*", NULL, ATA_HORKAGE_ZERO_AFTER_TRIM, },
That blacklist probably had nothing to do with the latest bug.Does this mean that Samsung also fixed/patched those Crucial/Micron SSD's?
Quite the irony.
It's been posted on the linux-raid list ~2 days ago. Look for "[PATCH] raid0: data corruption when using trim" and its followups.My bad. You are right. Algolia's bug is not related to "queued TRIM".
Though, as of today, I couldn't find any kernel patch or info submitted by samsung regarding this bug.
/me remembers their platter drives... that had a firmware bug that would cause data-corruption, by reading SMART data while writing data.
/me thinks Samsung storage has always been at the quality level of OCZ.
It's been posted on the linux-raid list ~2 days ago. Look for "[PATCH] raid0: data corruption when using trim" and its followups.
Thanks for that.Oh wow, by this theory of banning samsung ssd, let ban all the hdd manufacture because they all have x% return rate. I see people bitching about seagate failures. they must be ban.
I saw ios randomly reboot, apple should be ban too. I heard about all the security issues in android, lets ban google too.
Stop bitching about everything, nothing is perfect.
/me remembers their platter drives... that had a firmware bug that would cause data-corruption, by reading SMART data while writing data.
/me thinks Samsung storage has always been at the quality level of OCZ.
I'm surprised how the title hasn't been fixed yet, neither by the OP nor the mods, even after numerous posts detailing how it's a Linux only issue, guess that means no one here loves Samsung anymoreYes this thread title is misleading and should be corrected so as to not bring people here needlessly only to discover that their free OS is actually the culprit. My 840 pro is still performing its duties in my laptop quite nicely after a very pain staking rma process. Once samsung gets that ironed out they'll be able to sell more drives.
The Spinpoint 1 TB platter drives were the gold standard for good performance with low noise for many years, so me thinks your memory has suffered its own TRIM bug
I'm surprised how the title hasn't been fixed yet, neither by the OP nor the mods, even after numerous posts detailing how it's a Linux only issue, guess that means no one here loves Samsung anymore
Speaking of Samsung firmware bugs, wasn't there a big announcement a while ago that they were rolling out a second fix for that same issue because it resurfaced? I never received a firmware update notice.