RAID 5 data recovery

[Jeff7181]

Long story short, I was working on a server that uses a three disk RAID 5 array. One disk failed (it was in the field so we didn't have a drive to hot swap immediately) and it ran for a couple days on two disks, then a second disk failed. The disks themselves haven't been wiped, I'm sure the data hasn't been overwritten.

The data on these disks is worth no more than $10,000. Will a professional data recovery service be worth spending the money on to get that data back or should we officially cut our losses and maybe attempt amateur data recovery ourselves?

 

[Rifter]

Anything you do to try and get it yourself will reduce the chances of a professional getting your data back. You need to either send it in now or write it off and then try yourself but i wouldnt count on getting it.

I cant tell you if the data is worth it only your company can, you said no more than 10,000 so i would say if its going to cost less than that to get it back then its worth it.

This is why i NEVER run a RAID 5 array after a dropout, if it doesnt have a hot spare it gets shut off till i get another disk in there. RAID 5 is not the most reliable RAID setup out there, if holding data worth that much cash it really should have either had a hot spare or offsite backup or both.

 

[Jeff7181]

That's the problem... I don't know how much data recovery is going to cost. I also don't have the authority to get an estimate, although I will likely be tasked with recovering the data if they decide to do it in-house.

Oh... and I've always told them RAID 5 was ridiculous. We have 5 disks in these servers and only three of them are in this RAID 5 array, the others are not part of the array. Every time disk failure like this comes up I tell them we need to crack the image anyway to bake in new updates and software, we may as well change the RAID configuration to something like 0+1 or 6.

 

[sub.mesa]

Jeff: you would be able to recover the contents of the RAID5 using Linux or FreeBSD recovery, which uses its own RAID5 engine. This can be done without permanently changing any bit on your drives.

For this to succeed, you would need two of the three disks which made up the RAID5 intact and working. The disk that dropped out first is outdated as you said it ran a few days without (re-)syncing.

My question to you: what exactly do you mean with failed? Is your array split/broken because of a bad sector (minor issue; 100% data recovery expected) or did two of three disks fail totally? In the last case, you won't be able to recover your data but little bits and pieces.

So all depends on the actual state of your two HDDs that ran degraded for a couple of days.

 

[Jeff7181]

Yeah, that's the problem... a second dropped from the array before the first failed disk was replaced so I believe there's only one good disk.

I don't know exactly what the current state of the drives are (they're coming back from the field on Monday). I don't think they're physically broken because they do spin up, but I guess that's not a true indicator that it's not a physical failure.

I also just got a bit of info from another coworker. Disk 0, 1 and 2 were in the array. Disk 0 failed and of course the array survived on disk 1 and 2. Attempting to re-enable the drive and even reseat the drive in the bay didn't get it back online. A couple days later disk 2 failed. Reseating and trying to re-enable the drive didn't work with that one either. So my coworker started swapping disks into a new server to rule out a problem with the RAID controller. Disk 0 and disk 1 were inserted together and the machine booted. I find this odd because disk 0 and disk 1 should have been out of sync and the controller should have known that, shouldn't it?

Anyway... disk 1 and disk 2 have the data I want... I'm not even thinking about disk 0 right now because it's outdated and the array held the system partition so something most definitely has changed making it outdated. I'm just hoping no data was lost by using the outdated disk with one of the current disks and actually having it boot into Windows.

 

[sub.mesa]

Well, honestly the possibility is that you now have two corrupt arrays:

disk0 is not in sync with disks 1 and 2; still it would attempt to boot as the boot related data likely did not change during the time the array dropped to degraded mode.

But the real problem is that booting with disks 0 and 1 might also cause writes to have happened; which would make disk 1 to not be fully in sync with disk 2 either; thus possibly two corrupt versions now; though the corruption on disk1 could be minor.

I highly recommend not to write to the drives! If the drives spin up, you have a high degree of success on low-risk data recovery. Generally i recommend you buy 3 new disks of at least the same size and start copying all the data straight away.

If you like, i can offer my services in trying to recover your data. This would be done by writing the contents of the current disks to the 3 new disks, then creating a RAID5 array out of those three disks, then degrading the array with the long out-of-sync copy of disk0. This could work. The last step would be to copy the data from the RAID5 (including NTFS partitions) to a non-RAID single disk that is as large as the RAID5 size. The end result would be that you have a non-RAID disk with all data that was on the RAID5.

I do not need file access for this procedure, but i would need SSH access to a machine running FreeBSD, the three RAID disks, three empty disks and one big empty disk. This procedure is quite safe, the only real risk would be that the disks could be damaged so that spinning them up causes more damage; but personally i think its worth a try.

Let me know if you're interested. I've recovered many people's RAIDs.

 

[Russwinters]

Rebuilding the Raid 5 is easy.


The hard part:


What kind of disks are these?

What is the model number exactly.

What symptoms are they showing?


You can PM me for more info, I am a technician for a professional DR company.


Regards,

 

[Jeff7181]

Don't have that info in front of me right now, but I think things are better than I thought. I put all three drives in a different server and started it up. Checked the array config before it started booting the OS and it knows there's 4 logical drives on the array. I let it attempt to boot windows but it says boot.ini is corrupt. It does seem to recognize the c: drive though so even if my boss doesn't want to go through the expense of professional recovery, I think we'll be in good shape.

 

[taltamir]

1. There is something I keep on telling bosses that insist on having people servicing their own computers (which eventually ends very very badly), its akin to taking a guy off the street and demanding he perform a heart transplant. He would have no idea where to start and if you weren't his boss and threatening his job he would have probably flat out refused. This came about because I (I repair computers) actually had to fix a computer for a company where the guy literally DID tell his boss "no" when he told him to fix his computer, he took a chair and literally sat in front of his boss office saying "I haven't got any work done in 2 days, I can't fix my computer".
Naturally I fixed it in under 20 minutes, but its not something i'd expect anyone but an expert to know how to do.

Anyways... This brings me to the question of, how much do you know about computers? A hobbyist certainty is usually more knowledgeable then an on the job trained "professional". So I am not asking if you do this for a living, only what your knowledge and skill level are.
Also, are you paid (enough?) to do this kind of work?

2. RAID5 is downright stupid. I wouldn't even use RAID6... google with their amazing distributed first class storage system and teams of IT can do that, but unless you have a top notch IT team you do RAID1 and you like it.

3. What does "no more than 10,000$" mean? does it mean 9,999$? does it mean 900$? does it mean 90$? 9$?
Price should be significantly less than 10,000$ to get professional data recovery...

4. AFAIK companies that experience dataloss almost always go bankrupt within a year. You just can't survive dataloss. So 10,000$ seems on the low side. it should account for loss of business due to missed contractual obligations the delays bring, not just the cost to create it.

 

[Russwinters]

If you are not sure of the types of drives, and if any of them are exhibiting symptoms of physical failure, then you are risking everything.


Just because you can see the C: drive does not mean much with a RAID, especially a RAID 5.



I would highly recommend extra care if this is critical data.


Regards,

 

[Jeff7181]

I don't know how to answer your first question, I'm not going to bother posting my resume. Suffice it to say, I'm qualified to do my job which involves supporting around 500 servers just like this one in addition to others, but I have no professional data recovery experience or training.

I agree with your second point. Considering there are four 74 GB drives in each of these servers (there's about 500 of them nation wide) and the ~140 GB RAID 5 array rarely reaches 75% capacity at any site. Each of these servers also has an additional ~150 GB drive that I've never seen lower than 80% free space at any site. So yeah... it's a ridiculous design, but I'm not in the position to make that type of change at this point in time.

No more than $10,000 means potentially up to $10,000 but no more.

The company I work for is not in great financial shape, but a $10,000 loss absolutely in no way puts the company in jeopardy.

 

[taltamir]

I don't know how to answer your first question, I'm not going to bother posting my resume. Suffice it to say, I'm qualified to do my job which involves supporting around 500 servers just like this one in addition to others, but I have no professional data recovery experience or training.

I misunderstood your OP then, I thought you were saying that you are NOT an IT but have been fiddling with it on the side because your bosses heard you are "good with computers" and asked you to take a look. (aka, that you are an accountant or a doctor or a graphic designer or who knows what who was told to look at the server because it is giving problems)

If you are qualified to handle 500 computers then great... although this raises another alarm bell... that your bosses refuse to listen when you tell them to use raid 1+0 instead of RAID5. What makes them think they are more qualified then you to chose the type of raid to use?

No more than $10,000 means potentially up to $10,000 but no more.

This still is very vague figure. How am I to judge if a 1000$ repair is worthwhile in such a case?

The company I work for is not in great financial shape, but a $10,000 loss absolutely in no way puts the company in jeopardy.

That really depends on the data... what if the loss of the data results in delays that cause you to lose clients? Or merely loss of credibility with clients that causes loss of clients?
Is this the ONLY file-server the company owns or one of many? There are many questions here. I am sure there are things you just can't share with us, maybe it makes sense and is accurate... its just that I have a hard time imagining data that is "up to 10,000$" in worth. Is it customer data? work that has been done by employees (calculated to have cost 10,000 in wages)? who knows.

 

[Jeff7181]

It's sales data... I work for a retail chain. It's the store's POS server (among other roles).

My best guess is that the amount is between $7500 and $10,000 worth of sales data on those disks, but it could be lower given the fact that they were having issues with their point of sale system and no doubt there were times customers got sick of waiting and left. But judging by previous days sales, it should be in that range.

Obviously in a company that large I'm one of many in IT there. I'm not one of the engineers so what I recommend in terms of infrastructure changes is often heard, but ignored. Especially because in the three years that these type of servers have been deployed to 500-600 locations, this is the first time we've had two disks in the array fail before we could replace the first failed disk.

Regardless, management is dragging their feet with the approval to send the disks out so I'm thinking they're going to tell me to recover what ever I can... or maybe they'll turn them over to our SAN admins and let them have a look at it even though it's outside the scope of their work.

 

[taltamir]

I see now.
Well, if it is 7500$ to 10,000$ then AFAIK its worth more then the cost of professional data recovery. The only question is are they actually going to demand you recover it.

Didn't you mention backups though? might as well put them to use and recover the data from them.

From what you say it seems like you don't have anything to do until management decides what they actually want done.

 

[uOpt]

What RAID system?

If one of the disks decides to be functional again for a while (very likely, if you count in fridge/freezer tricks) then you can just force-assemble the RAID and mount the filesystem (do a readonly mount of course).

I had a couple of RAID5s double-dropped because of a bad controller just recently. You just force-assemble it with the last disk to drop (because you know no data has been written since then), copy off the data. Then re-sync with the last disk. Do not re-sync at the same time that you copy the data off. That is with Linux's md RAID system.