Outage

[Jason Clark]

Hi folks, well to say that yesterday wasn't a barrel of monkeeys is an understatement. We had a disk go bad in our raid (brand new disk), and that caused extensive data corruption in 3 of our main databases, totalling over 60gb. We've been at it since about 2pm yesterday removing data corruption, and re-instating the databases. We're 99% sure we didn't lose a single piece of data at this point.

Have a good day.

 

[kranky]

Been there, and I can easily imagine what a fun time that was.

:beer: time for you!

 

[imported_Phil]

Ouch!
:beer:

 

[Jason Clark]

Yeah, it was a blast. I've never seen a disk spawn media errors, and the controller not drop it from the array as failed. It hung around and decided to pooch some data along with it.

 

[MaceX]

I still love you

 

[Capt Caveman]

Thanks for all of your work! You guys rock!!!

 

[wallsfd949]

oh thanks.. I read this just after I discover two disks in a RAID10 just bit it and it's still 3+ hrs before the guy who has access to our cage in dallas can get there. Fortunately it happened to be the right two, but I'm still sweating..



I feel for you and hope I don't have to play the same game today...

 

[dug777]

Nice

 

[imported_Phil]

Originally posted by: Jason Clark
Yeah, it was a blast. I've never seen a disk spawn media errors, and the controller not drop it from the array as failed. It hung around and decided to pooch some data along with it.

I'm curious to know; what disks and which controller?

 

[FreshPrince]

can you explain what raid level you were using and which disk went bad?

I'm trying to understand why one disk going bad would cause database corruption.

This would help me troubleshoot my shop later if the same thing happens...

 

[EagleKeeper]

For those that might be superstitious, it was an sign to stop spending so much time on line and get out to see what the real world looks like.

 

[hardcandy2]

Think they were using RAID 0?

Which OS was this?

 

[Sunbird]

I went out and the real world still sucks, please don't let the disks make me cry again

 

[Doboji]

Ever feel like investing into some real data storage let me know. I work for Network Appliance...

RAID 4 Double Parity FTW

-Max

 

[FreshPrince]

Originally posted by: Doboji
Ever feel like investing into some real data storage let me know. I work for Network Appliance...

RAID 4 Double Parity FTW

-Max

do you guys have any real SAS solutions yet?

 

[Ricemarine]

Ouch.

Least its back!

 

[biostud]

Once at my work 13 out of 15 15K SCSI drives + the controller crashed at the same time. I don't think they ever found out why......

 

[jadinolf]

Originally posted by: MaceX
I still love you

:laugh:

 

[Doboji]

Originally posted by: FreshPrince

Originally posted by: Doboji
Ever feel like investing into some real data storage let me know. I work for Network Appliance...

RAID 4 Double Parity FTW

-Max

do you guys have any real SAS solutions yet?

No not yet, I'm hearing spring or fall. However I think 4gb Fibre Channel is more of a priority than SAS right now.

-Max

 

[Tu13erhead]

Originally posted by: Sunbird
I went out and the real world still sucks, please don't let the disks make me cry again

LOL, best post I've seen today.

:laugh:

 

[spyordie007]

Originally posted by: Tu13erhead

Originally posted by: Sunbird
I went out and the real world still sucks, please don't let the disks make me cry again

LOL, best post I've seen today.

:laugh:

Dont worry, it's still early

 

[DVK916]

Maybe you should have a backup system. A system that is a clone of the primary one. Where all of the data is duplicated and save, so if one goes down the other can just pop up and take its place.

 

[Blain]

Originally posted by: hardcandy2
Think they were using RAID 0?

Which OS was this?

Yep, we've all said how dangerous RAID 0 could be. That's why it's important to make regular CDR backups.
Maybe they were using Windows ME.

 

[BigToque]

Originally posted by: biostud
Once at my work 13 out of 15 15K SCSI drives + the controller crashed at the same time. I don't think they ever found out why......

:Q

 

[cliftonite]

Originally posted by: Blain

Originally posted by: hardcandy2
Think they were using RAID 0?

Which OS was this?

Yep, we've all said how dangerous RAID 0 could be. That's why it's important to make regular CDR backups.
Maybe they were using Windows ME.

haha :laugh: