2TB limit? How to overcome?

[AstroGuardian]

Helo everyone,

I have just been given an order to assemble and configure a temporary server for database hosting purposes. I have a GigaByte P43 + ICH10R Motherboard as a solution to do this.
I need to put 5 Western Digital disks 640GB each in RAID5 using the ICH10R controller. But i have remembered something like there is a 2Tb limit. Having in mind that 5 x 640 GB in RAID5 is 4 x 640Gb = 2560Gb which is more than 2TB (2048 Gb).

So is there any workaround how to bypass the 2Tb barrier? Could i use the whole capacity? Would partitioning for example first= 560Gb and second=2000Gb solve the problem or the limit applies to the whole volume? Or at least will i be good with using just the 2Tb?

Any solutions?
Thanx

 

[CalvinHobbes]

http://technet.microsoft.com/e.../library/cc773268.aspx
http://www.microsoft.com/whdc/...torage/GPT-on-x64.mspx

 

[Madwand1]

You need an OS past XP-32 in order to do this, with GPT. E.g. XP-64, Server 2003 or Vista (32 or 64).

You can access just the first 2 TiB without that.

 

[Nothinman]

So is there any workaround how to bypass the 2Tb barrier? Could i use the whole capacity? Would partitioning for example first= 560Gb and second=2000Gb solve the problem or the limit applies to the whole volume? Or at least will i be good with using just the 2Tb?

The problem is that PC BIOS partition tables only go up to 2TB so you can never have a partition that goes past that point. The only option is to use another partitioning format like GPT or don't use partitions at all. For example, in Linux you can set a whole block device with no partitions as a LVM physical volume and from there you can create a logical volume of any size.

 

[mooseracing]

IIRC I though over 2TB RAID 5 wasn't recommened because of error rates?

 

[dphantom]

Originally posted by: mooseracing
IIRC I though over 2TB RAID 5 wasn't recommened because of error rates?

Not completely. This link, while a wikipedia entry, is not bad at explaining various RAID configs and their limitations.

Link

 

[soflawill]

ah for the days of the, what was it? 32 gig limit? wow, either oldtimers disease is setting in or killed too many cells in the 70's. what was that limit?

 

[Nothinman]

ah for the days of the, what was it? 32 gig limit? wow, either oldtimers disease is setting in or killed too many cells in the 70's. what was that limit?

There have been a bunch of different limits over the years, it depends on what protocol and interface you're talking about.

 

[RebateMonger]

Offhand, I can think of 33 MB, 528 MB, 2.1 GB, 8.4 GB, 32 GB, 137 GB, 2 TB in the IBM/DOS/Windows world. But I've probably missed some. There are likely other combinations that existed with certain BIOSes.

You can sorta' combine multiple volumes into a drive that looks much larger using Volume Mount Points. Whether this makes sense depends on your particular needs.

GPT is the "sure thing". But not as a boot drive. But I doubt it's well supported by many drive utilities.

 

[AstroGuardian]

I forgot to mention that i will be using Windows Server 2003 Standard x86.

So if i make a 560Gb boot partition and install Windows on it, and if later at the disk management console i create 2Tb GPT partition for database storage, i am just fine right? Or i can just create a 2Tb normal partition and not messing up with GPT?

 

[Nothinman]

So if i make a 560Gb boot partition and install Windows on it, and if later at the disk management console i create 2Tb GPT partition for database storage, i am just fine right? Or i can just create a 2Tb normal partition and not messing up with GPT?

No, you have to choose between a PC BIOS partition table or a GPT partition table. Once you partition a drive with GPT it includes a faux MBR so that older utilities won't accidentally screw with it, but that's it.

 

[mooseracing]

Originally posted by: dphantom

Originally posted by: mooseracing
IIRC I though over 2TB RAID 5 wasn't recommened because of error rates?

Not completely. This link, while a wikipedia entry, is not bad at explaining various RAID configs and their limitations.

Link

But it has nothing about error problems with RAID and large arrays....I did find a reference on goolge that refreshed my memeory,

According to Zdnet, SATA drives often have unrecoverable read rates (URE) of 10^14, which implies that disk drives will not be able to read a sector once every 100,000,000,000,000 bits read. With hard drive capacities expected to reach two-terrabytes in 2009, the odds of a read error become practically unavoidable when recovering from a 7-drive RAID 5 disk failure. Upon encountering such a read error during a reconstruction process, it is claimed that the array volume will be declared unreadable and the recovery processes will be halted. Apparently all 12-terrabytes of data stored on the drives will be lost... or at least will require some extra effort and knowledge to recover.

I have heard from other admins that it doesn't even need to be that large of an array though.

 

[Nothinman]

But it has nothing about error problems with RAID and large arrays....I did find a reference on goolge that refreshed my memeory,

The problem has nothing to do with RAID, it's just that statistically the chance of problems grows as you add more drives and RAID5 is/was the most popular RAID level for arrays with more than 2 disks. If you're using RAID level with more redundancy like 6 the chances of problem go down a bit because you've got more parity to work with. If you have some other level of protection like the data checksumming in ZFS you should be pretty good as well.

 

[Snooper]

The problem is caused more by the number of spindles (aka: drives) that CAN fail than the size of the array. The probability of each single drive failing at any given time is the same. But the probability that ONE drive will fail goes up exponentially with the number of drives that COULD fail. Of course, that statement is dependent on me remembering my statistics and probability class correctly and what we were told in the NetApp Data OnTapp class that I took a few years ago...

One drive: 2 ^ 0 = 1
Two drives: 2 ^ 1 = 2
Three drives: 2 ^ 2 = 4
Four drives: 2 ^ 3 = 8
Five drives: 2 ^ 4 = 16
...
...
...

If I remember what the NetApp guys were saying, if you use RAID 5, then you should keep the number of spindles to 12 or less. Anything more than that and the failure rate multiplier starts to REALLY impact the overall failure rate of the array. If you had to have more spindles than that, they highly recommended you go with RAID 6 as the two parity drives made a huge improvement in the array failure rate probability.

All that is a very long way of saying that large arrays caused by large DRIVES do not adversely impact that failure probability of the array. Only large arrays caused by a large number of spindles.

On the FA880 I managed, we had 70 disks on 5 shelves, all very high quality SCSI drives. It was broke up into several different arrays (six in total) and I usually had one drive fail every two months or so. With the 10 or so redundant disks on that thing, it was usually rebuilt before I could even get down there to install a replacement disk. Also keep in mind that this system is VERY quick to flag a disk as bad and pull it out of an array, so the "failure rate" is probably quite a bit higher than most folks would expect for a desk top system.

 

[taltamir]

never ever ever ever EVER! use RAID5 on a mobo.

Either buy a 300$ controller, or do yourself a huge favor and make multiple raid1 (mirror) arrays.

 

[Nothinman]

never ever ever ever EVER! use RAID5 on a mobo.

While I don't ever recommend using a cheapo onboard RAID controller it has nothing to do with the RAID level. They're crap regardless of what kind of array you choose.

Either buy a 300$ controller, or do yourself a huge favor and make multiple raid1 (mirror) arrays.

I wouldn't go that far, I'd go with software RAID5 before wasting space of creating mulitple mirrors.

 

[RebateMonger]

TeraByte-plus hard drives and 8 GB-plus memory, especially at the consumer level, are kinda' scary. As noted, the inherent error rates of hard drives and memory (at least non-ECC memory) are likely to become bigger concerns that they've been in the past.

I suspect we'll see an resurgence of interest of ECC (and similar) memory at the consumer level. I was kinda' sorry when the parity bit in memory modules was dumped, but the only thing it could do was report the error and let the PC or OS or user decide what to do about it.

I'm not sure what will be done about errors on drives or, especially, on arrays. Sounds messy.

 

[taltamir]

Originally posted by: Nothinman

never ever ever ever EVER! use RAID5 on a mobo.

While I don't ever recommend using a cheapo onboard RAID controller it has nothing to do with the RAID level. They're crap regardless of what kind of array you choose.

You are COMPLETELY WRONG!
I received STRONG advice from people who handle it for a living to not do RAID5 on mobo, I tried it anyways, it was a disaster.
RAID5 and mobo loses the array whenever you upgrade the bios, when the battery goes low, when your OC is too high requiring a cmos clear, etc...
Not to mention that it is buggy as hell, with extremely poor performance.

RAID1 on a mobo on the other hand is resilient... if the raid1 array is lost you end up with two drives who have a perfect clone of all the data. You simply create a new raid array from them choosing to preserve the data on one of them, copying it to the other one (which still takes a while btw).

It is POSSIBLE to receover the lost array by deleting the degraded raid5 arrays showing up, then creating a new one with the exact same drive order (add them in the same order) and the exact same stripe size. And choosing NOT to clear / zero / initiate /whatever the array when asked. that will recover it, for most configurations.

RAID5 is a sensitive little ah heck and el cheapo mobo controllers are simply not suited for it. If you want to skimp get a server with linux or solaris or freenas / etc and do OS level raid5/6. Not mobo controller / driver level.
Or use a quality controller.

There is nothing inherantly "wrong" about mobo controllers, they are just always build like shit, all of them store settings in cmos instead of non volatile ram, and all (currently on the market) of them suck.

 

[Nothinman]

RAID5 and mobo loses the array whenever you upgrade the bios, when the battery goes low, when your OC is too high requiring a cmos clear, etc...

Flashing a BIOS is a rare operation and you should expect the new BIOS to start with default settings, including any arrays managed by that BIOS. And you're going to complain about it losing the array when the CMOS battery dies or you mess things up trying to OC? Come on.

Not to mention that it is buggy as hell, with extremely poor performance.

Which one? There's not only one BIOS RAID5 implementation out there.

There is nothing inherantly "wrong" about mobo controllers, they are just always build like shit, all of them store settings in cmos instead of non volatile ram, and all (currently on the market) of them suck.

Wait, so there's nothing wrong about them except for the fact that they're always implemented wrong? Nice...

 

[VirtualLarry]

Originally posted by: taltamir
There is nothing inherantly "wrong" about mobo controllers, they are just always build like shit, all of them store settings in cmos instead of non volatile ram, and all (currently on the market) of them suck.

I think that you are wrong about that. Most of them store the array metadata on a reserved part of the drive itself, not in the CMOS. That's how you can recover them by migrating your array to a compatible-RAID mobo, or using RAID2RAID to recover the array.

Edit: PS. CMOS is NVRAM.

 

[taltamir]

Originally posted by: VirtualLarry

Originally posted by: taltamir
There is nothing inherantly "wrong" about mobo controllers, they are just always build like shit, all of them store settings in cmos instead of non volatile ram, and all (currently on the market) of them suck.

I think that you are wrong about that. Most of them store the array metadata on a reserved part of the drive itself, not in the CMOS. That's how you can recover them by migrating your array to a compatible-RAID mobo, or using RAID2RAID to recover the array.

Edit: PS. CMOS is NVRAM.

cmos is not nvram, cmos is ram power by a 1.5v battery. if the charge on it falls too low the data on the cmos is lost... sometimes at just the right charge level it becomes corrupted and messes things up even more.

The firmware and cmos defaults however resides on non volatile memory

And you are right about the array data not being stored in the cmos, they do store it on the drive itself which is why you can migrate an array... The reason I had made that slip is because a cmos loss IS catastrophic to an array. My guess is that it causes the bios settings to revert to non raid mode which cases the array to be "lost", so when you turn raid back on it does not and never will detect it again because it has since already marked it as a faulty degraded array.

Point is, cmos is reset, array is lost. Move the drives to a compatible board, array is retained. That sort of loss whenever the cmos is reset (when you upgrade firmware, when your OC gets too high, when you make a change to hardware and need to clear cmos because it will not boot, etc etc etc) is catastrophic.

Besides which there is the raid5 writehole which puts your data in jeopordy, as well as horrible atrocious performance, and crappy drivers.

CAN there be a good high quality raid5 mobo implementation? yes... does nvidia, intel, amd, via, etc offer one? no!

 

[pugh]

AstroG,
Do yourself a huge favor and invest in a hardware RAID card if you are going to do RAID. Can save you a lot of trouble in the future instead of on board. That on board raid scares me.