filesystem for 5TB server

[acaeti]

So I am putting together a home server for media that will have a 5TB raid5. I am trying to decide what fs to use. I am kinda turned off to ext3 b/c building the system and any random fsck will take forever. ReiserFS seems to take forever to mount on reboot with large FS. And I just don't know enough about JFS/XFS. Any opinions out there?

Usage will be generally DVD isos and ~100 GB of MP3s. I built the RAID5 for data with a 1MB chunk size.

Disk speed is not really that big of an issue as most DVDs don't play at >10-12 Mbps and can't be ripped at >50 Mbps anyway. But I am worried about it crashing or losing power (UPS on the way, but it is no guarantee).

 

[cleverhandle]

Originally posted by: acaeti
But I am worried about it crashing or losing power (UPS on the way, but it is no guarantee).

Well, yeah... it is. That's kind of the point. Your UPS should be able to send a signal to the machine via serial or USB when it's almost out of power and then the machine should automatically perform a clean shutdown. I know there are several daemons out there for this, but the only one I've used is apcupsd, which is specific to APC units.

Can't help you on your actual question, though. Sorry.

 

[drag]

You'd probably want to divide your array up using lvm. Having a single large 5TB file system is pretty difficult. It's better to have smaller ones and keep a lot of extra space unused so that you can expand the FS's as you need them in the future. Also if you accidently end up with massive file system corruption for whatever reason then it's likely to only take out a portion of your data instead of the entire 5TB.

Also you probably won't want to go RAID5 if you can. Other raid levels like RAID 10 or RAID 6 are going to be more reliable and faster. For the most important stuff you'll probably want to keep mirrored on more then one machine... and backed up to DVD or tape if it's practical. Once you get to storage of this size the expense of it all starts getting a bit out of hand.

Probably XFS is a good bet for the FS. I think that XFS or Ext3 is what people generally use for this sort of application. If fsck time bothers you then look at XFS. Ext3 does do some extra data-protection stuff that other FSs in Linux don't do, but it does that those issues. This is another reason to use LVM. You can use XFS now, but when future versions of Ext3 like Ext4 stabilize and solve large volume problems (like fsck times) then you can migrate to that. A bit more future-proof.

 

[DivideBYZero]

You're going to boot from the massive FS? Not a good idea. Keep a small FS for the OS and keep several huge ones for data. Any OS corruption will only require a clean OS install and all your data will be safe and sound.

 

[TrixAreForKids]

Originally posted by: DivideBYZero
You're going to boot from the massive FS? Not a good idea. Keep a small FS for the OS and keep several huge ones for data. Any OS corruption will only require a clean OS install and all your data will be safe and sound.

This is an excellent point.

 

[ebaycj]

Originally posted by: TrixAreForKids

Originally posted by: DivideBYZero
You're going to boot from the massive FS? Not a good idea. Keep a small FS for the OS and keep several huge ones for data. Any OS corruption will only require a clean OS install and all your data will be safe and sound.

This is an excellent point.

yes, definitely go buy a tiny hard drive (40 or 80 gb should be more than enough), to boot off of.

 

[acaeti]

I do have a small 20 GB partition using LVM on the RAID for the OS and other stuff (LAMP goofing around, etc.) in ext3. And /boot on a bitty 256MB RAID1.

I guess I could setup the data dump as an expandable partition on an LVM base and grow it to my needs, but to be honest I already have ~3TB of data floating around on hard drives I swap in and out of USB disk enclosures. Part of the idea was to put it all in one place. Though I could always symlink stuff I suppose to make it all appear in the same samba share. Just a pain to have those stub 3-5 GB spaces left over that could be another DVD iso.

This is in no way enterprise/critical data or anything; there will be no tape backups or the like. Funny aside - I once lost 80 gigs worth of music due to winxp/redhat partition screw ups. This was circa 1999, when 80 gigs of music was unheard of. It's an addiction I guess - data loss and just rebuild it all.

WRT to XFS, I have read elsewhere that XFS can lead to data corruption if the machine loses power or crashes in the middle of ops (e.g. here). This is causing me to reconsider ReiserFS with data=journal, despite slow mounting and whatnot.

Thanks for all the replies thus far.

 

[Emulex]

don't feel bad i've gotten data corruption on NTFS too.

if the o/s corrupts the filesystem for whatever reason 40gb to 4000terabytes you are screwed. Best to segment into separate virtual arrays to prevent this.

Just based on my experience.

Also you should consider raid 6 which allows double drive failure). I swear it would be something i'd never see but if you do not have a RAIL for each drive, could smoke the array by dropping power to two drives. poof.

One power supply per drive or N+1 configuration only.

Think of any possible failure in a matrix and find a solution if you care about your data.

Go get an old NetAPP and a fiber card and you should have pretty darn reliable storage.

 

[Nothinman]

Any OS corruption will only require a clean OS install and all your data will be safe and sound.

Not true, since it's Linux it's usually simple to figure out which files are corrupt and just reinstall that package. If you're using Debian or Ubuntu you can use the debsums package to verify the checksums on every packaged file.

I am kinda turned off to ext3 b/c building the system and any random fsck will take forever. ReiserFS seems to take forever to mount on reboot with large FS. And I just don't know enough about JFS/XFS. Any opinions out there?

You'll only have to fsck ext3 in case of a problem so I wouldn't worry about that. Personally I'd use XFS, I've had very few problems with it, the userland tools are great and it supports huge filesystems like that.

Other raid levels like RAID 10 or RAID 6 are going to be more reliable and faster.

RAID6 writes may be a bit faster than RAID5 but RAID6 read speeds will be the same, maybe a little slower since 1 drive won't be used for reads. But in either case I really doubt the speed difference will be huge enough to care and definitely not huge enough to warrant losing all that space for RAID10.

WRT to XFS, I have read elsewhere that XFS can lead to data corruption if the machine loses power or crashes in the middle of ops (e.g. here). This is causing me to reconsider ReiserFS with data=journal, despite slow mounting and whatnot.

That's been fixed in 2.6.22 and even if it hadn't it's still no reason to go with any version of reiserfs.

if the o/s corrupts the filesystem for whatever reason 40gb to 4000terabytes you are screwed. Best to segment into separate virtual arrays to prevent this.

If the OS corrupts something then you probably have a hardware problem so segmentation won't help unless you get very lucky.

 

[drag]

I guess I could setup the data dump as an expandable partition on an LVM base and grow it to my needs, but to be honest I already have ~3TB of data floating around on hard drives I swap in and out of USB disk enclosures. Part of the idea was to put it all in one place. Though I could always symlink stuff I suppose to make it all appear in the same samba share. Just a pain to have those stub 3-5 GB spaces left over that could be another DVD iso.

You just organize your collection logically along a directory system.

For example the mp3s may be nice to organize alphabetically. A directory, B directory, C directory, etc etc. Each being a LVM. Label the LVMs themselves useful names and you can use simple bash scripts to generate the logical volumes and append their entry into /etc/fstab.

Even better it would be nice to arrange them in double-sided-DVD-sized chunks. With a 'A1', 'A2', 'A3' directory system to divide them up if the 'A' directory is to big to backup in one big thing. That way when you collect more music you just add more LVMs and keep the sizes relatively static. Once you back one LVM to DVD then you don't have to worry about losing it, and when you add more music you don't have to re-back-up everything up to take the changes. This way you can do effective backups spread over time and it won't be such a hassle.

In the future when you expand your collection further by adding another array or making a bigger server then transfering the logical volumes would be a much more simple affair then dealing with terrabytes of individual files.

Then if you want to get fancy with protecting your data, in combination with backups you can deploy something like AIDE to ensure that any files that may have some 'bitrot' due to some problem will be found quickly.

Also you should consider raid 6 which allows double drive failure). I swear it would be something i'd never see but if you do not have a RAIL for each drive, could smoke the array by dropping power to two drives. poof.

Yeppers. RAID 6 or RAID 10 are much much better for large amounts of data. RAID 5, due to how it functions, has a severe limitations that are starting to show up more and more as disk prices drop, disk sizes increase, and read/seek speeds are not keeping up. Having 2 drives fail, or having a second drive fail as your rebuilding your array from a previous failure gets more and more likely.

Of course having good backups is a actual solution to data corruption, but I can understand wanting to avoid any added expenses. After all this is not 'enterprise' this is something that's going to end up in a closet or in your basement or something. So I understand.

This is just best practices. Useful all around, take it or leave it and all that.
Of course doing this requires much more effort then just dumping everything in one huge volume, but I think the results will be worth the effort. When you got this much data I think putting more thought into it will pay off. Remember creating/mirroring/resizing/etc logical volumes can be done easily on the fly (especially with XFS), don't think of them as just fancy partitions. They are designed specifically to make things more manageable for administrators.

 

[drag]

RAID6 writes may be a bit faster than RAID5 but RAID6 read speeds will be the same, maybe a little slower since 1 drive won't be used for reads. But in either case I really doubt the speed difference will be huge enough to care and definitely not huge enough to warrant losing all that space for RAID10.

Well I said reliable AND faster. When you get big RAID5 hits some serious limitations when it comes to preventing losing a array. I don't know if the OP cares that much, I just want to give the best advice I can think of.

 

[Nothinman]

Remember creating/mirroring/resizing/etc logical volumes can be done easily on the fly (especially with XFS), don't think of them as just fancy partitions.

Except for shrinking, I think ext2/3 are the only filesystems that support shrinking at all.

 

[acaeti]

thanks for all the tips. I'm thinking XFS and RAID6 might be in my future. Especially since I have some sequential serial #'d HDs. If one goes, another may just be right behind it. Damn 500GB disks just aren't big enough . Can't the 750GBs/1TBs come down to $100/disk, say tomorrow?

FYI I believe reiser and XFS can be shrunk on the fly.

 

[Nothinman]

FYI I believe reiser and XFS can be shrunk on the fly.

No, XFS doesn't support shrinking at all and reiserfs doesn't support online shrinking or growing.

 

[RaiderJ]

Good thread. I have a 4 disk RAID 5 setup. Not really sure I ever want to expand beyond that... mostly because the overhead involved is just crazy.

What about having a RAID 6 setup, and doing sequential daily/weekly backups to a RAID 5 server? It seems that if you have that much data, investing in a few more drives for a separate backup system wouldn't really be a big expense. My motherboard has 8 SATA ports - could a pure software RAID 5 work as a sequential backup server?

 

[acaeti]

Originally posted by: Nothinman

FYI I believe reiser and XFS can be shrunk on the fly.

No, XFS doesn't support shrinking at all and reiserfs doesn't support online shrinking or growing.

I stand corrected! I should do a little more research before I open my mouth. I had forgotten that reiserfs needs to be unmounted.

 

[Nothinman]

I wish most people would just forget about reiserfs completely...

 

[Brazen]

Originally posted by: Nothinman
I wish most people would just forget about reiserfs completely...

Well I suppose they are just hurting themselves by going with reiserfs, but otherwise I do agree

XFS and Ext3 are the only two I would consider, and personally I would go with Ext3 for this job. Ext3's stability and reliability is still unmatched and you have a long way to go before running into the size limitations. And by the time you would run into those limitations, Ext4 should be stable and you'll be able to do an in place upgrade to that.

I only go with XFS when speed is more important than realiability (I almost always use XFS on my desktops/laptops). But since you will be accessing these through Samba, filesystem performance will not be an issue anyway.

As for the Raid, as Nothinman said, Raid10 is way overkill. Personally, I think for a home server the capacity loss for Raid6 is overkill, too, especially if you are going with few large capacity drives (700-1000GB). I would go with Raid5, but don't use more than 8 drives to an array. If you plan to use more than 8 drives, then create 2 smaller arrays and concatenate them with lvm. Of course, using 2 smaller arrays will also mean you lose more capacity than 1 big array, but IMO opinion the performance (especially rebuild time in case of a failure) is worth it (and when you get into that many drives, failures are to be expected).

I'll also reiterate that putting your OS on it's own drive is definitely a good idea here (but it looks like you already know that )

 

[Nothinman]

Ext3's stability and reliability is still unmatched and you have a long way to go before running into the size limitations.

Not that ext3 is bad, but no filesystem is perfect: http://www.mail-archive.com/li...rnel.org/msg01993.html

 

[drag]

I figure that with XFS backed up by a battery UPS that can talk to Linux over USB or serial is as good as anything out there. Between Ext3 vs XFS it's more personal preference then anything.


There is a reason why they are working on things like Ext4 and Chunkfs.

 

[acaeti]

Thanks everyone for all of the advice and discussion. I'm thinking a pair of RAID5's is going to be the order of the day, one with six disks, the other with five. LVM concatenation and XFS on top. and a stub of a boot/os drive.

So so long and thanks for all the fish!

 

[Brazen]

so how are you getting that many drives in a single computer? Do you have a case that can hold that many? or are you using something like an external SATA enclosure?

 

[acaeti]

I have an Antec P180 case with a 550 watt PSU. If I had known about it beforehand, I might've gone eSATA but the external enclosures sure are expensive. If they were say $100/per 5-bay enclosure, that'd be alright. But at ~$300/per 5-bay enclosure they seemed a little steep.

 

[Brazen]

Originally posted by: acaeti
I have an Antec P180 case with a 550 watt PSU. If I had known about it beforehand, I might've gone eSATA but the external enclosures sure are expensive. If they were say $100/per 5-bay enclosure, that'd be alright. But at ~$300/per 5-bay enclosure they seemed a little steep.

:shocked:wow that is a little steep. I ask because I'm thinking of doing something similar, but the only eSata enclosures I've found are only 1 or 2 bay. I would like a 10 bay rackmount enclosure, but I haven't seen one. But then I haven't looked seriously either, just glanced around Newegg and I'll probably have to look somewhere more "server room" oriented (my plan is to put a rack in the basement ).

 

[drag]

That's the problem with eSATA. Last time I looked there was only one company actually producing cables for it and they wanted a arm and a leg.

Same thing for external enclosures. All that stuff _should_ be very inexpensive, but it isn't.

Now if you want something fancy I bet you can go online and score a fiberchannel rack storage for about the same cost. Maybe be able to find 1TB storage relativeley inexpensively. To bad nothing comes close to the GB/$ that you can get from SATA drives nowadays.

If I had a couple grand to dump into a storage array what I'd do is head down to newegg and buy the biggest server case they got.

For example for 250 dollars you can get a nice Lian Li "PC-V2000Aplus II". It has 12 3.5 slots at the bottom and 7 5.25" at the top. If you stick a double-sided DVD burner in there then that leaves 6 5.25 drive bays, which you can stick 3.5" adapters in.

That leaves a total of 18 drive bays for 3.5" harddrives.

For the computer hardware inside the case I'd get a nice Intel motherboard with a cheap CPU.

for example:
http://www.newegg.com/Product/...?Item=N82E16813121060R

3 PCI-express slots. 8 onboard SATA. No onboard video so a cheap PCI card should be aquired.

Then maybe 3 http://www.newegg.com/Product/...x?Item=N82E16816103058
It's 4 ports sata controller that uses PCI-Express. That way I don't have to worry about and PCI buss limitations.


So 8 + 4 * 3 = 20 SATA ports.


Then get the cheapest core 2 duo cpu supported by that motherboard and 1gig of RAM. Also a nice 450-500 watt Seasonic power supply. Then a second Gb/s ethernet controller.

Ideally I'd setup 2 small drives in mirrored raid for the OS and boot. Then the rest in whatever array you'd like.

500GB drives seem reasonably priced. Assuming that you'd don't mind rigging up a couple dirty little mounts for the OS drives, then that still leaves me 18 ports to play around with.

About 8.5TB in JOBD.
About 7.5-8TB in RAID6 or 2 RAID 5 arrays.
Or about 4-4.5 TB in RAID 10.

Of course this is only for home storage. No hotspare drives left over, no hotswappable drive bays, no ECC ram, no half a dozen other little details that I'd need if I was doing this for a business. For home storage I don't mind downtime. If a drive goes south I'd rather just have the machine shutdown (properly) till I get a chance to look at it, rather then trying to keep running. For a 'real' setup you can expect probably 3-4x the price versus bare-bones non-critical setup. Probably 10x the cost if you throw the cost of backups into the mix (which is a hard requirement for businesses)

this essentially would turn the server into a over-glorified network disk controller. Not usefull for much else besides light-weight server duties. If I wanted to get fancy I'd use iSCSI to export LVs as disks. This way I can use the storage as if it's local to my machine with native FS and everything. Perfect for home directories or other such things.

There is good software emulation for iSCSI in Linux. Very fast, perfect for gigabit ethernet. For the iSCSI target (the server half) there is 'iSCSI enterprise target', and the Linux kernel has built-in iSCSI initiator (the client half) through the Open-iSCSI project.

If I wanted to get REALY fancy then I'd use the in-kernel support for Oracle's OCFSv2 for cluster storage file system so that multiple computers can access the same iSCSI shares at the same time. The difference between a 'clustering' file system and a 'regular' file system is that there is network communication between machines for syncronizing file system access and file locking mechanisms. This way you avoid file system corruption.

Perfect for any sort of high performance shared storage. Home directories, sharing programs through /usr/local/ mounts, media editing, etc.

Bonding 2 ethernet controllers from the file server to a switch, and with a network that supports 'Jumbo Frames' (all controllers and the switch must support large MTU sizes) you can probably get a 30-40% boost in file access performance over just plain 1000Mb/s ethernet.

All sorts of fun stuff.