Ideas for mass personal media storage

[yh125d]

I have atm a large and expanding media library consisting mostly of uncompressed BR rips. Currently it is at 1.36TB and growing quickly. Right now I have...


2x 1TB internal HD's
1x 1TB external (esata)
1x 2TB external (esata, 2x1TB internal)

for 5TB total. One 1tb drive is for os/scratch and is separate from the media library completely. Since my collection is rather large and has taken a year to accumulate, I want a full backup at all times. The way it is setup now I have the main "backup" folder on the 2TB drive, with the copy split across my 1TB external and my extra 1tb internal drive. This setup is... less than ideal so I'm thinking about upgrading/streamlining my setup and could use some ideas, storage is not my great forte


Requirements:

I'd want at least 3TB of storage with up to 3TB backup space.
Don't need hot-swap, nor is high performance required.
Compact and quiet is nice, but if something big or loud gets the job done better, I'll do it

Ideas:

Something akin to http://www.newegg.com/Product/...x?Item=N82E16817998043
Pros: up to 4TB storage, seems simple, not too expensive
Cons: Supposedly some only support up to 750gb or 1TB drives (I would like to have support for up to 2TB drives for the future), supposedly sort of unreliable/unrefined, loud fan, takes multiple SATA ports which for me are limited. Seems clunky and bothersome, don't know if I want to have all that bogging down my main gaming PC


---------

Scavenge the drives from my external and put 2TB drives in (if possible)

Pros: I reuse the external enclosure I have, and gain two 1TB internal drives
Cons: Don't even know if it can be done, 2TB drives still expensive


---------

Get a multibay NAS and fill with drives

Pros: Simple
Cons: Empty enclosures are pretty expensive, for like a 4 bay setup

---------

Build a file server box and fill with all the 1TB drives I can find

Pros: Simple enough, I have some spare parts I could use in it to make cheaper. Case, motherboard, ram
Cons: Big ol computer case sitting somewhere drawing power, doesn't seem like a very efficient solution.



So, any ideas/thoughts/criticisms/flaming?

 

[coolVariable]

Sorry, but why would you not encode with h.264 into mkv?

 

[yh125d]

Because that's no fun


No I'm sure it would lower my storage demands a lot, but I don't want to risk any quality loss whatsoever. I know it's probably stupid and I wouldn't notice the difference anyhow, but I want the highest fidelity I can manage. Plus I can't imagine how long it would take for me to encode over 1TB. But that's not why I made the thread

 

[coolVariable]

It's about 2 hours to encode a 2-3 hour movie with h.264 on my (rather old) HTPC.
2TB of BR movies is 40+ films (2TB / 50GB = 40+)
(BTW - That's not exactly a huge library to be honest)
If you just encode over night you should be done in 2 weeks.
And your storage needs will be cut by 2/3 or more.

Use a hi-qual setting and there is NO WAY anyone can ever tell the difference.
To the contrary, you might be able to improve quality with some filters.
I personally also like adding additional re-mixed audio tracks (original AC3/DTS, AAC Dolby Pro Logic II or Dolby Srnd, and mp3 Stereo).

 

[elconejito]

My $0.02 is to build your own fileserver. You won't run into limitations with ready-built NAS not supporting drives over X TBs, or limited # of drives. It does require more work and setup on your part, but it's fully customizable. If you're careful with part selection you can keep the power draw down, or you can just turn it on when you need it.

I built my own fileserver to hold media for the whole house, backup all the computers, and is a test web server for my freelance stuff. All the media is either compressed DVD (no BR or HDDVD) or MP3 so it's not a ton of storage space. I simply installed a handful of 500GB drives, installed linux, spent a few weeks cursing while doing trial & error on the command line. But now it works just peachy

 

[fleshconsumed]

Like elconejito I just built fileserver. Got myself adaptec RAID card and setup 8 drives in RAID5. That way I don't have to worry about backups, and the card supports online RAID expansion/migration so if I ever want to switch to RAID6 or add more diskspace, I can easily do so. I used WD greet drives with quadcore and the system typically draws around 100W in idle, not too much IMO, and if you do not need performance you can always use something like e7200/7300 to lower system power consumption by another 10-20W.

I installed Windows Server on it, but for your purposes you can install any OS that has drivers for your card and that supports guid partitioning so that you can create partitions greater than 2TB. I believe Vista supports it, but I'm not sure. If you do not want to buy hardware RAID card both windows and linux should be able to create software RAID, although it will be slow and I've never done it myself.

Biggest problem is noise from cooling fans. Adaptec 31605 which I'm using runs hot, so I have separate fan cooling just the card itself and use medium speed case fans just to be safe. Still it's not too bad.

 

[Winterpool]

When I started reading the OP, I simply presumed his storage was already on a file server, given the quantities involved, and it took me a few moments before I realised his drives are in his main PC.

File server is the obvious choice. How to build and configure is the interesting question.

 

[Winterpool]

Owing to poverty, stupidity, and sloth, I eventually let myself accumulate data across five or six ad hoc 'file servers' during the past few years. Right now, I'm trying to consolidate all these files onto one or two more modern servers, particularly the Hyper-V server I built last summer.

Have been strongly considering getting a multi-bay drive rack (eg Icy Dock) to hold (and keep somewhat ventilated) three or four TB drives in the 5.25-inch bays of a computer case. This would be much cheaper than, say, a Thecus NAS box but still provide some swapping capability.

Am still uncertain whether RAID (probably 5) is advisable in my situation, especially since (all together now) RAID isn't a backup solution. Anyone here used Linux 'software' RAID (mdadm)? If a box's cpu isn't going to be doing much other than serving files, it doesn't seem like the cpu usage of software RAID would be too problematic. But the ancient conventional wisdom was that software RAID is not a good idea.

Also considering Windows Home Server: the user interface is intriguing, but I'm a little wary of the drive pool (glorified JBOD?).

 

[VirtualLarry]

Check ME out

 

[StraightPipe]

I just stacked 4TB drives into my mdeia rig, and threw them onto a RAID controller card in RAID5.

I've also been looking at this box for work. It's an 8 drive enclousre, runs on e-sata:

http://www.newegg.com/Product/...51&Tpk=n82e16816111051

 

[elconejito]

A couple points people have brought up...

I use software RAID5 with Ubuntu server edition. No problems, easy setup. I found a couple guides online that walked me through it the first time, which was a little daunting since the commands are nothing like Windows. I ended up reinstalling before I went "live" and then I helped somebody else setup a little RAID1 server, and now it's like no big deal. I initially setup the disks when doing the install, and then used mdadm after installation to check up on the RAID. I haven't done anything like add or drop drives, or setup a RAID once linux was installed with mdadm though.

My write sustained write speeds with the software RAID vary from 20-40mb/s (according to Vista's speed info thingy when you're copying files) and reads are about 70-80mb/s. I don't know how much improvement I would get with a hardware RAID card, but for me that's plenty of performance for what I use it for which is unattended backups at night, and streaming media to a max of 4 computers in the house.

Windows Home Server is a good choice and makes things *really* easy, especially backups. Downside is it costs money. It does offer some redundancy in folder (drive?) duplication, but I don't think it's RAID.

If you have a well ventilated case noise shouldn't be too big of a concern.

I agree 100% that RAID is definitely NOT a backup. The data is on each computer, backed up to the fileserver, and also backed up to offsite storage (jungledisk). I just use the RAID for convenience with a little bit of redundancy.

I like an eSATA enclosure if it's just one computer in the house, but for serving multiple computers, got to go networked. I don't think the OP specified if it was one or many computers that need access to the media library.

 

[Keitero]

http://www.newegg.com/Product/...k=acer%20home%20server

Throw in 4x 2TB drives and that can get you a nice 7ish TB.

 

[yh125d]

I think I'm going to end up going home-made file server. I have some parts I oculd use to make it already

Antec 300 (will be able to use when I upgrade my case in the near future)
Silverstone 500w (when I upgrade my PSU soon)
A couple gigs of old DDR2

I'll probably pick up some cheap motherboard with 8 SATA and onboard raid and a celeron or sempron

 

[fleshconsumed]

elconejito
I get 200MB/s reads on my RAID5 array, don't know about writes, never really tested them, but definitely at least 80MB/s because that's how fast I copy files to it over network. Not bad for a bunch of 5400RPM drives.

Ironically, the reason I decided to pay a little extra for hardware RAID and put windows server on it because I didn't want to screw with linux. I play around with it occasionally in a virtual machine, and it's always a pain to work with. Cue linux lovers.....


yh125d
Onboard raid is always going to be software raid. And if you want software RAID5, then I don't think VISTA or HOME server allow it, you'd need Server 2003/2008, which can be quite expensive if you don't have technet/msdn license.

Anyway, look into SuperMicro SASLP-MV8 and SAT2-MV8 controllers if you need more SATA ports. These two cards are very popular with people who build RAID5 in linux. They are cheap, only $100 or so and have 8 SATA ports.

 

[StraightPipe]

Nope, software raid on Vista and XP works just fine, but if you are going to make your OS RAID'd you have to hit f6 when installing windows to add the RAID drivers.

Personally, i prefer hardware RAID, because it is faster, and it works independently of your mobo. So for example, if your mobo burns up, you can pull a hardware RAID card out and put it into another machine to get it back up and running.

There are software RAID cards too that are really cheap - like $30-70. They allow your to pull your arrays like a hardware card does, but they are a bit slower and usually have less features. (for example my Areca 1220 controller allows alot of advanced features, like taking a RAID5 set and transforming it to raid6)

I'm using RAID0 for my OS and apps, and RAID5 for my storage (pics, music, movies, game images, docs, etc)

 

[StraightPipe]

What you need to do first is decidde how you want to deal with your storage array.

NAS is a great option if you have multiple PC's in the network that all need access

Internal Array is great option too, if you have room in your case and a PSU that can handle the extra drain. You can always share your drives to the network, so if the box is on, the share is online.

External Array (DAS) (like the enclosure I linked to above) is great if you dont have the room in your box, or you want to be able to swap them around (we use these at work to store backups on, then swap them offsite weekly).

 

[StraightPipe]

Internal is cheapest, if you have the space, and if your mobo supports RAID.

 

[yh125d]

Internal might work if I upgrade to a full tower... I'm just going to contemplate for awhile


Thanks for the thoughts/suggestions all

 

[yh125d]

Would anyone know if I would be able to upgrade the HDs in my external to 2TB's for 4TB total in the future? I have http://www.newegg.com/Product/...x?Item=N82E16822101091

 

[Mango1970]

I hope I am not derailing you but with all these BR rips, are you then piping this out to a HDTV? I've gone every route that has been mentioned, from using my main PC, then built a fileserver, then a bloody HTPC and now I have a NAS. At the end of the day since my most important factor was being able to watch all these backed up movies on my living room HDTV, I not only had to figure out a storage solution but something that can play all these freaking movies (codecs, hardware decoding etc.).

I love the NAS approach but if you go cheap like I did, it's slow and has lots of limitations.

Building a very cool and quiet and efficient server turned out to be the best for my needs.
BTW how did you rip your BR movies? I mean what app did you use?

 

[yh125d]

I use AnyDVD HD to rip to video file then I steal the .m2ts file and rename it, deleting the rest. For normal DVDs I rip to image then use daemon tools to mount it then watch through PowerDVD or MPC. This is my drive http://www.newegg.com/Product/...x?Item=N82E16827136133


Right now I watch movies through my 1080p 24", but will be buying a nice big 1080p probably on black friday if I can wait that long


Oh and I only need file access to one computer, I plan on running video through my server to the main PC to be sent out to my receiver then TV

 

[alrightgame]

Out of curiosity, has anyone ever considered using a online file backup services like Carbonite or MozyHome Online Backup?
I'm curious about this, although my trust is lacking.
What would keep them from looking at the information, even with encryption?
They have unlimited storage plans for less then 100 dollars a year which is about the amount it would cost to buy a harddrive each year.

Cons
Access requires using your internet service that may or may not go above your services maximum data transfer rate (usually 5-10gigs capped a day).
Not nasdaq proof. If the company went bankrupt tomorrow, then you may never see your information again.
Security, who knows who can look at this stuff? I for one thing like my bootlegs that can't be found easily anymore.

Pros
Unlimited storage apparently, which I doubt, and could fall under a Con if they try to screw you.
Fireproof... you won't ever have to worry about natural disasters.
Cheaper.

 

[elconejito]

online backups are great for backups. The OP wanted something he can easily access on his computer. Trying to stream a multi-GB BR rip would be just about as awful a thing as I can think of.

Your cons are all valid (10GB caps? really? where is this? UK?). They let you encrypt the data locally before uploading so it is "supposed" to be all bits to them. However, if the FBI thinks something is up, I'm pretty sure they can get access to those servers and your data so putting something up illegal is probably ill-advised. Transfer speed is slow though.

The pros are valid.

 

[Knavish]

Originally posted by: alrightgame
Out of curiosity, has anyone ever considered using a online file backup services like Carbonite or MozyHome Online Backup?
I'm curious about this, although my trust is lacking.
What would keep them from looking at the information, even with encryption?
They have unlimited storage plans for less then 100 dollars a year which is about the amount it would cost to buy a harddrive each year.

One more con...

if you're on a cable modem with 384kbps upload limit, then it'll take you 2.5 days to upload 10GB with *no* interruption or slowdown. Uploading 5TB of movies would take you 3.5 YEARS of non-stop uploading ... lol

 

[Circlenaut]

Here's my setup:

http://forums.anandtech.com/me...=y&keyword1=fileserver

Though I ended up going with Windows Storage Server 2008 recently. Unfortunately you can't buy this OS, but you can "try" it with a Technet plus subscription

I also threw in an 8-bay Rosewill eSATA enclosure for backup of the server.