System for Massive Amounts of Encoding

[Trommsdorff]

I'm really out of the loop on hardware issues - it's hard for me to keep up when I'm not looking to buy.

I have to encode massive amounts of footage to WMV format - we're talking thousands of hours. I obviously need lots of horsepower. So, my considerations:

1) Number of processors?
2) Type of processors?
3) IDE, SCSI, RAID? I need at least 200 gigs of usable hard drive space at a time.
4) OS ?

Note that I don't have to CAPTURE the footage, just encode it.

Intel has their hyperthread processors out, is there any benefit to getting 2 or more of them?

I'd like to spend under $5000. Any suggestions ?

 

[DurocShark]

Raid SATA drives, 2 new P4's with hyperthreading, buttloads of ram, Win2k would probably be your most stable option.

 

[Trommsdorff]

Thanks for the reply DurocShark.

How is the RAID that's built into the newer motherboards (like Asus, my favorite) ? This would be my first RAID venture.

Dual Hyperthreading CPUs are better than Dual Xeons?

 

[Pariah]

The software you use will dictate how many CPU's would be necessary. If the encoding program is not SMP aware and can only have a single instance running at a time, any more than 1 CPU would be pointless. Get the fastest P4 available. Storage speed is irrelevant, as the CPU is the limiting factor in encoding. Any single 200GB+ ATA drive will work fine.

If you spend more than $2000, you're throwing money away on frivilous things that won't improve performance.

 

[thorin]

Originally posted by: Pariah
The software you use will dictate how many CPU's would be necessary. If the encoding program is not SMP aware and can only have a single instance running at a time, any more than 1 CPU would be pointless. Get the fastest P4 available. Storage speed is irrelevant, as the CPU is the limiting factor in encoding. Any single 200GB+ ATA drive will work fine.

If you spend more than $2000, you're throwing money away on frivilous things that won't improve performance.

I'll second all of that and add 'How are you getting the video onto/to the system for encoding?

Thorin

 

[dguy6789]

I recomend getting a system like this:

Intel Pentium 4 3.00GHz with 800 FSB
512 PC3200 DDR Memory
512 PC3200 DDR Memory
512 PC3200 DDR Memory
512 PC3200 DDR Memory(2 gigs of dual channel will work great)
AOPEN AX4C Max Mainboard link to it is here http://www.aopen.com/products/mb/AX4CMax.htm
get a couple 250Gb western digital 7200 RPM hard drives, if you have 2, raid them.
unless you play heavy games, get a cheap video card with good iumage quality, like a radeon 9000 or something
some case
and whatever else you might want to ad, this system would encode your stuff, record video, burn cds, and play ut2k3 all at the same time

 

[DaveSimmons]

RAID 0 striping is pontless for WMV encoding -- it's raw capturing that needs that kind of speed. RAID 1 mirroring is a good idea if you won't be backing up after each new bit of encoding but you shouldn't count on it as your only backup. There is a good RAID tutorial on the main AT website under Frequently Asked Questions. SATA is currently overpriced vs. regular IDE, especially if you want 200 MB of storage (or 400 mirrored)

A DVD burner for backups is a good idea. DVD-R media is cheaper than +R. Search "DVD" for threads on "+" vs. "-"

If your encoder doesn't support multiple processors you might want to buy -2- systems and a $30 KVM switch. With file sharing only one system needs the DVD burner, and you can put a smaller hard drive and $10 graphics card into box #2. Don't use integrated graphics as it steals memory and CPU cycles from the encoder.

Depending on the encoder, an XP 3200+ or P4 3.0C might be faster -- try to find benchmarks. Notice that the 3.0C (800 FSB) is up to 10% faster than the old 3.0B so adjust timing if you only findi an old 3.0B benchmark.

Actually, as an alternative to RAID 1, you could put 200 - 250 GB drives into both boxes and copy finished files from each to the other so each acts as the backup for the other.

box 1
P4 3.0C 800FSB -or- XP 3200+
512 MB or 1 GB of DDR 400
-x- graphics card
DVD burner
20 - 40 GB hard drive - operating system
200 - 250 GB hard drive - storage -or- 2 drives and IDE RAID 1 controller or backup by hand

box 2 - same except
- cheap graphics card
- no DVD burner (or might get one for faster overall backups),
- possibly smaller storage drive or a single combined OS / storage drive if using box 1 for backup

 

[Budman]

Originally posted by: DurocShark
Raid SATA drives, 2 new P4's with hyperthreading, buttloads of ram, Win2k would probably be your most stable option.

For Hyperthreading you'll need Windows XP win2k doesnt fully support HT.

 

[zephyrprime]

Xeons are hyperthreading now. They were the first processors to have hyperthreading. Even earlier than the regular P4's.

If you have thousands of hours of video, you may need 1 month or several months of encoding to get done. I wonder if you might be best off by buying 8 or more barebone cheap computers and then having one bigger server and link everything up with gigabit ethernet?
Is this for work? If it is, you may be able to use your office computers at night. This could be the fastest and cheapest route.

You should definitely learn how long it will take your encoder to encode each hour of footage you have. You shouldn't be flying blind for an operation like this.

 

[Trommsdorff]

Thanks for all the replies. To answer the questions:

The footage has already been captured (or whatever you do) from the betacam source. It's huge. I'm getting it on hard drives or else via FTP.

Since I'm making WMV files, I'm going to be using windows media encoder. All I've tested it on now is my main desktop, which is an 800 MHz AMD Athlon, 1 GB ram. Takes about 30 mins per minute for the settings I use. Nice huh?

Does Windows Media Encoder take advantage of multiple processors? Good question, I don't know the answer.

I'll definately get a DVD burner for this system - Currently i have a Sony DRU500Ax, and aside from not burning at 4.0 when I want it to, it does well.

I'm not really interested in setting up several computers (more than 2 more) - this is for a business (mine) and will be run in my house. Wifey gave me one room to "do my thing" in and that's all I've got.

It will definately take several months of encoding to get caught up, so I want something as fast as possible.

It seems like there should be some benchmark tests for encoding, I seem to remember those tests in previous anandtech and Sharky articles.

 

[TROGDORdBURNINATOR]

Storage speed is irrelevant

That's ridiculous and false. I capture using my AIW from analgue sources all the time. Then
just try to open it and automatically detect scenes or splice the footage up. It's entirely hard drive based. The CPU is not taxed and there is lots of ram left. The hard drive thrashes like crazy though.
For video editing a raid 0 setup is a MUST for good performance. Most raid cards have a stipe size preset of "video editing" to give you the best performance in this task (it's a very large stripe size.)

 

[DaveSimmons]

for a home office, you might want to go as quiet as possible for the 1 - 2 extra boxes, for example:
- evercase small midtower
- Antec True or Seasonic power supply (300w should be fine, 350w if you must)
- Albatron or Asus i875 mobo + P4 retail 2.8C or 3.0C
- single Seagate Baracuda V 60 GB - 120GB drive
- cheap ATI or geforce card

as the extra box(es) encode, back the results up to your main box where you have the 200+ GB of space (single, double or double as RAID 1).

 

[cow123]

i believe raid 0 does nothing for write performance, so maybe 2 drives each on its own channel would be faster - read the raw data off of one hdd/encode onto other

 

[0roo0roo]

Get the fastest P4 available. dual if you plan to use system during encodes. rest of system can be crap

 

[MercenaryForHire]

Originally posted by: cow123
i believe raid 0 does nothing for write performance, so maybe 2 drives each on its own channel would be faster - read the raw data off of one hdd/encode onto other

BAHAHAHAHAHROTFLMAO N000000000000B!

/deep breath

TrogDor has an excellent point - reading the huge video files will tax the storage system like mad.

And Off-Topic - why are you using WMV? Aren't you aware that it sucks?

- M4H

 

[sharkeeper]

i believe raid 0 does nothing for write performance, so maybe 2 drives each on its own channel would be faster - read the raw data off of one hdd/encode onto other

I suppose Seagate and LSi logic have been cheating like nVidia too! ATTO

-DAK-

 

[DaveSimmons]

Originally posted by: MercenaryForHire

Originally posted by: cow123
i believe raid 0 does nothing for write performance, so maybe 2 drives each on its own channel would be faster - read the raw data off of one hdd/encode onto other

BAHAHAHAHAHROTFLMAO N000000000000B!

/deep breath

TrogDor has an excellent point - reading the huge video files will tax the storage system like mad.

And Off-Topic - why are you using WMV? Aren't you aware that it sucks?

- M4H

For doing a lot of editing a RAID0 setup makes sense. For encoding of the final version it is no help at all. The editing step (as TrogDor says) is storage-dependent, but for WMV encoding it is CPU-dependent.

If a lot of editing is needed then the "main" system probably should have RAID0. The encoder box or boxes don't need it.

 

[Trommsdorff]

Ahh, I used to think so - but the new 9 is great, and the DRM is very secure. I went through a lot of them, including Divx pro, MPEG4, etc - I really think you get the best, or at least some of the best, compression/file size with windows media encoder. The app sucks though, that is for sure.

Originally posted by: MercenaryForHireAnd Off-Topic - why are you using WMV? Aren't you aware that it sucks?

- M4H

 

[randumb]

Windows Media Encoder has multithreading, so SMP should get some benefit. I would personally get a P4 w/ HT, because I'm not too familiar with good Xeon boards, and the Xeons are a bit pricey. Hyperthreading does help with Windows Media Encoder. According to this article: http://att.cnet.com/hardware/0-8079207-8-20688858-11.html

 

[JackBurton]

Originally posted by: TROGDORdBURNINATOR

Storage speed is irrelevant

That's ridiculous and false. I capture using my AIW from analgue sources all the time. Then
just try to open it and automatically detect scenes or splice the footage up. It's entirely hard drive based. The CPU is not taxed and there is lots of ram left. The hard drive thrashes like crazy though.
For video editing a raid 0 setup is a MUST for good performance. Most raid cards have a stipe size preset of "video editing" to give you the best performance in this task (it's a very large stripe size.)

Do you not read what he wants it for? He needs it for ENCODING. Not capturing or editing. With encoding the MOST important thing is the CPU. Since you are going to be using WMV I'd suggest a dual Xeon HT enabled setup. Spend the money on the fastest processors you can afford and get 1GB of ram with a 250GB hd and you are set. No need for RAID 0 or 2GB of ram. CPU is king when it comes to encoding. I know, I encode A LOT (but not in WMV). I just wish I had a $5000 budget for my system. My XP 1600+ needs to be upgraded REALLY bad.

 

[Dug]

Nice guide and info on WM9 HDTV encoding.
avsforum

I've watched quiet a bit of content with WM9 and can honestly say its the best I've seen.

So encoding with it may suck, but the final output in fantastic.

 

[sharkeeper]

Ram most certainly does help when you put your system under the hammer! :Q

-DAK-

 

[DurocShark]

R/W to the drives WILL make a difference when encoding because the source and output files need to move to and from the CPU as quickly as possible to keep it well fed.

RAM to help cache those files during encoding (remember that you're creating 2 images of the stream when encoding... one original and one output).

Dual procs: one to keep the OS happy and one for the software (assuming it can use the second proc... dunno what apps create WMF files) <-- Luxury anyway.

 

[TROGDORdBURNINATOR]

Do you not read what he wants it for? He needs it for ENCODING. Not capturing or editing. With encoding the MOST important thing is the CPU.

This sad forum never fails to dissapoint. How exactly do you think that he's going to get those multi-gigabyte files to and from the hard drives to be encoded? What exactly do you think is goign to happen to the output once it has been encoded? Have you tried opening a 2-hour raw capture? It takes about 20 minutes here. I guess that's all the CPU that's holding it up, I'm sure.

No one is arguing that the CPU isn't the most important thing. We're just saying that the storage medium is also very, very important since some people here have completely dismissed it. I guess maybe they've figured out a way to encode video which they haven't opened somehow.

 

[MercenaryForHire]

Originally posted by: Trommsdorff
Ahh, I used to think so - but the new 9 is great, and the DRM is very secure. I went through a lot of them, including Divx pro, MPEG4, etc - I really think you get the best, or at least some of the best, compression/file size with windows media encoder. The app sucks though, that is for sure.

Bolded the reason why WMP9 sucks.

And BTW, have you tried DivX Pro 5.0.5 yet? Pack that and a Vorbis stream into an OGM, and you're golden.

- M4H